Skip to main content

Measuring Tokenless Savings

中文版

Tokenless records payload size and estimated tokens before and after processing. It answers “how much a compression candidate shrank,” not “how much the model request or bill decreased.”

The database and CLI call the size fields “characters,” but the current writers store UTF-8 byte length. Stored token counts use an approximate ceil(bytes / 4) heuristic; they do not call a model tokenizer. Treat both as comparison metrics.

Understand the measurement scope​

Tokenless can measure:

  • Schema size before and after compression.
  • Tool/API response size before and after compression.
  • TOON size before and after encoding.
  • RTK output size before and after filtering, when a rewritten RTK command actually runs.
  • Active versus dry-run mode.
  • Session, agent, and tool-use identifiers.

Tokenless cannot directly measure:

  • Tokens generated by the model.
  • The system prompt or conversation history that bypasses Tokenless.
  • Final provider billing tokens.
  • Whether compression changed task quality.
  • Whether an additive adapter removed the original result from the final model request.

A rollout should compare both statistics and task-result quality.

View the cumulative summary​

tokenless stats summary

The current text output has this structure:

Tokenless Statistics Summary
============================================================
Total Records: ...

Character Savings:
  Before: ...
  After:  ...
  Saved:  ...

Token Savings:
  Before: ...
  After:  ...
  Saved:  ...

Breakdown by Operation:
----------------------------------------
  compress-response: ...

Character Savings and Chars in this output are the byte-based compatibility labels described above.

For machine-readable output:

tokenless stats summary --json

The summary reads at most the latest 10,000 records by default. Limit the query with:

tokenless stats summary --limit 1000

Inspect individual records​

List recent records:

tokenless stats list
tokenless stats list --limit 50

[ID:<n>] in the output is the record ID. Show the complete text before and after one operation:

tokenless stats show <record-id>

Explain the estimated saving and changed lines for that record:

tokenless stats diff <record-id>
tokenless stats diff <record-id> -U 5
tokenless stats diff <record-id> --json

When both endpoints are valid JSON, diff sorts object keys before comparison, so key-order-only changes are hidden in the display; it does not modify stored content. Use stats show when you need the verbatim payload or when the diff reports missing or oversized content.

Analyze end-to-end stages within one session:

tokenless stats diff --session <session-id>
tokenless stats diff --session <session-id> --sort time
tokenless stats diff --session <session-id> \
  --tool-use-id <tool-use-id>

The session overview contains metrics only. A tool-use report includes content diffs and links consecutive active stages only when their session/tool-use IDs match and the previous stored output exactly equals the next stored input. Disconnected stages, dry-run rows, and rows without a tool-use ID remain separate, preventing intermediate inputs from being counted twice.

For dry-run rows, after is the predicted compressed size while emitted remains the original before size. Operations with no estimated saving are not stored, so session reports cover saving records only.

Local statistics contain complete tool text. Do not paste stats show output into public issues, shared logs, or untrusted chats. See Configuration and data privacy.

Why no record appears​

No statistics record is added when:

  • The estimated token count did not decrease.
  • stats_enabled=false or TOKENLESS_STATS_ENABLED=0.
  • The adapter is not enabled or the old agent session was not restarted.
  • The hook or plugin cannot find tokenless.
  • The input did not pass through a supported Tokenless hook.

An additive adapter can create a record even though the host also retains the original result. In particular, the current Codex PostToolUse hook records compression candidates but does not replace the original tool output.

Start with:

tokenless stats status
anolisa adapter status tokenless

Then see No statistics appear after enabling the adapter.

Run a dry-run comparison​

Dry-run computes the compressed result and predicted savings but returns the original to its caller. A minimal reproducible comparison for the same input is:

TOKENLESS_COMPRESSION_ENABLED=0 \
  tokenless compress-response -f response.json \
  --session-id baseline-run

TOKENLESS_COMPRESSION_ENABLED=1 \
  tokenless compress-response -f response.json \
  --session-id active-run

tokenless stats summary --compare baseline-run active-run

For machine-readable output:

tokenless stats summary \
  --compare baseline-run active-run \
  --json

Notes:

  • --compare requires exactly two session IDs in baseline, active order.
  • The baseline should be a dry-run and the active session should apply compression. The CLI warns on a mode mismatch.
  • For real agent tasks, keep inputs, tool versions, and the environment as similar as possible.
  • Dry-run still writes the complete before/after text to the local statistics database.
  • Dry-run does not create Stash entries and does not disable RTK rewriting. RTK-written rows have no explicit mode and are read as active, so they can trigger a baseline mode warning.

Interpret the saving rate correctly​

The compression rate in stats summary covers only payloads handled by Tokenless. Estimate the whole-session effect with:

Estimated overall saving rate
= Tokenless payload compression rate × tool-payload share of session tokens

For example, a 60% payload compression rate with tool payloads representing 20% of the session gives an estimated overall saving of about 12%. This is still not a provider billing guarantee.

Local AgentSight display​

AgentSight's Token savings view can aggregate ~/.tokenless/stats.db read-only. When both run as the same user and AgentSight can access that database, SLS is not required to display local Tokenless statistics.

Check access with:

test -r ~/.tokenless/stats.db

See the AgentSight user guide for installation and dashboard use.

SLS JSONL​

SLS is a separate external ingestion path. It is not a prerequisite for AgentSight to read local statistics.

Default behavior:

  • sls_enabled=true.
  • The default target is /var/log/anolisa/sls/ops/tokenless.jsonl.
  • Tokenless appends only when the target file already exists; otherwise it skips the write.
  • ANOLISA SLS/Logtail infrastructure creates, rotates, and removes the file.
  • SLS records contain metrics and identifiers, never the original before/after text.
  • The bundled RTK statistics writer records rewrite-command rows only in local SQLite; it does not call the SLS writer.

Use a custom test file:

touch /tmp/tokenless-sls.jsonl
TOKENLESS_SLS_ENABLED=1 \
TOKENLESS_SLS_PATH=/tmp/tokenless-sls.jsonl \
  tokenless compress-response -f response.json

tail -n 1 /tmp/tokenless-sls.jsonl | jq .

TOKENLESS_SLS_PATH must be under /var/log/ or /tmp/. Production SLS endpoint, authentication, and Logtail configuration belong to platform operations and are outside this guide.

Clear statistics​

First confirm that historical comparisons are no longer needed:

tokenless stats clear --yes

This clears records but does not disable future recording. Stop new local records with:

tokenless stats disable

stats disable turns off only local SQLite statistics, not SLS. See Configuration and data privacy for the complete toggle behavior.