Measuring Tokenless Savings

Tokenless records payload size and estimated tokens before and after processing. It answers “how much a compression candidate shrank,” not “how much the model request or bill decreased.”

The database and CLI call the size fields “characters,” but the current writers store UTF-8 byte length. Stored token counts use an approximate ceil(bytes / 4) heuristic; they do not call a model tokenizer. Treat both as comparison metrics.

Understand the measurement scope

Tokenless can measure:

• Schema size before and after compression.
• Tool/API response size before and after compression.
• TOON size before and after encoding.
• RTK output size before and after filtering, when a rewritten RTK command actually runs.
• Active versus dry-run mode.
• Session, agent, and tool-use identifiers.

Tokenless cannot directly measure:

• Tokens generated by the model.
• The system prompt or conversation history that bypasses Tokenless.
• Final provider billing tokens.
• Whether compression changed task quality.
• Whether an additive adapter removed the original result from the final model request.

A rollout should compare both statistics and task-result quality.

View the cumulative summary

tokenless stats summary

The current text output has this structure:

Tokenless Statistics Summary
============================================================
Total Records: ...

Character Savings:
  Before: ...
  After:  ...
  Saved:  ...

Token Savings:
  Before: ...
  After:  ...
  Saved:  ...

Breakdown by Operation:
----------------------------------------
  compress-response: ...

Character Savings and Chars in this output are the byte-based compatibility labels described above.

For machine-readable output:

tokenless stats summary --json

The summary reads at most the latest 10,000 records by default. Limit the query with:

tokenless stats summary --limit 1000

--limit must be a positive integer. --limit 0 is rejected at parse time with a non-zero exit, matching stats diff --limit.

Saving-rate field definitions

Tokenless saving rates follow the “saved amount ÷ original uncompressed amount” definition; the fields differ only in aggregation scope and negative-value handling. Every percentage field emitted by tokenless stats is defined as follows:

Field	Where	Formula	Negative values
chars_saved_percent	stats summary --json (total and per operation)	(before_chars − after_chars) ÷ before_chars × 100%	Clamped: the saved amount never goes below 0
tokens_saved_percent	stats summary --json (total and per operation)	(before_tokens − after_tokens) ÷ before_tokens × 100%	Clamped: the saved amount never goes below 0
saved_percent	stats summary --compare --json	(baseline_tokens − tokenless_tokens) ÷ baseline_tokens × 100%	Clamped: the saved amount never goes below 0
saved_percent	stats diff --json (every chain and stage)	(before_tokens − after_tokens) ÷ before_tokens × 100%	Kept: a chain or stage that grows reports a negative percentage

Every field returns 0% when its denominator is 0.

saved_percent appears in two schemas with the same basic formula but different scopes and sign handling: the --compare value is computed from the two runs' totals (baseline_tokens and tokenless_tokens) and clamps a token increase to 0%, while stats diff --json reports one saved_percent for every chain and every stage, each computed from that object's own before_tokens and after_tokens and allowed to go negative. Worked example: with before=100 and after=150 tokens, --compare reports 0% while stats diff reports -50%.

Saved: N tokens (X%) in the text output corresponds to tokens_saved_percent: the denominator is the sum of before_tokens over the same records — the original, uncompressed size — not the session's total consumption and not any provider-side cache metric.

tokenless-stats does not emit savings_rate, cached_tokens, or total_cached_tokens fields, and it does not collect model-provider prompt-cache hit data. If another tool's report shows a savings_rate computed as cached_tokens ÷ total_tokens, that number describes the provider-side prompt-cache hit share; it does not represent Tokenless compression savings and is not produced by tokenless-stats.

Inspect individual records

List recent records:

tokenless stats list
tokenless stats list --limit 50

[ID:<n>] in the output is the record ID. Show the complete text before and after one operation:

tokenless stats show <record-id>

Explain the estimated saving and changed lines for that record:

tokenless stats diff <record-id>
tokenless stats diff <record-id> -U 5
tokenless stats diff <record-id> --json

When both endpoints are valid JSON, diff sorts object keys before comparison, so key-order-only changes are hidden in the display; it does not modify stored content. Use stats show when you need the verbatim payload or when the diff reports missing or oversized content.

Analyze end-to-end stages within one session:

tokenless stats diff --session <session-id>
tokenless stats diff --session <session-id> --sort time
tokenless stats diff --session <session-id> \
  --tool-use-id <tool-use-id>

The session overview contains metrics only. A tool-use report includes content diffs and links consecutive active stages only when their session/tool-use IDs match and the previous stored output exactly equals the next stored input. Disconnected stages, dry-run rows, and rows without a tool-use ID remain separate, preventing intermediate inputs from being counted twice.

For dry-run rows, after is the predicted compressed size while emitted remains the original before size. Operations with no estimated saving are not stored, so session reports cover saving records only.

Local statistics contain complete tool text. Do not paste stats show output into public issues, shared logs, or untrusted chats. See Configuration and data privacy.

Why no record appears

No statistics record is added when:

• The estimated token count did not decrease.
• stats_enabled=false or TOKENLESS_STATS_ENABLED=0.
• The adapter is not enabled or the old agent session was not restarted.
• The hook or plugin cannot find tokenless.
• The input did not pass through a supported Tokenless hook.

An additive adapter can create a record even though the host also retains the original result. Codex avoids this ambiguity: its PostToolUse hook does not run compression or record a response candidate because the host cannot replace the original output. Measure Codex savings from RTK rewrite records instead.

Start with:

tokenless stats status
anolisa adapter status tokenless

Then see No statistics appear after enabling the adapter.

Run the repository reference workload

The source tree includes deterministic fixtures for comparing compressor behavior across Tokenless revisions. On Linux, run the following command from src/tokenless/benchmark/l1-compressor in an ANOLISA source checkout:

cargo run --release --bin compression_rate -- --json

The report uses the committed src/tokenless/benchmark/l1-compressor/fixtures/tool_response.json and src/tokenless/benchmark/l1-compressor/fixtures/schema_search.json with the checked-out compressor defaults. The fixtures are generated by python/gen_fixtures.py without randomness and are byte-for-byte reproducible. Tokenless 0.8.2 at commit a30575361 produces this reference snapshot:

JSON field	Isolated stage and input	Saving
canonical.response.savings_pct	Response compression on the canonical response	36.3%
canonical.schema.savings_pct	Schema compression on the canonical schema	47.3%
canonical.response.toon_only_savings_pct	TOON encoding of the uncompressed canonical response	17.0%
canonical.schema.toon_only_savings_pct	TOON encoding of the uncompressed canonical schema	-2.3%

The same report measures mixed-workload stacking configurations over both canonical fixtures (stacking.configs), against their combined baseline of 5,551 estimated tokens — so single-track rows read lower than the isolated rates above:

Configuration	What runs	Saving
response_only	Response compression	34.0%
schema_only	Schema compression	3.0%
schema_response	Schema + response stacked	37.0%
response_toon	Response compression + TOON	47.4%
toon_only	TOON encoding of the raw payloads	15.8%
full_stack	Schema + response + TOON	50.3%

The TOON stacking rows are measured ungated (the benchmark encodes unconditionally); the deployed runtime adopts TOON only when it reduces the estimated token count, so deployed results can differ slightly from those rows. A negative isolated TOON result means that encoding made this input larger. In active mode, the runtime emits the original JSON whenever a candidate does not reduce the estimated token count. Snapshot numbers belong to the exact commit they were measured on; compression rates evolve across versions, so re-run the report after upgrading and cite the commit or version when quoting numbers.

For the full quality and adversarial suite plus this report (skips the criterion performance benches; takes a few minutes), run ./run-benchmarks.sh --quick from the same directory.

This is a regression workload, not a promised production range. The response fixture is synthetic and compression-friendly, the suite uses one response and one schema, and token counts use the approximate ceil(bytes / 4) rule rather than a model tokenizer. It also excludes adapter behavior and the share of tool data in a complete session. Use the snapshot only to check that the same source revision behaves comparably; use your own representative payloads and the dry-run comparison below to evaluate an actual workload. See the benchmark methodology and limitations for details.

Run a dry-run comparison

Dry-run computes the compressed result and predicted savings but returns the original to its caller. A minimal reproducible comparison for the same input is:

TOKENLESS_COMPRESSION_ENABLED=0 \
  tokenless compress-response -f response.json \
  --session-id baseline-run

TOKENLESS_COMPRESSION_ENABLED=1 \
  tokenless compress-response -f response.json \
  --session-id active-run

tokenless stats summary --compare baseline-run active-run

For machine-readable output:

tokenless stats summary \
  --compare baseline-run active-run \
  --json

Notes:

• --compare requires exactly two session IDs in baseline, active order.
• If either session has no records, the command exits with an error instead of reporting 0% savings.
• --limit must be a positive integer. --limit 0 is rejected at parse time instead of looking like a missing session.
• The baseline should be a dry-run and the active session should apply compression. The CLI warns on a mode mismatch.
• For real agent tasks, keep inputs, tool versions, and the environment as similar as possible.
• Dry-run still writes the complete before/after text to the local statistics database.
• Dry-run does not create Stash entries and does not disable RTK rewriting. RTK-written rows have no explicit mode and are read as active, so they can trigger a baseline mode warning.

Interpret the saving rate correctly

See Saving-rate field definitions for the definition of each percentage field. The compression rate in stats summary covers only payloads handled by Tokenless. Estimate the whole-session effect with:

Estimated overall saving rate
= Tokenless payload compression rate × tool-payload share of session tokens

For example, a 60% payload compression rate with tool payloads representing 20% of the session gives an estimated overall saving of about 12%. This is still not a provider billing guarantee.

Where the compression rate applies

The compression rate depends on how much removable material a payload contains, and varies widely by scenario:

• High savings: tools returning many uniform records (lists, tables, search results), payloads carrying redundant fields such as debug/trace/logs, or schemas with verbose descriptions.
• Moderate savings: shell output is truncated only beyond the Layer 2 thresholds (65,536-character strings, a 128-item array head window, depth 8); below them, lossless cleanup and record reduction (object arrays of at least 33 records) are what change the payload.
• Near-zero savings: responses shorter than the 200-character minimum gate; already-compact JSON without redundancy; any input that does not shrink (the size guard keeps the original).
• Not compressed: content-retrieval tool output (Read/Glob/Grep and the like, except the narrow native-Grep path-sharing case) and file-content results. Build/test logs, CSV/TSV tables, and supported API search listings do have their own compressors; Git diffs pass through unless the opt-in TOKENLESS_DIFF_COMPRESSION_ENABLED switch enables context cropping; other plain text, stack traces, HTML, and source code pass through unchanged for now.

Reference numbers always belong to the exact commit they were measured on — see Run the repository reference workload for the reproducible load, the current snapshot, and its limits. Real session savings must additionally be multiplied by the share of tool payloads in total session tokens; see Interpret the saving rate correctly. For the full trigger rules, see User manual · Compression trigger conditions and thresholds.

Local AgentSight display

AgentSight's Token savings view can aggregate ~/.tokenless/stats.db read-only. When both run as the same user and AgentSight can access that database, SLS is not required to display local Tokenless statistics.

Check access with:

test -r ~/.tokenless/stats.db

See the AgentSight user guide for installation and dashboard use.

SLS JSONL

SLS is a separate external ingestion path. It is not a prerequisite for AgentSight to read local statistics.

Default behavior:

• sls_enabled=true.
• The default target is /var/log/anolisa/sls/ops/tokenless.jsonl.
• Tokenless appends only when the target file already exists; otherwise it skips the write.
• ANOLISA SLS/Logtail infrastructure creates, rotates, and removes the file.
• SLS records contain metrics and identifiers, never the original before/after text.
• When the agent host or adapter injects a W3C traceparent into the environment Tokenless runs in, records also carry tokenless.trace_id and tokenless.span_id so an observability backend such as AgentLoop can attribute the savings to the trace that produced them. Nothing sets that variable automatically — OpenTelemetry keeps the active span in an in-process carrier — so both keys are omitted unless the launcher injected a usable context.
• The bundled RTK statistics writer records rewrite-command rows only in local SQLite; it does not call the SLS writer.

Use a custom test file:

touch /tmp/tokenless-sls.jsonl
TOKENLESS_SLS_ENABLED=1 \
TOKENLESS_SLS_PATH=/tmp/tokenless-sls.jsonl \
  tokenless compress-response -f response.json

tail -n 1 /tmp/tokenless-sls.jsonl | jq .

Correlate one run with the trace the launcher is in. The example sets the variable inline, which is exactly what a host or adapter has to do before spawning Tokenless:

touch /tmp/tokenless-sls.jsonl
TRACEPARENT=00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01 \
TOKENLESS_SLS_ENABLED=1 \
TOKENLESS_SLS_PATH=/tmp/tokenless-sls.jsonl \
  tokenless compress-response -f response.json

tail -n 1 /tmp/tokenless-sls.jsonl | jq '."tokenless.trace_id", ."tokenless.span_id"'

TOKENLESS_SLS_PATH must be under /var/log/ or /tmp/. Production SLS endpoint, authentication, and Logtail configuration belong to platform operations and are outside this guide.

Clear statistics

First confirm that historical comparisons are no longer needed:

tokenless stats clear --yes

This clears records but does not disable future recording. Stop new local records with:

tokenless stats disable

stats disable turns off only local SQLite statistics, not SLS. See Configuration and data privacy for the complete toggle behavior.