Saving tokens: the counter says 45%, the invoice 2%

One sentence sums up the whole problem, and it comes from the tool’s own benchmark: “rtk’s scoreboard said 96 million tokens saved while the invoice went up.” The rtk counter claimed 96 million tokens saved; the invoice, meanwhile, was going up. It’s in the JetBrains post comparing token-saving tools.

My trigger, personally: a LinkedIn post selling a “token-saving method” behind a paywall, blurred summary, link in the comments. I never bought it, never read it. That’s the point: you can take the claim apart with your own invoice, without paying for the method. Attack the measurement, not the person.

The number nobody looks at

ccusage monthly, July 2026 + Aug 1-4 window: 2.18 billion tokens, $1,703.64.

What it costs

Prices checked on August 4, 2026

Cache read 2,118,939,430 tokens

96.99%

Cache write 58,188,826 tokens

2.66%

Output 7,409,081 tokens

0.34%

Fresh input 123,360 tokens

0.006%

Total $1,703.64 (35 days)

cache_read / cache_write = 36.4. Every token written into context gets re-read ~36 times. It’s the only number worth remembering, and it turns the question “how do I send less text” into “how do I stop writing into a context I’ll re-read 36 times”.

Corollary that kills the marketing: raw output weighs 0.34% of tokens. Any tool that makes the agent “write less” plays on a tiny slice. Re-checked on 08/12: ratio ~36×, cache read ~97%, stable.

Announced vs measured

A series of paired tests from JetBrains (Harbor harness / SkillsBench, published protocol and budget), the only source comparing both sides:

Tool	Announced	Independently measured
rtk	−60 to −90%	+7.6% cost (p=0.004, low effort)
caveman	−65%	−8.5%
ponytail	−54% code / −20% cost	−15.4% code / −10.3% cost (p=0.004)
headroom	−60 to −95%	real compression, invoice went up

Range to remember: every saving actually reproduced by third parties lands between 0 and 15%. Never 60-90%.

The rtk case, measured on my machine

The counter, taken on 08/12: 4.4M “saved” tokens, 45.3%, across 9,892 cumulative commands. Hook installed in early April 2026, about four months. Pro-rated to the invoice’s 35-day window: ~1.17M tokens, against 58.19M of cache writes. Real reduction of written context: ~2%, not the 45% displayed.

$ rtk gain
Total commands:    9892
Tokens saved:      4.4M (45.3%)

The three causes, which are the real technical content:

1. The hook only fires on shell commands. Read, Grep, Glob, a good share of tool results, slip past it.
2. Claude Code already truncates large files. The tool counts as saved what would never have been transmitted anyway.
3. Retries. A broken rewrite adds one turn, and one turn re-reads the whole context at the 36× factor.

Bonus: the degradation is silent. The agent doesn’t know you compressed; I’ve already had a truncated Playwright output produce an undebuggable E2E failure.

The lever nobody sells

My permanent context, the instruction files loaded on every request: 7,381 tokens (measured 08/04). Across a 36-turn session: ~266,000 cache-read tokens, about 12% of my monthly volume, for files that only describe rules.

And the biggest line item (2,784 tokens) was a MEMORY.md full of agent, skill and hook inventory tables: state documentation, derivable from disk, already stale (it claimed 151 skills and listed 16). Paid 36 times per session.

The anecdote that carries the article: I found more savings by deleting my own documentation than by installing four compression tools.

Levers ranked by ROI:

1. Permanent context (CLAUDE.md, rules, memory): paid on every turn
2. MCP tool definitions: injected into every request; native deferred loading does the job, no need to stack a third-party layer
3. Session discipline: /clear between unrelated tasks, narrow scope, subagents for throwaway exploration; that’s where the 36× is decided, and no tool will do it for you
4. Output: 0.34% of tokens, ponytail’s turf
5. Shell output: rtk’s turf, ~2%

The protocol, if you want to measure yours

1. Baseline ccusage monthly --json over two weeks: note totalCost and the cacheRead/cacheCreation ratio.
2. One change at a time. Two layers at once = unreadable result.
3. Measure the invoice, not tokens. And the success rate: −80% tokens with +50% errors is a net loss.
4. Paired windows, same kind of work. Real effects are on the order of 10%, workload variance drowns them otherwise.

The ecosystem’s intuition is right: the 36× factor proves compressing context is the highest-leverage lever that exists. It’s their measurement that’s wrong, and none of the four touches the part of context that matters. The good news is the lever doesn’t need a tool: it’s called deleting your own documentation, and closing your sessions. The rest, my guide on costs covers.