RTK Has 80,000 Stars and a Counter That Cannot See Its Own Cost

What the Rust Token Killer actually compresses, why two independent paired benchmarks found the bill going up, and the category of measurement error that no disclosure can fix

RTK, the Rust Token Killer, sits between a coding agent and the shell. The agent runs git status, RTK runs it for real, and hands back * master / M a.txt / ?? b.txt instead of eleven lines of porcelain. It is a good piece of engineering, the compression is often elegant, and it has roughly 80,000 GitHub stars.

It also ships a counter, rtk gain, that reports how much it saved you. In one independent benchmark that counter read 349.2 million tokens saved across 445 agent attempts, an 89% reduction, while the measured bill for those same attempts went up 17% per task. In a second, run two months earlier by a different team, the counter read 96.2 million tokens saved, 99.8% of everything it touched, while the invoice rose 7.6%.

The interesting part is not that a number was wrong. It is that RTK's own documentation explains, precisely and in advance, why that number was never a cost figure, and the ecosystem repeated it anyway.

What RTK actually compresses

The dilution chain is spelled out on RTK's own docs page. RTK changes exactly one thing: the bytes a shell command sends back. Those bytes are one contributor to input tokens, alongside your prompt, the system prompt and conversation history. Input tokens are one part of the bill, which also counts output tokens. The doc states the consequence in a sentence: "A command showing 90% fewer output bytes does not make your session 90% cheaper."

The docs go further and explain the estimator. rtk gain computes tokens as bytes / 4. RTK ships no real tokenizer by design, because embedding one costs startup time and would need a tokenizer per model. The page is unusually candid about what that means: the percentage is reliable because the same estimator is applied to raw and filtered output, but "the absolute token counts are approximate," and it tells you to treat a line reading Input tokens: 45,230 as "an order of magnitude, not an invoice line."

That is a better disclosure than most commercial tools manage. Then the number went out into the world and did what big numbers do. One X post claiming RTK could cut Claude Code tokens by up to 60% reached 313,000 views. A YouTube video promised 90%. Guides sold "cut token costs 60 to 90 percent." A counter that prints eight figures travels further than a doc page that explains what the eight figures are not.

Two teams, two harnesses, the same answer

JetBrains ran it first, in July 2026, as part two of a series putting public token-saving add-ons through a paired A/B benchmark. Setup: Claude Code 2.1.201 headless, claude-sonnet-5 at low and high reasoning effort, SkillsBench across 86 tasks, Harbor 0.18, RTK v0.43.0 installed exactly as rtk init -g ships it. 425 billed trials, about $320.

They did a free ceiling analysis before spending anything, and it predicted the outcome. Claude Code reads files with its built-in Read and Grep tools, which bypass the Bash hook entirely. Half of what agents run in a shell is python3 and other uncovered commands, and a sixth uses pipes, heredocs and substitutions RTK deliberately refuses to rewrite. What remains is 33% of Bash calls carrying just under 20% of tool-result characters, and tool results are themselves a slice of billed input because context is re-read every turn. Squeeze RTK's entire share by 70% and the ceiling works out to about 3% of input tokens.

The measured result across 80 clean pairs: a median 7.6% more expensive per task at low effort (p=0.004), on 13.8% more turns (p=0.03) and 14.3% more cache reads (p=0.008). New input tokens, the only class RTK compresses, moved 3.2% with p=0.23, a flat null exactly where the ceiling analysis said the whole benefit had to live. At high effort the penalty disappeared into a median 0.1% (p=0.99). Task quality was statistically tied in both arms, 5 better and 4 worse against 71 ties at low effort. Their verdict: "Honest engineering, wrong counterfactual."

Quesma ran it again on September 11 with a different harness, different models and a different benchmark. Claude Code with Fable 5.0 and OpenCode with DeepSeek V4 Pro 0813, on Terminal-Bench 2.1, five attempts per task with RTK and five without on matched routes and timeouts. 85 Fable tasks, 89 DeepSeek tasks, 1,740 attempts, over $1,500 of tokens.

Total bill moved 5% down for Fable and 5% up for DeepSeek. Weighted per task, Fable came out 1% more expensive with a confidence interval straddling zero, DeepSeek 17% more expensive. Almost all of Fable's apparent savings traced to a single task, winning-avg-corewars, where RTK halved the turn count; across everything else it was under 1%. DeepSeek's average turn carried 7% less input under RTK, and there were 18% more turns overall.

Two independent teams, four model-harness combinations, and the same finding: the compression is real and the savings are not.

The category error

Here is what neither the disclosure nor the honest README addresses, because no disclosure can.

rtk gain is computed at the moment a single command executes. It compares the bytes the command produced against the bytes the agent received. That comparison is complete and correct for that one command. The cost RTK creates, when it creates any, arrives later, in the turns the agent takes after reading a shortened output. A self-measuring optimizer can see its own input and its own output. It cannot see the trajectory it changed.

Both benchmarks caught this from different angles. JetBrains found that the more commands the hook rewrote, the larger the penalty: heavily exposed task pairs cost about 24% more against 5% for pairs the hook barely touched, and controlling for task difficulty did not reproduce the pattern. Quesma found DeepSeek attempts took more turns on 58 tasks, 44 of which cost more, against fewer turns on 28, 23 of which cost less.

The counterfactual failures compound it. JetBrains logged 190 giant reads averaging around 506,000 "saved" tokens each, including one cat of a 1.2 MB CSV credited with 320,000 tokens, when Claude Code truncates tool results long before that and the agent would have received a few thousand either way. Quesma found two head -1 train.txt calls each credited 120.5 million tokens by comparing a one-line read against the whole file, together accounting for 69% of the entire savings counter. And chars-divided-by-four is priced at fresh-input rates while most of a session's input arrives as cached re-reads billed at a tenth for Fable and a thirtieth for DeepSeek.

None of that is dishonesty. It is a measurement that is structurally unable to reach the quantity people want it to report.

Put this into practice

If you already run RTK, keep it and stop believing the counter. Save% is a byte ratio and it is accurate as a ratio, which makes it genuinely useful for finding which commands flood your context. Treat the token and dollar columns as diagnostics for that and nothing else.

Before adopting any context-reduction tool, run five paired attempts. Same tasks, same model route, same timeouts, once with and once without. Compare the total bill and the per-task median, and publish the confidence interval next to the headline. Both teams independently converged on this shape because single runs are worthless: JetBrains measured that identical attempts of the same task in the same arm differ by a median 22% in cost, and their k=1 smoke test showed a scary 35% penalty that evaporated into noise at k=3.

Count turns, not just tokens. In both studies the turn delta predicted the cost delta better than the token delta did. If a tool changes how many round trips your agent makes, that is the number to watch.

Check the pricing assumption that makes all of this true. Terminal output is roughly 11% of Fable's input tokens and 40% of DeepSeek's, and in cached agentic loops most of it arrives as cache reads at a small fraction of fresh-input price. If your provider does not price cache reads that far below fresh input, or your workload re-reads uncached context every turn, the arithmetic changes and compression may genuinely pay.

Ask any savings counter two questions. What unit does it count, and can it see the turns its own intervention causes? For rtk gain the answers are estimated bytes over four, and no. Most of these counters will answer the same way.

Where this analysis breaks

The benchmarks are not the last word. JetBrains tested RTK v0.43.0 against Claude Code 2.1.201; Quesma tested v0.45.0 against Claude Code 2.1.220 and OpenCode 1.18.25. RTK is at v0.49.0 as of September 11 and is shipping fast enough that dozens of dev-*-rc tags land on the same days as stable ones. At least one specific bug has already been fixed: Quesma hit a git-multibranch attempt where an unsupported find flag got rewritten into an error loop that ran 339 consecutive failures over about 12 minutes and cost roughly nine times its matching baseline, and RTK fixed that in 0.46.0 after their runs. Both benchmarks are measurements of past versions.

Both are also single benchmarks. SkillsBench and Terminal-Bench 2.1 are terminal-heavy by design, which should favor RTK, and both teams say so. Quesma deliberately stayed on Terminal-Bench 2.1 rather than 3.0 or 4.0 because agents pass most 2.1 tasks and cost only matters on tasks that pass, which is defensible and also means the result may not transfer to harder work. Neither team tested the codebases you actually run agents against.

And Quesma's own position deserves naming: the company sells tooling for understanding what coding agents do, and its blog post ends by soliciting subscribers for a planned benchmark of a different tool. The methodology is published in enough detail to check, and the JetBrains result is independent corroboration from a team with a different commercial interest, which is the reason this article treats the finding as solid rather than as a vendor claim. That reasoning is the same reasoning you should apply to it.

What to take from this

RTK is worth installing if you want to see which commands are flooding your agent's context, and worth keeping if the compression happens to fit your workload. It is not a cost-saving tool, and its maintainers did not say it was.

The thing to carry past RTK is the shape of the error. Any tool that optimizes something and then reports on its own optimization is computing a counterfactual, and a counterfactual is a guess about a world that did not happen. When that guess is cheap to compute and prints a large number, it will outrun every caveat attached to it.

Go look at whichever tool in your agent stack reports its own savings. Find the line in its docs that defines the unit. If the docs do not have that line, that is the finding.

Sources: Quesma, "RTK reports huge token savings, but our cost benchmarks disagree" · JetBrains, "Does 'rtk' skill really cut agent tokens by 60–90%?" · RTK, "How RTK Savings Work" · rtk-ai/rtk

Medium metadata

Title: RTK Has 80,000 Stars and a Counter That Cannot See Its Own Cost
Subtitle: What the Rust Token Killer actually compresses, why two independent paired benchmarks found the bill going up, and the category of measurement error that no disclosure can fix
Tags: AI Agents, Claude Code, Developer Tools, Benchmarks, Open Source
Suggested kicker: Honest engineering, wrong counterfactual
Canonical: import from the fervorai.dev URL after publish