Prompt Compression Token Savings Benchmark [2026]: RTK vs Reality

If you’re trying to decide whether to ship prompt compression into an agentic coding stack, the wrong answer is “it saves 60% tokens.” The right answer is whether it reduces end-to-end cost per successful task without tanking success rate, breaking tool calls, or blowing up latency. That’s what a real prompt compression token savings benchmark should measure.

Key takeaways

• Token reduction is a component metric. For agents, cost-per-success is the only number that survives contact with reality.
• RTK-style tool output compression can save tokens and still increase spend when it causes extra turns or lowers pass rate by even 1–2 points.
• You need to log tool-call failure modes explicitly. “The agent failed” is not a diagnosis.
• Latency is part of cost. Compression adds CPU work and can increase turns, which moves your p95.
• Benchmarks that hide harness details (sampling, retries, refusal handling) are marketing, not measurement.

September 2026 is a perfect example of why this got contentious again. RTK (Rust Token Killer) went viral, hit ~79k GitHub stars (at the time Quesma wrote about it), and people started repeating “token savings” like it was the same thing as savings on the invoice. It isn’t.

What is prompt compression?

Prompt compression is the practice of shrinking the text you send to a large language model so you pay for fewer billed tokens. Usually you do that by removing redundancy, summarizing context, or compressing tool outputs before they’re fed back into the agent loop.

There are three different things people throw into the same bucket when they say “prompt compression.” Mixing them is how you end up benchmarking the wrong lever:

1. Tool output compression (RTK-style): compress terminal output (ls, git diff, test logs) before the model reads it.
2. Prompt/context compression (LLMLingua-style): compress instructions or retrieved context to fit more signal into a smaller window.
3. Memory compression: summarize chat history or agent state so long-running sessions don’t balloon.

This post is mostly about (1), because coding agents are terminal-heavy and because RTK is the current lightning rod. But the harness ideas apply to all three.

How RTK works

RTK (Rust Token Killer) sits between your coding agent and the shell. Instead of handing the model raw tool output, RTK rewrites it into a terser representation.

Quesma’s writeup shows the basic mechanism. RTK can rewrite common shell commands (including things like ls, git, test, package, and file ops) so the agent receives a compressed output that keeps “key facts” (filenames, permissions, sizes) while dropping verbosity (owners, timestamps, etc.). That’s a reasonable trade. Terminal output is noisy.

The important part is RTK’s own disclaimer, which Quesma quotes from the README:

• RTK may cut up to 90% of the bash output your agent reads.
• That is not the same as cutting your bill by 90%.

That disclaimer is basically the whole argument. Because your bill is dominated by the entire loop: retries, extra turns, dumb mistakes, and the agent’s own scaffolding.

Testing RTK on Terminal-Bench 2.1

If you want a benchmark for terminal-heavy coding agents, Terminal-Bench is at least aiming at the right shape of problem.

Quesma tested RTK on Terminal-Bench 2.1 specifically because newer versions (3.0, 4.0) were harder. Their reasoning was pragmatic: you can’t make strong claims about “cost savings” on tasks that almost never pass. That’s not a benchmark. That’s a failure generator.

Their setup (this is the kind of detail I wish more benchmark posts would just spell out, boringly, up front):

• Routes: Claude Code with Claude Fable 5.0, and OpenCode with DeepSeek V4 Pro 0813 via OpenRouter.
• Scheduling: each task ran 5× baseline and 5× with RTK, on the same route and timeouts.
• Filtering: they removed 4 security tasks for Fable due to refusals.
• Scale: after filtering, the comparison covered 85 Fable tasks and 89 DeepSeek tasks, totaling 1,740 attempts.
• Spend: they report “several days” and “over $1,500 spent on tokens.”

That last point matters. People love to post a “benchmark” with n=20 runs and call it science. It’s not.

Also, refusal handling is not optional anymore. Real models have policy constraints. Anthropic’s own support docs on Anthropic Support are a reminder that policy and gating can materially affect outcomes. Your harness needs a refusal taxonomy and a strategy, or your results are not repeatable.

Here’s the official explainer for prompt compression research-style claims (LLMLingua) if you want background. I’m including it because it’s exactly how people get misled into thinking “compression ratio” equals “money saved.”

Here’s the video:

The metric that matters: cost-per-success (and why token-savings lies)

I don’t care about “rtk gain.” I don’t even care about raw token delta.

If you’re building AI agents that touch tools, the number to optimize is:

Cost-per-success = total spend across all attempts / number of successful attempts

Total spend needs to include:

• input tokens
• output tokens
• tool-call overhead your platform bills for
• retries
• timeouts

This is the same “loop economics” framing I keep coming back to in AI in production. Once you ship an agent, your unit isn’t “a prompt.” Your unit is “a task completion.”

Quesma’s results are a clean illustration of why this matters:

• Pass rates dropped with RTK by about 1% for Fable and 2% for DeepSeek.
• When they divided all spending (including failed attempts) by number of passes, RTK was ~3% cheaper for Fable but ~7% more expensive for DeepSeek.

The part people keep missing: this happened even though RTK does what it claims. It reduces tool output tokens.

Why does a 1–2 point pass-rate change matter? Denominator math is brutal. If your baseline pass rate is 71% and you drop to 69%, cost-per-pass can easily go the wrong way unless per-attempt cost drops enough to compensate.

This is why token-only optimization pitches annoy me. The loop always wins.

Baseline vs RTK: a benchmark table you can sanity-check

Below is the directional comparison that matters, using Quesma’s published numbers (the same table they show in the post):

If you’re reading this and thinking “those dollar amounts look small for DeepSeek,” remember: cost-per-pass is a function of the provider’s pricing and the harness. The point is not the absolute number. The point is the sign.

Also, Quesma points out that one expensive task can dominate totals. That’s a real benchmarking trap. Your harness should report both aggregate and “equal-weighted per task.”

Failure modes: extra turns, tool-call breakage, and latency

Token compressors fail in three ways that matter operationally.

1) Extra turns erase savings

A compressor can reduce tool output tokens and still increase overall tokens if it causes the model to ask follow-up questions it didn’t need before.

In agent loops, a “turn” is not free. It’s:

• another model call
• another tool call
• more wall-clock time
• another opportunity for the agent to drift

This is why “terminal output is huge” does not automatically mean “terminal output dominates the bill.” In plenty of real agent stacks, the expensive part is reasoning and retries, not raw stdout.

2) Tool-call correctness breaks in boring ways

Compressed output can drop the one detail the model needed to form the next command. And it’s rarely dramatic. It’s usually one missing filename and then five minutes of the agent flailing.

The failure taxonomy I’ve found useful (and what I’d log in the harness) looks like:

• Parameter loss: path, flag, port, or filename missing after compression.
• Ordering loss: output reordered so the model picks the wrong line item.
• Error-context loss: stack trace or error header trimmed so the model misdiagnoses.
• Schema drift: tool output becomes “pretty” but no longer consistent, breaking downstream parsing.

If you don’t log these separately, you’ll end up arguing on vibes about whether compression “hurt accuracy.”

3) Latency gets worse even when spend looks better

Compression costs CPU time. It can also increase turns, which increases wall-clock latency.

If you’re using agents in a developer workflow, latency is UX. If you’re using agents in a user-facing flow, latency is revenue.

When I built the Walmart conversational commerce chatbot at Firework/Zealsight (2022–2024), we cared about sub-second responses at millions of queries daily. The lesson that stuck was simple: shaving tokens is cute, but latency and success rate dominate user experience. If your optimization adds 400–800ms and increases retries, it’s not an optimization.

Methodology: how to reproduce Quesma’s Terminal-Bench run

If you want other people to reproduce your results and build on them, publish a minimal “transparent harness spec.” Not a vibes paragraph. A spec.

Harness requirements (practical, not academic)

1. Deterministic scheduling: same task list, same number of runs per task (Quesma used 5 and 5).
2. Route pinning: record provider, model, and agent version. “Claude” is not a version.
3. Timeout policy: per-task timeout logged. Timeouts are failures and should be counted.
4. Retry policy: explicit max retries and backoff. Retries should be billed to the attempt.
5. Refusal handling: record refusal type, task excluded or kept, and why.

What to log per run

I’d log one JSON event per model call and per tool call, then aggregate into run-level metrics:

• prompt_tokens_in, prompt_tokens_out
• tool_tokens_in, tool_tokens_out (if your platform exposes it)
• rtk_gain (or compressor-specific metric), but only as a debug signal
• turn_count
• wall_clock_ms plus p50/p95 across runs
• tool_call_failures by category
• final_status: pass, fail, timeout, refusal

If you’re already doing OpenTelemetry for agents, this maps cleanly to a trace tree. I’ve written about this in agent orchestration adjacent posts like Execution Trace Tree for AI Agents and OpenTelemetry Instrumentation for AI Agents.

Pricing normalization across providers

If your benchmark compares different routes, you need a pricing table. At minimum:

• input $/M tokens
• output $/M tokens
• any platform fee (routing, tool execution)

Then compute cost-per-success with the same formula across providers.

This is also where local runs change the calculus. Based on the benchmark data I maintain at kunalganglani.com/llm-benchmarks, throughput and latency often dominate perceived “cost” for local inference. A model that is “free per token” but runs at 5 tok/s is not free in developer time.

When compression is worth it (rules of thumb I’d actually use)

I’m not anti-compression. I’m anti-people treating compression ratio like a win.

If you want thresholds you can operationalize, here are mine:

1. If pass rate drops by ≥1 point, treat it as a regression until proven otherwise. Quesma saw -1 and -2 points. That is enough to flip cost-per-pass.
2. If average turn count increases, assume you’re losing money. One extra turn can wipe out a lot of tool-output savings.
3. If p95 latency increases by >10–15%, you need a user-facing justification. Internal tools can tolerate more. Production flows rarely can.
4. If tool-call failure rate increases at all, you need a failure taxonomy and targeted fixes. “It failed sometimes” is not actionable.
5. If your baseline tool output is genuinely dominating tokens, then tool-output compression is a good bet. Measure it first. Don’t guess.

What I’d do first, before shipping RTK or any compressor into a serious workflow:

• Run a 20-task micro-benchmark in CI with your real repo and workflows.
• Add a cost-per-success gate the same way you’d add a p95 latency gate.
• Only then consider rolling it out broadly.

If you want a deeper cost model, I already have the scaffolding laid out in Agent Per-Task Cost Calculation and AI Agent Cost Per Task.

Verdict: tokens are the new vanity metric

The industry is going to keep selling “token savings” because it makes for an easy screenshot.

But for agentic workflows, it’s a vanity metric.

If you’re serious about production AI, benchmark compression like you’d benchmark a database migration. End-to-end. Reproducible. Tied to a metric the business actually cares about.

My prediction for 2027 is boring and inevitable. The teams that win won’t be the ones that compress the most tokens. They’ll be the ones with a harness that can tell them, in a week, whether a change improved cost-per-success and p95 latency across 100+ tasks. If you don’t have that harness, you’re not optimizing. You’re gambling.

Photo by Daniil Komov on Unsplash.