HarnessTax Says Your Coding Agent Harness Costs 2x for 2 Points, and the Bill Starts on the First Call

A Berkeley study ran seven models through Claude Code, Codex CLI, and Pi. The harness barely moved the pass rate. It doubled the invoice.

Claude Fable 5 solves 97.8% of a SWE-bench Lite sample inside Claude Code. Move the same model into Pi, a harness with four tools and no plan mode, and it solves 96.7%. The difference between those two numbers is one task in a hundred. The difference in cost is $1.33 per attempt versus $0.67.

That is the headline of HarnessTax, a study from Melissa Pan, Shuo Yang, Negar Arabzadeh, Wei-Lin Chiang, Ion Stoica, and Matei Zaharia at UC Berkeley and Arena, which hit Hacker News on the evening of September 16. It measures something almost nobody measures: what the software wrapped around the model is worth, in dollars, at the same success rate.

What the study did

Twenty-one model-harness pairs. Seven models: Claude Fable 5, Opus 4.8, Sonnet 4.6, Haiku 4.5, GPT-5.6 Sol, GPT-5.6 Luna, and Kimi K3. Three harnesses: Claude Code, Codex CLI, and Pi. Two benchmarks: SWE-bench Lite and Terminal-Bench 2.0. Each point is 30 randomly sampled tasks run three times.

Three findings came out, and the authors number them.

First, the harness moves cost more than correctness. Across the models that ran in all three, Claude Code cost about 2.0x Pi and 1.6x Codex on SWE-bench Lite, and 1.5x Pi on Terminal-Bench, using geometric means of the cost ratios. The average harness effect on success rate stayed within about 2 points on SWE-bench Lite and about 5 on Terminal-Bench.

Second, a simple harness competes. Pi sat on the cost-success Pareto frontier of both benchmarks with a tool surface of read, write, edit, and bash. Nothing else.

Third, models do not need their maker's harness. In nine of twelve model-benchmark comparisons across the six Anthropic and OpenAI models, a non-native harness posted the best success rate. GPT-5.6 Sol reached 83.3% on Terminal-Bench in Pi versus 78.9% in Codex, at $0.42 versus $0.76 per attempt. Sonnet 4.6 did slightly better in Codex than in Claude Code on SWE-bench Lite at similar cost.

The study's own phrase for the gap is a harness tax, borrowed from an April Portkey post that found Claude Code sending about 27,000 input tokens on a two-message session where Pi sent about 2,600.

Where the tax gets charged

The part I keep coming back to is Figure 3, the first-call context. Across all seven models, Claude Code's mean initial context was over 10x Pi's, and the authors attribute it to longer instructions and larger tool schemas. That is before the model has read a single line of the repository.

Think about what that means mechanically. Every turn of an agentic session resends the system prompt and the tool definitions. If the first call carries ten times the framework text, and a session runs fifteen turns (Fable 5 averaged 15.3 turns in Claude Code and 15.4 in Pi on SWE-bench Lite, near identical), the framework text is charged fifteen times. Prompt caching softens this, and the authors say so, but caching discounts the tokens; it does not delete them. The harness tax is not a one-time fee. It is a per-turn subscription you pay for features that the study says, on these tasks, bought you about two points.

This reframes the harness question. It is usually asked as "which harness has the features I want?" Plan mode, hooks, permission prompts, subagents, MCP management, a nicer TUI. Those are real, and the study does not claim they are worthless. It claims that on two open benchmarks their measured contribution to task success is small, and their contribution to cost is large, and that you have probably never looked at the second number because the default harness came with the model.

My position: harness choice is a procurement decision that has been sold to us as a productivity decision. The features are the pitch. The token bill is the price. And the price has been invisible because the harness and the model come from the same vendor, in the same box, and the box does not itemize.

Why "your Claude models may not need Claude Code" is the wrong takeaway

The study's own meta description says "your Claude models may not need Claude Code," and that line will get quoted out of context all week. Read the numbers instead. Fable 5 in Claude Code posted the highest success rate on SWE-bench Lite of any pair in the study. It was 1.1 points ahead of Pi at twice the cost. Whether that trade is good depends entirely on what a failed task costs you.

If you are running an agent over a thousand routine tickets, a 1.1-point difference is eleven extra failures and a doubled bill; Pi wins. If you are running one agent over one migration that takes a week and blocks a release, the cost of one failure is the week, and the 2x on tokens is noise; the heavier harness wins. The study does not pick for you. It gives you the exchange rate.

There is a subtler point in Finding 3. Provider-specific optimization is a real thing (OpenAI has described models optimized for Codex), and it still did not guarantee the best pairing. Model capability, the authors write, is "compatible, generalizable" across harnesses. That is good news for anyone building their own harness, and it is bad news for the idea that the harness is where the moat lives.

Put this into practice

You do not need a benchmark rig to measure your own harness tax. You need one week and a proxy.

Start with the first call. Route your coding agent through any gateway that logs raw requests (LiteLLM, Bifrost, Portkey, or your own reverse proxy) and read the input token count on turn one of a trivial session, before any file has been read. That number is your harness's fixed cost. Do it for each harness you have installed. The study found a 10x spread; yours may be smaller or larger, but you will have a number where you had a vibe.

Then run the same real tasks through two harnesses. Not a benchmark. Ten tickets from your own backlog, the kind you would hand to an agent anyway. Same model, same tasks, Claude Code and Pi (or Codex and Pi). Record cost per solved task, not pass rate alone. A harness that passes 9 of 10 at $12 and one that passes 8 of 10 at $5 are both defensible; you just need to know which one you are buying.

If you use Claude Code, version 2.1.274 (published September 16) added an effort attribute to the claude_code.llm_request OpenTelemetry span. If you already export OTel, you can now see per-request how hard the harness asked the model to think, which is the other half of the cost story the study did not measure.

Treat Pi as a control group rather than a replacement. Its four tools are the floor. Anything your heavier harness does better than the floor is a feature you can now price.

And keep a note of what the heavier harness actually did for you on the tasks where it won. Was it plan mode? Permission gating? A subagent? The study's finding is that these things are cheap to strip and expensive to carry, but that is on benchmarks. On your work, the answer might be different, and the only way to know is to look.

Honest limitations

The sample is small: 30 tasks per cell, three repetitions, on two public benchmarks. The authors say plainly that models "may have encountered" both during training, and that results "may differ on other benchmarks and workloads." Terminal-Bench 2.0 has already been superseded by later versions, and success rates in the mid-90s on SWE-bench Lite suggest a ceiling effect on the harder comparisons.

The cost figures are per attempt at the providers' list prices as the study ran them, and the page carries no publication date; its references are marked as accessed September 16, 2026. Turn counts are not comparable across harnesses, which the authors also note, so "same turns, double cost" is directional.

The study measures single-session benchmark tasks. It says nothing about multi-day work, sessions that compact, or the value of a permission prompt that stops an agent from deleting your test suite. Those are exactly the cases where a heavy harness might earn its keep, and they are unmeasured here.

Pi is not free of tradeoffs either. Its README states it "does not include a built-in permission system for restricting filesystem, process, network, or credential access," and recommends containerizing it. The cheap harness is cheap partly because it does less.

Closing

The number that matters from HarnessTax is not 2x. It is 10x, the first-call context ratio, because that is the one you can measure this afternoon on your own machine with a proxy and a text editor. Once you have it, the harness stops being the thing that came with the model and becomes a line item, and line items get questioned.

Go read your first request.

Sources: HarnessTax (UC Berkeley and Arena, references accessed September 16, 2026); Hacker News thread; Pi README; Portkey, "The Harness Tax" (April 13, 2026); Claude Code CHANGELOG and npm registry for 2.1.274.

Medium metadata

Title: HarnessTax Says Your Coding Agent Harness Costs 2x for 2 Points, and the Bill Starts on the First Call
Subtitle: A Berkeley study ran seven models through Claude Code, Codex CLI, and Pi. The harness barely moved the pass rate. It doubled the invoice.
Tags: AI Agents, Claude Code, Codex, Software Engineering, LLM
Canonical: fervorai.dev (import from the published URL)