The Coding Agent Feature Your Model Never Calls, and the 176-Setting Study That Measured It
A new arXiv study took a coding harness apart component by component. The tool that lets an agent recover context it dropped was called a median of zero times.
Somewhere in your coding agent there is a function that has never run.
Not a dead branch, not a deprecated path. A tool, registered in the schema, described in the system prompt, sitting in the model's action vocabulary waiting for a call that does not come. If you built a compaction layer that stores what it drops so the agent can pull it back later, that function is probably recall, or whatever you named it, and a paper published on September 17 has the receipts.
"An Empirical Study of Harness Design for Coding Agents" (arXiv 2609.20804) is 43 pages by Run-Ze Fan and eight coauthors, and it is the first work I have seen that treats a coding harness as a set of separable parts rather than a product you benchmark whole. The setup is simple enough to describe in a sentence. Fix the execution loop. Vary three things: planning, action space, context management. Run 176 matched settings across four models on SWE-Bench Verified and Terminal-Bench 2.1, at four context-window budgets, across five context-management tiers.
The finding that should stop you is buried in section 3.2, and it is not about accuracy. It is about a feature nobody uses.
Why "we made it recoverable" is the wrong reassurance
Every serious agent harness eventually solves the same problem. Long sessions overflow the context window, so something has to go. The crude answer is to delete old tool results. The polite answer is to summarize them. The answer everybody has converged on in the last year is a hybrid: drop the stale stuff, but store it somewhere retrievable, and hand the model a tool so it can pull a dropped observation back when it turns out to matter.
That design is intuitively correct. It converts a lossy operation into a lossless one. It lets you be aggressive about compaction because nothing is truly gone. Ship it, and the anxiety about deleting the wrong tool result goes away.
The paper models this exactly. It defines three mechanisms: elision (M1), recall (M2), and summarization (M3), and combines them into five tiers. Tiers T1 and T2 differ in one respect only, whether recall_event is available, which makes them a matched comparison of what recoverability actually buys.
Across 32 model, benchmark, and window comparisons, T2 beat T1 in 15 settings, lost in 14, and tied in three. The equal-weight mean difference is negative 0.36 percentage points: plus 0.40 on SWE-Bench, minus 1.12 on Terminal-Bench. That is noise shaped like a feature.
The usage data explains why. Among the 64 settings where recall was available, 36 of them, 56.3%, never called recall_event at all. The median invocation rate is zero. The mean falls from 0.540 calls per task at a 32k budget to 0.069 at 64k, 0.011 at 96k, and 0.007 at 128k. The 16 planning and action-space settings at the default tier with a 128k window record no recall calls whatsoever.
And where it does get used, it does not help. The heaviest-use configuration in the whole study, Nemotron-3 30B on Terminal-Bench at 32k under T2, averages 4.326 calls per task and scores 3.37% below the same setup without recall. The authors' own summary: "lossless storage adds machinery most models seldom use, and retrieving elided observations does not consistently translate into completed tasks."
Read that again with your own codebase in mind. The recovery path is not underperforming. It is unexercised.
The cheap pass beats the clever pass
The second finding lands in the same place from a different angle. The paper's default tier, T4, stages rule-based elision before LLM-based summarization, and it wins on cost profile while holding accuracy steady against the other managed strategies. Not because elision is smarter. Because it is free, and every token it removes is a token the summarizer never has to be paid to read.
The authors put it plainly: T4 "achieves the best overall cost profile by using cheap early elision to reduce reliance on LLM summarization."
The action-space results rhyme. Under the default configuration at 128k with planning on, the predefined tool set raises Nemotron-3 30B's success rate by 15.0% on SWE-Bench and 10.1% on Terminal-Bench, because the small model cannot reliably express its intentions in shell. Strip the tools away and 66% of its bash-only Terminal-Bench trajectories die after emitting a call the harness cannot resolve, shortening the average run from 71 turns to 15.
Go up to Nemotron-3 550B and the sign flips. Bash-only improves success by 3.6% on SWE-Bench and 5.6% on Terminal-Bench while cutting cost 53% and 30%, issuing 32% and 24% fewer calls. The big model composes denser shell commands; the predefined tools were overhead.
There is a pattern here worth naming. Three separate components, and in each case the sophisticated option is a subsidy for a weaker model rather than a general improvement. Planning behaves the same way: the paper finds it trades compute for accuracy on the weaker model and mainly saves cost on the stronger ones.
Your harness is probably carrying scaffolding built for a model you no longer run.
Put this into practice
You do not need to reproduce the study. You need three numbers from your own system, and two of them are one grep away.
Count your recovery calls. If your harness has a tool for retrieving compacted or elided context, search your trace logs for its name over the last thirty days. Not "is it working," but "how many times was it invoked, by which model, at what context pressure." If the answer is single digits across thousands of sessions, you have your verdict, and you can delete the storage layer, the tool schema, and the prompt text that describes it. That prompt text is not free; it occupies the same context you are fighting to preserve.
Check the order of your compaction pipeline. If your layer calls a summarizer on raw history, put a deterministic pass in front of it. Truncate oversized tool outputs, drop duplicate file reads, collapse repeated failed commands. Then summarize whatever survives. This is the T4 result, and it costs an afternoon.
Measure where your trajectories stop, not just whether they pass. The paper's trajectory analysis is the part that makes the results usable: context management extends trajectories without changing behavior much, planning changes where they stop, and the action space changes the granularity at which code gets written. If your agent fails, the useful question is which of those three shapes the failure has. A run that dies at turn 15 after emitting an unresolvable tool call is an action-space problem. A run that dies at turn 90 having never touched the right file is a planning problem. They have different fixes and you cannot tell them apart from a pass rate.
If you want one experiment to run this week: take your most-used model, turn off your recall tool, and compare a hundred tasks. The paper predicts you will see nothing. Nothing is the result.
What this study does not tell you
The four models are Nemotron-3 30B, 120B, and 550B, plus Mistral-Medium-3.5-128B, all open weights, all served locally with SGLang in BF16 at temperature 0. There is no Claude, no GPT, no Gemini in the entire evaluation. That matters more than it might seem, because the recall finding is partly a claim about whether a model reaches for an unfamiliar tool, and models differ enormously in how readily they do that. A frontier model trained hard on long-horizon tool use might call recall_event constantly. The paper cannot tell you, and neither can I.
The harness is also the authors' own lightweight implementation, not Claude Code, not Codex, not Cursor. The execution loop was held fixed precisely so the components could be compared, which is the right methodology and also means the absolute numbers do not transfer. What transfers is the shape of the effects.
It is a preprint, not peer reviewed, published the day before I read it. Token prices come from OpenRouter as accessed in August 2026, so the cost figures drift the moment anyone reprices. And the recall result specifically rests on a mechanism the authors implemented and named; a different recovery affordance, exposed differently, might get picked up. The honest version of the claim is narrower than the headline: this recovery interface, on these models, went unused.
I still think it generalizes, and here is why. The failure is not that recovery is a bad idea. It is that the model has no way to know what it lost. An elided observation leaves no reliable trace in the context that would prompt a model to go looking for it. You built a door, but you did not build the feeling of a missing room.
What to do with this
The transferable lesson is not about compaction at all. It is that an affordance only exists if something reaches for it, and you can measure that in an afternoon, and almost nobody does.
Every agent system accumulates these. The escalation tool that never fires. The clarifying-question path the model routes around. The structured output schema with an optional field that has been null in every row since launch. Each one costs you prompt tokens, maintenance, and a false sense that a risk is handled.
Go find one. Count its calls. Then decide whether you are keeping it because it works or because it made you feel better about a decision you did not want to make.
Sources: An Empirical Study of Harness Design for Coding Agents, arXiv 2609.20804, submitted September 17, 2026; HTML version for section 3.2 figures.
Medium metadata
Title: The Coding Agent Feature Your Model Never Calls, and the 176-Setting Study That Measured It Subtitle: A new arXiv study took a coding harness apart component by component. The tool that lets an agent recover context it dropped was called a median of zero times. Tags: AI Agents, Software Engineering, Machine Learning, Developer Tools, LLM Canonical: publish on fervorai.dev first, import to Medium from the canonical URL.