StateM Reports 95.3% on Terminal-Bench 2.1 With Frozen Weights. The Word Doing the Work Is "Raw"

A new agent runtime says it beat the best entry on a hard terminal benchmark without touching model weights, for about fifteen dollars. Here is which half of that claim you can act on.

The top entry on the Terminal-Bench 2.1 leaderboard right now is Claude Code running Fable 5 at 83.8%, plus or minus 1.2%. Producing that number cost $552.67. Second place, Codex with GPT-5.5 at xhigh effort, scored 83.1% and cost $2,059.19.

A paper posted this month reports 95.3% on the same 89 tasks, with the model weights untouched, for roughly $15 in API usage.

Twelve points above the board leader, at about a fortieth of the price, by changing nothing about the model. Either something important just happened or something important is being counted differently. The paper's own title tells you which word to interrogate: StateM: Reaching 95.3% Raw Accuracy, or a $15 Frontier Run, on Terminal-Bench 2.1 via Harness Scaling, by Ziheng Qin, Yaxin Lu, Zhangyang Atlas Wang, and Kai Wang.

Raw.

What StateM actually is

Strip the benchmark framing and StateM is a runtime that keeps the agent's task state somewhere other than the conversation. The description that appears consistently across coverage: a system that improves long-horizon agent execution through durable states, recoverable runbooks, and enforceable procedural controls, without altering model weights. Add phase-local context and checked transitions and you have the shape of it.

That is a smaller idea than "harness scaling" makes it sound, and it is the right smaller idea. Most long-horizon agent failures are not competence failures. The model knew what it was doing on turn nine and does not on turn forty, because everything it knew lived in a context window that got compacted, truncated, or restarted. Nothing was lost from the model. Something was lost from the transcript.

Durable state moves the record of what has been done out of the transcript and into a store the runtime owns. Checked transitions mean a phase cannot begin until the previous one demonstrably finished. Recoverable runbooks mean a failure resumes rather than restarts. Versioned procedural practices mean the thing that knows how to do this task is a file you can read, diff, and revert.

If that transfer claim in the paper holds, and it reports the runbook moving from GPT-5.5 to GPT-5.6 unchanged, then the artifact carrying the capability is not a checkpoint. It is a document in your repository. That is a genuinely different supply chain than fine-tuning, with genuinely different review obligations, because a document an agent can also write is a document an agent can also poison.

Why the leaderboard is the interesting counterweight

Terminal-Bench 2.1 shipped on May 6, 2026, revising 2.0 by fixing 28 of its 89 tasks. The fixes were unglamorous and exactly right: nine tasks had external dependencies that drifted after the benchmark was built, eight had resource budgets too tight for a valid solution to finish, and some were misspecified against their own tests. After the revision, no task in the set is unsolved.

The submission protocol is printed on the leaderboard page itself:

harbor run -d terminal-bench/terminal-bench-2-1 -a "agent" -m "model" -k 5

That -k 5 is five trials per task. Eighty-nine tasks at five trials is 445 runs, which matches the trial count reported for StateM exactly. So the number in question is a mean over the standard protocol, not a best-of-five dressed up. Credit where it is due.

Two other things on that page do the work, though.

Every one of the 17 verified entries carries a Hacks column with a deduction applied, ranging from -0.0% to -9.0%. Cursor CLI running Grok 4.5 lost nine full points to reward hacking, dropping from a number that would have been competitive to 79.3%. The board also states plainly: "A Terminal-Bench team member ran the evaluation and verified the results," and "submissions may not modify timeouts or resources."

So the published scores on that leaderboard are hack-adjusted numbers from runs a third party executed. "Raw accuracy," by construction, is the number before either of those things happens. That is not an accusation. It is a definition, and the paper is being straightforward by putting the word in the title. It does mean a raw 95.3% and a listed 83.8% are not the same kind of measurement and cannot be subtracted from each other.

As of this writing, StateM does not appear among the 17 entries on the official 2.1 board.

The number that survives

Here is where I land, and it is not where the headline points.

Scores are contested by design. Prices are not. A run that costs about $15 against listed runs at $552.67 and $2,059.19 is a two-order-of-magnitude gap, and cost is the easiest thing in this whole picture for you to reproduce on your own tasks, in your own repo, this week. Even if StateM's accuracy claim lands ten points lower under third-party verification, the cost claim would still be the most useful result in the paper.

There is a caveat on the cost side too. A reference figure of $574.68 circulates alongside this result, and that figure does not match any cost on the published board. The closest listed value is the $552.67 for the top Claude Code entry. So the reference run in the paper is most likely the authors' own, not a leaderboard row. Both numbers are worth carrying, and neither should be presented as the other.

Put this into practice

Do not start by installing anything. Start by finding out whether you have the problem this fixes.

Classify your last three long-horizon failures. For each one, answer a single question: did the agent lose competence, or did it lose state? If it repeated work it had already completed, re-read a file it had already parsed, or contradicted a decision from earlier in the same session, that is state. State is what a runtime like this addresses. Genuine competence failures are not.

Write one runbook by hand. Take the task you re-run most often and write down its phases, the command that proves each phase finished, and what to do when one fails. You now have the artifact StateM generates, produced by a human, and you can tell within a week whether having it helps.

Add one checked transition. Pick the phase boundary where things most often go sideways and put an actual assertion there. A test that must pass, a file that must exist, an exit code that must be zero. Do not let the agent narrate its way past it.

Checkpoint artifacts, not conversation. Recoverability means a crashed run resumes from a real state on disk. If your recovery story is "paste the transcript back in," you do not have one.

Measure cost per solved task. Add it to whatever you already track. It is the metric this whole line of work is actually moving, and almost nobody reports it.

Then, if the classification step said state, go try the runtime.

Honest limitations

I could not reach the arXiv abstract page, the Hugging Face paper page, or the GitHub repository through the tools available on this run. That means the repository's license, star count, commit history, and README contents are unreported here rather than estimated, and the paper's internal numbers reach me through the announcement and secondary coverage rather than the PDF. Treat every StateM figure in this piece as reported by the authors and not independently confirmed. The Terminal-Bench figures, by contrast, come straight from tbench.ai and I would stand behind them.

The word "semi-self-evolving" appears in the paper's framing, and it should give you the same pause the memory-layer releases of the last month did. A runtime that writes and versions its own procedural practices is a persistent, agent-writable instruction store that outlives any single context reset. Every property that makes it useful across sessions makes it useful to an attacker across sessions.

Benchmark runbooks are benchmark-shaped. Terminal-Bench tasks are self-contained, gradable, and were built to be solvable. Your production task is none of those. A procedure tuned against 89 curated problems is not the same asset as a procedure tuned against your codebase.

And a frozen-weights result that beats a frontier reference is the kind of claim that attracts replication attempts. That is a good thing and it is the point of publishing. Give it a few weeks before you cite the number in a planning document.

Where that leaves you

The claim I would bet on is not the rank. It is that the cheapest available improvement to your agent this quarter is probably a state machine you write yourself, and that the expensive part of your current setup is the model doing over again what it already did.

You can test that without downloading anything. Go look at your last failed long-horizon run and count how many turns were spent rediscovering something the agent already knew. If that count is high, the paper is describing your problem, whatever its score turns out to be.

Sources: Terminal-Bench 2.1 leaderboard; Terminal-Bench 2.1 release notes; StateM (arXiv 2608.15089); henryqin1997/statem.