One Success Isn't Reliability: The Agent Number Almost Nobody Reports

Microsoft's Thinkingbox benchmark runs every task twenty times. The strongest model falls from 65% to 25%, and most of what fails looks completely normal from the outside.

Send an agent through a stateful business workflow once and the best model tested finishes correctly about two times in three. Send it through the same workflow twenty times, from the same starting state, and it finishes correctly every single time on about one task in four.

Nothing changed except how many times somebody asked.

Those two numbers live in the same table of the same paper, and the gap between them is the entire distance between a demo that closes a deal and a system that survives contact with customers. One Success Isn't Reliability, posted to arXiv on August 20 by twelve authors out of the Microsoft Copilot Studio reinforcement-learning team, puts a number on something most teams have felt and never measured: 65.36% pass@1 against 25.25% pass^20.

Why the second number is the one that matters

Almost every agent result you have read this year is a pass@1 number. The team ran the task, it worked, they wrote it down. Sometimes they ran it a few times and reported the best result, which is worse, because best-of-N flatters a system exactly where it is weakest.

Pass^20 is the opposite question. Out of twenty independent attempts at the same task, how many tasks did the agent get right on all twenty? That is what a customer experiences. Nobody calls your support agent once. A refund workflow that lands 65% of the time is a workflow that generates a wrong bank record roughly one time in three, forever, on every ticket.

Thinkingbox builds this in rather than bolting it on. Every one of its 507 tasks runs N=20 repeated trials by default, and both scores come out of the same run. That design choice is the actual contribution here, more than the benchmark content. Once repetition is the default, "it worked" stops being a sentence you can say.

The benchmark itself covers five domains with deliberately similar task counts: retail (98 tasks), auto insurance (100), booking (104), neobank internal IT (104), and consulting IT and HR support (101). Every attempt gets graded by task-specific executable checks that read the terminal state of the backend, accepting any valid path to the right end state while rejecting wrong, missing, or extra effects. Designated tasks also check required properties of the final message to the user.

Grading the database instead of the transcript is what makes the rest of the findings possible.

The failures that look like successes

Here is the part that should change how you instrument agents.

The paper takes every failed trial and assigns it one dominant failure signature from the trajectory. Across the models in the table, the breakdown averages 77.5% tool usage errors, 12.1% wrong state update, 7.9% incomplete user resolution, and 2.5% no state-changing action at all.

Read those categories again with your dashboard in mind.

"Tool usage" here does not mean malformed JSON. The authors are explicit: these are traces where the environment returned an error, a failed precondition, or an empty lookup, and the agent failed to repair the workflow afterward. In some traces it simply continued as though the failed call had succeeded. Your tool-call success rate metric sees a call, sees a response, and moves on.

"Wrong state update" is worse. The agent invokes the mutating tool correctly. The tool returns without error. The agent confidently confirms completion to the user. And the record it wrote points at the wrong entity, the wrong date, the wrong eligibility decision, or the wrong refund amount. The paper's own phrasing is that these traces "can therefore appear operationally successful." For o3-pro that category is 27.8% of all failures. For Grok-4.3, 22.5%. For GPT-5.2, 18.3%.

Every agent observability product I have looked at this year sells response-level and tool-call-level telemetry. Clean termination, valid tool calls, no exceptions, latency inside budget. That is precisely the set of signals this paper says are not proxies for finishing the job. A clean green trace and a silently wrong refund are indistinguishable unless something reads the final state and compares it to what should have happened.

That is the argument in one line: if your evaluation cannot see the database, it cannot see failure.

The aggregate ranking hides more than it shows

A secondary finding is worth its own paragraph because it will bite anyone picking a model from a leaderboard.

Averaged across models, retail sits near 52% pass@1 while auto insurance sits near 23%. Task counts are comparable, so that is a difference in interaction and policy burden, not sample size. Only GPT-5.4 (65.36% average) and Claude Sonnet 4.6 (58.45%) clear 50% in every domain. Claude Opus 4.6 posts 74.90% on retail, competitive with anything in the table, then drops to 14.65% on auto insurance for an average of 37.91%. GPT-5.2 shows the same shape, 70.20% retail against 22.40% auto.

Among open weights, DeepSeek-V4-Pro leads at 43.26% average, close to GPT-5.2 overall. Mistral-Large-3 has hundreds of billions of parameters and lands at 4.66%, below Qwen3.6-27B at 32.94%. Size explains very little. Agentic post-training and tool-use behavior explain a lot.

So a model that tops your category might collapse in the one workflow family you actually sell.

Put this into practice

You do not need to adopt Microsoft's benchmark to get the benefit. You need to steal the method. Here is the smallest version that produces real signal, and you can start it this afternoon.

Pick one workflow that writes to a database. Refund, ticket status change, booking modification, account provisioning. Something with a side effect you can query afterward.
Write the check against the final state, not the reply. Query the record after the run. Assert the exact values that should have changed, and assert that nothing else changed. That second half is what catches collateral effects, and it is the half everyone skips.
Run it twenty times from an identical starting snapshot. Reset state between trials. Report both numbers side by side in whatever you show your team: pass@1 and pass^20. Once both numbers are on the slide, the conversation about readiness gets much shorter.
Read the failed traces by category. Sort them into the four buckets above. If most of yours are tool-usage recovery failures, your fix is error handling and retry policy, not a bigger model. If you have wrong-state updates, you have a correctness problem that no amount of prompt tuning reliably removes.
If you want the harness itself, the code is MIT-licensed and split across two repos: microsoft/thinkingbox for the framework and microsoft/thinkingbox-data for the datasets and MCP tool servers. It is Linux-only (WSL counts). Clone both, uv venv --python 3.12, uv sync --group dev, start the MCP session proxy with uv run tb mcp-start, then run the bundled cloud_drive smoke test to confirm your endpoint is wired up before you point it at anything real.

Start with five tasks, not five hundred. Twenty trials times five tasks times a multi-turn conversation with a simulated user is already a real bill, and five tasks is enough to tell you whether your pass^20 is a disappointment or a fire.

Honest limitations

The models in that leaderboard are one generation back. GPT-5.4, Claude Opus 4.6, GPT-5.2, o3-pro, Grok-4.3. Neither GPT-5.6 nor Opus 5 appears, so the absolute figures were dated the week they published. I would not quote 65.36% as the state of the art. The structural finding, that reliability collapses under repetition and that clean traces hide wrong outcomes, is what survives a model refresh, and it is the only part of this I would build a policy on.

The failure signatures are the authors' own diagnostic labels, and they say so: "observable diagnostics rather than unique causal explanations." One dominant signature per trace is a simplification of traces that often contain several problems.

The o3-pro row excludes 636 system and harness-error trials, so its denominators vary by domain and it is not strictly comparable to the rest.

The scenarios are business-process shaped. Retail, insurance, booking, banking, consulting support. If your agent writes code or does research, this benchmark will tell you very little about your workload, though the pass^k method transfers perfectly well.

The simulated user is another language model following a prompt. Real users are stranger, ruder, and more ambiguous than a well-behaved simulation, which means these numbers are probably optimistic.

And the repository is very new. When I read it, GitHub served 6 stars, 0 forks, and 11 commits on main, while this morning's trend data had recorded 19 stars. The README still says a paper is coming soon, which it is not; it is out. Treat the code as a reference implementation of an idea rather than a maintained product, and take the idea.

What to do with this

The next time someone shows you an agent completing a workflow, the useful question is not "how did it do that?" It is "how many times did you run it, and how many of those runs wrote the right row?"

Ask it about your own system first. Most teams have never run their flagship workflow twenty times from the same starting state, which means most teams do not know their real number. Finding out costs you one afternoon and one script. If the answer is fine, you have a defensible claim you did not have yesterday. If the answer is bad, you found out before your customers did.

A benchmark that grades the database instead of the transcript is not the exciting kind of progress. It is the kind that stops you shipping something confident and wrong.

Sources: One Success Isn't Reliability: Thinkingbox, a Sandbox and Benchmark for Agents in Stateful Business Workflows (arXiv:2608.19741), paper full text, microsoft/thinkingbox on GitHub. Figures read August 25, 2026.