The Consistency Gap: Your Agent's 77% Is Hiding a 24-Point Reliability Problem

IBM measured the distance between an agent that can do a task and one that will do it every time. Here is what causes it, how to measure yours, and why a bigger model won't close it.

Same prompt. Same model. Temperature zero. Five runs. Three pass, two fail, and nothing about the task changed between them.

That is not a bug report from a bad afternoon. It is the baseline result in a paper IBM Research put on arXiv on September 8 and wrote up for a general audience on September 15, and the number attached to it should make anyone shipping an agent uncomfortable. A ReAct agent running GPT-4.1 on the AppWorld benchmark succeeds 77.4% of the time on average. Ask it to succeed on the same task five times in a row and it manages that for only 53.0% of tasks. IBM calls the 24.4-point difference the consistency gap, and on the hardest tier of the benchmark it reaches 30 points.

Almost nobody reports the second number. Every leaderboard you have ever read reports the first.

Two numbers that share a benchmark and describe different products

The first number is Mean@k: run the benchmark k times, average the pass rate. That is what "accuracy" means on a model card, and k is often three, sometimes one.

The second is Pass^k: the fraction of tasks where the agent passes on every one of the k runs. Not Pass@k, the code-generation metric that asks whether at least one of k attempts worked. Pass@k is optimistic and useful when you can verify and retry. Pass^k is its mirror image, the pessimistic one, and it is the number a user experiences when they ask the same question twice. The ordering never changes: Pass^k is at most Mean@k, which is at most Pass@k.

My position is that Pass^k is the only one of the three that tells you whether an agent is a product. A reconciliation workflow that succeeds 77% of the time on average and every time on only half its tasks is not 77% reliable. It is a coin flip on a quarter of your inputs, dressed in a good average, and the average is what got it through the demo.

Why an agent flips when nothing changed

The mechanism is the part of the post worth reading twice.

Every decision an agent makes (which API to call, what argument to pass, whether to retry) comes out of a distribution over next tokens. What matters is the shape. A sharp distribution puts most of its mass on one token; the runners-up are far behind, and the same choice comes out run after run. A flat distribution spreads comparable mass across several near-tied tokens, and which one wins is close to a coin flip.

Sharp decisions survive noise. Flat ones do not, and hosted endpoints are full of noise: floating-point non-associativity on the GPU, request batching, and other platform effects that nudge probabilities by amounts far too small to reorder a clear winner and just large enough to reorder a near-tie. IBM's agent ran at temperature 0.0 throughout, so none of the observed variance is sampling. Greedy decoding and a fixed seed govern how a distribution becomes a token. They say nothing about the distribution itself, and on a hosted endpoint that distribution shifts slightly from call to call.

Then it compounds. A trajectory chains dozens of decisions. A small per-step chance of flipping becomes a large chance that some run goes differently, and that is where a 24-point gap comes from.

This is why the gap is orthogonal to capability. A stronger model raises the average. It does not necessarily sharpen the decisions that were flat, so it does not necessarily close the gap. An agent can be capable and inconsistent at the same time, and most of the ones I have run are.

How the Consistency Analyzer finds the flat steps

IBM's tool is a diagnostic, not a new model, and the design is cheap enough to run on production traffic.

Take one recorded trajectory. For each decision step, make one additional model call against the already-recorded context, asking for k completions at once (k=5 by default). Measure how much the output varies. That yields a consistency score per step, written into a scorecard that pinpoints where the run was one token-sample away from going somewhere else. No ground truth. No re-running the task against the environment. No logits or model internals, so it works against any endpoint that supports n completions. Detection is black-box on a trace you already have.

The second stage turns flagged steps into what IBM calls consistency guidelines, in the same format as the rest of its ALTK-Evolve toolkit, which stores agent-derived guidance and retrieves it into context on later runs. The post shows a real pair generated by GPT-4.1 from an AppWorld task about counting completed items in a note: use a line-anchored regex rather than a plain substring count because note titles often repeat the marker in a legend line, and verify search results return one match before proceeding. Neither is task trivia. Both are decision points that show up with high uncertainty across many tasks, and the analyzer caught them on a run the agent happened to get right.

The results, on the 168-task test_normal split: Pass^5 rose from 53.0% to 69.0% while Mean@5 rose from 77.4% to 81.0%, cutting the gap from 24.4 points to 12.0. Medium-difficulty tasks gained 22.9 points of Pass^5, hard tasks 14.3. Applied to a different task in the same AppWorld scenario, the guidelines still lifted Pass^5 by 13.0 points, three points below the same-task number, which is the evidence that they capture something reusable rather than patching one trace. On a weaker model, gpt-oss-120b, same-task Pass^5 moved from 10.1% to 16.1%, and the similar-task gain of 8.7 points exceeded the same-task gain.

Mean@5 never dropped at any difficulty level. IBM treated that as a hard requirement, and it should be yours too: a method that buys consistency by trading away accuracy is moving unreliability around, not fixing it.

Put this into practice

Start with the measurement, because it costs almost nothing and you may already have the data.

Compute Pass^k on the eval you already run. Run it three times instead of once. For each task, record whether it passed all three. Divide by the task count. Put that number next to your average in the same table. IBM's advice is that even k=3 surfaces a gap you did not know you had, and in my experience the gap widens on exactly the tasks you care most about, because those are the long ones with the most decisions to flip.

Expect the gap to be worst on your hardest tier. If you only report an aggregate, split it by difficulty once and look at the hard bucket. That is where a single averaged number is most misleading.

Try the analyzer on one trace. The toolkit installs with pip install altk-evolve (Python 3.12 or newer) and the blog post says the repository now includes the Consistency Analyzer and consistency-guideline generation used in the experiments. Note that as of September 16 the repo README does not document an analyzer command and its "latest" table stops at August 18, so budget time to read the source rather than expecting a one-liner. What you want out of it is the scorecard: which steps were flat.

Then take the flattest decisions away from the model. This is the step IBM's post does not spell out and the one I think matters most. A guideline is text in the context window, and text can be ignored. If the analyzer tells you the agent is near-tied on which of two tools to call, or on whether a string match should be anchored, that is a decision you can often make in code: a validator, an enum, a deterministic router, a rule that fires before the model sees the input. Guidelines are the cheap first move. Moving the decision out of the prompt is the durable one.

Sample in production. Because the analyzer needs one extra call per step and no replay, you can run it on a slice of live traffic. Pick 1% of trajectories, score them, and watch which steps stay flat across users. Those are your reliability backlog.

Where this breaks

Every published number is one model on one benchmark. GPT-4.1 on AppWorld is a reasonable proxy for tool-using agents, and the gpt-oss-120b result suggests the effect transfers, but a 24-point gap is not a universal constant, and your agent's gap could be five points or forty. Measure before you believe anything here applies to you.

The fix is more text in the context window. Consistency guidelines slot into ALTK-Evolve's storage and retrieval pipeline, which means they inherit every problem that pipeline has: retrieval that misses, guidelines that conflict, context that fills. The README's own retention and conflict-resolution machinery exists because those problems are real. A guideline that is not retrieved on the run where it mattered fixes nothing.

The analyzer measures flip-proneness, not correctness. It will flag a step the agent got right this time and might get wrong next time, which is the point, but it will also flag steps where either branch was acceptable. You still need a human to decide which flat decisions matter.

It needs an endpoint that returns multiple completions per call. If your provider does not expose that, the one-call-per-step economics become five.

And the technical report (arXiv 2609.08832) is licensed CC BY-NC-ND, so you can read it and cite it but not adapt it or use it commercially; the code is the part you can build on.

The number you decide to report

Nobody made you report Mean@k. It is a convention, and conventions change when enough people stop honoring them.

You can run your eval three times this week and publish the second number next to the first. You can find your three flattest decisions and move one of them into code. Or you can keep shipping on the average and find out where the gap is from your users, one flipped run at a time. The measurement is cheap. The choice about what to do with it is yours.

Sources: IBM Research, "Your Agent Aced the Task. Will It Do It Again?" (September 15, 2026); arXiv 2609.08832, "Closing the Consistency Gap" (submitted September 8, 2026); AgentToolkit/altk-evolve on GitHub.

Medium metadata

Title: The Consistency Gap: Your Agent's 77% Is Hiding a 24-Point Reliability Problem
Subtitle: IBM measured the distance between an agent that can do a task and one that will do it every time. Here is what causes it, how to measure yours, and why a bigger model won't close it.
Tags: AI Agents, LLM Evaluation, Agentic AI, Machine Learning, Software Engineering
SEO description: IBM's Consistency Analyzer found a 24-point gap between Mean@k and Pass^k on AppWorld. What causes agents to flip at temperature zero, how to measure your own gap, and what to do about the flat decisions.
Canonical: fervorai.dev (import to Medium from the published URL)