Terminal-Bench 2.1 vs 4.0: The Benchmark Version Number Is Now More Informative Than the Score

A model released this week beats Claude Opus 5.0 on one version of an agent benchmark and loses to it by twenty points on another. Both numbers are real. Here is how to read the gap.

DeepSeek-V4.1-Flash landed on Hugging Face overnight with a benchmark table that contradicts itself in public, and the contradiction is the most useful thing on the page.

On Terminal-Bench 2.1, it scores 90.6. Opus-5.0 scores 89.1. GPT-5.6 Sol scores 88.8. An open-weights MIT-licensed model beat both frontier labs, which is the headline every aggregator ran with.

Three rows down, on Terminal-Bench 4.0, the same model scores 31.2 and Opus-5.0 scores 51.8. On 3.0 it is 30.0 against 43.3. Same model, same evaluation run, same harness, same day. The gap between the two results is larger than the gap between most models people argue about.

Nobody is lying. DeepSeek published both rows in the same table, which is more than a lot of releases manage. But if you have been reading "beats Opus on Terminal-Bench" as a fact about model quality, that reading just broke, and the thing that broke it is a decimal point in a benchmark name.

The version number is a date stamp on the training target

Terminal-Bench 2.1 is not an easier version of Terminal-Bench 4.0 in the way that a lower difficulty setting is easier. It is an older version. And an older agent benchmark has been public for longer, which means more of its task distribution has soaked into the training pipelines, the synthetic data generators, and the eval-driven post-training loops of everything released since.

Look at what DeepSeek says about its own post-training. The model card describes the recipe as standard SFT, then RL, then on-policy distillation, "without algorithmic modifications," and says all the substantive changes are in the data pipeline: "large-scale automated synthesis of agent tasks and environments with progressive scaling of data, tasks, and rollouts."

That is a company telling you, plainly and without embarrassment, that its advantage comes from synthesizing agent tasks at scale. Synthesized against what? Against a target distribution. The public agent benchmarks are the most legible target distribution available. This is not an accusation of contamination and I am not making one. It is a description of how anybody would build this system, and DeepSeek describes it themselves.

Now the two rows make sense. On the benchmark generation that has been public long enough to be well-modeled, the cheap model with the aggressive data pipeline wins. On the generation that is new enough that nobody has had time to synthesize against it, the expensive model with more raw capacity wins by twenty points.

The score tells you how well a model does on a task distribution. The version number tells you how much time the industry has had to aim at that distribution. Right now the second fact is moving faster than the first.

The same pattern, from the other direction

OpenAI's GPT-6 Astra post from September 9 is the mirror image. It reports 57.9% on Terminal-Bench 4.0, against 37.3% for GPT-5.6 Sol and 55.8% for Claude Fable 5.1. Notice which version OpenAI chose to headline. Not 2.1, where the numbers are compressed into a two-point band at the top and an open model can beat you. The newest one, where the spread is wide and capacity still separates the field.

Both vendors are picking the benchmark generation that flatters them, and both are being straightforward about which one they picked. This is not a scandal. It is what a rational actor does when two versions of the same benchmark measure different things.

There is a third instance of this from a completely different scale, and it is the clearest of the three because the author had no incentive to hide it.

Hugo Vergnes trained a 3.8B model from scratch for $998 and scored 0.384 on CORE. He also ran the identical recipe at 1024-token context instead of 2048 and got 0.3384. Same model, same data, same steps, nearly identical eval loss at step 20,000 (2.0160 versus 2.0164, four decimal places apart), and a CORE gap of 0.034.

He then went and found out why. Three of CORE's 22 tasks have prompts that never fit in 1024 tokens. SQuAD in that bundle is a ten-shot task with a median prompt around 1,998 tokens, and his harness keeps the last max_seq_len tokens when a prompt is too long. So the test passage always survived and the ten worked examples were always cut. The model was being asked to produce exact-token matches in a format it had never been shown.

SQuAD did not plateau. It decayed monotonically to exactly zero: 0.1478, then 0.0617, then 0.0099, then 0.0007, then 0.0000. The model got steadily worse at the task the longer it trained, because an early high-entropy model occasionally emits something short and generic that happens to match, and a well-trained one commits to fluent prose that never does.

Strip SQuAD and boolq out and the other twenty tasks move by 0.008 combined. Two tasks out of 22 produced 83% of the headline improvement, and neither of them was measuring model quality. They were measuring truncation policy.

What this actually costs you

The practical damage is not that you get a wrong ranking. It is that you get a stable wrong ranking, which is much harder to notice.

If you evaluate three models on one benchmark version and one wins consistently, you will believe it. Consistency reads as signal. But if all three are being scored on a distribution that one of them was trained toward, the consistency is an artifact of the target, not of the models. You will make a migration decision on it, ship it, and then find out six weeks later that the cheap model falls apart on the twenty percent of your workload that does not look like the benchmark.

That is exactly the shape of the DeepSeek result. Terminal-Bench 4.0 is not measuring some exotic capability. It is measuring complex terminal-based tasks, software engineering, system configuration, data analysis. Ordinary work. A twenty-point gap there is not a rounding error.

Put this into practice

The lowest-friction version of this takes about an hour and needs no new tooling.

Pin the version in every number you write down. Not "Terminal-Bench," ever. "Terminal-Bench 4.0, Pass@1, N=3, 1M context, no network access." DeepSeek publishes all of those conditions for its runs. If a vendor's claim does not carry them, the claim is not comparable to anything.

Run both generations, and read the delta rather than the score. If you already run an agent eval, add the previous version of it alongside. The interesting output is not either number, it is the difference between them. A model that scores similarly on both is telling you its capability generalizes. A model with a wide spread is telling you where it was aimed.

Check the truncation policy on your own harness before you trust a single result. This is the Vergnes lesson and it is nearly free. For each task in your eval, print the prompt token count and the fraction of prompts that exceed your context limit. Then check which end your harness cuts from. If it keeps the tail, your few-shot demonstrations are being deleted and your model is being scored on a format it was never shown. If any task is cropping more than a few percent of prompts, that task is measuring your window, not your model.

Watch for scores that decline during training. Models do not normally get worse at things. When one does, the harness is almost always the reason, and the failure is usually silent. Vergnes only caught it because he logged per-task scores at every eval checkpoint rather than the aggregate. Log per-task.

When a model card gives you a continuous knob, map it yourself. DeepSeek-V4.1-Flash exposes reasoning effort as an integer from 1 to 100, and every published benchmark in the model card uses 100. The entire curve between 1 and 100 is undocumented. That is not a flaw, it is an invitation: bisect for the cheapest setting that still passes your eval, because the vendor has no reason to find that point for you and every reason to publish the top of the curve.

Honest limitations

I have not run Terminal-Bench 4.0 myself, and neither has almost anyone reading this. Everything above rests on numbers each vendor published about its own model in its own harness. DeepSeek says its base-model comparisons were run "in our internal framework under the same evaluation settings," which is the right thing to disclose and also means the comparison is not independent. OpenAI's Terminal-Bench 4.0 figures and its cost-per-task claims come with the word "estimated" attached.

The training-target explanation for the 2.1 versus 4.0 gap is my reading, not a claim either lab makes. There are competing explanations that fit the same data. Terminal-Bench 3.0 and 4.0 may simply contain longer-horizon tasks where a model with 16B activated parameters at decode runs out of capacity regardless of what it was trained on. Both could be true at once, and I have no way to separate them from outside. What I can say is that the pattern of "wins the old version, loses the new one" recurs across three unrelated releases this week, and a purely capacity-based explanation does not obviously predict that shape.

I also could not verify the Terminal-Bench version history independently while writing this, so I am relying on the fact that both vendors report multiple versions in the same tables and treat higher numbers as newer and harder. If that ordering is wrong, the argument inverts.

And the CORE example is a single person's single run. Vergnes lists four things he never ablated, including peak learning rate and the GQA ratio, and says outright that he inherited them from nanochat rather than testing them. That honesty is why I trust the truncation finding, which he did investigate in detail. It is not why you should trust his numbers as a general result.

What to do with this

The benchmark table is not the answer to your question. It is somebody else's answer to somebody else's question, published by a party with a stake in how it reads.

You already know what your workload looks like. Take one real task from it, run it through the two models you are actually choosing between, count the tokens yourself, and look at the outputs. That takes an afternoon and it beats every leaderboard in the world, because it is the only evaluation that is aimed at you.

The version number on a public benchmark tells you how long the industry has had to shoot at it. The version number on your own eval tells you how long you have had to shoot at your own problem. Only one of those is worth optimizing.

Sources: DeepSeek-V4.1-Flash model card, GPT-6 Astra, Hugo Vergnes, "Training a 3.8B LLM to 0.384 CORE for $998".

Medium metadata

Title: Terminal-Bench 2.1 vs 4.0: The Benchmark Version Number Is Now More Informative Than the Score
Subtitle: A model released this week beats Claude Opus 5.0 on one version of an agent benchmark and loses to it by twenty points on another. Both numbers are real.
Tags: AI Benchmarks, Machine Learning, DeepSeek, AI Agents, LLM Evaluation
Canonical: import from the fervorai.dev URL