Prime Intellect Ran 153 Autonomous Research Agents. The Ones That Won Measured the Noise First
Eighteen frontier models got the same task, the same eight GPUs, and no internet. They found nearly identical ideas. What separated first place from last was experiment protocol, and that part you can copy today.
Nobody invented anything.
That is the finding buried inside Prime Intellect's NanoGPT Speedrun Frontier, a set of 153 autonomous runs across 18 frontier models, each one turned loose on an 8xH200 node for up to eight and a half days with a single instruction: make this training run finish in fewer steps. The write-up says it plainly. "None of the runs produced a fundamentally new method; the winning ingredients are all similar to existing ones in the literature."
And yet the spread between models is enormous. Claude Fable 5 closed 81.7 percent of the gap to the human record. GPT-5.5 closed 8.1 percent. Same task, same node, same rulebook, roughly the same pile of ideas available to all of them. So where did a ten-fold difference in outcome come from?
Not from taste, exactly. From bookkeeping.
Why this is the most useful agent result of the month
Most agent evaluations tell you which model is better and leave you nothing to do about it. This one published the trajectories, and they show the losing behavior in enough detail that you can go check whether your own setup does the same thing.
Here is the position I'll defend for the rest of this piece: on any long-horizon task where results are noisy, the model's ideas are close to a commodity and the model's screening protocol is the product. If that's true, then a chunk of the gap between an expensive agent run and a cheap one is not intelligence you have to buy. It's discipline you can specify.
The setup, and the one number they got wrong
The speedrun trains a 124M-parameter GPT and counts the steps needed to reach a validation loss of 3.28. Prime Intellect's verified baseline sits at 3,290 steps. The human record claim, sitting in an open pull request upstream, is 2,600. Agents start at the baseline, know a better method exists, and have to find everything below it themselves.
Each run got the repo, a rulebook called program.md, and one injected goal prompt: read the rulebook, run fully autonomously, never ask for input, reach the loss in the fewest steps possible. The sandbox was bwrap plus a network namespace, and the only route outside was a logging proxy that allowed the model's own API and nothing else. No internet, no arXiv, no reading other people's pull requests. Prime Intellect says that restriction made models slightly more creative, not less.
To claim a record, a model had to run its recipe eight times on fixed seeds it could not touch, and a frozen verify.py accepted the claim only if the eight-run mean beat 3.27859. That margin makes passing on luck alone roughly one in a thousand.
Now the interesting part. Prime Intellect wrote an estimate of the benchmark's noise into program.md, and that estimate was slightly too large. Sixty-two of about a hundred runs did not trust it. They measured the noise themselves. In Prime Intellect's own words, "these runs are concentrated at the top of the results table."
Forty-two went further and found something the rulebook never mentioned: rerunning the identical recipe on the identical seed still moves the loss, because GPUs are not deterministic. That noise is much smaller than seed-to-seed noise, which means a model that discovers it can compare two recipes on a shared seed and resolve differences a normal screen cannot, at the same cost. Several models rebuilt their entire screening protocol around that one observation.
Read that again as an operational fact. The single most valuable move available to an autonomous researcher in this environment was to stop believing a number in its instructions file and go measure it.
What losing looks like, in detail
Prime Intellect's description of weaker runs reads like a code review of every rushed experiment I've ever run.
They kill whole families of ideas on a single seed. They treat their own crashes as evidence the idea was bad. They discard small gains that don't clear the bar on their own, which is fatal in a domain where the winning recipe is a stack of small gains that only work together. Grok 4.5 lost row normalization twice, both times because of its own scaling bugs, not because the idea failed.
The strong runs behave differently in ways that are almost boring to describe. They test borderline results on three seeds instead of one, and only pay for eight when their own noise model says the confirmation is worth it. After every merge they re-ablate the whole stack and drop whatever stopped helping. And they go back to old negatives, because a mechanism that did nothing under the previous recipe might matter under the new one.
The examples are specific. Opus 5 reopened β2 tuning under a revised recipe and it became a new record. Kimi K3 deleted two mechanisms that had produced its own previous record, after a new normalization made them redundant. When Fable ran out of gains from single knobs, it started testing pairs that were individually worse but jointly better, and one late re-probe of an old idea was worth thirty-one steps.
None of that is a capability you can only get by paying more per token. It's a policy.
The cost column nobody usually prints
Prime Intellect published token spend next to score, and the table is uncomfortable reading if you assume more spending buys more progress.
| Model | Harness | Gap closed | Total tokens | Experiments | Agent days |
|---|---|---|---|---|---|
| Fable 5 | claude-code · high | 81.7% | 800M | 811 | 8.7 |
| Opus 5 | claude-code · max | 53.6% | 183M | 292 | 2.9 |
| GPT-5.6 Sol | codex · xhigh | 35.9% | 2.9B | 963 | 6.1 |
| Grok 4.5 | grok-cli · xhigh | 24.6% | 46M | 399 | 2.7 |
| DeepSeek V4 Pro | claude-code · max | 12.3% | 26M | 189 | 1.1 |
GPT-5.6 Sol spent sixty-three times Grok 4.5's tokens and twenty-eight thousand tool calls to gain about eleven percentage points. Opus 5 got more than half the gap closed on 183M tokens, less than a sixteenth of Sol's spend. Fable led the table on 800M, which is a lot in absolute terms and about a quarter of what the sixth-place run burned.
Before anyone concludes that budget is the whole story, Prime Intellect built an equal-budget view that lets you re-rank every model by agent-hours, experiments, or output tokens, and their own conclusion cuts the other way: "Fable and Opus 5 lead however the budget is measured, and swapping hours for experiments barely changes the order, so the gap is not only about volume."
Both things are true. Spending more does not reliably buy you rank, and the models at the top would still be at the top on a tight budget. The variable that survives every normalization is what an experiment leaves behind.
Put this into practice
You don't need eight H200s to use any of this. You need an eval you already run and about an afternoon.
1. Measure your own noise before you trust a single delta. Run your evaluation on the same input, unchanged, five to ten times. Record the spread. That number is the floor below which every "improvement" you see is weather. Most teams I've talked to have never done this, and most agent instruction files contain a confidence threshold somebody guessed.
2. Put the measured number in the instructions file, not an estimate. This is the direct lesson from the 62 runs. If your agent is told "trust gains above 0.5 percent" and the real run-to-run spread is 1.2 percent, you have automated the acceptance of noise. Write the measured figure in, and re-measure it when the harness changes.
3. Split screening from confirmation, explicitly. One cheap run to screen, N runs to promote. Opus 5's trace shows exactly this policy written down in the agent's own scratchpad: trust gains above a threshold, confirm borderline wins with a second draw, let the eight-trial mean be the only thing that can declare a record. Two tiers, two different bars, stated in the prompt.
4. Hold one variable fixed to shrink the noise. The GPU non-determinism finding generalizes. Whenever you can compare two candidates against the same seed, same fixture, same cached context, do it, because the paired comparison resolves differences the unpaired one buries. This is the cheapest accuracy gain in the whole list.
5. Re-test your negatives after every accepted change. Keep a list of rejected ideas with the recipe version they were rejected under. When the stack shifts, replay the list. Fable's thirty-one-step win came from exactly this, and it's the step teams skip because rejected ideas feel finished.
6. Log tokens per accepted change, not tokens per run. A run that burned 2.9B tokens is not automatically a bad run. A run that burned 2.9B tokens across 963 experiments to accept a handful of them is telling you the screening bar was set wrong. That ratio is the number worth putting on a dashboard.
Steps one and two take an afternoon. Step five is a habit, not a task, and it's the one that pays for a month.
Honest limitations
This is one task in one domain, and the domain is unusually friendly to hill-climbing. Optimizer search on a fixed training script has a tight feedback loop and a scalar objective. Prime Intellect says so themselves: they don't have strong conviction that methods developed in this kind of speedrun are inherently scalable or would be used in real model training.
The benchmark is also loud. Their own stated variance is that two runs of the same model and harness land about 54 steps apart at 24 agent-hours, 43 apart at 100 experiments, and 40 apart at 300k output tokens. Against a total baseline-to-human span of 690 steps, that is a meaningful chunk of the scoreboard. They launched at least three seeds per model, compared after roughly 24 hours, and continued only the most promising one, which is a reasonable compute compromise and also a selection procedure that flatters every model's best number.
Model and harness are confounded. Fable, Opus 5 and Sonnet 5 ran on claude-code; the GPT-5.6 family ran on codex; Grok ran on grok-cli; Kimi K3 appears twice, once on kimi-code and once on Prime Intellect's own Prime Agent, with a seven-point difference between the two. You cannot cleanly separate "this model is better at research" from "this harness suited this model."
Several entries in the table are still marked running, and one, GLM 5.3, has no record at all yet. The ordering is provisional.
Prime Intellect sells compute and builds Prime Agent, which appears in the results as the third-place harness. They also stopped their independent hourly cheating monitor partway through, after hundreds of clean reports, and moved to checking during trace export. Both are defensible calls, and both are worth knowing before you quote the table as neutral ground truth. To their credit, the traces, scratchpads, monitor reports, per-run ledger and harness are all public, so the correction path is open to anyone willing to read.
One more thing that cuts against my own argument: the small internal inconsistency between "153 runs" in the headline and "62 out of ~100 runs" in the noise paragraph suggests the noise-measurement count covers a subset the write-up doesn't fully define. The direction of the finding is clear. The exact denominator isn't.
What I'd watch next
The result I keep turning over is the one about novelty. Eighteen frontier models, 153 runs, up to eight and a half days each, complete freedom over the method, and not one genuinely new idea. Prime Intellect flags this as the thing they were most surprised by, twice, and admits they can't tell whether it's an artifact of the speedrun setup or a real limit.
If it's the setup, the fix is more open-ended benchmarks and we'll know within a few months. If it's a real limit, then the thing these systems are extremely good at is running a disciplined search over a space someone else defined, and every deployment decision should follow from that sentence.
Either way, the practical move is the same and it's available this week. Go measure the noise in your own eval. If the number surprises you, everything your agents accepted last month was decided by a threshold nobody checked.
If you run that experiment, I want to hear what your spread turned out to be. My guess is it's wider than the threshold sitting in your config file.
Sources: NanoGPT Speedrun Frontier and Measuring Autonomous AI Research by Elie Bakouch and Prime Intellect, August 14 2026; the shared research repository. Related prior work referenced by the authors: METR's expenditure-horizon study and Intology's NanoGPT-Bench. All figures quoted here were read from the live results table and blog post on August 23 2026.