Real-SWE Is a Coding Benchmark You Are Not Allowed to See, and That Is the Point

A new benchmark runs frontier coding agents against private production codebases. The top score is 38.8%, the tasks are behind a request form, and both of those facts follow from the same design decision.

The best coding agent in the world resolved 38.8% of the tasks. Not on a hard research benchmark. On ten ordinary tickets from real companies: fix the invoice tax calculation, migrate a customer identity system, meter API tokens correctly. One task, an analytics stream reducer, was solved zero times across all 64 of its attempts by any of the eight agents tested.

Those numbers come from Real-SWE, released by Specific Labs in September 2026 and picked up on Hacker News on September 12 with 246 points. Before you argue about whether 38.8% is good, look at where the tasks come from, because the sourcing is the actual claim and everything else follows from it.

The contamination problem, stated plainly

A benchmark works by being a test the system has not already seen. Publish a benchmark and it goes onto the public internet. The next pretraining run scrapes the public internet. Now the answers are inside the model, and the score is measuring recall as much as capability.

Everyone in the field knows this. The usual responses are to build new tasks, to hold out a test split, or to argue that contamination is bounded and the numbers are still directionally useful. Real-SWE takes the harder option: license private production codebases from real businesses and build the tasks out of work their engineers actually did.

Specific states the reasoning without hedging. Tasks on private codebases are natively out of distribution, because this code is not available anywhere on the internet and is unlikely to have been trained on. They put a figure on the scale of what public benchmarks miss: 99% of tokens in real-world enterprises are hidden from frontier models. That figure is an assertion in their own post, not a cited measurement, and I would not build a plan on it. The directional claim underneath it is hard to dispute.

The codebases they picked are described but not named: a Luma and Partiful competitor with more than 200,000 users and a top-100 App Store ranking, a consumer fintech platform processing over 100,000 bank statements, and enterprise AI sales platforms.

What the scores say

Eight model-and-harness pairings, ten tasks, eight rollouts per task, 640 scored rollouts total. They evaluated combinations rather than models in isolation, on the reasoning that enterprise engineers use native harnesses and that is the real unit of deployment.

Rank	Model	Harness	Resolution	Cost per rollout
1	Fable 5.1	Claude Code	38.8%	$6.96
2	GPT-6 Astra	Codex CLI	33.8%	$4.67
3	Gemini 3.8 Flash	Gemini CLI	31.2%	$2.50
4	GLM 5.3	Claude Code	28.8%	$5.12
=5	Grok 4.6	Grok Build	23.8%	$3.44
=5	Muse Spark 1.3	Muse Code	23.8%	$2.74
7	Kimi K3	Kimi Code	18.8%	$3.90
8	GPT-5.6 Sol	Codex CLI	16.2%	$2.65

Resolution rate here is pass@1 averaged over eight independent runs per task.

Two things in that table deserve more attention than the ordering. First, cost does not buy correctness in any reliable way: Gemini 3.8 Flash lands third at $2.50 per rollout while GLM 5.3 lands fourth at more than double that. Second, six of the ten tasks resolve below 15%, and the difficulty is not evenly spread. Multi-region sweep resolves at 67.2%. Analytics stream reducer resolves at 0.0%. A benchmark where most of the mass sits at the bottom is measuring something real, but it is also fragile, because swapping two hard tasks for two medium ones moves every number on the board.

The failure taxonomy is more useful to a working engineer than the leaderboard is. Specific classified every failed run using the same categories across models, following DeepSWE's scheme, and the models fail in genuinely different ways. Grok 4.6 missed a required behavior in 67.2% of its failures. Kimi K3, 53.8%. On the other axis, GPT-5.6 Sol built on an unverified guess about the system in 43.3% of its failures, the worst on that measure, with GPT-6 Astra at 34.0%. Gemini 3.8 Flash barely guesses at all, 10.9%, but wires the right idea into the surrounding system incorrectly 49.1% of the time.

Those are three distinct engineering problems wearing one label. "The agent failed" tells you nothing. "The agent assumed the config key existed instead of grepping for it" tells you to change your prompt. "The agent wrote the right function and called it from the wrong place" tells you to change your review.

And the tasks are genuinely bigger than what public suites ask for. The reference solutions touch a median of 11 files, against 6 for both FrontierCode and DeepSWE. Instruction length is middling, a median of 1,742 characters against FrontierCode's 2,056 and Terminal-Bench 3's 1,584, which means the tasks are not underspecified so much as wide.

One finding cuts against a comfortable assumption. Letting the agent run longer did not help. Rollouts under ten minutes failed 71.4% of the time, 70 out of 98. Rollouts of ten minutes or longer failed 73.4% of the time, 398 out of 542. If your mitigation for a hard ticket is more thinking time, this data says you are buying nothing.

The trap in the design

Here is where the honest version of this article has to turn on the benchmark it just praised.

If a benchmark is only valid while it is private, then a valid benchmark cannot be audited. Those are the same sentence. Real-SWE's tasks sit behind a request-access form asking for your name, work email, company, role and intended use, and what you can request is a sample, not the suite. The page is marked noindex.

So a reader is being asked to accept a leaderboard produced by a company that sells into the market it is ranking, on tasks nobody outside can inspect, graded by verifiers nobody outside can inspect, with n=8 per cell on ten tasks. Every one of those is defensible on its own. Stacked, they add up to a number you cannot check.

I do not think Specific is acting in bad faith. Their methodology disclosure is better than most: they show per-task per-model pass counts, publish 95% confidence intervals on the headline rates, name the taxonomy they borrowed, flag which cost figures are incomplete, and state that tasks are in Harbor format with verifiers injected at grading time. That is more transparency than several vendor benchmarks published this year.

The problem is structural, not moral. Contamination resistance and auditability are in direct tension, and no amount of good faith dissolves that. A public benchmark is checkable and decays. A private benchmark resists decay and is uncheckable. There is no third option available to anyone publishing for a general audience.

Which means the conclusion is not "trust Real-SWE" or "ignore Real-SWE." It is that external benchmarks have a ceiling on what they can tell you about your codebase, and the ceiling just became visible.

Put this into practice

Build a held-out set of your own. It is smaller work than it sounds, and it is the only version of this that is actually yours.

Start with ten closed tickets. Go to your issue tracker and pull ten items your team finished in the last quarter. Pick ones with a real diff attached and a test that passes now. Do not pick the interesting ones; pick the median ones. The value is in representativeness.

Pin the commit before the fix. For each ticket, record the parent commit SHA. That is your starting state. The diff that closed the ticket is your reference solution, and the tests that shipped with it are your verifier. You already have all three artifacts, which is the whole trick: your tracker has been generating benchmark tasks for years.

Write the instruction from the ticket, not from the diff. Use the language the original reporter used. If you write the prompt after reading the solution, you will leak the answer into it without noticing. Specific's tasks run a median of 1,742 characters, which is a couple of paragraphs, not a spec.

Keep it off the internet. Private repo, no gists, no pasting it into a public issue to ask for help. The day it leaks is the day it stops measuring.

Run it on every model bump, and log the failure mode, not just the pass rate. Use Specific's five categories, since they are already defined and someone else did the taxonomy work: unverified assumption, missed requirement, integration error, regression, wrong file. After three or four runs you will know which failure your stack is actually prone to, which is the thing that changes what you do on Monday.

A concrete starting shape, whatever your runner:

tasks/
  TICKET-4412/
    base_commit          # SHA before the fix
    instruction.md       # the ticket text, written before anyone read the diff
    verify.sh            # the tests that shipped with the original fix
    reference.patch      # the diff that closed it, for your eyes only

Eight rollouts per task, like Real-SWE, if you can afford it. Three if you cannot. One rollout tells you almost nothing, because the variance on these tasks is large enough that a single run is a coin flip with extra steps.

One thing not to do: do not tune your agent against this set. The moment you start iterating on prompts until the numbers go up, you have converted your evaluation into a training target and Goodhart's law takes it from there. Keep a separate scratch set for tuning.

Honest limitations

Real-SWE is ten tasks. Ten. With eight rollouts each, the confidence intervals on the headline rates overlap considerably between adjacent ranks, which Specific shows on the chart. The 38.8% versus 33.8% gap between first and second is worth roughly half a task. Do not read the ordering as a ranking of models.

The harness is entangled with the model throughout. Fable 5.1 was run in Claude Code and GLM 5.3 was also run in Claude Code, so you can partially separate them there, but nowhere else. GPT-6 Astra and GPT-5.6 Sol both ran in Codex CLI and finished 17.6 points apart, which tells you the model matters, and tells you nothing about how Astra would do in a different harness.

The cost figures carry a caveat Specific prints themselves: Grok 4.6 and Kimi K3 report incomplete usage data, so their actual cost may be higher than the $3.44 and $3.90 listed.

The vendor sells in the market it measured. That does not make the data wrong. It does mean an independent replication would change how much weight the numbers deserve, and no independent replication is possible by construction.

The comparison figures are borrowed. The FrontierCode and DeepSWE file-count medians come from Cognition's published comparison rather than Specific's own measurement, which Specific footnotes.

And I have not run it. Nobody outside Specific has run the full suite. That is not a criticism I can level at them without also leveling it at the design, which is the point of this article.

What to actually do with this

The useful reading of Real-SWE is not the leaderboard. It is the failure taxonomy and the ten-minute finding, both of which you can act on tomorrow without access to a single one of their tasks.

Your agent probably fails in one dominant mode. Find out which. If it is unverified assumptions, the fix is a prompt that requires the agent to read before it writes, and a reviewer that checks whether it did. If it is integration errors, the fix is a wider test net, not a better model. If it is missed requirements, the fix is your ticket template.

And stop waiting for someone to publish the number that settles it. The benchmark that tells you whether an agent can do your team's work is a benchmark built from your team's work. Ten closed tickets and an afternoon. The infrastructure has existed in your issue tracker the entire time.

Sources: Real-SWE and its sample access page, both read 2026-09-13; the failure taxonomy follows DeepSWE; the FrontierCode and DeepSWE file-count medians are attributed by Specific to Cognition's published comparison.

Medium metadata

Title: Real-SWE Is a Coding Benchmark You Are Not Allowed to See, and That Is the Point
Subtitle: A new benchmark runs frontier coding agents against private production codebases. The top score is 38.8%, the tasks are behind a request form, and both of those facts follow from the same design decision.
Tags: AI Agents, Benchmarks, Software Engineering, Developer Tools, LLM Evaluation
Suggested kicker image: a sealed envelope on an otherwise empty leaderboard
Canonical: import from the fervorai.dev URL