SlopCodeBench Puts a Number on What Your Coding Agent Leaves Behind

Verbosity and erosion score the mess an agent makes, the ranking flips against the correctness leaderboard, and you can run both metrics on your own repository this week

There is a public leaderboard where the model that writes the cleanest code is not the model that solves the most problems, and the gap between those two rankings is wider than the gap between most model releases.

On the SlopCodeBench board, GPT 5.4 running in Codex posts a 25.5% isolated solve rate with an erosion score of 0.278 and a verbosity score of 0.193. GPT 5.2-Codex, one spot below it, posts 21.9% with an erosion of 0.728 and a verbosity of 0.398. The higher-ranked model is more correct and leaves behind less than half the structural damage. Two Opus entries sit at an identical 20.9% solve rate with erosion scores of 0.737 and 0.759, which means the newer model bought you nothing on correctness and cost you something on cleanliness. None of the benchmarks anyone quotes in a launch post would surface either fact.

That is the useful thing about SlopCodeBench. It is not another correctness board. It measures the second thing, the thing every engineer running agents has complained about for two years without being able to put a number on it.

The setup that hides the problem

Almost every agentic coding benchmark is single-shot. You hand the agent a complete specification, it writes a solution, hidden tests grade it. That format has a clean reward signal, which is exactly why coding became the first domain where models got reliably good. It is also why the benchmarks cannot see design decisions. A one-shot task never asks the agent to live with a choice it made twenty minutes ago.

SlopCodeBench inverts that. The benchmark, from Gabriel Orlanski, Devjeet Roy and eight co-authors across UW-Madison, Washington State and MIT, runs 36 problems across 196 checkpoints where the agent repeatedly extends its own earlier solution as new requirements arrive. Evaluation is black-box: the problem gives a CLI or API contract and nothing else, no prescribed architecture, no function signatures, no module boundaries. Early decisions compound because the agent has to keep building on them.

The result the paper reports for that setup is stark. Across 15 coding agents on open and closed models, no agent fully solved any problem end to end, and the best one passed 14.8% of checkpoints. Structural erosion rose across 77% of trajectories and verbosity across 75.5%. Compared against 473 open-source Python repositories, agent code came out 2.3 times more verbose and 2.0 times more eroded, and the human repositories degraded less often and by smaller margins across their own git histories.

Worth separating two numbers here, because they get conflated fast. The paper's own results table carries both as separate columns for the same model: a strict score, which is 14.8% for GPT 5.5, and an isolated score, which is 28.1%. Strict means every test passed at every checkpoint. Isolated means each checkpoint graded on its own. The live leaderboard at scbench.ai surfaces the isolated number, and it is now at v1.0 with 19 models against the paper's 15. Quote whichever one you mean and say which it is.

What the two metrics actually compute

Both are simple enough to reimplement, which is most of their value.

Verbosity is the count of lines flagged as duplicated or needlessly verbose, divided by total lines of code. The flagging comes from AST-Grep rules plus clone detection, so it is structural rather than textual: it catches the same logic written twice in different variable names, not just copy-pasted blocks.

Erosion measures how much of a codebase's weight sits in functions that have gotten out of hand. Each function gets a mass, defined as its cyclomatic complexity multiplied by the square root of its source lines. Erosion is the summed mass of every function with complexity above 10, divided by the summed mass of all functions. A codebase where the hard logic is spread across many small functions scores low. One where three monsters carry everything scores high.

Sebastian at Earendil, the company that ships the Pi agent harness, recomputed both on SlopCodeBench evaluation output and published the comparison on September 10. Established repositories averaged 0.15 verbosity and 0.31 erosion. Agent-generated code averaged 0.33 and 0.68. His ratios come out near 2.2 times on both axes, which does not exactly match the paper's 2.3 and 2.0, and that small disagreement is a feature: two parties computing the same metric on overlapping data landed close but not identical, which is what an honest metric looks like.

His post is also worth reading for what it rules out. The industry default for judging code quality is to ask a model. Scoring code 1 to 10 with an LLM, he writes, is "basically equivalent to a random number generator," and pairwise A-versus-B judging flips its preference when you rename the solutions. Human review is the real answer and does not scale to benchmarking every model against every harness. That is the gap verbosity and erosion are trying to fill: not perfect, but computable, and the same number every time you run it.

Put this into practice

The lowest-friction version of this takes an afternoon and needs nothing you do not already have.

Step one, get a baseline. Pick a repository with real history. Run a cyclomatic complexity tool over it (radon cc for Python, lizard for most other languages, gocyclo for Go) and compute the erosion ratio: total mass of functions above complexity 10, over total mass of all functions, where mass is complexity times the square root of source lines. That is roughly ten lines of script around the tool's output. Write the number down.

Step two, measure a session. Run your agent on a real feature, not a toy. Compute erosion again on the diff's blast radius, meaning the files it touched rather than the whole repo, so one change does not disappear into the average. Now "the agent made a mess" is a delta instead of a feeling, and you can argue about it in a pull request.

Step three, add verbosity if you use AST-Grep. Write rules for the duplication patterns your codebase actually suffers from. Do not start from someone else's rule set. The verbosity metric is a set of handcrafted heuristics by construction, which means it only measures what you tell it to look for, and the rules are where your team's taste gets encoded.

Step four, try explicit quality guidance and measure whether it helps. This is the paper's most actionable finding and its most sobering. Telling the agent up front to keep the code clean reduced initial verbosity and erosion by up to a third. It did not change the degradation rate. The code started better and got worse at the same speed. So a quality preamble in your agent's instructions is worth adding, and it is not a fix.

Where this breaks

The honest limitations are substantial, and they matter more than usual because these metrics are new enough that people will over-apply them.

The Goodhart problem is immediate and Earendil names it directly: start optimizing for verbosity or erosion and both stop measuring anything. An agent told to minimize erosion will split one coherent function into six incoherent ones and score beautifully. These are diagnostics for a human to interpret, not reward signals for a training loop and not merge gates.

The comparison corpus is 473 open-source Python repositories. Whether the 2x ratio holds for Rust, or for a mature Java enterprise codebase, or for anything written under a style guide the metrics were not tuned against, is unproven. Erosion's complexity-above-10 threshold is a convention, not a law of nature.

The metrics also miss most of what makes code bad. Neither one measures coupling between functions, cohesion within a module, naming, or whether the abstraction matches the problem. Earendil lists coupledness, churn and cohesion as directions still to explore, and notes one "notoriously vibey" open project that scored fine on both metrics anyway, probably because heavy coupling and a mass of unrelated functions dragged the averages down. A good score is not a clean bill of health.

Finally, the timing. The paper's v1 went up on 25 March 2026 and v2 on 7 May. The footnote on Earendil's post says the runs did not include Fable 5.1 or Astra. Every ratio in this article predates the current model generation, and the leaderboard's top entries are Codex and Claude Code pairings that have since been superseded. The ratios will move. The measurement will not.

What to do with this

The pitch is not that you should adopt SlopCodeBench's numbers as your quality standard. It is that there are now two axes where most teams have been tracking one, and the leaderboard proves they come apart. If you are choosing a model or a harness on solve rate alone, you are picking blind on the axis that decides whether anyone can maintain the output in six months.

Run the erosion number on your own repository before your next long agent session. It costs an afternoon. Either the number moves and you have learned something concrete about the tool you are already paying for, or it does not and you can stop worrying about it with evidence instead of vibes. The benchmark authors put the code, the problems and the leaderboard in the open specifically so you do not have to take their word for any of it.

Sources: SlopCodeBench (arXiv:2603.24755) · SlopCodeBench leaderboard · SprocketLab/slop-code-bench · Earendil, "Measuring the sloppiness of code"

Medium metadata

Title: SlopCodeBench Puts a Number on What Your Coding Agent Leaves Behind
Subtitle: Verbosity and erosion score the mess an agent makes, the ranking flips against the correctness leaderboard, and you can run both metrics on your own repository this week
Tags: AI Agents, Software Engineering, Coding Assistants, Benchmarks, Code Quality
Suggested kicker: Two axes, not one
Canonical: import from the fervorai.dev URL after publish