What "Comparable Performance" Actually Means in an AI Efficiency Claim
Three agent efficiency results landed in 48 hours. Read the shape of each comparison before you read the percentage.
NVIDIA's SoL-Pi paper says its harness "achieves performance comparable to Pi" while cutting token traffic by 44.7 to 49.0 percent. The project page for the same work, published alongside it, says SoL-Pi "retains roughly 94% of Pi's average score on both model backends."
Those are the same result. One of them is a number.
Six percentage points of score, given back in exchange for roughly half the tokens, is a good trade. I would take it on most workloads without thinking hard. But "comparable" and "retains 94%" do different work in a reader's head, and only one of them lets you decide whether the trade fits what you are building. The paper abstract chose the first. The blog chose the second. Nobody lied, and yet the version most people will quote is the one with no number in it.
This is worth dwelling on right now because three separate groups published agent efficiency results inside 48 hours this week, and all three are being summarized the same way on social feeds, as if they were the same kind of claim. They are not. They are three different comparison shapes. Learning to spot which shape you are looking at is the most portable skill in this whole news cycle, and it takes about thirty seconds per paper once you know what to check.
Why this matters more than it used to
For two years the interesting number in an AI release was a capability score. Bigger model, higher benchmark, done. Efficiency work sat in a side conversation, mostly among people running inference at scale.
That inverted this week. DeepSeek published DeepSeek-V4.1-Flash on September 17 and the abstract opens on long-horizon agents making workloads input-heavy, not on any benchmark. NVIDIA published SoL-Pi the same day, and it is entirely about spending less. Cactus put Needle 3 in front of Hacker News on September 18 with a pitch built around an 8 to 29 MB binary. Three teams, three layers of the stack, one shared premise: the agent's context is the bill, and the bill is now the design constraint.
When efficiency becomes the headline, the claim structure changes in a way that is easy to miss. A capability claim has one obvious failure mode, which is that the number is inflated. An efficiency claim has two. The savings can be inflated, and the thing being held constant can be loosened without anyone announcing it. Most of the reading energy goes to the first. Almost none goes to the second, which is where the interesting stuff hides.
The three shapes, using this week's three papers
Shape one: strictly better. DeepSeek-V4.1-Flash is a 552B-parameter multimodal Mixture-of-Experts with a million-token context. It activates 16B parameters per token during decode and only 8B during prefill. Cross-layer KV reuse in Compressed Sparse Attention 2, combined with FP4 KV caching, brings the global KV cache footprint that must live in HBM down to 890 bytes per token, about a quarter of DeepSeek-V4-Flash. A deployment technique called SWA Bounded Replay takes the persistent footprint on SSD or host memory to roughly an eighth.
And the abstract claims: "Despite its much smaller KV cache footprint, the model delivers substantially better performance than the baseline."
That is the strongest shape available. Less resource, more score, no give-back asserted. It is also the shape that should attract the most skepticism, because it is the hardest to achieve and the easiest to construct if you pick a weak baseline. The baseline here is DeepSeek's own previous model, which is a reasonable comparison and also an internal one.
Shape two: parity with an unstated give-back. SoL-Pi. Fourteen authors, arXiv on September 17, four mechanisms that survived an automated search process NVIDIA calls auto-research loops: Action Fusion, ObservationPack, an Evidence-Preserving Reducer, and Online Context Compact. Evaluated on EdgeBench, ByteDance Seed's public suite of long-horizon executable agent work, 51 tasks of it.
The abstract's word is "comparable." The project page's number is 94 percent retention. That gap is the whole lesson. A parity claim without a retention figure is not a parity claim, it is an efficiency claim with the cost of the efficiency left out of the sentence.
To NVIDIA's credit, three things here are genuinely well done, and I want to name them because the criticism is about phrasing, not conduct. EdgeBench is not theirs. They state explicitly that they reserved its tasks, verifiers and feedback for final held-out evaluation and did not use its data or verifiers as templates for environment synthesis, which is the discipline that makes the score mean something. They also note that on GPT-5.6 Sol the result exceeds that model's native Codex harness, which is the comparison an actual user cares about. And the dollar figures, $8.75 to $13.50 saved per hour against native Codex and Claude Code, are flagged as calculated using official API-equivalent pricing. That last one matters: if you have negotiated rates, the savings are not yours.
Shape three: a scoped win. Cactus Needle 3 does tool calls, structured extraction and text embedding, and refuses to chat. Every layer from 2 to 20 is a deployable subnetwork. On Mobile Actions, a 961-row phone-command suite scored on the exact call, the 20-layer model reaches 86.0 through the shipped 2-bit binary, against LFM2.5 1.2B at 82.4, Qwen3.5 0.8B at 76.0, and Apple's on-device model at 57.6, all of those at f16.
The claim getting quoted, though, is a different one: a 4-layer subnetwork passes DeepSeek V4 Flash. The page is honest about the conditions, stating that from 4 layers up "the tuned subnetwork passes DeepSeek V4 Flash" after fine-tuning on the downstream task, and that fine-tuning on DroidCall lifts every subnetwork by 18 to 36 points. So the real sentence is: a 29-million-parameter model, fine-tuned on one narrow task, beats a frontier model that was not fine-tuned on that task.
That is a genuinely useful fact and a completely different claim from "a 29M model beats DeepSeek." The scope is doing all the work, and the scope is the first thing that falls off in a repost.
Put this into practice
Four questions, in order. They take under a minute and they work on any efficiency claim, not just these three.
1. Find the retention number, or note that there isn't one. Search the paper, the abstract, the blog post and the README for a percentage attached to quality rather than cost. If the strongest quality word anywhere is "comparable," "matched," "on par" or "without sacrificing," and no figure appears next to it, treat that as an unstated give-back until you find the number. It is very often on the project page rather than in the abstract, which is exactly what happened here.
2. Ask what the cost baseline is denominated in. "One third less API cost" means one thing at list price and another at your price. Efficiency numbers priced at official rates are correct and are also the most flattering version available. If your organization has committed-spend discounts, recompute before you quote the dollar figure to anyone with a budget.
3. Check whether the benchmark was held out from the optimization. This is the question that separates real generalization from a tuned result, and it is usually answerable in one sentence of the methods section. SoL-Pi passes it explicitly. If a paper optimized a harness or a model against a benchmark and then reports a score on that benchmark, the number is a fit, not a finding.
4. Read the scope on every superlative. "Passes model X" almost always carries conditions: after fine-tuning, on one task family, at one quantization, with forced calls enabled. Find them. In Needle 3's case the conditions are printed right next to the claim, which is more than most releases manage.
If you want a single habit out of all four, make it this one: before you adopt an efficiency result, write down the sentence you would say to a colleague, and check that every qualifier in the source survived into your sentence. Most of the distortion in this space is not fraud. It is compression.
Honest limitations
This framing has real edges and I would rather name them than let you find them.
Retention percentages are not comparable across benchmarks. Ninety-four percent on a suite of two-to-twelve-hour long-horizon tasks, which is what EdgeBench measures, is a different animal from 94 percent on single-turn code completion. Long-horizon tasks compound, so a small per-step quality loss can show up as a much larger failure rate on the tasks you actually run. I do not know how SoL-Pi's six-point give-back distributes across task difficulty, and neither the paper abstract nor the project page breaks it out.
The four questions above also do nothing about the deeper problem, which is that none of this week's numbers are correctness numbers in the sense that matters operationally. They are benchmark scores. A harness that retains 94 percent of a benchmark score can still fail differently, and failure mode is not something a single percentage captures.
I have not run any of these three systems. Every figure here comes from the papers, project pages and product pages, cross-checked against each other, and the SoL-Pi retention figure specifically is NVIDIA reporting on NVIDIA's own evaluation. Nobody outside these teams has reproduced any of this work yet. That is normal for results a week old and it is still true.
And the shapes are not moral categories. A parity claim is not dishonest and an improvement claim is not automatically better. Shape two with a clearly stated 94 percent is more useful to me than shape one with a hand-picked baseline. The point is to know which one you are holding.
What to do with this
The next efficiency paper will land within a week, and it will get summarized into a percentage on a feed somewhere before most people read past the abstract. You can be the person in your team who checks the shape.
Start with the three from this week, because they are fresh and the sources are all public. Open the SoL-Pi project page, find the sentence with 94 percent in it, and compare it to the abstract's word. Open the Needle 3 page and find the conditions on the DeepSeek comparison. Open the DeepSeek abstract and decide for yourself whether "the baseline" is a fair fight. Twenty minutes, and the habit sticks.
Then apply it to whatever you were about to adopt.
Sources: SoL-Pi paper and project page, DeepSeek-V4.1-Flash, Cactus Needle 3, NVlabs/SoL-Pi.
Medium metadata
Title: What "Comparable Performance" Actually Means in an AI Efficiency Claim Subtitle: Three agent efficiency results landed in 48 hours. Read the shape of each comparison before you read the percentage. Tags: AI, Machine Learning, Software Engineering, AI Agents, Programming