Two API Settings Tripled a Benchmark Score. Nobody Touched the Model.

What OpenAI's ARC-AGI-3 rerun says about your own agent loop, and the two context settings worth checking today

A model scored 7.8% on a benchmark of 2D puzzle games. Same weights, same games, different harness, and it scored 38.3%. No fine-tuning, no new checkpoint, no prompt engineering contest. Two settings on an API.

That is the result OpenAI published on July 29 after digging into why GPT-5.6 Sol looked bad on ARC-AGI-3, a benchmark where an agent has to figure out how an unfamiliar game works without being told the rules. The model that had beaten Pokémon FireRed and Slay the Spire 2 was, in ARC's official harness, "dwelling a long time on each action and struggling to make progress," in OpenAI's own description. It looked slow. It looked confused. It looked like a model that could not learn.

It was a model that could not remember.

Capability moved into the scaffolding, and most teams have not repriced it

Here is the position, and it is not a subtle one: if two configuration flags can triple a score, then a lot of teams are running agent loops that are leaving real capability on the floor and blaming the model for it.

That is an uncomfortable claim to sit with when your quarter's plan is "wait for the next release." The bill for a bad context policy does not arrive as an error. It arrives as an agent that seems a bit dim, needs more turns than you expected, and costs more per task than the pricing page suggested. Every one of those symptoms reads as a model problem. Some of them are a memory problem you configured yourself.

The mechanism, which is more interesting than the number

Two things were happening inside the official harness.

First, after every game action, the model's private reasoning was thrown away. GPT-5.6 Sol could still see the record of what it had done and short notes about it, but not the thinking that produced those moves. Picture keeping a log that says "moved the purple block left" with no record of why purple mattered. Every turn, the model had to reconstruct its own theory of the game from the wreckage of its notes.

Second, the harness handled context limits with rolling truncation. Past 175,000 characters, the oldest messages were dropped. So the model was losing its reasoning turn by turn, and then losing the actions too.

OpenAI reimplemented the harness on its Responses API and changed exactly two things. Passing the previous response ID keeps reasoning attached across tool calls and turns, so the model's own working notes survive. Then compaction replaced rolling truncation, which means older context gets summarized rather than deleted.

Both changes point at the same failure. The result is what makes it worth your attention: roughly 3x the score at 6x fewer output tokens. Not a tradeoff. The model got better and cheaper at the same time, because it stopped paying repeatedly for conclusions it had already reached. There is a secondary effect OpenAI names too, which is that an agent running most of a task inside a nearly-full context window performs slightly worse for that reason alone. Truncation keeps you pinned at the ceiling. Compaction pulls you off it.

On one of the games, task cd82, no frontier model on the leaderboard clears a single level past the first. With the reimplemented harness, GPT-5.6 Sol finished all six.

The fight this started, because it matters for how you read every score

ARC Prize responded publicly and did not simply concede. Its verified scores use a deliberately generic no-harness setup, so that no lab can tune scaffolding to its own model's quirks: same observations, same system prompt, same action limits for everyone, with conversation state managed client-side through the completions-style interface that works across providers. Once labs start optimizing the harness, cross-model comparison dies.

Then ARC co-founder François Chollet drew a line that is genuinely useful. Harnesses built for the benchmark, or that encode knowledge of the benchmark's format, are out of bounds. General-purpose API settings that were not designed for ARC-AGI-3 and are available to every API customer are fair. By that standard, retained reasoning and compaction are fair, and ARC's own number for GPT-5.6 Sol undersold it. Chollet said as much, called different providers using different settings "a potential parity issue," and said he considers that acceptable as long as the settings and the cost are reported clearly. ARC says it is working with labs, OpenAI included, on how to fold server-side state management into verified testing.

Both parties are right about different things, which is what makes this worth reading twice. Yes, a generic harness protects comparability. Yes, a harness that discards reasoning measures something no production system does.

Put this into practice

You do not need an eval suite or a research budget for this. You need to know two facts about your own loop.

Find out whether you drop reasoning between steps. Open your agent loop and look at what you send on turn two. If you rebuild a message array from scratch each turn, or you store only user-visible text and tool results, the model's reasoning is on the floor. On the Responses API, pass the previous response ID and it persists. On other providers, check whether reasoning blocks are returned and whether you are dropping them on the way back in.

Find your truncation code and change what it does. Search for the place where you trim history to fit the window. If it slices off the oldest messages, that is rolling truncation and it silently deletes the beginning of the task, which is usually where the constraints live. Replace it with summarization: compaction if your provider offers it, or a summarize-then-continue step you write yourself. A worse summary beats a clean deletion.

Re-run your own eval before and after, and watch both axes. Score and output tokens. If the score moves, your model was never the bottleneck. If tokens drop while the score holds, you just found budget. Either outcome is worth the hour.

Log the context policy alongside every result. Whatever number you report internally, record which policy produced it. A score without its context policy attached is a number that cannot be compared to anything, including your own result from last month.

Do the cheap one first. Retaining reasoning is usually a one-line change and it is where the larger part of the gain lives in this case.

Honest limitations

The 38.3% is OpenAI's own figure, on the public set, from OpenAI's reimplementation of someone else's harness. It has not been replicated on the private set, where ARC's verified figure for the model stands at 7.78% across 25 environments. The comparison to Opus 5's 30.2% record is not effort-matched either: Sol's official 7.8% was run at the Max reasoning tier, Opus 5's 30.2% at High. Treat "beats Opus 5" as an unsettled claim, and treat the 13.3%-to-38.3% delta inside one harness family as the part with real evidence behind it.

The human baseline is an estimate. OpenAI puts the average human tester near 48% based on published gameplay logs, and the metric itself, Relative Human Action Efficiency, is scored against that baseline rather than being a raw solve rate.

Compaction is lossy. You are trading a deletion you understand for a summary you did not write, produced by the same class of model that will act on it. That is usually the better trade, and it is still a trade. Test the tasks where a dropped constraint would hurt you most.

Retained reasoning keeps the bad conclusions too. If a model talks itself into a wrong theory on turn three, that theory now rides along. Worse, if untrusted content reached the model and shaped its reasoning, that influence persists instead of being flushed. Retention is a memory feature, and memory holds whatever got in.

And server-side conversation state moves your agent's working memory into the vendor's custody. That is precisely why ARC keeps state client-side: portability. It cuts the other way too. MCP's 2026-07-28 revision went in the opposite direction, deleting the session from the transport and rebuilding it as an explicit handle threaded through tool arguments. One camp is moving state toward the provider, the other away from it, and you are picking a side whether or not you notice.

Cost changes shape. Fewer output tokens is a real saving, and longer retained state alters cache behavior and per-request input cost. Read your usage page, not the press release.

What to do with this

The useful takeaway is not that OpenAI won an argument with ARC Prize. It is that a widely-cited eval was measuring a memory policy and reporting it as intelligence, and nobody caught it for weeks, including the model's own maker.

Your agent loop has a context policy right now. You may not have chosen it. It may be whatever the framework you adopted does by default, or a truncation function someone wrote to stop a crash. Go read it. Then run your eval twice, once with reasoning retained and truncation replaced, and see which of your model complaints survive.

If the numbers move, you learned something about your harness. If they do not, you have earned the right to blame the model.

Sources: OpenAI, "How enabling two settings tripled our scores on the ARC-AGI-3 benchmark" (July 29, 2026); ARC Prize response and François Chollet on harness fairness; ARC-AGI-3 overview and methodology; ARC Prize verified Claude Opus 5 results; The Decoder coverage, updated July 30.