Fresh-Context Review: The Agent That Wrote Your Code Is the Worst Judge of It

Three agent projects shipped the same separation rule this week, and a study across seven model families explains what breaks when you skip it

Most agent setups ask one context window to hold three jobs at once. It takes the requirements, writes the code, then decides whether the code is any good. Nobody sits down and designs it that way. It happens because a chat session is a single thread, and typing "does this look right?" at the end of that thread costs nothing and feels like a review step.

It is not a review step. It is the author grading their own exam using the answer key they wrote an hour earlier.

Three separate projects on this week's GitHub momentum boards built a fix for that into their architecture, apparently without coordinating. A solo developer's Codex plugin makes a fresh model instance approve every diff before the session is allowed to say "done." A bug-bounty platform behind $1.5M in payouts routes every finding through a validation stage that runs scripts instead of asking the finder to double-check. Y Combinator's internal harness gives every person and every room a separate memory, permission set, and sandbox. Different problems, same underlying rule: the thing that produced the work does not get to certify it.

The position: your review pass is probably not one

Here is the claim I will defend. If your agent workflow ends with the same session that wrote the code saying the code looks correct, you have zero review coverage, not weak review coverage. And the failure mode is not random. It is concentrated exactly where you need review most.

Two lines of research say so. The NeurIPS 2024 paper LLM Evaluators Recognize and Favor Their Own Generations found that models can distinguish their own text from other models' text at better than chance, and that self-recognition ability correlates linearly with the strength of self-preference. The model knows its own handwriting, and the more reliably it knows it, the more it likes what it sees.

The follow-up matters more for anyone shipping code. Do LLM Evaluators Prefer Themselves for a Reason? ran the question on verifiable benchmarks (math, factual recall, code generation) where a right answer exists, across seven model families. Most self-preference turned out to be legitimate: stronger models favor their own answers largely because their answers are better. So far so reassuring.

Then finding two. Harmful self-preference, meaning the model favors an objectively worse response, persists specifically when the evaluator model erred as the generator. And stronger models show a more pronounced version of it. In the authors' framing, stronger models struggle more to recognize when they are wrong.

Read that against your workflow. The case where you desperately need the review to catch something is the case where the author got it wrong. That is the exact case where asking the author is most likely to return a confident approval. Self-review is not merely soft. It is inversely correlated with need.

What separation actually looks like in a shipped tool

sol-advisor is the clearest small example I have seen. It is a Codex-native plugin, MIT licensed, 134 stars and 6 forks and three commits on main as of this morning. Almost nothing to it. The whole idea fits in one table from its README:

The primary session runs GPT-5.6 Sol at High reasoning and never writes production code. It handles requirements, architecture, decomposition, routing, and acceptance. Routine implementation goes to a pinned Luna Max agent. Higher-risk, context-heavy, wider-blast-radius work goes to a pinned Terra Max agent. And the final review goes to a separate sol_advisor_sol_reviewer agent, a fresh Sol instance that requests a read-only sandbox and looks at the actual diff and verification evidence.

The orchestrator inspects every diff and reruns verification. The fresh reviewer returns one of exactly three verdicts: ship, fix-first, or rethink. The session cannot report completion until the reviewer returns ship.

Two design details in there are worth stealing whether or not you touch Codex.

The first is that the reviewer's verdict is a small closed vocabulary rather than prose. "Looks good with a few minor notes" is how a review dissolves. Ship, fix-first, rethink forces the reviewer to make a decision the orchestrator can act on mechanically.

The second is that routing is verified, not assumed. Before delegation and acceptance, sol-advisor requires the installed role files to pass a byte-for-byte check against the shipped templates, requires the native spawn metadata to expose all three exact agent names, and requires the observed model and reasoning effort to match the role pin. When the desktop app omits model or effort, a local inspector script reads the rollout file for those fields only, emits an allowlisted JSON object, and refuses if it finds zero matches, multiple matches, or an inconsistency. A missing, stale, conflicting, or unobservable role stops that lane with an actionable error. The README says it plainly: there is no silent model, reasoning, or agent-type fallback.

That is the part most homegrown setups get wrong. People build a two-agent workflow, one of the agents silently falls back to the default model or the default profile, and the workflow keeps reporting success while the separation it was built for has evaporated.

open-kritt attacks the same problem from the findings side. Its pitch opens by rejecting the obvious approach: pointing a model at an entire repository and asking it to find vulnerabilities rarely works well. Instead it breaks research into small well-defined tasks, runs them across agents in parallel with Codex or Claude Code, then applies post-scripts to validate issues and build proofs of concept, automatic de-duplication, and custom severity rankers. The validation stage is a different mechanism, not a differently worded prompt to the same agent. A proof of concept either runs or it does not.

The Kritt team says the internal version of this earned over $1,500,000 in bug-bounty payouts under the researcher name Blockian, with links to Immunefi and HackenProof profiles. I cannot audit that figure, and neither can you, so treat it as the team's claim rather than an independent result. The architecture is the transferable part regardless.

Put this into practice

You do not need a plugin, a marketplace, or a second vendor to get most of the benefit. Three steps, in order of friction.

Start the review in a new session. Not a new turn. A new context. Give it the diff, the original spec, and the test output, and nothing else. If you paste in the conversation that produced the code, you have recreated the problem you were trying to escape. This costs you one copy-paste and catches conversational assumptions, which are the errors most likely to survive a self-review because they never got written down anywhere the reviewer could see them.

Give the reviewer a closed verdict set and forbid edits. Steal sol-advisor's three: ship, fix-first, rethink. Tell the reviewer explicitly that it may not modify files. Then check that it did not, by comparing repository state before and after rather than trusting the instruction. sol-advisor's README is careful about this: its reviewer TOML requests read-only sandboxing, but the host permission profile may broaden that request, and if the sandbox cannot be observed or any mutation occurs, the review lane stops rather than claiming enforced isolation.

Make the reviewer think before it judges. The self-preference paper's third finding is the practical one: inference-time scaling strategies, such as generating a long chain of thought before evaluation, effectively reduce harmful self-preference. Asking for reasoning before the verdict is a measurable mitigation, not a style preference. Set the reviewer to a high reasoning effort even when the implementer runs cheap.

If you want the harder version afterward, pin the roles to specific models, verify the pin at runtime, and refuse to proceed when the evidence is missing. That is the sol-advisor design, and its install script installs missing templates only, verifies every copy byte-for-byte, and fails rather than overwriting a role you changed.

Where this breaks, honestly

A fresh context is not a different model. sol-advisor says this about itself in the README, which is more than most projects manage: the final review is context-independent, not model-family-independent. Sol reviews Sol's orchestration. That catches conversational assumptions. It does not catch a blind spot that lives in the weights. If a model family systematically misunderstands your ORM, a fresh instance of the same family will misunderstand it identically and approve.

The research I leaned on is about judging responses, not diffs. The verifiable-benchmark study covers math, factual knowledge, and code generation, then extends to LMArena. It does not test "read this 400-line diff against this spec." I am extrapolating from response-level judgment to diff-level review, and that extrapolation is reasonable but unproven.

A fresh context also loses information. The authoring session knew why an approach was rejected two hours ago. The reviewer does not, and will sometimes flag a deliberate decision as an error. You trade false approvals for false rejections. That is usually a good trade for production code and a bad one for a throwaway script.

The cost is real. Separate lanes plus a high-reasoning reviewer plus a chain of thought before every verdict means more tokens and more wall-clock time for the same feature. The sol-advisor author's pitch is that Luna Max on routine work more than pays for the reviewer, which is plausible and is also a claim about his usage, not yours.

And sol-advisor itself is three commits from one author, welded to specific GPT-5.6 model tiers, requiring a Codex CLI with plugins and native subagents enabled. It is a design to read, not a dependency to build a team on. Copy the orchestrator prompt and the verification discipline; think twice before making a 134-star, three-commit project a gate on your production diffs.

The question worth asking today

Open the last thing your agent shipped. Find the moment where something decided the work was finished. Was that a different context from the one that did the work? Did it have a verdict it could not talk its way around?

If the answer is no, you have not been reviewing. You have been asking, and the research says the answer comes back cheerful exactly when it should come back worried.

Sources: sol-advisor README, open-kritt README, Do LLM Evaluators Prefer Themselves for a Reason? (arXiv 2504.03846), LLM Evaluators Recognize and Favor Their Own Generations (NeurIPS 2024), yc-software/qm. Repository figures read directly from GitHub on 2026-08-03 and will have moved.