Independent AI intelligence Two editions daily · ET
Fervor AI

Analysis · November 17, 2026 · concept

Claude CodeClaude Code Code Reviewclaude-codeagent-harnessagent-securityai-skills

The Claude Code Team Stopped Reading Every AI Diff. One Rule Made That Safe.

The Claude Code team lets agents write most of its code. The trick that makes that safe is boring, cheap, and you can copy it this week.

Coding agents got fast enough that reading their work became the slow part.

You can start five agents before lunch. By three o'clock you have five pull requests, each a few hundred lines, each written by something that sounds confident whether it is right or not. Reading all of them carefully is a full-time job. Skimming them is how bugs ship.

So the interesting question stopped being "can the agent write this?" It became "how do I know it worked without reading every line?"

On November 16 the Claude Code team at Anthropic published its answer, in a post by Addy Osmani on the claude.dev blog. About 90% of that team's code is written and reviewed by Claude. Its answer is not "add more reviewers." It is a short list of rules about where checks live and what they are allowed to see. The most important one fits in a sentence:

If a change can modify the check that judges it, the check is broken.

That rule sounds obvious. Most agent setups break it anyway. Here is what it means, how the team applies it, and the smallest version you can set up on Monday.

The student who writes the answer key

Picture a student who is allowed to edit the answer key before the exam is graded. Even an honest student will drift. A question looks unfair, the key looks wrong, and a small fix seems reasonable. The grade stops meaning anything.

An agent working in your repo is in that position by default. The rules that judge its work (a review checklist, a lint config, a test file, a CLAUDE.md instruction) usually sit in the same repository it is editing. A pull request can change the code and, in the same diff, the rule that would have flagged it. No bad intent is needed. An agent told to "make the checks pass" will find the shortest path, and sometimes the shortest path runs through the check.

The Claude Code team closes that path in its product. Its Code Review feature reads review instructions from a file called REVIEW.md, and it reads that file from the pull request's base branch (the branch it will merge into) or the default branch, "never from the pull request's own changes". To change how reviews behave, you merge the REVIEW.md change first, through review, like any other change. (Code Review is a hosted feature that an organization Owner sets up; the rest of this playbook works without it.) The team applies the same idea everywhere: the rules that gate a change live on a branch the change cannot touch.

There is a catch that matters, and the docs are explicit about it. Several of REVIEW.md's powers only apply when the branch it is read from is protected (GitHub settings that stop people, and agents, from pushing to it directly). Lowering a finding's severity, excluding paths and capping nitpick comments are honored only from a protected default branch. If Code Review cannot confirm your protection settings, it reads no REVIEW.md at all. The docs' setup advice is two steps long for a reason: add the file, then protect your default branch.

That is the whole insight. Verification is less about how smart the checker is and more about who can edit it.

Run the change, not just the tests

The second rule is about what counts as evidence.

A unit test checks one small piece in isolation. An end-to-end test (E2E) drives the real app the way a user would: register, add an item to the cart, check out, see the confirmation page. The post's advice is direct. If you can only invest in one kind of test, pick end-to-end, because "either the whole feature works or it doesn't." A unit test can pass while the feature is broken, since neither the agent nor the test understands the whole task.

The team goes one step further and says running the app matters more than tests. Its method is a skill, a small folder of instructions an agent loads when it needs them. They create one named verify, put it in .claude/skills/ as close to the relevant code as they can, and have it describe how to start the app or simulator and drive it. Claude knows to use the verify skill before committing. For web changes that might mean the Chrome extension, for mobile an iOS or Android MCP server (a plug-in that gives the agent tools), for backend work running the service locally.

Then they treat the verify skill like production code. They ask Claude to iterate on it and stress-test it until it is reliable, because it is the smoke test every change will pass through.

The post is honest about how this fails. Flaky tests train the agent to rerun them instead of fixing the code. Slow tests mean the agent runs fewer of them. So the team's rule for flaky tests is strict: Claude cannot skip them, add sleeps or retry until they pass. If it can't find the root cause, it deletes the test and writes down why.

Three rounds, then a person

Nobody can read every change, so the team uses failures to decide what needs a human. Claude runs the checks and tries to fix what fails. If a check still fails after three rounds, Claude stops and brings in a person.

This is a small number with a big effect. Without a stop rule, an agent stuck on a failing check will keep trying things, and the things get stranger with each attempt. Three rounds is enough to fix a typo or a missed import. It is not enough to argue with a real design problem, which is exactly the case where you want a human anyway.

The same idea shows up in their reliability checklist. It has 39 items, each tied to a past incident and each with a trigger that can be read from a diff. One example: when session storage or message types change, check that old saved transcripts still load. Claude reads the diff (the list of lines the change adds and removes) and checks only the items whose trigger matches, usually 3 to 10. Nobody has to memorize old outages, and the agent doesn't burn time on checks that can't apply.

New gates start by watching

The last rule is the one most teams skip, and it is the reason the others survive contact with real work.

The Claude Code team built a commit gate that can block a commit if the session never ran /code-review or /verify. They did not switch it on. They gave it three modes: off, observe and enforce. In observe mode it only logs what it would have blocked, so the team can look for false positives before anyone gets stuck. In enforce mode it blocks, but a commit can carry an explanation in its message, and after two blocks in one session the next commit goes through so Claude doesn't loop forever. The post's own aside: "Claude Code isn't perfect."

They also stamp every commit with how it was verified, including the value "none," so they can later search for every commit that skipped verification.

This is the part worth copying even if you never touch Claude Code. A check that blocks on day one with no data is a check people will rip out by Friday. A check that watches for a week gives you the false-positive rate before it costs anyone an afternoon.

Put this into practice on Monday

You don't need Anthropic's setup to get most of the value. Here is the order I'd do it in, lowest effort first.

1. Protect your default branch. In GitHub, open the repo's Settings, then Rules (rulesets) or Branches, and require pull requests for main. Code Review counts a ruleset as protection when it restricts updates or requires pull requests with at least one approval that is dismissed or re-required after a new push. Expect your own direct pushes to main to start failing. That is the point.

2. Move every rule an agent could edit to that branch. If you use Code Review, add a REVIEW.md at the repo root with the things you want flagged, merge it, and only change it through pull requests. If you don't, apply the same idea to your CI config and lint rules: they change in their own reviewed pull request, never inside a feature diff.

3. Write one verify skill. Create .claude/skills/verify/SKILL.md. In plain words, tell the agent how to start the app, which one user flow to drive, and what success looks like. Something like:

---
name: verify
description: Start the app and check the change works end to end. Use before every commit.
---
1. Run `npm run dev` and wait for "ready on http://localhost:3000".
2. Open the app in the browser, sign in with the test account in .env.test,
   add one item to the cart and complete checkout.
3. Success: the confirmation page shows an order number.
4. If anything fails, report the step, the error and a screenshot. Do not commit.

Ask Claude to run it five times and fix whatever makes it flaky. The first runs will probably fail for boring reasons: a port already in use, a seed script that doesn't reset. Those failures are the skill getting better.

4. Add a stop rule to your instructions. One line in CLAUDE.md does it: "If a check still fails after three attempts, stop and summarize what you tried. Never skip, retry-until-green or edit a test to make it pass." It is a soft rule, since the agent can ignore it, but it changes behavior far more than you'd expect.

5. Run tests when Claude finishes. Claude Code hooks are shell commands that run at fixed points, and the Stop event fires when Claude finishes responding. A hook there can run your tests every time Claude stops. One detail matters: to make a failure send Claude back to work instead of just printing, the hook has to exit with code 2 (or return a block decision), and a normal failing test run exits with 1. So wrap the test command in a tiny script that turns a failure into exit code 2. Keep the hook in a settings file that is reviewed like code.

6. If you add a blocking gate, start it in observe mode. Whatever the gate is (a pre-commit script, a CI job, a required check), have it log "would have blocked" for a week before it blocks anything. Then read the log.

What this won't do

This playbook has real limits, and the post doesn't hide them.

It describes Anthropic's internal process. Only a few pieces ship as product behavior you get for free, mainly Code Review's base-branch REVIEW.md. The verify skill, the 39-item checklist, the commit gate and its observe mode are things the team built. You build yours.

A protected branch doesn't stop an agent from proposing a change to the rules. It stops that change from judging its own pull request, and it forces the edit through review. It does nothing about a person, or a compromised account, with permission to merge. Someone still has to review rule changes as carefully as code changes.

End-to-end checks cost time. They are slower than unit tests, they need a running app and test data, and they break for reasons unrelated to the change. The post's own warning applies: a flaky check trains the agent to retry rather than fix.

And the three-round stop and the CLAUDE.md rule are instructions, not enforcement. An agent can ignore an instruction. The things that actually enforce are the protected branch, a hook that exits with code 2, and any CI check you mark as required.

The shift

The post closes on a line worth sitting with: your job moves from reading every diff to "defining what 'done' looks like and reviewing the evidence."

That only works if the definition of done is something the work cannot rewrite. Start with the branch. Protect it today, move one rule onto it, and watch what your agents do when the shortcut is gone. You'll learn more about your setup from that one change than from another week of skimming diffs.

Sources: Addy Osmani, "How the Claude Code team thinks about verification," claude.dev, November 16, 2026; Claude Code docs, "Customize Code Review" and "Hooks reference," read November 17, 2026.