Human in the Loop Isn't a Real Check Once Your AI Agent Speaks First

Anthropic flipped a coin to decide which way 50 analysts' agents leaned. On their analytical choices, the analysts followed the coin 86% of the time. Here is how to keep your sign-off meaningful.

Almost every agent workflow ends with a person approving what the agent produced. That approval is the safety story, the reason teams feel fine letting an agent pick the metric or draft the recommendation.

It only works if the person brings their own view. A trial Anthropic published on October 27 suggests that view mostly disappears once the agent argues first: an opinion assigned by a coin flip steered most of the experts' analytical choices, and they mostly didn't notice.

The fix is a habit you can start this week: if a decision matters, write your answer down before the agent gives you its own. Here is the evidence, and a ten-minute setup for Claude Code.

What Anthropic actually tested

The study is called "How AI agents shape experts' judgment calls", and it is a randomized controlled trial, the same design medicine uses to separate cause from coincidence.

Anthropic recruited 61 professional analysts. Eleven chose not to use the agent, which left 50 in the main analysis. Most held a master's degree or doctorate, or had six or more years of experience. Each one got the same job: advise a fictional farming cooperative on whether to expand or end an advisory program.

The work involved four analytical choices, the kind that decide where an analysis lands:

Which outcome to measure
Which comparison method to use
Which farms to include
How to weight the data

Then came a final recommendation: roll out or not.

Here is the clever part. The analysts worked in Claude Code (Anthropic's coding agent, which runs in a terminal or editor) with Claude Opus 4.8. Before each of the five decisions, a coin flip secretly set which option the agent favored. The agent was told to "make the best honest case for its lean." So the agent's opinion carried no information about which option was better. It was random.

If analysts were weighing the agent's view as one input among many, their choices should have matched the agent's lean about half the time.

What happened

They matched it 86% of the time across the four analytical choices. Chance would be about 50%.

The final recommendation moved less, but it moved. When the agent leaned toward rollout, 54% of analysts recommended rollout. When it leaned the other way, 36% did. In the paper's preregistered main test (the analysis the authors committed to before collecting data), that direct effect on the final call fell just short of statistical significance (one-sided p = 0.069). The earlier choices did the heavier lifting: in an exploratory analysis (run afterward, on the same data), the four random leans on the analytical choices predicted 82% of final recommendations, against a 61% baseline.

The most uncomfortable numbers are about people who walked in with an opinion. Analysts who already had a preference switched 60% of the time when the agent disagreed with them, and 6% of the time when it agreed. The authors flag that this comparison rests on only 38 cases, so the exact gap is uncertain. The direction is not subtle.

Picture what that means in a normal week. You ask an agent to help evaluate a vendor, choose a database, or decide which customers a new pricing rule should cover. It proposes an option with good reasons. You read the reasons, they make sense, and you approve. The study suggests the reasons making sense is not evidence that your approval was independent. You would likely have found the opposite reasons convincing too.

Why nobody noticed

You might expect experienced analysts to catch themselves. They didn't, and the study shows why this is hard to see from the inside.

77% of analysts reused the agent's arguments in their own explanations. Only 15% credited the agent. And 52% rated the final recommendation as more their own than the agent's.

Confidence didn't separate the two groups either. Analysts who deferred and analysts who didn't reported nearly the same confidence (71 versus 73 on a 0 to 100 scale). Experience, expertise and most other traits the researchers tested did not predict who deferred.

The authors put the core problem plainly: "Deferral of this kind is hard to detect." The person doing the deferring feels like they made the call.

This is the part that should change how you think about review. A sign-off is usually treated as a property of the person: senior enough, careful enough, qualified enough. The study points at something else. Whether a review is independent depends mostly on the order of events, not on who is reviewing. If the agent speaks first, even a strong expert tends to end up checking the agent's reasoning instead of forming their own.

The agent also helped, and that matters

It would be easy to read this as "keep agents away from decisions." The same study argues against that.

Analysts who worked without an agent made serious methodological errors far more often: 6 of 11, versus 3 of 50 with an agent. They also took longer, with a median of 90 minutes against 65. The authors are careful here: the no-agent group was small and less experienced, and "the two groups were not directly comparable," so this doesn't prove the agent caused the improvement.

Anthropic published a second post the same day that points the same way. In its drug development update, Claude reviewed 1,061 pages of manufacturing documentation for the biotech company Enveda and caught all 11 issues human reviewers found, plus seven they missed.

So the honest picture is two things at once. Agents can make work better and make the human check weaker. The goal is to keep the first without paying for it with the second.

Your tools lean toward "just decide"

Look at the defaults you work with. Claude Code ships a built-in output style called Proactive (an output style is a set of instructions that shapes how Claude responds for a whole session). According to the docs, in Proactive, Claude "makes reasonable assumptions about routine decisions rather than stopping to ask."

For routine decisions, that is exactly what you want. Nobody needs an interview about variable names.

The trouble is that "routine" is doing a lot of work in that sentence. Choosing which farms count in an evaluation feels routine in the moment. So does picking which metric a dashboard reports, or which accounts a migration touches. Those are the judgment calls the study measured, and they're the ones an agent will happily decide for you if nothing in your setup says otherwise.

Anthropic's post floats two design fixes: agents that "make the best case for each option, or ask experts for their own view before offering one." It also names the catch right away: "this could quickly become overwhelming for users." You can't do this for every decision. You can do it for the ones that matter.

Put this into practice

None of this needs new software. It needs about ten minutes of setup and a habit.

1. Make a short list of decisions that need your own judgment. The paper's closing advice is that organizations should "decide deliberately which decisions need a person's own informed judgment." Do the personal version. Five to ten items is plenty. Good candidates: what to measure, what to exclude, which option to recommend to a customer or a boss, security exceptions, anything you'd have to defend in a meeting.

2. Commit before you look. When one of those decisions comes up, write one line before reading the agent's view: your pick and one reason. A comment in the ticket, a line in a notes file, anything with a timestamp. This is the cheapest fix there is, and it targets the exact mechanism the study found. Your later choice can still change. Now you'll know when it did.

3. Give Claude Code an "ask me first" style for decision work. Custom output styles are Markdown files saved in ~/.claude/output-styles (for you) or .claude/output-styles (for a project), per the output styles docs. Save this as ask-me-first.md in either folder:

---
name: Ask me first
description: Collect my view before offering yours on judgment calls
keep-coding-instructions: true
---

Before you recommend an option on any of these decisions, ask me for my
own pick and reason, and wait for my answer:
- which outcome or metric to measure
- which data, users or accounts to include or exclude
- which comparison or method to use
- the final recommendation in any analysis

After I answer, give the strongest case for each option, including mine,
then say where you disagree with me and why.
Handle routine implementation decisions yourself without asking.

Switch to it with /output-style Ask me first (Claude Code v2.1.269 or later), or pick it under /config. The keep-coding-instructions: true line keeps Claude Code's normal software engineering instructions in place, so only the decision behavior changes. If you create the file during a running session, restart Claude Code so it loads.

What to expect: on the listed decisions, Claude should ask a question and wait. On everything else, it works as usual. It's an instruction, not an enforced rule, so it can slip (more on that below). The first day this feels slow. That pause is the whole point.

4. For the big calls, ask for both cases. When a decision is expensive to reverse, a prompt like "make the strongest case for each option, then tell me which one you'd pick and what would change your mind" gets you the study's other design fix without a settings change.

5. Check yourself for two weeks. Keep the one-line commitments from step 2 and compare them with what you finally chose. If you changed your answer almost every time the agent disagreed, you have your own small version of the 60% figure, and a reason to keep the habit.

Honest limitations

This is one study, and its authors list the limits themselves.

The agent pushed harder than a normal one. Anthropic strengthened the leans on purpose to measure their effect. Everyday agents may lean less, and the effect may be smaller.
One task, one field, one model. Every analyst did the same farm evaluation with Claude Opus 4.8. The analysts were also new to farming, which may have made them lean on the agent more than they would in their own field.
Some numbers rest on small samples. The switching comparison uses 38 cases, the no-agent group had only 11 people, and the 82% prediction figure comes from an exploratory, in-sample analysis.
An output style is an instruction, not a lock. Claude Code's own docs say an output style "doesn't guarantee that something always happens or never happens." Claude might skip the question on a decision you didn't list, or when your request is phrased as "just do it."
It costs time. Committing first and reading both cases adds a minute or two per decision. Spend it where a wrong call is expensive, not everywhere.

The choice is yours to make on purpose

Agents are going to keep making more of the calls in your work, and on the evidence so far that is often a good trade. The study's closing line names the real question: "With the right tools, that split can be something we choose, instead of something we drift into."

Pick the five decisions you want to keep. Write your answer before the agent writes its own. Then you get to find out how often you agree, and that is information most people approving agent output today don't have.

Sources: Anthropic, "How AI agents shape experts' judgment calls" (Oct 27, 2026) and full paper (PDF); Anthropic, "Early progress on accelerating drug development with Claude" (Oct 27, 2026); Claude Code docs, Output styles, read October 28, 2026.