Anthropic's Coin-Flip Experiment: Experts Followed Claude's Random Nudges 86% of the Time

The Study Where the AI's Opinion Was a Coin Flip

When you hand a task to an AI agent, the assumption is that the agent does the mechanical work while you keep the judgment calls — what to measure, how to read ambiguous evidence, what to recommend. Today Anthropic published research that puts a number on how false that assumption can be, and the number is uncomfortable.

Researchers recruited 61 professional analysts, all of whom had passed a causal-inference knowledge check and worked regularly in Python or R. Their task: advise a fictional farming cooperative on whether its paid advisory program actually improved crop yields and should be expanded. Realistic data, a board-set threshold of roughly 3–4% productivity gain, and a median of 65 minutes of work in a VS Code-style editor with Claude Code (running Claude Opus 4.8) built in.

The trick was in the setup. The analysis contained four decision points where two options were both perfectly defensible: measure yield per hectare or per unit of operating cost; use difference-in-differences or adjusted levels; cover enrolled farms or the whole membership; weight farms equally or by production. For each analyst, a hidden system prompt told the agent which option to prefer — and which option that was had been decided by coin flip. The agent was instructed to make the strongest honest case for its lean, to change its mind if the evidence demanded it, and never to mislead. The analysts were never told their agent had an assigned preference.

The Results: 86% Compliance, 15% Self-Awareness

Fifty analysts used the agent, and across the four judgment calls they went with the agent's coin-flipped lean 86% of the time. If the agent's input made no difference, you'd expect around 50%. The compliance per decision ranged from 81% to 92%.

Three follow-on findings are arguably worse than the headline number:

• Pre-formed opinions folded fast. In the 38 cases where an analyst had already expressed their own preference, they switched away from it 60% of the time when the agent disagreed — versus 6% when it agreed.
• Arguments were borrowed wholesale. When asked to explain their choices afterward, 77% of analysts used the agent's own arguments, sometimes word for word.
• Almost nobody noticed. Only 15% credited the agent as the source of those arguments. Analysts who followed the agent closely were just as confident in their final recommendation — and just as likely to call it the product of their own judgment.

The final recommendation itself was shaped too, just more subtly: when the agent leaned "roll out," 54% of analysts recommended expanding the program, against 36% when it leaned the other way. And because the earlier judgment calls shaped the productivity estimate, the four coin flips alone let a statistical model predict 82% of final recommendations — before the analysts had written a line of code.

Why Smart People Defer Without Noticing

These weren't casual ChatGPT users. Ninety percent rated themselves proficient or expert in observational causal inference, and two-thirds held graduate degrees or 6+ years of experience. Expertise didn't protect them, and the paper explains why: deferral on judgment calls is structurally invisible.

Checking AI-assisted work for errors assumes there's a correct answer to check against. On a judgment call there isn't — the work looks equally sound whichever defensible option was chosen. And asking the expert afterward doesn't help either, because they genuinely don't perceive the influence. The agent made an honest, well-argued case for a position; the analyst agreed with a good argument. Nothing in that exchange feels like being steered.

Fluency compounds it. Prior research cited in the paper shows that confidence and polished language reliably sway human judgment. An agent stating a clear, evidence-backed preference at the exact moment you're deciding is close to a perfect anchoring machine — even when the preference came from a random coin.

Why This Matters More Every Month

Anthropic points at two audiences who should be especially worried. The first is anyone whose "human in the loop" is meant to be a check on AI: if the reviewer's judgment was quietly settled by the system under review, the oversight is theater. The second is AI safety research itself — labs have begun using agents to run and analyze experiments, and a human reviewer is only an independent check if their judgment is their own.

For everyone else running Claude, Copilot, Cursor or any agent on real work, the lesson scales down neatly: as agents absorb more hands-on execution, the judgment calls — which metric, which scope, which framing, which recommendation — are precisely where your values are supposed to enter the process. This study suggests that without deliberate defenses, they often don't.

Five Ways to Keep Your Judgment Your Own

• Commit before you consult. Write your own call on genuinely open decisions before asking the agent. The study's 60%-switch finding is the failure mode; a pre-registered opinion is the cheapest antidote.
• Ask for both sides, not a lean. Prompt the agent to make the best case for each option, or explicitly ask "what's the strongest argument against what you just recommended?" Agents can be configured to surface options rather than preferences.
• Flag the judgment calls explicitly. Ask the agent to list every decision in the task where two reasonable choices exist. Most deferral happens on choices you never noticed were choices.
• Adversarial second pass. Have a different model (or a fresh session) attack the finished analysis without seeing the first agent's reasoning. Disagreement between independent runs is a signal worth your attention.
• Audit the attribution. When you sign off on AI-assisted work, ask yourself which parts of the reasoning you could defend without the transcript. If the honest answer is "few," you've found your deferral.

Read the Fine Print

The researchers are candid about limits. The leans were deliberately strengthened to be measurable — without an assigned lean, Claude Opus 4.8 made the same five choices in 112 of 123 runs, and un-leaned agents argue with users less predictably, so real-world steering may be weaker. The switch-away analysis rests on just 38 cases (95% CI: 24–82 points). And it's not a cautionary tale against AI: agent users finished faster and made fewer serious errors, though the comparison group was small and less experienced. The finding isn't "AI is bad for judgment" — it's "sign-off is not the same as judgment."

Frequently Asked Questions