Anthropic: 86% of experts echoed Claude's recommendation without knowing it had been chosen by a coin flip

Imagine an A/B test in which half the users see a red button and half see a green one. If the red button gets clicked 3 times as often, the decision makes itself: roll it out and move on. Neither an expert nor AI is needed here. But real life is almost never that simple, and it is precisely for these situations that we are usually told we need a human.

When the data contradicts itself

A real A/B test looks more like this. The red button gets 5% more clicks. But users then choose a cheaper plan, and the total is 1% lower. That 1% is not statistically significant. Yet six months later, the red group has 1.5 times as many refunds. It seems strange to believe a refund depends on the color of a button during onboarding. And the green button also fits the design system better.

There is no clear answer here; the trade-offs need weighing. The industry's standard argument goes like this: a professional has experience and judgment, will weigh everything up and decide, while the model merely helps. The human in the loop acts as a safeguard.

How Anthropic tested this

The Anthropic team decided to measure rather than debate. The experiment involved 50 experts. Each worked with Claude to untangle a complicated scenario involving farmers and submitted a final conclusion.

The key trick in the design: a coin flip determined which position the agent would recommend. In other words, the direction of the recommendation did not depend on the facts; it was random. The participants did not know this. If the experts were exercising independent judgment, their conclusions should not have aligned so closely with the advice.

The figures reported in the account of the study are as follows:

• 86% gave the final recommendation the agent had steered them toward;
• 60% agreed with the AI at all four of four key decision points. Agreement by chance would have been 6.25%;
• 77% repeated the agent's reasoning in their explanations;
• 15% acknowledged where they had taken it from.

That last figure is perhaps the most telling. People did more than choose the same option: they presented someone else's arguments in their reports as their own, and almost no one noticed.

What this means in practice

First: saying "there is a human in the process" guarantees nothing on its own. If an expert sees the model's recommendation before forming an opinion, they will most likely become an extension of it rather than an independent check. This is the anchoring effect, except the anchor now speaks in persuasive, coherent prose.

Second: if you are building a product with a human in the loop, design oversight deliberately. A few ideas follow from the logic of the experiment:

• ask the person to write down their opinion before showing the model's answer;
• occasionally introduce deliberately opposing or incorrect advice and see whether people notice;
• measure the proportion of cases in which the person rejects the recommendation. If it is close to zero, that is a reason for concern, not evidence of the model's quality;
• separate the roles: one model or person makes a proposal, and another evaluates it blind.

Third: for product managers and founders, this is a question of responsibility. "An expert made the decision" sounds reassuring, but if the expert follows the agent in 86% of cases, the model made the actual decision and the human signed off on it.

Limitations

This is one experiment with 50 participants and a specific task involving farmers. The figures cannot be applied directly to code review, medicine or A/B testing. It is also unclear how much the outcome depends on the interface, participants' motivation and the time available to them. Anthropic itself also has a stake in the subject of agent oversight, so it is worth reading the original study as well as the account of it.

Still, the signal matters. The question is no longer whether a human is needed. It is whether that human is actually thinking, and how to verify it. Until there is an answer to the second question, human-in-the-loop remains more of a ritual than a safeguard.