Claude outperformed experts in peer review: editors chose AI in 87% of assessments

You have prepared an experiment report, but before sending it to management, you want someone to ask uncomfortable questions: where do the conclusions run ahead of the data, what explanation has been overlooked, and what needs another check? Harvard researchers tested a similar task using scientific manuscripts. The result offers a reason to try Claude for an initial critical review of team documents. For example, you could ask the model to identify other explanations for a rise in conversion before declaring a new interface the winner.

What Harvard compared

Gary King and Stephen Worthington used 124 manuscripts and 134 expert reviews from the rapidPeer service. The service provides preliminary reviews before papers are submitted to journals. The experts were paid; 88% of the 74 authors surveyed rated the feedback highly. Claude Opus 4.8 produced alternative reviews based on the manuscripts and instructions, both with web search and without external tools.

Three former journal editors assessed the same ten pairs of reviews without knowing who had written them. They chose Claude in 26 out of 30 assessments — roughly 87%. The editors did not examine the manuscripts themselves. A separate GPT-5.5 judge preferred AI in all 132 pairs with a clear winner in the version using search; the advantage also held without search. The authors did not measure improvements to the articles following the feedback. The work was published as a working paper on Harvard’s website.

The denominator is particularly important here: the thirty editorial assessments concern ten pairs reviewed by three people. Turning that finding into a promise that “AI is better than any expert at 87% of tasks” would therefore be a considerable overreach. This is the result of one specific comparison of reviews.

How to apply the idea to a product report

Consider a hypothetical example. A team changes the sign-up screen and writes: “Conversion increased by 12%; we’re rolling it out to everyone.” The product manager sends the model a description of the experiment, a results table, and the decision criteria. The request is to list weaknesses in the conclusion, identify supporting passages in the document, and suggest checks.

A useful response should ask specific questions. Were users assigned randomly? Were the groups observed for the same length of time? Is the 12% increase relative, or does it refer to percentage points? Did the new version receive more users from a different advertising channel? This is an example of the desired review, not an episode from the Harvard experiment.

An analyst then checks the feedback. If assignment was random and this is clearly stated in the appendix, the model’s criticism turns out to be an error. If the traffic mix really did differ, the team has a reason to reconsider its conclusion before launch. The value of this kind of assistant can be measured by the problems it finds and the time spent checking them.

A developer could run a similar review of an architecture proposal: attach the requirements, constraints, and diagram, and ask the model to identify failure scenarios. A founder could check a memo on product economics for hidden assumptions. Each such application requires its own evaluation: the Harvard comparison alone says nothing about the quality of a review of your architecture or financial model.

What to check before adoption

A practical starting point is a few completed projects where missed problems are already known. Prepare the documents as the team saw them before making its decision, and obtain a review from the model. Ask colleagues to assess the feedback without being told who wrote it. Count confirmed findings, false criticisms, and the time spent checking them separately. A well-written review can easily earn a high rating; a discovered error provides a more concrete criterion.

For ongoing work, a simple response format is useful: problem, location in the document, possible consequence, and method of verification. It helps you move quickly from reading to action. If the model writes that “the methodology is not convincing enough,” ask it to name a specific assumption and explain what result would require changing the conclusion.

The Harvard result deserves attention precisely as a reason to run this kind of experiment within your own team. An accessible initial review could help the team spot a weakness sooner. The decision to adopt it should follow testing on your own documents: how many useful comments you received, and how much time you spent sorting through the rest.