Epoch AI and Anthropic: AI Agents Overstate Their Results, and the Best One Scored 15%

The idea of an "AI scientist" that comes up with its own hypotheses, runs experiments and publishes results remains one of the industry's biggest promises. Two independent evaluations show it is still far off. Epoch AI and Anthropic, without coordinating, reached the same conclusion: GPT-5.6 Sol and Claude Fable 5 can churn through experiments, but they don't know how to doubt what they get. Sol's best result was 15% of the human reference score. And even that 15% came from methods the researchers already knew.

What exactly they found

Both studies point to the same problem. The agents handle the mechanics well: writing code, launching a run, collecting metrics, drafting a conclusion. But once a task demands scientific skepticism, quality drops. The models tend to overstate results, take a lucky run for a confirmed pattern, and fail to look for reasons the numbers might be an artifact.

According to the studies, the models' most serious weakness is their inability to critically check their own conclusions. For science, this is a fundamental defect. What separates a researcher from an executor is that the researcher tries to refute their own hypothesis before a reviewer does.

The 15% ceiling

The figure of 15% of the human benchmark should be read with a caveat: it was reached using methods that were already known. Sol did not propose new approaches; it competently applied what already exists in the literature. That points to skilled assembly of known material, not an ability to push the boundaries of knowledge. The models still lack the creative thinking that distinguishes discovery from reproduction.

The publicly available summary does not disclose the details of the methodology: the composition of the tasks, how the reference score was calculated, or the number of runs. So these 15% should not be compared with other benchmarks without the reports themselves.

Key facts

• Studies: Epoch AI and Anthropic, conducted independently and reaching a consistent conclusion.
• Models tested: GPT-5.6 Sol and Claude Fable 5.
• Best result: 15% of the human reference score (Sol).
• Source of the result: methods already known to the researchers.
• Main weakness: no critical assessment of their own results.
• Second weakness: a lack of genuinely creative thinking.

How to work with this

For teams already building agents into R&D, the takeaway is practical. An agent suits routine work: reproducing known experiments, sweeping parameters, preparing tables and draft reports. But every one of its conclusions should be treated as a hypothesis, not a result.

A few rules help. First, independent verification: a human or a separate pipeline, without the same agent, reruns the key numbers. Second, mandatory baselines: any claimed improvement is compared against a simple baseline and against the variance between runs. Third, logging: store the code, seeds and data of every run so an inflated result can be found after the fact. Fourth, don't let the agent word its conclusions without a specialist's editing.

This is also a signal for the market. Claims about "autonomous research agents" should be tested by the model's ability to find its own errors, not by the number of experiments it launches. As long as two independent labs, including the developer of one of the models tested, say the same thing, autonomous science remains a task for the future. A human in the verification loop is still required.