PhAI Labs: Claude Opus 5 Ranks Last in Scientific Hypothesis Test With 2.3%

PhAI Labs has published a comparison of models on their ability to propose scientific hypotheses, and the first reaction in Chinese media is surprise: Claude Opus 5, at 2.3%, came in last among five groups. The GPT group scored 2.5%. The gap is two tenths of a percentage point, and the full table, judging by the available excerpt, includes several more models, among them GLM.

How the test works

The methodology is unusual in that it relies neither on expert graders nor on a benchmark with known answers. Models propose research hypotheses, and these are then matched against scientific literature published later. If a model's idea coincided with a direction that actually developed in papers, it counts as a useful pointer for future research. The percentage in the report apparently shows the share of such hits.

The approach has an obvious strength: the future cannot be fitted to the answer. It also has weak spots. Overlap with later publications reflects what the scientific community worked on, not necessarily what would have been valuable. A bold hypothesis that nobody got around to testing scores zero.

What we know about the results

• Claude Opus 5: 2.3%, last place out of five groups.
• GPT: 2.5%.
• GLM and the other participants: values are cut off in the available excerpt of the material.
• Criterion: overlap of hypotheses with literature published after they were generated.
• Related effort: DiscoveryRSI, a PhAI Labs project on self-evolution of research agents.

The main thing to keep in mind is that all values fall within a range of a few percent. When the leader and the last-place model are separated by fractions of a percentage point, it is hard to claim the ranking is stable without confidence intervals, the number of runs, or the sample size. A gap of 0.2 pp may well be noise. Headlines about the "bottom" of the ranking sound louder than the data itself.

DiscoveryRSI and agent self-evolution

In the broadcast announcement, the model comparison is tied to DiscoveryRSI. The abbreviation hints at recursive self-improvement applied to scientific discovery: the agent not only proposes hypotheses but also improves its own way of proposing them. The available text gives no details of the architecture, so it is too early to judge how the loop is actually implemented. The logic, however, is clear. If a system has a measurable quality signal, for example the share of hypotheses confirmed by later literature, that signal can be used to select and refine agent strategies.

This is where the PhAI Labs test becomes more interesting than its table. A metric that works as an external judge is suitable not only for comparing models but also as a reward function for training agents.

How to read rankings like this

For teams building scientific or research agents, the takeaway is pragmatic. A single hypothesis benchmark does not replace testing in your own domain: biology, materials science, and mathematics produce different distributions of "successful" ideas. The result for a specific model version, with a specific prompt and a specific agent harness, does not carry over to the whole model family.

Before switching providers because of these numbers, it is worth waiting for the full report and checking three things: how many hypotheses were generated per model, how a "match" with the literature was defined, and whether the results reproduce across repeated runs. Until then, it is more sensible to read 2.3% versus 2.5% as a sign that all participants still handle this task poorly, not as a verdict on any one model.