SciΠvot benchmark: the best LLM picks the right plan 63.5% of the time, versus 73.4% for humans

LLM-based agents can run scientific experiments for months with almost no human involvement, but they lack judgment at decision forks. The authors of the SciΠvot benchmark collected 248 cases in which senior researchers corrected agents, then tested whether a model would choose the human's correction. The best result among the tested models in the project-context setting is 63.5%. Fresh researchers who saw the same material scored 73.4%. On the selected cases, four of the five frontier models scored below random guessing.

How the test works

The data comes from real work. For six months, senior researchers ran LLM agents on 26 projects across six scientific fields and regularly stepped in when an agent made a poor research decision. Each intervention became a case. The model gets two options: the LLM's original plan and the alternative corrected by a human. The only context is the project record that existed before the correction, so the model cannot peek at the outcome.

This setup removes the main difficulty in evaluating scientific judgment. Research decisions are open-ended and depend on a project's history, so they have no single reference answer. Here the reference is the human's choice in a specific situation where the agent had already gone wrong.

• Size: 248 cases from 26 projects, 6 scientific fields
• Format: choosing between the LLM's plan and the human-corrected version
• Best model accuracy with the project record: 63.5%
• Fresh researcher accuracy: 73.4%
• Frontier models below chance on the selected cases: 4 of 5
• Factors tested: reasoning effort, access to the project record, reconsideration

What the results showed

At medium reasoning effort and with access to the project record, models systematically prefer the LLM's original plan. The authors call this a bias: an option that resembles the model's own way of thinking seems more reasonable to it. The description offers no direct data on the cause, but the pattern is consistent with the known tendency of language models to favor text close to what they would have written themselves.

The second unexpected result concerns context. The project record should help: it shows goals, past experiments, and constraints. In practice, at medium reasoning effort, every model was more accurate without the record than with it. A long project history does not turn into a better decision, and it may reinforce the very logic that led the agent into error.

Reconsideration does not rescue the models either. It does not produce a reliable gain in overall accuracy. Models struggle even in a convenient situation: the right alternative is on the table, yet the choice is still weak. The gap with humans is about ten percentage points, and the humans in the test were fresh researchers, not the authors of the corrections.

What teams building agents should do

First takeaway: you cannot judge long-horizon autonomy solely by whether the agent finished the experiment. Months of uninterrupted operation say nothing about the quality of its decision forks. Human checkpoints belong exactly where the agent changes a hypothesis, an experiment design, or a success criterion.

Second takeaway: an agent's self-review cannot be considered independent. If a model prefers its own plans, an "agent proposes, same agent critiques" setup will let some errors through. You need an external reviewer: a human, or a model with a different task framing.

Third takeaway is practical. Any team running research agents is accumulating a log of corrections, and that log can already serve as an internal benchmark on the SciΠvot pattern. Pairs of "agent plan, human correction" give an honest test on your own data before you expand autonomy. Extra context and more reasoning deserve separate testing rather than being taken on faith: according to this data, they do not guarantee improvement.