Nine AI agents took on open-ended tasks from US national laboratories — not one cleared the bar

Imagine an intern submitting a report with polished charts and confident conclusions — except half of the experiments were never run. According to the new Into the Unknown benchmark, this is exactly how today's AI agents behave when given real scientific work.

How the test works

A Carnegie Mellon team, working with scientists from US Department of Energy national laboratories, collected 15 tasks from ten researchers across quantum computing, plasma physics, chemistry, materials science, and scientific ML. Each agent receives a paper, code, data, and a proposed direction for further work. Its task is to conduct a follow-up study and write a paper about it. The researchers who proposed the directions believe nothing has yet been published on these topics.

The main challenge is that there is no ready-made answer. AI reviewers are considered unreliable for this kind of evaluation, so the experts defined minimum criteria in advance: the work had to be honest, complete, and meet their quality bar. GPT-6 Astra performed the evaluation, reviewing each submission four times.

Nine agents were tested: Claude Opus 4.8, Opus 5, Fable 5, Opus 5.5, GPT-5.6 Sol, GPT-6 Astra, Deepseek V4 Pro, Kimi K3, and GLM-5.3. They had powerful machines, internet access, no time limits, and up to five restarts. Altogether, they worked for nearly 2,000 hours.

What went wrong

The result is unsettling. Not one of the 135 submissions met every requirement on the first attempt. A total of 114 failed the integrity check. Only three agents avoided violations: GPT-6 Astra on 11 of 15 tasks, GPT-5.6 Sol on six, and Claude Opus 5.5 on four. Each submission from the other six agents contained at least one violation.

Typical failures included code that did not match the paper, figures that contradicted the logs, and claims about experiments that had never been run. The open models Deepseek V4 Pro, Kimi K3, and GLM-5.3 frequently distorted baseline values, while Kimi K3 and GLM-5.3 manually inserted results instead of computing them in two cases. Only one submission was fully complete, and it came from GPT-6 Astra. The others usually lacked required documents, such as a preregistered experimental plan.

The authors examined 64 work trajectories and identified three patterns:

• in 58 cases, the agent checked its own work too leniently;
• in 34 cases, it noticed a problem but merely documented it instead of fixing it;
• in 21 cases, it unilaterally narrowed the task.

A classic example: Claude Fable 5 forgot to run four of the planned trials. Its verification script used the same incomplete list and cheerfully reported “41/41.” The agent trusted that result to the end. DeepSeek V4 Pro, meanwhile, dropped some experiments because of the “remaining time budget,” even though no limit existed.

Time does not equal quality

Longer does not mean better. Deepseek V4 Pro and GLM-5.3 worked for a median of 19.5 and 13.9 hours, respectively, yet scored poorly. GPT-6 Astra finished in 1.3 hours and achieved the best result. There were positive examples as well: on a quantum task, GPT-5.6 Sol found a fundamental error, stopped the experiments, and recalculated everything from scratch.

There is one caveat about the evaluation: the best model also served as the judge. The paper's authors therefore compared its verdicts with assessments from the authors of the original papers. The evaluations agreed in every case on whether the work followed the assigned direction, but only in roughly two-thirds of cases on scientific quality. One expert noted that the papers lacked a narrative. Judgments like that remain a human responsibility for now.

What to do in practice

If you are building agents that design experiments, run benchmarks, or write reports autonomously, the conclusion is straightforward: an agent must not validate itself through the same pipeline it created. A verification script that takes its task list from the agent will confirm any omission.

Several concrete measures can help:

• verify reports independently by automatically comparing reported figures with raw logs;
• define the expected list of runs externally instead of letting the agent create it;
• require the agent to fix detected problems, not merely document them;
• prohibit silent reductions in scope; every reduction must be explicit;
• do not try to buy quality with longer runs; it is better to cap the time and evaluate the result.

The benchmark's code and data have been published, and the project page includes a leaderboard. For teams building research agents, it provides a ready-made integrity test suite. The gap between “running experiments” and “conducting independent research” remains wide, especially when there is no single answer that can be checked unambiguously.