FrontierCS 2 tested 12 AI agents on R&D tasks: the best passed 58%

The FrontierCS 2 benchmark asks a question the industry usually avoids: can an LLM agent do more than solve a textbook problem and make a contribution worthy of a research paper? The set consists of 48 open tasks across 13 domains. Of the 12 agents tested, the best, Claude Opus 5.5, passes 58% of its runs. On 16 tasks, no run passed, so a third of the set scores zero.

How a paper becomes an exam

The design is simple and fairly strict. The authors take a research paper and the code its creators released publicly, then delete from the repository exactly the code that implements the paper's contribution. The removed part becomes the human reference. The bar is not an abstract threshold but what the researchers actually came up with.

The agent gets a working repository without the key component and must restore it or beat it. The tasks span 13 domains: machine learning systems, high-performance computing, databases, security, robotics and others. Beyond the 48 published tasks, FrontierCS 2 has a closed set of more than 100 tasks, which will be announced later. It guards against overfitting to the public benchmark: you can't tune an agent to tasks it has never seen.

A research environment instead of a single attempt

Previous open benchmarks usually handed the agent a problem statement and scored a single program after the run finished. FrontierCS 2 changes the format: the agent works inside a research environment and can submit a patch at any moment. For each patch, the judge returns a score on a development set. That gives the agent a signal it can use to improve its solution iteratively, as a human would.

The evaluator itself, the held-out final test set and the reference solution all stay closed. The final submission is checked on the hidden set and compared with the reference. An agent can tune its solution to the visible metric, but it will be tested on data it has not seen.

A high score is not a pass

The most interesting part of the results is not the 58% figure but the gap between a score and a pass. Most runs that beat the reference on total final score still failed, because they fell short on an individual metric or an individual workload. For systems work, this is a typical pattern. A database or HPC-kernel optimization that speeds up one scenario and breaks another is useless in production, and the benchmark accounts for that.

The authors also built a common ability scale for agents and tasks. The median task turned out to be about as hard as the strongest agent can solve. The set is not too easy, or results would hit a ceiling, and it is not unreachable by design. It leaves room to measure the progress of the next models.

Key parameters

• 48 open tasks in 13 domains, plus a closed set of 100+ tasks
• 12 agents tested
• Best result: Claude Opus 5.5, 58% of runs passed
• 16 tasks on which no run passed
• The agent can submit patches at any moment and receives a score on the development set
• The evaluator, the final test set and the reference are hidden from the agent

What teams building agents should watch

For those building coding agents and research pipelines, the benchmark offers three practical takeaways. First, evaluate across a set of metrics and workloads, not a single aggregate number: an average gain hides regressions. Second, test separately the ability to iterate on development-set feedback and still hold up on the hidden set, because this is where an agent differs from a code generator. Third, the failure on 16 tasks shows where the ceiling is. These tasks demand research intuition on top of programming skill, and the closed set will show whether that intuition improves in the next generations of models.