SciGame Tests Five LLM Agents: Familiar Formulas Get in the Way of Finding Altered Laws

Researchers have introduced SciGame, a benchmark that tests whether LLM agents can discover scientific laws rather than recall them from training data. The authors package each formula from a scientific paper into a structured "card" and build two environments with an identical interface from it: an original and a counterfactual. In the second, the relationship is altered, but the dimensions of the quantities stay consistent. An agent that relies on memorized knowledge will fail in the second environment. Experiments on five frontier models showed that this is exactly what happens.

What earlier benchmarks lacked

The authors identify four problems with existing tests. First, they often change the task, the observation protocol, and the metric all at once, so the regimes cannot be compared on a single interface. Second, many metrics score data fitting rather than prediction on unseen inputs, so a plausible explanation can earn a high score even when the predictor is wrong. Third, most datasets are static and contain almost no laws published after a model's knowledge cutoff. Fourth, paper-based tasks mix law discovery with programming and experiment reproduction.

Other works address some of these aspects: DiscoveryWorld and Auto-Discovery-Bench study hypothesis revision and experiment selection, LLM-SRBench covers equation search, Gravity-Bench changes the physical context, PaperBench tests research reproduction, and SciLaws-Bench builds tasks from papers. But none of them compares the same task with and without candidates. In SciLaws-Bench, the Real and Parallel settings use different observation and evaluation protocols, so their accuracy figures are not comparable.

How the evaluation works

The card records the relationship between quantities, parameters, variable definitions, units of measurement, the admissible input domain, and the conditions of applicability. From it, the authors generate environments with a deterministic simulator. The counterfactual version is created through structural mutations modeled on NewtonBench (a benchmark with altered physical laws): coefficients and exponents change, and nonlinear transformations and cross-dependencies between inputs are added. The authors verified that the reference formulas of each environment fail evaluation on observations from the paired environment. This means the environments can be told apart empirically.

There are two types of tasks. In Identification, the agent picks a law from a set of candidates: it is given a formula that combines the original and counterfactual expressions through an unknown parameter. The simulator always sets this parameter to zero or one, but the agent is not told so. In Discovery, there are no candidates, and the agent must construct the relationship between inputs and observables from the description, constraints, and observations. In both cases, the agent submits executable expressions, an explanation, and conditions of applicability. Each task family yields four tasks.

Each task runs under three protocols:

• Prior-only: there are no observations, and the agent relies on memorized knowledge;
• Fixed: the agent receives a predefined set of observations;
• Adaptive: the agent proposes experiments itself and decides when to submit an answer.

The benchmark includes two collections. Historical contains canonical laws from the history of science as interpretable references. Temporal is built by a rerunnable pipeline: it admits only papers whose journal publication date and first preprint date both fall after the knowledge cutoff of the model being evaluated. One LLM extracts quantitative relationships, and another independently checks the cards for accuracy and completeness, fixing or removing incomplete ones. The time window shifts along with new models to reduce the risk of memorization.

What the experiments showed

There are three scores: prediction accuracy on held-out inputs, hierarchical episode success, and a rubric-based explanation score. The authors draw three conclusions. First, familiar formulas help on original laws but hurt when the relationship is altered. Second, adaptive experiments benefit the strongest models but hurt weaker ones under a limited budget. Third, a good explanation can hide a quantitative failure: the text is convincing, but the predictor is wrong.

Recovering the exact symbolic form remains the main bottleneck, even when the agent interprets the observations correctly. Incorrect functional dependence is the main type of error. The authors conclude that modern agents can actively explore systems, but deriving exact laws in unfamiliar conditions remains an open problem.

The takeaway for developers of scientific agents is practical. Prediction on unseen inputs should be measured separately, and a coherent explanation should not be counted as success. An adaptive experiment budget makes sense only for models that can manage it. The available excerpt of the paper contains no tables with scores, so specific models cannot be compared by the numbers. The authors have released the code and data.