Claude Opus 5 built simple baselines that closed 95.5% of the gap to DL models from Nature and Science

A team at Carnegie Mellon University has proposed using a language model to vet high-profile deep learning papers by automatically building simple baselines for them. In the experiment, Claude Opus 5 received 36 tasks drawn from papers in Nature, Science and Cell-family journals, and wrote a script with a simple model for each one. The median result was surprisingly high: the baselines closed 95.5% of the gap between a trivial predictor and the published neural network, and the average across tasks was 85.7%. On 12 of the 36 tasks, the simple model matched deep learning or beat it.

How baseline strength was measured

The authors introduced a metric called Simple Baseline Strength (SBS). It is a dimensionless ratio that runs from the result of a trivial predictor (a constant such as the mean of the target or the most frequent class) to the result of the published DL model. An SBS of 0 means the baseline is no better than the trivial predictor. A value of 1 means it matched the neural network, and a value above 1 means it beat it. Normalization makes it possible to compare tasks that use different metrics. For metrics where lower is better, the formula works the same way.

Tasks were selected from open-access papers published between 31 January 2024 and 18 September 2026. The requirement was public data, so the sample reflects only studies that released their datasets. The tasks span eight scientific fields, with at least three tasks in each, and at least three tasks from each Nature-family journal. Baseline results were not used when selecting tasks.

Each attempt ran in a container with no network access, apart from access to the model provider's API. Test labels and result files were kept outside the container. The agent wrote and ran a single script with a 30-minute limit, on CPU only and without pretrained weights, since checkpoints trained on public data could have seen the test set. The prompt asked for a "reasonable baseline" rather than necessarily the simplest one, so neural networks were allowed. Of the 157 scoreable scripts, 47 used PyTorch and 85 used gradient boosting over trees.

Where simple models fell short

Each task was repeated five times. Of the 180 attempts, 157 produced valid predictions; 16 tasks had at least one failed attempt, and three were left with no more than two scoreable runs. If failures are counted as zeros, the average drops to 79.6%, while the median stays at 95.5%.

Six tasks showed a median SBS below 0.5. All six had at least one failed attempt. In all six, the agent was given not tabular features but raw sequences, string descriptions of structures or atomic coordinates. But the failure cannot be explained by input type alone: several tasks with the same kind of data came out near 1 or higher. For two of the six, the median is based on only two scoreable repeats. The agent's own validation score correlates only roughly with the test score, and repeats of the same task usually give similar results.

The authors also compared setups with different amounts of compute on a separate set of nine tasks. Some of these runs used earlier builds of the environment in which task documentation was hidden, so a direct comparison calls for caution. They also analyze the effect of the level of research context, but the available text gives no figures for it. The authors did not test differences between journals and fields and draw no conclusions from their ordering: the groups are small, with three to six tasks per field.

Why this matters for reviewers

The motivation is that peer review cannot keep up with the flow of papers. Springer Nature reported 3.1 million submitted manuscripts in 2025, 30% more than a year earlier. ICLR submissions grew from 11,603 to 19,525, and NeurIPS submissions from 9,467 in 2020 to 21,575 in 2025. On arXiv, the number of papers grew by 36.4% over 2023–2025, from 208,493 to 284,486.

Missed baselines have already distorted the picture. In predicting cell responses to perturbations, five foundation models did not outperform simple methods. In long-term time-series forecasting, single-layer linear models beat the Transformer on nine datasets. And on 45 tabular datasets, trees remained stronger than neural networks.

The practical takeaway for authors and reviewers is that an automatic baseline can be run as a cheap check on a claimed improvement. It does not prove that the DL method is useless: it may offer gains in interpretability or on other data. But if a simple model on a CPU recovers almost the entire gap in 30 minutes, the authors should explain exactly what justifies the complex architecture.