Anthropic: Infrastructure Configuration Shifts Terminal-Bench Scores by 6 Percentage Points

Infrastructure can shift the result of an agentic coding benchmark by several percentage points, and sometimes by more than the gap that separates the leaders on a leaderboard. That is the conclusion Anthropic engineers reached after running Terminal-Bench 2.0 in six resource configurations. Between the strictest and the most generous setup, the difference was 6 pp at p < 0.01. The model, the harness and the task set stayed the same throughout.

How the problem was found

Anthropic runs Terminal-Bench 2.0 on a Google Kubernetes Engine cluster. During calibration, the team noticed that its scores did not match the official leaderboard and that the rate of infrastructure errors was unexpectedly high: up to 6% of tasks failed because of pod crashes that had nothing to do with the model's abilities.

The cause turned out to be how the limits were applied. The Kubernetes implementation treated a task's recommended resources as both a guaranteed minimum and a hard ceiling. A container got exactly the amount of memory specified and was killed at the slightest overshoot. A brief spike in consumption triggered an OOM-kill, even though the task could have finished successfully. The sandbox provider used by the Terminal-Bench leaderboard is more lenient: it allows temporary overshoots and does not kill the container.

What the experiments showed

The authors ran the benchmark at settings ranging from strict enforcement of the specs (1x) to no limits at all. The key figures:

• the infrastructure error rate falls monotonically: from 5.8% at 1x to 0.5% with no limits;
• moving from 1x to 3x cuts errors from 5.8% to 2.1% (p < 0.001);
• within the 1x–3x range, success rates fluctuate within noise (p = 0.40);
• above 3x, success rises faster than errors fall: a further 1.6 pp drop in errors yields almost +4 pp in success;
• the total gain with no limits relative to 1x is +6 pp (p < 0.01);
• on SWE-bench (227 tasks, 10 samples each, RAM up to 5x), the gain is only 1.54 pp.

Up to the 3x mark, extra resources cure instability: the benchmark becomes more reliable, but not easier. Beyond that, the agent starts solving what it could not solve before: it pulls in heavy dependencies, launches expensive subprocesses and runs resource-hungry tests. The tasks rstan-to-pystan and compile-compcert improved the most.

What the benchmark actually measures

Tight limits reward frugal strategies; generous ones reward brute-force approaches. One example is the bn-fit-modify task, which involves fitting a Bayesian network. Some models immediately install pandas, networkx and scikit-learn. With generous limits, this works; with strict ones, the pod crashes from lack of memory during installation. There is also a lightweight path: implement the math with the standard library, and some models choose exactly that. The resource configuration decides which habit pays off.

The effect reproduced across different Anthropic models, although its magnitude varied. On other model families the trend apparently holds, but this was not rigorously tested. There are also less-studied factors: Anthropic anecdotally observed pass rates fluctuating with the time of day, probably because of API latency.

Practical takeaways

The authors recommend specifying two parameters for each task, a guaranteed amount and a container-kill threshold, rather than a single value. The gap between them should be calibrated so that scores at the lower and upper bounds differ only within noise. For Terminal-Bench 2.0, a 3x ceiling cut infrastructure errors by about two thirds and barely inflated the result. The multiplier should be published alongside the results.

For those who read leaderboards, the takeaway is straightforward: treat a lead of under 3 pp with skepticism if the configuration is not documented. The spread across the moderate resource range is a little under 2 pp, and it overlaps with the usual binomial confidence intervals of 1–2 pp, while at the extremes of the range it reaches 6. Labs should treat resources as a full-fledged experimental variable, on par with prompt format and temperature. Benchmark maintainers should publish not only the recommended specs but also the methodology for applying them. Otherwise, a lead of a couple of points may simply mean a more powerful virtual machine.