Evaluating AI Models Now Costs More Than Training Them, Creating a New Barrier for Researchers

Running a full evaluation of an AI agent in 2026 costs between $2,800 and $40,000 per run. The EvalEval Coalition has published an extensive report: benchmarking is no longer a line item next to model training in the budget. It has become a computational and financial barrier in its own right, with all the consequences that carries for the independence of evaluation.

The numbers behind the benchmarks

The researchers collected concrete data on eight widely used evaluation systems:

• HAL (a comprehensive agentic leaderboard): $40,000 for 21,730 runs across 9 models and 9 benchmarks
• GAIA: up to $2,829 per run without caching
• PaperBench: $4,200 to $9,500 depending on the protocol
• The Well (ML for physics tasks): ~$2,400 per architecture, ~$9,600 for a full sweep
• MLE-Bench: ~$5,500 per seed (75 Kaggle tasks × 24 hours on GPU + API)

A single GAIA run is comparable to a graduate student's typical annual travel budget. Comparing six models across three seed runs costs roughly $150,000. Some benchmarks require actual training, and there the compute cost of evaluation exceeds the cost of the training itself by roughly a factor of one hundred.

Why agentic tests can't be compressed

For static language benchmarks, compression has long worked: Flash-HELM shrinks a test by 100–200 times without losing ranking accuracy, and tinyBenchmarks cut MMLU from 14,000 examples to 100 with an error of about 2%.

Agentic benchmarks resist the same techniques. Task costs within a single agentic test vary by a factor of 10,000. Expensive tasks do not yield proportionally more accurate results: on Mind2Web, a 9× price difference corresponds to an accuracy difference of just 2%. The maximum compression gain is 2–3.5×, two orders of magnitude worse than for static benchmarks.

Reliability adds another multiplier. On τ-bench, the same model scored 60% on a single run and only 25% over eight. A statistically sound measurement requires at least k=8 repetitions, which multiplies the cost by 8: a $10,000 test becomes an $80,000 one.

"The conventional view is that the main constraint is model capability.

But the evaluation points to a different bottleneck: reliability," says the EvalEval Coalition.

Independent verification becomes a privilege

When three seed runs for six models cost $150,000, academic groups are priced out. Only large labs have the budgets for statistically sound evaluation, and they are the same labs that build the systems being evaluated. This is a structural conflict of interest: external verification disappears not because nobody wants it, but because nobody can pay for it.

The EvalEval Coalition proposes a pragmatic answer: stop running the same tests over and over. Today every group starts from scratch because other groups' results are buried in PDF papers with no machine-readable data. The coalition has launched Every Eval Ever, a Hugging Face repository where results are submitted with full metadata, logs and parameters. By its calculation, even a twofold reuse of data would save more than all compression techniques combined.

What this means

The economics of AI evaluation have flipped: evaluation is no longer a minor budget line but a major operating expense and a tool of influence. Whoever can pay for a benchmark writes the leaderboard. If independent verification keeps getting more expensive, external oversight of AI systems risks ending up entirely in the hands of the labs that build them.