AI Model Evaluation Now Costs More Than Training — A New Barrier for Researchers

Running a full evaluation of an AI agent in 2026 costs between $2,800 to $40,000 per run. EvalEval Coalition released an extensive report: benchmarking has stopped being a line item in the budget next to model training and has become a standalone computational and financial barrier — with all the implications for evaluation independence.

Benchmark Figures

Researchers collected specific data on eight widely used evaluation systems:

• HAL (comprehensive agent leaderboard) — $40,000 for 21,730 runs across 9 models and 9 benchmarks
• GAIA — up to $2,829 per single run without caching
• PaperBench — from $4,200 to $9,500 depending on protocol
• The Well (ML for physics tasks) — ~$2,400 for architecture, ~$9,600 for full sweep
• MLE-Bench — ~$5,500 per seed (75 Kaggle problems × 24 hours on GPU + API)

A single GAIA run is comparable to a typical annual travel budget for a graduate student. Running three seeds across six models costs approximately $150,000. Some benchmarks require actual training — and there, the computational cost of evaluation exceeds the cost of training itself by roughly a hundred times.

Why Agent Tests Can't Be Compressed

For static language benchmarks, compression has long worked: Flash-HELM shrinks a test 100–200 times without losing ranking accuracy, and tinyBenchmarks reduced MMLU from 14,000 examples to 100 with roughly 2% error. Agent benchmarks resist the same techniques. The cost of tasks within a single agent benchmark varies by a factor of 10,000. Yet expensive tasks don't yield proportionally accurate results: on Mind2Web, a 9× price difference corresponds to only a 2% accuracy difference. Maximum compression effect is 2–3.5 times, two orders of magnitude worse than static benchmarks.

An additional multiplier is reliability. The same model on τ-bench showed 60% in one run but only 25% in eight runs. Statistically valid measurement requires a minimum of k=8 repetitions, automatically multiplying the cost by 8: a $10,000 test becomes $80,000.

"It's conventionally believed that model capability is the main limiting factor.

But evaluation shows: the real bottleneck is reliability," — EvalEval Coalition.

Independent Verification Becomes a Privilege

When three seed runs for six models cost $150,000, academic groups are physically knocked out of the game. Only large laboratories have budgets for statistically sound evaluation — the same ones creating the systems being evaluated. This is a structural conflict of interest: external verification doesn't disappear because people don't want it, but because nobody can afford it.

EvalEval Coalition proposes a pragmatic solution: stop running the same tests over and over. Currently each group starts from scratch because other results are buried in PDF papers without machine-readable data. The coalition launched the Every Eval Ever project — a repository on Hugging Face where results are submitted with full metadata, logs, and parameters. It's been calculated that even just reusing data twice would save more than all compression techniques combined.

What This Means

The economics of AI evaluation have flipped: evaluation is no longer a minor budget line item but a primary operational cost and instrument of influence. Whoever can afford to pay for a benchmark writes the leaderboard. If independent verification continues to become more expensive, external oversight of AI systems risks becoming entirely concentrated in the hands of the laboratories that create them.