MIT Technology Review: Why Standard AI Benchmarks No Longer Show Real-World Value

MIT Technology Review writes that familiar AI benchmarks are increasingly poor at showing how systems behave in real work. A model can win an isolated test and still slow a team down once it is built into a live process.

Why the tests diverge

For decades, progress in AI was measured as a contest of "machine versus human." The approach is convenient: give a model chess, math problems, coding tasks or essays, then compare its result with how an individual person performs. Such tests are easy to standardize, turn into rankings and use in marketing. That is why an entire industry of headline numbers, leaderboards and comparisons has grown around benchmarks, all of which look great in presentations.

The problem is that AI is almost never used the way it is tested. In real work, a system does not exist in a vacuum: it is embedded in teams, processes, rules, deadlines and internal standards. Its value shows up not in a single answer but in a series of interactions over weeks and months. So a high score on an isolated task does not yet mean the model will speed up work, reduce errors or even prove useful to an organization.

Where evaluation breaks down

The article gives a telling example from medicine. Some image-analysis systems read scans faster and more accurately than experienced radiologists in tests. On paper, that looks like a ready-made recipe for higher productivity.

But in a hospital, a decision is rarely made by one specialist at one moment. Radiologists, oncologists, physicists, nurses and other team members may all work around a single case, and the treatment plan changes as new data arrives. When such tools enter the real workflow, it turns out that staff need extra time to interpret the model's outputs, reconcile them with local reporting standards and check compliance with regulatory requirements.

As a result, a system that promised a speedup in testing sometimes creates delays in practice. Moreover, it can reinforce early "anchoring" on a plausible but incomplete answer, increase cognitive load and carry errors further down the chain. This is how the "AI graveyard" appears: products with high scores that fail to take root in real work.

What the author proposes instead

Instead of narrow tests, the author proposes HAIC benchmarks, short for Human-AI, Context-Specific Evaluation. This approach assesses not only the model itself but also how it behaves inside a specific team, process and organizational environment. The point is to bring evaluation closer to real use rather than a lab demonstration.

• Shift the focus from an individual task to the work of the team and the whole process
• Measure the effect not in a single test run but over the long term
• Treat as important not only speed and accuracy but also coordination, the quality of joint decisions and how visible errors are
• Look not only at the model's answer but at the consequences before and after it is applied

The author also describes early examples of this approach. In one British hospital system, the question was posed not as "has diagnosis become more accurate" but as "does AI change the quality of collective discussion and interaction between specialists." In the humanitarian sector, a similar system was observed for 18 months, with separate tracking of how easily people notice and correct the model's errors. Such a long horizon makes it possible to design safeguards for a specific context rather than hope that a high test score by itself guarantees safety and usefulness.

What this means

The industry is gradually hitting the limit of the old metrics: they show well what a model can do on its own, but poorly what happens when it becomes part of a living organization. For business and government, this is a signal to look beyond leaderboards and ask whether AI helps teams work more reliably, faster and more safely in real conditions.