MLCommons launches AILuminate: how to put a price tag on AI risk

The risk is already on the books

AI risk-management theory isn't some new philosophy. Any mature organization already runs a risk-management cycle: identify vulnerabilities, estimate expected losses, apply controls proportional to the size of the risk, and report to whoever is accountable. Insurers value this, auditors test for it, and boards sign off on it.

But the cost of failures is rising. IBM's 2025 Cost of a Data Breach study found the average breach costs $4.44 million globally and a record $10.22 million in the US. 63% of breached organizations either had no AI governance policy at all or were still drafting one. Companies with significant "shadow AI" — systems deployed haphazardly and without oversight — lose an average of $670,000 more.

Regulation adds further penalties. GDPR and CCPA, standards conceived before generative AI even existed, already sharply limit what decisions an AI system is allowed to make. The EU AI Act provides for fines of up to €35 million or 7% of global turnover — a strong incentive for proactive risk management.

How aviation became safe

The payoff from good management of a new risk isn't just avoided losses — it's the health of an entire industry. In 1959, US commercial aviation had roughly 40 fatal accidents per million departures. Within a decade, that fell below 2. Over the following 50 years, the rate kept roughly halving each decade.

Aviation became safe because the industry built a risk-management system and kept refining it. Measurement built trust, and trust let flying become routine. Several billion passengers a year, plus insurers and lenders backing more than 30,000 commercial aircraft, now treat flying as utterly ordinary.

The end of self-grading

Managing AI risk requires honest measurement. Pharmaceutical companies don't judge their own clinical trials. Public companies don't audit their own financial statements.

Yet in the AI industry, vendors still grade their own work, run their own benchmarks, and report their own results. Take the code-review case: a vendor ran its own benchmark and reported 82% bug-catch accuracy; a competitor reran it on the same repositories and got 45%; when an independent lab built its own benchmark, no vendor scored above 63%. The products hadn't changed.

In December 2024, MLCommons launched AILuminate — a family of independent, consensus-governed benchmarks that assess how reliably an AI system avoids harmful responses. They're built by working groups that include AI companies, academia, and civil society. Vendors don't write their own exam, don't see the questions, and don't hold the scorecard.

What to do with the scores

Scores broken out by harm type show where risk concentrates for your specific use case. That's raw material for a risk register.

Results should drive controls. If a system's score drops under adversarial pressure, that's a concrete, evidence-backed signal to build system-level defenses — not a vague mandate to "be careful."

Test your own configuration. This is the step most programs skip. As the buyer of a system, test exactly what you'll deploy, using the same test the vendor used. The evaluation needs to reflect real-world performance with your specific integration — the difference between a car's label fuel-economy figure and what you actually get on your road. The label describes the test track, not your commute.

Agent Reliability Profile

A model-level evaluation shows how a model performs straight off the assembly line. But it says nothing about how it will perform once deployed. Most companies don't deploy raw models — they deploy wrapped ones: embedded in prompts, connected to internal data, equipped with tools, sometimes operating as agents. That wrapping is exactly what makes a model useful — and fragile.

MLCommons' financial-services working group developed the Agent Reliability Profile — a per-agent artifact that records a concrete, falsifiable claim: this system reliably operates at a specified level of autonomy, within a specified operational scope, under a specified control envelope. Not a vague "vibe" about how "reliable" an agent feels, but a concrete claim an auditor or insurer can actually verify. This profile was a finalist at the CDIR hackathon.

Treat AI like any other risk

AI risk can be identified, priced, managed proportionally, and reported — just as companies have done for decades with credit risk, currency exposure, and workplace safety. This isn't a brake on AI adoption; it's a precondition for it. The risk is already on your books. The only question is whether you'll measure it.