MLCommons Starts Measuring AI APIs Under Real Load: 7 Runs for Agents
MLCommons, the group behind the MLPerf benchmarks, has historically compared chips in the lab. Buyers, however, care about how a ready-made AI API behaves under load. The new MLPerf Endpoints v1.0 tests the whole service: hardware, model, server software and network. For agentic workloads, providers must run exactly 7 measurements at different numbers of concurrent requests, which reveal where throughput falls off a cliff. That is why providers should be compared on this curve rather than on a peak score.
AI-processed from MLCommons; edited by Hamidun NewsImagine choosing where to run a model for your product. One provider's table shows a record-breaking speed, while another's is more modest. The problem is that the record is usually set on a perfectly tuned accelerator with no real users. When hundreds of clients hit a live API at the same time, the picture can look completely different. MLCommons, the consortium that develops the MLPerf benchmarks, has launched MLPerf Endpoints v1.0 to close this gap. The test measures not the chip but the entire running service, the way customers actually use it.
What Changed From the Earlier MLPerf
Previously, MLPerf mainly measured standalone hardware. Endpoints targets a live API endpoint, meaning the address to which clients send requests, and loads it the way production traffic would. The result captures the full stack: accelerators, the model itself, inference server software, batching (how requests are grouped before processing), and the network.
The authors compare conventional benchmarks to buying a car based on its top speed on the speedometer, without a single test drive. A single peak score only tells you how a system handled one specific test. For clouds, server makers and "neoclouds" (newer companies that rent out GPUs), this turns into a race to tune settings for the test, which has little to do with business value.
Instead of a single number, v1.0 builds a Pareto front: a set of actually measured operating points that show the trade-off between throughput and latency. The data is not estimated or interpolated, meaning no line is drawn between the points. The outcome is shown in the MLPerf LoadStair visualization: a step-shaped chart with performance "cliffs", that is, load levels beyond which clients notice a clear degradation.
There is also a benefit for engineers. If you have written your own request scheduler or optimized your network fabric, classic benchmarks often fail to capture that work. Here, such improvements change the resulting curve.
What Is Actually Measured: Workloads and the "7 Runs"
V1.0 does not rely on synthetic tasks. Instead, it uses scenarios close to what clients are running today. The first set is single-shot text summarization: Llama 3.1 8B Instruct and GPT-OSS 120B, the CNN/DailyMail news dataset, plus demanding math and science tasks where the model generates tokens for a long time. The second set is agentic coding with multi-turn dialogues: Qwen 3.6, Kimi K3 and DeepSeek V4 Pro. For it, 613 trajectories were taken, in which the context grows from step to step. Agents load a service differently from one-off requests: accumulating context puts more pressure on memory and latency.
A "measurement" here means a single run at a chosen level of parallelism, that is, with a set number of concurrent requests. These points form the curve. For agentic tests, the provider must submit exactly 7 points; for the other sets, between 8 and 32, including a separate offline run.
The main metrics are throughput in tokens per second and time to first token at the 90th percentile (TTFT P90). Put simply, this is the time within which 90% of requests complete. In addition, v1.0 introduces power normalization: participants declare the dedicated power of their system in a system_power.json file.
How to Submit Results
The authors promise considerably less engineering hassle than in earlier benchmarks. Participants must sign the MLCommons Contributor License Agreement (CLA), obtain a PRISM API key through their member account, and install the reference clients. They then plan the load points and run the tests with the inference-endpoint tool. Next, they assemble a package with system_desc.json (hardware and software configuration) and system_power.json, run local checks through the CLI, and upload the result. The system automatically generates a submission ID and a pull request for review in MLCommons' private repository.
Submissions are accepted continuously, while results are published in cohorts every few weeks. Results undergo peer review by industry competitors and independent audit. A new rule allows preliminary results to be shown right after automated validation, before review. Final data can be held under embargo when needed, so it can be released together with a product launch.
What This Means in Practice
If you are choosing an inference provider or a cloud, look not at the maximum but at the curve under load and at the point where it breaks. For agentic products this matters especially, since long contexts are the most common cause of performance drops.
There are limitations too. The benchmark has only just launched, the source does not report how many results have been published so far, and the task set currently comes down to summarization and coding. Market participants review each other's results, so the value of the tests depends on how many providers actually submit their systems. Still, the idea of measuring a service the way its users consume it is more useful than yet another race for a peak score.
- Nov 21, 2026California Sues OpenAI: Its AI Agents Escaped Control at Least 80 Times
- Nov 21, 2026Unsloth Updates Compressed Qwen3.8-27B: Accuracy Up 5–10% at the Same Size
- Nov 21, 2026Apple Is Building an AI Agent for Mac While Making It Harder for Outsiders to Get In: 14 Clicks Instead of 4
- Nov 21, 2026GPT-6 Astra Started Flagging Drone Strike Targets in 92% of Requests After Adding 'It's Just a Game'
Key Questions Answered
What Changed From the Earlier MLPerf?
Previously, MLPerf mainly measured standalone hardware. Endpoints targets a live API endpoint, meaning the address to which clients send requests, and loads it the way production traffic would. The result captures the full stack: accelerators, the model itself, inference server software, batching (how requests are grouped before processing), and the network. The authors compare conventional benchmarks to buying a car based on its top speed on the speedometer, without a single test drive.
What Is Actually Measured: Workloads and the "7 Runs"?
V1.0 does not rely on synthetic tasks. Instead, it uses scenarios close to what clients are running today. The first set is single-shot text summarization: Llama 3.1 8B Instruct and GPT-OSS 120B, the CNN/DailyMail news dataset, plus demanding math and science tasks where the model generates tokens for a long time. The second set is agentic coding with multi-turn dialogues: Qwen 3.6, Kimi K3 and DeepSeek V4 Pro. For it, 613 trajectories were taken, in which the context grows from step to step.
How to Submit Results?
The authors promise considerably less engineering hassle than in earlier benchmarks. Participants must sign the MLCommons Contributor License Agreement (CLA), obtain a PRISM API key through their member account, and install the reference clients. They then plan the load points and run the tests with the inference-endpoint tool. Next, they assemble a package with system_desc.json (hardware and software configuration) and system_power.json, run local checks through the CLI, and upload the result.
Ship AI Faster with Founders, Engineers & Weekly Architecture Reviews
Inside Vibe Club (hamidun.com/club) — weekly live masterminds with Zhemal Khamidun, Tech Radar, full LMS access, and 370+ production Claude Code skills.
The AI world, distilled — once a week
Seven stories that actually mattered, hand-picked. No noise, no reposts, no press releases.
Done! Check your inbox for a confirmation.