Free Intensive Nov 21–22Free 2-day hands-on Vibecoding Intensive on Nov 21–22 at 16:00 MSK (Sat–Sun) · Live Cohort 3 starts Nov 23Join Free →Cohort 3 (Nov 23)
ReleaseAPI / Product · ActionableMLCommons→ original

MLCommons Starts Measuring AI APIs Under Real Load: 7 Runs for Agents

MLCommons, the group behind the MLPerf benchmarks, has historically compared chips in the lab. Buyers, however, care about how a ready-made AI API behaves under load. The new MLPerf Endpoints v1.0 tests the whole service: hardware, model, server software and network. For agentic workloads, providers must run exactly 7 measurements at different numbers of concurrent requests, which reveal where throughput falls off a cliff. That is why providers should be compared on this curve rather than on a peak score.

AI-processed from MLCommons; edited by Hamidun News
Listen to article
Article Fact Sheet & Key Figures
ParameterDetails
Primary sourceMLCommons ↗
Editor-in-Chief
Key metric #1Agentic coding is run across 613 multi-turn scenarios
Key metric #2Standard tests require 8 to 32 measurements, including an offline run
Time to first token90% of requests complete within this window

Imagine choosing where to run a model for your product. One provider's table shows a record-breaking speed, while another's is more modest. The problem is that the record is usually set on a perfectly tuned accelerator with no real users. When hundreds of clients hit a live API at the same time, the picture can look completely different. MLCommons, the consortium that develops the MLPerf benchmarks, has launched MLPerf Endpoints v1.0 to close this gap. The test measures not the chip but the entire running service, the way customers actually use it.

What Changed From the Earlier MLPerf

Previously, MLPerf mainly measured standalone hardware. Endpoints targets a live API endpoint, meaning the address to which clients send requests, and loads it the way production traffic would. The result captures the full stack: accelerators, the model itself, inference server software, batching (how requests are grouped before processing), and the network.

The authors compare conventional benchmarks to buying a car based on its top speed on the speedometer, without a single test drive. A single peak score only tells you how a system handled one specific test. For clouds, server makers and "neoclouds" (newer companies that rent out GPUs), this turns into a race to tune settings for the test, which has little to do with business value.

Instead of a single number, v1.0 builds a Pareto front: a set of actually measured operating points that show the trade-off between throughput and latency. The data is not estimated or interpolated, meaning no line is drawn between the points. The outcome is shown in the MLPerf LoadStair visualization: a step-shaped chart with performance "cliffs", that is, load levels beyond which clients notice a clear degradation.

There is also a benefit for engineers. If you have written your own request scheduler or optimized your network fabric, classic benchmarks often fail to capture that work. Here, such improvements change the resulting curve.

What Is Actually Measured: Workloads and the "7 Runs"

V1.0 does not rely on synthetic tasks. Instead, it uses scenarios close to what clients are running today. The first set is single-shot text summarization: Llama 3.1 8B Instruct and GPT-OSS 120B, the CNN/DailyMail news dataset, plus demanding math and science tasks where the model generates tokens for a long time. The second set is agentic coding with multi-turn dialogues: Qwen 3.6, Kimi K3 and DeepSeek V4 Pro. For it, 613 trajectories were taken, in which the context grows from step to step. Agents load a service differently from one-off requests: accumulating context puts more pressure on memory and latency.

A "measurement" here means a single run at a chosen level of parallelism, that is, with a set number of concurrent requests. These points form the curve. For agentic tests, the provider must submit exactly 7 points; for the other sets, between 8 and 32, including a separate offline run.

The main metrics are throughput in tokens per second and time to first token at the 90th percentile (TTFT P90). Put simply, this is the time within which 90% of requests complete. In addition, v1.0 introduces power normalization: participants declare the dedicated power of their system in a system_power.json file.

How to Submit Results

The authors promise considerably less engineering hassle than in earlier benchmarks. Participants must sign the MLCommons Contributor License Agreement (CLA), obtain a PRISM API key through their member account, and install the reference clients. They then plan the load points and run the tests with the inference-endpoint tool. Next, they assemble a package with system_desc.json (hardware and software configuration) and system_power.json, run local checks through the CLI, and upload the result. The system automatically generates a submission ID and a pull request for review in MLCommons' private repository.

Submissions are accepted continuously, while results are published in cohorts every few weeks. Results undergo peer review by industry competitors and independent audit. A new rule allows preliminary results to be shown right after automated validation, before review. Final data can be held under embargo when needed, so it can be released together with a product launch.

What This Means in Practice

If you are choosing an inference provider or a cloud, look not at the maximum but at the curve under load and at the point where it breaks. For agentic products this matters especially, since long contexts are the most common cause of performance drops.

There are limitations too. The benchmark has only just launched, the source does not report how many results have been published so far, and the task set currently comes down to summarization and coding. Market participants review each other's results, so the value of the tests depends on how many providers actually submit their systems. Still, the idea of measuring a service the way its users consume it is more useful than yet another race for a peak score.

Story

Key Questions Answered

What Changed From the Earlier MLPerf?

Previously, MLPerf mainly measured standalone hardware. Endpoints targets a live API endpoint, meaning the address to which clients send requests, and loads it the way production traffic would. The result captures the full stack: accelerators, the model itself, inference server software, batching (how requests are grouped before processing), and the network. The authors compare conventional benchmarks to buying a car based on its top speed on the speedometer, without a single test drive.

What Is Actually Measured: Workloads and the "7 Runs"?

V1.0 does not rely on synthetic tasks. Instead, it uses scenarios close to what clients are running today. The first set is single-shot text summarization: Llama 3.1 8B Instruct and GPT-OSS 120B, the CNN/DailyMail news dataset, plus demanding math and science tasks where the model generates tokens for a long time. The second set is agentic coding with multi-turn dialogues: Qwen 3.6, Kimi K3 and DeepSeek V4 Pro. For it, 613 trajectories were taken, in which the context grows from step to step.

How to Submit Results?

The authors promise considerably less engineering hassle than in earlier benchmarks. Participants must sign the MLCommons Contributor License Agreement (CLA), obtain a PRISM API key through their member account, and install the reference clients. They then plan the load points and run the tests with the inference-endpoint tool. Next, they assemble a package with system_desc.json (hardware and software configuration) and system_power.json, run local checks through the CLI, and upload the result.

ZK
Zhemal Khamidun — CPO at Alpina Digital, product manager of AlpinaGPT. Building AI products since 2019. Hamidun News is a pet project that grew into a production aggregator with 6,000+ readers.
✦ Private AI Builders Community · Vibe Club

Ship AI Faster with Founders, Engineers & Weekly Architecture Reviews

Inside Vibe Club (hamidun.com/club) — weekly live masterminds with Zhemal Khamidun, Tech Radar, full LMS access, and 370+ production Claude Code skills.

Join Vibe Club →Explore Hamidun Academy
What do you think?
Loading comments…