Free Intensive Nov 21–22Free 2-day hands-on Vibecoding Intensive on Nov 21–22 at 16:00 MSK (Sat–Sun) · Live Cohort 3 starts Nov 23Join Free →Cohort 3 (Nov 23)
InfrastructureResearch Paper · For R&DMLCommons→ original

MLPerf Training: 10,240-GPU clusters and 49% faster training in six months

MLCommons has published MLPerf Training v6.1, an exam for clusters: everyone trains the same model to the same quality target, and the clock decides. The largest runs grew from 8 192 to 10 240 GPUs. On the four unchanged benchmarks, average training time fell by a factor of 1.49 in six months: from 100 minutes to about 67. Part of that gain comes from software, so when choosing a cluster, check recent results rather than hardware spec sheets.

AI-processed from MLCommons; edited by Hamidun News
MLPerf Training: 10,240-GPU clusters and 49% faster training in six months
Source: MLCommons. Collage: Hamidun News.
Listen to article
Article Fact Sheet & Key Figures
ParameterDetails
Primary sourceMLCommons ↗
Editor-in-Chief
Key metric #125 submitters and 250 submissions, versus 24 and 229 in v6.0
Multi-node systems76% of Closed results, versus 61% in v6.0
New benchmarksagentic RL on Qwen3.5-397B and the DLRMv4 recommender

Imagine you are choosing where to train a model: rent GPUs in the cloud or build your own cluster. Vendors' spec-sheet numbers tell you little. MLPerf Training is a common exam: every participant solves the same task on the same data to the same target quality, and the measure is time to result.

In the latest round, v6.1 from MLCommons, the largest runs grew to 10 240 GPUs (the previous ceiling was 8 192). On the four benchmarks that did not change since the last round, the best time fell by a factor of 1.

49 on average, or roughly 49% in speed. In rough terms, training that took 100 minutes six months ago now finishes in 67. For you, this is a guide to how much real speedup you can expect from changing hardware and software settings.

What changed in this round

MLCommons received 250 submissions from 25 participants; v6.0 had 229 from 24. The largest runs used 10 240 GPUs: NVIDIA on GB300 and Azure on GB200 trained DeepSeek-V3. Google ran the same model on 8 192 TPU7x (Ironwood) chips in a single slice with a fast internal ICI network. Ironwood and NVIDIA Rubin debuted in the Available category, meaning the systems are already available to customers. In total, the round featured 12 accelerators from three vendors and four interconnect technologies between nodes.

The typical system got bigger. The median number of accelerators in a Closed result stayed at 8 from v4.0 through v6.0; now it is 32. Results on 64 or more accelerators rose to 111 from 73. Multi-node systems account for 76% of Closed results (117 of 153), versus 61% in v6.0. About 28% of submissions came from the cloud, so virtualized infrastructure is keeping pace with the rest. The University of Florida ran the benchmark on half of a B200 SuperPOD while the other half kept serving researchers.

Two benchmarks were added. The first is agentic reinforcement learning: a 397-billion-parameter model (Qwen3.5-397B) is trained to fix real software projects; seven participants ran it, with 29 submissions. The second is the generative recommender DLRMv4 (HSTU), which replaced the old DLRM-DCNv2: eight participants, 20 submissions. The large dense pretraining model was dropped after three rounds: the industry has moved to mixture-of-experts architectures, where only part of the model works on each token. DeepSeek-V3 and Qwen3.5-397B now cover that niche.

Where the 49% comes from, and why the gains are uneven

1.49 is the geometric mean of the improvement in best results across the four unchanged benchmarks. It is not an official MLPerf result but an index that MLCommons calculates itself. The systems in the comparison are the same size: 512 accelerators, except for Llama 3.1 8B, where v6.0 used 1 024. The gain is unevenly distributed: FLUX.1 sped up noticeably over six months, while the small Llama 3.1 8B barely changed. Part of the gain comes from software: Firmus and KRAI showed notable improvements over baseline implementations, including through AI-assisted optimizations.

Why more GPUs does not mean faster

The benchmark measures training time on a fixed task, so a larger cluster should finish the same work sooner. But gradient synchronization between accelerators does not shrink as the cluster grows, so 100 accelerators will not train a model 100 times faster. Batch size is also limited: Llama 3.1 8B processes about 64 sequences per step, so on hundreds of accelerators each sequence has to be split, adding data exchange inside every layer.

DeepSeek-V3 has a batch of about 16 000 sequences, but its problem is different: memory. The training state of a 671-billion-parameter model takes more than 10 TB, while an accelerator's HBM is 186–288 GB. So the model is split across pipeline stages and experts, and some computations are redone to save memory. On NVL72 racks and TPU7x slices, the fast internal network spans many nodes, so how a job divides traffic between the fast and slow networks increasingly determines speed.

Processors are a separate matter. In agentic RL (for example, Qwen3.5 GRPO), CPUs run the environments and tests that turn the model's answers into a reward. Participants used from 500 to about 3 150 parallel sandboxes. Host CPU capacity will matter more as agentic tasks grow.

What to do with this

When choosing a platform, look not at peak GPU count but at results for a task similar to yours, and at the network design. For small models, hundreds of extra accelerators will barely help. For agentic pipelines, budget CPU capacity for sandboxes.

Caveats: participants submit the results themselves, the first results on new benchmarks partly reflect their novelty, and in this round only Korea's TTA measured energy.

Story

Key Questions Answered

What changed in this round?

MLCommons received 250 submissions from 25 participants; v6.0 had 229 from 24. The largest runs used 10 240 GPUs: NVIDIA on GB300 and Azure on GB200 trained DeepSeek-V3. Google ran the same model on 8 192 TPU7x (Ironwood) chips in a single slice with a fast internal ICI network. Ironwood and NVIDIA Rubin debuted in the Available category, meaning the systems are already available to customers. In total, the round featured 12 accelerators from three vendors and four interconnect technologies between nodes.

Where the 49% comes from, and why the gains are uneven?

1.49 is the geometric mean of the improvement in best results across the four unchanged benchmarks. It is not an official MLPerf result but an index that MLCommons calculates itself. The systems in the comparison are the same size: 512 accelerators, except for Llama 3.1 8B, where v6.0 used 1 024. The gain is unevenly distributed: FLUX.1 sped up noticeably over six months, while the small Llama 3.1 8B barely changed.

Why more GPUs does not mean faster?

The benchmark measures training time on a fixed task, so a larger cluster should finish the same work sooner. But gradient synchronization between accelerators does not shrink as the cluster grows, so 100 accelerators will not train a model 100 times faster. Batch size is also limited: Llama 3.1 8B processes about 64 sequences per step, so on hundreds of accelerators each sequence has to be split, adding data exchange inside every layer.

ZK
Zhemal Khamidun — CPO at Alpina Digital, product manager of AlpinaGPT. Building AI products since 2019. Hamidun News is a pet project that grew into a production aggregator with 6,000+ readers.
✦ Unified AI Workspace · Nomad

Test Frontier LLMs & Autonomous Agents in One Workspace

Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.

Try Nomad AI →Launch Nomad Agent
What do you think?
Loading comments…