AI Training Efficiency: Why Speed Isn't Everything
Training modern language models with hundreds of billions of parameters takes thousands of accelerators and months of work. Efficiency has traditionally been measured by throughput: how many tokens are processed per second. The industry is now turning to a new metric, goodput, which counts not just how fast data is processed but how much real training progress the model makes. The shift is fundamental. Instead of racing for raw performance, companies are starting to optimize useful work, which may sharply cut the cost of training the next generation of AI systems.
AI-processed from TNW; edited by Hamidun News
When the talk turns to training large language models, it almost always comes down to two things: how many GPUs are in use and how fast the system chews through data. Tokens per second have become the industry's currency, and more is better. But what if this metric, for all its clarity, tells only half the story? That is the question raised by the growing concept of goodput, which promises to change how AI training efficiency is assessed.
Pretraining a modern model with a hundred billion parameters or more is an engineering marathon that stretches over weeks and months. Thousands of accelerators work in parallel, processing enormous volumes of text. Success has traditionally been measured by two indicators. The first is throughput: how many tokens the system can process per unit of time. The second is training progress: how much the model actually improves with each iteration. The problem is that the two do not always correlate as neatly as engineers would like.
Throughput is deceptively simple. It shows how quickly data moves through the compute pipeline but says nothing about the quality of that work. Picture a factory line stamping out parts at record speed while half of them end up as scrap. On paper, productivity is high; the real yield is something else entirely. In AI training the analogy is strikingly accurate. A system can post impressive throughput figures while a significant share of its computation is wasted: reprocessing data after failures, idling during synchronization between nodes, and distributing load poorly across accelerators. All the while, the token counter keeps spinning, creating an illusion of progress.
This is where goodput comes in. It tries to measure not raw throughput but useful work, counting only the computation that actually brings the model closer to finishing training. Suppose a cluster of four thousand GPUs processes a trillion tokens a day, but twenty percent of that work is lost to hardware failures, checkpoint restarts and communication overhead between nodes. The real goodput is then only eight hundred billion tokens. The difference looks academic until you convert it into dollars: when renting a large GPU cluster costs millions of dollars a day, a twenty percent loss amounts to hundreds of millions over a training cycle.
The move from throughput to goodput as the key metric reflects a deeper shift in the industry. The era in which AI progress was defined purely by scale (more data, more parameters, more compute) is gradually giving way to an era of optimization. Companies are realizing that clusters cannot be grown indefinitely, either economically or in terms of energy. By various estimates, training a single frontier model already costs hundreds of millions of dollars, and the next generation may cross the billion-dollar mark. Under those conditions, every percentage point of real efficiency matters enormously. Optimizing goodput is no longer a theoretical exercise but a direct tool for cutting costs.
The practical consequences run through the whole chain, from data center design to the architecture of training software frameworks. At the hardware level, this means closer attention to fault tolerance: if one of thousands of accelerators fails, the system should redistribute the load without losing progress, rather than rolling back to the last checkpoint and losing hours of work. At the software level, it means smarter checkpointing strategies, asynchronous gradient update methods and advanced sharding algorithms that minimize communication overhead between nodes. Google, Meta* and other major players are already investing heavily in infrastructure where goodput is a first-class metric in the design of training systems.
There is one more aspect that often gets overlooked. Goodput forces us to think not only about how fast data is processed, but about which data is processed. Not all tokens are equally useful for training. Approaches such as curriculum learning and intelligent data selection, in which the model receives the most informative examples at the right moment in training, directly raise goodput in its broader sense: a measure of the model's real progress per unit of compute spent.
In essence, the goodput concept is an acknowledgment that the industry is maturing. When a technology is young, everyone chases the biggest numbers on paper. As it grows up, the focus shifts to real returns. For companies training the next generation of language models, the gap between throughput and goodput is the gap between burning hundreds of millions of dollars and investing sensibly in progress. And those who first learn to maximize the useful work of their clusters will gain a decisive competitive edge in the race for next-generation artificial intelligence.
*Meta is designated an extremist organization and banned in the Russian Federation.
- Nov 18, 2026Devin Bot built an Android app on its own in 44 days: 1 134 PRs from 183 messages
- Nov 18, 2026Google finds AI agents that discover vulnerabilities and hack websites on their own
- Nov 18, 2026AI agent ShakeFix fixes a pharmacy app bug in 18 minutes at GitLab hackathon
- Nov 18, 2026Nvidia's data-center revenue jumps 110% to $107.6 billion: the AI pause didn't hurt
Test Frontier LLMs & Autonomous Agents in One Workspace
Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.
The AI world, distilled — once a week
Seven stories that actually mattered, hand-picked. No noise, no reposts, no press releases.
Done! Check your inbox for a confirmation.