AI training efficiency: why speed isn't everything

When it comes to training large language models, the conversation inevitably centers on two things: how many GPUs are involved and how fast the system processes data. Tokens per second have become a kind of currency in the industry — the more, the better. But what if this metric, for all its clarity, tells only half the story? This is precisely the question raised by the growing concept of goodput, which promises to transform the very approach to assessing AI training efficiency.

Pre-training a modern model at the scale of hundreds of billions of parameters and beyond is an engineering marathon stretched over weeks and months. Thousands of accelerators work in parallel, processing colossal masses of text data. Traditionally, the success of this process was measured by two metrics. The first is throughput — the bandwidth: how many tokens the system can process per unit of time. The second is training progress: how much the model actually improves with each iteration. The problem is that these two metrics don't always correlate with each other as engineers would like.

Throughput is deceptively simple. It shows how fast data flows through the computing pipeline, but says nothing about the quality of that work. Imagine a factory conveyor belt that stamps out parts at record speed, but half of them are defective. Formally, productivity is high; actual output is something else entirely. In the context of AI training, the analogy works surprisingly well. The system can demonstrate impressive throughput figures, yet a significant portion of computations are wasted — on reprocessing data after failures, on downtime due to synchronization between nodes, on suboptimal load distribution across accelerators. All this time the token counter keeps spinning, creating an illusion of progress.

This is where goodput enters the stage — a metric that attempts to measure not raw throughput, but useful work. Goodput accounts only for those computations that truly bring the model closer to completion of training. If a cluster of four thousand GPUs processes a trillion tokens a day, but twenty percent of this work is lost due to hardware failures, checkpoint restarts, and communication overhead between nodes, then the real goodput is only eight hundred billion tokens. The difference seems academic until you translate it to dollars: at the cost of renting a large GPU cluster in the millions of dollars per day, a twenty percent loss is hundreds of millions per training cycle.

The transition from throughput to goodput as a key metric reflects a deeper shift in the industry. The era when AI progress was defined exclusively by scale — more data, more parameters, more computing — is gradually giving way to an era of optimization. Companies are realizing that scaling clusters indefinitely is impossible either economically or energetically. By various estimates, training a single frontier model already costs hundreds of millions of dollars, and the next generation could cross the billion-dollar mark. Under such conditions, every percent of real efficiency is of immense importance. Optimizing goodput becomes not a theoretical exercise, but a direct tool for reducing costs.

The practical consequences of this approach affect the entire chain — from data center design to the architecture of software training frameworks. At the hardware level, this means heightened attention to fault tolerance: if one of thousands of accelerators fails, the system should redistribute the load without losing progress, rather than rolling back to the last checkpoint and losing hours of work. At the software level — this means smarter checkpointing strategies, asynchronous gradient update methods, and advanced sharding algorithms that minimize communication overhead between nodes. Google, Meta, and other major players are already actively investing in infrastructure where goodput is a first-class metric in designing training systems.

There is yet another aspect often overlooked. Goodput forces us to think not only about how fast data is processed, but also about what data is processed. Not all tokens are equally useful for training. Approaches like curriculum learning and intelligent data selection, where the model receives the most informative examples at the right point in training, directly increase goodput in its broader sense — as a metric of real model progress per unit of computation spent.

The concept of goodput is essentially an acknowledgment of industry maturity. When technology is young, everyone chases maximum numbers on paper. As it matures, the focus shifts to real returns. For companies training the next generation of language models, the difference between throughput and goodput is the difference between burning hundreds of millions of dollars and wisely investing in progress. And those who first learn to maximize useful work from their clusters will gain a decisive competitive advantage in the race for next-generation artificial intelligence.