NVIDIA described nonuniform tensor parallelism for LLM training on GPUs

NVIDIA published an article in its corporate Developer Blog about nonuniform tensor parallelism — an approach designed to increase goodput, that is, the share of useful computational work, when training large language models on clusters of thousands of GPU.

What Is Goodput in LLM Training

Training modern large language models is increasingly performed on clusters of thousands of graphics processors and stretches over weeks, and sometimes even months of continuous operation. With such duration, the cost of failures rises sharply: the failure of a single node, hanging of a GPU "straggler" or the need to restart from a checkpoint consume part of the total computational time. Goodput is a metric that reflects not just the total GPU load, but the fraction of genuinely useful, productive work adjusted for such losses. Among typical causes of its reduction are network congestion between nodes, uneven load on individual GPUs, and time spent on saving and recovering intermediate checkpoints after a failure.

• The material was published in the NVIDIA Developer Blog and is devoted specifically to increasing goodput in LLM training
• This concerns infrastructure designed for tasks spanning thousands of GPU simultaneously
• Such training runs can last a long time without stopping
• The proposed method is called nonuniform tensor parallelism — "nonuniform tensor parallelism"

How Nonuniform Tensor Parallelism Works

Tensor parallelism is one of the basic methods of distributed training: individual matrix operations within model layers are broken into pieces and calculated simultaneously on different GPUs, allowing you to train models that physically do not fit in the memory of one graphics card. The classical approach applies the same degree of partitioning to all model layers indiscriminately. The nonuniform variant described by NVIDIA selects the degree of parallelism individually for different parts of the model — taking into account their computational load and the volume of data that needs to be exchanged between GPUs.

The goal is to reduce idle time and communication overhead between devices, increasing the overall goodput without purchasing additional hardware. Such techniques are usually implemented in conjunction with distributed training libraries like Megatron-LM and the NeMo platform, which NVIDIA develops specifically for training large models on its GPUs.

Tensor parallelism is not the only way to distribute training between GPUs: alongside it, pipeline parallelism is used, where different layers of the model are attached to different GPUs and processed in a pipeline fashion, and data parallelism, where each GPU stores a full copy of the model, and only the training examples themselves are shared between devices. In practice, large training clusters combine all three approaches simultaneously, and how well this combination is selected for a particular model architecture directly determines the overall goodput.

Why This Matters for AI Infrastructure

Even a small percentage gain in efficiency translates into significant savings when counting thousands of GPUs and weeks of continuous training. Training a truly large modern model requires computational resources costing tens and hundreds of millions of dollars, so even a one-percent gain in goodput translates into meaningful savings in time, electricity, and money. NVIDIA, as the largest supplier of accelerators for AI, regularly publishes such engineering breakdowns so that its clients — companies training their own LLMs on NVIDIA infrastructure — can squeeze more efficiency from already existing clusters without expanding their equipment fleet.

What This Means

As the scale of large language model training grows, not only the quantity and power of chips matter, but also systems software engineering — how efficiently the distributed infrastructure uses the already purchased GPUs.