NVIDIA details nonuniform tensor parallelism for training LLMs on GPUs

NVIDIA has published a post on its corporate Developer Blog about nonuniform tensor parallelism, an approach designed to raise goodput, the share of compute that does useful work, when training large language models on clusters of thousands of GPUs.

What is goodput in LLM training?

Modern large language models are increasingly trained on clusters of thousands of GPUs, and training runs can last weeks or sometimes months without interruption. At that scale, failures get expensive fast. A crashed node, a stalled straggler GPU, or a restart from a checkpoint each eats into total compute time. Goodput measures more than raw GPU utilization: it captures the share of genuinely useful, productive work after accounting for these losses. Common causes of lower goodput include network congestion between nodes, uneven load across individual GPUs, and the time spent saving and restoring intermediate checkpoints after a failure.

• The post appeared on the NVIDIA Developer Blog and focuses specifically on improving goodput in LLM training
• It covers infrastructure built for jobs that span thousands of GPUs at once
• These training runs can continue for long stretches without stopping
• The proposed method is called nonuniform tensor parallelism

How does nonuniform tensor parallelism work?

Tensor parallelism is one of the basic techniques of distributed training. It splits individual matrix operations inside a model's layers into pieces and computes them simultaneously on different GPUs, which makes it possible to train models that physically don't fit in a single GPU's memory. The classic approach applies the same degree of splitting to every layer, regardless of what that layer does.

The nonuniform variant NVIDIA describes instead picks the degree of parallelism individually for different parts of the model, based on their compute load and the volume of data they must exchange between GPUs. The goal is to cut idle time and inter-device communication overhead, raising overall goodput without buying more hardware. Techniques like this are typically implemented alongside distributed training libraries such as Megatron-LM and the NeMo platform, which NVIDIA develops specifically for training large models on its GPUs.

Tensor parallelism isn't the only way to spread training across GPUs. It is used alongside pipeline parallelism, which assigns different layers of the model to different GPUs and processes them as a pipeline, and data parallelism, in which each GPU holds a full copy of the model and the training examples themselves are divided among devices. In practice, large training clusters combine all three approaches at once, and how well that combination fits a given model architecture directly determines the resulting goodput.

Why does this matter for AI infrastructure?

Even a small percentage gain in efficiency adds up to real savings when you're dealing with thousands of GPUs and weeks of continuous training. Training a truly large modern model requires compute worth tens to hundreds of millions of dollars, so even a one-percent goodput gain translates into meaningful savings in time, electricity, and money. As the largest supplier of AI accelerators, NVIDIA regularly publishes engineering breakdowns like this one so that customers — companies training their own LLMs on NVIDIA infrastructure — can squeeze more efficiency out of the clusters they already have without expanding their hardware fleet.

What this means

As LLM training scales up, success depends not only on the number and power of chips but also on systems software engineering: how efficiently distributed infrastructure uses the GPUs a company has already bought.