NVIDIA explains how to train transformers at lower precision without losing quality

NVIDIA has published an in-depth guide on its Developer Blog to optimizing transformer architectures for training at reduced numerical precision, using FP8 and BF16. The article targets engineers who want to lower the cost of training runs without losing model quality.

Why teams need this

Transformers underpin most modern language and generative models. As models grow from billions to tens of billions of parameters, the cost of a single training run rises exponentially. Each experiment iteration takes more GPU-hours, which slows development and makes it more expensive.

In practice, slow training is not just a technical inconvenience. It limits how many hypotheses a team can test in a quarter and how large a model it can afford. NVIDIA describes speeding up transformers not as an optimization but as a condition of staying competitive.

What low-precision training is

Standard neural network training uses a 32-bit format (FP32), which offers high numerical precision but consumes a lot of memory and runs slower on modern GPUs. Reducing bit width lets you fit more data into video memory and speeds up matrix operations:

• FP16: 16-bit floating-point numbers; supported by most modern GPUs
• BF16: Brain Float 16; a wider dynamic range, better suited to unstable training of large models
• FP8: an 8-bit format available on the Hopper architecture (H100, H200); delivers twice the matrix-operation throughput of BF16
• INT8: 8-bit integer; used more often for inference than for training

The main challenge is preserving numerical stability at such low precision. A naive switch from FP32 to FP8 leads to diverging gradients and unstable training.

Techniques NVIDIA recommends

Simply swapping the format does not work, so NVIDIA describes several proven approaches.

Mixed precision. Weights are stored in FP32, while the forward and backward passes run in FP16 or BF16. This combines the speed of low-precision computation with the reliability of full-precision parameter storage, and it is the de facto standard for most modern training pipelines.

Loss scaling. FP16 represents very small numbers poorly, so gradients in later layers can underflow to zero. Loss scaling artificially increases the loss value before the backward pass, then scales the gradients back down. Modern implementations do this automatically and adaptively.

TransformerEngine. NVIDIA's specialized library automatically manages precision at the level of individual transformer layers. It supports FP8 on Hopper and integrates with PyTorch, JAX and Megatron-LM. Instead of rewriting the whole training code, an engineer plugs in TransformerEngine and gets FP8 acceleration with minimal changes.

"As models grow, training runs consume ever more GPU-hours and engineering time.

This directly affects how quickly teams can experiment and how large a model they can afford," — NVIDIA Developer Blog.

What this means

The guide arrives at a time when training efficiency has become as important as model accuracy. Teams on H100 or H200 get concrete instructions: FP8 via TransformerEngine is one of the most accessible ways to trim a GPU budget without reworking the architecture. For small labs, this may be the difference between training a 70-billion-parameter model and abandoning it because of cost.