NVIDIA develops a method for training neural networks in 4-bit precision

NVIDIA has unveiled NVFP4, a new methodology for training neural networks in 4-bit precision. It saves substantial memory and compute when training large models.

How it works

The standard approach uses 8-bit (FP8) or 16-bit (BF16) precision to store intermediate results and training gradients. NVIDIA managed to halve these memory requirements by moving to the 4-bit NVFP4 format.

The method does not simply cut precision. It combines several techniques: selective use of the more precise BF16 on critical layers of the model, special mathematical transforms of the gradient input data (16×16 random Hadamard transforms), and stochastic rounding during computation.

Traditionally, 4-bit training was considered risky: over long training runs, rounding errors accumulate and degrade the model. The company tested NVFP4 on a hybrid Mamba-Transformer model with 12 billion parameters, training it on 10 trillion tokens, the longest public experiment with 4-bit training. This shows that, with the right methodology, numerical errors do not accumulate catastrophically.

Results exceeded expectations

The key metric was accuracy on the MMLU-Pro benchmark, a broad knowledge test covering mathematics, natural sciences, the humanities and other areas. The NVFP4 model scored 62.58%, just 0.04% below the model trained with the traditional FP8 method (62.62%). For practical applications this difference is wholly insignificant: the accuracy is within measurement error.

Against the backdrop of a twofold memory saving, this is a rare case where lowering the precision of number representation did not noticeably reduce the quality of results. This means NVFP4 does not sacrifice correctness to save resources.

• Memory reduction: 2x compared with FP8
• Benchmark accuracy loss: less than 0.1%
• Experiment scale: 10 trillion tokens
• Architecture: hybrid Mamba-Transformer model with 12 billion parameters

What this means for the industry

The result matters for companies that train models from scratch. A 2x memory saving means the same amount of computation can run faster and cheaper, or the saved resources can go toward training larger models. If your company trains a model for 1000 A100 GPU-days, NVFP4 could cut that to 500 GPU-days while preserving quality.

For researchers, this opens new possibilities for experimenting with architectures, data volumes and hyperparameters. It will become easier to test new ideas on larger models within a single day than on smaller models over a week.

However, the method still needs further validation on other types of models, especially pure transformers and models with a different structure. So far NVIDIA has shown results only on the hybrid Mamba-Transformer architecture.

It is also important to understand that 4-bit training is a specialized technique that requires dedicated software optimizations and hardware support (for now, full support exists only on NVIDIA GPUs).