16-bit formats halve model weight memory without changing the architecture

For the same 7-billion-parameter model, weights may occupy roughly 28 GB in FP32, 14 GB in FP16 or BF16, and 7 GB at eight bits. The architecture does not change—the way numbers are stored does. The benefit seems obvious: less memory and less data moving between memory and compute units, so the model should respond faster.

But this is exactly where confusion usually begins. Fewer bits do not guarantee a speedup. If the processor or GPU cannot efficiently execute operations in the chosen format, the runtime may add conversions, switch to a slower kernel, or fail with an error. An overly narrow value range can also turn a large activation into infinity or erase small differences between numbers.

FP16 and BF16 solve different problems

A floating-point number stores a sign, an exponent, and a significand. The exponent determines the range of magnitudes, while the significand determines the ability to distinguish nearby values. Reducing the bit count forces a choice about what to sacrifice.

FP16 allocates more space to the significand, so it distinguishes neighboring numbers more precisely. Its range is relatively narrow, however: large activation values can cause overflow. BF16 stores as many exponent bits as FP32, so it supports much larger and smaller magnitudes. The cost is lower precision for nearby values.

Imagine a layer that receives the values 1.001 and 1.002, then later encounters an activation of 100,000. FP16 will better preserve the difference between the first two numbers but may struggle with the large value. BF16 will handle the outlier more easily, but it may round the nearby numbers to the same value. Which option is better depends on the model, data, and hardware—not on the format's name.

There is another detail: the type used to store the weights does not reveal the whole picture. A matrix multiplication may read FP16 inputs, accumulate the sum in FP32, and write the result back in FP16. Performance reports should therefore specify the precision of inputs, accumulation, and outputs.

Mixed precision protects sensitive operations

A model does not have to be converted entirely to one format. Matrix multiplications can run in FP16 or BF16, while normalization, softmax, reductions, and selected layers remain in FP32. This approach preserves acceleration for the heaviest operations while reducing the risk of numerical errors.

Manual type switching is not free, however. Unnecessary conversions consume time, and mismatched input and weight types can sometimes break execution. For a pretrained model, it is more sensible to start with the mixed-precision mode supported by its library, inspect the profiler, and only then change individual layers.

A large checkpoint is best loaded directly in the target format. If the FP32 version is placed in memory first and only then converted to FP16, memory will briefly be required for both copies. With a large model, that can be enough to trigger an out-of-memory error before the first request.

What to measure before launch

Formats should be compared across three dimensions. First, memory: weight size, peak consumption, KV cache, and temporary buffers. Second, speed: prompt-processing latency, token-generation latency, and overall throughput at realistic batch sizes, measured separately. Third, quality: perplexity, application-specific metrics, and responses to representative prompts.

A single matching text output is not enough. With greedy token selection, responses can look identical even when the probability distributions have already diverged noticeably. To find problems, compare logits, especially around decisions where two tokens receive nearly equal scores.

The practical conclusion is simple: start with FP32 as a numerical baseline, test FP16 and BF16 on the target hardware, warm up the model, and synchronize the device before taking measurements. Then choose the most compact format that preserves the required quality and actually runs faster on your hardware. Optimal precision is not the fewest possible bits, but a measured balance of memory, speed, and reliability.