RunPod Breaks Down 4 Quantization Methods: Going From FP16 to INT4 Cuts Memory 4x

RunPod, a GPU cloud provider, has published an overview of four ways to compress models before deployment: post-training quantization (PTQ), quantization-aware training (QAT), mixed precision and dynamic quantization. The piece contains no benchmarks of its own; it is a reference comparison of the trade-offs. But the topic directly shapes your inference bill. The weights of a 70B-parameter model take up about 140 GB in FP16, about 70 GB in INT8 and about 35 GB in INT4. The difference between "you need two 80 GB accelerators" and "one is enough" comes down to these numbers.

Four ways to lose a little accuracy

PTQ takes an already trained model, runs a small calibration dataset through it and converts the weights, and sometimes the activations, to INT8 or INT4. No retraining is needed, and the process takes minutes or hours. This is how the popular GPTQ and AWQ work. The price is a drop in quality, especially at 4 bits and on smaller models.

QAT takes a different route. It emulates quantization during training or fine-tuning, so the model learns to work with coarsened weights. The result is usually closer to the original accuracy, but it requires data, time and GPU hours, which is a significant expense for a large model.

Mixed precision keeps most operations in FP16 or BF16 and leaves sensitive spots, such as accumulators, normalization and softmax, at a higher bit width. Dynamic quantization stores weights as integers and computes activation ranges on the fly at runtime. It is the easiest to apply to an existing model and suits CPU inference well, but the overhead of computing the ranges eats into part of the gain.

Bit-width reference points

Speed and memory improve as bit width drops, but not uniformly. A few reference values are useful for estimates:

• FP16/BF16: 2 bytes per parameter, baseline accuracy; a 70B model needs about 140 GB for weights alone.
• INT8: 1 byte per parameter, half the memory; quality loss on most tasks is usually small.
• INT4: 0.5 bytes per parameter, about 35 GB for 70B; a noticeable risk of degradation on reasoning and code.
• Overhead: the KV-cache and activations are not included in these figures and grow with context length and the number of concurrent requests.
• Hardware support: you only get a speedup from what tensor cores can compute natively. On the H100 that means FP8 and INT8, while INT4 often needs special kernels.

Where the catch is

The main trap is that compressing weights does not equal faster inference. Inference on large models is most often bound by memory bandwidth, so smaller weights do speed up token generation. But if the runtime performs constant dequantization without optimized kernels, the gain disappears. The second problem is that average metrics such as perplexity hide failures on specific tasks: arithmetic, long context, rare languages and code generation.

This suggests a practical order of operations. For a first attempt, take INT8 PTQ, or an 8-bit format your hardware supports, and compare it with FP16 on your own data rather than on public tests. If quality holds, try INT4 and look at the worst cases, not the average. If the drop is critical and you have a budget for fine-tuning, move to QAT. Dynamic quantization fits where there is no GPU at all or you need to test a hypothesis quickly. Count the KV-cache separately: a 70 GB INT8 model on an 80 GB card leaves only about 10 GB for context and concurrent requests, and at that point INT4 wins not on speed but on free memory.