How to Compress an LLM to 8 Bits End to End: SmoothQuant and FP8 Against Outliers
To make an LLM run faster, its numbers are compressed to 8 bits. Weights can be quantized once, but activations, the intermediate values computed for each request, shift constantly and contain outliers. SmoothQuant divides the noisy activation channels by a coefficient and multiplies the matching weights by the same coefficient, so the layer's output stays unchanged. W8A8 uses 8 bits for both weights and activations (INT8 or FP8). However, it is not always faster than W4A16 or BF16, so measure on your own workload.
AI-processed from Machine Learning Mastery; edited by Hamidun News
To make an LLM run faster and take up less memory, the numbers inside it are stored in 8 bits instead of heavier formats such as BF16 or FP32. With model weights this is simple: they are fixed after training, so they can be compressed once before deployment. Activations are harder.
These are the intermediate values the model computes anew for each request. They depend on the input token, its position and the preceding layers, and certain channels constantly contain abnormally large numbers, or outliers. W8A8 mode (8 bits for both weights and activations) runs straight into this problem.
In a new Machine Learning Mastery piece, Adrian Tam shows how it is worked around using SmoothQuant, INT8 and FP8, and what can already be run in vLLM.
Why activations are harder than weights
Imagine a ruler that must mark both a child's height and the height of a skyscraper. If you stretch the scale to fit the skyscraper, you cannot measure the child accurately: all the values land within a couple of marks. If you clip the scale, the skyscraper hits the ceiling and its information is lost. Outliers in activations work the same way: a wide range makes the quantization step too coarse for ordinary values, while clipping improves accuracy for ordinary values but destroys the outlier.
The scale can be set in two ways. Static scaling is chosen in advance from calibration data: it is fast, but inputs outside the calibration range may be clipped. Dynamic scaling is measured during the request itself: it adapts to each input, but adds a maximum search and scale calculation to the inference path. A common approach for transformers is per-token dynamic scaling: one scale for all hidden features of a token. The example in the article uses symmetric INT8, where the zero point is always zero, which keeps matrix multiplication arithmetic simpler.
SmoothQuant: shifting the difficulty onto weights
SmoothQuant removes outliers with a clever substitution. Activation channels with large values are divided by chosen coefficients, and the corresponding weight columns are multiplied by the same coefficients. Before quantization, the layer's output is mathematically unchanged, but the activation range is evened out. Weights are static and usually tolerate an uneven range better when quantized per channel.
The author notes a distinction from AWQ, covered in an earlier part of the series. AWQ uses calibration activations to protect important weights in a format such as W4A16, while activations stay in a more precise format during inference. SmoothQuant prepares both sides for INT8: it modifies the stored weights and makes on-the-fly activation quantization easier.
The practical route is the LLM Compressor library from the vLLM project. In the example on TinyLlama-1.1B, 128 calibration samples from the ultrachat_200k dataset, up to 512 tokens long, are used. A SmoothQuantModifier with a smoothing strength of 0.8 and a GPTQModifier for the W8A8 scheme are then applied, and the lm_head layer is left untouched. The checkpoint stores per-channel INT8 weights and instructions on how to quantize activations during inference, but the activations themselves are not saved. vLLM loads such a checkpoint without a separate quantization argument.
FP8: not just bit truncation
FP8 is also quantization with a scale. The tensor is divided by a scale and rounded to the nearest value the FP8 format can represent. The author stresses that this is not simply cutting bits off an FP32 value. This matters especially for Blackwell GPU owners: vLLM's current INT8 kernels target NVIDIA GPUs from Turing to Hopper, so on Blackwell you should choose a supported FP8 path.
What to consider in practice
- A short generation on TinyLlama only verifies that the checkpoint loads and runs. A small model's answers may be inaccurate, so quantization quality must be compared against the original model on representative prompts.
- The checkpoint format, model architecture, vLLM version and GPU must all support the same W8A8 scheme.
- W8A8 does not guarantee a win over W4A16 or BF16. INT8 matrix hardware performs well during prefill and with large batches, while W4A16 reads fewer weight bytes and can be more advantageous for decoding with small batches.
The takeaway for anyone calculating inference costs: there is no single best scheme. First define your workload profile: long prompts or short answers, large or small batches. Then build two or three checkpoint variants and benchmark prefill and decode separately, and do not forget to evaluate quality. One caveat: the walkthrough uses a tiny model, so real speed and accuracy figures for large LLMs will need to be verified independently.
- Nov 21, 2026A local phi3 was jailbroken in 51% of attempts — the developer put a security gateway in front of it
- Nov 21, 2026Thinking Machines Releases Open-Weight Inkling 2 Mini with 26.4B Parameters for 32 GB Macs
- Nov 21, 2026NAND flash for SSDs drops 19–32% in spot deals — drives could get cheaper
- Nov 21, 2026Unsloth Updates Compressed Qwen3.8-27B: Accuracy Up 5–10% at the Same Size
Key Questions Answered
Why activations are harder than weights?
Imagine a ruler that must mark both a child's height and the height of a skyscraper. If you stretch the scale to fit the skyscraper, you cannot measure the child accurately: all the values land within a couple of marks. If you clip the scale, the skyscraper hits the ceiling and its information is lost. Outliers in activations work the same way: a wide range makes the quantization step too coarse for ordinary values, while clipping improves accuracy for ordinary values but destroys the outlier.
What are the key details on SmoothQuant: shifting the difficulty onto weights?
SmoothQuant removes outliers with a clever substitution. Activation channels with large values are divided by chosen coefficients, and the corresponding weight columns are multiplied by the same coefficients. Before quantization, the layer's output is mathematically unchanged, but the activation range is evened out. Weights are static and usually tolerate an uneven range better when quantized per channel. The author notes a distinction from AWQ, covered in an earlier part of the series.
What are the key details on FP8: not just bit truncation?
FP8 is also quantization with a scale. The tensor is divided by a scale and rounded to the nearest value the FP8 format can represent. The author stresses that this is not simply cutting bits off an FP32 value. This matters especially for Blackwell GPU owners: vLLM's current INT8 kernels target NVIDIA GPUs from Turing to Hopper, so on Blackwell you should choose a supported FP8 path.
Test Frontier LLMs & Autonomous Agents in One Workspace
Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.
The AI world, distilled — once a week
Seven stories that actually mattered, hand-picked. No noise, no reposts, no press releases.
Done! Check your inbox for a confirmation.