Quantization Makes INT8 LLM Weights Half the Size of FP16: A TorchAO Guide
To generate each token, an LLM rereads all of its weights from memory, and at low load that takes longer than the computation itself. Quantization stores the weights as 8-bit integers (INT8) instead of 16-bit floating-point numbers (FP16). That cuts the storage in half, and TorchAO, the library in PyTorch, does it in a single function call without retraining. The speedup depends on the GPU and its kernels, so after compressing, measure the memory, speed, and quality of the whole model.
AI-processed from Machine Learning Mastery; edited by Hamidun News
Imagine that for every word a chatbot writes, it has to flip through its entire encyclopedia again. That is roughly how an LLM runs on a GPU: for each new token, the transformer reads all of its weights from memory. With small batches (when few requests are being processed), moving the weights takes longer than the math itself.
Quantization targets this problem by storing the weights in fewer bits. For example, an 8-bit integer (INT8) takes half the space of a 16-bit floating-point number (FP16 or BF16). In addition, many AI accelerators compute at low precision faster than at FP16.
Adrian Tam of Machine Learning Mastery worked through the process in practice, from the formula to ready-made checkpoints, including through TorchAO, the quantization library inside PyTorch.
How Weights Become Integers
The easiest way to think about it is as a ruler. The range of weight values is divided into a finite number of steps, and each weight is recorded as the number of its nearest step. Two parameters are needed for this: the scale (the distance between adjacent steps) and the zero point (the step that corresponds to a true zero). To use the value, the step number is converted back into an approximate number. The difference between the original and the reconstructed value is the rounding error.
The author warns of a trap. A teaching example in PyTorch stores values as uint8, and each one occupies a full byte, even if you asked for 4 bits. To get real savings, a production format has to pack values into less than a byte. Another detail: a single scale for the entire tensor degrades quality. For LLMs, per-channel or per-group schemes are usually better, where each row of the matrix or each block of weights has its own range.
Weight-only mode means that only the weights are compressed, while the layer's inputs (activations) stay in FP16 or BF16. An efficient kernel uses the integer weights directly and does not reassemble a full copy of the matrix.
TorchAO, GPTQ, and AWQ: Three Ways to Get a Compressed Model
The shortest path is TorchAO. The quantize_() function replaces the weights of suitable Linear layers in an already trained model in place. No fine-tuning is required, which is why the method is called post-training quantization. In the example, the author requests asymmetric per-channel INT8 with BF16 activations. Through the transformers library, the configuration can be applied right when the model loads, for example INT4 with group size 128. The author also notes separately that explicitly selecting the HQQ algorithm and tile packing let him bypass the optional dependency on the MSLK kernel in TorchAO 0.17 on a tested RTX 4090. On other versions and cards, the available layouts may differ.
Simple rounding looks at each weight in isolation and does not account for how much a given input channel affects the output. GPTQ and AWQ use calibration examples, meaning a small set of typical texts. GPTQ compresses layer by layer and adjusts the not-yet-compressed weights along the way, compensating for accumulated error. In transformers, it now runs on the GPT-QModel backend, and in the example calibration runs on the c4 dataset, with the parameter sym=False enabling the asymmetric scheme. AWQ finds the weights that matter most for typical activations and computes scaling factors that protect them before compression.
There are pitfalls too. Asymmetric GPTQ checkpoints work with fewer kernels: the Marlin backend does not support them. The reference AWQ implementation uses 128 samples of 512 tokens each from pile-val-backup for calibration, and you cannot choose another dataset through its command line. A raw .pt file after quantization is not a complete Hugging Face checkpoint: it lacks the model configuration and tokenizer, which must be packaged separately.
What to Do in Practice
- Choose the scheme and backend together. A mathematically correct configuration is not always the fastest, and the runtime must support the exact packing format.
- Don't estimate savings from the formula alone. INT8 gives half the raw storage of FP16 or BF16, but the outcome depends on packing, groups, and scales.
- Measure the whole model. Speed depends on hardware, kernels, and matrix shapes. Look at memory, latency, and quality on your own tasks, not on a single layer.
- Check calibration. GPTQ and AWQ results depend on how closely the examples resemble real traffic. For code, legal text, or another language, a generic dataset may not be a good fit.
The material also covers converting to the GGUF format for llama.cpp (which requires a built copy of the repository) and evaluating the effects of quantization. The original source in our version breaks off midway through the GPTQ section, so it contains no figures for final speed or quality loss. For teams that track inference costs, this is a useful cheat sheet. A sensible order is: take the TorchAO variant without calibration as a baseline, measure memory, speed, and quality, and then try GPTQ or AWQ where you need stronger compression.
- Nov 21, 2026Eight Open-Source Tools for AI Agents: Browser Use Leads with 121,000 Stars
- Nov 21, 2026Thinking Machines Releases Open-Weight Inkling 2 Mini with 26.4B Parameters for 32 GB Macs
- Nov 21, 2026AURA 2.0 Video Model Starts Talking on Its Own: Response Audio Arrives in 344 ms
- Nov 21, 2026Anthropic raised a record $81.3B in its IPO, but the founders kept control
Key Questions Answered
How Weights Become Integers?
The easiest way to think about it is as a ruler. The range of weight values is divided into a finite number of steps, and each weight is recorded as the number of its nearest step. Two parameters are needed for this: the scale (the distance between adjacent steps) and the zero point (the step that corresponds to a true zero). To use the value, the step number is converted back into an approximate number. The difference between the original and the reconstructed value is the rounding error.
What are the key details on TorchAO, GPTQ, and AWQ: Three Ways to Get a Compressed Model?
The shortest path is TorchAO. The quantize_() function replaces the weights of suitable Linear layers in an already trained model in place. No fine-tuning is required, which is why the method is called post-training quantization. In the example, the author requests asymmetric per-channel INT8 with BF16 activations. Through the transformers library, the configuration can be applied right when the model loads, for example INT4 with group size 128.
What to Do in Practice?
Choose the scheme and backend together. A mathematically correct configuration is not always the fastest, and the runtime must support the exact packing format. Don't estimate savings from the formula alone. INT8 gives half the raw storage of FP16 or BF16, but the outcome depends on packing, groups, and scales. Measure the whole model. Speed depends on hardware, kernels, and matrix shapes. Look at memory, latency, and quality on your own tasks, not on a single layer. Check calibration.
Test Frontier LLMs & Autonomous Agents in One Workspace
Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.
The AI world, distilled — once a week
Seven stories that actually mattered, hand-picked. No noise, no reposts, no press releases.
Done! Check your inbox for a confirmation.