How to Shrink a Language Model 3x: A Guide to FP8, GPTQ and SmoothQuant

The open-source tool llmcompressor can compress pretrained language models to a size suitable for production use. A new practical guide shows how to apply quantization to ready-made instruction-tuned models and how to pick the best method for your use case.

What is model quantization

Quantization reduces the precision of the numbers a model works with. Instead of the usual 16-bit (FP16) or 32-bit (FP32) numbers, the model can run on 8-bit (int8) or even 4-bit values. This makes the model smaller and faster, but it may degrade the quality of its answers.

There are two approaches: quantization-aware training (QAT) and post-training quantization (PTQ). The first is more accurate but requires retraining the model on data. The second is faster: it is applied to an already finished model right before use, with no need for retraining.

llmcompressor specializes in PTQ, or post-training quantization. This lets you apply compression in a matter of hours instead of weeks of retraining. An engineer loads a ready-made model into llmcompressor, chooses a quantization method, and a few hours later has a compressed version ready to run on modest hardware.

Three quantization methods compared

The guide tests three different approaches in detail on the same base model:

• FP8 dynamic quantization — the simplest, runs in minutes and needs no calibration on additional data. All numbers (weights and activations) are converted to an 8-bit format. Downside: the lowest accuracy, with a loss of up to 5% in answer quality.
• GPTQ (W4A16) — compresses only the model weights to 4 bits, while activations stay at the original 16 bits. It requires a small calibration on a data sample. A good balance between speed and quality.
• SmoothQuant with GPTQ (W8A8) — the most accurate of the three, with weights and activations in 8 bits but with a smart redistribution of values. It is slower than the others and needs more calibration data, but results stay close to the original, with a loss of under 1%.

Each method was tested on a real task: generating text from user prompts. The authors measured file size on disk, generation speed (latency and throughput), and the model's perplexity, a metric of how much the model errs on test data.

Benchmark results

Size on disk can shrink by 3-4x. A 16-bit model takes up considerably more space than an 8-bit version of the same model. Inference speed rises in proportion to the compression, most noticeably on mobile devices, where battery life is critical.

Accuracy depends on the method chosen. FP8 loses up to 5% in answer quality, SmoothQuant less than 1%. For production scenarios where every percentage point of accuracy matters, people choose SmoothQuant, even if it is slower. FP8 suits brainstorming, drafts and auxiliary tasks, where the compute savings justify the quality loss.

The guide's practical takeaway: if you need speed and low costs, choose FP8. If accuracy is critical and you are willing to spend more time on inference, choose SmoothQuant.

Who needs this

The tool and approach are useful for companies that want to run their own language model in production:

• on the edge (on the user's device) without sending data to the cloud
• in a private cloud with limited hardware and budget
• at scale: the smaller the model, the cheaper batch processing and cloud bills

Startups and enterprises already use quantization actively. Meta* released Llama 2 with official int8 quantization support. Hugging Face released bitsandbytes, a library that simplifies quantization for engineers. Now llmcompressor lets engineers do it with precise control over the method.

What this means

Quantization is moving from experiment to standard tool in the ML pipeline. It closes the last mile: tools like llmcompressor let an engineer choose the trade-off between size, speed and quality in hours instead of weeks of experiments. For the industry as a whole, this means large language models become more accessible, cheaper to run and safer for privacy, because they can run locally without the cloud.

*Meta is designated an extremist organization and banned in Russia.