Qwen3-32B Sped Up 49% Without a New GPU: Weights Compressed from 16 to 8 Bits
A large model can be sped up without a new GPU by storing its weights in 8 bits (FP8) instead of 16 (BF16). At an NVIDIA workshop, Qwen3-32B on A100 GPUs went from 16.4 to 24.5 tokens per second. That is a 49% gain for a single user, but across the whole server with many concurrent requests, the gain is smaller: from 639 to 719 tokens per second. The measurement should therefore be repeated on your own workload, and the quality of responses should be checked.
AI-processed from Habr AI; edited by Hamidun News
If you run a large language model yourself, the pain is familiar: responses are slow, and a new GPU costs as much as a used car. At an NVIDIA workshop at GTC Berlin 2026, the company showed that part of the speed can be gained with a single configuration change. The weights of the Qwen3-32B model (its "memory," the numbers it is made of) are stored in 8 bits per number instead of 16.
These formats are called BF16 and FP8. As a result, on four NVIDIA A100 GPUs the model generated 24.5 tokens per second instead of 16.
4. A token here is roughly a syllable or a short word. For a single user, this is a 49% speedup, and no hardware had to be changed.
What exactly was done with Qwen3-32B
The workshop was held on October 20 by NVIDIA architects Amit Kushwaha and Yesika Ramirez Cardenas. Each participant had their own Kubernetes cluster with four 80 GB NVIDIA A100 GPUs. The material retells an article by engineer Siim Tolk dated November 4, 2026.
The model was launched in two variants and compared using the NVIDIA AIPerf tool. In BF16, each weight takes 16 bits; in FP8, 8 bits. The data becomes half the size and can be read from GPU memory faster. 240 requests were sent to each variant.
The results were as follows:
- average time between tokens: 61 ms in BF16 and 42 ms in FP8;
- generation speed for a single user: 16.4 versus 24.5 tokens per second, i.e. +49.4%;
- total server throughput: 639 versus 719 tokens per second, i.e. +12.5%.
The difference between the two gains is explainable. The speed of a single stream and the aggregate performance under many parallel requests depend on different things: batching, memory consumption, and the nature of the load. For a chat with one person, FP8 helps noticeably. For a server serving dozens of users simultaneously, the gain is much more modest.
What cannot be considered proven in this result
There are three caveats to keep in mind before switching everything to FP8.
First, the A100 does not support native FP8 Tensor Cores. They appeared in newer NVIDIA architectures. Therefore, the FP8 label in the configuration does not yet tell us which computations actually ran in the 8-bit format. The gain may have come mainly from savings in transferring weights from memory. The original article contains no details of the implementation.
Second, the quality of responses after quantization was not tested in the article. Speed without a quality assessment is only half a measurement. 8-bit weights usually carry over reasonably well, but on a specific task, such as code generation or working with long context, the result may decline.
Third, the 49.4% figure applies to one model, one piece of hardware, and one load test. On a different GPU or with a different model, the number may turn out quite different.
What else was covered at the workshop and what is useful
Besides quantization, participants went through two more blocks. In the first, they set up Envoy AI Gateway: one address instead of separate addresses for three models, API keys, token-based limits, retries on errors, and gradual rollout of a new model version to a small share of traffic. Token-based limits are more sensible than request-based limits: a single request with a 10,000-token context loads the server more than ten short ones.
The second block covered NVIDIA Dynamo. It splits prompt processing (prefill) and response generation (decode) across different groups of GPUs. It can also route a request to where the cache of the dialogue's beginning is already stored. This is useful for long chats and RAG. But transferring the cache between GPUs itself consumes resources, and at low load a conventional setup on the same GPUs may prove more efficient. Scaling via KEDA based on vLLM metrics, that is, the number of running requests and queue length, also makes sense only for multi-user systems.
For a home machine with a single GPU, most of this set is excessive. Quantization, FlashAttention, and inference engine settings are applicable.
Practical takeaway
Do not take someone else's 49% on faith; repeat the measurement yourself. Compare not only tokens per second but also time to first token, GPU memory usage, and response quality on your own tasks. Separately, check how speed changes with one user versus under load. For a chat for one person, FP8 can help noticeably, while for a service with a hundred clients, the gain will most likely be closer to 12.5%.
- Nov 21, 2026llama.cpp 1.0: Stable Interface, 35 Functions Removed, and a Cache for Large MoE Models
- Nov 21, 2026Gemma 4 on a mini PC writes 63 tokens per second, Qwen3.8 only 12.8: a test of 14 LLMs
- Nov 21, 2026Typing "ghbdtn" Instead of "привет" (Hello) Costs an LLM 4–7x More: 350–500 Tokens vs 70–90
- Nov 21, 2026Lab Releases Vacuum Deposition Emulator and 46-Task Benchmark for AI Agents
Key Questions Answered
What exactly was done with Qwen3-32B?
The workshop was held on October 20 by NVIDIA architects Amit Kushwaha and Yesika Ramirez Cardenas. Each participant had their own Kubernetes cluster with four 80 GB NVIDIA A100 GPUs. The material retells an article by engineer Siim Tolk dated November 4, 2026. The model was launched in two variants and compared using the NVIDIA AIPerf tool. In BF16, each weight takes 16 bits; in FP8, 8 bits. The data becomes half the size and can be read from GPU memory faster. 240 requests were sent to each variant.
What cannot be considered proven in this result?
There are three caveats to keep in mind before switching everything to FP8. First, the A100 does not support native FP8 Tensor Cores. They appeared in newer NVIDIA architectures. Therefore, the FP8 label in the configuration does not yet tell us which computations actually ran in the 8-bit format. The gain may have come mainly from savings in transferring weights from memory. The original article contains no details of the implementation.
What else was covered at the workshop and what is useful?
Besides quantization, participants went through two more blocks. In the first, they set up Envoy AI Gateway: one address instead of separate addresses for three models, API keys, token-based limits, retries on errors, and gradual rollout of a new model version to a small share of traffic. Token-based limits are more sensible than request-based limits: a single request with a 10,000-token context loads the server more than ten short ones. The second block covered NVIDIA Dynamo.
Test Frontier LLMs & Autonomous Agents in One Workspace
Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.
The AI world, distilled — once a week
Seven stories that actually mattered, hand-picked. No noise, no reposts, no press releases.
Done! Check your inbox for a confirmation.