Free Intensive Nov 21–22Free 2-day hands-on Vibecoding Intensive on Nov 21–22 at 16:00 MSK (Sat–Sun) · Live Cohort 3 starts Nov 23Join Free →Cohort 3 (Nov 23)
InfrastructureInsight · On RadarHabr AI→ original

Qwen3-32B Sped Up 49% Without a New GPU: Weights Compressed from 16 to 8 Bits

A large model can be sped up without a new GPU by storing its weights in 8 bits (FP8) instead of 16 (BF16). At an NVIDIA workshop, Qwen3-32B on A100 GPUs went from 16.4 to 24.5 tokens per second. That is a 49% gain for a single user, but across the whole server with many concurrent requests, the gain is smaller: from 639 to 719 tokens per second. The measurement should therefore be repeated on your own workload, and the quality of responses should be checked.

AI-processed from Habr AI; edited by Hamidun News
Qwen3-32B Sped Up 49% Without a New GPU: Weights Compressed from 16 to 8 Bits
Source: Habr AI. Collage: Hamidun News.
Listen to article
Article Fact Sheet & Key Figures
ParameterDetails
Primary sourceHabr AI ↗
Editor-in-Chief
Key metric #1Time between tokens fell from 61 to 42 ms across 240 requests in the test
Key metric #2The A100 has no native FP8 kernels, so the source of the speedup needs to be verified
Key metric #3The original article did not measure response quality after compression

If you run a large language model yourself, the pain is familiar: responses are slow, and a new GPU costs as much as a used car. At an NVIDIA workshop at GTC Berlin 2026, the company showed that part of the speed can be gained with a single configuration change. The weights of the Qwen3-32B model (its "memory," the numbers it is made of) are stored in 8 bits per number instead of 16.

These formats are called BF16 and FP8. As a result, on four NVIDIA A100 GPUs the model generated 24.5 tokens per second instead of 16.

4. A token here is roughly a syllable or a short word. For a single user, this is a 49% speedup, and no hardware had to be changed.

What exactly was done with Qwen3-32B

The workshop was held on October 20 by NVIDIA architects Amit Kushwaha and Yesika Ramirez Cardenas. Each participant had their own Kubernetes cluster with four 80 GB NVIDIA A100 GPUs. The material retells an article by engineer Siim Tolk dated November 4, 2026.

The model was launched in two variants and compared using the NVIDIA AIPerf tool. In BF16, each weight takes 16 bits; in FP8, 8 bits. The data becomes half the size and can be read from GPU memory faster. 240 requests were sent to each variant.

The results were as follows:

  • average time between tokens: 61 ms in BF16 and 42 ms in FP8;
  • generation speed for a single user: 16.4 versus 24.5 tokens per second, i.e. +49.4%;
  • total server throughput: 639 versus 719 tokens per second, i.e. +12.5%.

The difference between the two gains is explainable. The speed of a single stream and the aggregate performance under many parallel requests depend on different things: batching, memory consumption, and the nature of the load. For a chat with one person, FP8 helps noticeably. For a server serving dozens of users simultaneously, the gain is much more modest.

What cannot be considered proven in this result

There are three caveats to keep in mind before switching everything to FP8.

First, the A100 does not support native FP8 Tensor Cores. They appeared in newer NVIDIA architectures. Therefore, the FP8 label in the configuration does not yet tell us which computations actually ran in the 8-bit format. The gain may have come mainly from savings in transferring weights from memory. The original article contains no details of the implementation.

Second, the quality of responses after quantization was not tested in the article. Speed without a quality assessment is only half a measurement. 8-bit weights usually carry over reasonably well, but on a specific task, such as code generation or working with long context, the result may decline.

Third, the 49.4% figure applies to one model, one piece of hardware, and one load test. On a different GPU or with a different model, the number may turn out quite different.

What else was covered at the workshop and what is useful

Besides quantization, participants went through two more blocks. In the first, they set up Envoy AI Gateway: one address instead of separate addresses for three models, API keys, token-based limits, retries on errors, and gradual rollout of a new model version to a small share of traffic. Token-based limits are more sensible than request-based limits: a single request with a 10,000-token context loads the server more than ten short ones.

The second block covered NVIDIA Dynamo. It splits prompt processing (prefill) and response generation (decode) across different groups of GPUs. It can also route a request to where the cache of the dialogue's beginning is already stored. This is useful for long chats and RAG. But transferring the cache between GPUs itself consumes resources, and at low load a conventional setup on the same GPUs may prove more efficient. Scaling via KEDA based on vLLM metrics, that is, the number of running requests and queue length, also makes sense only for multi-user systems.

For a home machine with a single GPU, most of this set is excessive. Quantization, FlashAttention, and inference engine settings are applicable.

Practical takeaway

Do not take someone else's 49% on faith; repeat the measurement yourself. Compare not only tokens per second but also time to first token, GPU memory usage, and response quality on your own tasks. Separately, check how speed changes with one user versus under load. For a chat for one person, FP8 can help noticeably, while for a service with a hundred clients, the gain will most likely be closer to 12.5%.

Story

Key Questions Answered

What exactly was done with Qwen3-32B?

The workshop was held on October 20 by NVIDIA architects Amit Kushwaha and Yesika Ramirez Cardenas. Each participant had their own Kubernetes cluster with four 80 GB NVIDIA A100 GPUs. The material retells an article by engineer Siim Tolk dated November 4, 2026. The model was launched in two variants and compared using the NVIDIA AIPerf tool. In BF16, each weight takes 16 bits; in FP8, 8 bits. The data becomes half the size and can be read from GPU memory faster. 240 requests were sent to each variant.

What cannot be considered proven in this result?

There are three caveats to keep in mind before switching everything to FP8. First, the A100 does not support native FP8 Tensor Cores. They appeared in newer NVIDIA architectures. Therefore, the FP8 label in the configuration does not yet tell us which computations actually ran in the 8-bit format. The gain may have come mainly from savings in transferring weights from memory. The original article contains no details of the implementation.

What else was covered at the workshop and what is useful?

Besides quantization, participants went through two more blocks. In the first, they set up Envoy AI Gateway: one address instead of separate addresses for three models, API keys, token-based limits, retries on errors, and gradual rollout of a new model version to a small share of traffic. Token-based limits are more sensible than request-based limits: a single request with a 10,000-token context loads the server more than ten short ones. The second block covered NVIDIA Dynamo.

ZK
Zhemal Khamidun — CPO at Alpina Digital, product manager of AlpinaGPT. Building AI products since 2019. Hamidun News is a pet project that grew into a production aggregator with 6,000+ readers.
✦ Unified AI Workspace · Nomad

Test Frontier LLMs & Autonomous Agents in One Workspace

Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.

Try Nomad AI →Launch Nomad Agent
What do you think?
Loading comments…