Four 2017 V100 GPUs run a 27B-parameter model at 218 tokens/s
Minsk service ZuBrIQ has put Qwen3.8-27B (27 billion parameters) into production on four Tesla V100 cards from 2017, even though official vLLM does not start 4-bit models on them. With the 1Cat-vLLM fork, the quantized model weighs about 21 GB, reads up to 262,000 tokens of context (hundreds of pages of text), and generates about 218 tokens per second across 24 simultaneous requests. Old cards are viable for production, provided you calculate memory by hand.
AI-processed from Habr AI; edited by Hamidun News
Many teams don't have fresh H100s, but their racks hold graphics cards that are already written off. Engineers at the Belarusian service ZuBrIQ showed that four Nvidia Tesla V100s (2017 server cards with 16 GB of memory each) can still run the dense model Qwen3.8-27B. The "27B" designation means 27 billion parameters. Its context window is 262K, meaning about 262,000 tokens: hundreds of pages of text in a single request. Generation speed is about 218 tokens per second in total across 24 concurrent requests (a token is roughly a word or part of a word). The team's write-up is practical, covering launch commands, figures, and pitfalls that are easy to fall into.
Why V100s at all
The zubr model is the only one at ZuBrIQ whose requests never leave Belarus. Prompts, documents, and responses stay on servers in Minsk. For banks, the public sector, and companies handling personal data, this is a requirement without which they can't operate. New cards are difficult and expensive to bring into the country, so the team is squeezing the maximum out of what they have: four V100 SXM2 cards with 16 GB each, 64 GB in total.
The difficulty lies in the Volta architecture (SM70). It has no bfloat16, the format most modern models are distributed in, and no FP8, and the official vLLM refuses to run 4-bit models on cards with an index below 75. The workaround is the 1Cat-vLLM fork under Apache-2.0, which includes 4-bit kernels for Volta, an attention backend for V100, and CUDA Graph support. It runs natively via systemd inside an LXC container on Proxmox, without Docker.
The model was taken in INT4 (weights stored in 4 bits instead of 16): Qwen3.8-27B from cyankiwi weighs about 21 GB, roughly 5.3 GB per card. There are two caveats. The quantization is stored in the compressed-tensors format, which requires setting the environment variable VLLM_SM70_COMPRESSED_TENSORS_TURBOMIND=1. In addition, the --dtype float16 flag is mandatory, because Volta has no bfloat16.
Where the memory goes
The first trap: in the fork, the model weights are barely subtracted from the KV-cache budget (the memory where the model stores the context it has already read). Because of this, with a setting of 0.50, the process occupied nearly 14 GB per card instead of 8.
The team set the cache size explicitly via --kv-cache-memory-bytes. The calculation is straightforward: the engine reported that a 131,072-token window needs 2.08 GiB, which works out to about 17 KB per token.
For 262,144 tokens, 4.6 GB is enough, with a small margin. The calculation has to be based on the most loaded card: if an embedding service also lives on the same GPU, the cache is limited by that GPU's free memory.
The second trap: CUDA graphs are captured for each batch size and sit outside the budget. At max-num-seqs 96, they took about 5 GB beyond the plan. After lowering the value to 24, throughput barely dropped, and the memory was recovered.
The third: two instances on the same cards. A separate Qwen3-VL-4B for vision was running alongside, and when both processes restarted simultaneously, the initialization peak exceeded 16 GB. One worker hit an OOM, and the main one hung on the NCCL watchdog: systemd showed active, but the port was closed. In September the problem went away: Qwen3.8-27B reads images itself and handled a table test correctly in 3 out of 4 cases, while the separate 4B managed 0 out of 3. The 4B was shut down, freeing about 8 GB, and the window was raised from 131,072 to 262,144 tokens.
MoE versus dense, and the naming rule
Until September, production ran the MoE model Qwen3.6-35B-A3B: 35 billion parameters, of which about 3 billion are active for each token. Both models were benchmarked with the same script: 300 requests, 24 clients, 512 input tokens and 256 output tokens. The dense 27B (dense means all parameters are always active) produced 218.5 tokens per second against 200.2 (9% more), and each token printed 14% faster. The authors suspect the MoE kernels on SM70 are simply poorly optimized.
The price was paid in the first token: the median wait was 1.31 s versus 0.59 s, and for the slowest 1% of requests (P99) it was 14.2 s versus 2.3 s. The dense model engages all 27 billion parameters when processing the prompt, while the MoE engaged about 3 billion. The team's caveat: they compared speed, not quality, which was checked only with smoke-test requests.
A separate lesson concerns names. When the old name qwen3.6-35b was removed during the weight swap, the support chatbot, which called vLLM directly, got a 404 and began sending every customer to a human operator. Now the first name in --served-model-name (zubr) never changes, and versioned names such as zubr-2 describe specific weights. The truth about the weights is shown by the root field in /v1/models, because vLLM returns the requested alias in its response.
Takeaways
- On older cards without bfloat16 and FP8, a 4-bit quant plus a vLLM fork is a working path, but it requires float16 and the correct environment variables.
- Set the KV cache explicitly and account for memory used by CUDA graphs.
- Don't run two heavy processes on the same GPUs at once.
- Separate the public model name from the weight version.
The limitations are also clear: the results come from a single configuration, model quality was not compared, and the fork is not an official vLLM branch, so the team will have to maintain support on its own.
- Nov 21, 2026Qwen3-32B Sped Up 49% Without a New GPU: Weights Compressed from 16 to 8 Bits
- Nov 21, 2026Letter Urging Huang's Return to Nvidia Gathers 175,000 Signatures, 8,766 from 'Employees'
- Nov 21, 2026Nvidia Loses More Than $1 Trillion in Market Value After Jensen Huang's Exit
- Nov 21, 2026Nvidia Board Ousts Jensen Huang, Putting $12.9B Hugging Face Deal at Risk
Key Questions Answered
Why V100s at all?
The zubr model is the only one at ZuBrIQ whose requests never leave Belarus. Prompts, documents, and responses stay on servers in Minsk. For banks, the public sector, and companies handling personal data, this is a requirement without which they can't operate. New cards are difficult and expensive to bring into the country, so the team is squeezing the maximum out of what they have: four V100 SXM2 cards with 16 GB each, 64 GB in total. The difficulty lies in the Volta architecture (SM70).
Where the memory goes?
The first trap: in the fork, the model weights are barely subtracted from the KV-cache budget (the memory where the model stores the context it has already read). Because of this, with a setting of 0.50, the process occupied nearly 14 GB per card instead of 8. The team set the cache size explicitly via --kv-cache-memory-bytes. The calculation is straightforward: the engine reported that a 131,072-token window needs 2.08 GiB, which works out to about 17 KB per token.
What are the key details on MoE versus dense, and the naming rule?
Until September, production ran the MoE model Qwen3.6-35B-A3B: 35 billion parameters, of which about 3 billion are active for each token. Both models were benchmarked with the same script: 300 requests, 24 clients, 512 input tokens and 256 output tokens. The dense 27B (dense means all parameters are always active) produced 218.5 tokens per second against 200.2 (9% more), and each token printed 14% faster. The authors suspect the MoE kernels on SM70 are simply poorly optimized.
Test Frontier LLMs & Autonomous Agents in One Workspace
Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.
The AI world, distilled — once a week
Seven stories that actually mattered, hand-picked. No noise, no reposts, no press releases.
Done! Check your inbox for a confirmation.