NVIDIA Speeds Up Text Generation of a 124B Model by 42.5% on a Desktop DGX Spark
A neural network slows down not only because of weak compute: for every word, it has to read a huge amount of weights from memory. NVIDIA sped up the 124-billion-parameter MoE model Ling-3.0-Flash on a desktop DGX Spark using three techniques. Generation speed on the GSM8K test rose by 42.5%, average latency fell by 24.5%, and accuracy did not change. Before buying new hardware, check your kernels, quantization, and decoding.
AI-processed from X: @crazyMLguy; edited by Hamidun News
When a neural network writes its answer slowly, the first instinct is to buy a more powerful graphics card. In a technical blog post, NVIDIA showed that often something else is enough: the same hardware can deliver noticeably more if you work on the software side. The team ran the Ling-3.0-Flash model on a desktop DGX Spark computer. It is an MoE model with 124 billion parameters. After three optimizations, text generation speed (so-called decoding throughput, meaning how many tokens per second the model outputs) rose by 42.5%. Average response latency dropped by 24.5%. The model was tested on the GSM8K benchmark of school-level math problems, and accuracy remained practically unchanged.
What Was Actually Accelerated
MoE (mixture of experts) is an architecture in which only a subset of the model's "experts" works on each token. This means fewer computations, but the weights still number in the billions and must be constantly read from memory. On compact machines like the DGX Spark, memory often becomes the bottleneck during text generation. That is why NVIDIA targeted three areas at once.
The first is optimized compute kernels, which the material calls Humming kernels. On their own, they added about 9.3% to throughput. The gain is modest, but it was achieved without changing the model itself.
The second is FP8 quantization for the output layer. At each step, this layer accesses a large table of weights to select the next word. If the table is stored in a more compact format, less data is read from memory at every step. It's like a courier carrying the same parcels in lighter boxes.
The third is speculative decoding. A standard model outputs one token per forward pass. With the speculative approach, several candidate tokens are proposed first, and then the main model verifies them in a single round. Correct candidates are accepted in a batch, so more text is produced in the same time. It works like autocomplete on a phone: instead of checking every letter, you scan an entire phrase at once.
Combined Results
Together, all three optimizations produced these results on GSM8K:
- decoding speed is 42.5% higher;
- average latency is 24.5% lower;
- accuracy is practically unchanged.
The last point matters as much as the first two. Quantization and speculative schemes usually raise the question of whether speed was bought at the cost of answer quality. According to NVIDIA, quality on GSM8K stayed at the same level. The 42.5% figure is the result of combining the techniques. It is not the sum of their individual contributions, and the 9.3% from the kernels is only part of the overall picture.
What Practitioners Should Do
For anyone running large models locally or on small servers, the takeaway is simple: before looking for more expensive hardware, check three things.
- Which kernels your inference engine uses.
- Whether the most memory-hungry parts of the model can be compressed without losing quality.
- Whether your stack supports speculative decoding for your model.
This is especially relevant for teams building private assistants, local agents, and prototypes on desktop workstations. For them, every percent of speed means lower dialogue latency and the ability to serve more requests with the same budget.
Limitations
This is a single example: one model, one type of hardware, and one benchmark. GSM8K tests arithmetic reasoning, and on long dialogues, code, or multilingual tasks, quantization and speculative decoding may behave differently. In addition, the figures come from NVIDIA's own technical blog, and there are no independent measurements yet. Before deploying these techniques in production, they should be verified on your own data and with your own quality metrics.
Still, the overall conclusion is clear: in inference, a significant share of the gains comes from engineering work on memory and decoding, not from buying a new accelerator.
- Nov 21, 2026Thinking Machines Releases Open-Weight Inkling 2 Mini with 26.4B Parameters for 32 GB Macs
- Nov 21, 2026NAND flash for SSDs drops 19–32% in spot deals — drives could get cheaper
- Nov 21, 2026Unsloth Updates Compressed Qwen3.8-27B: Accuracy Up 5–10% at the Same Size
- Nov 21, 2026AI Writes Vulnerable Code in 44% of Tests: Positive Technologies Gives Agents a Scanner
Key Questions Answered
What Was Actually Accelerated?
MoE (mixture of experts) is an architecture in which only a subset of the model's "experts" works on each token. This means fewer computations, but the weights still number in the billions and must be constantly read from memory. On compact machines like the DGX Spark, memory often becomes the bottleneck during text generation. That is why NVIDIA targeted three areas at once. The first is optimized compute kernels, which the material calls Humming kernels. On their own, they added about 9.3% to throughput.
What are the key details on Combined Results?
Together, all three optimizations produced these results on GSM8K: decoding speed is 42.5% higher; average latency is 24.5% lower; accuracy is practically unchanged. The last point matters as much as the first two. Quantization and speculative schemes usually raise the question of whether speed was bought at the cost of answer quality. According to NVIDIA, quality on GSM8K stayed at the same level. The 42.5% figure is the result of combining the techniques.
What Practitioners Should Do?
For anyone running large models locally or on small servers, the takeaway is simple: before looking for more expensive hardware, check three things. Which kernels your inference engine uses. Whether the most memory-hungry parts of the model can be compressed without losing quality. Whether your stack supports speculative decoding for your model. This is especially relevant for teams building private assistants, local agents, and prototypes on desktop workstations.
Test Frontier LLMs & Autonomous Agents in One Workspace
Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.
The AI world, distilled — once a week
Seven stories that actually mattered, hand-picked. No noise, no reposts, no press releases.
Done! Check your inbox for a confirmation.