NVIDIA speeds up Blackwell inference by up to 15x with DFlash Speculative Decoding

NVIDIA has published a detailed description of DFlash Speculative Decoding — an inference method for the Blackwell GPU architecture that accelerates language model token generation in scenarios with strict latency requirements by up to 15x compared to the standard approach.

Problem of Sequential Generation

Autoregressive language models operate on a simple principle: each next token is computed only after the previous one is ready. This fundamental limitation of transformer architecture means that the GPU spends most of its time waiting for one operation to complete before moving to the next. Computational power is used unevenly, and system throughput is bottlenecked by this sequential step. The problem is exacerbated when moving to multi-agent systems. When multiple AI agents interact sequentially — one requests another, which addresses a third — the latency of each individual inference compounds and quickly becomes the bottleneck of the entire chain. In production scenarios with thousands of simultaneous agent calls, even small latency overhead becomes a serious scaling problem.

Speculative decoding is a known technique to combat this limitation. A small draft model predicts several next tokens at once, and the main large model verifies them all in a single batch. If the draft tokens match — they are accepted without additional computation. In case of mismatch, a rollback occurs, but even accounting for recomputation, the GPU is loaded more densely than in the standard sequential scheme.

What DFlash Adds

DFlash is a specific implementation of speculative decoding optimized for the hardware characteristics of Blackwell. The key difference from other implementations: the method is built on top of Flash Attention — an algorithm already embedded in most modern LLM frameworks and requiring no separate configuration from the user.

Method characteristics:

• Specialized CUDA kernels written for Blackwell tensor cores
• Parallel verification of draft tokens as a single batch of attention operations
• Compatibility with popular inference libraries without code rewriting
• Zero quality degradation: model responses are statistically identical to baseline
• Up to 15x speedup in scenarios with long contexts and accurate draft models

Important caveat: 15x is the upper bound under optimal conditions. Actual gains depend on draft model accuracy, context length, and request patterns. For short single-turn queries or poorly matched draft models, the improvement will be more modest.

Why Blackwell is Special

The Blackwell architecture brings several hardware improvements that make DFlash particularly effective. Increased HBM3e memory bandwidth allows faster loading of weights for both models. Faster tensor cores accelerate parallel matrix operations. An improved compute scheduler reduces overhead when switching between draft and main models. When the draft model generates 4–8 tokens ahead and the main model verifies them in a single batch, the GPU workload transforms: from a narrow sequential chain it becomes a wide parallel operation for which Blackwell is optimized at the hardware level.

"As multi-agent system complexity grows, latency requirements become even stricter.

DFlash is one of the tools that allows keeping latency within reasonable bounds while scaling," explain the authors in the NVIDIA Developer blog.

What This Means

For teams building production LLM services on Blackwell cards, DFlash offers a choice without quality compromises: either significantly reduce GPU costs for the same traffic, or serve substantially more requests on existing hardware. For multi-agent pipelines, the effect is non-linear — reducing latency at the beginning of the chain yields gains at each subsequent step.