NVIDIA unveils SPEED-Bench, a single benchmark for speculative decoding

NVIDIA has introduced SPEED-Bench on Hugging Face, a new benchmark for speculative decoding, a technique for speeding up inference of large language models. Its goal is to measure not laboratory peak performance but how models and inference engines behave on tasks closer to real-world operation.

How SPEED-Bench is built

The authors start from a simple problem: existing tests are fragmented. Some evaluate draft-model quality on samples that are too small, others measure throughput on short prompts at batch size 1, and still others depend on a specific stack that reflects production poorly. As a result, it is hard to compare speculative decoding methods with one another: the same algorithm can look excellent on a toy dataset and noticeably worse on long contexts or under high request concurrency.

SPEED-Bench is split into two parts and complemented by a single measurement framework. The qualitative split contains 880 prompts from 18 public sources, spread across 11 categories, from coding and math to roleplay, RAG, summarization and multilingual. Each category has 80 examples, selected to reduce semantic duplication and cover as many different scenarios as possible. For selection, the authors embedded candidates with the text-embedding-3-small model and minimized the average pairwise similarity within each category.

• The qualitative split measures acceptance rate and acceptance length across domains
• The throughput split tests speed on input sequences from 1k to 32k tokens
• Each length has three difficulty levels: low-, mixed- and high-entropy
• One bucket holds 1,536 prompts, which allows stable throughput curves at batch sizes up to 512
• The framework works with TensorRT-LLM, vLLM and SGLang

The problem of fair engine comparison is handled separately. Different inference systems apply chat templates, BOS tokens and tokenization differently, so the same model can receive slightly different inputs. In SPEED-Bench, prompt preparation is moved outside the engines: they receive already pre-tokenized sequences. This reduces the influence of incidental differences and makes it possible to compare the speculative decoding algorithms themselves rather than side effects of preprocessing. The framework also records detailed telemetry on step latency, user TPS and overall output throughput.

What the tests showed

The first results show that speculative decoding depends strongly on the type of task. In low-entropy domains such as coding and math, acceptance length is higher: it is easier for the drafter to guess the next tokens. In more open-ended tasks such as roleplay and writing, the figures are lower.

In the examples from the paper, native MTP heads in Qwen3-Next reach an average acceptance length of 2.81, EAGLE3 on GPT-OSS 120B reaches 2.25, and N-Gram on Llama 3.3 70B reaches 1.41. At batch size 32, N-Gram even falls to an average slowdown of up to 0.88x instead of delivering a speedup.

Another finding concerns aggressive optimizations. The authors look separately at vocabulary pruning in EAGLE3, a technique that lowers the cost of the final projection. On coding and math its effect is almost unnoticeable, but on the long tail of user queries, especially multilingual, RAG and summarization, acceptance length drops more sharply. In other words, an optimization that looks harmless on a narrow dataset can degrade real behavior on a broader set of tasks.

The most practical observation concerns synthetic workloads. Running inference on random tokens is still popular in the industry, but for speculative decoding this mode distorts the picture. The model recognizes noise, answers with boilerplate and artificially raises acceptance length. In SPEED-Bench measurements, this inflates throughput by roughly 23% compared with realistic workloads. For teams, this is a direct warning: synthetic benchmarks can lead to the wrong choice of draft length, or even of the entire acceleration scheme.

What this means

SPEED-Bench is an attempt to bring the evaluation of speculative decoding closer to what really matters for teams serving LLMs in production: long contexts, high batch sizes, different domains and comparable conditions across engines. If the benchmark catches on, the discussion about LLM acceleration may shift from impressive numbers on synthetic tests to reproducible data on where exactly the speedup works and where it does not. For infra and research teams, that is more useful than yet another record on one convenient dataset.