Parallel speculative decoding speeds up SGLang by up to 4 times and goes open source

If you run LLMs in production, you have almost certainly heard of speculative decoding. The idea is simple: a small draft model quickly predicts several tokens ahead, and a larger model checks them in a single pass. If the predictions are correct, you get several tokens for the cost of one step, with no change in response quality. That is why the method has become almost standard in production inference stacks. Now a new study takes it further: parallel drafting that preserves causal dependencies. It reports up to 4 times higher throughput in SGLang, and the code has been released publicly.

The problem with EAGLE-3

EAGLE-3 is considered one of the strongest approaches to speculative decoding. Its drafting is autoregressive: to predict the fifth token, it must first predict the fourth, and so on down the chain. Its predictions are accurate, but that accuracy costs time. Every additional draft token means another sequential forward pass. The result is a familiar trap: you want to predict further ahead, but drafting itself starts slowing things down and eats into the gains.

Imagine an assembly line where every worker waits for the previous one to finish. Adding workers is pointless: the length of the chain determines the speed.

What replaces the chain

DFlash took an earlier step toward a solution. Inspired by diffusion language models, it generates the entire draft in parallel: one pass through the draft network produces a whole block of tokens. Latency no longer grows with draft length. But full parallelism comes at a price: tokens within the block are predicted almost independently of one another, whereas in language the next word depends heavily on the previous one. This can reduce the proportion of tokens accepted by the verifying model.

Judging by its title, the new study addresses precisely this trade-off. Causal dependencies between draft tokens are built into the depth of the draft network itself: instead of sequential steps over time, the dependency is encoded in the layers traversed by a single parallel pass. Put simply, token order is accounted for within one pass, rather than through several consecutive passes. Drafting therefore retains the speed of parallel approaches while moving closer to autoregressive accuracy.

For specific architectural details, model sizes and complete benchmark tables, consult the paper and repository. The figure of up to 4 times refers to throughput in SGLang and, as always with an "up to" claim, describes the best-case scenario rather than a typical result.

What this means in practice

First: if you use SGLang and already have speculative decoding enabled, it makes sense to test the new method on your own traffic. The gains depend heavily on the model, response length, batch size and the proportion of tokens the verifying model accepts. With short responses or large batches, the improvement may be considerably smaller than the headline suggests.

Second: look beyond tokens per second. Time to first token, behavior under heavy load and the draft network's GPU memory consumption also matter. The additional memory allocated to drafting can leave less room for the KV cache and reduce the number of concurrent requests.

Third: training the draft network requires data and compute. If you fine-tune or replace the base model, you will need to retrain the draft model; otherwise, the token acceptance rate will fall. Factor that into implementation costs.

Limitations and conclusion

For now, these results come from a single publication, with few independent reproductions. The method is tied to a particular stack, and porting it to vLLM, TensorRT-LLM or custom kernels will take work. Still, the direction is clear: the race is not for a smarter base model, but to spend fewer sequential steps on each token. For teams whose GPU bill has become their biggest expense, open-source code promising a severalfold gain is worth testing in the next sprint: compare throughput and token acceptance rates on real requests before and after.