AWS and vLLM roll out P-EAGLE, speeding up large LLM inference by up to 1.69x

AWS and the vLLM team have unveiled P-EAGLE, a new way to speed up inference for large language models without changing the main model. The approach is already integrated into vLLM, and in tests with GPT-OSS 20B it delivered up to 1.69 times more throughput than standard EAGLE-3.

Where the bottleneck was

Speculative decoding has long been considered one of the most practical ways to accelerate LLM inference. A helper model proposes several upcoming tokens in advance, and the main model quickly checks which of them can be accepted. The EAGLE method already brought a noticeable gain and was used in vLLM, SGLang and TensorRT-LLM.

But one important problem remained: to generate K draft tokens, the drafter model had to run K sequential forward passes. The deeper the speculation, the more latency the draft itself added. As a result, classic EAGLE hit a hidden ceiling. On paper, it made sense to speculate deeper and accept more tokens per round, but in practice the drafter's extra work started to eat the gains.

The P-EAGLE authors remove exactly this limitation: all K draft tokens are generated in a single pass. This shifts the balance toward more aggressive speculation, especially for long responses and coding tasks, where every extra sequential operation shows up in latency and throughput.

How P-EAGLE works

P-EAGLE uses a two-step architecture. First, the target model processes the prompt and predicts the next token, as usual. Along the way, the system saves the internal hidden states for the prompt positions and for the new token.

Then the drafter assembles inputs for all future positions in parallel: for the parts of the sequence that already exist, it uses real embeddings and hidden states, and for positions that don't exist yet it substitutes learnable masks and a shared hidden vector. After that, several future tokens are predicted in one forward pass rather than a chain of several steps.

Training on long sequences is a separate challenge. AWS writes that for GPT-OSS 120B on UltraChat, the median sequence length, prompt and generation combined, reached 3891 tokens, and the 90th percentile reached 10800 tokens. With parallel draft decoding, memory grows very quickly because the number of positions becomes N × K. To handle this, the authors added a sequence partition algorithm: it splits one long sequence into contiguous chunks, preserves correct attention dependencies between them, and allows gradients to accumulate within a single example, not only across different batches.

Integration and numbers

The practical side goes beyond the paper: P-EAGLE has already been added to vLLM, starting with version 0.16.0. To enable it, you use speculative decoding with the flag parallel_drafting: true and attach a compatible drafter head. AWS has already published ready-made checkpoints for GPT-OSS 120B, GPT-OSS 20B and Qwen3-Coder 30B, so the technology can be tried without training from scratch.

• The integration appeared in vLLM in version 0.16.0
• The mode is enabled via the flag parallel_drafting: true
• Ready-made P-EAGLE head models are available for GPT-OSS 120B, GPT-OSS 20B and Qwen3-Coder 30B
• On NVIDIA B200, the gain over standard EAGLE-3 ranged from 1.05× to 1.69×
• P-EAGLE's best throughput in the tests came at a speculation depth of K=7

The benchmarks look consistent. On MT-Bench, HumanEval and SPEED-Bench, the new method gained 55–69% at low concurrency and kept a 5–25% advantage even under high load.

Beyond speed, acceptance length also increased: the average number of draft tokens the verifier accepts per round. For example, at K=7 on HumanEval, P-EAGLE reached 3.94 versus 3.03 for EAGLE-3, and on SPEED-Bench 3.38 versus 2.59.

AWS separately notes that running GPT-OSS 20B with an EAGLE drafter still requires a one-line patch in vLLM, which is expected to land in one of the upcoming releases.

What this means

For teams already running vLLM in production, P-EAGLE looks like a rare improvement that doesn't require rebuilding the whole stack: the new scheme is built into a familiar runtime and is activated by a config setting plus a compatible checkpoint. If the ecosystem quickly gets more drafter models trained for parallel decoding, this variant of speculative decoding could become the new standard for fast, cheap LLM inference.