Tencent Open-Sources AngelSpec, a Framework for Speeding Up LLM Inference With Speculative Decoding

On July 30, 2026, Tencent open-sourced AngelSpec, a torch-native framework for training speculative decoding draft models. On Hy3 models, it raised the average acceptance rate of proposed tokens from 52.8% to 66.4% without any loss in generation quality.

What is speculative decoding

Speculative decoding is a way to speed up generation in a large language model without lowering answer quality. A lightweight draft model proposes several future tokens at once, and the main (target) model verifies all of them in a single pass through rejection sampling. The method is lossless: the final text matches what the main model would have produced on its own. The speedup depends on two things: how many draft tokens are accepted, and how long the full draft-and-verify cycle takes.

• Release date: July 30, 2026; open-source license
• The framework is torch-native and works with models from the Hy3 family
• It covers two approaches: autoregressive MTP and the block-parallel DFlash family (DFly, D-cut)
• Average token acceptance rate: 52.8% → 66.4% at T=0
• Average accepted draft length: 2.58 → 2.99 tokens

Why one universal model falls short

A single universal draft model falls short because real traffic is heterogeneous and the two speed metrics pull in opposite directions. In open-ended, high-entropy conversation, dozens of continuations are plausible, so acceptance drops quickly as the proposal gets deeper, and generating a long block does not pay off. Code and math behave differently: syntax, repeated identifiers, and step-by-step derivations constrain future tokens more tightly and create long, predictable stretches. So AngelSpec ships two draft models instead of one compromise: MTP is trained on diverse conversational data, while the block-diffusion model is reinforced with code and math examples.

"AngelSpec treats workload heterogeneity as a first-class design constraint," the

MarkTechPost write-up says.

How the MTP path works

The MTP path addresses the mismatch between training and inference with a scheme that shares parameters across several depths. The original Hy3 model was trained with a single MTP layer and no recurrent unrolling, so tokens at the second and third draft positions were accepted noticeably less often. AngelSpec keeps D logical depths but reuses a single physical MTP block, unrolling it over D steps during training. Following the Training-Time Test principle from EAGLE-3, depth k+1 receives not the ground-truth token but the argmax prediction from depth k. In addition, the main backbone and the language head are frozen, and training runs on rollouts generated by the target model itself.

The effect concentrates where it is needed. According to Tencent, on the GSM8K benchmark, third-position acceptance rose from 0.290 to 0.706, and on HumanEval from 0.387 to 0.757, while first-position acceptance barely changed (0.799 → 0.814).

What this means

AngelSpec shows that inference speedups are better built around the real heterogeneity of a workload than around an averaged benchmark, using different draft models for conversation and for code. For teams serving LLMs in production, it is an open tool for squeezing more speed out of the same hardware without a drop in quality.

Frequently asked questions

What is speculative decoding in simple terms?

It is a technique for speeding up LLMs in which a small model guesses several upcoming tokens ahead, and a large model checks them in a single pass. If the guesses are right, generation runs faster; the quality of the text does not change.

How does AngelSpec differ from a single draft model?

AngelSpec trains two specialized models: MTP for high-entropy conversation, and block-parallel for code and math with long predictable stretches. According to Tencent's measurements, this raised the average token acceptance rate from 52.8% to 66.4%.