ECHO teaches LLMs to predict tokens ahead without add-on modules, at half the cost of MTP

The ECHO authors propose a way to train a language model to predict several tokens ahead without separate heads or modules. The model's own final layer predicts each horizon. Training overhead is half that of multi-token prediction trained at every depth. Accuracy gains on downstream tasks concentrate in reasoning and code. At inference, the same model acts as its own drafter for speculative decoding.

Where classic MTP breaks down

Multi-token prediction methods are auxiliary by design. The multi-horizon loss mostly trains modules attached to the backbone. Each new horizon means another module and another pass. The further ahead you want to look, the more expensive pretraining gets. The authors point to a second effect: the auxiliary modules saturate, and extra depth stops turning into quality.

The approach has a downside at inference, too. Drafters trained after the fact lose alignment with the target model on open-ended, high-entropy generation, which is exactly where speedups matter most.

How ECHO works

The idea is to make predicting the future part of the autoregressive task rather than an attachment to it. The backbone's final layer predicts all horizons. Supervision lands in the very parameters that generate the next token.

Dynamic Depth Sampling (DDS) handles the expensive depth. At each training step, the depth is chosen at random, with probabilities that fall as depth grows. Shallow horizons are trained often and deep ones less often, but the model still sees them. DDS is what delivers the savings: half the overhead of MTP trained at every depth.

What the authors claim

The abstract has no tables of absolute metrics, so below is only what it says:

• Training cost: half the overhead of MTP trained at every depth.
• Pretraining: added depth converts into downstream accuracy where auxiliary modules saturate. The effect is concentrated in reasoning and code.
• Inference: the pretrained model is its own drafter and can predict deeper than it was trained to.
• Draft–target alignment: it holds on open-ended, high-entropy generation, where post-hoc drafters lose it.
• Robustness: the advantage holds with a doubled token budget, a repeated pretraining run, and after supervised finetuning.

The last point matters: pretraining effects often dissolve after finetuning or with a different seed. The authors claim that did not happen here. You can only verify that from the full text of the paper.

What to check in practice

For teams that pretrain their own models, ECHO is interesting as a way to get a dense signal from future tokens without growing the number of modules. This setup needs no extra drafter for speculative decoding: it is already built into the weights. That simplifies deployment, since you don't have to train, version, and keep a second model in sync.

Before bringing the approach into your own pipeline, a few things need clarifying. Which model sizes and token budgets were used. What the real speedup looks like in tokens per second, not just quality. What share of draft tokens gets accepted in open-ended dialogue. How the probability distribution in DDS was chosen and how sensitive it is to tuning. None of this is in the abstract.

If the results hold at scale, the effect will show up in two places at once: in code and reasoning quality at the pretraining stage, and in inference cost on long free-form generation. For now, this is a claim from a research group without independent reproduction, and it is best treated as a strong hypothesis to test on your own data.