ECHO: Multi-Step Forecasting Without Add-Ons, Half the Cost of MTP

ECHO: Multi-Step Prediction Without Auxiliary Modules

Researchers at Baidu (ernie-research) have proposed ECHO, a method for embedding multi-step prediction directly into a language model's backbone, removing the need for separate auxiliary modules. The approach costs half as much as the standard multi-token prediction paradigm (MTP) and delivers a notable boost on reasoning tasks and code generation.

Traditional approaches to multi-step prediction work as an extension bolted onto the base model: each prediction horizon needs its own module, each module requires a separate forward pass through the network, and training each one is a separate task. ECHO changes this fundamentally—instead of auxiliary modules, the backbone network's final layer directly predicts all horizons.

ECHO's Three Components

The method rests on three key ideas:

A final layer for all horizons. Rather than attaching modules to predict tokens at +1, +2, +3 steps ahead, ECHO moves all of that logic into the base network's final layer. The objective function acts directly on the parameters that generate the next token. The result: gains keep growing as more horizons are added, right where auxiliary modules have long since plateaued.

Dynamic Depth Sampling (DDS). The probability of training on the +1 horizon is higher than +2, which is higher than +3. This makes deep-horizon prediction affordable while preserving its benefits.

Extrapolation beyond the trained horizon. The model turns out to be able to predict further than what it was trained on. At inference time, this lets it serve as its own draft model for speculative decoding.

Results Across Four Task Categories

ECHO's biggest gains show up in its primary categories: • Reasoning — logical inference and multi-step analysis • Code generation — writing programs and algorithms

On secondary categories, the effect is noticeable but more modest: • QA (question answering) — comprehension and information retrieval • Multiple choice — selecting among given options

After supervised finetuning, the gains spread across all four categories. The method proves robust: the advantage holds up under a doubled token budget and across repeated pretraining runs.

Production Deployment

ECHO is already deployed in Baidu's production systems on models at the 200-billion-parameter scale and above. At inference time, the model generates its own drafts, maintaining high acceptance rates even when predicting beyond the depth it was trained on. This fully eliminates the need for auxiliary draft models—the network is immediately ready for accelerated speculative decoding with no after-the-fact tuning required.