DAGFormer Plans Connections Between Layers and Cuts Perplexity by 17–25%

Researchers have presented DAGFormer, a transformer-based language model in which a topology predictor, trained jointly with the main network, plans additional connections between layers. On OLMo-2 pairs ranging from 75M to 1B parameters, WikiText-2 perplexity drops by 17–25% compared with a vanilla transformer. The 1B-parameter version, trained on 5B tokens, outperforms the vanilla one trained on 10B. The work was published in Hugging Face Daily Papers.

How the routing works

In a standard transformer, layers are linked by residual connections: each block receives the output of the previous one plus a "skip." DAGFormer adds direct connections from early layers to later ones, turning the network into a directed acyclic graph. The strength of each connection is not fixed. The topology predictor determines it from the input tokens, so the route through the layers can differ from one text to another.

The main difference from existing cross-layer routing architectures is the scale of planning. Earlier approaches tune each layer's connections separately, without a view of the whole network. DAGFormer plans the routing of the entire model in a single pass, and the authors see this as the source of the gains.

What the measurements showed

The comparison used matched OLMo-2 pairs with identical data and budget. Key figures from the paper:

• perplexity on WikiText-2 is 17–25% lower than the vanilla transformer (sizes from 75M to 1B);
• 1B DAGFormer trained on 5B tokens beats the vanilla model trained on 10B tokens;
• lowest perplexity among four routing baselines at 75M, 300M and 1B;
• across 13 tasks (commonsense, science, reading, math), average accuracy is 2.0–2.5 points higher at 300M and 1B;
• after instruction finetuning, the AlpacaEval win rate against the vanilla model is 63–70%, higher than for any routing baseline;
• under structural pruning, DAGFormer degrades the least in five of six settings.

The most practical result here is arguably data efficiency: half the tokens for comparable or better quality. The pruning result also deserves attention, since it is exactly what teams face when compressing models for production. The accuracy gain on downstream tasks is more modest than the perplexity gain: two points on average is noticeable, but not dramatic. AlpacaEval relies on scoring by another model acting as judge, so 63–70% is better read as a strong signal than as a final verdict.

What to check before adopting it

Three things are unclear from the description. The first is the cost of the predictor itself: how many parameters, how much memory and how much inference latency it adds. The second is scale: the experiments stopped at 1B, and carrying the result over to 7B and above remains a hypothesis. The other half of that question is whether the advantage over the baselines holds once long contexts and mixtures of experts come into play. The third is whether such a topology can be built into already trained checkpoints, or whether the architecture has to be designed in from scratch. The text does not say.

For teams that train small models from scratch and are constrained by a limited token budget, the work offers a concrete idea to reproduce: replace fixed connections with planned ones and compare on your own dataset. For those working with ready-made open weights, it is enough for now to watch for a code release and independent replications. If the gain holds at 7B and above, and the overhead turns out to be small, the approach will become a notable candidate for the standard toolkit of architectural techniques.