Hybrid LLMs: Recurrent Layers Suppress Attention, and Staged Training Helps

Hybrid language models, in which linear recurrence works alongside softmax attention, do not train the way their designers assumed. The authors of Competing Learning in Hybrid Architectures show that layers of different types compete for the same function. The layer that learns it first suppresses the training signal for the others. Recurrence usually wins, and attention is left without a retrieval skill. Halting the weight updates of the recurrent layers midway through training brings attention back into play. On models up to 1B parameters, the staged scheme improved language modeling loss, common-sense reasoning and retrieval tasks across different context lengths.

What the conflict is

Hybrids are popular because of a trade-off. Modern recurrent layers offer expressive token mixing and high throughput because they compress context into a fixed-size state. The cost is well known: when extracting specific information from context, they fall short of softmax attention, which accesses tokens directly. The idea behind a hybrid is to keep attention where precise lookup is needed and hand the rest of the work to cheap recurrence.

According to the authors, this logic breaks down during training. Layer attribution analysis showed that, under typical data distributions, modern recurrence learns retrieval functions faster than attention. Once one layer has learned a function, the training signal for the other layers on that function is suppressed. As a result, in most cases attention never learns to look things up. The architecture remains hybrid on paper, but the layer it was designed around does not do its job.

Staged training

The test turned out to be simple. If the weight updates in the recurrent layers are stopped midway through training, attention has time to learn retrieval. The authors built a staged training procedure for pretraining hybrids on natural language on this observation: recurrence gets to work first, then makes room for attention.

The abstract does not give details: at what point to freeze the layers, how the stages are structured, and what happens to the recurrence after it is frozen. These parameters determine whether the method is reproducible, and readers will have to look for them in the full text of the paper.

• Scale of the test: models up to 1B parameters.
• Phenomenon: Competing Learning, in which mixers compete for the same function.
• Diagnostic tool: layer attribution analysis.
• Technique: stopping updates to the recurrent layers midway through training.
• Metrics: language modeling loss, common-sense reasoning, retrieval tasks at different context lengths.
• The abstract gives no numerical gains and claims only a "substantial improvement."

What to check in practice

Teams that pretrain their own hybrids should check whether the attention layer actually works on retrieval. Attribution analysis and an ablation that freezes the recurrent layers at different stages are suited to this. In-context retrieval tests should run at several lengths, not just one. A "hybrid versus pure recurrence" comparison can hide the problem: attention is formally present but contributes nothing to lookup.

The result has limitations. 1B parameters is far from the sizes at which modern commercial models are trained. It is unknown whether the effect holds at tens of billions of parameters and on other data mixes. The claimed gain is not yet backed by numbers in the available description. Still, the diagnostic itself is useful: if layers crowd each other out, tuning the ratio of recurrent to attention layers alone will not solve the problem. The order in which they are trained has to be managed.