Layer Duplication During Training Lifts GSM8K from 12% to 19% on a 3B Model

Researchers Ferdinand Kapl, Kaitlin Maile, Stefan Bauer and Johannes von Oswald compared three ways to build a language model of up to 3 billion parameters. The first is a standard dense network. The second is a network that gradually gets deeper during training by copying its middle layers (the LIDAS method).

The third is a "looped" network, in which the same layers are run several times in a single pass. The team looked at in-context learning (ICL): a model's ability to understand a task from a couple of examples in the prompt, without fine-tuning. Both depth growth and looping turned out to learn ICL noticeably better than the dense model.

At the 3B scale, LIDAS reached about 19% exact match on GSM8K with 8 examples in the prompt, while the dense model scored about 12%.

Two Ways to Reuse Depth

In a standard model, the number of layers is fixed in advance, and every layer is trained from scratch. LIDAS works differently: training starts with a shallow network, then the middle layers are duplicated, and the network grows as it trains. At first the copies are nearly identical, then they diverge. The early stage of training runs on a small model, so it is cheaper. According to the authors, at 3B LIDAS uses 26% fewer training FLOPs (floating-point operations) and still beats the dense model.

Looping is simpler. The model takes a small set of layers, and the data passes through it several times in a single forward pass. The result is fewer unique parameters and a greater effective depth.

The experiments covered models from 360M to 3B parameters. The code is built on Hugging Face's nanotron library. The authors are not releasing their own code or weights.

Where the Models Win

GSM8K is a set of grade-school math word problems. Eight solved examples go into the prompt, and the test checks whether the model's answer matches the reference. Results for the 3B models:

• LIDAS: about 19% exact match;
• dense model: about 12%.

The second test is RegBench: new formal languages the model has not seen before. From a few examples, it must pick up the rules and predict the next symbol. Quality is measured with total variation distance, the difference between the predicted probabilities and the true ones. Lower is better:

• depth growth: ~0.119;
• looping: ~0.125;
• dense model: ~0.137.

This is where it is genuinely ICL: the test gives the model nothing to recall from its weights, so all it can do is read the context.

Duplicated Induction Heads and the Price of the Gain

The authors explain the result through induction heads. These are attention heads that find an earlier occurrence of the current token in the context and suggest what followed it. They are considered one of the main mechanisms behind ICL.

In the 3B model trained with LIDAS, 10 of the 12 strongest induction heads descend from two ancestors in the early four-layer network. A circuit found in the small model gets copied across the full depth as the network grows. It looks as though it is not later displaced by the knowledge the network stores in its weights.

Another check was ablation. The authors switched off the top 20% of such heads and measured how much the average F1 fell across 19 few-shot tasks:

• dense model: a drop of about 0.08;
• LIDAS: about 0.18;
• looped model: about 0.28.

So the grown and looped models rely more heavily on these circuits, and their entire ability to learn from examples is concentrated in them.

What Engineers Should Keep in Mind

For teams that pretrain models, the takeaway is practical. If you want to save compute and get good ICL, gradual depth growth looks like a reasonable option. At 3B it delivered both better results and 26% savings in FLOPs.

Looping carries a higher price. Such a model has fewer unique parameters and is worse at retrieving knowledge stored in its weights. The ICL gain has to be paid for: the model reads context better but remembers facts worse.

There is also a caveat about reliability. Every configuration from 360M to 3B was trained with only one seed, that is, one initial state of the random number generator. The variation between runs has not been measured. A seven-percentage-point gap on GSM8K looks convincing, but without repeated runs it is impossible to say how robust it is. A scale of up to 3B is also small by the standards of modern models, and whether the conclusions transfer to tens of billions of parameters needs to be tested separately.

Still, the result is unexpected. The dense model, trained in the conventional way, came out worse precisely where it was considered strong: in learning from examples in the context. The authors' explanation is early copying of attention circuits, and it can be tested at other model sizes.