Moonshot AI unveils Attention Residuals, an alternative to residual connections in transformers

Moonshot AI has released Attention Residuals, an architectural update for transformers that changes one of the model's most basic elements: residual connections. Instead of a fixed sum of the outputs of all previous layers, the team proposes depth-wise attention, so the model itself decides which representations from across the network's depth it actually needs.

Where the bottleneck is

In most modern LLMs, each new layer doesn't just process its input; it adds its result to a shared hidden state. This scheme, inherited from the residual approach and especially popular in PreNorm architectures, helps train deep networks without gradients collapsing. But it comes at a cost: all earlier representations are mixed with equal weight, and their contribution gets diluted over time. As depth grows, hidden states increase roughly linearly, and early signals become harder and harder to extract in a useful form.

Moonshot AI calls this the problem of depth-wise aggregation: models have learned to pick out important tokens along a sequence and to route experts in MoE, but along the depth of the network they still rely on fixed summation. The researchers draw a direct analogy with the RNN era: sequences were once compressed into a single state, until attention gave each token access to all previous steps. The proposal here is to do almost the same thing, only across layers rather than across time.

How AttnRes works

In Attention Residuals, each layer receives not the plain sum of all previous outputs but a weighted combination of earlier representations computed via softmax attention. The weights depend on the layer itself and on the input data, so the network can amplify useful signals and suppress noise instead of inheriting everything equally.

The practical variant uses a very lightweight mechanism: one learnable pseudo-query vector per layer. That makes the idea look less like a radical overhaul of the transformer and more like a relatively compact replacement for the familiar residual scheme.

The full version of AttnRes requires storing all previous states, so for large models Moonshot AI proposes Block AttnRes. Layers are split into blocks; ordinary accumulation remains inside each block, while attention is applied between the summary representations of the blocks. According to the team, a configuration with roughly eight blocks retains most of the full version's gain, reduces memory and communication requirements from O(Ld) to O(Nd), and keeps additional inference latency below 2%.

What the tests showed

Moonshot AI tested the approach not only in scaling-law experiments but also on the large pretrained Kimi Linear model, with 48 billion parameters of which 3 billion are active, after training on 1.4 trillion tokens. The key claim: Block AttnRes reaches the same loss as a baseline model trained with 1.25 times the compute budget. In other words, this is not cosmetic tuning but potentially more cost-effective scaling.

• GPQA-Diamond: 36.9 → 44.4
• HumanEval: 59.1 → 62.2
• MMLU: 73.5 → 74.6
• C-Eval: 79.6 → 82.5
• Inference latency: less than 2%

Training dynamics matter separately. In its report, the team writes that AttnRes mitigates the PreNorm dilution effect: the magnitude of hidden states does not spread out with depth, and gradient norms are distributed more evenly across layers. In practice, this means more manageable training and a lower chance that part of the model's depth acts as expensive but barely useful ballast. The most noticeable gains came in multi-step reasoning and code generation, which makes the work especially interesting for future LLMs and agent systems.

What this means

This is not a new chatbot or a user-facing feature, but an attempt to rewrite one of the basic building blocks of transformers. If Moonshot AI's results are confirmed on other architectures and in industrial stacks, the race for LLM quality will increasingly be run not only through more data and GPUs, but also through smarter internal mechanics of the models themselves.