A no-hair theorem for transformers: the prompt is forgotten as t^-α, α≈0.55–0.7

How much does a transformer remember about its prompt after generating t tokens? The authors of a paper featured on Hugging Face Daily Papers measured this directly and found a power law. The distance between the hidden states of two different prompts with an identical continuation decays roughly as t^{-α}. For trained models, the exponent α is 0.55–0.7; for untrained ones, 0.2–0.5. For reference, α=1 matches the rate at which the share of a fixed prompt in uniform attention shrinks as the context grows. The smaller α is, the longer the memory lasts.

How forgetting was measured

The experiment is simple. Take two different prompts and append the same continuation text to both. Any difference in hidden states at position t is then determined only by the prompt, not by what was generated afterward. What remains is to watch how that difference fades as t grows.

The result is not obvious. A trained network forgets faster than a randomly initialized one: an exponent of 0.55–0.7 versus 0.2–0.5. Training pushes the model toward the reference value α=1 but does not reach it. A fixed share of the prompt in uniform attention gives exactly 1/t, while real models decay more slowly.

The theorem and its conditions

For finite depth L and prompt length P, the authors prove a bound of O(P(log t)^{L-1}/t). It holds for pre-LN transformers with softmax attention and bounded attention logits, plus one more condition: pre-normalization activations must not approach zero. That condition is what makes the theorem conditional.

The mechanism is geometric. The prompt's direct influence is diluted because attention is split among a growing number of tokens. There is also a detour that the authors call a relay. Continuation tokens read something from the prompt and pass it along. But this passing adds only logarithmic factors to the decay and does not change the exponent. Hence the name: by analogy with the "no-hair" theorem for black holes, little information about what is inside remains visible from outside. If you fix the text that has already been generated, the prompt's influence on the next token goes to zero. Whatever persists, such as the language of the response, has to be carried by the text itself.

What the checks confirmed

An exactly solvable linear model with a relay shows that the bound cannot be improved. The same model reproduces the growth of the local exponent toward 1 that is observed in untrained networks.

In a separate control experiment, the authors added a bias to the logits that compensates for dilution. Early in training, the decay stops; later it reverses. So dilution is necessary for the measured forgetting. Untrained networks and the solvable model show that it is also sufficient.

Key figures from the paper:

• Decay of the distance between hidden states: t^{-α}.
• Trained models: α≈0.55–0.7.
• Untrained models: α≈0.2–0.5.
• Dilution reference: α=1.
• Proven bound: O(P(log t)^{L-1}/t), where P is prompt length and L is depth.
• Conditions: finite depth, pre-LN, softmax, bounded logits, activations away from zero.

What this means for engineers

The work is theoretical and offers no direct recipes for production. But the direction is clear. In long generation, a system prompt loses its direct influence following a power law, and the model will keep following instructions only to the extent that they are reflected in the text already written.

So it is useful to anchor important constraints in the first tokens of the response or to periodically return them to the context. For architecture developers, the takeaway is different. If you need long memory of the start of the context, you will have to change attention itself, for example by adding logit biases, as in the control experiment.

That is a separate engineering task: the theorem describes a standard pre-LN transformer, not every scheme with memory.

Caution is warranted in another respect too. The bound is an upper bound, and the measurements concern hidden states, not response quality. How exactly α relates to instruction following on real tasks remains to be tested separately.