ResonatorLM Taught to Read Memory by Token: Perplexity at 1024 Drops from 12.81 to 3.76

Researchers have proposed a small change to ResonatorLM, a fixed-state-size model that reads memory from a bank of decaying complex oscillators. The read phase, previously constant for each head, is now a function of the current token. This added 0.2% more parameters. In experiments with character-level models of 6M parameters (WikiText-2), quality at the training length did not change. But when trained on 256 tokens and tested on 1024, perplexity fell from 12.81 to 3.76.

The Fork in the Road Where the Work Started

Fixed-state-size models avoid the quadratic cost of attention, but they force a trade-off. If the state transition depends on the input, the memory becomes content-dependent. The price is FFT-based training, which can no longer be used in its usual form. If the transition is fixed, the convolutional path with O(T log T) complexity survives, but memory is read the same way for any input.

ResonatorLM belongs to the second type. Each head reads the oscillator bank through a fixed, learned phase. The authors of the new work left the transition as it was and changed only the read: the phase is now computed from the current token. Input dependence appears only at the read stage, so the convolution stays exact and training follows the same scheme as before.

What the Analysis of the Untrained Bank Showed

Before running experiments, the authors analyzed the oscillator bank without any training. The phase does encode elapsed time. But the memory turns out to be short: an ordinary token drops below the background level within 2–8 steps. No choice of decay extends linear reconstruction beyond roughly 16 tokens.

Key figures from the paper:

• Parameter increase: 0.2%.
• Model scale: 6M parameters, character level, WikiText-2.
• Trained on 256, tested on 1024 tokens: perplexity of 12.81 for the original model and 3.76 for the variant with input-dependent phase.
• At the training length, results match the original.
• Linear reconstruction horizon in the untrained bank: up to ~16 tokens.
• A token falls below the background within 2–8 steps.

Where the Failure Beyond the Training Length Comes From

The main diagnostic finding concerns the long-memory heads. They accumulate more history than they saw during training, and on long inputs they read the state in a regime the model has never encountered. This explains why the original ResonatorLM degrades beyond its training length. Making the phase input-dependent eliminates the problem: the gap between 12.81 and 3.76 shows how large it is.

The mechanism turned out to be more modest than one might expect. The learned phase tracks the current character, not the distance to past tokens, and retrieval quality does not improve. In other words, the model did not learn to pull the right fragment out of the past more accurately. It simply stopped breaking when more history accumulates than it saw in training.

Where This Is Useful

For anyone working with convolutional state-space models with a fixed transition, the takeaway is practical: test length extrapolation separately. It is enough to train a model on a short context and compare perplexity on a context 4 times longer. A gap of this size can hide behind good metrics at the training length. Diagnosing long-memory heads, as in the paper, points to where to look for the cause.

There are limitations. The experiments used models with 6M parameters, at the character level, on a single dataset. Transfer to token-level models with billions of parameters remains untested. Retrieval does not improve, so for tasks that need exact extraction from context, the method is not a solution. As cheap insurance for long inputs at almost no parameter cost, it looks convincing.