NX-AI's KronLSTM Changes the Delta Rule and Stabilizes Memory Without Normalization

NX-AI, the company of Sepp Hochreiter and the developer of the xLSTM family, has open-sourced KronLSTM, a linear attention architecture with a new update rule for associative memory. The authors' main claim is that the memory behaves predictably at any input length, its expected norm is uniformly bounded, and on long-context benchmarks the model competes with the (Gated) DeltaNet family and often outperforms it. The abstract contains no table of numbers, so the size of the advantage can be judged only from the paper itself and the published weights.

The problem: memory that forgets the wrong things

Linear attention replaces the transformer's growing KV-cache with a fixed-size matrix. The model writes key–value pairs into it and reads them out by query. Compute cost is linear in sequence length, and the memory used for state is constant. The price is interference: every new write disturbs older ones. The delta rule in DeltaNet and Gated DeltaNet partly addresses this by first subtracting from memory what is already associated with the key, then writing the new value. But even this correction touches more information than one would like.

KronLSTM introduces the Kronecker Delta Rule. Its memory revision mechanism corrects target associations and changes the matrix along only one direction in matrix space. According to the authors, this minimally invasive intervention reduces interference with previously stored content compared with the delta rule. This matters for long context: the more tokens go by, the more each unnecessary overwrite costs.

The math: a Markov chain instead of manual stabilization

The paper puts particular emphasis on analysis. The memory dynamics can be described as a Markov chain that converges to a stationary distribution. Throughout, the expected norm of the memory matrix stays uniformly bounded across the entire input sequence.

In practice, this removes a whole layer of engineering workarounds. In many recurrent architectures, the state has to be kept in check with normalization, decay gates or additional stabilizers. The authors say KronLSTM does without extra normalization and state-stabilization mechanisms. They call the rule an analytically transparent foundation for stable associative memory in recurrent linear architectures.

What we know about the model

• Type: linear attention, an extension of the xLSTM family
• Key idea: Kronecker Delta Rule, memory revision along a single direction
• Comparison: (Gated) DeltaNet and other linear attention architectures
• Result, as claimed by the authors: competitive scores, often better, especially on long context
• Stability guarantee: the expected norm of the memory matrix is uniformly bounded
• License and access: code on GitHub (NX-AI/KronLSTM), models in the NX-AI collection on Hugging Face

Practical takeaway

For teams building long-context models and counting GPU memory spent on the KV-cache, KronLSTM is worth adding to the list of architectures to compare against Gated DeltaNet. Code and weights are open, so you can test it on your own tasks: search over long documents, associative retrieval, work with code repositories.

Three things are worth watching. First, how quality degrades as length grows beyond the training length. Second, training and inference speed on real kernels, since theoretical linearity does not guarantee a wall-clock win. Third, behavior in hybrid setups where linear layers alternate with standard attention. The absence of extra normalization simplifies implementation, but the final verdict depends on independent measurements. For now this is a strong claim with a careful theoretical foundation, and the community will be the one to test it.