mLSTM и ортогонализация чтения: прирост recall — это обучаемость, а не память

In July 2026, a preprint appeared on arXiv (arXiv:2607.19390) that re-examined a widely discussed result: orthogonalizing the memory matrix of the recurrent mLSTM network during reading (five iterations of the Newton–Schulz algorithm, trained end-to-end) sharply improves noisy associative retrieval. The effect reproduces, but, as the authors show, it is not an improvement in memory — it is an acceleration of learning.

A plateau, not memory

Training on this task is a long plateau at the level of random guessing, followed by a sharp jump to the solution. According to the preprint, orthogonalized reading works precisely during the plateau: it reconditions the optimization problem itself rather than expanding the amount of stored information. Direct decoding of the memory state showed that "failed" models already hold about half of the associations in a linearly recoverable form — the plateau turns out to be a reading failure over a half-written store, not an absence of writing.

Three tell-tale properties

The authors present three properties that prove the effect is about trainability, not memory capacity.

• Consistency. Exact reading via the recursive least-squares method (the Mesa layer) reproduces the effect, while straight-through estimation, delta-rule writing, frozen random keys, and ordinary normalization do not.
• Uniformity. Across a grid of "learning rate × complexity," the trick multiplies the probability of escaping the plateau by roughly six times, with no noticeable dependence on complexity, and widens the workable learning-rate corridor.
• Removability. Applied to already-failed models at inference time, the trick does not rescue a single one; switched off via annealing right after the jump, it leaves ordinary mLSTMs at full accuracy.

Why the gain disappears without the architecture

Most of the published gain requires no architectural changes at all. The share of solved tasks at a fixed budget, as the authors note, measures the probability of escaping the plateau, and that probability obeys a "heat and noise" law: elasticity with respect to learning rate of +3.0, with respect to gradient noise of −1.65. Within this framework, the original result on a 96-token vocabulary (vocab-96) turns out to be a low-noise condition caused by large batches, rather than a manifestation of memory capacity.

"Memorization benchmarks used for architecture selection partially

measure trainability," the abstract of preprint arXiv:2607.19390 states.

What this means

The result strikes at two industry habits. First, recall benchmarks used to select model architectures partly measure not memory but how easily a model trains. Second, the system turns out to be a vivid "model organism" of emergence: a sharp threshold in behavior arises from a censored metric layered over a gradually accumulating structure — not from the sudden appearance of a new capability.

Frequently Asked Questions

What is orthogonalized reading in mLSTM?

Orthogonalized reading makes the mLSTM memory matrix orthogonal at the moment of reading — here, via five iterations of the Newton–Schulz algorithm, trained end-to-end. The preprint shows that it improves not data storage, but optimization conditions.

Why can't the gain be considered a memory improvement?

The gain is "removable": the trick does not help already-trained failed models at inference time, and it is fully switched off via annealing after the model makes its jump, with no drop in the accuracy of the ordinary mLSTM. According to the preprint, about half of the associations are stored in linearly recoverable form even without it.

What is emergence in this context?

Emergence here is a sharp behavioral threshold that arises not from a jump in capability, but from a censored metric layered on top of smoothly accumulating structure inside the model.