Parallax: improved local linear attention with covariance correction

Researchers presented Parallax — a parameterized variant of Local Linear Attention (LLA), which retains the familiar softmax mechanism and adds a learnable covariance correction branch. On May 31, 2026, the publication MarkTechPost reported this: according to the publication, the new method replaces the per-query solver used in LLA with a learnable projector, which doubles the arithmetic intensity of computations and improves perplexity of models at scales of 0.6 and 1.7 billion parameters.

What is the limitation of linear attention

The classical attention mechanism (self-attention) in transformers is one of the main sources of computational costs in training and inference of language models: its complexity grows quadratically with the length of the input sequence. Linear attention was proposed as a way to overcome this limitation, bringing computations closer to linear complexity, but usually at the cost of some loss in quality compared to full softmax attention. Local Linear Attention (LLA) is one of the approaches in this direction, which for each query solves a local approximation task, but this creates its own computational efficiency bottlenecks.

Key facts about Parallax:

• Method is built on the basis of local linear attention (LLA).
• Per-query solver is replaced by a learned projector.
• Softmax attention mechanism is retained, a covariance correction branch is added.
• Arithmetic intensity of computations doubles compared to the baseline approach.
• Perplexity improvement is recorded on models with 0.6 and 1.7 billion parameters.

What the learnable projector changes

The key architectural difference of Parallax is the replacement of the solver, which in the original LLA variant was recalculated separately for each query, with a single learnable projector. This replacement reduces the computational load specific to each individual query and transfers part of the "intelligence" of the attention mechanism to parameters that are learned in advance together with the rest of the model. The added covariance correction branch, judging by the method's name, is designed to compensate for inaccuracies in linear approximation, bringing the quality of the mechanism closer to full softmax attention while maintaining the speed advantage characteristic of linear attention methods.

What the experiments showed

The authors note two practical results. First, doubling of arithmetic intensity — a metric that shows how efficiently computing hardware is used for each byte of data transmitted: the higher the arithmetic intensity, the less computations are bottlenecked by memory bandwidth rather than the performance of the computing cores themselves. Second, the method showed perplexity improvement — the standard quality metric for language models reflecting how well the model predicts the next token — on models of relatively small scale: 0.

6 and 1.7 billion parameters. For the research community working on efficient transformer architectures, such results are interesting primarily as proof that linear attention mechanisms can be brought closer to the quality of full attention without sacrificing the gains in computational efficiency.

Why this research direction is important

The search for alternatives to classical quadratic attention is one of the most active areas of current research in language models: in recent years, in addition to linear attention, other architectural approaches to reducing computational costs for processing long sequences have been proposed — recurrent and state-space architectures, various variants of sparse attention and hybrid schemes combining multiple mechanisms in a single model. The common motivation for all these directions is the same: the length of context that models need to process — from short queries to entire books, code repositories and multi-hour dialogues — is growing faster than computing becomes cheaper, which means the efficiency of each attention mechanism directly determines how long a context can be afforded to process at a given infrastructure budget. Against this background, Parallax testing is so far limited to comparatively small models — 0.

6 and 1.7 billion parameters — and whether the claimed advantages will be preserved when scaling to the size of modern industrial LLMs remains a question for further research.

The practical value of such architectural improvements rarely manifests itself instantly: the path from a research publication to applying the method in industrial-scale models usually takes months or years and requires independent verification on larger architectures and diverse datasets. Nevertheless, the very idea of retaining the softmax mechanism, rather than completely abandoning it for linear complexity, distinguishes Parallax from more radical attention alternatives — it is rather an evolutionary refinement of an existing approach than an attempt to invent a fundamentally new architecture from scratch.