LISA: sparse attention ускоряет длинные рассуждения LLM на 50% без переобучения

Researchers have introduced LISA (Linear-Indexed Sparse Attention) — a drop-in replacement for the attention mechanism that in July 2026 showed a 50% inference speedup at a context of 16,000 tokens and a 5.6% boost in reasoning quality, without requiring the model to be trained from scratch. The work was published on arXiv (2607.19358).

How LISA works

LISA is embedded into an existing model as a plug-and-play replacement for standard self-attention and combines two parallel branches. The first is Linear Attention with O(n) complexity, responsible for "long-term memory" and long-range dependencies. The second is the Lightning Indexer mechanism, which selects the top-M most important tokens from the entire context and passes them to Sparse Self-Attention. The branches are combined via a gating mechanism.

• Name: LISA (Linear-Indexed Sparse Attention), arXiv preprint 2607.19358
• Inference complexity drops from O(n²) to O(nM), where M is much smaller than n
• Two components: Linear Attention (O(n) memory) + Lightning Indexer + Sparse Self-Attention
• Requires no pretraining from scratch — the module is embedded into a ready-made model
• Tests were run on DeepSeek-distilled-Qwen models

Why this matters for inference

Standard self-attention has quadratic complexity O(n²), which causes inference cost to grow sharply on long sequences. This has become a bottleneck for models with long chain-of-thought reasoning, such as DeepSeek-R1: the test-time scaling paradigm forces them to generate ever-longer reasoning context, and costs grow along with it. According to the authors, this is precisely what limits the deployment of long-CoT models in production.

LISA attacks the problem architecturally, rather than through more powerful hardware. By replacing part of full attention with sparse and linear attention, the method shifts the generation of n tokens from O(n²) mode to O(nM) mode — with M (the number of selected tokens) noticeably smaller than the context length n.

What the tests showed

On models from the DeepSeek-distilled-Qwen family, LISA delivered a 50% inference speedup at a context of 16K tokens while simultaneously raising the average score on reasoning benchmarks by 5.6% — including AIME and MATH-500. In other words, the method doesn't just speed things up, it also doesn't sacrifice quality.

Training is built in two stages. In the first, linear attention for long-range dependencies is integrated into the model, supplemented by a sliding-window mechanism; it is optimized via knowledge distillation to approximate the distribution of full self-attention from a "frozen" teacher model. In the second stage, the static sliding window is replaced by the Indexer, trained through a new loss function — a per-head KL divergence that aligns its token selection with the teacher's attention patterns.

"LISA delivers a 50% inference speedup at a context of 16K tokens,

while simultaneously improving the average score by 5.6% on reasoning benchmarks, including AIME and MATH-500," the study's abstract on arXiv states.

What it means

Long reasoning is expensive precisely because of quadratic attention, and LISA shows a way to cut costs without retraining the model from scratch — it's enough to embed the module into an existing network. For teams deploying reasoning models in production, this is a direct way to cut inference costs without sacrificing accuracy.

Frequently asked questions

What is LISA?

LISA (Linear-Indexed Sparse Attention) is a drop-in replacement module for the attention mechanism in language models. It combines linear attention and sparse attention with selection of important tokens, reducing inference complexity from O(n²) to O(nM).

Does the model need to be retrained for LISA?

No, training from scratch is not required. LISA is a plug-and-play replacement for self-attention; it is embedded into a ready-made model and tuned in two stages via knowledge distillation from a teacher model.

How much does LISA speed up inference?

According to the abstract, on DeepSeek-distilled-Qwen models the method delivers a 50% inference speedup at a context of 16K tokens while also raising average reasoning quality by 5.6% on the AIME and MATH-500 benchmarks.