LISA: Sparse Attention Speeds Up Long LLM Reasoning by 50% Without Retraining

Researchers have introduced LISA (Linear-Indexed Sparse Attention), a drop-in replacement module for the attention mechanism. In July 2026, it showed a 50% inference speedup at a 16-thousand-token context and a 5.6% improvement in reasoning quality, without requiring the model to be trained from scratch. The work is published on arXiv (2607.19358).

How LISA works

LISA slots into an existing model as a plug-and-play replacement for standard self-attention and combines two parallel branches. The first is Linear Attention with O(n) complexity, which handles "long memory" and long-range dependencies. The second is the Lightning Indexer, which selects the top-M most important tokens from the entire context and passes them to Sparse Self-Attention. The branches are merged through a gating mechanism.

• Name: LISA (Linear-Indexed Sparse Attention), arXiv preprint 2607.19358
• Inference complexity drops from O(n²) to O(nM), where M is much smaller than n
• Two components: Linear Attention (O(n) memory) + Lightning Indexer + Sparse Self-Attention
• Does not require pretraining from scratch: the module is inserted into an existing model
• Tests were run on DeepSeek-distilled-Qwen models

Why this matters for inference

Standard self-attention has quadratic complexity, O(n²), so inference cost climbs sharply on long sequences. This has become a bottleneck for models with long chain-of-thought reasoning, such as DeepSeek-R1: the test-time scaling paradigm pushes them to generate ever longer reasoning contexts, and costs grow along with them. According to the authors, this is precisely what limits production inference for long-CoT models.

LISA attacks the problem architecturally rather than with more powerful hardware. By replacing part of full attention with sparse and linear attention, the method moves the generation of n tokens from O(n²) to O(nM) mode, with M (the number of selected tokens) significantly smaller than the context length n.

What the tests showed

On models from the DeepSeek-distilled-Qwen family, LISA delivered a 50% inference speedup at a 16K-token context and at the same time raised the average result on reasoning benchmarks by 5.6%, including AIME and MATH-500. In other words, the method does not just speed things up; quality does not drop either.

Training is built in two stages. In the first, linear attention is integrated into the model to handle long-range dependencies, supplemented by a sliding-window mechanism. It is optimized through knowledge distillation to approximate the full self-attention distribution of a frozen teacher model. In the second stage, the static sliding window is replaced by the Indexer, which is trained with a new loss function: a per-head KL divergence that aligns its token selection with the teacher's attention patterns.

"LISA delivers a 50% inference speedup at a 16K-token context while

improving the average result by 5.6% on reasoning benchmarks, including AIME and MATH-500," the study's abstract on arXiv states.

What this means

Long reasoning is expensive precisely because of quadratic attention, and LISA shows a way to make it cheaper without retraining the model from scratch: it is enough to insert the module into an existing network. For teams deploying reasoning models in production, this is a direct way to cut inference costs without sacrificing accuracy.

Frequently asked questions

What is LISA?

LISA (Linear-Indexed Sparse Attention) is a drop-in replacement module for the attention mechanism in language models. It combines linear attention with sparse attention that selects important tokens, reducing inference complexity from O(n²) to O(nM).

Does LISA require retraining the model?

No, training from scratch is not required. LISA is a plug-and-play replacement for self-attention; it is inserted into an existing model and tuned in two stages through knowledge distillation from a teacher model.

How much does LISA speed up inference?

According to the abstract, on DeepSeek-distilled-Qwen models the method delivers a 50% inference speedup at a 16K-token context while also raising average reasoning quality by 5.6% on the AIME and MATH-500 benchmarks.