MiniMax releases MSA: sparse attention speeds up million-token context processing by 28x

MiniMax has published research on MSA (MiniMax Sparse Attention), a new sparse attention mechanism that processes million-token contexts 28 times faster than standard Grouped Query Attention, with no loss of quality on benchmarks.

How MSA works

MSA is built on top of Grouped Query Attention (GQA) and uses a two-branch, block-sparse architecture. Traditional attention works on an "everyone with everyone" principle: each query token attends to all keys and values in the context. At a million tokens, this requires an astronomical number of operations and becomes computationally prohibitive.

MiniMax's solution splits the process into two independent stages. First, a lightweight Index Branch, a specialized module, selects the Top-k most relevant key and value blocks for each query and each GQA group. This selection is fast and cheap. Then the Main Branch performs full, exact attention, but only over the selected blocks rather than the whole context.

Importantly, MSA's block-sparse structure is compatible with existing optimized CUDA kernels. The team did not need to build custom GPU implementations: the mechanism runs efficiently on standard hardware.

Training scale and results

MSA was trained as part of a 109-billion-parameter MoE (Mixture of Experts) model with a total budget of 3 trillion tokens. In terms of scale, it is one of the largest publicly documented experiments with sparse attention mechanisms.

Key results:

• Per-token compute load reduced by 28.4x at a 1 million token context compared with full GQA
• Quality on standard downstream benchmarks is comparable to baseline GQA, with negligible degradation
• Architectural compatibility: MSA slots in on top of existing GQA without reworking the rest of the model
• Runs on standard CUDA kernels with no custom implementations

"MSA preserves the accuracy of GQA while cutting attention compute costs 28-fold at a context of one million tokens," says the

MiniMax technical report.

Why long context is so hard to scale

Processing long sequences is a long-standing problem for transformer architectures. KV cache memory grows linearly with context length, while the number of attention operations grows quadratically. For a 1 million token context, that means trillions of query-key pairs per model layer.

Most existing solutions make trade-offs: they truncate the context, use a sliding window, or switch to linearized approximations of attention, losing accuracy. Alternative architectures such as Mamba change the model structure fundamentally, which complicates transferring already trained weights and compatibility with transformer infrastructure.

MSA takes a different route: keep the Main Branch exact (no approximations) and delegate the selection of relevant blocks to the lightweight Index Branch. This division of labor is what delivers the 28x speedup without noticeable quality loss.

What this means

If the results hold up in production conditions, MSA could become a practical path to cheap long contexts for working with large documents, codebases and multimodal inputs. MiniMax has opened up the technical details of the method, which lets other teams adapt the approach to their own scales.