MiniMax released MSA: sparse attention speeds up 1M-token context processing by 28x

MiniMax published research on MSA (MiniMax Sparse Attention) — a new sparse attention mechanism that allows processing contexts of up to one million tokens 28 times faster than standard Grouped Query Attention without quality loss on benchmarks.

How MSA Works

MSA is built on top of Grouped Query Attention (GQA) and uses a dual-branch block-sparse architecture. Traditional attention mechanisms operate on an "all-to-all" principle: each query token accesses all keys and values in the context. For a context of one million tokens, this requires an astronomical number of operations and becomes computationally infeasible.

MiniMax's solution divides the process into two independent stages. First, a lightweight Index Branch works — a specialized module that for each query and each GQA group selects the Top-k most relevant blocks of keys and values. This selection happens quickly and cheaply. Then the Main Branch performs full accurate attention, but only over the selected blocks — not the entire context.

Importantly, the block-sparse structure of MSA is compatible with existing optimized CUDA kernels. The team did not need to create custom GPU implementations: the mechanism works efficiently on standard hardware.

Training Scale and Results

MSA was trained as part of a 109-billion MoE model (Mixture of Experts) with a total token budget of 3 trillion tokens. In scale, this is one of the largest publicly documented experiments with sparse attention mechanisms.

Key results:

• Reduction in computational load per token — 28.4 times for a 1 million token context compared to full GQA
• Quality on standard downstream benchmarks — comparable to baseline GQA, degradation is negligible
• Architectural compatibility: MSA integrates on top of existing GQA without reworking the rest of the model
• Works on standard CUDA kernels without custom implementations

"MSA preserves GQA accuracy while simultaneously reducing computational costs of attention 28 times for a million-token context," — from

MiniMax's technical report.

Why Long Context is Hard to Scale

Processing long sequences is a longstanding problem for Transformer architectures. Memory for the KV cache grows linearly with context length, while the number of attention operations grows quadratically. For a context of 1 million tokens, this means trillions of query-key pairs per model layer.

Most existing solutions make a trade-off: they truncate context, use a sliding window, or switch to linearized attention approximations, losing accuracy. Alternative architectures like Mamba fundamentally change the model structure, which complicates weight transfer and compatibility with Transformer infrastructure.

MSA offers a different path: keep the Main Branch exact (without approximations), and delegate the task of selecting relevant blocks to a lightweight Index Branch. This division of labor is exactly what provides 28x speedup without noticeable quality loss.

What This Means

If results reproduce in production conditions, MSA could become a practical way to achieve cheap long contexts — for working with large documents, codebases, and multimodal inputs. MiniMax has disclosed the technical details of the method, allowing other teams to adapt the approach to their own scales.