Xiaomi MiMo Moves to Encoder-Decoder: 5x Less Prefill at 1M Context

Long context in LLMs has long since stopped being a feature and become a hardware bill. Load a million tokens into a model and you pay twice: first in time and compute for prefill, then in memory for the KV cache, which balloons along with the context. Now Xiaomi, according to SemiAnalysis, is trying to solve this at the architecture level.

What exactly is changing

SemiAnalysis analysts write that the next generation of Xiaomi MiMo models is moving to a causal encoder-decoder. This comes after DeepSeek V4.1 Flash chose a similar path. The architecture is called HySparse2, and the key details in the post are these: the prompt passes through just 25 of 49 layers, and the decoder's KV cache is not computed from scratch but projected from the encoder's states.

In a conventional decoder model, every layer processes every token of the prompt and stores its own keys and values. Here the scheme is different: the encoder takes on the heavy work of reading the long input, while the decoder receives ready-made states from it and builds its own KV from them.

Imagine a lawyer handed a 500-page contract. You could make them reread the whole thing every time they answer a question. Or they could write a summary once and work from it afterward. The analogy is crude, but it captures the point: the expensive reading of the input is moved to a separate stage, handled by only some of the layers.

The numbers

At a 1M-token context, compared with MiMo-V2, SemiAnalysis cites two results:

• 5x less compute for prefill;
• a 4.5x smaller KV cache.

The first figure means time to first token on huge prompts could drop noticeably, and the cost of processing input will fall. The second hits memory: a smaller cache means more concurrent requests on the same GPU, or long context on a more modest configuration.

It is important to understand where the comparison comes from. It is a comparison with Xiaomi's own previous model, not an independent benchmark against competitors. And the post contains no quality data: how the new architecture holds up in accuracy on long context is not yet known.

Why it matters

The main signal here is the trend. Two prominent Chinese labs in a row are moving toward a scheme in which input and generation are handled differently. For an industry that lived for years in a "decoder-only" paradigm, this is a notable shift. Long-context inference efficiency is becoming a competitive advantage on par with quality.

For those building products on long context — RAG without aggressive chunking, codebase analysis, agents with long memory — this is a potential reduction in the cost of scenarios that currently run up against memory and latency.

Limitations and what to do

First, what we don't know. The post has no data on quality, on how the model behaves on long-context tasks, on when exactly the models will be released, or whether the weights will be open. The figures are given for 1M tokens; on short prompts the gain may be far more modest. The source is an analysts' post, not a technical report from Xiaomi.

Practical takeaways for developers and product managers:

• don't rewrite your stack for the new architecture until there are open weights and independent measurements;
• if you are modeling the economics of long context, include a scenario in which the cost of prefill and memory drops several-fold;
• watch for encoder-decoder support in inference engines: how quickly the gains reach real products depends on it.

If the claimed 5x and 4.5x are confirmed in practice without loss of quality, this will be one of the most tangible steps toward a cheap million tokens.