Xiaohongshu speeds up its sparse-attention indexer: DoSA halves TTFT

Chinese social network Xiaohongshu (RedNote) has gone after a bottleneck that appears once sparse attention is in place. Its Dots team developed DoSA: in tests, the method filtered out about 97% of candidates in the indexer ahead of time, left the final Top-K unchanged and, according to the headline of the Jiqizhixin publication, cut time to first token (TTFT) in half at a context of roughly 1M tokens.

Where the bottleneck hides

Sparse attention reduces computation by having each query look not at all previous tokens but only at the K most relevant ones. A separate lightweight module, the indexer, selects them. It scores every query–token pair and picks the best. That makes the main attention cheap, but the indexer still scans the entire context. Its cost grows quadratically with sequence length, and at a million tokens it is the indexer that starts to determine prompt processing time. This design is used, among others, by the DeepSeek V3.2 family of models with their "lightning" indexer.

The problem is most noticeable at the first token. During prefill, the indexer has to go through the whole long prompt, and the user waits longer for a response than they would like.

How DoSA works

The full text of the source article is truncated on our end, so we give implementation details as described by the authors. For each candidate, DoSA computes an upper and a lower bound on its score. This is a cheap operation compared with exact calculation. The pruning logic is simple: if a candidate's upper bound is below a lower bound that already guarantees a place in the top K, that candidate cannot make the Top-K. There is no need to compute its exact score.

The key property is that the method does not approximate the result: it produces the same set of tokens as a full brute-force pass. That is why the slogan "提速不降智" ("faster without losing intelligence") fits here: model quality should not depend on the speedup, because token selection does not change.

What the tests show

• About 97% of candidates were pruned before exact scores were computed.
• The Top-K result matches the original algorithm.
• The context in the tests was about 1M tokens.
• Time to first token dropped by roughly half (per the publication's headline).

Some data is missing from the available fragment: the model name (the text cuts off at "Dots3…"), the hardware, the batch size, absolute latency values and the breakdown of the gain by stage. The "halved" figure applies to a specific configuration and should not be carried over to other stacks without testing. The share of filtered candidates likely depends on the data: the sharper the score distribution, the more effective the bounds.

The practical side

For teams serving long-context models with sparse attention, DoSA is interesting because it speeds up inference without retraining and without the risk of changing model behavior. Since token selection stays the same, quality regression tests should pass unchanged. That is a strong argument for production, where any deviation in output requires separate verification.

The first step is to measure what share of prefill time the indexer takes in your stack. If that share is noticeable at 128K–1M tokens, the method will pay off. If most of the time goes to the MoE part or to data transfer between GPUs, the effect will be more modest. It is also worth waiting for open-source code or a technical report: at the CUDA kernel level, the overhead of computing the bounds could change the overall picture considerably.

For Xiaohongshu itself, this is also a question of economics. The cheaper it is to process long prompts, the easier it is to keep large-context scenarios in the product: feed search, document analysis, agent chains.