Wildberries cut its AI assistant's response time from 10.9 s to 3.6 s with DFlash 2
Each message to Wildberries' AI assistant triggers about 26 LLM calls, and the user waits for the entire chain. The team adopted DFlash 2: a small draft model guesses tokens ahead, and the large model verifies them in one pass. In production, the median message processing time dropped from 10.9 s to 3.6 s, although more than decoding changed. For short JSON responses, parallel drafters come out ahead.
AI-processed from Habr AI; edited by Hamidun News
When you message an AI assistant in a store, a single message kicks off a whole chain of language model calls. The Wildberries shopper assistant averages around 26 of them: it parses context, extracts preferences, generates search queries, selects filters, and only then writes the reply. Some calls run in parallel, but the user waits for the entire chain.
To shorten that wait, the team moved the assistant to speculative decoding. The idea is simple: a small draft model (the drafter) guesses several next tokens ahead, and the large model checks the guess in one pass. In production, after switching to DFlash 2, the median message processing time fell from 10.
9 to 3.6 seconds. But first, Wildberries compared seven methods on a single testbed.
Why checking guesses pays off
Generating text with a large model is limited not by compute but by memory. To produce one token, the GPU reads all the weights involved, yet it does very little arithmetic. If you feed in eight tokens at once, the pass costs almost the same: the weights are read once.
So the drafter proposes a continuation, and the large model accepts the longest correct prefix. At the first mismatch, it substitutes its own token. With correct verification, the target model's output distribution does not change. A weak drafter does not corrupt the text; it simply guesses right less often. The speedup is not guaranteed. It depends on the cost of drafting and verification and on the share of accepted tokens. Under heavy load the gain shrinks: each step becomes compute-bound rather than memory-bound.
The assistant's profile is unusual. An average call is a prompt of about 5,000 tokens and a response of about 45 tokens, most often JSON following a schema. The responses are short but numerous, and they sit on the critical path. The production model is a MoE in FP8 with hybrid attention: some layers use linear attention, others use standard full attention.
What the testbed showed
You cannot measure on production: the assistant calls adjacent systems that have strict RPS limits. So the team built an isolated testbed with the same code, the same model, and the same vLLM flags. The run covered 449 turns from 300 sessions that were not in the training data. They measured the time of a whole turn under load, not tokens per second, because that is what the user actually sees. The same method varied from run to run by up to 9% at low load and up to 4.5% at 128 users. The authors treat methods whose differences fall within that spread as equal.
The results:
- The best performers are the parallel drafters DFlash, DSpark, and DFlash 2. At low load, their turns run about twice as fast; at 128 concurrent users, they are 1.4x faster. Among themselves, they fall within the noise.
- The model's own MTP head, fine-tuned on the domain, delivers ×1.7–1.8 at low load.
- EAGLE-3 accepts nearly as many tokens per step as DFlash but is 11–23% slower. Its draft is built sequentially, while a parallel drafter produces its draft in a single pass.
- Off-the-shelf drafters work without training. On an open model, its MTP head and the public DFlash speed up turns by 1.6–2x at low load and by 1.2–1.3x at 128 users.
The drafters were trained on responses from the same model that runs in production, so they predict that model's continuations rather than generically correct text. Each node had 6 GPUs running vLLM servers with the large model, and 2 more GPUs training the drafter. The pilot budget was 847 thousand rows and 39 million response tokens.
Where production is easy to break
The most practical finding concerns methods that need no training. In vLLM 0.30, n-gram and suffix decoding break responses on a hybrid model with linear attention. The authors identified the pattern and ran a control experiment. The cause: linear attention keeps a single fixed-size state, and the contribution of rejected tokens cannot be precisely and cheaply removed from it. The vLLM code paths for such layers are younger than the path for the standard KV cache. There, the team found two limitations: one breaks responses, and the other prevents enabling adaptive verification.
There is also an honest caveat. The drop in the median from 10.9 to 3.6 s is the result of the production switch, not a clean experiment: more than decoding changed.
The practical takeaway for teams with similar systems: if you run a hybrid model on vLLM, compare the share of malformed JSON for the baseline and for each method. Correct speculation must not change it. For short structured responses, look at parallel drafters. It is better to train your own drafter or MTP head on your own model's responses.
- Nov 21, 2026Open-Weight Inkling 2 Mini Matches Qwen3.8-27B on Independent Benchmark, Runs on 16 GB of Memory
- Nov 21, 2026A Developer Had Claude Install 900 Apps and Built a Free Mac Cleaner
- Nov 21, 2026Qwen3-0.6B Fine-Tuned on a Mac mini: Errors Dropped to 3 out of 30 from 9
- Nov 21, 2026The Programmer Stopped Being an AI Agent Dispatcher: Multica Takes Over the Handoff Routine
Key Questions Answered
Why checking guesses pays off?
Generating text with a large model is limited not by compute but by memory. To produce one token, the GPU reads all the weights involved, yet it does very little arithmetic. If you feed in eight tokens at once, the pass costs almost the same: the weights are read once. So the drafter proposes a continuation, and the large model accepts the longest correct prefix. At the first mismatch, it substitutes its own token. With correct verification, the target model's output distribution does not change.
What the testbed showed?
You cannot measure on production: the assistant calls adjacent systems that have strict RPS limits. So the team built an isolated testbed with the same code, the same model, and the same vLLM flags. The run covered 449 turns from 300 sessions that were not in the training data. They measured the time of a whole turn under load, not tokens per second, because that is what the user actually sees. The same method varied from run to run by up to 9% at low load and up to 4.5% at 128 users.
Where production is easy to break?
The most practical finding concerns methods that need no training. In vLLM 0.30, n-gram and suffix decoding break responses on a hybrid model with linear attention. The authors identified the pattern and ran a control experiment. The cause: linear attention keeps a single fixed-size state, and the contribution of rejected tokens cannot be precisely and cheaply removed from it. The vLLM code paths for such layers are younger than the path for the standard KV cache.
Test Frontier LLMs & Autonomous Agents in One Workspace
Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.
The AI world, distilled — once a week
Seven stories that actually mattered, hand-picked. No noise, no reposts, no press releases.
Done! Check your inbox for a confirmation.