Free Intensive Nov 21–22Free 2-day hands-on Vibecoding Intensive on Nov 21–22 at 16:00 MSK (Sat–Sun) · Live Cohort 3 starts Nov 23Join Free →Cohort 3 (Nov 23)
ReleaseInsight · On RadarHabr AI→ original

Wildberries cut its AI assistant's response time from 10.9 s to 3.6 s with DFlash 2

Each message to Wildberries' AI assistant triggers about 26 LLM calls, and the user waits for the entire chain. The team adopted DFlash 2: a small draft model guesses tokens ahead, and the large model verifies them in one pass. In production, the median message processing time dropped from 10.9 s to 3.6 s, although more than decoding changed. For short JSON responses, parallel drafters come out ahead.

AI-processed from Habr AI; edited by Hamidun News
Wildberries cut its AI assistant's response time from 10.9 s to 3.6 s with DFlash 2
Source: Habr AI. Collage: Hamidun News.
Listen to article
Article Fact Sheet & Key Figures
ParameterDetails
Primary sourceHabr AI ↗
Editor-in-Chief
Key metric #1In the testbed, the top methods doubled step speed, and at 128 concurrent users they were 1.4x faster
Key metric #2EAGLE-3 is 11–23% slower than DFlash because its draft is built one token at a time
Key metric #3The model's own MTP head, fine-tuned on the domain, delivers ×1.7–1.8 versus ×2

When you message an AI assistant in a store, a single message kicks off a whole chain of language model calls. The Wildberries shopper assistant averages around 26 of them: it parses context, extracts preferences, generates search queries, selects filters, and only then writes the reply. Some calls run in parallel, but the user waits for the entire chain.

To shorten that wait, the team moved the assistant to speculative decoding. The idea is simple: a small draft model (the drafter) guesses several next tokens ahead, and the large model checks the guess in one pass. In production, after switching to DFlash 2, the median message processing time fell from 10.

9 to 3.6 seconds. But first, Wildberries compared seven methods on a single testbed.

Why checking guesses pays off

Generating text with a large model is limited not by compute but by memory. To produce one token, the GPU reads all the weights involved, yet it does very little arithmetic. If you feed in eight tokens at once, the pass costs almost the same: the weights are read once.

So the drafter proposes a continuation, and the large model accepts the longest correct prefix. At the first mismatch, it substitutes its own token. With correct verification, the target model's output distribution does not change. A weak drafter does not corrupt the text; it simply guesses right less often. The speedup is not guaranteed. It depends on the cost of drafting and verification and on the share of accepted tokens. Under heavy load the gain shrinks: each step becomes compute-bound rather than memory-bound.

The assistant's profile is unusual. An average call is a prompt of about 5,000 tokens and a response of about 45 tokens, most often JSON following a schema. The responses are short but numerous, and they sit on the critical path. The production model is a MoE in FP8 with hybrid attention: some layers use linear attention, others use standard full attention.

What the testbed showed

You cannot measure on production: the assistant calls adjacent systems that have strict RPS limits. So the team built an isolated testbed with the same code, the same model, and the same vLLM flags. The run covered 449 turns from 300 sessions that were not in the training data. They measured the time of a whole turn under load, not tokens per second, because that is what the user actually sees. The same method varied from run to run by up to 9% at low load and up to 4.5% at 128 users. The authors treat methods whose differences fall within that spread as equal.

The results:

  • The best performers are the parallel drafters DFlash, DSpark, and DFlash 2. At low load, their turns run about twice as fast; at 128 concurrent users, they are 1.4x faster. Among themselves, they fall within the noise.
  • The model's own MTP head, fine-tuned on the domain, delivers ×1.7–1.8 at low load.
  • EAGLE-3 accepts nearly as many tokens per step as DFlash but is 11–23% slower. Its draft is built sequentially, while a parallel drafter produces its draft in a single pass.
  • Off-the-shelf drafters work without training. On an open model, its MTP head and the public DFlash speed up turns by 1.6–2x at low load and by 1.2–1.3x at 128 users.

The drafters were trained on responses from the same model that runs in production, so they predict that model's continuations rather than generically correct text. Each node had 6 GPUs running vLLM servers with the large model, and 2 more GPUs training the drafter. The pilot budget was 847 thousand rows and 39 million response tokens.

Where production is easy to break

The most practical finding concerns methods that need no training. In vLLM 0.30, n-gram and suffix decoding break responses on a hybrid model with linear attention. The authors identified the pattern and ran a control experiment. The cause: linear attention keeps a single fixed-size state, and the contribution of rejected tokens cannot be precisely and cheaply removed from it. The vLLM code paths for such layers are younger than the path for the standard KV cache. There, the team found two limitations: one breaks responses, and the other prevents enabling adaptive verification.

There is also an honest caveat. The drop in the median from 10.9 to 3.6 s is the result of the production switch, not a clean experiment: more than decoding changed.

The practical takeaway for teams with similar systems: if you run a hybrid model on vLLM, compare the share of malformed JSON for the baseline and for each method. Correct speculation must not change it. For short structured responses, look at parallel drafters. It is better to train your own drafter or MTP head on your own model's responses.

Story

Key Questions Answered

Why checking guesses pays off?

Generating text with a large model is limited not by compute but by memory. To produce one token, the GPU reads all the weights involved, yet it does very little arithmetic. If you feed in eight tokens at once, the pass costs almost the same: the weights are read once. So the drafter proposes a continuation, and the large model accepts the longest correct prefix. At the first mismatch, it substitutes its own token. With correct verification, the target model's output distribution does not change.

What the testbed showed?

You cannot measure on production: the assistant calls adjacent systems that have strict RPS limits. So the team built an isolated testbed with the same code, the same model, and the same vLLM flags. The run covered 449 turns from 300 sessions that were not in the training data. They measured the time of a whole turn under load, not tokens per second, because that is what the user actually sees. The same method varied from run to run by up to 9% at low load and up to 4.5% at 128 users.

Where production is easy to break?

The most practical finding concerns methods that need no training. In vLLM 0.30, n-gram and suffix decoding break responses on a hybrid model with linear attention. The authors identified the pattern and ran a control experiment. The cause: linear attention keeps a single fixed-size state, and the contribution of rejected tokens cannot be precisely and cheaply removed from it. The vLLM code paths for such layers are younger than the path for the standard KV cache.

ZK
Zhemal Khamidun — CPO at Alpina Digital, product manager of AlpinaGPT. Building AI products since 2019. Hamidun News is a pet project that grew into a production aggregator with 6,000+ readers.
✦ Unified AI Workspace · Nomad

Test Frontier LLMs & Autonomous Agents in One Workspace

Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.

Try Nomad AI →Launch Nomad Agent
What do you think?
Loading comments…