Synchronous MoE decoding makes GPUs wait for the busiest group

Adding GPUs to an LLM service cluster and getting less of a throughput gain than expected is a familiar picture in large online deployments. Jiqizhixin reports that in setups for MoE models like DeepSeek, the culprit is waiting. Several groups of cards process their own requests in parallel, but they finish each decoding step together, through inter-group communication. The step runs in lockstep: until the busiest group finishes computing, the others stand still. The available excerpt of the article contains no concrete numbers and no description of the method itself. What follows is a breakdown of the mechanics of the problem, based on what the excerpt does say.

Why a step takes exactly as long as the slowest group

In MoE deployments, work is distributed across groups of cards. Each group has its own pool of requests, and the load differs from group to group: some handle long conversations with heavy context, others short exchanges. But decoding does not reduce to independent computations. At each step, the groups need to exchange data to get through the model's layers, including routing tokens to experts that sit on different cards.

This exchange works as a barrier. A group that finishes quickly cannot move ahead and has to wait. Step time is set by the maximum across groups, not the average. If load variance grows along with the number of groups, the chance of hitting one badly overloaded node grows too. There are more cards, yet the share of time they spend idle increases.

Where the gains disappear

The problem is not a shortage of compute. Expensive accelerators wait on their neighbors, and the bill for them runs at the full rate. On paper, utilization looks high; in practice, part of the time goes to waiting.

The description points to the following properties of the situation:

• Mode: online serving with a large number of concurrent requests.
• Architecture: an MoE model deployed in parallel across several groups of cards.
• Bottleneck: a synchronous decoding step with inter-group communication.
• Symptom: throughput grows more slowly than the number of cards.
• Main source of loss: load imbalance between groups, which forces the fast ones to wait for the slow ones.

I infer the last point from the logic of the setup, not from a quotation. But the phrase "until the busiest group has finished computing, the others wait" has no other plausible explanation.

What is known about the solution

The headline promises a "new solution" to waiting in inference. The available excerpt covers only the problem statement. It is unknown which method the authors propose, which models and hardware they tested it on, and what speedup they achieved. Any speedup figures without the full text would be invented, so there are none here.

The general classes of solutions to problems like this are well known: balancing load across groups, smarter distribution of requests among them, and overlapping computation with communication. Which of these the work uses cannot be said.

Practical takeaway

A team running its own MoE inference should check what is actually blocking scaling. First, measure the spread of step times across card groups, not just total throughput. If the slowest group lags noticeably behind the average, it is too early to buy more GPUs: some of them will sit idle. Next, look at how requests are distributed across groups, because context length and the number of active requests strongly affect step time. Only after that does it make sense to calculate token cost at different card counts.

Read the full text of the study before porting the method to production. For now, all it tells us is where to look for the losses.