UltraEP от Xiaohongshu: балансировка MoE-экспертов за 300 мкс, 94,3% от идеала

The dotsinfra team at Xiaohongshu (小红书, "Xiaohongshu") has introduced UltraEP — a runtime that balances the load of MoE model experts in real time on rack-scale systems and achieves 94.3% of ideal training and inference throughput. The paper was published on arXiv on June 2, 2026.

What problem does UltraEP solve

UltraEP eliminates the load imbalance between experts in MoE models that leaves expensive rack-scale GPU systems idle. In rack-scale nodes such as the Huawei Atlas 900 A3 SuperPoD and NVIDIA NVL72, dozens of GPUs are joined into a single high-speed interconnect domain, with model experts distributed across them (Expert Parallelism). Tokens select experts unevenly: some GPUs become overloaded and turn into "compute stragglers," causing bottlenecks in all-to-all communication and spikes in activation memory.

Existing balancers reshuffle experts periodically, relying on historical load data. According to the authors, in real production deployments with a non-stationary load pattern this approach is unreliable — the imbalance between ranks reaches up to 4.01× relative to the average, and some cards in the rack sit idle while overloaded GPUs slow down the entire step.

How balancing works in 300 microseconds

UltraEP rebalances every microbatch and every layer right on the critical compute path, rather than once every N steps. The system reacts to load immediately after gating using a quota-driven scheduler and performs irregular transfers of expert states via persistent tile streaming, which is "native" to rack-scale nodes, with a relay mechanism against fan-out overload. The additional overhead is about 300 microseconds per rebalancing.

• 94.3% of ideal (force-balanced) throughput, averaged across training and inference
• 1.49× speedup compared to running without balancing
• Load imbalance between ranks reduced from 1.30–4.01 to 1.01–1.04
• Rebalancing overhead — about 300 microseconds
• Testing — multi-RSN deployment of up to 256 GPUs
• Models from 106 to 671 billion parameters; arXiv 2606.04101, submitted June 2, 2026

"UltraEP is the first balancer with precise real-time load accounting for training and prefill inference of large

MoE models on rack-scale nodes," the dotsinfra team writes in the paper on arXiv.

Why this matters for MoE training

UltraEP moves expert balancing from a "periodic" plane into real time, recovering up to 1.49× throughput on the same hardware. For models at the scale of 671 billion parameters (the scale of DeepSeek-V3), this means racks of dozens of GPUs stop sitting idle because of "hot" experts that receive a disproportionate share of tokens. The key to the speed is the rack-scale topology itself: the extended connectivity of dozens of GPUs within a node allows experts to be shuffled between cards faster than classic periodic balancers can manage. UltraEP is implemented as a standalone runtime and, according to the authors, integrates into existing training and inference stacks without rewriting them.

What this means

Load balancing is becoming the next front in the battle for MoE efficiency — after throughput and communication speed. UltraEP shows that, given rack-scale node connectivity, experts can be reshuffled literally at every step, squeezing near-ideal utilization out of hardware worth millions of dollars.

Frequently asked questions

Which models was UltraEP tested on?

UltraEP was tested on state-of-the-art MoE models ranging from 106 to 671 billion parameters in a multi-RSN deployment of up to 256 GPUs, averaging results across training and prefill inference scenarios.

How much does UltraEP speed up MoE?

According to the paper, UltraEP delivers a 1.49× throughput gain relative to running without balancing and reaches 94.3% of the theoretical maximum under ideally balanced load.

What are rack-scale systems?

These are rack-scale nodes such as the Huawei Atlas 900 A3 SuperPoD and NVIDIA NVL72, where dozens of GPUs are joined into a single high-speed interconnect domain. It is precisely their extended connectivity that lets UltraEP transfer expert states between cards in microseconds.