Xiaohongshu's UltraEP Balances MoE Experts in 300 µs, Reaching 94.3% of Ideal

The dots infra team at Xiaohongshu (小红书, "Xiaohongshu") has unveiled UltraEP, a runtime that balances MoE expert load in real time on rack-scale systems and reaches 94.3% of ideal training and inference throughput. The paper was published on arXiv on June 2, 2026.

What problem does UltraEP solve

UltraEP eliminates load skew across MoE experts, which leaves expensive rack-scale GPU systems idle. In rack-scale nodes such as the Huawei Atlas 900 A3 SuperPoD and NVIDIA NVL72, dozens of GPUs are joined into a single high-speed interconnect domain, and the model's experts are distributed across them (Expert Parallelism). Tokens pick experts unevenly: some GPUs are overloaded and become compute stragglers, all-to-all communication jams up, and activation memory spikes.

Existing balancers rearrange experts periodically, based on historical load. According to the authors, this approach is unreliable in real production deployments with non-stationary load patterns: skew between ranks reaches 4.01× relative to the mean, and part of the rack's cards sit idle while overloaded GPUs slow down the entire step.

How balancing in 300 microseconds works

UltraEP rebalances every microbatch and every layer directly on the critical compute path, rather than once every N steps. The system reacts to load immediately after gating using a quota-driven scheduler, and it carries out irregular transfers of expert state through persistent tile streaming native to rack-scale nodes, with a relay mechanism to prevent fan-out overload. The additional overhead is about 300 microseconds per rebalance.

• 94.3% of ideal (force-balanced) throughput, averaged across training and inference
• 1.49× speedup compared with running without balancing
• Load skew between ranks reduced from 1.30–4.01 to 1.01–1.04
• Rebalancing overhead of about 300 microseconds
• Tests on multi-RSN deployments of up to 256 GPUs
• Models from 106 to 671 billion parameters; arXiv 2606.04101, submitted on June 2, 2026

"UltraEP is the first real-time, load-exact balancer for training and prefill inference of large

MoE models on rack-scale nodes," the dots infra team writes in its arXiv paper.

Why this matters for MoE training

UltraEP moves expert balancing from a "periodic" regime to real time, recovering up to 1.49× throughput on the same hardware. For models at the 671-billion-parameter scale (the scale of DeepSeek-V3), this means racks of dozens of GPUs stop idling because of "hot" experts that receive a disproportionate share of tokens. The key to the speed is the rack-scale topology itself: the extended connectivity of dozens of GPUs within a node lets experts be shuffled between cards faster than classic periodic balancers can manage. UltraEP is implemented as a standalone runtime and, according to the authors, plugs into existing training and inference stacks without rewrites.

What this means

Load balancing is becoming the next front in the fight for MoE efficiency, after bandwidth and communication speed. UltraEP shows that, given the connectivity of rack-scale nodes, experts can be reshuffled literally on every step, squeezing near-ideal utilization out of hardware that costs millions of dollars.

Frequently asked questions

What models was UltraEP tested on?

UltraEP was tested on frontier MoE models from 106 to 671 billion parameters in multi-RSN deployments of up to 256 GPUs, with results averaged across training and prefill-inference scenarios.

How much does UltraEP speed up MoE?

According to the paper, UltraEP delivers a 1.49× throughput gain relative to running without balancing and reaches 94.3% of the theoretical maximum under a perfectly balanced load.

What are rack-scale systems?

These are rack-level nodes such as the Huawei Atlas 900 A3 SuperPoD and NVIDIA NVL72, in which dozens of GPUs are joined into a single high-speed interconnect domain. It is their extended connectivity that lets UltraEP move expert state between cards in microseconds.