OpenAI unveils MRC, a networking protocol for training AI clusters with 100,000 GPUs

OpenAI has detailed MRC, a new networking protocol for the supercomputers that train large AI models. The company says it is already deployed across all of its largest NVIDIA GB200 clusters, including the OCI site in Abilene and Microsoft's Fairwater systems, and that it keeps performance steady even when links and switches fail.

Why MRC was needed

Training frontier models is constrained not only by the GPUs themselves but also by the network that connects them. Every training step involves millions of data exchanges, and if even one packet or one flow arrives noticeably later than the rest, some accelerators sit idle. Small clusters can still tolerate these delays, but at Stargate scale the problem becomes systemic: the more nodes you have, the higher the odds of congestion, latency jitter, and hardware failures.

For OpenAI, this is no longer a side engineering problem. The company says more than 900 million people use ChatGPT every week, which turns compute infrastructure into a foundational layer of the service. So over the past two years, the team has worked with AMD, Broadcom, Intel, Microsoft, and NVIDIA to rebuild its networking stack to deliver not just high speed, but predictable behavior under load and during partial failures.

How the network works

The core idea behind MRC is to stop treating a network interface as one big 800 Gbps pipe. Instead, OpenAI splits it into several smaller links — for example, eight 100 Gbps lanes, each running to its own switch. The result is a multi-plane network in which the same traffic can travel over many independent paths. OpenAI estimates that this configuration can support a network of roughly 131,000 GPUs with just two tiers of Ethernet switches, whereas a conventional 800 Gbps design would need three or four tiers.

That's where the protocol itself comes in, extending the familiar RoCE stack for AI training workloads. Rather than sending an entire transfer down a single route, MRC "sprays" the packets of one transfer across hundreds of paths at once. Packets may arrive out of order, but that's fine: each one carries its final memory address, so the receiver places the data where it belongs as it arrives. This lets the network use available links more evenly and handle local congestion much better.

• A single transfer is split across many parallel paths through different network planes
• When it detects signs of congestion, the protocol drops the problematic path and replaces it with another
• When a packet is lost, MRC quickly assumes a failure and resends the data
• If a packet is lost due to congestion on the receiver side, packet trimming helps: only the header is forwarded, as an explicit request for retransmission

OpenAI specifically highlights that MRC can route around network failures on a microsecond scale, whereas a traditional fabric could take seconds or even tens of seconds to reconverge. This matters most for synchronous training, where each compute step is gated not by the average transfer but by the slowest one in the cluster. With this kind of balancing, multiple jobs can share a single cluster with less risk of interfering with one another.

What changes in operations

Another major shift is dropping conventional dynamic routing in favor of source routing based on SRv6. In a typical network, switches recompute routes themselves using protocols such as BGP, which adds complexity and new classes of failures. With MRC, the sender encodes the packet's path in the IPv6 address, and switches simply execute that route hop by hop using static tables. The idea sounds radical, but OpenAI says it is exactly what simplifies the control plane and removes the need to constantly fix network logic by hand.

For OpenAI, practice matters more than theory, and here the company has concrete numbers. It says its training networks consist of millions of links, and in real runs, multiple brief link flaps between tier-0 and tier-1 switches can occur every minute — with no measurable impact on synchronous pretraining. During training of one of the recent frontier models for ChatGPT and Codex, engineers had to reboot four tier-1 switches, and doing so required no coordination with the teams running the training at the time. If one port fails on an eight-port network interface, bandwidth drops by at most one-eighth, but the job keeps running instead of crashing entirely.

What it means

MRC shows that the race for stronger models is increasingly shifting toward infrastructure. OpenAI isn't just speeding up training on its own clusters; it is also contributing the protocol to the Open Compute Project in an attempt to turn its in-house engineering into an industry standard. If other labs and cloud providers adopt the approach, large AI clusters will become cheaper, easier to operate, and more resilient to failures without constant manual network tuning.