How Distributed AI Training Is Reshaping Inter-Datacenter Networks

Training large language models has long outgrown a single datacenter. Google trained Gemini synchronously across clusters in different geographic locations. Microsoft linked AI datacenters in Wisconsin and Georgia into a single distributed supercomputer. AWS connected AI compute clusters over long distances for Anthropic, which develops Claude. Meta built high-performance inter-datacenter links to support model training. CoreWeave and Google Cloud announced cross-cloud training over a private interconnect; Azure plans to follow later this year.

This geographic dispersion stems from the limits of a single datacenter or campus cluster, which can't meet power and cooling demands. Cisco estimates that training a modern model requires clusters with tens of thousands of GPUs. By 2030, the largest frontier training runs will consume 4–16 GW of power, according to researchers at Epoch AI. Distributing compute lets companies build in locations with more available power, space, and fewer permitting constraints.

Synchronization as the critical bottleneck

Training a neural network is a cyclical repetition of compute and synchronization. In each cycle, thousands of accelerators (GPUs, AWS Trainium, Google TPUs) must exchange results and synchronize before the next stage. The networks aren't carrying text or images for training — they're carrying large arrays of numerical data: gradients and intermediate results.

When thousands of accelerators finish their compute phase simultaneously and start communicating, incast problems arise: many senders converge on one receiver, and traffic arrives faster than the link can handle. The fundamental problem is synchronization itself: in a synchronous training job, the slowest group of GPUs blocks all the others. Network delay becomes compute delay. Ramesh Sivakolundu, a Cisco Silicon One architect, explains: "Any training job is a repetition of compute and synchronization. Compute weights, communicate them to synchronize every GPU, then restart compute. Bursty traffic happens periodically, and the entire network needs to complete the exchange before the next stage."

Coherent optics and deep buffers

Inside a single datacenter, full network connectivity between GPU racks is achieved through engineered bandwidth. But for scale-across (distributed training), traffic to another datacenter narrows dramatically — nearly 14 times less than a typical DCI baseline, according to Cisco estimates. A synchronized burst can turn that narrow pipe into a bottleneck.

Traditional DCI connects independent datacenters for redundancy and workload distribution; its traffic is asynchronous — one flow doesn't need to finish in lockstep with others. In scale-across AI training, the inter-site network becomes part of a single coordinated computation, requiring sufficient bandwidth and predictable delivery.

Bandwidth scales with distance. Inside a rack, speeds reach up to 800 Gb/s and 1.6 Tb/s over short copper links. At the facility scale, pluggable optics are used for scale-out links spanning hundreds of meters to about 2 km. Scale-across extends this to metro or regional distances, creating new requirements.

Short-range datacenter optics can't handle 400–800 Gbps over hundreds of kilometers. Coherent optics are needed: sophisticated modulation and digital signal processing (DSP) compensate for physical distortion over long distances. The required port count is enormous: connecting two 100-megawatt AI sites needs 12,000–32,000 coherent optical ports (versus 1,000–2,000 for a typical DCI between comparable facilities). At 400–800 Gb/s per port, that yields multi-petabit aggregate capacity.

A smart solution is coherent pluggables: DWDM wavelengths are generated directly in router ports, avoiding a separate bank of DWDM transponders. Cisco's Optical Transport 3000 with a multi-rail design cuts power per rail by 75% and rack space by 80%. Where a standalone transport system is needed, the NCS 1014 delivers 12.8 Tb/s from a 1RU line card.

Latency, buffering, and congestion management

Over long distances, congestion feedback arrives slower. On an 800 Gb/s link spanning 100 km, roughly 100 MB of data is already in flight by the time the sender learns about a problem. Shallow-buffer architectures designed for low latency inside a datacenter aren't up to the task: deep buffers are required.

Deep buffering absorbs temporary bursts while traffic management and congestion signaling resolve the issue. A shared buffer is a single memory pool dynamically allocated across ports, letting a busy link absorb larger bursts without packet loss.

Cisco uses the Silicon One P200 (a 51.2 Tb/s programmable routing processor) in its 8223 and N9000 platforms, paired with 400G and 800G coherent pluggables for long-haul links. The P200 features a programmable network processor and P4 tooling, allowing features to be added in software without changing the silicon. For scale-across, where topology and traffic management are still experimental, that flexibility is critical.

Cisco's strategy is co-design: the networking ASIC is developed years ahead of the finished router, accounting for the predictable collective communication patterns in AI training. Proactive congestion management steers or schedules traffic ahead of predictable bursts, while deep buffering serves as a safety net for transient congestion and link failures. The two approaches are complementary: one avoids queuing altogether, the other gives the network room for traffic when a queue forms anyway. Hardware-based encryption is integrated as well, avoiding an additional processing bottleneck on the training fabric.