Meta Reveals Prometheus: A Gigawatt Cluster That Trains Models With 85% Useful Time
Meta has shown the inside of Prometheus, a gigawatt-scale cluster in Ohio where it trained its Muse models. When one card fails in a training job spanning thousands of GPUs, the whole job stops, much like an assembly line halted by a single broken machine. That is why Meta tracks goodput: the share of time the cluster is actually training the model. For Muse Wonder, goodput exceeded 85% during pretraining and 90% during RL. The takeaway is that at this scale, spare capacity and failure handling matter more than the raw GPU count.
AI-processed from Meta Engineering; edited by Hamidun News
When people talk about the AI lab race, they usually count GPUs and megawatts. But large training jobs have an awkward property: they work like an assembly line, where stopping one machine slows the entire line. If one of thousands of servers breaks, the whole job suffers. Meta Engineering has published a breakdown of Prometheus, a supercluster of roughly one gigawatt in New Albany, Ohio, where it trained its new Muse model family. The key metric here is goodput, meaning the share of time the cluster is actually training the model rather than sitting idle or recovering from a failure. For Muse Wonder pretraining, goodput exceeded 85%; for reinforcement learning (RL) training, it reached 90%.
What the Machine Is and How It Is Built
Two years ago, the New Albany site was an ordinary data center serving web traffic. It was converted for training frontier models. Prometheus now consists of hundreds of thousands of GPUs spread across several connected buildings, including conventional facilities, buildings specifically optimized for AI, and quickly erected structures. Everything is linked by kilometers of fiber optic cable and operates as a single machine. The first phase is already complete, and at full scale the cluster is expected to grow to about 1 million GPUs. Cooling relies on a closed-loop liquid system.
The cluster currently runs two Nvidia platforms, with a third on the way. The first, Catalina, is built on the GB200 NVL72: 72 GPUs connected via NVLink into a single domain, 13.4 TB of HBM3e memory (192 GB per GPU), 720 petaflops in FP8, and about 120 kW per system. Meta chose the Ariel variant with two 400G network cards per GPU, or 800G per accelerator. The company argues that the GB200 improved on every parameter relative to the H100 except networking, and that this imbalance would have cost performance.
The second platform, Clemente, uses the GB300 NVL72: 288 GB of memory per GPU (50% more), over one exaflop in FP4, with all 72 GPUs fitting in a single rack. The 800G network per GPU comes standard here, there are fewer cables, and maintenance is simpler. Cooling is either modular air-liquid or factory-built liquid cooling if the building is new.
Networking: GPUs Have to Talk, Not Wait
A typical data center routes requests and responses between services that do not need tight synchronization. Training does need it, which is why Meta added a second, dedicated Ethernet network a few years ago for direct GPU-to-GPU communication. Within a building, each such fabric connects up to 20,736 GPUs. Meta first used the Disaggregated Scheduled Fabric, which separates large, long-lived data flows so they do not collide. When network cards learned to reorder packets on their own, the company moved to the Non-Scheduled Fabric with 51.2T switches.
Between buildings, an aggregation network runs on Arista 7800 chassis with Broadcom Jericho3 chips, with links ranging from 2 to over 100 km. At those distances the speed of light becomes a constraint, so Meta worked with its Superintelligence Labs to design a parallelism scheme that keeps latency-sensitive traffic inside a single model replica. The traffic that crosses between replicas is the kind that tolerates latency better. In total, the network comprises tens of thousands of switches and millions of optical modules.
Reliability: Spare GPUs and Analyzing Every Interruption
Moving from NVL8 to NVL72 increased the blast radius of a single failure ninefold: instead of eight GPUs in a connected domain, there are now 72. For that reason, the rack became the primary unit of planning. Training is easier to divide into power-of-two groups, so each rack runs 64 GPUs while the remaining eight serve as a hot spare. If a card fails, a spare takes its place, and the job restarts from the last checkpoint without waiting for a repair.
For rare failures that take out an entire rack, Meta also keeps about 2% of racks in reserve. A run on 32,768 active GPUs requires 523 NVL72 racks, or 37,656 GPUs, which is about 13% above active capacity. Nearly all of that 13% (11%) is simply the geometry of "64 out of 72" rather than a reliability tax.
Physical complexity grows as well: a single NVL72 domain contains 5,184 links at 200 Gbit/s, and each can fail at several points, including the GPU's SerDes, the compute tray board, cable cartridges, or switch interfaces.
Meta also describes its working model with Nvidia. Vendor engineers sit inside the data centers, exchange telemetry, and jointly trace each interruption to its root cause. Meta owns scheduling and infrastructure, while Nvidia owns firmware, signal diagnostics, and memory handling. This openness reduces disputes over responsibility and speeds up fixes.
The result: the mean time between interruptions (MTBI) across Muse Wonder runs was about 47K GPU-hours, and after firmware updates across the whole fleet it rose to more than 54K.
What It Means
For anyone building their own AI infrastructure, the lesson is simple: cluster size is only half the job. Spare capacity in every rack, fast restarts from checkpoints, and root-cause analysis of interruptions keep goodput above 85–90%. There is a caveat as well: Meta itself publishes all of these figures, and there is no independent verification. The logic still applies to smaller clusters, though. Calculate what share of paid GPU-hours goes to actual training, and fix the causes of downtime.
- Nov 22, 2026Wells Fargo and four other US banks stop taking responsibility for customers' AI agent mistakes
- Nov 22, 2026Yandex Grant Funds an LLM Engine: 35B Model on RTX 2080 Ti Runs Three Times Faster Than llama.cpp
- Nov 22, 2026Analysts estimate: by 2028, gas will supply 87% of Anthropic's power and coal 11%
- Nov 22, 2026AWS Quick gets a Meta Ads connector: campaigns and reports from a text prompt
Key Questions Answered
What the Machine Is and How It Is Built?
Two years ago, the New Albany site was an ordinary data center serving web traffic. It was converted for training frontier models. Prometheus now consists of hundreds of thousands of GPUs spread across several connected buildings, including conventional facilities, buildings specifically optimized for AI, and quickly erected structures. Everything is linked by kilometers of fiber optic cable and operates as a single machine.
What are the key details on Networking: GPUs Have to Talk, Not Wait?
A typical data center routes requests and responses between services that do not need tight synchronization. Training does need it, which is why Meta added a second, dedicated Ethernet network a few years ago for direct GPU-to-GPU communication. Within a building, each such fabric connects up to 20,736 GPUs. Meta first used the Disaggregated Scheduled Fabric, which separates large, long-lived data flows so they do not collide.
What are the key details on Reliability: Spare GPUs and Analyzing Every Interruption?
Moving from NVL8 to NVL72 increased the blast radius of a single failure ninefold: instead of eight GPUs in a connected domain, there are now 72. For that reason, the rack became the primary unit of planning. Training is easier to divide into power-of-two groups, so each rack runs 64 GPUs while the remaining eight serve as a hot spare. If a card fails, a spare takes its place, and the job restarts from the last checkpoint without waiting for a repair.
Test Frontier LLMs & Autonomous Agents in One Workspace
Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.
The AI world, distilled — once a week
Seven stories that actually mattered, hand-picked. No noise, no reposts, no press releases.
Done! Check your inbox for a confirmation.