Eight GPUs cut training from 32 days to 5, but GPU-hour costs rose 25%
If a model takes 32 days to train on one GPU, it seems logical to use eight and finish in 4 days. In practice it took 5. The cards spend time exchanging gradients and waiting for each other, so scaling efficiency is 80%, not 100%. Meanwhile, consumption rose from 768 to 960 GPU-hours, or 25%: faster, but more expensive. Before buying more cards, profile your training and compare the speedup with the increase in cost.
AI-processed from Habr AI; edited by Hamidun News
Imagine your model takes 32 days to train on a single GPU. You add eight cards (Multi-GPU mode) and expect to finish in 4 days, but it takes 5. The cards spend time exchanging gradients (the adjustments the model learns from), moving data, and waiting for their slowest neighbors. As a result, the speedup is 6.4x instead of 8x, which means a scaling efficiency of 80%. And the billable GPU-hours came to 960 instead of 768, an increase of 25%. Training finished 27 days sooner, but it cost more. Below, we look at where these losses come from and how to decide whether you need extra cards.
Why eight cards don't give an eightfold speedup
If you train models, sooner or later you run into one of two problems: training drags on too long, or the model stops fitting in video memory. The natural reaction is to add GPUs. But Multi-GPU is not a free speedup: every new card brings overhead, and you pay for it in time and money.
Let's run the numbers on the hypothetical example from the source. Speedup: 32 / 5 = 6.4. Scaling Efficiency: 6.4 / 8 × 100% = 80%. Now the accounting side. One card for 32 days is 768 GPU-hours. Eight cards for 5 days is 960 GPU-hours, 192 more. At the same hourly price, training became 25% more expensive but finished 27 days sooner. If the deadline matters more than the budget, the trade is a good one; if not, the extra hours are wasted.
Four approaches and where each one hurts
Data Parallelism (PyTorch DDP). Each GPU holds a full copy of the model and processes its own share of the batch. In the backward pass, the processes synchronize gradients through an All-Reduce operation. DDP can overlap communication with computation, but with small batches, communication takes as long as the computation itself, and the gain melts away.
Tensor Parallelism. The parameters and computations of individual layers are split across cards, which constantly exchange intermediate results. Performance therefore depends heavily on the interconnects: PCIe, NVLink and NVSwitch topologies.
Pipeline Parallelism. The model's layers are spread across accelerators, and microbatches pass through them in a chain. While the pipeline fills and drains, some cards sit idle. These intervals are called pipeline bubbles.
FSDP. Parameters, gradients and optimizer states are distributed across the cards. With full sharding, this saves video memory (VRAM) but requires additional All-Gather and Reduce-Scatter operations.
A rough rule: DDP fits when the model fits on one card and you simply need to speed it up. FSDP, Tensor Parallelism and Pipeline Parallelism are for when it doesn't fit.
What to check before buying or renting cards
If the speedup is lower than expected, check data loading, NCCL communication time (NVIDIA's library for data exchange between GPUs) and the load balance across processes. If the cards are waiting on the DataLoader, no cluster will help. PyTorch Profiler can find delays in computation and communication, and the nvidia-smi topo -m command shows the topology of connections between GPUs.
A practical sequence looks like this:
- measure throughput (data processing speed) and step time on a single card;
- repeat the measurement on several cards and calculate scaling efficiency;
- compare the speed gain with the growth in GPU-hours and cost;
- only then decide whether you need the next step.
Sometimes optimizing the DataLoader or switching the parallelism scheme gives a bigger gain than adding accelerators.
Limitations of the example
The 80% efficiency is an illustration, not a benchmark. On real workloads, the figure depends on model size, batch size, the connections between cards and code quality. The material was published on the blog of the cloud provider Reg.cloud (Рег.облако) as an overview, without its own measurements. But the logic is universal: speed and cost grow out of step, and a project lead benefits from seeing both numbers, not just the training time.
- Nov 19, 2026Social Department replaces a week of binge-watching with four QuickSearch queries on AWS
- Nov 19, 2026Open-Source Rust Video Editor FilmCraft Hits 18.5K GitHub Stars and Lets AI Agents Drive the Engine
- Nov 19, 2026Kling Image 4.0 Renders in Native 8K and Generates 10-Frame Series from 2.33 ₽ per Image
- Nov 19, 2026Google's Gemini 4 Neon runs twice as fast as Carbon and costs 17% less per task
Key Questions Answered
Why eight cards don't give an eightfold speedup?
If you train models, sooner or later you run into one of two problems: training drags on too long, or the model stops fitting in video memory. The natural reaction is to add GPUs. But Multi-GPU is not a free speedup: every new card brings overhead, and you pay for it in time and money. Let's run the numbers on the hypothetical example from the source. Speedup: 32 / 5 = 6.4. Scaling Efficiency: 6.4 / 8 × 100% = 80%. Now the accounting side. One card for 32 days is 768 GPU-hours.
What are the key details on Four approaches and where each one hurts?
Data Parallelism (PyTorch DDP). Each GPU holds a full copy of the model and processes its own share of the batch. In the backward pass, the processes synchronize gradients through an All-Reduce operation. DDP can overlap communication with computation, but with small batches, communication takes as long as the computation itself, and the gain melts away. Tensor Parallelism. The parameters and computations of individual layers are split across cards, which constantly exchange intermediate results.
What to check before buying or renting cards?
If the speedup is lower than expected, check data loading, NCCL communication time (NVIDIA's library for data exchange between GPUs) and the load balance across processes. If the cards are waiting on the DataLoader, no cluster will help. PyTorch Profiler can find delays in computation and communication, and the nvidia-smi topo -m command shows the topology of connections between GPUs.
Test Frontier LLMs & Autonomous Agents in One Workspace
Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.
The AI world, distilled — once a week
Seven stories that actually mattered, hand-picked. No noise, no reposts, no press releases.
Done! Check your inbox for a confirmation.