Free Intensive Nov 21–22Free 2-day hands-on Vibecoding Intensive on Nov 21–22 at 16:00 MSK (Sat–Sun) · Live Cohort 3 starts Nov 23Join Free →Cohort 3 (Nov 23)
InfrastructureInsight · On RadarHabr AI→ original

Eight GPUs cut training from 32 days to 5, but GPU-hour costs rose 25%

If a model takes 32 days to train on one GPU, it seems logical to use eight and finish in 4 days. In practice it took 5. The cards spend time exchanging gradients and waiting for each other, so scaling efficiency is 80%, not 100%. Meanwhile, consumption rose from 768 to 960 GPU-hours, or 25%: faster, but more expensive. Before buying more cards, profile your training and compare the speedup with the increase in cost.

AI-processed from Habr AI; edited by Hamidun News
Eight GPUs cut training from 32 days to 5, but GPU-hour costs rose 25%
Source: Habr AI. Collage: Hamidun News.
Listen to article
Article Fact Sheet & Key Figures
ParameterDetails
Primary sourceHabr AI ↗
Editor-in-Chief
Key metric #1DDP, FSDP, Tensor Parallelism and Pipeline Parallelism: four schemes with different overheads
Key metric #2PyTorch Profiler and nvidia-smi topo -m reveal delays and the links between GPUs
Key metric #3Training finished 27 days sooner, but cost 192 more GPU-hours

Imagine your model takes 32 days to train on a single GPU. You add eight cards (Multi-GPU mode) and expect to finish in 4 days, but it takes 5. The cards spend time exchanging gradients (the adjustments the model learns from), moving data, and waiting for their slowest neighbors. As a result, the speedup is 6.4x instead of 8x, which means a scaling efficiency of 80%. And the billable GPU-hours came to 960 instead of 768, an increase of 25%. Training finished 27 days sooner, but it cost more. Below, we look at where these losses come from and how to decide whether you need extra cards.

Why eight cards don't give an eightfold speedup

If you train models, sooner or later you run into one of two problems: training drags on too long, or the model stops fitting in video memory. The natural reaction is to add GPUs. But Multi-GPU is not a free speedup: every new card brings overhead, and you pay for it in time and money.

Let's run the numbers on the hypothetical example from the source. Speedup: 32 / 5 = 6.4. Scaling Efficiency: 6.4 / 8 × 100% = 80%. Now the accounting side. One card for 32 days is 768 GPU-hours. Eight cards for 5 days is 960 GPU-hours, 192 more. At the same hourly price, training became 25% more expensive but finished 27 days sooner. If the deadline matters more than the budget, the trade is a good one; if not, the extra hours are wasted.

Four approaches and where each one hurts

Data Parallelism (PyTorch DDP). Each GPU holds a full copy of the model and processes its own share of the batch. In the backward pass, the processes synchronize gradients through an All-Reduce operation. DDP can overlap communication with computation, but with small batches, communication takes as long as the computation itself, and the gain melts away.

Tensor Parallelism. The parameters and computations of individual layers are split across cards, which constantly exchange intermediate results. Performance therefore depends heavily on the interconnects: PCIe, NVLink and NVSwitch topologies.

Pipeline Parallelism. The model's layers are spread across accelerators, and microbatches pass through them in a chain. While the pipeline fills and drains, some cards sit idle. These intervals are called pipeline bubbles.

FSDP. Parameters, gradients and optimizer states are distributed across the cards. With full sharding, this saves video memory (VRAM) but requires additional All-Gather and Reduce-Scatter operations.

A rough rule: DDP fits when the model fits on one card and you simply need to speed it up. FSDP, Tensor Parallelism and Pipeline Parallelism are for when it doesn't fit.

What to check before buying or renting cards

If the speedup is lower than expected, check data loading, NCCL communication time (NVIDIA's library for data exchange between GPUs) and the load balance across processes. If the cards are waiting on the DataLoader, no cluster will help. PyTorch Profiler can find delays in computation and communication, and the nvidia-smi topo -m command shows the topology of connections between GPUs.

A practical sequence looks like this:

  • measure throughput (data processing speed) and step time on a single card;
  • repeat the measurement on several cards and calculate scaling efficiency;
  • compare the speed gain with the growth in GPU-hours and cost;
  • only then decide whether you need the next step.

Sometimes optimizing the DataLoader or switching the parallelism scheme gives a bigger gain than adding accelerators.

Limitations of the example

The 80% efficiency is an illustration, not a benchmark. On real workloads, the figure depends on model size, batch size, the connections between cards and code quality. The material was published on the blog of the cloud provider Reg.cloud (Рег.облако) as an overview, without its own measurements. But the logic is universal: speed and cost grow out of step, and a project lead benefits from seeing both numbers, not just the training time.

Story

Key Questions Answered

Why eight cards don't give an eightfold speedup?

If you train models, sooner or later you run into one of two problems: training drags on too long, or the model stops fitting in video memory. The natural reaction is to add GPUs. But Multi-GPU is not a free speedup: every new card brings overhead, and you pay for it in time and money. Let's run the numbers on the hypothetical example from the source. Speedup: 32 / 5 = 6.4. Scaling Efficiency: 6.4 / 8 × 100% = 80%. Now the accounting side. One card for 32 days is 768 GPU-hours.

What are the key details on Four approaches and where each one hurts?

Data Parallelism (PyTorch DDP). Each GPU holds a full copy of the model and processes its own share of the batch. In the backward pass, the processes synchronize gradients through an All-Reduce operation. DDP can overlap communication with computation, but with small batches, communication takes as long as the computation itself, and the gain melts away. Tensor Parallelism. The parameters and computations of individual layers are split across cards, which constantly exchange intermediate results.

What to check before buying or renting cards?

If the speedup is lower than expected, check data loading, NCCL communication time (NVIDIA's library for data exchange between GPUs) and the load balance across processes. If the cards are waiting on the DataLoader, no cluster will help. PyTorch Profiler can find delays in computation and communication, and the nvidia-smi topo -m command shows the topology of connections between GPUs.

ZK
Zhemal Khamidun — CPO at Alpina Digital, product manager of AlpinaGPT. Building AI products since 2019. Hamidun News is a pet project that grew into a production aggregator with 6,000+ readers.
✦ Unified AI Workspace · Nomad

Test Frontier LLMs & Autonomous Agents in One Workspace

Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.

Try Nomad AI →Launch Nomad Agent
What do you think?
Loading comments…