RunPod: 4 A100 GPUs in the cloud cost $124 000 less than on-prem over 3 years

RunPod has published a total-cost-of-ownership calculation on its blog. It says four NVIDIA A100s in the company's cloud will cost more than $124 000 less over three years than a comparable on-prem cluster. That equates to about $41 000 a year and roughly $3 440 a month. Divide the sum by 105 120 GPU-hours (four cards, around the clock, 36 months) and you get a difference of about $1.18 per GPU-hour. One important caveat: the cloud provider did the math, and the announcement does not disclose the detailed methodology.

Where the gap comes from

An in-house cluster requires money before the first job runs. You have to buy servers and cards, networking, a rack, and power supplies. Then the recurring costs begin: electricity, cooling, floor space, replacing failed components, and the time of the engineer who looks after all of it. The cloud turns all of this into an hourly rate, and you pay only for the hours you actually use.

The age of the hardware also matters. The A100 launched in 2020, and over a three-year horizon its market value falls noticeably as the H100 and newer generations become more available. A purchased card loses value on the balance sheet, while a renter simply switches to a newer architecture.

Where the calculation may not hold

There is one main variable in comparisons like this: utilization. If four A100s are busy with training or inference almost around the clock, idle hours approach zero and the cloud's advantage shrinks. If the cards are needed only at peak periods, such as before a model release or during a stretch of experiments, then your own hardware spends most of its time idle, heating the room.

Here are a few more assumptions to check in any calculation of this kind:

• the price of electricity and cooling in your region;
• the cost of an administrator's time and the degree of automation;
• cloud fees for data storage and egress traffic;
• availability of the cards you need at the moment you need them, especially in peak weeks;
• data locality and compliance requirements.

Key figures from the announcement:

• configuration: 4× NVIDIA A100;
• comparison horizon: 36 months;
• claimed savings: more than $124 000;
• on average: about $41 300 a year and $3 440 a month;
• equivalent at 24/7 operation: about $1.18 per GPU-hour (our calculation from the claimed sum).

How to check this against your own numbers

Start with actual utilization. Pull the metrics for the last three months and see how many hours a week the GPUs are really computing rather than waiting for data. Then find the break-even point: the share of busy hours at which your own cluster becomes cheaper than renting. A team with 20–30% utilization and a team with 85–90% will get opposite answers.

Next, run a pilot. Move one typical workload, whether fine-tuning or production inference, to the cloud for a month and compare the real bill with your internal estimate. A hybrid often wins: baseline load on your own hardware, spikes on rented capacity.

RunPod's argument is convincing for startups and research groups with uneven schedules. For a steady, round-the-clock workload, treat the $124 000 figure as a reason to do your own math, not as a ready-made conclusion.