AWS adds 3 checks to model training to catch silent GPU errors

Training is running, the loss curve looks normal and the GPUs report no problems. Yet one accelerator has already calculated an incorrect number, and the other model replicas have incorporated it into their shared update. The team keeps spending compute time even though the training results may already be compromised. AWS has described how to look for these failures directly in the PyTorch DDP training loop: repeat some calculations on another node and compare the results.

This is a useful engineering walkthrough for teams training models across multiple GPU nodes. It includes code for the checks and a recovery procedure using SageMaker HyperPod. Its central point applies beyond any particular cloud: the absence of hardware warnings does not mean the calculations are correct.

How one error spreads to every replica

In data-parallel training, multiple model replicas process different batches of examples. The system then averages their gradients, the values used to update the weights. Each replica receives the same averaged value and performs the same update.

Suppose three GPUs calculate gradients of 0.20, 0.40 and 0.30. The average is 0.30. But if a fault causes the second accelerator to return 4.00 instead of 0.40, all three replicas receive an incorrect average. Their weights will still match after the update. A simple comparison of the replicas will suggest everything is fine, even though the error has already spread.

Standard checks protect other parts of the system. Memory checks help detect data corruption in memory, while network checks detect transmission errors. A checksum for a saved model confirms the file's integrity. But a correctly written file can still contain weights corrupted by an earlier calculation.

To illustrate the scale of the problem, AWS cites data from the training of Llama 3 405B: over 54 days, Meta attributed six of 419 unexpected interruptions to silent data corruption. Those figures cover detected incidents that could be linked to a fault; they do not reveal the frequency of all undetected errors.

Three checks in the training loop

The first check looks for NaN and Inf in local gradients before averaging, then checks the averaged result before the weights are updated. This makes it possible to stop a step before an obviously invalid value becomes part of the model. But a number such as 4.00 looks perfectly valid and will pass this check.

Catching that requires the second check: at selected steps, each GPU reprocesses a microbatch from another node. If identical data and conditions produce different results, there is reason to investigate. The comparison covers the loss value and compact gradient fingerprints. The approach requires a third node to resolve ambiguous comparisons.

The third check runs after the update, comparing weight fingerprints across model replicas. If one replica differs from a strict majority, the suspect instance can be identified. The check runs periodically and before checkpoints are saved, to avoid writing detected corruption into the next checkpoint.

When an alert is raised, the team stops training, maps the suspect process to its node and prevents new jobs from being scheduled there. It then examines metrics, runs diagnostics if necessary and replaces the node after checking it. The recommendation is to resume training from a checkpoint saved before the suspect interval.

What to consider before deployment

AWS tested the solution on four nodes with 32 NVIDIA H100 GPUs, injecting faults through software. The tests were also repeated on three nodes, the minimum required for the proposed approach. These tests show how the checks behave in specific scenarios, but do not prove they can detect every hardware fault.

There are also specific blind spots. An incorrect finite number produced within the averaging operation can pass all the checks. Recomputation checks only selected steps and requires reproducible data; gradient comparisons also rely on deterministic calculations. Matching compact fingerprints provide strong evidence that tensors are equal, but not mathematical proof.

The practical takeaway: expensive distributed training runs warrant checks on the calculation results themselves, along with recovery preparations made in advance. A starting point is to check for NaN and Inf before updates and saves. Next, measure the cost of recomputation on your own workload and determine how often to run the audit. Checkpoints must be stored outside the node that may be replaced: its local disks are destroyed during replacement.