How to Speed Up Transformer Training with NVIDIA Apex: FusedAdam, FusedLayerNorm and torch.amp
NVIDIA Apex combined with torch.amp can speed up transformer training by 1.5–2.5× without any change to the model architecture. The key tools are FusedAdam in place of standard Adam, FusedLayerNorm for normalization, and native autocast for mixed precision. The walkthrough covers building Apex from source and a step-by-step benchmark of each component.
AI-processed from MarkTechPost; edited by Hamidun News
Training transformers is one of the most resource-hungry tasks in ML. Standard PyTorch works correctly, but it leaves a lot of performance on the table: most operations run sequentially even where they could be fused. NVIDIA Apex supplies a set of optimized CUDA kernels, and native torch.amp adds mixed precision. Together they produce the same result faster and with less GPU memory.
What is NVIDIA Apex, and why build it from source
Apex is an open-source NVIDIA library that extends PyTorch with optimized CUDA extensions. Its core mechanism is "fused" kernels: operations that combine several computation steps into a single GPU call. This cuts the number of memory accesses and the synchronization overhead, which matters most for small operations, where launching a kernel takes longer than the computation itself.
An important point: a standard pip install apex does not include the CUDA extensions. You need to build from source manually with the --cuda_ext --cpp_ext flags. That requires compatible versions of PyTorch, the CUDA Toolkit and a C++ compiler. After installation, check programmatically that the kernels are available via apex.optimizers and apex.normalization, to confirm you are using the native versions and not slow Python fallbacks.
FusedAdam, FusedLayerNorm and torch.amp
Three tools deliver most of the performance gain when training transformers:
- FusedAdam is the Adam optimizer rewritten in CUDA. It merges the weight update, the first and second moment computation, norm clipping and weight decay into a single kernel. It makes far fewer GPU memory accesses than the standard implementation.
- FusedLayerNorm is Layer Normalization as a single CUDA kernel instead of a chain of sequential PyTorch operations (mean, variance, subtract, divide, scale, shift). For transformers with dozens of normalization layers, the cumulative effect is noticeable.
- torch.amp is PyTorch's built-in automatic mixed precision, exposed through
autocast()andGradScaler. It switches computation to BF16 or FP16 where that is safe, keeping FP32 for gradient accumulation. It roughly halves GPU memory use and speeds up matrix multiplications on GPUs with tensor cores.
The key advantage is how little code changes: FusedAdam is a drop-in replacement for torch.optim.Adam, FusedLayerNorm replaces nn.LayerNorm, and torch.amp is added as a wrapper around the forward pass without rewriting the rest of the training logic.
Step-by-step benchmark
To isolate the contribution of each component, the optimizations are enabled one at a time, recording iteration time and peak GPU memory at each step:
- Baseline Adam + FP32 — the reference point
- FusedAdam + FP32 — the gain from swapping the optimizer
- FusedAdam + FusedLayerNorm + FP32 — adding fused normalization
- FusedAdam + FusedLayerNorm + torch.amp (BF16) — the full configuration
This incremental approach avoids relying on the aggregate result of a "black box": the contribution of each step is visible. The final configuration improves iteration time by 1.5× to 2.5× over the baseline; the exact figure depends on model size, batch size and GPU architecture.
"Mixed precision combined with fused kernels is the de facto standard for production transformer training."
What this means
For ML engineers, this is a reproducible recipe with a measurable result: build Apex from source, swap two components and add an autocast context manager, with no changes to the data structure, metrics or validation logic. It is especially relevant on a tight GPU-hour budget: running the same experiments in less time means direct savings for teams training large models on their own hardware or in the cloud.
- Nov 21, 2026SGLang with Humming Kernels Cut Time to First Token by 7–23% and Output Throughput by 17–34%
- Nov 21, 2026CoreWeave Owes $47 Billion on Nvidia Chips and Lost $709 Million in One Quarter
- Nov 21, 2026A local phi3 was jailbroken in 51% of attempts — the developer put a security gateway in front of it
- Nov 21, 2026Thinking Machines Releases Open-Weight Inkling 2 Mini with 26.4B Parameters for 32 GB Macs
Key Questions Answered
What is NVIDIA Apex, and why build it from source?
Apex is an open-source NVIDIA library that extends PyTorch with optimized CUDA extensions. Its core mechanism is "fused" kernels: operations that combine several computation steps into a single GPU call. This cuts the number of memory accesses and the synchronization overhead, which matters most for small operations, where launching a kernel takes longer than the computation itself. An important point: a standard pip install apex does not include the CUDA extensions.
What are the key details on FusedAdam, FusedLayerNorm and torch.amp?
Three tools deliver most of the performance gain when training transformers: FusedAdam is the Adam optimizer rewritten in CUDA. It merges the weight update, the first and second moment computation, norm clipping and weight decay into a single kernel. It makes far fewer GPU memory accesses than the standard implementation. FusedLayerNorm is Layer Normalization as a single CUDA kernel instead of a chain of sequential PyTorch operations (mean, variance, subtract, divide, scale, shift).
What are the key details on Step-by-step benchmark?
To isolate the contribution of each component, the optimizations are enabled one at a time, recording iteration time and peak GPU memory at each step: Baseline Adam + FP32 — the reference point FusedAdam + FP32 — the gain from swapping the optimizer FusedAdam + FusedLayerNorm + FP32 — adding fused normalization FusedAdam + FusedLayerNorm + torch.amp (BF16) — the full configuration This incremental approach avoids relying on the aggregate result of a "black box": the contribution of each step is visible.
Test Frontier LLMs & Autonomous Agents in One Workspace
Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.
The AI world, distilled — once a week
Seven stories that actually mattered, hand-picked. No noise, no reposts, no press releases.
Done! Check your inbox for a confirmation.