Free Intensive Nov 21–22Free 2-day hands-on Vibecoding Intensive on Nov 21–22 at 16:00 MSK (Sat–Sun) · Live Cohort 3 starts Nov 23Join Free →Cohort 3 (Nov 23)
ReleaseResearch Paper · For R&DMarkTechPost→ original

How to Speed Up Transformer Training with NVIDIA Apex: FusedAdam, FusedLayerNorm and torch.amp

NVIDIA Apex combined with torch.amp can speed up transformer training by 1.5–2.5× without any change to the model architecture. The key tools are FusedAdam in place of standard Adam, FusedLayerNorm for normalization, and native autocast for mixed precision. The walkthrough covers building Apex from source and a step-by-step benchmark of each component.

AI-processed from MarkTechPost; edited by Hamidun News
How to Speed Up Transformer Training with NVIDIA Apex: FusedAdam, FusedLayerNorm and torch.amp
Source: MarkTechPost. Collage: Hamidun News.
Listen to article

Training transformers is one of the most resource-hungry tasks in ML. Standard PyTorch works correctly, but it leaves a lot of performance on the table: most operations run sequentially even where they could be fused. NVIDIA Apex supplies a set of optimized CUDA kernels, and native torch.amp adds mixed precision. Together they produce the same result faster and with less GPU memory.

What is NVIDIA Apex, and why build it from source

Apex is an open-source NVIDIA library that extends PyTorch with optimized CUDA extensions. Its core mechanism is "fused" kernels: operations that combine several computation steps into a single GPU call. This cuts the number of memory accesses and the synchronization overhead, which matters most for small operations, where launching a kernel takes longer than the computation itself.

An important point: a standard pip install apex does not include the CUDA extensions. You need to build from source manually with the --cuda_ext --cpp_ext flags. That requires compatible versions of PyTorch, the CUDA Toolkit and a C++ compiler. After installation, check programmatically that the kernels are available via apex.optimizers and apex.normalization, to confirm you are using the native versions and not slow Python fallbacks.

FusedAdam, FusedLayerNorm and torch.amp

Three tools deliver most of the performance gain when training transformers:

  • FusedAdam is the Adam optimizer rewritten in CUDA. It merges the weight update, the first and second moment computation, norm clipping and weight decay into a single kernel. It makes far fewer GPU memory accesses than the standard implementation.
  • FusedLayerNorm is Layer Normalization as a single CUDA kernel instead of a chain of sequential PyTorch operations (mean, variance, subtract, divide, scale, shift). For transformers with dozens of normalization layers, the cumulative effect is noticeable.
  • torch.amp is PyTorch's built-in automatic mixed precision, exposed through autocast() and GradScaler. It switches computation to BF16 or FP16 where that is safe, keeping FP32 for gradient accumulation. It roughly halves GPU memory use and speeds up matrix multiplications on GPUs with tensor cores.

The key advantage is how little code changes: FusedAdam is a drop-in replacement for torch.optim.Adam, FusedLayerNorm replaces nn.LayerNorm, and torch.amp is added as a wrapper around the forward pass without rewriting the rest of the training logic.

Step-by-step benchmark

To isolate the contribution of each component, the optimizations are enabled one at a time, recording iteration time and peak GPU memory at each step:

  1. Baseline Adam + FP32 — the reference point
  2. FusedAdam + FP32 — the gain from swapping the optimizer
  3. FusedAdam + FusedLayerNorm + FP32 — adding fused normalization
  4. FusedAdam + FusedLayerNorm + torch.amp (BF16) — the full configuration

This incremental approach avoids relying on the aggregate result of a "black box": the contribution of each step is visible. The final configuration improves iteration time by 1.5× to 2.5× over the baseline; the exact figure depends on model size, batch size and GPU architecture.

"Mixed precision combined with fused kernels is the de facto standard for production transformer training."

What this means

For ML engineers, this is a reproducible recipe with a measurable result: build Apex from source, swap two components and add an autocast context manager, with no changes to the data structure, metrics or validation logic. It is especially relevant on a tight GPU-hour budget: running the same experiments in less time means direct savings for teams training large models on their own hardware or in the cloud.

Story

Key Questions Answered

What is NVIDIA Apex, and why build it from source?

Apex is an open-source NVIDIA library that extends PyTorch with optimized CUDA extensions. Its core mechanism is "fused" kernels: operations that combine several computation steps into a single GPU call. This cuts the number of memory accesses and the synchronization overhead, which matters most for small operations, where launching a kernel takes longer than the computation itself. An important point: a standard pip install apex does not include the CUDA extensions.

What are the key details on FusedAdam, FusedLayerNorm and torch.amp?

Three tools deliver most of the performance gain when training transformers: FusedAdam is the Adam optimizer rewritten in CUDA. It merges the weight update, the first and second moment computation, norm clipping and weight decay into a single kernel. It makes far fewer GPU memory accesses than the standard implementation. FusedLayerNorm is Layer Normalization as a single CUDA kernel instead of a chain of sequential PyTorch operations (mean, variance, subtract, divide, scale, shift).

What are the key details on Step-by-step benchmark?

To isolate the contribution of each component, the optimizations are enabled one at a time, recording iteration time and peak GPU memory at each step: Baseline Adam + FP32 — the reference point FusedAdam + FP32 — the gain from swapping the optimizer FusedAdam + FusedLayerNorm + FP32 — adding fused normalization FusedAdam + FusedLayerNorm + torch.amp (BF16) — the full configuration This incremental approach avoids relying on the aggregate result of a "black box": the contribution of each step is visible.

ZK
Zhemal Khamidun — CPO at Alpina Digital, product manager of AlpinaGPT. Building AI products since 2019. Hamidun News is a pet project that grew into a production aggregator with 6,000+ readers.
✦ Unified AI Workspace · Nomad

Test Frontier LLMs & Autonomous Agents in One Workspace

Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.

Try Nomad AI →Launch Nomad Agent
What do you think?
Loading comments…