Operator Fusion and FlashAttention: Keeping GPUs from Idling While They Wait for Memory
During inference, a GPU often isn't computing so much as waiting: each small operation launches separately and writes its intermediate result to memory. Operator fusion merges these steps into a single pass, much like a cook chopping a salad in one bowl instead of five. FlashAttention computes attention in blocks and never builds the enormous T×T matrix in memory. The weights and formulas stay the same, so enable the ready-made backends first and profile the whole workload.
AI-processed from Machine Learning Mastery; edited by Hamidun News
When language model inference runs slower than you'd like, the first instinct is usually to buy a more powerful GPU. But often the accelerator isn't computing so much as waiting. Each small operation in a layer launches as a separate "kernel" (a program executed on the GPU), and its intermediate result is written to the main video memory, HBM, and then immediately read back.
It's like making a salad by moving each chopped ingredient into a new bowl: you spend more time carrying dishes than wielding the knife. Operator fusion merges several steps into a single pass, and FlashAttention does the same for the attention mechanism. The model's weights and formulas remain unchanged.
Adrian Tam described this optimization layer in a educational walkthrough on Machine Learning Mastery. The article includes no speed measurements of its own, but it clearly lays out the causes of the slowdowns and how to address them.
Why "correct" code turns out slow
A transformer layer consists of familiar steps: linear projections, attention, normalization, activation functions, and the addition of the original signal (the residual). A straightforward implementation launches a separate kernel for each step and writes every intermediate tensor to memory. The math is correct, but moving values around and launching kernels takes more time than the useful arithmetic.
A typical example is the SwiGLU block in modern models. It contains three matrix multiplications, a SiLU activation, and an element-wise multiplication. Every boundary between operations means a write to HBM and a read back. If you fuse SiLU with the multiplication, a single pass over the data replaces two.
Other fusion candidates include the residual addition combined with RMSNorm or LayerNorm, bias with activation, rotary position embeddings (RoPE) for queries and keys, updating the KV cache, and dequantization together with matrix multiplication. But fusion has limits. If one huge matrix multiplication consumes nearly all the time, the gain is small. An overly large kernel overloads the GPU's registers, reduces the occupancy of compute units, and can end up slower than several well-tuned kernels. The author advises keeping the simple code as a reference and choosing kernels empirically, by profiling the entire workload.
FlashAttention and paged KV cache
Naive attention builds a matrix of scores Q·Kᵀ of size T² for each head, where T is the sequence length. Writing this matrix to memory and reading it again for the softmax consumes bandwidth, and on long prefill (the initial processing of the prompt) this becomes the bottleneck. FlashAttention sidesteps the problem: it processes queries, keys, and values in tiles, updates the softmax statistics, and accumulates the result in fast on-chip memory. The full matrix never reaches main memory.
In PyTorch this is available through scaled_dot_product_attention. The framework selects a backend automatically based on the device, data type, tensor shapes, and mask. You can set the priority manually with sdpa_kernel, but hard-coding the choice is risky: if neither FlashAttention nor the cuDNN backend supports your inputs or hardware, the code will fail.
It's important to distinguish the phases. FlashAttention is especially useful during prefill, where many tokens are processed together. During decode, when the model generates a response one token at a time, there is one or only a few new queries, and the bottleneck is more often reading the growing KV cache, that is, the stored data about tokens already processed.
This is where paged attention comes in. A request's KV cache grows token by token and ends at an unpredictable length. If you require one contiguous block of memory for it, some space is reserved and wasted, and moving and extending the cache becomes expensive. The paged approach divides memory into fixed-size blocks and keeps a table mapping logical blocks to physical ones. The kernel follows this table and cannot assume that K[t+1] sits next to K[t]. Blocks are allocated as needed, returned to the pool when a request finishes, and can be shared between sequences with a common prefix. This doesn't reduce the number of logical KV elements; the gain is less fragmentation and a smaller reserve.
In the vLLM example, the author explicitly sets block_size=16 and the TRITON_ATTN backend. The block_size parameter by itself does not select the compute backend.
torch.compile and CUDA Graphs
Writing kernels by hand in CUDA or Triton is specialist work. The alternative is TorchDynamo, which captures the graph of operations, while TorchInductor, the default backend of torch.compile, optimizes it, reuses buffers, and fuses compatible element-wise operations into a single Triton kernel. But not everything gets fused. Graph breaks caused by data-dependent branches, unsupported operations, or converting tensors to Python scalars split the model into several graphs.
The reduce-overhead mode adds CUDA Graphs. Fusion reduces the number of kernels and memory traffic. A CUDA Graph records the remaining sequence of GPU operations and launches it with a single command from the CPU, removing launch overhead. This suits repetitive small-batch tasks best, such as decode. If request shapes change constantly, the benefit is smaller, and the memory for working buffers can grow.
What to do in practice
- Profile the whole application, not an individual kernel: a smaller number of kernels in the profiler is only a hint of a gain.
- Set TORCH_LOGS=perf_hints and output_code before starting Python to see which operations Inductor fused.
- Check results against a tolerance suited to your data type: the order of operations changes, and small numerical differences are normal.
- Watch for dynamic shapes: they trigger recompilation and can reduce the effect of fusion.
The material is educational and contains no measurements of its own, but as a map of causes and remedies for anyone tuning inference on vLLM or PyTorch, it is useful.
- Nov 21, 2026A viral GTA 6 Claude clip looks fake: half an hour of control burned through the 5-hour Pro limit
- Nov 21, 2026Claude Code Speeds Up Snowboard Kids 2 Decompilation: From 20% to 45% of Functions in Three Months
- Nov 21, 2026VibeCBS collects project nicknames in advance: the agent finds 2.8 times more documents
- Nov 21, 2026FDE job postings grew 4.6x in a year: what a forward deployed engineer actually does at the client's site
Key Questions Answered
Why "correct" code turns out slow?
A transformer layer consists of familiar steps: linear projections, attention, normalization, activation functions, and the addition of the original signal (the residual). A straightforward implementation launches a separate kernel for each step and writes every intermediate tensor to memory. The math is correct, but moving values around and launching kernels takes more time than the useful arithmetic. A typical example is the SwiGLU block in modern models.
What are the key details on FlashAttention and paged KV cache?
Naive attention builds a matrix of scores Q·Kᵀ of size T² for each head, where T is the sequence length. Writing this matrix to memory and reading it again for the softmax consumes bandwidth, and on long prefill (the initial processing of the prompt) this becomes the bottleneck. FlashAttention sidesteps the problem: it processes queries, keys, and values in tiles, updates the softmax statistics, and accumulates the result in fast on-chip memory. The full matrix never reaches main memory.
What are the key details on torch.compile and CUDA Graphs?
Writing kernels by hand in CUDA or Triton is specialist work. The alternative is TorchDynamo, which captures the graph of operations, while TorchInductor, the default backend of torch.compile, optimizes it, reuses buffers, and fuses compatible element-wise operations into a single Triton kernel. But not everything gets fused. Graph breaks caused by data-dependent branches, unsupported operations, or converting tensors to Python scalars split the model into several graphs. The reduce-overhead mode adds CUDA Graphs.
Test Frontier LLMs & Autonomous Agents in One Workspace
Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.
The AI world, distilled — once a week
Seven stories that actually mattered, hand-picked. No noise, no reposts, no press releases.
Done! Check your inbox for a confirmation.