Google Built Fuzzing Into the XLA Autotuner and Caught 142 Silent Compiler Bugs on TPU

If a compiler crashes with an error, you notice. If it silently produces wrong numbers, you find out a week into training, when the loss behaves strangely and the cluster bill has already arrived. These silent bugs are exactly what the Google team examines in its work XLA from the Trenches, and their solution is surprisingly down-to-earth.

Why silent bugs are worse than crashes

XLA turns a model's computation graph into machine code for TPU. Along the way it fuses operations into large kernels (fusion), lays data out across vector registers and memory, and performs dozens of other optimizations. Each of them can be correct on its own and still break on a rare combination of shapes, types, and parameters.

Static tests in CI check cases that someone thought up in advance. Even millions of such tests do not cover all the combinations that arise in real models. The bug doesn't show up as a crash: everything runs, everything is fast, the result is just slightly off. When training a large model, the cost of such an error is measured in millions of dollars.

How it works

The key idea is not to build separate, expensive verification infrastructure but to hook into something that already works. The performance autotuner already tries different compilation variants of the same kernel every day in search of the fastest one. The authors added result comparison to this loop: the output of each variant is checked against the output of the default fused kernel. If there is a discrepancy, one of the variants is being compiled incorrectly.

If the suspicion falls on a bug inside the graph, the system also compares the result with an unfused reference, where operations are executed separately. This makes it possible to tell whether fusion is specifically to blame.

Next comes a two-stage isolator. Simply splitting a kernel into parts often masks the bug, because the problem lives specifically in the fused version. So the isolator reads the hardware vector registers step by step inside the fused kernel itself and finds the instruction where the values first diverge. In addition, the authors use synthetic program generation and a swarm of AI agents that write patches for the problems found.

What they found

On Google's TPU clusters, the system helped fix more than 142 bugs missed by CI. The distribution is interesting: 43% of them were not in high-level graph transformations but in low-level code generators for vector registers and memory. In other words, things broke where ordinary graph-level tests barely look.

Limitations and takeaways

To be fair, the work has weak spots. All 142 bugs are counted as a single figure, with no ablations by component, so it is unclear which part of the system delivers the most value: the fuzzing, the isolator, or the agents. Nondeterministic memory races, true heisenbugs, remain outside the scope of the method. Moreover, comparing against the default kernel assumes the default itself is correct, which is not always the case.

The practical takeaway is broader than TPU. If you have an autotuner, a JIT compiler, or any mechanism that iterates over equivalent implementations of one operation, you are already paying to compute several answers on the same inputs. Adding a result check with a tolerance for numerical noise is cheap, and it catches precisely the errors nobody would otherwise see. For teams training large models on their own stack, this is one of the cheapest ways to catch defects before they eat the budget.