DeepMind Finds Two Rounding Errors in Softmax: Gradients Vanish Too Early
Google DeepMind has found two rounding errors in softmax and cross-entropy, the functions that almost every neural network learns with. When a model is almost 100% confident in its answer, the computer rounds a tiny gradient, the correction to the weights, down to zero, even though on paper it is nonzero. The second error appears during distillation, when a small model copies a larger one. The fix is to rearrange the operations in the formula: the math stays the same, but small numbers are no longer lost.
AI-processed from TG: @mishin_learning (Мишин Лернинг); edited by Hamidun News
Softmax and cross-entropy sit in almost every neural network, from image classifiers to large language models. Softmax turns a model's raw scores into probabilities, and cross-entropy compares those probabilities with the correct answer and computes the error. These two functions are so basic and so 'solved' that almost nobody looks inside them. Google DeepMind researchers did look, and they found two numerical errors that can cause training to go differently than intended. The paper is called "A Fix for Softmax and Cross-Entropy" and is published on arXiv.
An important caveat: the error is not in the math. The formulas are correct. The problem lies in how the computer evaluates them. Floating-point numbers are stored with limited precision, and very small values simply get lost.
Where the gradient disappears
The gradient is the signal that says which way, and by how much, to adjust the weights. The first error appears when the model becomes too confident in the correct answer. Suppose it is 99.999999% sure a photo shows a cat. On paper, it still lacks a tiny bit to reach the ideal, so the gradient remains small but nonzero. The computer, however, rounds it down to zero prematurely: for it, the confidence already equals one, and there seems to be nothing left to learn.
The simplest analogy is a cash register that discards anything less than a cent. One purchase changes nothing. But training involves millions of steps, and some of the useful signal disappears at each one.
The second error arises during distillation, the name for training in which a small student model reproduces the answers of a large teacher model. According to the authors, the numerical error appears even when the student perfectly reproduces the teacher. The goal has already been reached, yet the calculation is still distorted.
How bad it is
The researchers themselves stress that the situation is not as dire as it might seem. Long-established techniques already partly protect against the first problem. Label smoothing keeps the correct answer from reaching 100% confidence. Low temperature also prevents the model from becoming overly confident. In such cases, the gradient does not get the chance to reach the zone where rounding zeroes it out.
But these techniques treat the symptom, not the cause. They do not fix the computational error itself. Moreover, they change the training objective. Label smoothing deliberately makes the model less categorical, which can get in the way where honest probability calibration matters.
What the authors propose
The solution turns out to be simple: rewrite the formula, that is, change the order of operations. The mathematical result stays the same, but the computer stops losing small numbers. No new layers, different architecture, or additional hyperparameters are required. It is a fix at the level of how the function is implemented.
For developers, there are two practical takeaways:
- If you write your own kernel for softmax or cross-entropy, for example in CUDA or Triton, or train models in low precision, check how it handles small probabilities.
- If you use label smoothing or temperature out of habit, remember that they may mask a numerical problem rather than remove it.
What we still don't know
The summary of the research contains no figures. It is unclear how much the fix improves model quality, which tasks show a noticeable effect, and how it plays out in large language models. Popular frameworks may already account for some of these subtleties. To judge the real gain, you need to read the paper itself and examine the experiments.
Still, the finding is telling. Even in the most basic building blocks of AI, there are places where imprecise arithmetic quietly affects training. The fix is cheap, yet it potentially touches everyone who trains neural networks. Those who work on training and on custom kernels should at least skim this work.
- Nov 21, 2026Softmax Rounding Bug Explains Grokking: After the Fix, the Sudden Generalization Disappears
- Nov 21, 2026Developer Fits DeepSeek V3 into 190 Lines of JAX and Trains It on a MacBook Air in 2.6 Hours
- Nov 21, 2026Jensen Huang Steps Down as NVIDIA CEO After 33 Years, CFO Colette Kress Becomes Interim CEO
- Nov 21, 2026OpenAI Disbands Robotics Team Again: GPT-6 Astra Completed 74.8% of Tasks in Simulation Versus 51%
Key Questions Answered
Where the gradient disappears?
The gradient is the signal that says which way, and by how much, to adjust the weights. The first error appears when the model becomes too confident in the correct answer. Suppose it is 99.999999% sure a photo shows a cat. On paper, it still lacks a tiny bit to reach the ideal, so the gradient remains small but nonzero. The computer, however, rounds it down to zero prematurely: for it, the confidence already equals one, and there seems to be nothing left to learn.
How bad it is?
The researchers themselves stress that the situation is not as dire as it might seem. Long-established techniques already partly protect against the first problem. Label smoothing keeps the correct answer from reaching 100% confidence. Low temperature also prevents the model from becoming overly confident. In such cases, the gradient does not get the chance to reach the zone where rounding zeroes it out. But these techniques treat the symptom, not the cause. They do not fix the computational error itself.
What the authors propose?
The solution turns out to be simple: rewrite the formula, that is, change the order of operations. The mathematical result stays the same, but the computer stops losing small numbers. No new layers, different architecture, or additional hyperparameters are required. It is a fix at the level of how the function is implemented.
Test Frontier LLMs & Autonomous Agents in One Workspace
Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.
The AI world, distilled — once a week
Seven stories that actually mattered, hand-picked. No noise, no reposts, no press releases.
Done! Check your inbox for a confirmation.