Rounding Bug Found in Softmax and Cross-Entropy: 85% Accuracy Instead of 23%
Softmax and cross-entropy appear in every classifier and LLM, and researchers have now found a rounding bug in both. When a model is very confident, the probability of the correct answer rounds to 100%, the error is computed as zero, and learning from that example stalls. After a fix, accuracy on a test set for the a² + b² mod 97 task reached 85%, while the standard function fell from 61% to 23%. If you train models in bf16, it is worth testing a replacement for this loss function.
AI-processed from TG: @gonzo_ML (gonzo-обзоры ML статей); edited by Hamidun News
Softmax and cross-entropy are the most heavily used functions in deep learning. Softmax turns the raw scores a model produces for each possible answer (called logits) into probabilities. Cross-entropy compares those probabilities with the correct answer and outputs a number that shows how wrong the model was. This number is called the loss, and the network learns from it. This pairing appears in classifiers, language models, and distillation. Jonathan Barron, Dan Gnanapragasam, and Michael Chennen argue in the paper "A Fix for Softmax and Cross-Entropy" that every mainstream implementation of these functions contains two rounding errors. The errors went unnoticed for more than a decade of active use.
How rounding stalls training
The authors call the first bug Softmax Collapse. Picture a student who is so sure of an answer that the teacher stops correcting them. Something similar happens in a model. When the model is very confident, the probability of the correct class rounds to exactly 100% in computer arithmetic. The learning signal is roughly "one minus the probability," which comes out to exactly zero. The gradient that updates the weights disappears, and the model stops learning from that example. The cause is not the math itself but the fact that floating-point numbers have a finite number of significant digits.
The second bug, Softmax Cancellation, shows up in distillation. This is the process of training a small student model to reproduce the answers of a large teacher model. If the student's logits already match the teacher's, the gradient should become zero. But when two nearly equal numbers are subtracted, rounding leaves a small nonzero remainder. That remainder keeps pushing the weights, and they drift. The effect is strongest in bf16, a format that stores few significant digits yet is used today to train many large models.
What the authors proposed and what they measured
The authors rewrote softmax, log-softmax, and cross-entropy in logarithmic space, covering both hard labels, where there is one correct answer, and soft labels, where the target is a set of probabilities. The idea is straightforward: avoid subtracting nearly identical numbers, and compute in a way that keeps small differences from being lost. There is no separate repository. The paper includes self-contained code snippets in JAX.
They tested the approach on modular arithmetic, where grokking is clearly visible. This term describes a situation where a model memorizes training examples for a long time and then suddenly starts to generalize.
- On the task "addition modulo 31" (a + b mod 31), in bf16, fp32, and fp64, the fixed cross-entropy reached 98.9–100% test accuracy. The standard version reached 93.3–100%.
- Alternatives performed worse. Zero-sum Projection, which subtracts the mean from the gradients, reached 48.9–97.9%. StableMax, which replaces softmax with a piecewise rational function, reached 95.6–97.4%.
- On the task a² + b² mod 97 without regularization, the fixed variant reached 85% accuracy. Standard cross-entropy first rose to 61% and then fell to 23%.
The fix also removes the sharp loss spikes that grokking researchers call the Slingshot effect. In bf16 distillation, the weight drift disappears, so there is no need to switch to fp64, which is considerably more expensive in memory and speed.
What to do about it in practice
For engineers, the key point is that this is a drop-in replacement. Only the loss function changes; the architecture and data stay the same. If you train or distill models in bf16 and see unexplained loss spikes or stagnation on very confident examples, this fix is worth testing first. It costs less than moving the entire training run to higher-precision formats.
There are also honest caveats. All experiments were run on small modular arithmetic tasks, where large gaps between logits and near-zero margins between student and teacher are created deliberately. Such extreme regimes are rare in large-scale training, so the gains on large language models have not yet been demonstrated. Any effect in production may appear only in specific scenarios: long training runs with high model confidence, aggressive distillation, or low-precision arithmetic.
The conclusion is this. It is too early to promise that your models will immediately improve. But the fixed cross-entropy is worth adding to your set of A/B checks, especially in bf16 and distillation pipelines. If the effect holds at real scale, the improvement will come from repairing the foundation, not from a new architecture.
- Nov 21, 2026Softmax Rounding Bug Explains Grokking: After the Fix, the Sudden Generalization Disappears
- Oct 26, 2026DeepMind Finds Two Rounding Errors in Softmax: Gradients Vanish Too Early
- Oct 29, 2026Google DeepMind finds softmax flaw: gradient drops to zero at a logit gap of 16.6
- Oct 28, 2026Researchers Remove the Noise from LLM MTP Training with a Single Line of Code
Key Questions Answered
How rounding stalls training?
The authors call the first bug Softmax Collapse. Picture a student who is so sure of an answer that the teacher stops correcting them. Something similar happens in a model. When the model is very confident, the probability of the correct class rounds to exactly 100% in computer arithmetic. The learning signal is roughly "one minus the probability," which comes out to exactly zero. The gradient that updates the weights disappears, and the model stops learning from that example.
What the authors proposed and what they measured?
The authors rewrote softmax, log-softmax, and cross-entropy in logarithmic space, covering both hard labels, where there is one correct answer, and soft labels, where the target is a set of probabilities. The idea is straightforward: avoid subtracting nearly identical numbers, and compute in a way that keeps small differences from being lost. There is no separate repository. The paper includes self-contained code snippets in JAX. They tested the approach on modular arithmetic, where grokking is clearly visible.
What to do about it in practice?
For engineers, the key point is that this is a drop-in replacement. Only the loss function changes; the architecture and data stay the same. If you train or distill models in bf16 and see unexplained loss spikes or stagnation on very confident examples, this fix is worth testing first. It costs less than moving the entire training run to higher-precision formats. There are also honest caveats.
Test Frontier LLMs & Autonomous Agents in One Workspace
Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.
The AI world, distilled — once a week
Seven stories that actually mattered, hand-picked. No noise, no reposts, no press releases.
Done! Check your inbox for a confirmation.