Free Intensive Nov 21–22Free 2-day hands-on Vibecoding Intensive on Nov 21–22 at 16:00 MSK (Sat–Sun) · Live Cohort 3 starts Nov 23Join Free →Cohort 3 (Nov 23)

DeepMind Finds Two Rounding Errors in Softmax: Gradients Vanish Too Early

Google DeepMind has found two rounding errors in softmax and cross-entropy, the functions that almost every neural network learns with. When a model is almost 100% confident in its answer, the computer rounds a tiny gradient, the correction to the weights, down to zero, even though on paper it is nonzero. The second error appears during distillation, when a small model copies a larger one. The fix is to rearrange the operations in the formula: the math stays the same, but small numbers are no longer lost.

AI-processed from TG: @mishin_learning (Мишин Лернинг); edited by Hamidun News
DeepMind Finds Two Rounding Errors in Softmax: Gradients Vanish Too Early
Source: TG: @mishin_learning (Мишин Лернинг). Collage: Hamidun News.
Listen to article
Article Fact Sheet & Key Figures
ParameterDetails
Primary sourceTG: @mishin_learning (Мишин Лернинг) ↗
Editor-in-Chief
Key metric #1Label smoothing and low temperature soften the first error but do not fix the calculation
Key metric #2The second error shows up even when the student perfectly reproduces the teacher
Key metric #3No new layers or hyperparameters are needed; only the order of operations changes

Softmax and cross-entropy sit in almost every neural network, from image classifiers to large language models. Softmax turns a model's raw scores into probabilities, and cross-entropy compares those probabilities with the correct answer and computes the error. These two functions are so basic and so 'solved' that almost nobody looks inside them. Google DeepMind researchers did look, and they found two numerical errors that can cause training to go differently than intended. The paper is called "A Fix for Softmax and Cross-Entropy" and is published on arXiv.

An important caveat: the error is not in the math. The formulas are correct. The problem lies in how the computer evaluates them. Floating-point numbers are stored with limited precision, and very small values simply get lost.

Where the gradient disappears

The gradient is the signal that says which way, and by how much, to adjust the weights. The first error appears when the model becomes too confident in the correct answer. Suppose it is 99.999999% sure a photo shows a cat. On paper, it still lacks a tiny bit to reach the ideal, so the gradient remains small but nonzero. The computer, however, rounds it down to zero prematurely: for it, the confidence already equals one, and there seems to be nothing left to learn.

The simplest analogy is a cash register that discards anything less than a cent. One purchase changes nothing. But training involves millions of steps, and some of the useful signal disappears at each one.

The second error arises during distillation, the name for training in which a small student model reproduces the answers of a large teacher model. According to the authors, the numerical error appears even when the student perfectly reproduces the teacher. The goal has already been reached, yet the calculation is still distorted.

How bad it is

The researchers themselves stress that the situation is not as dire as it might seem. Long-established techniques already partly protect against the first problem. Label smoothing keeps the correct answer from reaching 100% confidence. Low temperature also prevents the model from becoming overly confident. In such cases, the gradient does not get the chance to reach the zone where rounding zeroes it out.

But these techniques treat the symptom, not the cause. They do not fix the computational error itself. Moreover, they change the training objective. Label smoothing deliberately makes the model less categorical, which can get in the way where honest probability calibration matters.

What the authors propose

The solution turns out to be simple: rewrite the formula, that is, change the order of operations. The mathematical result stays the same, but the computer stops losing small numbers. No new layers, different architecture, or additional hyperparameters are required. It is a fix at the level of how the function is implemented.

For developers, there are two practical takeaways:

  • If you write your own kernel for softmax or cross-entropy, for example in CUDA or Triton, or train models in low precision, check how it handles small probabilities.
  • If you use label smoothing or temperature out of habit, remember that they may mask a numerical problem rather than remove it.

What we still don't know

The summary of the research contains no figures. It is unclear how much the fix improves model quality, which tasks show a noticeable effect, and how it plays out in large language models. Popular frameworks may already account for some of these subtleties. To judge the real gain, you need to read the paper itself and examine the experiments.

Still, the finding is telling. Even in the most basic building blocks of AI, there are places where imprecise arithmetic quietly affects training. The fix is cheap, yet it potentially touches everyone who trains neural networks. Those who work on training and on custom kernels should at least skim this work.

Story

Key Questions Answered

Where the gradient disappears?

The gradient is the signal that says which way, and by how much, to adjust the weights. The first error appears when the model becomes too confident in the correct answer. Suppose it is 99.999999% sure a photo shows a cat. On paper, it still lacks a tiny bit to reach the ideal, so the gradient remains small but nonzero. The computer, however, rounds it down to zero prematurely: for it, the confidence already equals one, and there seems to be nothing left to learn.

How bad it is?

The researchers themselves stress that the situation is not as dire as it might seem. Long-established techniques already partly protect against the first problem. Label smoothing keeps the correct answer from reaching 100% confidence. Low temperature also prevents the model from becoming overly confident. In such cases, the gradient does not get the chance to reach the zone where rounding zeroes it out. But these techniques treat the symptom, not the cause. They do not fix the computational error itself.

What the authors propose?

The solution turns out to be simple: rewrite the formula, that is, change the order of operations. The mathematical result stays the same, but the computer stops losing small numbers. No new layers, different architecture, or additional hyperparameters are required. It is a fix at the level of how the function is implemented.

ZK
Zhemal Khamidun — CPO at Alpina Digital, product manager of AlpinaGPT. Building AI products since 2019. Hamidun News is a pet project that grew into a production aggregator with 6,000+ readers.
✦ Unified AI Workspace · Nomad

Test Frontier LLMs & Autonomous Agents in One Workspace

Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.

Try Nomad AI →Launch Nomad Agent
What do you think?
Loading comments…