Free Intensive Nov 21–22Free 2-day hands-on Vibecoding Intensive on Nov 21–22 at 16:00 MSK (Sat–Sun) · Live Cohort 3 starts Nov 23Join Free →Cohort 3 (Nov 23)
ReleaseInsight · On RadarTG: @dealerAI (Dealer.AI)→ original

Softmax Rounding Bug Explains Grokking: After the Fix, the Sudden Generalization Disappears

Softmax turns model outputs into probabilities, and researchers have found two rounding bugs in its implementations in PyTorch, JAX, and TensorFlow. When the model is almost 100% confident, the gradient for the correct answer drops to zero while the gradients for wrong answers remain, and the balance breaks. In distillation, a student that already matches its teacher still receives a nonzero gradient. After the fix, grokking, the sudden generalization that follows overfitting, disappears, and standard LLM pretraining will barely change.

AI-processed from TG: @dealerAI (Dealer.AI); edited by Hamidun News
Softmax Rounding Bug Explains Grokking: After the Fix, the Sudden Generalization Disappears
Source: TG: @dealerAI (Dealer.AI). Collage: Hamidun News.
Listen to article
Article Fact Sheet & Key Figures
ParameterDetails
Primary sourceTG: @dealerAI (Dealer.AI) ↗
Editor-in-Chief
Key metric #1The precision cliff for bf16 is around 5.5, and it is reached quickly.
Key metric #2Barron joked that the default eps=1e-8 in Adam lands right on the fp32 cliff.
Key metric #3For roughly 90% of standard training the effect is nearly zero; the main gain is in distillation.

Softmax and cross-entropy appear in almost every training loop. Softmax converts a model's raw outputs (logits) into probabilities, and cross-entropy measures how wrong the model is. That error produces the gradient, the signal that tells the weights which direction to move. On October 22, a paper appeared claiming that standard implementations of these functions in all popular frameworks (PyTorch, JAX, and TensorFlow) contain two numerical bugs. The work sparked interest and debate, and it also called into question one of the best-known phenomena in neural network training: grokking.

The Two Bugs

The first bug is called Softmax Collapse. Consider a classifier that has become overconfident and assigns the correct answer a probability of nearly 100%. With limited numerical precision, that value rounds to one. As a result, the gradient for the correct answer becomes exactly zero, while the gradients for the incorrect options remain small but nonzero. Normally these gradients sum to zero, like two sides of a balanced scale. Here one side drops to zero and the other does not, so the balance breaks. Training receives a skewed signal.

The second bug, Softmax Cancellation, appears during distillation, when a small student model learns to reproduce a larger teacher model. As the student's logits approach the teacher's, the error should become zero. But because of numerical imprecision, the gradient stays nonzero even when the match is perfect. The student keeps adjusting itself to the scale of the teacher's distribution (its embeddings or probabilities), even though there is nothing left to adjust. It resembles a perfectionist who keeps tweaking a finished piece of work and ruins it.

What Happened to Grokking

The most striking consequence concerns grokking. This term describes a situation where a model first memorizes the training examples for a long time and then abruptly begins to generalize. Sharp spikes in the loss during this phase, so-called slingshot spikes, were considered a hallmark of this regime. After the fix, models that previously grokked stop doing so. In other words, slingshot spikes turn out to be an artifact of the softmax bug, not a mysterious phase transition.

Without the bug, the picture is more mundane. Either grokking speeds up with weight decay (a penalty on overly large weights), or without weight decay accuracy simply does not drop. Barron, the paper's author, joked that the default eps=1e-8 in the Adam optimizer lands exactly on the fp32 precision cliff, where the bug is triggered.

Who Should Care and Who Shouldn't

Barron himself writes in the paper that the extreme confidence margins and near-zero student-teacher discrepancies tested in the experiments are rare in typical large-scale training. Large language models learn to predict the next token during pretraining. Their distributions are far from "I am 99.999% certain," so logits rarely hit the fp32 limit. For roughly 90% of standard training, almost nothing changes.

Real effects are expected in four scenarios:

  • Distillation. A student approaching its teacher was receiving spurious nonzero gradients and drifting without cause. The fix removes this. For production distillation pipelines, which are common in industry, this is a clear improvement.
  • Training in bf16. The precision cliff for bf16 is only around 5.5, and it is reached very quickly. Nearly everyone trains in bf16 today, so the bug may trigger constantly. Here the fix delivers the largest gain in stability.
  • Fine-tuning and RLHF. When a model is adapted to a specific task, it becomes very confident. In RLHF with PPO or GRPO, where the model generates responses and receives scores, the bug may distort gradients in later stages.
  • Small datasets with weight decay. This is the classic grokking regime on toy tasks. The fix removes slingshots and speeds up convergence, but such setups are almost absent in industry.

What Not to Expect

The fix will not make GPT-6, Claude, or Gemini noticeably better. Their pretraining will not break without it, and it will not become several times stronger with it. It also does not change training or inference speed. It is not a new attention mechanism or a new activation function; it is a numerical correction.

The practical takeaway is this. If you distill models, train in bf16, or run late-stage RLHF, compare your loss curves between the standard and corrected implementations. If you are doing ordinary pretraining, there is no need to panic. And when reading articles about grokking, keep in mind that some of the effects may have been explained by rounding rather than by properties of the network.

Caveat: this is a summary of the work based on a Telegram channel post, not a full analysis of the paper. The authors are still estimating the size of the effect on real large-scale pipelines, and the community debate continues. The primary source is the arXiv preprint, linked in the original post.

Story

Key Questions Answered

What are the key details on The Two Bugs?

The first bug is called Softmax Collapse. Consider a classifier that has become overconfident and assigns the correct answer a probability of nearly 100%. With limited numerical precision, that value rounds to one. As a result, the gradient for the correct answer becomes exactly zero, while the gradients for the incorrect options remain small but nonzero. Normally these gradients sum to zero, like two sides of a balanced scale. Here one side drops to zero and the other does not, so the balance breaks.

What Happened to Grokking?

The most striking consequence concerns grokking. This term describes a situation where a model first memorizes the training examples for a long time and then abruptly begins to generalize. Sharp spikes in the loss during this phase, so-called slingshot spikes, were considered a hallmark of this regime. After the fix, models that previously grokked stop doing so. In other words, slingshot spikes turn out to be an artifact of the softmax bug, not a mysterious phase transition.

Who Should Care and Who Shouldn't?

Barron himself writes in the paper that the extreme confidence margins and near-zero student-teacher discrepancies tested in the experiments are rare in typical large-scale training. Large language models learn to predict the next token during pretraining. Their distributions are far from "I am 99.999% certain," so logits rarely hit the fp32 limit. For roughly 90% of standard training, almost nothing changes. Real effects are expected in four scenarios: Distillation.

ZK
Zhemal Khamidun — CPO at Alpina Digital, product manager of AlpinaGPT. Building AI products since 2019. Hamidun News is a pet project that grew into a production aggregator with 6,000+ readers.
✦ Unified AI Workspace · Nomad

Test Frontier LLMs & Autonomous Agents in One Workspace

Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.

Try Nomad AI →Launch Nomad Agent
What do you think?
Loading comments…