Grokking

Grokking is a delayed generalization phenomenon in neural network training where a model first memorizes training data, then — after prolonged training well past apparent overfitting — abruptly learns the underlying rule and achieves strong generalization.

Grokking is a training dynamics phenomenon in which a neural network transitions, often abruptly and after many additional gradient steps, from a memorization regime (near-perfect training accuracy, poor validation accuracy) to genuine generalization on held-out data. The term was introduced by Alethea Power and colleagues at DeepMind in a 2022 paper studying small transformer models trained on modular arithmetic tasks.

In the original experiments, models memorized the training set quickly but showed no validation improvement for thousands of additional steps — then suddenly generalized nearly perfectly. The transition is thought to involve a structural shift in internal representations: a high-norm "memorization circuit" is progressively replaced by a more weight-efficient "algorithmic circuit" encoding the true rule. Weight decay, by penalizing large-norm solutions, accelerates this shift and is a reliable way to induce grokking in practice.

Grokking matters because it challenges the conventional early-stopping heuristic and the assumption that continued training after overfitting is permanently harmful. It demonstrates that regularization combined with sufficient compute can produce qualitative jumps in generalization even when standard loss curves appear to have stagnated, reframing overfitting as a potentially transient rather than permanent failure mode.

As of 2026, grokking remains an active area of theoretical research. Subsequent work has documented similar phase transitions in larger models and more naturalistic settings beyond toy arithmetic. It has become a key reference point in discussions of emergent capabilities in large language models, double descent phenomena, and the mechanisms by which structured knowledge is consolidated during neural network training.