Overfitting

Overfitting occurs when a machine learning model learns the training data too closely—including its noise and idiosyncrasies—resulting in high accuracy on training examples but poor generalization to unseen data.

Overfitting is a failure mode in which a model captures statistical patterns specific to its training set rather than the underlying data-generating distribution. The model effectively memorizes training examples instead of learning transferable features, so its training loss is very low while its validation or test loss is substantially higher. This gap between training and held-out performance is the diagnostic signature of overfitting.

Overfitting is more likely when model capacity—the number of parameters or the expressiveness of the function class—is high relative to the amount of training data. A deep neural network with millions of parameters trained on a small dataset can fit training examples exactly while performing near chance on new inputs. The bias-variance tradeoff formalizes this: high-capacity models have low bias (they can represent complex functions) but high variance (their outputs are sensitive to the specific training sample used).

Standard countermeasures include regularization techniques such as L2 weight decay, dropout, and batch normalization; data augmentation to artificially expand training diversity; early stopping based on validation-set loss; and reducing model size. Cross-validation is the standard diagnostic tool for detecting overfitting before deployment.

An important nuance emerged from the deep learning era: very large models sometimes exhibit "double descent," where test error initially rises with model complexity (classical overfitting) but then falls again at very high parameter counts. Foundation models with hundreds of billions of parameters trained on internet-scale corpora often generalize well despite being capable of near-exact memorization, challenging classical intuitions. Understanding the conditions under which scale suppresses overfitting remains an active research question as of 2026.