Loss Function

A loss function is a mathematical function that measures the discrepancy between a model's predictions and the true target values, producing a scalar score that optimization algorithms minimize during training.

A loss function (also called a cost function or objective function) quantifies how well a machine learning model's outputs match desired targets. It maps model predictions and corresponding ground-truth labels to a single non-negative scalar; the lower the value, the closer the predictions are to the targets. Optimization during training consists of finding the model weights that minimize this scalar.

The choice of loss function depends on the task. For regression, mean squared error (MSE) penalizes large deviations quadratically. For binary classification, binary cross-entropy measures the log-likelihood of correct class assignments. For multi-class problems, categorical cross-entropy is standard. Specialized tasks use custom losses: object detection models often combine a localization loss with a classification loss; diffusion models use a denoising score-matching objective; language models minimize cross-entropy over next-token predictions, equivalent to maximizing the log-likelihood of the training corpus.

The loss function is central to both model design and theoretical analysis. It encodes the inductive bias of the learning system—what "good" means for a given task—and its landscape over parameter space determines the difficulty of optimization. A poorly chosen loss function can produce models that optimize the proxy metric while failing on the real task, a dynamic sometimes framed as Goodhart's Law applied to machine learning.

In large language model training as of 2026, standard next-token cross-entropy loss is commonly supplemented by reinforcement learning from human feedback (RLHF), where a learned reward model replaces or augments the base loss to align outputs with human preferences. Research into loss functions that better capture semantic quality, factual accuracy, and safety properties is ongoing at labs including Anthropic, Google DeepMind, and OpenAI.