How ADWIN and Online Learning Help Models Cope with Data Drift

A machine learning model can post excellent metrics on tests and quickly lose accuracy in production when user behavior, demand or the market changes. The phenomenon is called concept drift, and a Habr article looks at how to fight it by combining online learning with the ADWIN algorithm.

What is concept drift

Concept drift is a situation in which the statistical patterns a model was trained on gradually stop matching reality. A model that showed high accuracy on historical data starts making mistakes not because it was poorly trained, but because the world it describes has changed: users behave differently, demand shifts, the market reacts to new factors.

• The problem: the model loses accuracy in production while test metrics stay unchanged
• The cause: user behavior, demand or market conditions change over time
• The weak spot of the classic approach: scheduled retraining often lags behind the moment drift has already begun
• The proposed solution: combining online learning with the ADWIN algorithm

Why scheduled retraining does not solve it

The standard approach is to retrain the model from scratch on fresh data at regular intervals. The problem is that the retraining schedule is set in advance and is not tied to the actual moment the data began to change. A model may run on "outdated knowledge" from one retraining cycle to the next, losing accuracy precisely when changes happen faster than the schedule anticipates.

How pairing with ADWIN helps

ADWIN (ADaptive WINdowing) is an algorithm that tracks the statistics of an incoming data stream in a sliding window and signals when the data distribution has changed in a statistically significant way. Combined with online learning, in which the model is updated incrementally on new examples rather than retrained from scratch, this lets the model adapt to new data without fully discarding its accumulated knowledge. In other words, it can react to drift almost immediately after detecting it, instead of waiting for the next scheduled retraining cycle.

How this differs from classic retraining

The difference between the two approaches lies in reaction speed and in how much knowledge is lost. When a model is retrained from scratch, it effectively starts over on a new slice of data, and everything it learned from older but still relevant patterns is lost. Incremental online learning with a drift detector such as ADWIN, by contrast, keeps the accumulated knowledge and only adjusts the model in response to a confirmed statistical change. This lowers the computational cost of constant retraining and at the same time shortens the time the model spends stuck with "outdated" knowledge.

What this means

For teams that run ML models in production, concept drift is a constant source of silent quality degradation that test-set metrics will not reveal. Combining online learning with drift detection via ADWIN is a practical way to narrow the gap between the moment the data changed and the moment the model accounted for it, without having to maintain a separate full retraining cycle for every hiccup in production.