Model drift: why an AI system that works well today fails tomorrow

A model that was 92% accurate at delivery can drop to 78% a year later, without a single line of code changing. The problem isn't in the model, it's in the world around it.

No bug, no incident, no deployment that broke something. An AI model rarely fails because it was built wrong yesterday. It fails because the world in which it must decide today is no longer the same world it learned from.

Two types of drift

// Data drift versus concept drift

Data driftinput changes shape

Concept driftthe input-outcome relationship changes

Data drift means the model sees new situations that deviate from what it learned. The input changes: different customer groups, new products, changing behaviour.

Concept drift means the meaning of that input changes. The signals the model could previously rely on no longer predict the outcome in the same way.

That's why AI testing shifts from a one-off quality check to continuous quality assurance.

How you detect drift

Statistical monitoring of the input data

Continuously compare the distribution of incoming data with the training data. A significant deviation is an early warning, even before performance visibly declines.

Delayed performance monitoring

Where possible, compare predictions against the actual outcome after the fact. That gives the most direct signal, but often only arrives after a delay.

Proxy metrics when real labels are missing

When the actual outcome is only known much later (for example with credit risk), you use interim signals to suspect drift before it's confirmed.

A drift measurement isn't a goal in itself. The question isn't "has our data changed?", but "is that change large enough to reduce trust in the predictions?"

Tools like Evidently AI and NannyML are built specifically for this: they track the statistical distance between production data and training data, and flag when it crosses a threshold.

From one-off validation to continuous assurance

Decide in advance what drift is acceptable and what requires reactivating the validation process.
Set up monitoring as part of the production pipeline, not as a separate dashboard nobody consults.
Plan retraining as a process, not a last resort. Those who only retrain once complaints come in are structurally behind the problem.
Document every retraining as a new validation cycle: an updated model is a different model, with its own testing obligation.

For high-risk systems under the EU AI Act, post-market monitoring isn't a recommendation but an obligation (Article 72). Continuous monitoring of model behaviour and potential drift is an important way to fulfil that obligation.

An example from practice

A recommendation model at an e-commerce client performed well until the company acquired a new product category through an acquisition. Within six weeks, the click-through rate on recommendations dropped by nearly a third, with no release, bug, or incident to point to.

// What the monitoring showed

Input data drift (week 1-2)slightly elevated

Input data drift (week 5-6)well above threshold

Click-through rate drop-31%

Monitoring flagged the data drift two weeks before the click-through rate visibly dropped. Without that early warning, the problem would only have surfaced through customer complaints, weeks later.

What we set up in practice at clients

A defined drift threshold, agreed with the business beforehand: what level of deviation triggers an investigation, what level triggers automatic retraining?
Monitoring as part of the pipeline, with clear ownership: who gets the alert, and what's the expected response time?
A fixed validation cycle for every retraining, even when it happens "automatically": an updated model goes through the same tests as a new one.

// Standing still isn't stability

Do you know if your model still performs today as it did at delivery?

We set up drift monitoring and show you when a model is due for revision, before customers notice.

Book a call