What is Model Drift?

Two different failures with one name

Data drift (covariate shift) means the inputs changed. A new customer segment arrives, a marketing campaign shifts traffic mix, a sensor is recalibrated. The relationship between features and target still holds; the model is simply being asked about regions of input space it saw rarely in training.

Concept drift means the relationship itself changed. The same inputs now imply a different outcome — fraudsters adopt a new technique, consumer preferences move, a competitor's pricing changes what a given signal means. This is the more serious case, because no amount of input monitoring reveals it and only outcomes do.

A third, non-drift cause is worth ruling out first: an upstream pipeline change, a schema alteration, or a unit change. When accuracy drops suddenly rather than gradually, suspect a broken pipeline before concluding the world changed.

Detecting it

Input monitoring is cheap and immediate. Compare live feature distributions against the training distribution using population stability index, Kolmogorov-Smirnov tests, or simple percentile tracking. Alert on sustained shifts, not single-day noise. This catches data drift but is blind to concept drift.

Prediction monitoring watches the distribution of model outputs. If a fraud model's flag rate doubles with no known cause, something changed even if you cannot yet say what.

Outcome monitoring is the only true measure of accuracy, and it requires ground truth, which usually arrives late — a loan default is known months after the decision. Build the labelling loop deliberately: sample predictions for human review at a fixed rate so you have current labels, and design the product to capture natural feedback where possible.

In the meantime, proxy metrics carry real information: override rates by human reviewers, downstream conversion, complaint volume, and the model's own confidence distribution.

Responding well

Scheduled retraining on recent data handles gradual drift and is the sensible default for most systems. Choose the cadence from the observed rate of decay rather than from habit — quarterly is fine for many business processes, daily is necessary in adversarial settings such as fraud and abuse.

Triggered retraining fires when monitoring crosses a threshold. It responds faster and needs guardrails, because automatically retraining on drifted data can encode the very problem you are reacting to.

Two traps deserve naming. Feedback loops: if the model influences the data it later trains on — a recommender only observing outcomes for items it chose to show — retraining amplifies its own bias. Mitigate with deliberate exploration and by logging what was shown, not just what was clicked. And silent success: never deploy a retrained model on the assumption it is better. Validate it on a held-out recent period, compare on important slices, and roll it out behind a canary as described in MLOps. Retraining is a deployment, with all the same risks.