Model drift

Why does a model that nobody changed get worse over time?

A model is deployed. Its weights are frozen; the file's checksum is the same today as it was eighteen months ago. Nobody has retrained it, and the code around it has been stable. Its accuracy has fallen by nine points.

Something got worse while nothing changed. That sentence cannot be true as stated, so one of its two halves must be wrong — and the interesting question is which. What, exactly, was measured as "accuracy", and against what?

Here is the step that unlocks it. A model is not a description of the world; it is a description of a sample of the world, taken at a particular moment. Accuracy is not a property the model carries around with it — it is a property of the pairing between the model and the data it is currently being shown. Freeze one side of that pair and the other side keeps moving.

So the model is not degrading. The relationship is. Which raises the useful question: what can move on the data side?

Three things, and they are worth separating because they call for different responses.

The inputs can change while the underlying rule holds. Your fraud model was fitted when most transactions came from desktop browsers, and now most come from phones; the mix of inputs has shifted, and the model is being asked more questions from regions of the space where it saw little training data. This is covariate shift.

The base rates can change. A model that classified one call in twenty as an escalation now faces a population where one in five is, perhaps because a product change altered who calls. Its calibrated thresholds were tuned against the old proportion, so the same scores now imply something different. This is prior shift.

Or — and this is the deepest one — the relationship between inputs and outcomes can itself change. The word "cloud" once implied weather in your support tickets and now implies hosting. The same input now genuinely warrants a different answer. That is concept drift, and no amount of reweighting old data will fix it, because the old labels have become wrong.

Now, before reaching for any of these, I would ask a deflating question. Are you sure the world changed, and not your pipeline? In practice, the most common cause of a sudden accuracy drop is not the slow drift of human behaviour at all. It is that an upstream team changed a units field from miles to kilometres, or started sending an empty string where they used to send a null, or renamed a category, or replaced the service that generated one of your features. The model then receives inputs that mean something different from what its training assumed — and this arrives as a step change overnight, not as a gentle decline. Distinguishing the two by their shape over time is the fastest diagnostic available.

And there is one more source, the one that makes drift a story about path dependence rather than weather. What if the model's own outputs change the data it will later be trained on?

That last point deserves its own treatment, because it is the mechanism people miss. A model that acts on the world creates a feedback loop, and the loop biases the data.

A credit model approves some applicants and rejects others. Which applicants generate repayment outcomes you can later learn from? Only the approved. So the next training set is a sample selected by the current model's own beliefs, and the regions where it was wrongly pessimistic never get corrected, because it never sees the evidence. The same shape appears in recommendation, where the model determines what users are shown and therefore what they can click, and in fraud, where the adversary is deliberately searching for exactly the inputs your model handles badly.

Two misconceptions to correct gently. First, drift detected in the inputs is not the same as harm. Input distributions shift constantly for benign reasons, and a model can be entirely robust to a shift that a statistical test flags loudly. If you alert on input drift alone you will train your team to ignore alerts — so drift monitoring is a leading indicator whose value is that it arrives before the labels do, not a verdict.

Second, retraining is not automatically the remedy, and running it on a schedule can make things worse. If the recent data is contaminated by the feedback loop above, or if the labels are delayed and therefore mostly missing for the newest period, an automatic retrain will faithfully learn the bias. The honest practices are duller: keep a slice of randomly-handled traffic so you have unbiased outcomes to learn from, hold a stable reference set for comparison, watch the shape of the change to separate a pipeline break from real drift, and treat retraining as a change that must pass evaluation like any other.

• Detecting shift in practice — population stability indices, two-sample tests, and why they alarm more often than they should.
• Training-serving skew: how feature computation differs between the training pipeline and the live one, and why it looks exactly like drift.
• Exploration policies that keep a small randomly-handled slice of traffic, and the ethical questions that raises in lending and medicine.

Every term the collection defines is gathered in the glossary.