Change Attribution: Model Drift vs Data Drift vs Model Update

Three reasons a metric can move

A monitored metric slips, task success falls, cost rises, quality degrades, and the pressure is to fix it fast. But there are three fundamentally different reasons the metric could have moved, and each demands a different mitigation. Reaching for a fix before attributing the cause is how teams apply a data-drift remedy to a model-version problem and wonder why nothing improves. This evaluate-level knowledge point is about telling the three causes apart, so the mitigation matches the actual cause rather than the assumed one.

Model drift: same inputs, shifting behaviour

Model drift is a gradual change in the model's behaviour on stable inputs over time. The key qualifier is that the inputs are held constant, if you fed the model the same inputs it saw before, the outputs would gradually differ. Because it is gradual, model drift rarely trips a sudden threshold alert; it creeps, which is why it is caught not by simple thresholds but by periodically comparing the output distribution over time. When the same inputs start producing measurably different outputs, drift is the candidate.

Data drift: same model, shifting inputs

Data drift is the mirror image: the model is unchanged, but the distribution of inputs it receives has shifted. Users start asking different kinds of questions, documents take on new formats, a seasonal pattern changes the mix. The model behaves exactly as it always did on each input; it is just now seeing inputs it was not optimized for, so the aggregate metric moves. Here the behaviour is constant and the inputs are the variable, the exact opposite of model drift. The mitigation is different too, you address the inputs (expand coverage, adapt retrieval, update the eval set to the new distribution) rather than the model.

Model update effects: a discrete version change

Model update effects are not drift at all in the gradual sense. They occur when the model version changed, an update, a swap, sometimes an unannounced one, and the new version behaves differently on the existing, unchanged inputs. The signature is discrete rather than gradual: the metric steps at the moment of the version change rather than creeping. The mitigation is operational, pin model versions, monitor deprecation notices, and re-run the eval suite whenever the version changes, so a version-driven behaviour change is caught and validated rather than mistaken for drift.

Why misattribution produces the wrong fix

The whole point of separating these is that their mitigations do not transfer. Expanding input coverage does nothing for a metric that moved because the model version changed; pinning a version does nothing for a genuine input-distribution shift; comparing output distributions over time addresses gradual model drift but not a discrete update. Defaulting to "it must be model drift" whenever a metric moves is the central error, because it applies the drift mitigation to causes that are actually data drift or a version update, and the metric stays broken. Correct attribution, checking whether the inputs changed, whether the version changed, or whether the same inputs are drifting gradually, is what points to the right fix.

What the exam trips candidates on

The first trap is diagnosing every metric regression as model drift by default, without checking whether the input distribution or the model version changed instead. A scenario will show a metric dropping and a team assuming drift; the credited reading insists on checking all three causes, was it the inputs, the version, or genuine gradual drift, before choosing a mitigation.

The second trap is applying a prompt-level fix intended for data drift to a regression that was actually caused by an unannounced model version update. A scenario will show a metric stepping down right when a version changed, with the team tweaking prompts as if the inputs had shifted; the correct answer attributes it to the model update, pins the version, and re-runs the evals rather than chasing the inputs.

Common misreadings to avoid

How this shows up on the exam

Domain 4 questions on this knowledge point describe a metric that moved and offer a default diagnosis, usually model drift, or a mismatched fix. The reliable approach checks the signature (gradual creep versus a version-aligned step), verifies whether inputs or the version changed, and applies the mitigation that matches the actual cause: distribution comparison for drift, coverage and retrieval changes for data drift, version pinning and eval re-runs for model updates.

This is the evaluate-level capstone of the diagnosis task statement, extending fixing model mismatch, whose unannounced-swap concern is the model-update cause here, and building on orchestrator-workers failure attribution as another exercise in tracing a symptom to its true cause. Detection depends on anomaly detection and alerting, and misattribution here rhymes with the false-confidence failures in a stale eval suite and an underpowered experiment.