- In short
- When a metric moves, three distinct causes must be told apart: model drift (the model's behaviour on stable inputs changed), data drift (the input distribution itself changed, not the model), and model update effects (the model version changed and the new version handles existing inputs differently). Each cause requires a different fix, so instrumentation must distinguish them rather than simply flagging that a metric moved.
Three causes behind one moved metric
Detecting that a metric moved is only half the job; the other half is attributing why. The exam names three distinct causes that can move the same metric, and the whole point of the knowledge point is that they are genuinely different and demand different fixes. Model drift is the model behaving differently on inputs that have not changed. Data drift is the inputs changing while the model stays the same. Model update effects are a version change producing different behaviour on existing inputs. A dashboard that only says 'quality dropped' cannot tell you which of these happened, and applying the wrong fix, rolling back a version when the data actually shifted, wastes effort and leaves the real cause in place.
So instrumentation has to be built to distinguish the three, not merely to detect movement. That means capturing enough about inputs, model versions, and behaviour over time to separate 'the model changed,' 'the inputs changed,' and 'the version changed.'
- Change attribution
- The discipline of attributing a moved metric to one of three distinct causes, model drift (behaviour changed on stable inputs), data drift (the input distribution changed), or model update effects (the version changed and handles existing inputs differently), because each requires a different fix and cannot be told apart by flagging movement alone.
Telling the three apart
The three causes are distinguished by asking what actually changed. Model drift: the inputs are stable and the version is unchanged, yet behaviour has shifted, this is the gradual drift that periodic distribution comparison catches. Data drift: the model and version are unchanged, but the input distribution has moved, users are asking different questions, documents have changed shape, so the same model faces a different workload. Model update effects: the version changed, and the new version handles the existing inputs differently, so the movement is tied to the update event and shows on inputs that previously produced other results.
The instrumentation to separate them follows from the four-layer logging model. The request layer records the model version, so a movement can be lined up against version changes to spot update effects. Input characteristics over time reveal data drift. Output distribution on a held-stable input set reveals model drift. With all three tracked, a moved metric can be attributed rather than guessed at.
Each cause has its own fix
Attribution matters because the fixes diverge. If the cause is a model update effect, the response is to evaluate the new version and, if it regresses, roll back or re-tune against it, a version decision. If the cause is data drift, rolling back the version does nothing, because the model never changed; the fix is adapting to the new input distribution, updating prompts, retrieval, or handling for the shifted inputs. If the cause is model drift on stable inputs, the fix is re-grounding or re-evaluating the model's behaviour. Apply the model-update fix to a data-drift problem and you have changed the one thing that was not broken while leaving the actual change, the inputs, unaddressed.
This is why 'a metric moved' is never an actionable conclusion on its own. The action depends entirely on which of the three causes is responsible, and only attribution gets you there.
What the exam trips candidates on
Two traps. The first is applying a model-version rollback when the real cause is a shift in the input distribution, a mismatched fix that changes the model when the data changed. The second is assuming any metric movement following a model provider update is automatically a model update effect without checking for concurrent data drift, jumping to attribution without ruling out a coincident input change. The credited answer distinguishes the three causes with instrumentation before choosing a fix.
Worked example
A support system's answer-quality metric drops the same week a new model version is deployed. The team concludes it is a model update effect and prepares to roll back. Before they do, a colleague notes that a large new customer segment, asking a very different mix of questions, onboarded that same week. How should the attribution proceed?
The team is about to fall into both traps at once. They saw a metric drop coincide with a model-version deployment and jumped straight to 'model update effect,' which is the second trap, assuming that movement after a provider update is automatically an update effect. But the colleague has identified a confound: a large new customer segment with a very different question mix onboarded the same week, which is a textbook data-drift event, the input distribution changed while the model's behaviour on old inputs may be unchanged. Two candidate causes overlap in time, so the coincidence with the version deployment is not proof of an update effect.
Proper attribution separates them with instrumentation rather than assuming. Re-run a held-stable set of the old segment's representative inputs against both the old and new versions: if quality on those unchanged inputs is the same across versions, the update did not degrade behaviour and the drop is data drift from the new segment; if quality on those same inputs is worse on the new version, that is a genuine model update effect. Checking the input distribution over the week will separately confirm the data shift. The fixes then diverge accordingly: if it is data drift, rolling back the version, the first trap, would change the one thing that was not broken and leave the new segment mishandled, so the right fix is adapting prompts and retrieval to the new question mix; if it is truly an update effect, then evaluating and rolling back or re-tuning the version is correct. The point is that the rollback decision must follow attribution, not precede it.
Common misreadings to avoid
Misconception
When quality drops, rolling back to the previous model version is a safe first response.
What's actually true
Misconception
A metric moving right after a model provider update is, by definition, a model update effect.
What's actually true
How this shows up on the exam
Expect a moved metric with two plausible causes overlapping in time, often a version change and an input-distribution change, and a question about how to attribute it. The reliable reading is to distinguish model drift, data drift, and model update effects with instrumentation before choosing a fix. This knowledge point builds on anomaly detection and model drift, relies on the four-layer logging model for the version and input signals, and unlocks the failure taxonomy.
A quality metric drops the same week a new model version ships and a new customer segment with a different question mix onboards. What is the correct first step?
People also ask
What are the three causes of a moved metric?
How is model drift different from data drift?
Why must instrumentation distinguish these causes?
Watch and learn
Official Anthropic Academy lessons first, then hand-picked walkthroughs. Videos load only when you press play.
No videos curated for this concept yet
We are still curating the best official and community videos for this topic.
Official prep for this domain
Anthropic's own free prep module for this part of the syllabus, on the official prep course. Free with an Anthropic Academy sign-in.
References & primary sources
Master this concept with Archie
Practice it inside an adaptive study session. Archie, your Socratic AI tutor, tracks your mastery with Bayesian Knowledge Tracing and schedules the perfect next review.