Anomaly Detection Thresholds and Model Drift

Two shapes of change need two detection methods

Not every problem announces itself the same way, and the exam's applied point here is that the detection method has to match the shape of the change. Some changes are sudden and sharp: cost jumps, latency crosses the SLA line. Threshold alerts, fire when a metric crosses a set boundary, are exactly right for these. Other changes are gradual: the model's output distribution drifts slowly over weeks on inputs that have not changed. Thresholds are exactly wrong for these, because a slow drift never spikes across a line. Detecting drift needs periodic comparison of output distributions instead. Using one method for both leaves half your failures unmonitored.

The skill is diagnostic: given a described change, decide whether it is a sharp event a threshold catches or a slow drift that needs distribution comparison, and instrument accordingly.

Thresholds catch the spikes

A threshold alert watches a metric and fires when it crosses a boundary: cost exceeding, say, 150% of its rolling average, or p95 latency crossing the SLA. These are the right tool for sudden, sharp changes because such changes cross the boundary decisively and the alert fires at the moment it matters. A misconfigured deployment that doubles cost, a dependency slowdown that pushes latency over the SLA, a runaway loop, all trip thresholds cleanly. For anything that moves fast enough to spike, thresholds give timely, actionable alerts.

The value of thresholds is their immediacy: they turn a sharp movement into an alert in near real time. The cost is that immediacy depends on the change being sharp. A change that arrives slowly, spread over many small increments none of which crosses the line, is invisible to a threshold no matter how significant its cumulative effect.

Drift needs distribution comparison

Model drift is the canonical slow change: a gradual shift in the distribution of the model's outputs over time, on inputs that are themselves stable. Nothing spikes. Each day's outputs look a little different from a month ago, but no single day crosses a threshold, so threshold alerting sees nothing while quality slowly degrades. Catching drift requires a different instrument: periodically comparing the output distribution now against the distribution from an earlier baseline, and flagging when they have diverged beyond a tolerance. This is a comparison of distributions over time, not a single-point check against a line.

So the two methods are complementary, not interchangeable. Thresholds watch for the sharp events; periodic distribution comparison watches for the slow drift. A monitoring design that only has thresholds is blind to drift, and one that only compares distributions is slow to catch spikes. This drift-detection idea recurs in monitoring retrieval quality, where retrieval metrics drift as the index and corpus change.

What the exam trips candidates on

Two traps. The first is relying solely on threshold alerts and assuming they will catch slow, gradual quality degradation, which they structurally cannot. The second is confusing a sudden cost spike, which is threshold-detectable, with gradual model drift, which requires distribution comparison, and reaching for the wrong instrument. The credited answer matches the detection method to whether the change is a sharp event or a slow drift.

Common misreadings to avoid

How this shows up on the exam

Expect a scenario contrasting a caught sharp event with an uncaught slow decline, and a question about why thresholds missed the drift. The reliable reading is that thresholds catch spikes while gradual drift needs periodic distribution comparison. This knowledge point builds on aggregate vs per-request metrics, unlocks change attribution, which distinguishes the causes once a change is detected, and connects to monitoring retrieval quality.