Anomaly Detection Thresholds and Alerting

Two kinds of change need two kinds of detection

Once metrics are being captured and decomposed, you need something watching them, but not everything worth catching looks the same. Some changes are sudden, a cost spike, a latency jump, and some are gradual, a slow shift in output quality over weeks. The apply-level skill here is matching the detection method to the shape of the change: threshold alerts for sudden shifts, distribution comparison for gradual drift, and using each metric's own baseline rather than one generic rule for all of them.

Threshold alerts for sudden shifts

Threshold alerting is the right tool for changes that happen suddenly. You set a line on a metric that matters and fire an alert when the metric crosses it: a cost spike that exceeds, as a worked example, 150% of the trailing 7-day average, or a p95 latency that crosses the SLA threshold. These work well because the failures they target are step changes, a spike or a breach that clearly exceeds the recent baseline, so a threshold catches them promptly. The specific figure is a design choice, but pairing a concrete multiple with a defined trailing window, rather than a bare "watch the cost," is what turns an intention into an alert that actually fires.

The key discipline is that each metric needs its own baseline and its own threshold logic. A cost spike and a latency SLA breach are different phenomena, measured on different scales against different baselines, and a single generic threshold cannot serve both. Cost alerts key off the trailing average; latency alerts key off the SLA line. Setting one blanket threshold for everything either misses real anomalies on one metric or floods you with false alarms on another.

Distribution comparison for gradual drift

Threshold alerts have a blind spot, and it is exactly the failure mode LLM systems are prone to: model drift, a gradual change in the output distribution on stable inputs over time. Because drift is slow, it rarely crosses a sudden threshold in a single step, it creeps below the alert line, day after day, never triggering, until the cumulative shift is large. A threshold tuned to catch spikes simply will not fire on a slow creep.

Catching drift needs a different method: periodic distribution comparison. Rather than waiting for a line to be crossed, you compare the output distribution now against the distribution from an earlier period, and a gradual shift shows up as a growing difference between the two even though no single day tripped a threshold. So the two methods are complementary, thresholds for sudden shifts, distribution comparison for gradual drift, and a monitoring design that uses only thresholds is blind to drift by construction.

Alerts point to attribution

Detecting that a metric moved is only the first half; the second half is knowing why. Change-attribution instrumentation should be able to distinguish the three causes a metric can move for, model drift (gradual behaviour change on stable inputs), data drift (the input distribution changed), and model update effects (the version changed), because each demands a different mitigation. An alert that fires without the instrumentation to attribute the cause leaves the team guessing, and guessing defaults to the wrong fix. Good alerting is wired to the change-attribution evidence that turns "something moved" into "here is what moved and why."

What the exam trips candidates on

The first trap is relying only on hard threshold alerts to catch model drift, when drift is gradual and rarely crosses a sudden threshold. A scenario will show a team expecting a threshold to catch a slow quality decline; the credited reading explains that drift creeps below the alert line and must be caught by periodic distribution comparison instead.

The second trap is setting a single generic alert threshold for both cost spikes and latency SLA breaches, when each requires its own baseline and threshold logic. A scenario will apply one blanket rule to different metrics; the correct answer separates them, keying cost off the trailing average and latency off the SLA, each with its own threshold.

Common misreadings to avoid

How this shows up on the exam

Domain 4 questions on this knowledge point present a threshold expected to catch drift, or one generic threshold across different metrics. The reliable moves are to use threshold alerts with per-metric baselines for sudden shifts, add distribution comparison for gradual drift, and wire alerts to change-attribution so a firing alert points toward the right cause.

This builds on request-level tracing and aggregate metrics, which supply the metrics being watched, and on the four logging categories beneath them. The drift-versus-sudden distinction is the detection side of change attribution, latency alerting targets the p95 design target, and it complements discernment, which catches quality problems no metric alert can express.