P95 Latency as the Design Target

Design for the requests that breach, not the ones that don't

A latency SLA is not breached by the typical request, it is breached by the slow ones. That single fact is why designing a system to its median latency is a mistake: the median describes the request in the middle, which was never going to breach anything, while the requests that do breach live out at the tail where the median cannot see them. This apply-level knowledge point is about using p95 as the design target, understanding why a single-request demo hides it, and knowing what actually drives latency so you optimize the right variable.

Why the median lies and p95 tells the truth

The median (p50) is the latency of the middle request, half are faster, half are slower. It is a fine description of a typical request and a poor description of the requests that cause problems. Since an SLA is breached whenever a request exceeds the threshold, and the requests that exceed it are the slow ones at the top of the distribution, the median simply does not describe the population you care about. Optimize to the median and you have built a system that is fast for the requests that were never at risk while leaving the tail free to breach.

P95, the value below which 95% of requests complete, deliberately reaches into that tail. It says: 95% of requests are at least this fast, and only the slowest 5% are worse. That is a target aligned with the SLA, because it constrains the part of the distribution where breaches actually happen. Designing to p95 (or a higher percentile for stricter SLAs) is designing for the requests that decide whether you meet the SLA.

A single-request demo cannot show you p95

There is a practical reason p95 is easy to miss: demos do not surface it. A demo typically issues one request at a time, with no competing load, so what you observe is close to a best-case single-request latency, essentially the median under no contention. Production sends many requests concurrently, and under that concurrency the tail stretches, some requests wait, some coincide with heavier ones, and p95 climbs well above the tidy number the demo showed. So a system that looked comfortably within its SLA in a demo can breach it in production, not because anything changed but because the demo never measured the metric that matters. P95 has to be measured under concurrent load that mimics production, not from one-at-a-time trials.

What actually drives latency

Optimizing latency requires knowing what moves it, and the drivers are task complexity, model size, and output length: harder tasks, larger models, and longer generated responses all take more time. Those are the levers, simplify the task, choose an appropriately sized model, cap or reduce output length.

What is not primarily a latency driver, at moderate request rates, is raw volume. High daily volume pressures cost, more requests, more tokens, more spend, but it does not automatically threaten latency if the requests are well spread and the system is not saturated. A large "requests per day" figure can average out to comfortably spaced requests that each complete quickly. Assuming high volume must breach a latency SLA confuses the cost axis with the latency axis; the two are driven by different things.

What the exam trips candidates on

The first trap is designing a system to meet a latency SLA using median latency figures instead of p95, missing the tail requests that actually breach the SLA. A scenario will present a design signed off on median latency; the credited reading switches the target to p95, measured under concurrent load, because the median hides the breaching tail.

The second trap is assuming a high daily request volume automatically threatens a latency SLA, when the load may actually average out to well-spaced sequential requests. A scenario will treat a big volume number as a latency problem; the correct answer separates the axes, noting volume primarily pressures cost, and that latency is driven by task complexity, model size, and output length, so a well-spread high volume need not breach the SLA.

Common misreadings to avoid

How this shows up on the exam

Domain 4 questions on this knowledge point present a latency SLA signed off on median or demo figures, or a high-volume number framed as a latency threat. The reliable moves are to design to p95 measured under concurrent load, to tune the real latency drivers (complexity, model size, output length), and to keep the cost axis (volume) separate from the latency axis.

This is the latency half of cost and latency modeling inputs, and its tail reasoning parallels token distribution skew, where the tail drives cost just as it drives latency here. Caching helps the latency side by not reprocessing the prefix, connecting to prompt caching, and p95 is exactly the kind of tail metric that anomaly detection and alerting watches against the SLA line, with the tail protected in production by layered reliability controls.