Prompt Caching for Cost and Latency Reduction

The biggest cost lever, with conditions

When a cost model comes in over the ceiling, prompt caching is often the most powerful lever available, but it is a lever with conditions, and applying it where the conditions do not hold either saves nothing or introduces a correctness risk. This apply-level knowledge point is about knowing when caching genuinely reduces cost and latency, how its pricing actually works, and the two situations, low reuse frequency and live-state content, where it backfires.

When caching pays off

Caching works by preserving the processed prefix of the prompt, so the model does not reprocess those tokens on every request. That means the win comes from two things multiplied together: how long the cached prefix is, and how often it is reused. A long, stable system prompt hit across many requests is the ideal case, lots of tokens saved, many times over. The prefix has to be stable, though: if the cached portion changes between requests, there is nothing to reuse, and the benefit disappears.

So the profile for caching is specific: a long system prompt (or other prefix) that stays the same across a high volume of requests. That is exactly the shape of many production deployments, which is why caching is such a common and effective cost lever, but it is not automatic.

The pricing has two rates and a write cost

Caching does not make the cached tokens free; it reprices them. On the first request that establishes the cache, the cached tokens incur a cache write cost that is higher per token than standard input, because writing the cache is extra work. On every subsequent request that hits the cache, those same tokens are billed at the cache read rate, which is much lower than the standard input rate. The net saving is the difference between paying standard input every time and paying one elevated write plus many cheap reads.

This is why the cost model has to account for the write cost on first use, not just the read savings. If you only model the cheap reads, you overstate the benefit; the honest calculation nets the one-time write premium against the repeated read savings, and the more reuse there is, the more the reads dominate and the bigger the net win.

Two ways caching backfires

The first way caching disappoints is when reuse is too infrequent relative to the cache TTL. The cache entry lives only for a short time, a default of roughly five minutes, and then expires. If requests arrive less often than the TTL, every request finds an expired cache and pays the write cost again, never reaching the cheap reads that make caching worthwhile. A workload has to be frequent enough, relative to the TTL, to keep hitting a live cache, or the "savings" are just repeated write premiums.

The second way is a correctness problem, not a cost one. Cached content is frozen for the life of the cache entry, so if you cache content that must reflect current, live state, the served responses can be stale for the duration of the consistency window. For a use case that depends on up-to-the-moment data, that staleness can be unacceptable, caching has traded correctness for cost. The rule is to cache stable content (the unchanging system prompt), not live-state content that has to be current. (TTL length and the exact read/write rates are set by current pricing and should be confirmed when modelling.)

What the exam trips candidates on

The first trap is assuming prompt caching always reduces cost, without checking that request frequency is high enough relative to the cache TTL to actually reuse the cache. A scenario will apply caching to a low-frequency workload; the credited reading points out that if requests arrive less often than the TTL, the cache expires between uses and only the write premium is paid, so caching adds cost rather than saving it.

The second trap is caching content that needs to reflect current, live state, introducing stale-data risk inside the cache's consistency window. A scenario will cache a prefix containing live data; the correct answer flags the consistency window and restricts caching to the stable portion of the prompt, keeping live-state content out of the cache.

Common misreadings to avoid

How this shows up on the exam

Domain 4 questions on this knowledge point present caching applied to a low-frequency workload or to live-state content. The reliable moves are to confirm the prefix is long and stable and reused frequently relative to the TTL, to model the write-once/read-cheap pricing honestly, and to keep live-state content out of the cache to avoid the consistency-window risk.

This is the main optimization lever built on cost and latency modeling inputs, and it directly attacks the input-cost side that token distribution skew inflates. Its latency benefit connects to p95 latency as the design target, and caching sits alongside the other production controls in layered reliability controls as part of a well-engineered call path.