Cost and Latency Levers: Model Tier, Tokens, and Volume

The three inputs every cost model needs

Before an architect commits to a configuration, they build a cost and latency model, and the exam wants you to know what feeds it. Three inputs: call volume (how many requests per day or per month), token budget per request (input tokens plus expected output tokens), and model tier. From those three you can estimate monthly cost and worst-case latency and check them against a budget ceiling and an SLA before any code is written. Everything else in the cost-latency topic, caching, batching, tier choice, is a lever that moves one of these three.

Treating cost as a first-class design input, computed up front rather than discovered on the first bill, is the mindset this knowledge point installs. The three inputs are the vocabulary for that conversation.

Input and output tokens are priced differently

A cost model that lumps all tokens together is wrong from the start. Input tokens (the system prompt, retrieved context, and user message) and output tokens (the model's response) are priced at different rates, and those rates differ across model tiers. So the estimate has two multiplications, not one: input token count at the input rate, output token count at the output rate, summed, then multiplied by volume. A workload that is input-heavy (long context, short answers) and one that is output-heavy (short prompt, long generations) can have very different costs even at the same total token count, because the mix hits different rates.

The tier dimension compounds this. Moving between tiers changes both the input and output rates, which is why tier selection is a genuine cost lever and not just a quality choice. A cheaper tier can bring an over-budget configuration into range purely through its lower per-token rates, if the task tolerates it.

The max_tokens cap is a ceiling, not a quality dial

A specific misconception the exam targets: raising the max_tokens cap does not make outputs better. The cap sets the maximum length the model may generate; it does not instruct the model to produce more or better content. What raising it actually does is raise the worst-case: a request that runs to the higher ceiling costs more and takes longer, so a generous cap inflates worst-case latency and cost without buying quality. Quality comes from the prompt, the tier, and the task design, not from headroom the model may never use, and when it does use it, you pay for it.

The practical reading is to set max_tokens to what the task genuinely needs plus a sensible margin, not to a large value "for safety." A large cap is a latent cost and latency risk that surfaces on exactly the long requests that already hurt most.

The skewed tail dominates spend

The subtlest input is the shape of the token distribution, not its average. Token counts are usually skewed: most requests are short, but a tail of long requests consumes a disproportionate share of total tokens and therefore total cost. A cost model built on the average token count assumes a uniform distribution that rarely exists, and it can understate real spend by a large factor, often two or three times, because the expensive tail is averaged away. The average request is cheap; the total bill is driven by the long ones.

So a sound cost model looks at the distribution, models the typical case and the tail separately, and sizes against both. This is the same tail-versus-average lesson that makes p95 the right latency target: in both cost and latency, the tail is where the design gets decided.

What the exam trips candidates on

Two traps. The first is assuming a higher output token cap always improves response quality; it does not, it only raises worst-case cost and latency. The second is modelling cost from average token counts alone, ignoring that a small share of long requests can dominate total spend. The credited answer separates input and output rates, sizes max_tokens to need rather than to safety, and models the token distribution rather than its average.

Common misreadings to avoid

How this shows up on the exam

Expect a sizing scenario with a blended rate, an average-token assumption, or a large 'safety' max_tokens, and a question about what the cost model gets wrong. The reliable reading is three inputs (volume, token budget, tier), separate input/output rates, a cap that is a ceiling not a quality dial, and a distribution modelled with its tail. This knowledge point is the base of the cost-latency topic: it unlocks prompt caching, p95 latency as the design target, and batch processing, and it feeds the capstone on justifying a configuration under dual constraints.