Cost and Latency Modeling Inputs

Know the numbers before you build the system

A system that behaves correctly can still be the wrong system if it cannot afford to run at production volume. The cost and latency model is the instrument that tells you, in advance, whether the design fits the budget, and the CCAR-P exam treats building it before any code as a core architect responsibility. This remember-level knowledge point establishes the three inputs the model needs and the arithmetic that turns them into a monthly cost you can check against the ceiling.

The three inputs

The model needs exactly three inputs. Call volume is how many requests the system will make per day or per month, and it comes from the business requirement, the actual number of conversations, documents, or transactions, not a developer's guess. Token budget per request has two parts: the input tokens (system prompt, retrieved context, and user message) and the expected output tokens (the response length). Model tier determines the per-token rates that apply.

With these three, the cost model is a calculation rather than a hope. Change any of them, higher volume, a longer prompt, a more capable tier, and the projected cost moves in a way you can see before committing to the architecture.

The arithmetic: input and output priced separately

Monthly cost is built by pricing the two token types separately and summing them. Multiply call volume by the input token count at the input token rate to get the input cost; multiply call volume by the output token count at the output token rate to get the output cost; add the two. The reason you cannot shortcut this with a single blended "average tokens per request" figure is that input and output tokens are priced at different rates on every model tier, output typically costs more per token than input, so a request that is mostly output costs more than a request with the same total token count that is mostly input. Collapsing the two into one average throws away exactly the distinction the pricing is built on.

Build it before the code, check it against the ceiling

The timing is not optional. The cost model is built before any production code, and its purpose is to be checked against the budget ceiling that was agreed at project approval. If the projection exceeds the ceiling, the architecture has to change, a cheaper tier, a shorter prompt, caching, batching, before a line of code locks the design in. The Batch API is one concrete lever worth modelling explicitly: it prices asynchronous work at roughly half the standard rate and accepts large request batches, so any workload whose SLA tolerates delayed processing can be modelled as a cost alternative. For regulated data, confirm the batch route is covered by the relevant compliance configuration before routing sensitive fields through it. Deferring the model until after the system is built means discovering the mismatch at the first billing cycle, when renegotiating the architecture is far more expensive than a spreadsheet. (Because per-token rates and batch discounts change, the model should be filled in against current published pricing at the time it is built.)

What the exam trips candidates on

The first trap is estimating monthly cost using only an average token count per request instead of separately pricing input and output tokens at their distinct rates. A scenario will present a blended per-request average as the cost basis; the credited reading splits input and output, applies each rate, and notes that the two are priced differently on every tier.

The second trap is skipping the cost model until after the architecture is built, discovering the budget mismatch only at the first billing cycle. A scenario will show a team that built first and modelled later; the correct answer moves the model to the front, before any code, so the design can be changed while changing it is still cheap.

Common misreadings to avoid

How this shows up on the exam

Domain 4 questions on this knowledge point present a cost estimate that blends input and output tokens, or a plan to model cost after building. The reliable moves are to price input and output separately at the tier's distinct rates, drive the model from call volume, token budget, and tier, and build it before any code so it gates the design against the budget ceiling.

This is the foundation of the optimization task statement and two of the five evaluation dimensions. It sets up token distribution skew, which warns that averages understate a long-request tail, prompt caching, the main lever to reduce input cost, and p95 latency, the latency side of the same model. The per-request data it needs is exactly what request-level tracing captures in production.