Stratified Eval and Partial Migration for the CCAR-P Exam

When the average lies

Evaluating a model downgrade with a single overall score is the mistake this knowledge point exists to prevent. Suppose you are moving a workload from a more capable tier to a cheaper one and the cheaper model scores, on average, close to the original. That average can conceal a genuine problem: the cheaper model may hold quality on most of the work while collapsing on a specific subtype, and the strong performance elsewhere masks the collapse in the mean. The Claude Certified Architect - Professional (CCAR-P) exam treats this as an evaluate-level skill because it asks you to judge a migration decision that a naive average would get wrong.

The two tools that fix it are a stratified eval, which breaks the score out by subtype, and a rollback criterion committed before the results arrive. Together they turn a downgrade from a gamble into a defensible decision, and often reveal a third option the average would have hidden entirely.

Stratify by subtype, not by average

The point of stratifying is that different subtypes of a task can behave very differently under a cheaper model. A document-extraction workload might handle invoices and receipts easily on the cheaper tier while struggling with two harder document types. Report only the average and those two hard types are diluted into an acceptable-looking number. Report per subtype and the two failures stand out immediately. So the eval set must be built to keep per-subtype scores meaningful, which means sampling enough of each subtype that its score is not statistical noise. The stratified view is what makes the regression visible; the average is what makes it disappear.

Fix the rollback criterion before the data

The second tool is a rollback criterion decided in advance, for example, "reject the migration if any single subtype drops below 0.85." Deciding this before the eval runs is what gives it force. Once you have seen that a subtype scored 0.82, it becomes tempting to argue that 0.82 is "close enough" or that the subtype is "rare anyway." A criterion fixed beforehand forecloses that negotiation: 0.82 is below the line, the line was drawn before the data, and the migration as proposed is rejected. This is the same discipline as eval-gated model swaps, sharpened for the stratified case where the temptation to explain away a single weak subtype is strongest.

Partial migration: the option the average hides

The most valuable move here is recognising that a stratified failure is not all-or-nothing. When some subtypes pass the threshold and others fail, you do not have to choose between accepting the whole downgrade and rejecting it entirely. You can route by subtype: send the passing subtypes to the cheaper tier and keep the failing ones on the more capable tier. This captures most of the cost savings without taking the regression on the difficult subtypes. It is the same routing logic as per-step model tiering, applied across subtypes of one step rather than across steps. Crucially, this option is invisible to an aggregate eval, because the aggregate never told you which subtypes were the problem.

What the exam trips candidates on

The two traps are the two ways the average misleads. The first is accepting a downgrade because the overall average score looks acceptable while specific subtypes fail badly. A scenario will foreground a reassuring average; the credited answer stratifies and catches the buried regression. The second is treating a stratified eval failure as all-or-nothing instead of considering a partial, subtype-routed migration. A scenario will frame the choice as accept-everything or reject-everything; the credited answer routes by subtype to salvage most of the savings.

Common misreadings to avoid

How this shows up on the exam

Evaluate-level questions present a proposed downgrade with an average score and ask you to judge it. The reliable answer stratifies by subtype, applies a rollback criterion fixed before the results, rejects the full downgrade when a subtype fails the floor, and considers a partial subtype-routed migration to keep most of the savings.

This knowledge point extends eval-gated model swaps with stratification and a rollback rule, applies the routing idea from per-step model tiering across subtypes, and often follows the remediation of an undeclared model-tier default, where per-step and per-subtype tiering are reconstructed together.