- In short
- An eval-gated model swap treats any change of model as a behaviour change to the whole system and gates it like a code release. A defensible swap requires three things fixed in advance: a curated test set representative of production traffic, a grading function, and a delta threshold set before the eval runs. Setting the acceptance threshold after seeing results turns a standard into after-the-fact justification.
A one-line config change that rewrites the product
Swapping the model behind a system looks trivial: change one identifier and redeploy. That appearance is the trap. A model change is a behaviour change to the entire system, because every output the product produces now comes from a different model with different tendencies. The Claude Certified Architect - Professional (CCAR-P) exam treats this as an apply-level skill and expects you to handle a model swap with the same discipline you would bring to any code release. If you would not ship a code change untested, you should not ship a model change untested either.
This is the natural consequence of the Sonnet-first default heuristic: once you accept that tier changes must be evidence-driven, you need a concrete mechanism for producing that evidence. That mechanism is the eval gate.
- Eval-gated model swap
- Treating any model change as a release gated by evaluation. Three elements are fixed before the swap: a curated test set representative of production traffic, a grading function that scores outputs against known-good answers, and a delta threshold set in advance below which the swap is not shipped. The threshold decided beforehand is what makes the eval a standard rather than a justification.
The three things a defensible swap requires
At minimum, gating a model swap needs three components, and the order in which you fix them matters.
The first is a curated test set of prompts with known-good outputs that covers the real distribution of work the system sees. A test set drawn from actual production traffic is what makes the result mean something; a handful of cherry-picked prompts does not. The second is a grading function: either model-graded against a rubric, or programmatic where the check can be expressed in code. The grader turns outputs into comparable scores. The third is a delta threshold set in advance, below which you do not ship. This is the number that says how much regression, if any, is acceptable.
Fix all three before you run the swap. The test set and grader make the comparison possible; the pre-committed threshold makes the comparison decisive.
Why the threshold must come first
The single most important discipline here is setting the acceptance threshold before you run the eval. Set it afterward and you are no longer setting a standard, you are writing the acceptance criteria after the build. Once you have seen that the new model scored, say, 0.88, it is dangerously easy to decide that 0.88 is "good enough" precisely because it is what you got. The threshold has to exist before the data so the data cannot define it.
This is a subtle failure because it feels rigorous. You ran an eval, you looked at numbers, you made a decision. But if the number itself moved the goalposts, the eval gated nothing. A threshold committed in advance is the only version that can actually reject a swap.
Benchmarks are not your eval
A common shortcut is to skip the held-out eval because "the new model is generally better on public benchmarks." Public benchmarks measure general capability on someone else's task distribution, not your system's behaviour on your traffic. A model that scores higher in aggregate can still regress on the specific slice of work your product depends on. The benchmark tells you the model is broadly capable; only your own eval tells you it is right for your workload. This is also why a prompt tuned for one tier may need retesting when the model changes, a link explored in prompt portability across model tiers.
What the exam trips candidates on
The two traps target the two ways teams fake the gate. The first is running an eval but choosing the pass/fail threshold only after seeing how the new model scored. This looks like diligence and is actually post-hoc justification. The credited answer fixes the threshold before the run. The second is skipping a held-out eval set because the new model is better on public benchmarks. The credited answer insists on an eval against representative production traffic regardless of benchmark results.
Worked example
A team wants to migrate a classification service from Sonnet to a newly released, cheaper model. They point to the new model topping public leaderboards and propose shipping immediately, adding, 'If it scores above whatever Sonnet got, we'll keep it.' What is wrong, and what should they do?
Two things are wrong, and both are classic gate failures. First, the leaderboard argument substitutes a public benchmark for an eval on the team's own traffic. The new model may be broadly stronger and still regress on this service's specific class distribution, which a general benchmark cannot reveal. Second, the acceptance rule, "above whatever Sonnet got," is defined by the data rather than in advance, so it cannot reject anything; whatever the new model scores becomes the standard retroactively.
The disciplined approach fixes three things before the swap. Curate a test set drawn from real production classifications with known-good labels. Choose a grading function, here a programmatic exact-match against the labels is natural. Then commit a delta threshold in advance, for example, the new model must score within one point of Sonnet's measured baseline with no class dropping below an agreed floor. Only after those are fixed do they run both models against the set.
Now the eval can actually gate. If the new model meets the pre-committed threshold, the cheaper model ships with evidence behind it. If it does not, the leaderboard result is irrelevant and the migration is rejected. The benchmark never entered the decision, and the threshold could not be bent to fit the outcome.
Common misreadings to avoid
Misconception
Running an eval and then deciding the threshold based on the score is rigorous enough.
What's actually true
Misconception
A model that leads public benchmarks doesn't need a project-specific eval.
What's actually true
How this shows up on the exam
Apply-level questions describe a proposed model swap and ask how to gate it, or present a flawed gating process and ask what is missing. The reliable answer names all three elements, a representative test set, a grading function, and a threshold fixed before the run, and rejects both post-hoc thresholds and benchmark-only justifications.
This knowledge point follows the Sonnet-first default heuristic and underpins both per-step model tiering, where each step needs its own gated decision, and stratified eval and partial migration, which extends the gate with a subtype-stratified test set and a pre-committed rollback criterion.
A team runs their new candidate model against a test set, sees it score 0.89, and declares 0.89 the passing bar because 'that's clearly acceptable.' Why does this fail as a release gate?
People also ask
Why treat a model swap like a code release?
What do you need to gate a model change?
Are public benchmarks enough to justify a model swap?
Watch and learn
Official Anthropic Academy lessons first, then hand-picked walkthroughs. Videos load only when you press play.
No videos curated for this concept yet
We are still curating the best official and community videos for this topic.
Official prep for this domain
Anthropic's own free prep module for this part of the syllabus, on the official prep course. Free with an Anthropic Academy sign-in.
References & primary sources
Master this concept with Archie
Practice it inside an adaptive study session. Archie, your Socratic AI tutor, tracks your mastery with Bayesian Knowledge Tracing and schedules the perfect next review.