Evals as Pre-Build Acceptance Criteria

The order of operations decides whether evals help

The instinctive way to build a system is to write the code, look at the output, decide it seems right, and add some tests later if there is time. The Claude Certified Architect - Professional (CCAR-P) exam treats that order as a mistake. An eval suite is only fully useful when it exists before the production code, because its real job is not to grade a finished system but to define what "finished" means in the first place.

Putting evals first is an understand-level skill: you need to grasp why the timing changes what the eval can do for you, not just that it should happen early. Written first, the eval suite is a specification. Written last, it is a rubber stamp on whatever the system already happens to do.

Three things writing evals first forces

Defining evals before code forces three things that are otherwise easy to defer indefinitely.

First, it forces the team to state success in measurable terms. "The assistant should summarize well" is not something you can build against or verify. Turning it into an eval forces the harder question of what a good summary actually contains and how you would check it, and that question is far cheaper to answer before implementation than after.

Second, it exposes design assumptions early, while changing them is still cheap. The act of writing checks surfaces the places where the requirement is ambiguous or the architecture is making a bet. Discovering that bet on day one costs a conversation; discovering it after the system is built costs a rebuild.

Third, it gives you a gate that can tell whether any later change actually improved the system. Without a suite defined in advance, a prompt tweak or a model swap is a guess: it might be better, it might be worse, and there is no instrument to say which.

The gate that every change must pass

Once the suite exists, it becomes the checkpoint for every subsequent change to the production system. Swap the model to a cheaper tier? Run the suite. Revise the system prompt? Run the suite. Change the retrieval configuration? Run the suite. This is the only reliable way to know whether a change moved the system forward or quietly broke something, because an LLM system rarely announces a regression, it just starts producing subtly worse output on inputs nobody re-checked.

The deeper principle is that verifiability is a property you have to design in. If a behavior cannot be written as an eval, then every change you make to the system is unverifiable with respect to that behavior. You are flying on the assumption that it still works, and assumptions are exactly what production erodes.

What the exam trips candidates on

The first trap is building the system first and writing evals afterward as a QA formality. A scenario will present a team that shipped, then added evals to document what the system does. The credited reading recognizes that an eval written to match existing behaviour cannot catch the design flaws an eval written first would have exposed, and it cannot serve as an honest gate because it was fitted to the answer.

The second trap is trusting manual spot-checks. A scenario will show a team that tried the system on a handful of familiar inputs, saw it respond correctly, and declared it validated. Spot-checks only confirm behaviour on the inputs already tried; they say nothing about the inputs nobody thought of, which is precisely where production surfaces failures. Recognizing spot-checks as an anecdote rather than a suite is the point.

Common misreadings to avoid

How this shows up on the exam

Domain 4 questions on this knowledge point describe a team that either wrote evals too late or leaned on spot-checks, then hit a regression. The dependable reading is that the eval suite belongs at the start of the build as the acceptance criteria and the gate, and that manual checks are not a substitute for a representative suite.

This builds directly on the five evaluation dimensions, which are the axes the acceptance criteria are written against, and it leads into translating a business requirement into a measurable threshold, which is the mechanics of writing the criteria. It also underpins the five-stage eval workflow and pairs with diagnosing a stale eval suite, the failure that appears when a suite defined early is never kept current.