The Five-Stage Eval Workflow

Five stages, each feeding the next

An eval is not a single action; it is a pipeline, and the stages have to run in order because each one produces the artifact the next one needs. Skip a stage or run them out of sequence and the workflow produces a number that does not map to anything. The apply-level skill this knowledge point tests is running the workflow end to end, knowing what each stage takes in, what it hands off, and where the workflow quietly breaks if a stage is shortchanged.

Stage 1 and 2: define the task, then build the dataset

Stage one defines the task. You state the behaviour being evaluated in specific, measurable terms, write the prompt you will test it with, and set the pass criteria. A vague task definition produces a vague eval, so the concreteness of both the behavioural specification and the prompt is what makes every later number meaningful. The artifact is a task specification with a test prompt and pass criteria.

Stage two builds the golden dataset. You assemble the inputs the system will face, including edge cases and counterexamples, and, critically, label each with its expected output. The order matters: the dataset is built to match the task defined in stage one, not before it, because the task specification tells you what inputs and expected outputs are relevant. The artifact is a labelled dataset. Skip the labels and the dataset is just raw inputs with nothing to grade against.

Stage 3 and 4: automated checks, then judge scoring

Stage three runs the automated code-based checks. Each prompt goes through the system and its output is compared against the expected output for the unambiguous behaviours, format compliance, schema validation, exact-match lookups. These are fast and cheap, so they run across the whole dataset. The artifact is a pass/fail record per item.

Stage four scores the behaviours that need interpretation with a judge. Tone, reasoning quality, appropriateness of edge-case responses, anything a function cannot assess, goes to a calibrated LLM judge that returns a score and its reasoning per item. Stages three and four are the grading ladder in action: cheap deterministic checks first, judge only where the behaviour demands it. The artifact is a score with reasoning per item.

Stage 5: interpret and act, past the mean

Stage five aggregates everything into an overall score and, just as importantly, a per-category breakdown. The aggregate tells you roughly where the system stands and whether a change moved it; the breakdown tells you where. This distinction is the heart of the stage. A change that raises the mean score while quietly degrading performance on edge-case or adversarial categories has not made the system better, it has traded away the cases that matter for a nicer average. Only the per-category breakdown reveals that trade, which is why reporting the mean alone is a stage-five failure.

What the exam trips candidates on

The first trap is building the golden dataset before defining the task and pass criteria. A scenario will show a team that collected a dataset first, then tried to fit a task to it, producing a dataset that does not map to a specific measurable behaviour. The credited reading restores the order: define the task, then build the dataset to match it.

The second trap is reporting only the aggregate mean after a change and concluding the system improved. A scenario will present a change that lifted the mean and ask whether it is an improvement; the correct answer withholds that conclusion until the per-category breakdown is checked for edge-case regressions, because a higher mean can hide a worse system on the categories that matter.

Common misreadings to avoid

How this shows up on the exam

Domain 4 questions on this knowledge point either scramble the stage order, usually building the dataset before defining the task, or stop at the aggregate mean. The reliable moves are to run the stages in sequence with each artifact feeding the next, insist the dataset carry labelled expected outputs, and always read the per-category breakdown before declaring an improvement.

This workflow operationalizes translating a requirement into a threshold as stage one and stitches together the three eval types via the grading ladder in stages three and four. Its interpret-and-act stage is where a stale eval suite should be caught, its judge stage relies on a calibrated rubric, and it extends to whole conversations in multi-turn eval design.