The Eval Grading Ladder

A ladder, climbed only as far as the behaviour forces

Knowing the three eval types is one thing; choosing among them under cost pressure is another. The grading ladder is the decision rule that connects them: it says to start at the bottom, with the cheapest method that can reliably grade the behaviour, and climb only when the behaviour genuinely demands it. The understand-level insight is why the ladder runs in this direction, that cheapness and reliability, not thoroughness for its own sake, are what maximize the regressions you catch per dollar.

Start at the bottom: code-based wherever it fits

The bottom rung is code-based grading, and the rule is that if a behaviour can be checked in code, it should be. Deterministic checks, schema validation, exact match, length, presence, run in milliseconds, cost almost nothing, and never drift. Because they are essentially free, they can run on every single change without a budget conversation, and because they are deterministic, their verdicts are trustworthy without calibration.

Starting anywhere above this rung for a behaviour that code could handle is a category error: you are spending a scarce, expensive resource on a problem the cheapest resource solves perfectly. The ladder's first instruction is to exhaust the code-based option before considering anything else.

Climb only when interpretation is required

The second rung, LLM-as-judge, is reached only when the behaviour cannot be reduced to a deterministic rule, when grading it genuinely requires interpretation of tone, reasoning, or appropriateness. The word "only" is load-bearing. You climb because the behaviour forces you to, not because a judge feels more thorough. And when you do climb, the judge has to be made rigorous, detailed rubrics, constrained verdicts (a small fixed set of labels rather than a free-form score), and calibration against human-labelled examples, so that the interpretation it supplies is actually reliable. One further discipline is easy to overlook: grade with a different model than the one whose outputs you are evaluating. A judge scoring its own family of outputs tends toward self-preference, quietly rating them higher, so separating the graded model from the grading model removes a bias that a rubric alone will not catch.

The top rung, human grading, is the last resort, reserved for high-stakes or novel behaviours where neither code nor a calibrated judge can yet be trusted. It is the most expensive and least scalable option, so you climb to it only when nothing below it will do, and even then usually on a sampled slice.

Volume beats perfection

The ladder has a corollary that shapes how you build the dataset: favour volume over perfection. A large set of automatically-gradable cases catches more regressions than a small set of painstakingly hand-graded ones, because broad, cheap coverage can run on every change and touches far more of the input distribution. A team that pours its effort into a handful of meticulously human-scored examples buys precision on those few cases while giving up coverage everywhere else, and coverage is where regressions hide.

This is why the ladder tilts toward the cheap rungs: they are not a compromise on quality but the way to buy the most quality assurance per unit of effort. The expensive rungs are for the behaviours that truly need them, not the default.

What the exam trips candidates on

The first trap is defaulting to human review for a behaviour that code could check deterministically. A scenario will send a schema or format check to a human queue; the credited answer moves it down to a code-based check and notes that human time is the most expensive resource, wasted here on a cheap problem.

The second trap is choosing a small, meticulously hand-graded eval set over a much larger automatically-graded one. A scenario will present a team proud of its fifty carefully human-scored examples; the correct reading recognizes that the small set sacrifices regression coverage, and that many cheap automatic cases would catch more. Both traps are the same error in different directions: mismatching the grading effort to the behaviour and to the goal of maximizing coverage.

Common misreadings to avoid

How this shows up on the exam

Domain 4 questions on this knowledge point present a grading plan that either over-invests in human review or over-values a tiny hand-graded set, and ask how to restructure it. The reliable moves are to push every unambiguous behaviour down to code-based grading, climb to a judge only for interpretive behaviours, reserve humans for the highest stakes, and favour broad cheap coverage over a small perfect sample.

This applies the three eval types as an ordered decision rule, and it sets up calibrating an LLM-as-judge rubric, the discipline that makes rung two trustworthy when you climb to it. It reinforces cost and latency modeling, since eval cost is a real budget line, and it complements covering adversarial and edge-case inputs, because cheap grading is what lets a large representative dataset run on every change.