Code-Based, Model-Based, and Human-Review Eval Types

Three ways to grade, matched to three kinds of behaviour

Not every behaviour you need to evaluate can be checked the same way, and picking the wrong method either wastes money or produces a grade you cannot trust. Some outputs have exactly one correct form, valid JSON or not, the right field value or the wrong one, while others, tone, reasoning quality, appropriateness, require a judgment call that no simple function can make. The CCAR-P exam expects you to name the three eval methodologies and match each to the kind of behaviour it fits. This is a remember-level foundation for the grading decisions that follow.

Code-based evals: fast, cheap, deterministic

A code-based eval is a function that inspects the output programmatically, does it parse as valid JSON, does it match the expected schema, does a regex find the required pattern, is it under the length limit, does the extracted value equal the authoritative value. These checks run in milliseconds, cost essentially nothing because there is no model call, and never drift, the same input always produces the same verdict.

Their strength is also their boundary. A function can only check what can be reduced to a deterministic rule. It can confirm that output is valid JSON; it cannot judge whether a summary captured the important points, because "important" requires interpretation a function cannot supply. Wherever a behaviour has a single unambiguous correct answer, the code-based eval is the right tool and any heavier method is waste.

Model-based evals: judgment at scale, at a price

A model-based eval, often called LLM-as-judge, hands the output to a judge model along with the original prompt and a scoring rubric, and the judge returns a verdict with reasoning. This is how you score behaviours that require interpretation, response quality, instruction-following, reasoning soundness, safety, handling of ambiguous inputs, at a scale humans could never reach.

The cost is roughly one model call per item evaluated, at the judge model's per-token rate, which adds up across a large dataset but is still far below human review. The limitation is reliability on borderline cases: a judge can be inconsistent, and without forcing it to produce reasoning alongside its score, that inconsistency is hard to detect. A model-based eval is a system that itself needs engineering and validation, not a free oracle.

Human-review evals: the expensive last resort

Human-review evals put a person in front of the output to score it against a rubric or annotate it freely. This is the method for high-stakes or novel behaviours where neither a function nor a judge model can yet be trusted, safety-critical edge cases, brand-new capability areas without established rubrics, anything where a wrong grade carries real risk. Human review is also how you calibrate and validate a model judge in the first place.

Its cost is human time and its ceiling is human throughput, which makes it the slowest and most expensive option and unusable at full scale. In practice it runs on sampled subsets, and humans bring their own inconsistency, so even here the process needs rubrics and structure.

What the exam trips candidates on

The first trap is using an LLM judge to check something a function should check, such as whether output is valid JSON. A scenario will route a deterministic check through a model judge; the credited answer moves it to a code-based check, which is faster, cheaper, and deterministic, and points out that spending a model call to validate JSON is waste.

The second trap is the reverse, assuming a code-based check can score a subjective quality like tone, helpfulness, or reasoning. A scenario will try to grade "professional tone" with a regex or keyword match. The correct reading recognizes that these behaviours require interpretation a function cannot supply and belong to a model-based or human eval. The whole skill is matching the mechanism to whether the behaviour is unambiguous or interpretive.

Common misreadings to avoid

How this shows up on the exam

Domain 4 questions on this knowledge point list a set of behaviours and ask you to sort each into code-based or model-based (with human review for the highest-stakes). The reliable test is whether the behaviour has a single unambiguous correct answer, if so, code-based; if it needs interpretation, model-based or human.

This foundation feeds directly into the eval grading ladder, which turns the three types into a cost-ordered decision rule, and into calibrating an LLM-as-judge rubric, which makes the model-based option trustworthy. It also supports the five-stage eval workflow, whose automated-checks and judge-scoring stages are exactly these methods in sequence, and it connects back to translating a requirement into a threshold, where the behaviour's nature determines its check.