- In short
- Translating a business requirement into a measurable threshold means rewriting a vague requirement such as "summarize accurately" into a specific, checkable behavior, such as extracting named fields correctly, and attaching a numeric passing threshold derived from the business requirement itself rather than from whatever the first prototype happens to score. Each named failure mode becomes its own category in the eval dataset, so the eval reports where the system fails, not just whether it passed on average.
From a sentence a stakeholder says to a number a system must hit
Business requirements arrive as sentences: "summarize claims accurately," "handle customer questions well," "flag risky contracts." None of those is something a machine can check. The apply-level skill this knowledge point tests is the translation, taking a sentence that sounds like a requirement and turning it into a specific behavior with a numeric bar that an eval can actually measure. Get the translation right and the eval becomes a decision instrument; get it wrong and you have an eval that runs, produces a number, and tells you nothing about whether the system is good enough to ship.
- Measurable threshold
- A specific, checkable behavior paired with a numeric passing bar derived from the business requirement. Translating a requirement into a threshold means rewriting a vague behavior into a concrete one, decomposing compound requirements into separate checks, naming each failure mode as its own eval category, and setting the passing number from what the business needs rather than what the prototype scores.
Rewrite the behavior to something checkable
The first move is to replace the vague verb with a concrete, observable one. "Summarize claims accurately" does not say what to measure. Rewritten, it might become "extract the filer's name, claim number, incident date, and claimed amount from each document, and reproduce them exactly." Now there is something a check can look at: the four fields are either present and correct or they are not.
This rewrite usually also decomposes a compound requirement into several separate ones. "Accurately" was hiding at least two different demands, extracting the right fields and not inventing fields that were not there, and each deserves its own check. Writing a single pass/fail for a compound requirement collapses distinct behaviors into one number and loses the ability to see which behavior actually failed.
Set the threshold from the requirement, not the prototype
The second move is where the most consequential mistake lives. Once you have a checkable behavior, you have to decide what score counts as passing. The threshold must come from the business requirement: a claims workflow feeding a system of record might require 100% accuracy on structured fields, a hallucination rate under 2%, and schema compliance at 99.5%, because those are the numbers the downstream process can tolerate.
What you must not do is set the threshold to whatever the first prototype happens to achieve. That is circular: it tells you only whether the system still performs like the prototype, which is not the question. The prototype's score is a fact about the current implementation; the threshold is a fact about what the business needs. They are allowed to differ, and when they do, the gap is exactly the information you wanted, the system is not yet good enough.
Turn each failure mode into a category
The third move is to name the ways the system can be wrong and make each one a category in the eval dataset. For the claims example the failure modes might be a fabricated claim number, a missing incident date, and a value pulled from the wrong claim. Each becomes a labelled slice of the dataset, so the eval does not just report an aggregate, it reports which failure mode is occurring and how often. A single blended score hides this; per-failure-mode categories surface it, and surfacing it is what lets you fix the right thing.
What the exam trips candidates on
The first trap is setting the passing threshold to match whatever the first prototype scores. A scenario will describe a team that ran the prototype, saw 91%, and declared 91% the bar. The credited answer recognizes this as circular and insists the threshold come from the business requirement, even, especially, when that means the prototype currently fails.
The second trap is writing a single pass/fail check for a compound requirement. A scenario will present a requirement bundling several behaviors and an eval that checks them as one. The correct reading decomposes the requirement into separate measurable behaviors and separate failure-mode categories, so the eval can say which part is broken rather than only that something is.
Worked example
A stakeholder asks for a system that 'reads support tickets and routes them correctly.' A developer builds a prototype, measures 87% routing accuracy on a sample, and proposes 87% as the eval threshold. What is wrong with this, and how should the eval be defined instead?
Two problems. First, "routes them correctly" is a compound behavior masquerading as a single one. Routing correctly means at least identifying the ticket's category, assigning it to the right queue, and not silently dropping tickets it cannot classify. Collapsed into one 87% number, the eval cannot tell whether the failures are misclassifications, wrong-queue assignments, or drops, and each of those has a different fix.
Second, adopting 87% as the threshold because that is what the prototype scored is circular. It defines "good enough" as "whatever the current build does," which is not a statement about the business at all. The threshold has to come from what the routing process can tolerate: perhaps misrouting above 3% overwhelms the manual-correction queue, which sets the real bar.
The correct definition rewrites the requirement into separate checkable behaviors, category identified, queue assigned, no silent drops, makes each failure mode its own dataset category so the eval reports which one is failing, and sets each threshold from the operational requirement. If the prototype's 87% falls short of that bar, that gap is the finding, not a reason to lower the bar to meet it.
Common misreadings to avoid
Misconception
The passing threshold should be whatever the first working prototype achieves.
What's actually true
Misconception
A compound requirement can be checked with a single pass/fail eval.
What's actually true
How this shows up on the exam
Domain 4 questions on this knowledge point hand you a vague requirement or a prototype-derived threshold and ask what a sound eval definition requires. The reliable moves are to rewrite the behavior into something checkable, derive the threshold from the business need rather than the prototype, and split failure modes into their own categories.
This applies the discipline of evals as acceptance criteria to the concrete act of writing the criteria, and it feeds directly into covering adversarial and edge-case inputs, since the failure-mode categories you name here become the slices your dataset must cover. It also connects to the five-stage eval workflow, where task definition is stage one, and to the technical-to-business metric translation layer that keeps these numbers tied to what stakeholders track.
A requirement reads 'extract invoice data reliably.' Which approach best turns it into a usable eval?
People also ask
How do you set a passing threshold for an eval?
Why should thresholds come from the business requirement, not the prototype score?
What is a failure-mode category in an eval dataset?
Watch and learn
Official Anthropic Academy lessons first, then hand-picked walkthroughs. Videos load only when you press play.
No videos curated for this concept yet
We are still curating the best official and community videos for this topic.
Official prep for this domain
Anthropic's own free prep module for this part of the syllabus, on the official prep course. Free with an Anthropic Academy sign-in.
References & primary sources
Master this concept with Archie
Practice it inside an adaptive study session. Archie, your Socratic AI tutor, tracks your mastery with Bayesian Knowledge Tracing and schedules the perfect next review.