- In short
- Designing explicit review criteria means specifying exactly which issues an automated reviewer must report (such as bugs, security vulnerabilities, and broken logic) and which it must skip (such as style preferences and formatting), with code examples that mark the boundary between the two. The explicit boundary is what keeps a review system from flagging noise or missing real defects.
What designing explicit review criteria involves
Designing explicit review criteria is the work of writing down, in advance, the two lists that govern an automated reviewer: the things it must always raise, and the things it must always leave alone. A review system without those lists is not neutral; it simply invents the lists on the fly, differently each time. The whole value of authoring explicit review criteria is that you, not the model, decide where the line between signal and noise falls.
The report list is the easy half. Most teams can name what they want surfaced: logic bugs, security vulnerabilities, broken error handling, race conditions. The half people forget is the skip list, and it is where reviews go wrong. If you never tell the reviewer that local formatting, naming taste, and conventional in-house patterns are out of scope, it will dutifully comment on all of them, burying the bug report under a pile of preferences. Explicit review criteria are defined by both halves working together.
- Explicit review criteria
- A written specification of exactly which issues an automated reviewer must report and which it must skip, with boundary examples marking the dividing line, so the system raises real defects without drowning them in subjective noise.
Why the skip list matters as much as the report list
The skip list is what makes a review usable. A finding only has value relative to the things that were correctly left silent. If the reviewer comments on everything, a human still has to triage everything, and you have moved the work rather than removed it. Naming what to skip is therefore not a politeness; it is the lever that turns a verbose model into a focused one. Anthropic's guidance on being specific captures the principle directly: telling the model precisely what to do and not do is how you get the behaviour you want.
There is a second, quieter reason the skip list earns its place. Subjective comments are the ones developers resent most, because they read as the tool second-guessing a deliberate choice. A reviewer that lectures about a variable name the team has used for years feels broken even when its bug-finding is excellent. By writing the conventional patterns of your codebase into the skip list, you align the system with how your engineers already work, which is what keeps them reading it.
Boundary examples: the hard cases live on the line
Naming the two lists handles the obvious cases. The difficulty is always the cases that sit on the boundary, and that is exactly where a boundary example earns its keep. A boundary example is a snippet that could plausibly belong to either list, paired with a ruling on which side it falls and why. Showing the model one or two of these does more than a paragraph of qualification, because it demonstrates the threshold rather than describing it.
Consider a function that is slightly too long. Is that a reportable maintainability problem or a skippable style preference? The answer depends on your team, which is precisely why it has to be shown, not assumed. A boundary example fixes the ruling: "a function over eighty lines that mixes three responsibilities is reportable; a tidy ninety-line function with one clear job is not." With that example in the prompt, the model stops guessing and starts applying your standard, and the consistency of the whole review improves.
How explicit criteria shape the review pipeline
Explicit criteria are not a separate stage so much as the contract the review prompt enforces on every diff. The model reads the change, checks each candidate finding against the report and skip lists, consults the boundary examples for the ambiguous ones, and emits only the findings that clear the bar. Everything downstream, from severity to human triage, inherits the discipline of that first filter.
A worked example: a reviewer that buries the bug
Worked example
A fintech team adds an automated reviewer to its pipeline, but its first real catch, a missing authorisation check, is lost among twenty comments about naming and import order.
The initial instruction was broad: "Review this pull request and report any issues you find." The model interpreted any issues literally and returned a wall of comments, most of them about whitespace, import grouping, and variable names that did not match its own preferences. Buried at comment fourteen was a genuine and serious finding: an endpoint that skipped the authorisation middleware. A reviewer dismissed the whole batch as noise and nearly missed it.
The team rewrote the criteria explicitly. The report list named logic errors, security and authorisation gaps, broken error handling, and data-integrity risks. The skip list named formatting, import order, naming preferences, and any pattern already common in the repository. They added one boundary example: an unvalidated request body is reportable, but a slightly verbose log message is not. The prompt was otherwise unchanged.
The next review returned three comments instead of twenty, and the authorisation gap was the first of them. By naming what to skip as carefully as what to report, the team turned a noisy tool into a trusted one. The point this knowledge point drives home is that an explicit boundary, backed by an example, is what lets a reviewer be both thorough about defects and quiet about taste.
Common misreadings to avoid
This is an apply-level skill, so exam items tend to test whether you would actually specify the boundary or merely gesture at it. The two misreadings below are the frequent ones.
Misconception
Listing what to report is enough; the reviewer will naturally ignore everything else.
What's actually true
Misconception
Explicit criteria mean writing an exhaustive rulebook covering every possible finding.
What's actually true
Where this sits in the knowledge graph
Designing explicit review criteria builds on severity calibration with code examples: once findings are graded consistently, the next question is which findings should appear at all. It also leans on explicit categorical criteria, the broader principle that categories must be named, not implied. Get the report-and-skip boundary right and you have the substrate for the harder judgement ahead.
That judgement is the precision versus recall trade-off in review prompts, which this knowledge point directly unlocks: a tighter skip list raises precision, a broader report list raises recall, and explicit criteria are how you steer between them. When a single category keeps misbehaving, you move into category-specific prompt iteration. The motive behind all of it is the false positive trust problem: noise destroys trust, and explicit criteria are the cure.
Why local conventions belong on the skip list
The hardest part of a skip list is not the universal noise like trailing whitespace; it is the patterns that are conventional in your codebase but would look wrong to a model trained on everyone else's. A house style might favour a particular error-wrapping idiom, a specific test layout, or a naming scheme that an outside reviewer would flag on sight. None of those are defects, yet a reviewer with no knowledge of your conventions will report them tirelessly, and each report reads as the tool misunderstanding the project.
Writing your conventions into the skip list is how you teach the reviewer the local dialect. It is not enough to say skip style; you have to name the specific patterns that are deliberate here, ideally with a one-line example of each. This turns a generically competent reviewer into one that fits your repository, and it removes the single most demoralising class of false positive, the one where the tool argues with a decision the team made on purpose years ago.
Anchor findings in quoted evidence
A report-versus-skip boundary still depends on the model judging, for each candidate, whether the issue is actually present. On long or noisy diffs that judgement is where speculative findings creep in, because the model infers a problem from a variable's name or a function's shape rather than from the code's behaviour. Anthropic's prompting guidance offers a direct countermeasure for long-input review: ask the model to quote the exact lines that trigger a finding before it analyses or reports them. Grounding the decision in quoted source first forces an evidence check, and a candidate finding that cannot point to the lines that justify it is usually one that belongs on the skip side of the boundary.
The technique pairs cleanly with the report and skip lists. The lists decide which categories of issue are in scope; the quote-first rule decides whether a specific instance in this diff is real enough to surface. In practice you add a single instruction: for each finding, quote the lines it refers to and state which report category it matches, and if you cannot quote supporting lines, do not raise it. That requirement lifts precision without narrowing the report list, because it removes the inferred, unprovable findings rather than the substantive ones, and it leaves every surviving comment carrying the evidence a human needs to confirm or dismiss it fast.
Validating criteria before you trust them
Explicit criteria are a hypothesis until you test them. The cheapest validation is to run the new prompt against a sample of past pull requests whose outcomes you already know, then read what it reports and, just as importantly, what it stays silent on. If it raises a formatting nit you meant to skip, the skip list has a gap; if it misses a defect you know was there, the report list or a boundary example needs work. The sample does not need to be large to be useful; a dozen representative diffs will expose most boundary problems.
This validation step is where authoring criteria meets the broader discipline of evaluation. Anthropic's guidance on defining success criteria frames it the same way: state what a good outcome looks like, then measure against held-out examples instead of trusting your intuition about the prompt. Running your criteria over known cases converts a plausible-sounding specification into one you have actually watched behave, which is the difference between a criteria document you hope works and one you can defend.
Explicit criteria as living documentation
A well-written set of report and skip criteria does double duty: it steers the reviewer and it records what the team considers a defect. New engineers can read it to learn the standards, and disagreements about whether something should have been flagged become concrete edits to the criteria rather than recurring arguments. Because the boundary is written down and example-backed, the review system improves through small, reviewable changes, the same way the codebase does, instead of through quiet shifts in one person's private judgement.
How it shows up on the exam
Task Statement 4.1 lives most often inside the continuous-integration review scenario, where an architect is handed a reviewer that is technically working but practically useless because it cannot tell signal from noise. The questions describe a tool burying real findings under cosmetic comments and ask for the design change that fixes it. The answer that earns credit is to define explicit report and skip criteria with a boundary example, not to tune sampling, swap the model, or tell it to try harder. The exam is checking that you reach for specification over hope, which is the defining habit of this skill.
An automated reviewer returns dozens of comments per pull request, mostly about formatting and naming, and developers have started ignoring it entirely. The prompt says only: review this PR and report any issues. What is the strongest fix?
People also ask
What should an automated code reviewer report versus skip?
How do you write explicit criteria for an LLM reviewer?
Why do vague review criteria produce inconsistent findings?
Watch and learn
Official Anthropic Academy lessons first, then hand-picked walkthroughs. Videos load only when you press play.
Being specific
Why watch: Spelling out exactly what to do and not do is how you design explicit criteria for reliable behaviour.
More videos for this concept
References & primary sources
Master this concept with Archie
Practice it inside an adaptive study session. Archie, your Socratic AI tutor, tracks your mastery with Bayesian Knowledge Tracing and schedules the perfect next review.