- In short
- Diagnosing failure patterns means applying the hallucination and inconsistency taxonomy to an output that looks clean on the surface, identifying which specific pattern is present and why it would pass a casual read. Matching the observed error to its named pattern, fabricated specific, contradiction, confident-tone masking, capability hallucination, or completeness gap, determines the right fix and the specific verification action to take.
From spotting to diagnosing
Recognising that something is wrong is only the start. The Claude Certified Associate - Foundations (CCAO-F) exam pushes to the analyze level here: given an output that looks clean, identify which specific failure pattern is present, explain why it would pass a casual read, and name the verification action that follows. Diagnosis is the bridge between the taxonomy and the fix, because the fix depends entirely on which pattern you are dealing with.
The starting premise is uncomfortable but essential: none of these outputs looks broken. Fluent generation produces failures that are, by their nature, built to survive a normal read. So diagnosis is not a matter of waiting for a jarring note; it is a matter of actively checking the clean output against each named pattern and seeing which one it matches.
- Diagnosing failure patterns
- Applying the failure-pattern taxonomy to a surface-clean output to identify which specific pattern is present, fabricated specific, internal contradiction, confident-tone masking, capability hallucination, or completeness gap, and why it passes a casual read. The matched pattern determines the fix and points to a specific verification action, such as tracing a citation, running a consistency pass, confirming a claimed action, or supplying the missing element.
Active checking, not passive noticing
Because the failures are designed to read cleanly, the diagnostic stance must be active. You do not scan for something that looks wrong; you run the checks that each pattern demands and see what they turn up. Is there a precise figure with no traceable source? Do the figures reconcile across distant sections? Does the output claim an action, and was a tool for it available? Is anything the requirements demanded simply absent? Each question targets a pattern, and the honest answer to one of them is where the diagnosis lands. Waiting for an error to announce itself is the failure mode the whole lesson is built to correct.
Matching the pattern to the fix
Diagnosis matters because the right fix is pattern-specific. A fabricated specific is fixed by tracing or demanding a real citation, or by removal. An internal contradiction is fixed by a consistency pass that reconciles the conflicting figures. A capability hallucination is fixed by confirming the action against the world and checking tool availability. A completeness gap is fixed by supplying what is missing, not by re-verifying what is present. Landing on the wrong pattern sends you to the wrong fix, which is why the diagnosis, not just the alarm, is what the exam tests.
Missing is not the same as invented
The sharpest diagnostic distinction is between a completeness failure and a hallucination. A hallucination adds something false; a completeness gap omits something true and necessary. They feel similar in that both leave the output unfit to use, but they are opposite defects and take opposite fixes. Calling a missing element a hallucination sends you hunting for a false claim that is not there, while the real problem, an absence, goes unaddressed. Keeping accuracy and completeness as independent checks is what keeps this distinction clean during diagnosis.
What the CCAO-F exam trips candidates on
The first trap is labelling every kind of output problem as a hallucination even when the actual issue is a missing element or a stale figure. "Hallucination" becomes a catch-all that hides the real diagnosis and points to the wrong fix. The credited answer names the specific pattern, distinguishing an invented claim from an omission from a contradiction.
The second trap is concluding an output is safe because no single sentence, read alone, contains an obviously false claim. That is precisely the condition under which contradictions and completeness gaps thrive, and it is the passive-noticing stance the lesson warns against. The exam rewards active checking against every pattern, not the absence of a locally false sentence.
Worked example
Claude produces a reconciliation report that reads cleanly: every sentence is grammatical, no single claim looks obviously false, and it is well formatted. A colleague says 'nothing jumps out, so it's fine.' Walk through how to diagnose it properly.
Start by rejecting "nothing jumps out" as a verdict. Fluent generation produces failures built to pass exactly this kind of read, so the absence of a jarring sentence is not evidence of safety; it is the normal appearance of the failures worth catching. Diagnosis means actively running each pattern's check rather than waiting to be alarmed.
Run them in turn. Are there precise figures with no traceable source? If a headline number appears uncited, that is a fabricated-specific candidate, fixed by tracing or demanding a real citation. Do the figures reconcile across the whole report, does the subtotal match the line items, does a total on the last page match the components introduced earlier? A mismatch here is an internal contradiction, invisible to sequential reading and fixed by a consistency pass. Does the report claim any action was taken, a file saved, a system updated, and was a tool for that even available? If so, that is a capability hallucination, fixed by confirming the action and checking tool availability. Finally, against the requirements, is anything that was asked for simply absent? That is a completeness gap, an omission, not an invention, and fixed by supplying the missing element rather than by fact-checking what is present.
Suppose the check reveals that the subtotal does not match the line items above it. The correct diagnosis is a specific one, an internal contradiction in the numbers, and it points to a specific verification action: recompute and reconcile the figures rather than reformat or regenerate. Crucially, do not file this under the generic label "hallucination," because that would send you looking for an invented fact instead of reconciling the real conflict. The value of the diagnosis is that it named the pattern, and the pattern named the fix.
Common misreadings to avoid
Misconception
Any problem in an AI output can be called a hallucination.
What's actually true
Misconception
If no single sentence is obviously false, the output is safe.
What's actually true
How this shows up on the exam
Domain 2 questions on this knowledge point present a clean-looking output and ask you to diagnose the specific problem and its fix. The dependable approach is to actively run each pattern's check, match the observed error to a named pattern, keep omissions distinct from inventions, and state the specific verification action the pattern requires.
This is the analyze-level capstone of task statement 2.2, drawing together spotting fabricated specifics, internal contradiction detection in long outputs, and capability hallucination from the hallucination pattern taxonomy. The completeness-versus-invention distinction depends on accuracy and completeness as independent checks.
A clean, well-formatted reconciliation report has no obviously false sentence, but its subtotal does not match the line items above it. What is the correct diagnosis and fix?
People also ask
How do you diagnose what is wrong with an AI output?
Why label the specific failure pattern?
Is a missing element a hallucination?
Watch and learn
Official Anthropic Academy lessons first, then hand-picked walkthroughs. Videos load only when you press play.
No videos curated for this concept yet
We are still curating the best official and community videos for this topic.
Official prep for this domain
Anthropic's own free prep module for this part of the syllabus, on the official prep course. Free with an Anthropic Academy sign-in.
References & primary sources
Master this concept with Archie
Practice it inside an adaptive study session. Archie, your Socratic AI tutor, tracks your mastery with Bayesian Knowledge Tracing and schedules the perfect next review.