Failure Taxonomy: Prompt Failure, Hallucination, Model Mismatch, Orchestrator Failure

Four failure classes, four different fixes

When a production Claude system fails, the instrumentation tells you a metric moved; diagnosis tells you what kind of failure moved it. The exam's analyse-level taxonomy names four classes, and the reason to learn them is that each has a distinct fix. Misclassify the failure and you apply the wrong fix, most commonly reaching for stronger prompt wording when the real problem is elsewhere. The four are prompt failure, hallucination, model mismatch, and orchestrator-workers failure. Classifying a described failure into the right one, and naming the matching fix, is the skill being tested.

This taxonomy sits downstream of change attribution: attribution tells you whether the model, data, or version moved a metric; the taxonomy tells you what kind of failure a given wrong output actually is. Both share the discipline of diagnosing before fixing.

Prompt failure and hallucination look similar but differ

The two classes most often confused are prompt failure and hallucination, because both surface as a wrong output. They are distinct. A prompt failure happens when the instruction was ambiguous or underspecified, so the model filled the gap in a way you did not intend. The fault is in what you asked, and the fix belongs in the prompt: specify the missing constraint, disambiguate the instruction. A hallucination is different: the model produced confident, fluent content that is simply not grounded in the input or any reliable source. The instruction may have been perfectly clear; the model went beyond what the evidence supported.

The fix divergence is the crux. A hallucination is not fixed by stronger instructions, because the problem is missing grounding, not a missing rule. Telling a model 'do not make things up' does not supply the facts it lacked. The fix is grounding: retrieval to bring in the real source, tool use to fetch the true value, or a verification step to check the claim. Confusing the two leads to the signature mistake of adding prompt wording to a problem that needed grounding.

Model mismatch and orchestrator failure

Model mismatch is when the selected tier is wrong for the task, either it never fit, or it was swapped without re-evaluation and the new tier handles the task worse. The fix is not prompt tweaking or grounding; it is eval-gated model selection, choosing the tier by running it against an eval rather than by assumption, so the tier is matched to the task on evidence. This ties model choice back to the discipline of evaluating every change before it ships.

Orchestrator-workers failure is the multi-agent class, and it is distinguished by where you have to look. A failure in a coordinator-subagent system cannot be diagnosed from a single-agent view; you have to trace across the orchestrator and its subagents. The trace is what lets you tell a recoverable subagent failure (one worker failed and can be retried or flagged) from an unrecoverable orchestrator failure (the coordinator itself failed, so retrying a worker will not help). Without a trace spanning both levels, a multi-agent failure is easily misread as a single-agent prompt failure.

What the exam trips candidates on

Two traps. The first is trying to fix a hallucination purely by adding stronger wording to the prompt instead of adding grounding, which mistakes a grounding gap for an instruction gap. The second is misclassifying a multi-agent failure as a single-agent prompt failure because the trace does not span both the orchestrator and its subagents, missing that the failure lives across levels. The credited answer classifies the failure first and applies the fix that matches its class.

Common misreadings to avoid

How this shows up on the exam

Expect a described failure and a tempting 'just strengthen the prompt' fix, sometimes inside a multi-agent system. The reliable reading is to classify the failure, prompt, hallucination, model mismatch, or orchestrator-workers, and apply its matching fix, grounding for hallucination, eval-gated selection for mismatch, and a cross-level trace for orchestrator failures. This knowledge point builds on change attribution, shares its diagnose-before-fixing discipline with diagnosing capability bloat and diagnosing retrieval failures, and unlocks observability as a precondition for agent autonomy.