- In short
- Root-cause auditing a degraded configuration means tracing subtly wrong output back to specific stale elements across multiple configuration mechanisms at once. Subtly wrong output with no error points to a maintenance audit across instructions, knowledge, Skills, and Memory, not a new prompt. Multiple elements can drift simultaneously, such as a standing instruction and a Memory entry both pinning a Project to a prior reporting period, even while the knowledge base already holds the correct current documents. The fix updates or removes only the specific stale elements identified, leaving correct elements untouched.
The capstone: several stale elements at once
The maintenance knowledge points each handle one mechanism. This one, the domain's hardest and an evaluate-level skill on the CCAO-F exam, puts them together: diagnosing output that has gone subtly wrong when more than one configuration element has drifted at the same time. The scenario is realistic and unforgiving, because the fix requires identifying exactly which elements are stale and touching only those.
The starting signal is familiar from the configuration drift problem: output is subtly wrong, and there is no error. That signature is not a prompting problem and not a model problem; it is a maintenance audit waiting to happen, across all four mechanisms at once. The skill is running that audit precisely.
- Root-cause configuration audit
- The evaluation of tracing subtly wrong output, produced with no error, back to specific stale elements across instructions, knowledge, Skills, and Memory simultaneously. Multiple elements can drift at once, and some mechanisms may be correct while others are stale. The fix updates or removes only the identified stale elements, leaving correct elements, including up-to-date knowledge-base documents, untouched.
Subtly wrong plus no error means audit, not re-prompt
The first judgement is refusing two wrong branches. When a Project's output is subtly wrong with no error, it is tempting either to write a new, more elaborate prompt to steer around the problem or to conclude the model is underpowered. Both are mistakes. A new prompt fights the symptom while the stale configuration keeps reasserting itself, and blaming the model misreads drift as capability.
The correct branch is a maintenance audit. Subtle, unexplained wrongness is the fingerprint of stale configuration, so the response is to inspect the mechanisms, not the model and not the prompt. This is the review cadence run reactively and thoroughly: go through instructions, knowledge, Skills, and Memory looking for what has aged.
Multiple elements drift together, and some do not
What makes the capstone hard is that drift is not always in one place. Several elements can be stale at once, and they can reinforce each other. A classic pattern: a standing instruction still formats to last period's template and cites last period's targets, and separately a Memory entry still records the prior reporting period as current. Two mechanisms, drifted independently, both pinning the Project to a stale period, which is why the output is consistently, subtly wrong.
The subtler twist is that not everything is stale. The knowledge base may already hold the correct current documents, the up-to-date template, this period's targets, even while the instruction and Memory still point at the old ones. This is the trap that separates a careful audit from a careless one: the correct, current documents sit right beside the genuinely stale instruction and Memory, and a hasty auditor can blame the wrong element. The fix must update or remove the stale instruction and Memory while leaving the correct knowledge-base documents untouched.
What the CCAO-F exam trips candidates on
The exam sets two traps. The first is deleting or blaming a correct, up-to-date knowledge-base document because it happens to sit alongside genuinely stale instructions or Memory. The audit must distinguish the current documents from the stale elements and leave the correct ones in place; removing a right document because it is near the wrong ones only creates a new gap. Blaming the nearest thing rather than the actual stale element is the tell.
The second is concluding that subtle, unexplained drift means the model itself is underpowered rather than auditing the configuration for stale elements. The scenario invites a model-capability explanation; the credited answer identifies the specific stale instruction and Memory entry and fixes them. Both traps reward the disciplined audit: subtly wrong plus no error means find and fix the specific stale elements across mechanisms, touching only what is actually stale.
Worked example
A recurring monthly-report Project has started producing subtly outdated figures with no error. Its standing instruction reads 'Format every report using the Q2 FY25 template and cite the FY25 targets.' Its knowledge base holds Report_Template_FY26.docx and FY26_Targets.xlsx. Its Memory records 'Reporting period is FY25; targets are in the FY25 plan.' The current period is FY26. Which elements are driving the drift, and what is the fix?
Two elements have drifted together, while a third is already correct, and the fix must target only the stale ones.
Run the audit rather than re-prompting or blaming the model. The standing instruction is stale: it still pins reports to the "Q2 FY25 template" and the "FY25 targets," steering output to the prior period. The Memory entry is separately stale: it records "Reporting period is FY25," reinforcing the same wrong period from a second mechanism. These two, drifted independently, are jointly driving the subtly outdated figures, which is exactly the multiple-elements-at-once pattern.
The knowledge base is already correct: it holds Report_Template_FY26.docx and FY26_Targets.xlsx, the current documents. This is the trap, deleting or blaming those correct FY26 documents because they sit beside the stale instruction and Memory would be wrong and would create a new gap. The fix is to update the standing instruction to reference the FY26 template and targets and to edit or remove the stale Memory entry so it reflects FY26, while leaving the correct knowledge-base documents untouched. Concluding instead that the model is underpowered would miss the actual cause entirely. Touch only the stale elements, and the figures return to correct.
Common misreadings to avoid
Misconception
If output is subtly wrong, a correct-looking document near the stale settings is probably part of the problem and should be removed.
What's actually true
Misconception
Subtle, unexplained drift with no error means the model is underpowered.
What's actually true
How this shows up on the exam
Domain 5's hardest questions present a full configuration and subtly wrong output, with several elements listed. The credited answer identifies the specific stale elements, often more than one, and fixes only those, explicitly leaving correct current documents in place and rejecting both the new-prompt and the underpowered-model explanations.
This capstone integrates the whole maintenance task statement: the review cadence as the audit method, Skills versioning and the Memory lifecycle as sources of drift, and the configuration drift problem as the reason the output degraded silently in the first place.
A monthly-report Project produces subtly outdated figures with no error. Its standing instruction cites the FY25 template and FY25 targets; its Memory records the period as FY25; its knowledge base holds the FY26 template and FY26 targets. The current period is FY26. Which is the correct diagnosis and fix?
People also ask
How do I audit a Claude Project producing wrong output?
Can more than one configuration element drift at once?
Is subtly wrong output a model problem?
Watch and learn
Official Anthropic Academy lessons first, then hand-picked walkthroughs. Videos load only when you press play.
No videos curated for this concept yet
We are still curating the best official and community videos for this topic.
Official prep for this domain
Anthropic's own free prep module for this part of the syllabus, on the official prep course. Free with an Anthropic Academy sign-in.
References & primary sources
Master this concept with Archie
Practice it inside an adaptive study session. Archie, your Socratic AI tutor, tracks your mastery with Bayesian Knowledge Tracing and schedules the perfect next review.