Failure-Boundary Design in Orchestration (CCAR-P)

Where multi-agent systems actually break

A fan-out/fan-in orchestration can look correct on its happy path, decompose, run, synthesize, and still be dangerous, because the danger lives at the failure boundaries, not the happy path. The Claude Certified Architect - Professional (CCAR-P) exam treats failure-boundary design as an analyse-level skill because reviewing an orchestration means looking specifically for what happens when a subagent fails, when an irreversible action fires, and when coverage is incomplete. Multi-agent systems fail silently by default: a dropped unit or an unreviewed irreversible action produces no error, so the controls that catch them have to be designed in deliberately.

The controls are retry, gap-flagging, and human gates. Retry re-runs a failed subagent so its unit is not lost. Gap-flagging surfaces a unit that could not be completed rather than omitting it. A human gate stops an irreversible action from firing without review. Together they form the failure boundary that keeps failures and irreversible actions from passing silently into the final output.

Silent drops: the missing retry or gap-flag

When a subagent fails and there is neither a retry nor a gap-flag, its unit of work simply vanishes from the synthesis. The orchestrator aggregates whatever came back, and the result looks complete because nothing announces the absence. This is the most insidious multi-agent failure: a due-diligence report that silently omits one filing, a classification pass that quietly skips a batch, a coverage sweep missing a section. The output is not obviously wrong, it is incomplete in a way no one can see. The fix is to retry failed units and, when a unit genuinely cannot be completed, flag the gap explicitly so the synthesis reports partial coverage rather than passing itself off as complete.

Irreversible actions need a gate

The second failure boundary is around irreversible downstream actions. An action that cannot be undone, auto-archiving records, auto-sending a message, committing a transaction, taken with no human gate is a control-boundary defect regardless of how well the rest of the pipeline works. The reasoning is that the quality of the decompose-run-synthesize path does not matter if a single wrong output triggers an irreversible action no one reviewed. The severity is set by the irreversibility, not by the average correctness of the system. Any such action belongs behind a human gate, so a person signs off before the irreversible step fires.

Flag the gap, do not hide it

The unifying principle is honesty about coverage. A well-designed orchestration flags coverage gaps rather than silently presenting partial results as complete. Whether a unit failed, was skipped, or could not be reached, the synthesis should say so, because a partial result labelled complete is worse than a partial result labelled partial: it invites decisions made on incomplete information. Underpinning all of this is a shared trace ID propagated to every subagent, which is what makes a failure diagnosable after the fact, so you can reconstruct which unit failed and why.

Recoverable and unrecoverable failures are not symmetric

Where a failure lands decides whether it can be recovered. A subagent failure is usually recoverable: if one unit fails, the orchestrator can retry it, route it elsewhere, or drop it and flag the gap while the rest of the work proceeds. An orchestrator failure is usually not: the agent that owns the goal and holds the synthesis loses its thread, the whole run fails, and any partial subagent work is stranded. Design for that asymmetry, make subagent work idempotent and retryable, and protect the orchestrator's state by checkpointing progress so a failed run resumes rather than restarts. A third boundary sits at synthesis itself: when two subagents return conflicting results, the orchestrator needs an explicit conflict-resolution rule or an escalation to a human, not a silent pick.

The coverage check that turns gap-flagging from an intention into a guarantee is a count reconciliation: the number of results returned must equal the number of units dispatched, or the run flags the difference before anyone reads the synthesis. Without that rule, a fan-out that dispatched fifty units and received forty-eight reports "forty-eight reviewed" instead of "two are missing," and the confident, complete-looking summary is exactly what makes the gap invisible.

What the CCAR-P exam trips candidates on

The exam tests two traps. The first is reviewing an orchestration diagram and approving it because the happy path of decompose, run, and synthesize looks correct, missing the failure-handling gaps. The credited answer looks past the happy path to ask what happens when a subagent fails and whether any irreversible action fires unreviewed.

The second is treating "the subagents ran successfully" as sufficient evidence the synthesis is complete, without checking for silently dropped failed units. A scenario will report success and imply completeness, and the reliable reading notices that without retry and gap-flagging, a failed unit could have vanished, so "no errors surfaced" does not mean "nothing was dropped."

Common misreadings to avoid

How this shows up on the exam

Domain 1 questions on this knowledge point present an orchestration diagram and ask which components carry a control or failure-boundary defect. The reliable reading looks past the happy path for three things: a missing retry or gap-flag that would silently drop a failed unit, an irreversible action with no human gate, and any synthesis that presents partial coverage as complete rather than flagging the gap.

Failure-boundary design builds on orchestrator-subagent fan-out and fan-in and feeds observability across a multi-agent pipeline, which the shared trace ID supports. The human-gate control is the same one from automation vs augmentation and designing the feedback loop, and it sets up assigning work by deterministic guarantee.