The System Vulnerability Walkthrough for the CCAR-P Exam

Walk the path, not just the front door

Naming risk categories tells you what to look for; the system vulnerability walkthrough tells you where to look. The CCAR-P exam treats it as an analyse-level skill because the method is what turns a list of categories into a concrete assessment of one system. The technique is disciplined and simple: walk the request and data paths together, stop at every entry point, and ask two questions - what could an adversary do here, and which control currently stands in the way.

The reason to walk the whole path is that attacks do not politely confine themselves to the front door. Every place data enters or leaves the system is a candidate surface: user input, retrieved content, tool outputs, the model's own output, and the logs. A review that inspects only the initial request examines one of five points and declares the system assessed, which is exactly the incomplete review that passes and then fails in production.

Every entry point gets the same two questions

The power of the walkthrough is its uniformity. At each point, you ask what an adversary could do and which control answers it. At user input, the adversary crafts a direct injection or an oversized payload; input screening stands in the way. At retrieved content, the adversary plants an indirect injection; retrieved-content screening stands in the way. At tool outputs, a tool response carries an instruction the model trusts; the same screening applies. At the model's output, the model produces something unsafe; output screening stands in the way. And at the logs, a sensitive field is written where it should not be; log minimisation and access controls stand in the way.

The rule that makes the walkthrough rigorous is this: any point with no assigned control is a plausible attack surface, regardless of how unlikely it seems. You do not get to skip a point because an attack there feels improbable - improbable is not the same as impossible, and the walkthrough's job is to make every surface explicit so the improbable ones are decided deliberately rather than ignored by default.

The logs are a surface, and they are the one people skip

The single most commonly omitted point is the logging pipeline. Reviews naturally gravitate toward the model - its inputs and its outputs feel like where the action is - and stop at "the model's response." But logs are a data-path point that can leak sensitive fields even when the model's output is entirely clean. A response can pass every content filter and still have written a Social Security number or a full account record into a log store along the way, where a different set of eyes and a different retention policy govern it.

Because data exposure through logging is independent of the model's behaviour, no output filter would ever catch it. That independence is precisely why the walkthrough forces you to treat the log as its own entry point with its own control. Skipping it is not a small omission - it is skipping the surface where a large share of real-world data-exposure incidents actually happen.

What the CCAR-P exam trips candidates on

The first trap is stopping the walkthrough at "the model's response" and never considering the logging pipeline as an exposure point. A scenario details careful input and output screening and asks whether the system's data is protected; candidates who accept the model-centred boundary miss the log. The credited reading names the logs as an unexamined surface where sensitive fields can leak with clean output.

The second trap is assuming a control that covers one entry point automatically covers a different one. A scenario shows strong input screening and asks whether retrieved content is protected; the credited reading is no - input screening does not inspect retrieved content, so that point still needs its own control. The walkthrough exists precisely to prevent one control's coverage from being silently extended to a point it never touches.

Common misreadings to avoid

How this shows up on the exam

Domain 5 items hand you an architecture and ask you to identify its vulnerabilities or what a review missed. The reliable method is to walk all five entry points - input, retrieved content, tool outputs, model output, logs - and, at each, name the adversary action and the control. Flag every point without a control, and never let one control's coverage extend implicitly to a point it does not touch. The logs are the point candidates forget.

This method operationalises the LLM system risk taxonomy by mapping each category onto a point on the path, and it pairs directly with the three-point guarded request path whose checkpoints are the controls you check for. Its output feeds the risk assessment as a written deliverable, where each open surface becomes a documented row with a mitigation and an owner.