The Five Components of a Structured A/B Test

The anatomy every structured experiment shares

When you want to know whether a prompt change, a model swap, or a new retrieval strategy actually improves a live Claude system, guessing from a few examples will not do. A structured A/B test is the instrument, and it has five parts that must all be present for the result to mean anything. The CCAR-P exam expects you to name them and to understand the two properties, probabilistic outputs and the need for pre-specification, that make LLM experiments distinctive. This is a remember-level foundation for everything else in the experimentation task statement.

The five parts, one at a time

The hypothesis is the specific, falsifiable claim the experiment tests, naming the change, the metric it should move, and by how much. Without it, any result can be reinterpreted as a win after the fact.

The treatment group receives the new version, the changed prompt, the swapped model, the revised retrieval, and the control group receives the current version unchanged. The comparison between them is the whole point: the control is the baseline that tells you what "no change" looks like over the same period.

The primary metric is the single measure, defined before the experiment starts, that decides the outcome, task success rate, cost per completion, a satisfaction proxy. And the sample size is the number of observations per group needed for the difference between them to be statistically meaningful rather than noise. Undersize it and the experiment cannot tell a real effect from randomness.

Why LLM experiments are different

Classical A/B testing usually compares deterministic systems: the same input produces the same output, so the only variation is the change under test. LLM outputs are probabilistic, the same prompt can produce different outputs, which adds a layer of variance on top of everything else. That extra noise makes results harder to read and interaction effects, where a change helps on some input types and hurts on others, harder to control.

The practical consequence, developed further in the sample-size and diagnosis knowledge points, is that LLM experiments generally need larger samples than traditional software tests to reach the same confidence. The noise floor is higher, so you need more data to see a real signal above it.

Two rules that keep the comparison honest

Two of the components carry rules that are easy to state and easy to violate. Assignment to treatment or control must be random, and consistent for a given user or session, so the two groups are comparable and a user does not bounce between versions mid-conversation. Assign non-randomly, by time of day, by request type, and the groups differ in ways that have nothing to do with the change, so any observed gap is confounded.

And the primary metric must be pre-specified. Choosing it after seeing which metric moved favourably is outcome-shopping: with enough metrics on the dashboard, one will have drifted in the right direction by chance, and picking it turns an experiment into a search for a flattering number. Naming the metric in advance is what makes the result a test.

What the exam trips candidates on

The first trap is running a comparison with no pre-specified primary metric, then picking whichever metric moved favourably after the fact. A scenario will describe a team that ran two versions, looked at the dashboard, and reported the metric that happened to improve; the credited reading names this as outcome-shopping and requires the metric to be fixed before the run.

The second trap is non-random assignment, such as routing requests to treatment or control by time of day. A scenario will split traffic in a way that makes the groups non-comparable; the correct answer flags that the apparent difference may be an artifact of the assignment rather than the change, and requires random, per-user-consistent assignment.

Common misreadings to avoid

How this shows up on the exam

Domain 4 questions on this knowledge point present an experiment missing one or more of the five components, usually a non-random split or a post-hoc metric, and ask what is wrong. The reliable checklist is hypothesis, treatment, control, pre-specified primary metric, and adequate sample size, plus random, per-user-consistent assignment and awareness that LLM noise raises the bar.

This foundation leads into writing a falsifiable hypothesis, which sharpens component one, and into sample size and statistical power, which quantifies component five. It sets up the choice between live A/B and shadow testing and the compound-failure diagnosis in an underpowered, outcome-shopped experiment.