Diagnosing an Underpowered, Outcome-Shopped Experiment

Three ways a convincing result can be worthless

The most dangerous experiment result is the one that looks like a clear win and is not. This evaluate-level knowledge point is about recognizing the compound failure that produces exactly that: an undersized sample, an uncontrolled input distribution between the arms, and a metric chosen after the fact. Each is independently fatal, and they often appear together, reinforcing one another into a result that feels solid and means nothing. The skill is diagnosing all three and understanding that fixing only one still leaves the result invalid.

Failure one: the sample is too small

The first failure is a sample too small for the effect being claimed. A comparison of 50 sessions per arm on a noisy LLM metric cannot distinguish a real effect from random variation, because a several-point gap is well inside the noise floor at that size. The gap looks like signal, but it is the kind of fluctuation that appears and vanishes as the sample grows. Its signature is that the apparent effect disappears when the experiment is rerun at an adequate scale, which is exactly what happens when the small win settles back to nothing weeks later.

Failure two: the arms saw different inputs

The second failure is an uncontrolled input distribution. If assignment is not properly randomized and held consistent, one arm can happen to receive an easier or harder mix of inputs than the other. When that happens, the metric gap between the arms reflects which inputs each arm saw, not the change under test. The treatment arm might look better simply because it drew fewer edge cases, a sampling artifact wearing the costume of a real improvement. This failure is independent of sample size: even a large experiment produces a biased result if the arms are not comparable.

Failure three: the metric was chosen afterward

The third failure is post-hoc metric selection, outcome-shopping. If the primary metric is picked after seeing which metric moved favourably, the experiment stops being a test. With several metrics on the dashboard and noisy outputs, one is likely to have drifted in a flattering direction by chance, and selecting it retroactively guarantees a "win" that carries no evidential weight. The tell is that had a different metric moved, the team would have reported that one instead, meaning no possible result could have counted as a failure.

Any one is fatal, and they compound

The crucial evaluative point is that these three are independent and each is individually sufficient to invalidate the result. A perfectly sized, perfectly balanced experiment is still worthless if the metric was chosen after the fact. A perfectly pre-specified metric on a huge sample is still worthless if one arm saw easier inputs. You do not get to fix the most obvious problem and salvage the conclusion; a sound result requires all three to be right at once. And in practice they cluster, the same informal, rushed process that undersizes a sample also tends to skip randomization and shop for a metric, so a flawed experiment usually carries more than one of these defects.

What the exam trips candidates on

The first trap is deploying a change because a 50-session comparison showed a 6-point improvement, without checking sample size, input-distribution balance, or metric pre-specification. A scenario will present exactly that small comparison as grounds to ship; the credited reading refuses, and names which of the three failures apply.

The second trap is attributing an early apparent win entirely to the change when the treatment arm happened to receive an easier input mix. A scenario will show an apparent gain that is really an input-distribution artifact; the correct answer separates the change's effect from the sampling artifact and declines to credit the change. Both traps reward diagnosing the specific failures rather than accepting a convincing-looking number.

Common misreadings to avoid

How this shows up on the exam

Domain 4 questions on this knowledge point present a small, informal comparison declared a winner and ask you to diagnose it. The reliable approach checks all three independent failures, sample size, input-distribution balance, and metric pre-specification, recognizes that any one is fatal, and declines to credit the change until a properly designed experiment confirms it.

This is the evaluate-level capstone of the experimentation task statement, drawing the undersizing failure from sample size and statistical power and the missing-metric failure from a falsifiable hypothesis, both grounded in the five components of an A/B test. It rhymes structurally with diagnosing a stale eval suite and change attribution, two other places where a confident-looking signal hides a broken measurement.