Sample Size and Statistical Power for LLM Experiments

The number that decides whether the result means anything

You can have a perfect hypothesis, random assignment, and a pre-specified metric, and still learn nothing, if the experiment is too small. Sample size is the component that determines whether the difference you observe between treatment and control is a real effect or just noise. The apply-level skill here is knowing what inputs set the required sample size, why LLM systems push that number higher, and why even a statistically significant result still has to clear a second, separate bar before you act on it.

What sets the sample size

Three inputs determine how many observations per group you need. The minimum detectable effect is the smallest difference you care about catching, if you want to detect a 2% improvement you need far more data than if a 10% improvement is the smallest that would interest you, because small effects hide more easily in noise. The baseline metric value is where the control currently sits, which shapes the variance around the measurement. And the required confidence level is how sure you insist on being that the result is not chance, higher confidence demands more data.

These three feed a sample-size calculation you run before the experiment, not after. Running first and asking about power later is how teams end up with a result they cannot interpret. The confidence side is usually anchored to two conventions: a required confidence of about 95% (a 5% significance level) and 80% statistical power. To make the scale concrete, detecting a 5-point improvement over a 70% baseline success rate at those conventions needs on the order of 1,500 sessions per arm, so a live split running 800 requests a day reaches a usable sample in roughly four days. The number climbs steeply as the effect you want to catch shrinks.

Why LLM experiments need more data

The variance of the thing you are measuring drives how much data you need to see through it, and LLM outputs are noisier than deterministic software. The same prompt can produce different outputs, and quality varies across inputs in ways a deterministic system does not exhibit. That extra variance raises the noise floor, so a real effect of a given size is harder to distinguish, and the sample needed to distinguish it is correspondingly larger.

The practical takeaway is a rule of thumb: an LLM experiment generally needs a bigger sample than a traditional software A/B test aiming for the same confidence and the same effect size. Sizing an LLM experiment as if it were a deterministic one systematically underpowers it.

Two failures: underpowered, and significant-but-small

An underpowered experiment is one whose sample is too small for the effect it is trying to detect. Its defining property is that it cannot tell a real effect from noise, no matter how the numbers look. A team that eyeballs a 6-point gap on 50 sessions per arm and calls it a win has an underpowered experiment: the gap is inside the noise floor and can vanish at scale. The results looking convincing is not evidence; underpowered results always look like something.

The second failure is confusing statistical with practical significance. A statistically significant result means the observed difference is unlikely to be chance given the sample. It does not mean the difference is big enough to be worth having. A change can be significant yet tiny, and shipping it means taking on the operational cost of maintaining a new version for a gain too small to justify it. Both questions, is it real, and is it big enough to matter, must be answered before declaring a winner.

What the exam trips candidates on

The first trap is running a comparison on a small sample, such as 50 sessions per group, and treating an observed gap as a real effect without checking statistical power. A scenario will show a small-sample comparison with an encouraging difference; the credited reading identifies it as underpowered, the gap within the noise floor, and refuses to conclude an effect without an adequate sample.

The second trap is declaring a statistically significant result a winner without asking whether the effect size justifies the operational cost. A scenario will present a significant but small improvement and ask whether to ship; the correct answer separates statistical from practical significance and weighs the effect against the cost of shipping and maintaining the change.

Common misreadings to avoid

How this shows up on the exam

Domain 4 questions on this knowledge point present either a small-sample comparison treated as conclusive or a significant-but-tiny result treated as a win. The reliable moves are to size the experiment in advance from the minimum detectable effect, baseline, and confidence, allow for higher LLM variance, and separate statistical significance from practical significance before acting.

This quantifies the sample-size component of the five components of an A/B test and takes the numeric threshold from a falsifiable hypothesis as its minimum detectable effect. Undersizing is one leg of an underpowered, outcome-shopped experiment, the choice of test volume interacts with live versus shadow testing, and the variance intuition connects to token distribution skew, where averages likewise mislead.