Writing a Specific, Falsifiable A/B Test Hypothesis

A hypothesis is a commitment made before the data

The hypothesis is the first of the five A/B-test components, and it is the one that governs whether the experiment can be honest. Its job is to commit, in advance, to exactly what result would count as success and what would count as failure. If it cannot do that, the experiment has no way to fail, and an experiment that cannot fail is not a test, it is a search for a flattering number. The understand-level skill here is knowing what a hypothesis must contain to be falsifiable, and recognizing the vague statements that only look like hypotheses.

The four things a hypothesis must name

A usable hypothesis names four things. First, the treatment: the exact change being tested, not "a better prompt" but the specific revision. Second, the expected direction of the primary metric: which metric should move, and which way. Third, a numeric threshold: how much it must move to count as a win, because "goes up" is not a decision rule, "goes up by at least 5%" is. Fourth, constraints on secondary metrics: what must not get worse for the win to count, so that a primary-metric gain bought at the cost of a secondary-metric loss is not mistaken for progress.

Put together, a well-formed hypothesis reads like a sentence you could be wrong about: "Replacing the summarize instruction with an extract-the-three-action-items instruction will increase task success rate by at least 5% without degrading latency p95." Every clause is a commitment the data can contradict.

Why the vague version fails

"The new prompt is better" fails on every count. It names no treatment (better how, changed how?), no metric (better on what?), and no threshold (better by how much?). Because it commits to nothing, no result can contradict it. If task success rises, it was better; if cost falls, it was better; if nothing moves but a user says a nice thing, it was better. A claim compatible with every outcome is a claim about nothing, and it leaves the door open to declaring victory whatever the data shows.

This is the deep reason falsifiability matters. Without a hypothesis that states in advance what failure looks like, the experiment becomes reinterpretable after the fact, and post-hoc reinterpretation is exactly how noise gets shipped as a win.

The threshold is not optional

It is worth isolating the threshold, because a hypothesis can name a treatment and a primary metric and still be unusable if it omits the number. "The new prompt will increase task success rate" has a treatment and a metric but no bar, so any positive blip, however tiny or noisy, technically satisfies it. The threshold is what lets you decide before the run what magnitude of change is worth acting on, and it is the input the sample-size calculation needs. A hypothesis without a threshold cannot tell you in advance what counts as a win, which defeats the purpose of writing it down.

What the exam trips candidates on

The first trap is accepting a vague hypothesis like "the new version performs better" as sufficient grounds to launch an experiment. A scenario will present exactly that sentence as the team's hypothesis; the credited reading rejects it as unfalsifiable and rewrites it to name the treatment, the primary metric, a numeric threshold, and secondary-metric constraints.

The second trap is a hypothesis with a primary metric but no threshold. A scenario will offer "the change will increase task success rate," which sounds specific but sets no bar, so any positive movement satisfies it. The correct answer supplies the numeric threshold, making it possible to say in advance what counts as a win.

Common misreadings to avoid

How this shows up on the exam

Domain 4 questions on this knowledge point hand you a vague or threshold-less hypothesis and ask you to fix it. The reliable rewrite names the treatment, the primary metric and its direction, a numeric threshold, and secondary-metric constraints, so the experiment has a defined failure condition before it runs.

This sharpens the hypothesis component from the five components of an A/B test, and its numeric threshold is the input to sample size and statistical power. The discipline of deriving a threshold mirrors translating a requirement into a measurable threshold on the eval side, and a missing hypothesis is one leg of the compound failure in an underpowered, outcome-shopped experiment.