When an A/B testing tool says a result is “significant at 95%”, most people hear “there is a 95% chance the variant is better”. That is not what it means, and acting on that misreading is one of the most expensive mistakes in CRO.
What significance actually measures
A standard A/B test starts with a null hypothesis: the change makes no difference to the metric. The test then asks:
If the change truly made no difference, how surprising would data like ours be?
That surprise is the p-value: the probability, assuming no real difference, of seeing a difference at least as large as the one observed. If the p-value falls below a threshold chosen before the test (the significance level, alpha, often 0.05), the result is called statistically significant.
Three things a p-value is not
The American Statistical Association’s statement on p-values is blunt about the common misreadings (Wasserstein and Lazar, 2016):
- It is not the probability that the variant is better (or that the null hypothesis is true).
- It does not measure the size or importance of the effect. A tiny, commercially meaningless lift can be significant with enough traffic.
- A threshold alone should not drive the decision. Context, effect size, costs and risks matter.
A worked example
Control converts 400 of 10,000 visitors (4.0%). Variant converts 460 of 10,000 (4.6%).
- Absolute difference: +0.6 percentage points. Relative lift: +15%.
- A two-proportion z-test gives roughly z ≈ 2.09, p ≈ 0.037 (two-sided).
- An approximate 95% confidence interval for the difference is about +0.04 to +1.16 points.
So the result is significant at alpha 0.05. The useful part is the interval: the true lift could plausibly be anything from almost nothing to more than a point. If shipping the variant has a real cost, “almost nothing” might not pay for it. That is a business judgement, not a statistical one.
Errors you are trading off
- Type I error (false positive): declaring a winner when there is no real effect. Alpha sets how often this happens when there is truly no effect.
- Type II error (false negative): missing a real effect. Its probability is beta; power is 1 minus beta. See power and minimum detectable effect.
Lowering one error rate raises the other unless you add traffic. That is why sample size has to be planned before launch with the sample size calculator.
Mistakes that break significance
- Peeking and stopping early. Checking daily and stopping on the first significant day inflates false positives well beyond 5% (Kohavi, Tang and Xu, 2020). Fix the sample size or use a sequential method.
- Testing many metrics or segments and reporting the best one. With 20 independent comparisons at alpha 0.05, you expect about one “significant” result by chance alone.
- Ignoring data quality. A sample ratio mismatch (traffic not split as planned) can invalidate a test however small the p-value.
- Confusing significance with importance. Always state the effect size and its interval.
How this fits WXM
In the Validate phase, every test should have a primary metric, guardrails, a planned sample size and a decision rule written down before launch. Significance is one input to that rule, not the rule itself.