Before you launch a test, ask: if the change really works, how likely is this test to notice? That probability is statistical power. Running tests with low power is like searching a dark room with a weak torch: when you find nothing, you have learned very little.
Two definitions
- Power is the probability that the test detects an effect of a specified size, if that effect truly exists. It equals 1 minus the type II (false negative) error rate. A common target is 80%.
- Minimum detectable effect (MDE) is the smallest true effect the test is designed to detect with that power at your chosen significance level. It can be stated as relative (a 10% lift on a 4% rate, giving 4.4%) or absolute (+0.4 percentage points).
The trade-off in one example
With a 4% baseline conversion rate, 5% significance (two-sided) and 80% power, the standard two-proportion formula gives roughly:
| Relative MDE | Expected rate | Visitors per variant |
|---|---|---|
| 20% | 4.8% | about 10,300 |
| 10% | 4.4% | about 39,500 |
| 5% | 4.2% | about 154,000 |
Halving the MDE roughly quadruples the sample. Try it yourself in the sample size calculator.
How to choose an MDE
- Start from value. What is the smallest lift that would pay for building, shipping and maintaining the change? That is your business MDE.
- Check feasibility. Using your real daily traffic, how long would a test at that MDE take? Most teams cap a single test at four to six weeks, in whole weeks to cover weekly cycles.
- If it doesn’t fit, change the plan, not the maths. Options: test bolder changes that could plausibly produce bigger effects; test higher up the funnel where there is more traffic; use a more sensitive metric that still reflects value; or decide without a test, using research, and monitor after release.
Low power has a hidden cost
Underpowered tests don’t just miss real effects. When they do reach significance, the estimated effect tends to be exaggerated, because only lucky, inflated estimates cross the threshold. Teams then ship “winners” whose real impact is smaller than reported (Kohavi, Tang and Xu, 2020).
In the WXM cycle
The MDE belongs in the Architect phase plan and is locked before Validate begins. Recording the planned MDE and power with every test makes inconclusive results interpretable later.