“How long should I run this test?” has a precise answer, and it is decided before the test starts.
The four inputs
- Baseline conversion rate of the primary metric on the tested page or step.
- Minimum detectable effect (MDE): the smallest effect worth detecting. See power and MDE.
- Significance level (alpha): commonly 5%.
- Power: commonly 80%.
The sample size calculator applies the standard two-proportion formula to these inputs and shows the exact maths it used.
From sample size to duration
days = total sample across all variants ÷ (daily eligible visitors × share of traffic in the test)
Example: the calculator says 10,300 visitors per variant for a two-variant test, so 20,600 in total. Your product page gets 1,500 eligible visitors a day and the full audience is in the test: 20,600 ÷ 1,500 ≈ 14 days. Run 14 days: exactly two full weeks.
Rules that keep the duration honest
- Round up to whole weeks. Behaviour differs by weekday; partial weeks over-represent some days.
- Run at least one, ideally two, full weeks even if traffic is huge.
- Stop at the planned sample, not at a nice-looking day. Stopping at the first significant reading inflates false positives (Kohavi, Tang and Xu, 2020).
- Watch out for unusual periods. Sales events, campaigns and outages can make the sample unrepresentative. Note them in the readout.
- Check the split. If a 50/50 test shows a clearly unequal split, investigate a sample ratio mismatch before reading results.
- visitors per variant
- total · about - days (- full weeks recommended)
When not to run the test
If the duration comes out at many months, the test cannot answer your question in a useful time. That is a finding in itself: in the WXM Architect phase, such ideas either become bolder, move to a higher-traffic step, or are decided on research evidence and monitored after release.