- visitors per variant
- total · about - days (- full weeks recommended)
Method
Per-variant sample size for a two-proportion comparison:
n = ( z₁₋α/₂ · √(2·p̄·(1−p̄)) + z₁₋β · √(p₁(1−p₁) + p₂(1−p₂)) )² ÷ (p₂ − p₁)²
where p₁ is the baseline rate, p₂ the rate at your MDE, p̄ their average, α the significance level and 1−β the power. Duration = total sample ÷ (daily eligible visitors × share of traffic in the test), rounded up to full days; run in whole weeks.
Assumptions and limits
- A binary metric (converted or not) and independent visitors.
- A fixed-horizon test: decide the sample in advance and analyse once at the end. If you check results continuously, use a sequential method instead.
- No correction for multiple variants. With more than two arms, consider a stricter α.
- Traffic estimates are yours; real tests lose some users to consent and tracking gaps.
Validation
The calculator’s code is unit-tested against independent reference values computed with Python’s statistics.NormalDist. For example, a 10% baseline with a 20% relative MDE at α = 0.05 (two-sided) and 80% power gives 3,841 visitors per variant.
Learn the reasoning in sample size and test duration.