An A/B test (or split test) shows randomly selected visitors either the current experience (A, the control) or a changed version (B, the variation), then compares how each group behaves on a metric chosen in advance.
You changed the headline and sales went up. Was it the headline? Maybe. Or maybe it was payday, a competitor’s outage, or the email that went out on Tuesday. An A/B test is how you find out.
Why randomization matters
Comparing “this month versus last month” mixes your change with everything else that changed: campaigns, seasonality, competitors, the news. Random assignment means the two groups differ, on average, only in the experience you changed. That is what lets you say the change caused the difference, within the uncertainty the statistics describe (Kohavi, Tang and Xu, 2020).
When A/B testing is the right tool
- You have a specific, research-backed hypothesis.
- The page or step has enough traffic to detect a worthwhile effect in a few weeks. Check with the sample size calculator.
- The change is reversible and safe to show to part of your audience.
When those don’t hold, use research, usability testing or a staged rollout with monitoring instead.
The process, mapped to WXM
Phase 1 of 4
Wonder: First, ask better questions.
Look before touching anything: analytics and tag audits, session recordings, heatmaps and funnel baselines show who arrives, what they do, where they hesitate and when they leave.
Phase 2 of 4
Architect: Turn the "why" into a plan worth building.
Diagnose why people do not convert, then translate the evidence into a structured program. Each hypothesis earns its place through impact, evidence, effort and test feasibility.
Phase 3 of 4
Validate: Build the variation. Let the evidence decide.
Design and build variations, QA the experience and tracking, and launch with defined success metrics. Tests run to an appropriate decision point with guardrails and statistical rigor.
Phase 4 of 4
Expand: Roll out what works. Build on what you learn.
Roll winners out wherever the learning applies, raise testing velocity and feed new questions back into Wonder, with senior review at every step.
Each cycle makes the next one smarter: what Expand learns becomes the starting point for Wonder. WXM is the Wave Experimentation Model by Wavesy.
Wonder: find the problem. Analytics show where people drop off; research shows why. See CRO research methods.
Architect: write the hypothesis and plan.
- A hypothesis that links evidence, change and expected effect.
- One primary metric tied to the business outcome, plus guardrail metrics that must not get worse (for example refund rate, average order value, page speed).
- The MDE, significance level, power and sample size, and therefore the duration.
- A decision rule: what you will do for a win, a loss and an inconclusive result.
- Prioritize the idea against others with PXL.
Validate: build, QA, run, analyse.
- QA every variant on real devices and browsers, including accessibility and tracking.
- Launch to the planned allocation, then leave it alone until the planned sample is reached.
- Check data quality first (is the split as planned?), then read the effect and its confidence interval. See statistical significance.
Expand: decide and share.
- Roll out winners; consider where else the learning applies.
- Write up every result, including losses and inconclusive tests, so the next Wonder phase starts smarter.
Summary: plan everything that decides the result (metric, guardrails, sample size, decision rule) before launch, then leave the test alone.
Pro tip: if building a polished variation is what’s slowing you down, test a quick, honest version first. A clean wireframe-level change that’s live this week teaches you more than a pixel-perfect one that ships next quarter. Done is better than perfect, as long as the variation is QA’d and accessible.
Common mistakes
- Testing without a research-based reason.
- Changing the test mid-flight (allocation, targeting, variant content).
- Stopping early on a good day (“peeking”).
- Reporting the best of many metrics or segments as if it were the plan.
- Ignoring guardrails, then shipping a “winner” that hurts margin.
A/B testing and SEO
Google’s guidance is straightforward: don’t show search engines different content from users (cloaking), point variant URLs to the original with rel="canonical", use 302 redirects for redirect tests, and end the test once you have your answer (Google Search Central).