The short answer
Every test readout answers three questions, in this order:
- Can I trust it? Validity checks before any metric.
- What happened? The effect and its confidence interval, not just “significant or not”.
- What do we do now? Ship, stop, or learn and retest.
Skip step 1 and you might roll out a broken result. Skip step 3 and the test was just an expensive opinion poll.
Step 1: Check validity first
- Sample ratio mismatch. Did each group get the planned share of users? If not, stop here. See sample ratio mismatch.
- Planned duration reached. Did the test run its full, pre-planned sample and whole weeks? Reading early inflates false positives (Kohavi, Tang and Xu, 2020).
- Tracking parity. Does the primary metric fire the same way in both variations? QA screenshots and event counts help.
- Nothing else changed. No sitewide sale, outage or campaign that hit one group differently.
Step 2: Read the effect, not just the p-value
A p-value tells you how surprising the data would be if there were no difference. It does not tell you how big the effect is or how sure to be about it (Wasserstein and Lazar, 2016). The confidence interval does more work.
Worked example: inconclusive
20,000 users per group. Control converts 800 (4.0%), variation 860 (4.3%).
- Difference: +0.30 percentage points
- 95% confidence interval: −0.09 to +0.69 points
The interval includes zero, so the result is inconclusive at 95%. It also tells you something useful: a lift of more than about 0.7 points is unlikely. If you needed a 1-point lift for this to be worth it, you’ve learned the idea probably can’t deliver that.
Worked example: a win
Same traffic, variation converts 900 (4.5%).
- Difference: +0.50 points (a relative lift of 12.5%)
- 95% confidence interval: +0.10 to +0.90 points
The whole interval sits above zero. Report it as “about half a point, plausibly anywhere from 0.1 to 0.9”, not as “12.5% uplift guaranteed”.
Step 3: Decide
| Result | What it means | What to do |
|---|---|---|
| Win (interval above zero, guardrails fine) | The change very likely helps | Roll it out, then look for where else the learning applies |
| Loss (interval below zero) | The change very likely hurts | Don’t ship. Ask which part of the hypothesis was wrong |
| Inconclusive | Effect smaller than you could detect | Decide if a small effect still matters; ship on other grounds, iterate bolder, or move on |
| Guardrail broken | Primary metric up, something important down | Usually don’t ship. Understand the trade-off first |
Write the decision rule before the test launches. Then this step is a lookup, not a debate.
Turn results into learnings
A result is a fact about one page and one period. A learning is what it tells you about your users. Record both in a simple test library:
- Hypothesis, evidence behind it, and the mechanism you expected
- Result with confidence interval and the decision
- What it says about the mechanism (supported, contradicted, unclear)
- The next question it raises
Over time, the library shows which kinds of ideas work for your audience. That is how you get faster.
Common mistakes
- Peeking and stopping on a good day. Plan the sample size with the sample size calculator.
- Slicing until something wins. Segment results are hypotheses for the next test, not conclusions.
- Calling flat tests failures. If the team only celebrates wins, people stop testing bold ideas.
What to do next
Learn how to write test reports people actually read, so your decision lands with the people who make it.