The short answer
Write the report for the person who decides, not for the person who analysed. That means:
- Decision first. One sentence: what we’re doing because of this test.
- Result in business terms, as a range. Orders, leads, revenue, not “conversion rate +0.5pp”.
- What we learned about customers, in plain words.
- What happens next.
- The stats in a small box at the end, for anyone who wants them.
This is a practitioner method, not a rule from research. It comes from a simple observation: a correct report that nobody reads changes nothing.
Why most test reports don’t land
They follow the order the analyst worked in: setup, data, statistics, result. Stakeholders read the first lines, see p-values and confidence intervals, and skip to the end looking for “did it win?”. If the answer is “inconclusive”, the test looks like a waste.
Flip the order and the same content becomes useful.
A one-page structure
Decision: Roll out the delivery date line on all product pages.
The question: Do shoppers finish checkout more often when they see delivery cost and date on the product page?
What happened: Yes. Probably about 100 more orders a week at current traffic, and very likely somewhere between 20 and 190. Average order value and refunds didn’t change.
What we learned: Shoppers were leaving because of a late surprise, not the delivery price itself. Showing information earlier works better than discounting it.
Next: Show the same date in the cart and confirmation email. Research next: do returning customers want faster delivery options?
Details: 2 weeks, 41,800 users, 50/50 split (SRM check passed). Completed orders per visitor +0.5 points, 95% CI +0.1 to +0.9.
Numbers in this example are made up to show the format.
Explain uncertainty without the jargon
| Instead of | Say |
|---|---|
| “Statistically significant at 95%” | “Very unlikely to be a fluke” |
| “95% CI +0.1 to +0.9pp” | “Probably between 20 and 190 extra orders a week” |
| “Not significant” | “Any effect is too small for this test to see. It’s at most about X” |
| “p = 0.03” | Leave it in the details box |
Translate percentage points into the unit the business plans in. “0.5 points” means nothing to a finance lead. “About 100 orders a week” does.
Make flat tests and losses worth reading
A loss reported well is a gift: it stopped you shipping something harmful. Lead with that.
Decision: Keep the current pricing page. What happened: The new layout likely reduced sign-ups. We avoided rolling out a change that would have cost leads. What we learned: Visitors used the comparison table more than we thought. Removing it made decisions harder.
For inconclusive tests, say what range you ruled out: “If there is an effect, it’s smaller than 1 point, so this idea won’t deliver the 3-point lift we hoped for.”
Common mistakes
- Leading with the method. Nobody needs to know the tool’s name before the decision.
- Relative lifts on tiny bases. “+50%” on 4 orders sounds huge. Show absolute numbers.
- Burying guardrails. If something important got worse, it belongs in “What happened”, not the appendix.
- No next step. Every report should hand a question back to research.
What to do next
Before writing the report, make sure the result is solid: interpreting A/B test results. And if the hardest part is getting tests approved in the first place, read let the boss’s idea run first.