The short answer
You planned a 50/50 split. You got 50.8% and 49.2%. That looks harmless, but with 100,000 users it is very unlikely to be chance. That gap is a sample ratio mismatch (SRM): the split you got differs from the split you designed by more than randomness allows.
SRM matters because random assignment is the whole reason an A/B test works. If one group quietly lost some users, for example everyone on an old browser, the groups are no longer alike and any difference in conversion could come from that, not from your change (Fabijan et al., 2019; Kohavi, Tang and Xu, 2020).
How to check for SRM
Use a chi-square goodness-of-fit test on the number of users (or sessions, whatever your unit of assignment is) in each group.
- Write down the planned split, for example 50/50.
- Compute the expected count per group: total users × planned share.
- For each group, compute (observed − expected)² ÷ expected, and add them up. That sum is the chi-square statistic.
- Look up the p-value with one degree of freedom for two groups (number of groups minus one in general).
Most testing tools and any spreadsheet can do this. In Google Sheets, =CHISQ.TEST(observed_range, expected_range) returns the p-value directly.
Worked example
| Group | Planned share | Expected users | Observed users |
|---|---|---|---|
| A (control) | 50% | 50,000 | 50,800 |
| B (variation) | 50% | 50,000 | 49,200 |
- A: (50,800 − 50,000)² ÷ 50,000 = 12.8
- B: (49,200 − 50,000)² ÷ 50,000 = 12.8
- Chi-square = 25.6, p ≈ 0.0000004
A split this uneven would happen by chance far less than once in a million tests. This test has SRM. Don’t read the conversion numbers yet.
Compare a smaller gap: 50,250 versus 49,750 gives chi-square 2.5 and p ≈ 0.11. That is well within normal noise.
What usually causes SRM
Work through these in order. The first ones are the most common in practice (Fabijan et al., 2019):
- Redirects and page speed. A redirect test or a slower variation loses users before the tracking fires. The slower group shows fewer users.
- Tracking that differs between variations. The variation changed the page so the analytics tag fires later, twice, or not at all for some users.
- Bots and crawlers. Automated traffic often lands in one group or gets filtered differently.
- Targeting and eligibility. A condition that is evaluated after assignment (logged-in state, cookie consent, device) excludes users unevenly.
- Assignment bugs. Users switching groups, cached pages serving the wrong variation, or a broken hashing setup.
- Data processing. Joins, filters or deduplication in your reporting that treat the groups differently.
Segment the split by browser, device, country and day. SRM that appears in only one segment points straight at the cause.
Common mistakes
- Checking only at the end. Run the check early (after a day or two) so you can stop a broken test instead of wasting weeks.
- Ignoring it because the result looks good. A “winner” with SRM is still untrustworthy. It might even be a winner because of the broken split.
- Using conversions instead of users. The check is about how many units entered each group, not how many converted.
What to do next
Add the SRM check to your test QA and to the top of every readout. Then read how to interpret wins, losses and inconclusive results, and refresh statistical significance if the p-value step felt shaky.