Module 5 of 8

Experiment science

Sample sizes, significance and how to read a result honestly. Plain words first, maths second.

Lessons

1. 1

   Causal inference and randomization

   Explain why randomization supports causal claims

   Coming in a monthly release

2. 2

   Conversion rates and uncertainty

   Read a conversion rate with its interval

   Coming in a monthly release

3. 3

   Power and minimum detectable effect

   Plan for an effect worth detecting

   Coming in a monthly release

4. 4

   Sample size and duration

   Compute sample size and run time

   Coming in a monthly release

5. 5

   Frequentist inference and stopping rules

   Avoid peeking and know your stopping rule

   Coming in a monthly release

6. 6

   Bayesian inference and prior sensitivity

   Interpret posteriors and check priors

   Coming in a monthly release

7. 7

   Multiple comparisons and sample-ratio mismatch

   Control false discoveries and detect SRM

   Coming in a monthly release

8. 8

   Interpret wins, losses and inconclusive results

   Decide honestly whatever the outcome

   Coming in a monthly release