Research Recaps

p-Hacking in A/B Testing

Key Takeway

Data on 2000+ commercial experiments show that 57% of online A/B experimenters p-hack, inflating the number of experiments that show significant effects. This award-winning study shows just how difficult it is to design new and effective online innovations: 70% of the effects of all experiments are truly null, i.e., not expected to yield any improvement over the baseline.

By using MSI.org you agree to our use of cookies as identifiers and for other features of the site as described in our Privacy Policy.