p-Hacking and False Discovery in A/B Testing – MSI

Working Paper

p-Hacking and False Discovery in A/B Testing

Ron Berman, Leonid Pekelis, Aisling Scott, and Christophe Van den Bulte

Oct 18, 2018

Uses data containing 2,101 commercial experiments from the Optimizely platform, and a regression discontinuity design to detect p-hacking, i.e., the causal effect of reaching a particular p-value on stopping behavior.

Topics

marketing analytics marketing experiments

research recap

p-Hacking in A/B Testing

Data on 2000+ commercial experiments show that 57% of online A/B experimenters p-hack, inflating the number of experiments that show significant effects. This award-winning study shows just how difficult it is to design new and effective online innovations: 70% of the effects of all experiments are truly null, i.e., not expected to yield any improvement over the baseline.

By using MSI.org you agree to our use of cookies as identifiers and for other features of the site as described in our Privacy Policy.