Key Takeway

Data on 2000+ commercial experiments show that 57% of online A/B experimenters p-hack, inflating the number of experiments that show significant effects. This award-winning study shows just how difficult it is to design new and effective online innovations: 70% of the effects of all experiments are truly null, i.e., not expected to yield any improvement over the baseline.