A Bias Correction Approach for Interference in Ranking Experiments – MSI

Working Paper

A Bias Correction Approach for Interference in Ranking Experiments

Ali Goli

University of Washington

Anja Lambrecht

Hema Yoganarasimhan

University of Washington

Mar 28, 2022

Demonstrates the presence of interference bias in the Total Average Treatment Effect (TATE) estimates of ranking algorithms based on A/B tests and provides a novel solution that can recover the true TATE of a ranking algorithm based on past A/B tests, even if those tests suffer from a combination of interference issues.

Topics

marketing analytics marketing experiments Marketing ROI Omnichannels & Customer Journey

research recap

Correcting Bias in A/B Testing of Ranking Algorithms

A/B tests to optimize ranking algorithms used by website marketplaces may be misleading if outcomes observed under one treatment depend on treatments assigned to the rest of the population. A novel approach can recover the true Total Average Treatment Effects (TATEs) of ranking algorithms based on past A/B tests, even where they suffer from such interference issues.

By using MSI.org you agree to our use of cookies as identifiers and for other features of the site as described in our Privacy Policy.