October 17, 2023 | 12:00 pm - 12:30 pm ET 


MSI Webinar: A Bias Correction Approach for Interference in Ranking Experiments

Online marketplaces use ranking algorithms to determine the rank-ordering of items sold on their websites. The standard practice is to determine the optimal algorithm using A/B tests. We present a theoretical framework to characterize the Total Average Treatment Effect (TATE) of a ranking algorithm in an A/B test and show that naive TATE estimates can be biased due to interference. We propose a bias-correction approach that can recover the TATE of a ranking algorithm based on past A/B tests, even if those tests suffer from a combination of interference issues. Our solution leverages data across multiple experiments and identifies observations in partial equilibrium in each experiment, i.e., items close to their positions under the true counterfactual equilibrium of interest. We apply our framework to data from a travel website and present comprehensive evidence for interference bias in this setting. Next, we use our solution concept to build a customized deep learning model to predict the true TATE of the main algorithm of interest in our data. Counterfactual estimates from our model show that naive TATE estimates of click and booking rates can be biased by as much as 15% and 29%, respectively.


Ali Goli

University of Washington

Ali Goli is an Assistant Professor of Marketing at the Michael G. Foster School of Business, University of Washington. His research focuses on marketing and public policy, field experiments on two-sided platforms, advertising, and online education. He earned his Ph.D. from the University of Chicago Booth School of Business.

Pricing & Registration

October 17, 2023 | 12:00 pm - 12:30 pm ET

Price: $0.00


*Please note that candid photos will be taken during all in-person and virtual events. We may use your photo on our website, in our newsletter or other digital media. All Q&A sessions throughout the event will be captured. Please note that any questions and related discussion will be included in the recordings of the event.

By using MSI.org you agree to our use of cookies as identifiers and for other features of the site as described in our Privacy Policy.