Working Paper

Fields of Gold: Web Scraping for Consumer Research

Johannes Boegershausen

Erasmus University

Abhishek Borah


Andrew Stephen

University of Oxford

Jan 24, 2021


Outlines the key challenges, state-of-the-art remedies, best practices, and corresponding standards for evaluation for web scraping in consumer research. Provides a structured workflow designed to achieve consistency and standardization in how web scraping is conducted, documented, reported, and evaluated in both the research and peer review process. The authors propose four interdependent facets necessary for generating credible scientific findings from web scraping-based research: (1) design transparency, (2) analytic reproducibility, (3) analytic robustness, and (4) effect replicability and generalizability.

By using you agree to our use of cookies as identifiers and for other features of the site as described in our Privacy Policy.