[go: up one dir, main page]
More Web Proxy on the site http://driver.im/ skip to main content
10.1145/3292500.3330722acmconferencesArticle/Chapter ViewAbstractPublication PageskddConference Proceedingsconference-collections
research-article

Diagnosing Sample Ratio Mismatch in Online Controlled Experiments: A Taxonomy and Rules of Thumb for Practitioners

Published: 25 July 2019 Publication History

Abstract

Accurately learning what delivers value to customers is difficult. Online Controlled Experiments (OCEs), aka A/B tests, are becoming a standard operating procedure in software companies to address this challenge as they can detect small causal changes in user behavior due to product modifications (e.g. new features). However, like any data analysis method, OCEs are sensitive to trustworthiness and data quality issues which, if go unaddressed or unnoticed, may result in making wrong decisions. One of the most useful indicators of a variety of data quality issues is a Sample Ratio Mismatch (SRM) ? the situation when the observed sample ratio in the experiment is different from the expected. Just like fever is a symptom for multiple types of illness, an SRM is a symptom for a variety of data quality issues. While a simple statistical check is used to detect an SRM, correctly identifying the root cause and preventing it from happening in the future is often extremely challenging and time consuming. Ignoring the SRM without knowing the root cause may result in a bad product modification appearing to be good and getting shipped to users, or vice versa. The goal of this paper is to make diagnosing, fixing, and preventing SRMs easier. Based on our experience of running OCEs in four different software companies in over 25 different products used by hundreds of millions of users worldwide, we have derived a taxonomy for different types of SRMs. We share examples, detection guidelines, and best practices for preventing SRMs of each type. We hope that the lessons and practical tips we describe in this paper will speed up SRM investigations and prevent some of them. Ultimately, this should lead to improved decision making based on trustworthy experiment analysis.

References

[1]
Auer, F. and Felderer, M. 2018. Current State of Continuous Experimentation: A Systematic Mapping Study. Proceedings of the 2018 44rd Euromicro Conference on Software Engineering and Advanced Applications (SEAA) (Prague, Czechia., 2018).
[2]
Bakshy, E., Eckles, D. and Bernstein, M.S. 2014. Designing and deploying online field experiments. Proceedings of the 23rd international conference on World wide web - WWW '14 (New York, New York, USA, 2014), 283--292.
[3]
Barik, T., Deline, R., Drucker, S. and Fisher, D. 2016. The Bones of the System: A Case Study of Logging and Telemetry at Microsoft. (2016).
[4]
Chen, N., Liu, M. and Xu, Y. 2018. Automatic Detection and Diagnosis of Biased Online Experiments. arXiv preprint arXiv:1808.00114. (2018).
[5]
Deng, A., Lu, J. and Litz, J. 2017. Trustworthy Analysis of Online A/B Tests. Proceedings of the Tenth ACM International Conference on Web Search and Data Mining - WSDM '17 (New York, New York, USA, 2017), 641--649.
[6]
Devore, J.L. and Berk, K.N. 2012. Modern Mathematical Statistics with Applications.
[7]
Dmitriev, P., Frasca, B., Gupta, S., Kohavi, R. and Vaz, G. 2016. Pitfalls of long-term online controlled experiments. 2016 IEEE International Conference on Big Data (Big Data) (Washington, DC, USA, Dec. 2016), 1367--1376.
[8]
Dmitriev, P., Gupta, S., Dong Woo, K. and Vaz, G. 2017. A Dirty Dozen: Twelve Common Metric Interpretation Pitfalls in Online Controlled Experiments. Proceedings of the 23rd ACM SIGKDD international conference on Knowledge discovery and data mining - KDD '17 (Halifax, Nova Scotia, Canada, 2017).
[9]
Fabijan, A., Dmitriev, P., Holmstrom Olsson, H. and Bosch, J. 2018. Effective Online Controlled Experiment Analysis at Large Scale. 2018 44th Euromicro Conference on Software Engineering and Advanced Applications (SEAA) (Prague, Czechia., Aug. 2018), 64--67.
[10]
Fabijan, A., Dmitriev, P., McFarland, C., Vermeer, L., Holmström Olsson, H. and Bosch, J. 2018. Experimentation growth: Evolving trustworthy A/B testing capabilities in online software companies. Journal of Software: Evolution and Process. (Nov. 2018), e2113.
[11]
Fabijan, A., Dmitriev, P., Olsson, H.H. and Bosch, J. 2018. Online Controlled Experimentation at Scale: An Empirical Survey on the Current State of A/B Testing. Proceedings of the 2018 44rd Euromicro Conference on Software Engineering and Advanced Applications (SEAA) (Prague, Czechia., 2018).
[12]
Fabijan, A., Dmitriev, P., Olsson, H.H. and Bosch, J. 2017. The Benefits of Controlled Experimentation at Scale. Proceedings of the 2017 43rd Euromicro Conference on Software Engineering and Advanced Applications (SEAA) (Vienna, Austria, Aug. 2017), 18--26.
[13]
Gupchup, J., Hosseinkashi, Y., Dmitriev, P., Schneider, D., Cutler, R., Jefremov, A. and Ellis, M. 2018. Trustworthy Experimentation Under Telemetry Loss. to appear in: Proceedings of the 27th ACM International on Conference on Information and Knowledge Management - CIKM '18 (Lingotto, Turin, 2018).
[14]
Gupta, S., Ulanova, L., Bhardwaj, S., Dmitriev, P., Raff, P. and Fabijan, A. 2018. The Anatomy of a Large-Scale Experimentation Platform. 2018 IEEE International Conference on Software Architecture (ICSA) (Seattle, USA, Apr. 2018), 1--109.
[15]
Kaufman, R.L., Pitchforth, J. and Vermeer, L. 2017. Democratizing online controlled experiments at Booking. com. arXiv preprint arXiv:1710.08217. (2017), 1--7.
[16]
Kevic, K., Murphy, B., Williams, L. and Beckmann, J. 2017. Characterizing Experimentation in Continuous Deployment: A Case Study on Bing. 2017 IEEE/ACM 39th International Conference on Software Engineering: Software Engineering in Practice Track (ICSE-SEIP) (May 2017), 123--132.
[17]
Kluck, T. and Vermeer, L. 2017. Leaky Abstraction In Online Experimentation Platforms: A Conceptual Framework To Categorize Common Challenges. (Oct. 2017).
[18]
Kohavi, R., Deng, A., Frasca, B., Walker, T., Xu, Y. and Pohlmann, N. 2013. Online controlled experiments at large scale. Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining - KDD '13 (Chicago, Illinois, USA, 2013), 1168.
[19]
Kohavi, R., Deng, A., Longbotham, R. and Xu, Y. 2014. Seven rules of thumb for web site experimenters. Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining - KDD '14 (New York, New York, USA, 2014), 1857--1866.
[20]
Kohavi, R. and Longbotham, R. 2017. Online Controlled Experiments and A/B Testing. Encyclopedia of Machine Learning and Data Mining. C. Sammut and G.I. Webb, eds. Springer US. 922--929.
[21]
Kohavi, R., Longbotham, R., Sommerfield, D. and Henne, R.M. 2009. Controlled experiments on the web: survey and practical guide. Data Mining and Knowledge Discovery. 18, 1 (Feb. 2009), 140--181.
[22]
Kohavi, R. and Thomke, S. 2017. The Surprising Power of Online Experiments. Harvard Business Review.
[23]
Leaky Abstractions: 2018. https://booking.ai/leaky-abstractions-in-online-experimentation-platforms-ae4cf05013f9.
[24]
Lindgren, E. and Münch, J. 2015. Software development as an experiment system: A qualitative survey on the state of the practice. Lecture Notes in Business Information Processing (Cham, May 2015), 117--128.
[25]
List of browsers that support 128-bit and 256-bit encryption:.
[26]
Mayring, P. 2002. Qualitative content analysis - research instrument or mode of interpretation. The Role of the Researcher in Qualitative Psychology. 139--148.
[27]
Mckinney, E.H. 1966. Generalized Birthday Problem. The American Mathematical Monthly. 73, 4 (Apr. 1966), 385.
[28]
Microsoft Experimentation Platform: http://www.exp-platform.com.
[29]
MIT Code: 2016. http://bit.ly/Code2016Kohavi.
[30]
Pearson, K. 1900. X. On the criterion that a given system of deviations from the probable in the case of a correlated system of variables is such that it can be reasonably supposed to have arisen from random sampling. The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science. 50, 302 (Jul. 1900), 157--175.
[31]
Runeson, P. and Höst, M. 2008. Guidelines for conducting and reporting case study research in software engineering. Empirical Software Engineering. 14, 2 (2008), 131--164.
[32]
SRM Interview Guide: 2019. https://www.dropbox.com/s/h0291u1fcqg0eze/SRM Interview Guide.pdf?dl=0.
[33]
Tang, D., Agarwal, A., Brien, D.O., Meyer, M., O'Brien, D. and Meyer, M. 2010. Overlapping experiment infrastructure. Proceedings of the 16th ACM SIGKDD international conference on Knowledge discovery and data mining - KDD '10 (New York, New York, USA, 2010), 17.
[34]
Xu, Y., Chen, N., Fernandez, A., Sinno, O. and Bhasin, A. 2015. From Infrastructure to Culture. Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining - KDD '15 (New York, New York, USA, 2015), 2227--2236.
[35]
Zhao, Z., Chen, M., Matheson, D. and Stone, M. 2016. Online Experimentation Diagnosis and Troubleshooting Beyond AA Validation. 2016 IEEE International Conference on Data Science and Advanced Analytics (DSAA) (Oct. 2016), 498--507.

Cited By

View all

Recommendations

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences
KDD '19: Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining
July 2019
3305 pages
ISBN:9781450362016
DOI:10.1145/3292500
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 25 July 2019

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. a/b testing
  2. online controlled experiments
  3. sample ratio mismatch
  4. srm

Qualifiers

  • Research-article

Conference

KDD '19
Sponsor:

Acceptance Rates

KDD '19 Paper Acceptance Rate 110 of 1,200 submissions, 9%;
Overall Acceptance Rate 1,133 of 8,635 submissions, 13%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)63
  • Downloads (Last 6 weeks)3
Reflects downloads up to 02 Dec 2024

Other Metrics

Citations

Cited By

View all
  • (2024)Improving Ego-Cluster for Network Effect MeasurementProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining10.1145/3637528.3671557(5713-5722)Online publication date: 25-Aug-2024
  • (2024)A/B testingJournal of Systems and Software10.1016/j.jss.2024.112011211:COnline publication date: 2-Jul-2024
  • (2023)Estimating Effects of Long-Term TreatmentsSSRN Electronic Journal10.2139/ssrn.4352459Online publication date: 2023
  • (2023)Variance Reduction Using In-Experiment Data: Efficient and Targeted Online Measurement for Sparse and Delayed OutcomesProceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining10.1145/3580305.3599928(3937-3946)Online publication date: 6-Aug-2023
  • (2023)Balancing Approach for Causal Inference at ScaleProceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining10.1145/3580305.3599778(4485-4496)Online publication date: 6-Aug-2023
  • (2023)Zero to Hero: Exploiting Null Effects to Achieve Variance Reduction in Experiments with One-sided TriggeringProceedings of the Sixteenth ACM International Conference on Web Search and Data Mining10.1145/3539597.3570413(823-831)Online publication date: 27-Feb-2023
  • (2023)Investigating the effects of product popularity and time restriction: The moderating role of consumers’ goal specificityProduction and Operations Management10.1111/poms.1400332:9(2723-2739)Online publication date: 1-Sep-2023
  • (2023)A/B Integrations: 7 Lessons Learned from Enabling A/B Testing as a Product FeatureProceedings of the 45th International Conference on Software Engineering: Software Engineering in Practice10.1109/ICSE-SEIP58684.2023.00033(304-314)Online publication date: 17-May-2023
  • (2023)Online Controlled Experiments and A/B TestsEncyclopedia of Machine Learning and Data Science10.1007/978-1-4899-7502-7_891-2(1-13)Online publication date: 8-Mar-2023
  • (2022)Anytime-valid inference for multinomial count dataProceedings of the 36th International Conference on Neural Information Processing Systems10.5555/3600270.3600474(2817-2831)Online publication date: 28-Nov-2022
  • Show More Cited By

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media