More Web Proxy on the site http://driver.im/

research-article

The evolution of continuous experimentation in software product development: from data to a data-driven organization at scale

Authors:

Aleksander Fabijan,

Pavel Dmitriev,

Helena Holmström Olsson,

Jan BoschAuthors Info & Claims

ICSE '17: Proceedings of the 39th International Conference on Software Engineering

Pages 770 - 780

https://doi.org/10.1109/ICSE.2017.76

Published: 20 May 2017 Publication History

Abstract

Software development companies are increasingly aiming to become data-driven by trying to continuously experiment with the products used by their customers. Although familiar with the competitive edge that the A/B testing technology delivers, they seldom succeed in evolving and adopting the methodology. In this paper, and based on an exhaustive and collaborative case study research in a large software-intense company with highly developed experimentation culture, we present the evolution process of moving from ad-hoc customer data analysis towards continuous controlled experimentation at scale. Our main contribution is the "Experimentation Evolution Model" in which we detail three phases of evolution: technical, organizational and business evolution. With our contribution, we aim to provide guidance to practitioners on how to develop and scale continuous experimentation in software organizations with the purpose of becoming data-driven at scale.

A/B testing; continuous experimentation; data science; customer feedback; continuous product innovation; Experimentation Evolution Model; product value; Experiment Owner

References

[1]

D. J. Patil, "Building Data Science Teams," Oreilly Radar, pp. 1--25, 2011.

[2]

A. Fabijan, H. H. Olsson, and J. Bosch, "Customer Feedback and Data Collection Techniques in Software R&D: A Literature Review," in Software Business, ICSOB 2015, 2015, vol. 210, pp. 139--153.

[3]

G. Westerman, M. Tannou, D. Bonnet, P. Ferraris, and A. McAfee, "The Digital Advantage: How Digital Leaders Outperform their Peers in Every Industry," MIT Sloan Manag. Rev., pp. 1--24, 2012.

[4]

R. Kohavi and R. Longbotham, "Online Controlled Experiments and A/B Tests," in Encyclopedia of Machine Learning and Data Mining, no. Ries 2011,2015, pp. 1--11.

[5]

H. H. Olsson and J. Bosch, The HYPEX model: From opinions to data-driven software development. 2014.

[6]

H. H. Olsson and J. Bosch, "Towards continuous customer validation: A conceptual model for combining qualitative customer feedback with quantitative customer observation," in Lecture Notes in Business Information Processings, vol. 210, pp. 154--166.

[7]

F. Fagerholm, A. S. Guinea, H. Mäenpää, and J. Munch, "Building Blocks for Continuous Experimentation," Proc. 1st Int. Work. Rapid Contin. Softw. Eng., pp. 26--35, 2014.

Digital Library

[8]

F. Fagerholm, A. S. Guinea, H. Mäenpää, and J. Munch, "The RIGHT model for Continuous Experimentation," J. Syst. Softw., vol. 0, pp. 1--14, 2015.

[9]

M. Kim, T. Zimmermann, R. DeLine, and A. Begel, "The emerging role of data scientists on software development teams," in Proceedings of the 38th International Conference on Software Engineering - ICSE '16, 2016, no. MSR-TR-2015-30, pp. 96--107.

Digital Library

[10]

P. Rodriguez et al., "Continuous Deployment of Software Intensive Products and Services: A Systematic Mapping Study," J. Syst. Softw., 2015.

[11]

A. Fabijan, H. H. Olsson, and J. Bosch, "The Lack of Sharing of Customer Data in Large Software Organizations: Challenges and Implications," in 17th International Conference on Agile Software Development XP2016, 2016, pp. 39--52.

[12]

R. C. Martin, Agile Software Development, Principles, Patterns, and Practices. 2002.

Digital Library

[13]

H. H. Olsson, H. Alahyari, and J. Bosch, "Climbing the `Stairway to heaven' - A mulitiple-case study exploring barriers in the transition from agile development towards continuous deployment of software," in Proceedings - 38th EUROMICRO Conference on Software Engineering and Advanced Applications, SEAA 2012, 2012, pp. 392--399.

Digital Library

[14]

S. Mujtaba, R. Feldt, and K. Petersen, "Waste and lead time reduction in a software product customization process with value stream maps," in Proceedings of the Australian Software Engineering Conference, ASWEC, 2010, pp. 139--148.

Digital Library

[15]

E. M. Goldratt and J. Cox, The Goal: A Process of Ongoing Improvement, vol. 2nd rev. e, no. 337 p. 2004.

[16]

D. Ståhl and J. Bosch, "Continuous integration flows," in Continuous software engineering, vol. 9783319112, 2014, pp. 107--115.

[17]

E. Ries, The Lean Startup: How Today's Entrepreneurs Use Continuous Innovation to Create Radically Successful Businesses. 2011.

[18]

G. Castellion, "Do It Wrong Quickly: How the Web Changes the Old Marketing Rules by Mike Moran.," J. Prod. Innov. Manag., vol. 25, no. 6, pp. 633--635, 2008.

[19]

The Standish Group, "The standish group report," Chaos, vol. 49, pp. 1--8, 1995.

[20]

J. Manzi, Uncontrolled: the surprising payoff of trial-and-error for business, politics, and society. Basic Books, 2012.

[21]

P. Bosch-Sijtsema and J. Bosch, "User Involvement throughout the Innovation Process in High-Tech Industries," J. Prod. Innov. Manag., vol. 32, no. 5, pp. 1--36, 2014.

[22]

H. H. H. H. Olsson and J. Bosch, "From opinions to data-driven software R&D: A multi-case study on how to close the `open loop' problem," in Proceedings - 40th Euromicro Conference Series on Software Engineering and Advanced Applications, SEAA 2014, 2014, pp. 9--16.

Digital Library

[23]

M. L. T. Cossio et al., A/B Testing - The most powerful way to turn clicks into customers, vol. XXXIII, no. 2. 2012.

[24]

R. C. Van Nostrand, "Design of Experiments Using the Taguchi Approach: 16 Steps to Product and Process Improvement," Technometrics, vol. 44, no. 3, pp. 289--289, Aug. 2002.

[25]

H. Hohnhold, D. O'Brien, and D. Tang, "Focusing on the Long-term," in Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining - KDD '15, 2015, pp. 1849--1858.

Digital Library

[26]

R. Kohavi, A. Deng, and R. Longbotham, "Seven Rules of Thumb for Web Site Experimenters," Kdd, pp. 1--11, 2014.

Digital Library

[27]

R. Kohavi, R. Longbotham, D. Sommerfield, and R. M. Henne, "Controlled experiments on the web: Survey and practical guide," Data Min. Knowl. Discov., vol. 18, pp. 140--181, 2009.

Digital Library

[28]

R. Kohavi, A. Deng, B. Frasca, T. Walker, Y. Xu, and N. Pohlmann, "Online controlled experiments at large scale," in Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining, 2013, pp. 1168--1176.

Digital Library

[29]

D. Tang, A. Agarwal, D. O'Brien, and M. Meyer, "Overlapping experiment infrastructure," in Proceedings of the 16th ACM SIGKDD international conference on Knowledge discovery and data mining - KDD '10, 2010, p. 17.

Digital Library

[30]

T. Barik, R. Deline, S. Drucker, and D. Fisher, "The Bones of the System: A Case Study of Logging and Telemetry at Microsoft," 2016.

Digital Library

[31]

P. Runeson and M. Höst, "Guidelines for conducting and reporting case study research in software engineering," Empir. Softw. Eng., vol. 14, no. 2, pp. 131--164, 2008.

Digital Library

[32]

P. Mayring, "Qualitative content analysis - research instrument or mode of interpretation," in The Role of the Researcher in Qualitative Psychology, 2002, pp. 139--148.

[33]

K. M. Eisenhardt, "Building Theories from Case Study Research.," Acad. Manag. Rev., vol. 14, no. 4, pp. 532--550, 1989.

[34]

J. Bosch and U. Eklund, "Eternal embedded software: Towards innovation experiment systems," in Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 2012, vol. 7609 LNCS, no. PART 1, pp. 19--31.

Digital Library

[35]

A. Fabijan, H. H. Olsson, and J. Bosch, "Time to Say `Good Bye': Feature Lifecycle," in 42th Euromicro Conference on Software Engineering and Advanced Applications (SEAA), Limassol, Cyprus. 31 Aug.-2 Sept. 2016, 2016, pp. 9--16.

[36]

"Hypothesis Kit for A/B testing." {Online}, Available: http://www.experimentationhub.com/hypothesis-kit.html.

[37]

D. Yuan, S. Park, and Y. Zhou, "Characterizing logging practices in open-source software," in Proceedings - International Conference on Software Engineering, 2012, pp. 102--112.

Digital Library

[38]

Q. Fu et al., "Where do developers log? an empirical study on logging practices in industry," Companion Proc. 36th Int. Conf. Softw. Eng. - ICSE Companion 2014, pp. 24--33, 2014.

Digital Library

[39]

K. Rodden, H. Hutchinson, and X. Fu, "Measuring the User Experience on a Large Scale: User-Centered Metrics for Web Applications," Proc. SIGCHI Conf. Hum. Factors Comput. Syst., pp. 2395--2398, 2010.

Digital Library

[40]

"Optimizely." {Online}, Available: https://www.optimizely.com/.

[41]

"Mixpanel." {Online}, Available: https://mixpanel.com/.

[42]

"Oracle Maxymiser." {Online}. Available: https://www.oracle.com/marketingcloud/products/testing-and-optimization/index.html.

[43]

W. Wood, M. G. Witt, and L. Tam, "Changing circumstances, disrupting habits," J. Pers. Soc. Psychol., vol. 88, no. 6, pp. 918--33, 2005.

Cited By

Kedziora DSiemon DElshan ESońta M(2024)Towards stability, predictability, and quality of intelligent automation services: ECIT product journey from on-premise to as-a-serviceProceedings of the 7th ACM/IEEE International Workshop on Software-intensive Business10.1145/3643690.3648595(15-23)Online publication date: 16-Apr-2024
https://dl.acm.org/doi/10.1145/3643690.3648595
Holmström Olsson HBosch J(2024)How To Get Good At Data: 5 StepsProceedings of the 7th ACM/IEEE International Workshop on Software-intensive Business10.1145/3643690.3648242(32-39)Online publication date: 16-Apr-2024
https://dl.acm.org/doi/10.1145/3643690.3648242
Quin FWeyns DGalster MSilva C(2024)A/B testingJournal of Systems and Software10.1016/j.jss.2024.112011211:COnline publication date: 2-Jul-2024
https://dl.acm.org/doi/10.1016/j.jss.2024.112011
Show More Cited By

Recommendations

Software Entropy in Agile Product Evolution
HICSS '10: Proceedings of the 2010 43rd Hawaii International Conference on System Sciences

As agile software development principles and methods are being adopted by large software product organizations it is important to understand the role of software entropy. That is, how the maintainability of a system may degrade over time due to ...
Software evolution in agile development: a case study
OOPSLA '10: Proceedings of the ACM international conference companion on Object oriented programming systems languages and applications companion

The agile development method (ADM) is characterized by continuous feedback and change, and a software system developed using ADMevolves continuously through short iterations. Empirical studies on evolution of software following agile development method ...
The impact of agile principles on market-driven software product development

Agile development methods such as extreme programming (XP), SCRUM, Lean Software Development (Lean SD) and others have gained much popularity during the last years. Agile methodologies promise faster time-to-market, satisfied customers and high quality ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

ICSE '17: Proceedings of the 39th International Conference on Software Engineering

May 2017

816 pages

ISBN:9781538638682

General Chair:
Sebastian Uchitel
University of Buenos Aires, Argentina and Imperial College London, UK
,
Program Chairs:
Alessandro Orso
Georgia Institute of Technology
,
Martin Robillard
McGill University, Canada

Sponsors

SIGSOFT: ACM Special Interest Group on Software Engineering
IEEE-CS: Computer Society
SADIO: SADIO

Publisher

IEEE Press

Publication History

Published: 20 May 2017

Check for updates

Qualifiers

Research-article

Conference

ICSE '17

Sponsor:

SIGSOFT
IEEE-CS
SADIO

ICSE '17: 39th International Conference on Software Engineering

May 20 - 28, 2017

Buenos Aires, Argentina

Acceptance Rates

Overall Acceptance Rate 276 of 1,856 submissions, 15%

Upcoming Conference

ICSE 2025

2025 IEEE/ACM 46th International Conference on Software Engineering

April 26 - May 3, 2025

Ottawa , ON , Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

21
Total Citations
View Citations
1,444
Total Downloads

Downloads (Last 12 months)10
Downloads (Last 6 weeks)0

Reflects downloads up to 07 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Kedziora DSiemon DElshan ESońta M(2024)Towards stability, predictability, and quality of intelligent automation services: ECIT product journey from on-premise to as-a-serviceProceedings of the 7th ACM/IEEE International Workshop on Software-intensive Business10.1145/3643690.3648595(15-23)Online publication date: 16-Apr-2024
https://dl.acm.org/doi/10.1145/3643690.3648595
Holmström Olsson HBosch J(2024)How To Get Good At Data: 5 StepsProceedings of the 7th ACM/IEEE International Workshop on Software-intensive Business10.1145/3643690.3648242(32-39)Online publication date: 16-Apr-2024
https://dl.acm.org/doi/10.1145/3643690.3648242
Quin FWeyns DGalster MSilva C(2024)A/B testingJournal of Systems and Software10.1016/j.jss.2024.112011211:COnline publication date: 2-Jul-2024
https://dl.acm.org/doi/10.1016/j.jss.2024.112011
Bernardes MMarczak S(2023)On the Understanding of the Role of Continuous Experimentation in Technology-Based StartupProceedings of the XXXVII Brazilian Symposium on Software Engineering10.1145/3613372.3613414(21-30)Online publication date: 25-Sep-2023
https://dl.acm.org/doi/10.1145/3613372.3613414
Fabijan ADmitriev PArai BDrake AKohlmeier SKwong ALiu AMuccini H(2023)A/B Integrations: 7 Lessons Learned from Enabling A/B Testing as a Product FeatureProceedings of the 45th International Conference on Software Engineering: Software Engineering in Practice10.1109/ICSE-SEIP58684.2023.00033(304-314)Online publication date: 17-May-2023
https://dl.acm.org/doi/10.1109/ICSE-SEIP58684.2023.00033
Gottschalk SBhat RWeidmann NKirchhoff JEngels GKühn TSousa V(2022)Low-code experimentation on software productsProceedings of the 25th International Conference on Model Driven Engineering Languages and Systems: Companion Proceedings10.1145/3550356.3561572(798-807)Online publication date: 23-Oct-2022
https://dl.acm.org/doi/10.1145/3550356.3561572
de Boer Rvan Haalem Svan der Horst DWesseling T(2022)When experimentation starts as a solution to raise ROIProceedings of the 26th International Conference on Evaluation and Assessment in Software Engineering10.1145/3530019.3534983(270-271)Online publication date: 13-Jun-2022
https://dl.acm.org/doi/10.1145/3530019.3534983
Issa Mattos DLiu Y(2022)On the Use of Causal Graphical Models for Designing Experiments in the Automotive DomainProceedings of the 26th International Conference on Evaluation and Assessment in Software Engineering10.1145/3530019.3534979(264-265)Online publication date: 13-Jun-2022
https://dl.acm.org/doi/10.1145/3530019.3534979
Reklaitė AGamper J(2022)Offline assessment of interference effects in a series of AB testsProceedings of the 26th International Conference on Evaluation and Assessment in Software Engineering10.1145/3530019.3534978(262-263)Online publication date: 13-Jun-2022
https://dl.acm.org/doi/10.1145/3530019.3534978
Anderson KVisser DMannen JJiang Yvan Deursen AHarman MMiller H(2022)Challenges in applying continuous experimentationProceedings of the 44th International Conference on Software Engineering: Software Engineering in Practice10.1145/3510457.3513052(107-114)Online publication date: 21-May-2022
https://dl.acm.org/doi/10.1145/3510457.3513052
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents