The Impact of Code Ownership of DevOps Artefacts on the Outcome of DevOps CI Builds
Authors: 
Ajiromola Kola-Olawuyi, Nimmi Rashinika Weeraddana, 
Meiyappan NagappanAuthors Info & Claims
Pages 543 - 555
Abstract
DevOps is a key element in sustaining the quality and efficiency of software development. Yet, the effectiveness of DevOps methodologies extends beyond just technological expertise. It is greatly affected by the manner in which teams handle and engage with DevOps artefacts. Grasping the intricacies of code ownership and contribution patterns within DevOps artefacts is vital for refining strategies and ensuring they deliver their full potential.
There are two main strategies to manage DevOps artefacts as suggested in prior work: (1) all project developers need to contribute to DevOps artefacts, and (2) a dedicated group of developers needs to be authoring DevOps artefacts. To analyze which strategy works best for Open-Source Software (OSS) projects, we conduct an empirical analysis on a dataset of 892,193 CircleCI builds spanning 1,689 OSS projects. We employ a two-pronged approach to our study. First, we investigate the impact of chronological code ownership of DevOps artefacts on the outcome of a CI build on a build level. Second, we study the impact of the Skewness of DevOps contributions on the success rate of CI builds at the project level.
Our findings reveal that, in general, larger chronological ownership and higher Skewness values of DevOps contributions are related to more successful build outcomes and higher rates of successful build outcomes, respectively. We further find that projects with low Skewness values could have high build success rates when the number of developers in the project is relatively small. Thus, our results suggest that while larger software organizations are better off having dedicated DevOps developers, smaller organizations would benefit from having all developers involved in DevOps.
References
[1]
Rabe Abdalkareem, Suhaib Mujahid, and Emad Shihab. 2021. A Machine Learning Approach to Improve the Detection of CI Skip Commits. IEEE Transactions on Software Engineering 47, 12 (12 2021), 2740--2754.
[2]
Rabe Abdalkareem, Suhaib Mujahid, Emad Shihab, and Juergen Rilling. 2021. Which Commits Can Be CI Skipped? IEEE Transactions on Software Engineering 47, 3 (3 2021), 448--463.
[3]
Roozbeh Aghili, Heng Li, and Foutse Khomh. 2023. Studying the characteristics of AIOps projects on GitHub. Empirical Software Engineering 28, 6 (11 2023), 143.
[4]
H. Akaike. 1974. A new look at the statistical model identification. IEEE Trans. Automat. Control 19, 6 (12 1974), 716--723.
[5]
Olusegun Akinwande, H G Dikko, and Samson Agboola. 2015. Variance Inflation Factor: As a Condition for the Inclusion of Suppressor Variable(s) in Regression Analysis. Open Journal of Statistics 05 (1 2015), 754--767.
[6]
Eman Abdullah AlOmar, Moataz Chouchen, Mohamed Wiem Mkaouer, and Ali Ouni. 2022. Code Review Practices for Refactoring Changes: An Empirical Study on OpenStack. (3 2022).
[7]
Christian Bird, Nachiappan Nagappan, Brendan Murphy, Harald Gall, and Premkumar Devanbu. 2011. Don't touch my code!. In Proceedings of the 19th ACM SIGSOFT symposium and the 13th European conference on Foundations of software engineering. ACM, New York, NY, USA, 4--14.
[8]
Benjamin M Bolker, Mollie E Brooks, Connie J Clark, Shane W Geange, John R Poulsen, M Henry H Stevens, and Jada-Simone S White. 2009. Generalized linear mixed models: a practical guide for ecology and evolution. Trends in Ecology & Evolution 24, 3 (2009), 127--135.
[9]
Bihuan Chen, Linlin Chen, Chen Zhang, and Xin Peng. 2020. Buildfast: History-aware build outcome prediction for fast feedback and reduced cost in continuous integration. In Proceedings of the 35th International Conference on Automated Software Engineering.
[10]
Jacob Cohen. 1968. Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit. Psychological Bulletin 70, 4 (1968), 213--220.
[11]
Breno B. Nicolau de França, Helvio Jeronimo, and Guilherme Horta Travassos. 2016. Characterizing DevOps by Hearing Multiple Voices. In Proceedings of the XXX Brazilian Symposium on Software Engineering. ACM, New York, NY, USA, 53--62.
[12]
Elisa Diel, Sabrina Marczak, and Daniela S. Cruzes. 2016. Communication Challenges and Strategies in Distributed DevOps. In 2016 IEEE 11th International Conference on Global Software Engineering (ICGSE). IEEE, 24--28.
[13]
Paul Donner. 2016. Enhanced self-citation detection by fuzzy author name matching and complementary error estimates. Journal of the American Society for Information Science and Technology 67, 3 (2016), 662.
[14]
Thomas Durieux, Claire Le Goues, Michael Hilton, and Rui Abreu. 2020. Empirical study of restarted and flaky builds on Travis CI. In Proceedings of the 17th International Conference on Mining Software Repositories.
[15]
Paul M Duvall, Steve Matyas, and Andrew Glover. 2007. Continuous integration: improving software quality and reducing risk.
[16]
Floris Erich. 2019. DevOps is Simply Interaction Between Development and Operations. 89--99.
[17]
Csaba Faragó, Péter Hegedűs, and Rudolf Ferenc. 2015. Code ownership: Impact on maintainability. In Computational Science and Its Applications-ICCSA 2015: 15th International Conference, Banff, AB, Canada, June 22-25, 2015, Proceedings, Part V 15. Springer, 3--19.
[18]
Matthieu Foucault, Jean-Rémy Falleri, and Xavier Blanc. 2014. Code ownership in open-source software. In Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering. ACM, New York, NY, USA, 1--9.
[19]
Tanner Fry, Tapajit Dey, Andrey Karnauch, and Audris Mockus. 2020. A Dataset and an Approach for Identity Resolution of 38 Million Author IDs extracted from 2B Git Commits. In Proceedings of the 17th International Conference on Mining Software Repositories. ACM, New York, NY, USA, 518--522.
[20]
Keheliya Gallaba, John Ewart, Yves Junqueira, and Shane Mcintosh. 2020. Accelerating continuous integration by caching environments and inferring dependencies. Transactions on Software Engineering (2020).
[21]
Keheliya Gallaba, John Ewart, Yves Junqueira, and Shane McIntosh. 2022. Accelerating Continuous Integration by Caching Environments and Inferring Dependencies. IEEE Transactions on Software Engineering 48, 6 (6 2022), 2040--2052.
[22]
Keheliya Gallaba, Maxime Lamothe, and Shane McIntosh. 2022. Lessons from eight years of operational data from a continuous integration service. In Proceedings of the 44th International Conference on Software Engineering. ACM, New York, NY, USA, 1330--1342.
[23]
Keheliya Gallaba, Christian Macho, Martin Pinzger, and Shane McIntosh. 2018. Noise and heterogeneity in historical build data: an empirical study of travis ci. In Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering. 87--97.
[24]
Taher Ahmed Ghaleb, Daniel Alencar Da Costa, and Ying Zou. 2019. An empirical study of the long duration of continuous integration builds. Empirical Software Engineering 24 (2019).
[25]
Taher A Ghaleb, Safwat Hassan, and Ying Zou. 2022. Studying the Interplay between the Durations and Breakages of Continuous Integration Builds. Transactions on Software Engineering 49 (2022).
[26]
Rubén Grande, Aurora Vizcaíno, and Félix O. García. 2024. Is it worth adopting DevOps practices in Global Software Engineering? Possible challenges and benefits. Computer Standards & Interfaces 87 (1 2024), 103767.
[27]
Michaela Greiler, Kim Herzig, and Jacek Czerwonka. 2015. Code Ownership and Software Quality: A Replication Study. In 2015 IEEE/ACM 12th Working Conference on Mining Software Repositories. IEEE, 2--12.
[28]
Foyzul Hassan and Xiaoyin Wang. 2017. Change-Aware Build Prediction Model for Stall Avoidance in Continuous Integration. In 2017 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM). IEEE, 157--162.
[29]
Jordan Henkel, Christian Bird, Shuvendu K Lahiri, and Thomas Reps. 2020. Learning from, Understanding, and Supporting DevOps Artifacts for Docker. In Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering (ICSE '20). Association for Computing Machinery, New York, NY, USA, 38--49.
[30]
Michael Hilton, Timothy Tunnell, Kai Huang, Darko Marinov, and Danny Dig. 2016. Usage, costs, and benefits of continuous integration in open-source projects. In Proceedings of the 31st international conference on automated software engineering.
[31]
Dongyang Hu, Tao Wang, Junsheng Chang, Gang Yin, and Yang Zhang. 2018. Multi-Discussing across Issues in GitHub: A Preliminary Study. In 2018 25th Asia-Pacific Software Engineering Conference (APSEC). 406--415.
[32]
Ramtin Jabbari, Nauman bin Ali, Kai Petersen, and Binish Tanveer. 2016. What is DevOps?. In Proceedings of the Scientific Workshop Proceedings of XP2016. ACM, New York, NY, USA, 1--11.
[33]
Ramtin Jabbari, Nauman bin Ali, Kai Petersen, and Binish Tanveer. 2016. What is DevOps? A Systematic Mapping Study on Definitions and Practices. In Proceedings of the Scientific Workshop Proceedings of XP2016 (XP '16 Workshops). Association for Computing Machinery, New York, NY, USA.
[34]
Xianhao Jin and Francisco Servant. 2021. What Helped, and what did not? An Evaluation of the Strategies to Improve Continuous Integration. In 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE). IEEE, 213--225.
[35]
Eirini Kalliamvakou, Georgios Gousios, Kelly Blincoe, Leif Singer, Daniel M. German, and Daniela Damian. 2014. The promises and perils of mining GitHub. In Proceedings of the 11th Working Conference on Mining Software Repositories. ACM, New York, NY, USA, 92--101.
[36]
Noureddine Kerzazi and Bram Adams. 2016. Who needs release and devops engineers, and why?. In Proceedings of the international workshop on continuous software evolution and delivery. 77--83.
[37]
Noureddine Kerzazi, Foutse Khomh, and Bram Adams. 2014. Why do automated builds break? an empirical study. In 2014 IEEE International Conference on Software Maintenance and Evolution. IEEE, 41--50.
[38]
Yang Luo, Yangyang Zhao, Wanwangying Ma, and Lin Chen. 2017. What are the Factors Impacting Build Breakage?. In 2017 14th Web Information Systems and Applications Conference (WISA). IEEE, 139--142.
[39]
Rungroj Maipradit, Dong Wang, Patanamon Thongtanunam, Raula Gaikovina Kula, Yasutaka Kamei, and Shane McIntosh. 2023. Repeated Builds During Code Review: An Empirical Study of the OpenStack Community. In Proc. of the International Conference on Automated Software Engineering.
[40]
Ade Miller. 2008. A hundred days of continuous integration. In Agile 2008 conference. IEEE, 289--293.
[41]
Sikender Mohsienuddin Mohammad. 2017. DevOps automation and Agile methodology. International Journal of Creative Research Thoughts (IJCRT), ISSN (2017), 2320--2882.
[42]
Nuthan Munaiah, Steven Kroh, Craig Cabrey, and Meiyappan Nagappan. 2017. Curating GitHub for engineered software projects. Empirical Software Engineering 22, 6 (12 2017), 3219--3253.
[43]
Reza Nadri, Gema Rodriguez-Perez, and Meiyappan Nagappan. 2022. On the Relationship Between the Developer's Perceptible Race and Ethnicity and the Evaluation of Contributions in OSS. IEEE Transactions on Software Engineering 48, 8 (8 2022), 2955--2968.
[44]
Ansong Ni and Ming Li. 2017. Cost-Effective Build Outcome Prediction Using Cascaded Classifiers. In 2017 IEEE/ACM 14th International Conference on Mining Software Repositories (MSR). IEEE, 455--458.
[45]
Zi Peng, Tse-Hsun Chen, and Jinqiu Yang. 2020. Revisiting test impact analysis in continuous testing from the perspective of code dependencies. IEEE Transactions on Software Engineering 48, 6 (2020), 1979--1993.
[46]
Jackson A Prado Lima, Willian DF Mendonça, Silvia R Vergilio, and Wesley KG Assunção. 2022. Cost-effective learning-based strategies for test case prioritization in continuous integration of highly-configurable software. Empirical Software Engineering 27, 6 (2022), 133.
[47]
Foyzur Rahman and Premkumar Devanbu. 2011. Ownership, experience and defects: a fine-grained study of authorship. In Proceedings of the 33rd International Conference on Software Engineering. 491--500.
[48]
Ayushi Rastogi, Nachiappan Nagappan, Georgios Gousios, and André van der Hoek. 2018. Relationship between geographical location and evaluation of developer contributions in github. In Proceedings of the 12th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement. ACM, New York, NY, USA, 1--8.
[49]
Leah Riungu-Kalliosaari, Simo Mäkinen, Lucy Ellen Lwakatare, Juha Tiihonen, and Tomi Männistö. 2016. DevOps adoption benefits and challenges in practice: A case study. In Product-Focused Software Process Improvement: 17th International Conference, PROFES 2016, Trondheim, Norway, November 22-24, 2016, Proceedings 17. Springer, 590--597.
[50]
Islem Saidani, Ali Ouni, Moataz Chouchen, and Mohamed Wiem Mkaouer. 2020. Predicting continuous integration build failures using evolutionary search. Information and Software Technology 128 (2020), 106392.
[51]
Islem Saidani, Ali Ouni, and Mohamed Wiem Mkaouer. 2022. Improving the prediction of continuous integration build failures using deep learning. Automated Software Engineering 29 (2022).
[52]
Mark Santolucito, Jialu Zhang, Ennan Zhai, Jurgen Cito, and Ruzica Piskac. 2022. Learning CI Configuration Correctness for Early Build Feedback. In 2022 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER). IEEE, 1006--1017.
[53]
Steven Sawyer. 2009. Analysis of Variance: The Fundamental Concepts. Journal of Manual & Manipulative Therapy 17 (4 2009), 27E--38E.
[54]
Gideon Schwarz. 1978. Estimating the Dimension of a Model. The Annals of Statistics 6, 2 (1978), 461--464. http://www.jstor.org/stable/2958889
[55]
Tushar Sharma, Marios Fragkoulis, and Diomidis Spinellis. 2016. Does your configuration code smell?. In Proceedings - 13th Working Conference on Mining Software Repositories, MSR 2016. Association for Computing Machinery, Inc, 189--200.
[56]
August Shi, Wing Lam, Reed Oei, Tao Xie, and Darko Marinov. 2019. iFixFlakies: A framework for automatically fixing order-dependent flaky tests. In Proceedings of the 27th Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering.
[57]
Jacek Śliwerski, Thomas Zimmermann, and Andreas Zeller. 2005. When do changes induce fixes? ACM SIGSOFT Software Engineering Notes 30, 4 (7 2005), 1--5.
[58]
Jens Smeds, Kristian Nybom, and Ivan Porres. 2015. DevOps: A Definition and Perceived Adoption Impediments. 166--177.
[59]
Eliezio Soares, Daniel Alencar da Costa, and Uirá Kulesza. 2023. Continuous Integration and Software Quality: A Causal Explanatory Study. (9 2023).
[60]
Eliezio Soares, Gustavo Sizilio, Jadson Santos, Daniel Alencar da Costa, and Uirá Kulesza. 2022. The effects of continuous integration on software development: a systematic literature review. Empirical Software Engineering 27, 3 (2022), 78.
[61]
Stephen Orban. 2015. Enterprise DevOps: Why You Should Run What You Build.
[62]
Ahmed Tamrawi, Hoan Anh Nguyen, Hung Viet Nguyen, and Tien N Nguyen. 2012. Build code analysis with symbolic evaluation. In 2012 34th International Conference on Software Engineering (ICSE). IEEE, 650--660.
[63]
Patanamon Thongtanunam, Shane McIntosh, Ahmed E Hassan, and Hajimu Iida. 2016. Revisiting Code Ownership and Its Relationship with Software Quality in the Scope of Modern Code Review. In Proceedings of the 38th International Conference on Software Engineering (ICSE '16). Association for Computing Machinery, New York, NY, USA, 1039--1050.
[64]
Bogdan Vasilescu, Yue Yu, Huaimin Wang, Premkumar Devanbu, and Vladimir Filkov. 2015. Quality and productivity outcomes relating to continuous integration in GitHub. In Proceedings of the 2015 10th joint meeting on foundations of software engineering. 805--816.
[65]
Carmine Vassallo, Gerald Schermann, Fiorella Zampetti, Daniele Romano, Philipp Leitner, Andy Zaidman, Massimiliano Di Penta, and Sebastiano Panichella. 2017. A tale of CI build failures: An open source and a financial organization perspective. In 2017 IEEE international conference on software maintenance and evolution (ICSME). IEEE, 183--193.
[66]
Carmine Vassallo, Fiorella Zampetti, Daniele Romano, Moritz Beller, Annibale Panichella, Massimiliano Di Penta, and Andy Zaidman. 2016. Continuous delivery practices in a large financial organization. In 2016 IEEE International Conference on Software Maintenance and Evolution (ICSME). IEEE, 519--528.
[67]
Mario Villamizar, Oscar Garcés, Lina Ochoa, Harold Castro, Lorena Salamanca, Mauricio Verano, Rubby Casallas, Santiago Gil, Carlos Valencia, Angee Zambrano, et al. 2017. Cost comparison of running web applications in the cloud using monolithic, microservice, and AWS Lambda architectures. Service Oriented Computing and Applications 11 (2017), 233--247.
[68]
Yuqing Wang, Maaret Pyhäjärvi, and Mika V Mäntylä. 2020. Test automation process improvement in a DevOps team: experience report. In 2020 ieee international conference on software testing, verification and validation workshops (icstw). IEEE, 314--321.
[69]
Anna Wiedemann, Manuel Wiesche, Heiko Gewald, and Helmut Krcmar. 2023. Integrating development and operations teams: A control approach for DevOps. Information and Organization 33, 3 (9 2023), 100474.
[70]
Pavlina Wurzelová, Fabio Palomba, and Alberto Bacchelli. 2019. Characterizing women (not) contributing to open-source. In 2019 IEEE/ACM 2nd International Workshop on Gender Equality in Software Engineering (GE). IEEE, 5--8.
[71]
Jing Xia and Yanhui Li. 2017. Could We Predict the Result of a Continuous Integration Build? An Empirical Study. In 2017 IEEE International Conference on Software Quality, Reliability and Security Companion (QRS-C). IEEE, 311--315.
[72]
Jing Xia, Yanhui Li, and Chuanqi Wang. 2017. An Empirical Study on the Cross-Project Predictability of Continuous Integration Outcomes. In 2017 14th Web Information Systems and Applications Conference (WISA). IEEE, 234--239.
[73]
Jiaxin Zhu, Minghui Zhou, and Audris Mockus. 2014. Patterns of Folder Use and Project Popularity: A Case Study of Github Repositories. In Proceedings of the 8th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM '14). Association for Computing Machinery, New York, NY, USA.
Recommendations
A qualitative study of DevOps usage in practice
Organizations are introducing agile and lean software development techniques in operations to increase the pace of their software development process and to improve the quality of their software. They use the term DevOps, a portmanteau of development ...
CMMI guided process improvement for DevOps projects: an exploratory case study
ICSSP '16: Proceedings of the International Conference on Software and Systems ProcessVery recently, an increasing number of software companies adopted DevOps to adapt themselves to the ever-changing business environment. While it is important to mature adoption of the DevOps for these companies, no dedicated maturity models for DevOps ...
Assessing the Maturity of DevOps Practices in Software Industry: An Empirical Study of HELENA2 Dataset
EASE '22: Proceedings of the 26th International Conference on Evaluation and Assessment in Software EngineeringCurrently, the software development organizations are adopting DevOps practices in order to develop quality product. Due to the lack of definition of DevOps, the principles, practices, and methods adopted in DevOps to determine success have changed ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
April 2024
788 pages
ISBN:9798400705878
DOI:10.1145/3643991
- Chair:
- Diomidis Spinellis,
- Program Chair:
- Alberto Bacchelli,
- Program Co-chair:
- Eleni Constantinou
Copyright © 2024 Copyright is held by the owner/author(s). Publication rights licensed to ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 02 July 2024
Check for updates
Author Tags
Qualifiers
- Research-article
Conference
MSR '24
Sponsor:
MSR '24: 21st International Conference on Mining Software Repositories
April 15 - 16, 2024
Lisbon, Portugal
Upcoming Conference
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 35Total Downloads
- Downloads (Last 12 months)35
- Downloads (Last 6 weeks)8
Reflects downloads up to 09 Jan 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in