[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to main content

Chapter 12 The Evolution of Continuous Experimentation in Software Product Development: From Data to a Data-Driven Organization at Scale

  • Chapter
  • First Online:
Accelerating Digital Transformation

Abstract

Software development companies are increasingly aiming to become data-driven by trying to continuously experiment with the products used by their customers. Although familiar with the competitive edge that the A/B testing technology delivers, they seldom succeed in evolving and adopting the methodology. In this paper, and based on an exhaustive and collaborative case study research in a large software-intense company with highly developed experimentation culture, we present the evolution process of moving from ad-hoc customer data analysis towards continuous controlled experimentation at scale. Our main contribution is the “Experimentation Evolution Model” in which we detail three phases of evolution: technical, organizational and business evolution. With our contribution, we aim to provide guidance to practitioners on how to develop and scale continuous experimentation in software organizations with the purpose of becoming data-driven at scale.

Reprinted with permission from IEEE. Originally published in Proceedings of the 39th International Conference on Software Engineering, 2017, pp. 770–780. DOI: 10.1109/ICSE.2017.76

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Subscribe and save

Springer+ Basic
£29.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
GBP 19.95
Price includes VAT (United Kingdom)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
GBP 103.50
Price includes VAT (United Kingdom)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
GBP 129.99
Price includes VAT (United Kingdom)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
GBP 129.99
Price includes VAT (United Kingdom)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

Similar content being viewed by others

References

  1. Ablett, R., Sharlin, E., Maurer, F., Denzinger, J., Schock, C.: Buildbot: Robotic monitoring of agile software development teams. In: RO-MAN 2007-The 16th IEEE International Symposium on Robot and Human Interactive Communication, pp. 931–936. IEEE (2007)

    Google Scholar 

  2. Abrahamsson, P.: Is management commitment a necessity after all in software process improvement? In: Proceedings of the 26th Euromicro Conference. EUROMICRO 2000. Informatics: Inventing the Future, vol. 2, pp. 246–253. IEEE (2000)

    Google Scholar 

  3. Abrahamsson, P.: Measuring the success of software process improvement: the dimensions. arXiv preprint arXiv:1309.4645 (2013)

    Google Scholar 

  4. Abrahamsson, P., Warsta, J., Siponen, M., Ronkainen, J.: New directions on agile methods: a comparative analysis. In: Proceedings of the International Conference on Software Engineering, pp. 244–254 (2003). DOI 10.1109/ICSE.2003.1201204

    Google Scholar 

  5. Abran, A.: Software metrics and software metrology. John Wiley & Sons (2010)

    Google Scholar 

  6. Agarwal, A., Shankar, R., Tiwari, M.: Modeling the metrics of lean, agile and leagile supply chain: An anp-based approach. European Journal of Operational Research 173(1), 211–225 (2006)

    MathSciNet  MATH  Google Scholar 

  7. Agarwal, P.: Continuous scrum: agile management of saas products. In: Proceedings of the 4th India Software Engineering Conference, pp. 51–60 (2011)

    Google Scholar 

  8. Aghabozorgi, S., Shirkhorshidi, A.S., Wah, T.Y.: Time-series clustering–a decade review. Information Systems 53, 16–38 (2015)

    Google Scholar 

  9. Albuquerque, C., Antonino, P., Nakagawa, E.: An investigation into agile methods in embedded systems development. In: Computational Science and Its Applications, Lecture Notes in Computer Science, vol. 7335, pp. 576–591. Springer (2012). URL http://www.springerlink.com/content/38uk703767811277/abstract/

  10. Allamanis, M., Barr, E.T., Bird, C., Sutton, C.: Learning natural coding conventions. In: Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering, pp. 281–293. ACM (2014)

    Google Scholar 

  11. Alshayeb, M., Li, W.: An empirical study of system design instability metric and design evolution in an agile software process. Journal of Systems and Software 74(3), 269–274 (2005)

    Google Scholar 

  12. Alyahya, S., Ivins, W.K., Gray, W.: A holistic approach to developing a progress tracking system for distributed agile teams. In: 2012 IEEE/ACIS 11th International Conference on Computer and Information Science, pp. 503–512. IEEE (2012)

    Google Scholar 

  13. Ambler, S.: Agile modeling: effective practices for extreme programming and the unified process. John Wiley & Sons (2002)

    Google Scholar 

  14. Ambler, S.W., Lines, M.: Disciplined Agile Delivery, 1 edn. IBM Press (2012). URL http://disciplinedagiledelivery.wordpress.com/

  15. Amershi, S., Begel, A., Bird, C., DeLine, R., Gall, H., Kamar, E., Nagappan, N., Nushi, B., Zimmermann, T.: Software engineering for machine learning: A case study. In: 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), pp. 291–300. IEEE (2019)

    Google Scholar 

  16. Antinyan, V., Staron, M.: Rendex: A method for automated reviews of textual requirements. Journal of Systems and Software 131, 63–77 (2017)

    Google Scholar 

  17. Arazy, O., Kopak, R.: On the measurability of information quality. Journal of the American Society for Information Science and Technology 62(1), 89–99 (2011)

    Google Scholar 

  18. Arpteg, A., Brinne, B., Crnkovic-Friis, L., Bosch, J.: Software engineering challenges of deep learning. In: 2018 44th Euromicro Conference on Software Engineering and Advanced Applications (SEAA), pp. 50–59. IEEE (2018)

    Google Scholar 

  19. Auer, F., Felderer, M.: Current state of research on continuous experimentation: a systematic mapping study. In: 2018 44th Euromicro Conference on Software Engineering and Advanced Applications (SEAA), pp. 335–344. IEEE (2018)

    Google Scholar 

  20. Avgeriou, P., Kruchten, P., Ozkaya, I., Seaman, C.: Managing technical debt in software engineering (dagstuhl seminar 16162). In: Dagstuhl Reports, vol. 6. Schloss Dagstuhl-Leibniz-Zentrum fuer Informatik (2016)

    Google Scholar 

  21. Axelsson, S., Baca, D., Feldt, R., Sidlauskas, D., Kacan, D.: Detecting defects with an interactive code review tool based on visualisation and machine learning. In: the 21st International Conference on Software Engineering and Knowledge Engineering (SEKE 2009) (2009)

    Google Scholar 

  22. Bach, J.: Exploratory Testing. https://www.satisfice.com/exploratory-testing (2020). [Online; accessed July 18, 2020]

  23. Baldassarre, M.T., Caivano, D., Visaggio, G.: Comprehensibility and efficiency of multiview framework for measurement plan design. In: Empirical Software Engineering, 2003. ISESE 2003. Proceedings. 2003 International Symposium on, pp. 89–98. IEEE (2003)

    Google Scholar 

  24. Bardsiri, V.K., Jawawi, D.N.A., Hashim, S.Z.M., Khatibi, E.: Increasing the accuracy of software development effort estimation using projects clustering. IET Software 6(6), 461–473 (2012)

    Google Scholar 

  25. Barik, T., DeLine, R., Drucker, S., Fisher, D.: The bones of the system: A case study of logging and telemetry at microsoft. In: 2016 IEEE/ACM 38th International Conference on Software Engineering Companion (ICSE-C), pp. 92–101. IEEE (2016)

    Google Scholar 

  26. Baskerville, R., Wood-Harper, A.T.: A Critical Perspective on Action Research as a Method for Information Systems Research. Journal of Information Technology 11(2), 235–246 (1996)

    Google Scholar 

  27. Basri, S., Dominic, D.D., Murugan, T., Almomani, M.A.: A proposed framework using exploratory testing to improve software quality in sme’s. In: International Conference of Reliable Information and Communication Technology, pp. 1113–1122. Springer (2018)

    Google Scholar 

  28. Batsaikhan, O., Lin, Y.: Building a shared understanding of customer value in a large-scale agile organization: A case study. Master’s thesis, Chalmers—University of Gothenburg, Dept. of Computer Science and Engineering (2018)

    Google Scholar 

  29. Baumeister, J., Reutelshoefer, J.: Developing knowledge systems with continuous integration. In: Proceedings of the 11th International Conference on Knowledge Management and Knowledge Technologies, pp. 1–4 (2011)

    Google Scholar 

  30. Beaumont, O., Bonichon, N., Courtès, L., Dolstra, E., Hanin, X.: Mixed data-parallel scheduling for distributed continuous integration. In: 2012 IEEE 26th International Parallel and Distributed Processing Symposium Workshops & PhD Forum, pp. 91–98. IEEE (2012)

    Google Scholar 

  31. Beck, K.: Embracing change with extreme programming. Computer 32(10), 70–77 (1999)

    Google Scholar 

  32. Beck, K.: Extreme programming explained: embrace change. addison-wesley professional (2000)

    Google Scholar 

  33. Beck, K., Beedle, M., van Bennekum, A., Cockburn, A., Cunningham, W., Fowler, M., Grenning, J., Highsmith, J., Hunt, A., Jeffries, R., Kern, J., Marick, B., Martin, R.C., Mellor, S., Schwaber, K., Sutherland, J., Thomas, D.: Manifesto for the Agile Software Development (2001)

    Google Scholar 

  34. Berger, C., Eklund, U.: Expectations and challenges from scaling agile in mechatronics-driven companies–a comparative case study. In: International Conference on Agile Software Development, pp. 15–26. Springer (2015)

    Google Scholar 

  35. Bernardi, L., Mavridis, T., Estevez, P.: 150 successful machine learning models: 6 lessons learned at booking. com. In: Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 1743–1751 (2019)

    Google Scholar 

  36. Besker, T., Martini, A., Bosch, J.: A systematic literature review and a unified model of atd. In: 2016 42th Euromicro Conference on Software Engineering and Advanced Applications (SEAA), pp. 189–197. IEEE (2016)

    Google Scholar 

  37. Besker, T., Martini, A., Bosch, J.: The pricey bill of technical debt: When and by whom will it be paid? In: 2017 IEEE International Conference on Software Maintenance and Evolution (ICSME), pp. 13–23. IEEE (2017)

    Google Scholar 

  38. Bisio, R., Malabocchia, F.: Cost estimation of software projects through case base reasoning. In: International Conference on Case-Based Reasoning, pp. 11–22. Springer (1995)

    Google Scholar 

  39. Bjarnason, E., Unterkalmsteiner, M., Borg, M., Engström, E.: A multi-case study of agile requirements engineering and the use of test cases as requirements. Information and Software Technology 77, 61–79 (2016)

    Google Scholar 

  40. Bjarnason, E., Wnuk, K., Regnell, B.: A case study on benefits and side-effects of agile practices in large-scale requirements engineering. In: proceedings of the 1st workshop on agile requirements engineering, pp. 1–5 (2011)

    Google Scholar 

  41. Boehm, B.: Get ready for agile methods, with care. Computer 35(1), 64–69 (2002)

    Google Scholar 

  42. Boehm, B.W., et al.: Software engineering economics, vol. 197. Prentice-hall Englewood Cliffs (NJ) (1981)

    Google Scholar 

  43. Boetticher, G., Menzies, T., Ostrand, T.: Promise repository of empirical software engineering data. West Virginia University, Department of Computer Science (2007)

    Google Scholar 

  44. Booch, G.: Object oriented design with applications. Benjamin-Cummings Publishing Co., Inc. (1990)

    MATH  Google Scholar 

  45. Bosch, J.: Building products as innovation experiment systems. In: International Conference of Software Business, pp. 27–39. Springer (2012)

    Google Scholar 

  46. Bosch, J., Eklund, U.: Eternal embedded software: Towards innovation experiment systems. In: International Symposium On Leveraging Applications of Formal Methods, Verification and Validation, pp. 19–31. Springer (2012)

    Google Scholar 

  47. Bosch, J., Olsson, H.H., Crnkovic, I.: It takes three to tango: Requirement, outcome/data, and ai driven development. In: SiBW, pp. 177–192 (2018)

    Google Scholar 

  48. Bosch, J., Olsson, H.H., Crnkovic, I.: Engineering ai systems: A research agenda. In: Artificial Intelligence Paradigms for Smart Cyber-Physical Systems, pp. 1–19. IGI Global (2021)

    Google Scholar 

  49. Bosch-Sijtsema, P., Bosch, J.: User involvement throughout the innovation process in high-tech industries. Journal of Product Innovation Management 32(5), 793–807 (2015)

    Google Scholar 

  50. Bowyer, J., Hughes, J.: Assessing undergraduate experience of continuous integration and test-driven development. In: Proceedings of the 28th international conference on Software engineering, pp. 691–694 (2006)

    Google Scholar 

  51. Brar, H.K., Kaur, P.J.: Static analysis tools for security: A comparative evaluation. International Journal 5(7) (2015)

    Google Scholar 

  52. Braun, V., Clarke, V.: Using thematic analysis in psychology. Qualitative research in psychology 3(2), 77–101 (2006)

    Google Scholar 

  53. Briand, L., El Emam, K., Morasca, S.: Theoretical and empirical validation of software product measures. International Software Engineering Research Network, Technical Report ISERN-95-03 (1995)

    Google Scholar 

  54. Briand, L.C.: Novel applications of machine learning in software testing. In: 2008 The Eighth International Conference on Quality Software, pp. 3–10. IEEE (2008)

    Google Scholar 

  55. Briand, L.C., Morasca, S., Basili, V.R.: Property-based software engineering measurement. Software Engineering, IEEE Transactions on 22(1), 68–86 (1996)

    Google Scholar 

  56. Briand, L.C., Wüst, J., Daly, J.W., Victor Porter, D.: Exploring the relationships between design measures and software quality in object-oriented systems. Journal of systems and software 51(3), 245–273 (2000)

    Google Scholar 

  57. Brooks, G.: Team pace keeping build times down. In: Agile 2008 Conference, pp. 294–297. IEEE (2008)

    Google Scholar 

  58. Brown, N., Cai, Y., Guo, Y., Kazman, R., Kim, M., Kruchten, P., Lim, E., MacCormack, A., Nord, R., Ozkaya, I., et al.: Managing technical debt in software-reliant systems. In: Proceedings of the FSE/SDP workshop on Future of software engineering research, pp. 47–52 (2010)

    Google Scholar 

  59. Brun, Y., Ernst, M.D.: Finding latent code errors via machine learning over program executions. In: Proceedings of the 26th International Conference on Software Engineering, ICSE ’04, pp. 480–490. IEEE Computer Society, Washington, DC, USA (2004). URL http://dl.acm.org/citation.cfm?id=998675.999452

  60. Bruneliere, H., Burger, E., Cabot, J., Wimmer, M.: A feature-based survey of model view approaches. Software & Systems Modeling (2017). DOI 10.1007/s10270-017-0622-9

    Google Scholar 

  61. Buglione, L., Abran, A.: Introducing root-cause analysis and orthogonal defect classification at lower cmmi maturity levels. Proc. MENSURA p. 29 (2006)

    Google Scholar 

  62. Bures, M., Frajtak, K., Ahmed, B.S.: Tapir: Automation support of exploratory testing using model reconstruction of the system under test. IEEE Transactions on Reliability 67(2), 557–580 (2018)

    Google Scholar 

  63. Calpur, M.C., Arca, S., Calpur, T.C., Yilmaz, C.: Model dressing for automated exploratory testing. In: 2017 IEEE International Conference on Software Quality, Reliability and Security Companion (QRS-C), pp. 577–578. IEEE (2017)

    Google Scholar 

  64. Campbell-Pretty, E.: Tribal unity: Getting from teams to tribes by creating a one team culture (2016)

    Google Scholar 

  65. Cannizzo, F., Clutton, R., Ramesh, R.: Pushing the boundaries of testing and continuous integration. In: Agile 2008 Conference, pp. 501–505. IEEE (2008)

    Google Scholar 

  66. Castellion, G.: Do it wrong quickly: how the web changes the old marketing rules by mike moran (2008)

    Google Scholar 

  67. Catal, C., Diri, B.: A systematic review of software fault prediction studies. Expert systems with applications 36(4), 7346–7354 (2009)

    Google Scholar 

  68. Chappelly, T., Cifuentes, C., Krishnan, P., Gevay, S.: Machine learning for finding bugs: An initial report. In: Machine Learning Techniques for Software Quality Evaluation (MaLTeSQuE), IEEE Workshop on, pp. 21–26. IEEE (2017)

    Google Scholar 

  69. Chow, T., Cao, D.B.: A survey study of critical success factors in agile software projects. Journal of systems and software 81(6), 961–971 (2008)

    Google Scholar 

  70. Cicchetti, A., Ciccozzi, F., Pierantonio, A.: Multi-view approaches for software and system modelling: a systematic literature review. Software & Systems Modeling pp. 1–27 (2019). DOI 10.1007/s10270-018-00713-w

    Google Scholar 

  71. Clancy, T.: The standish group report. Chaos report (1995)

    Google Scholar 

  72. Cockburn, A.: Agile software development: the cooperative game. Pearson Education (2006)

    Google Scholar 

  73. Codabux, Z., Williams, B.: Managing technical debt: An industrial case study. In: 2013 4th International Workshop on Managing Technical Debt (MTD), pp. 8–15. IEEE (2013)

    Google Scholar 

  74. Cohan, S.: Successful customer collaboration resulting in the right product for the end user. In: Agile 2008 Conference, pp. 284–288. IEEE (2008)

    Google Scholar 

  75. Cook, T.D., Campbell, D.T., Day, A.: Quasi-experimentation: Design & analysis issues for field settings, vol. 351. Houghton Mifflin Boston (1979)

    Google Scholar 

  76. Cossio, M., et al.: A/b testing-the most powerful way to turn clicks into customers, vol (2012)

    Google Scholar 

  77. Mascarenhas Hornos da Costa, J., Oehmen, J., Rebentisch, E., Nightingale, D.: Toward a better comprehension of lean metrics for research and product development management. R&D Management (2014)

    Google Scholar 

  78. Crook, T., Frasca, B., Kohavi, R., Longbotham, R.: Seven pitfalls to avoid when running controlled experiments on the web. In: Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 1105–1114. ACM (2009)

    Google Scholar 

  79. Cunningham, W.: The wycash portfolio management system. ACM SIGPLAN OOPS Messenger 4(2), 29–30 (1992)

    Google Scholar 

  80. Cusomano, M., Selby, R.: Microsoft secrets—how the world’s most powerful software company creates technology, shapes markets, and manages people (1995)

    Google Scholar 

  81. Dagan, I., Engelson, S.P.: Committee-based sampling for training probabilistic classifiers. In: Machine Learning Proceedings 1995, pp. 150–157. Elsevier (1995)

    Google Scholar 

  82. Dahlmeier, D.: On the challenges of translating nlp research into commercial products. In: Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 92–96 (2017)

    Google Scholar 

  83. Dajsuren, Y., Gerpheide, C., Serebrenik, A., Wijs, A., Vasilescu, B., van den Brand, M.: Formalizing Correspondence Rules for Automotive Architecture Views. In: Proceedings of the 10th international ACM Sigsoft conference on Quality of software architectures, pp. 129–138. ACM (2014). DOI 10.1145/2602576.2602588

    Google Scholar 

  84. Daskalantonakis, M.K., Yacobellis, R.H., Basili, V.R.: A method for assessing software measurement technology. Quality Engineering 3(1), 27–40 (1990)

    Google Scholar 

  85. Davis, A.M.: Just Enough Requirements Management: Where Software Development Meets Marketing. Dorset House Publishing (2005)

    Google Scholar 

  86. Desharnais, J.M., Abran, A.: How to succesfully implement a measurement program: From theory to practice. In: Metrics in Software Evolution, pp. 11–38. Oldenbourg Verlag, Oldenburg (1995)

    Google Scholar 

  87. Dess, G.G., Shaw, J.D.: Voluntary turnover, social capital, and organizational performance. Academy of Management Review 26(3), 446–456 (2001)

    Google Scholar 

  88. D’haeseleer, P.: What are dna sequence motifs? Nature biotechnology 24(4), 423 (2006)

    Google Scholar 

  89. Di Nucci, D., Palomba, F., Tamburri, D.A., Serebrenik, A., De Lucia, A.: Detecting code smells using machine learning techniques: are we there yet? In: 2018 IEEE 25th International Conference on Software Analysis, Evolution and Reengineering (SANER), pp. 612–621. IEEE (2018)

    Google Scholar 

  90. Diaz-Ley, M., Garcia, F., Piattini, M.: Implementing a software measurement program in small and medium enterprises: a suitable framework. IET Software 2(5), 417–436 (2008)

    Google Scholar 

  91. Dikert, K., Paasivaara, M., Lassenius, C.: Challenges and success factors for large-scale agile transformations: A systematic literature review. Journal of Systems and Software 119, 87–108 (2016)

    Google Scholar 

  92. Dingsyr, T., Nerur, S., Balijepally, V., Moe, N.B.: A decade of agile methodologies: Towards explaining agile software development. Journal of Systems and Software 85(6), 1213–1221 (2012). DOI 10.1016/j.jss.2012.02.033. URL http://www.sciencedirect.com/science/article/pii/S0164121212000532

    Article  Google Scholar 

  93. Dösinger, S., Mordinyi, R., Biffl, S.: Communicating continuous integration servers for increasing effectiveness of automated testing. In: 2012 Proceedings of the 27th IEEE/ACM International Conference on Automated Software Engineering, pp. 374–377. IEEE (2012)

    Google Scholar 

  94. Downs, J., Hosking, J., Plimmer, B.: Status communication in agile software teams: A case study. In: 2010 Fifth International Conference on Software Engineering Advances, pp. 82–87. IEEE (2010)

    Google Scholar 

  95. Downs, J., Plimmer, B., Hosking, J.G.: Ambient awareness of build status in collocated software teams. In: 2012 34th International Conference on Software Engineering (ICSE), pp. 507–517. IEEE (2012)

    Google Scholar 

  96. Dubinsky, Y., Talby, D., Hazzan, O., Keren, A.: Agile metrics at the israeli air force. In: Agile Conference, 2005. Proceedings, pp. 12–19. IEEE (2005)

    Google Scholar 

  97. Durelli, V.H., Durelli, R.S., Borges, S.S., Endo, A.T., Eler, M.M., Dias, D.R., Guimarães, M.P.: Machine learning applied to software testing: A systematic mapping study. IEEE Transactions on Reliability 68(3), 1189–1212 (2019)

    Google Scholar 

  98. Durisic, D., Staron, M., Tichy, M., Hansson, J.: Assessing the impact of meta-model evolution: a measure and its automotive application. Software & Systems Modeling 18(2), 1419–1445 (2019)

    Google Scholar 

  99. Duvall, P.M., Matyas, S., Glover, A.: Continuous integration: improving software quality and reducing risk. Pearson Education (2007)

    Google Scholar 

  100. Dybå, T., Dingsøyr, T.: Empirical studies of agile software development: A systematic review. Information and Software Technology 50(9-10), 833–859 (2008). DOI 10.1016/j.infsof.2008.01.006. URL http://www.sciencedirect.com/science/article/pii/S0950584908000256

    Article  Google Scholar 

  101. Dyer, R., Nguyen, H.A., Rajan, H., Nguyen, T.N.: Boa: A language and infrastructure for analyzing ultra-large-scale software repositories. In: Proceedings of the 2013 International Conference on Software Engineering, pp. 422–431. IEEE Press (2013)

    Google Scholar 

  102. Dzamashvili Fogelström, N., Gorschek, T., Svahnberg, M., Olsson, P.: The impact of agile principles on market-driven software product development. Journal of software maintenance and evolution: Research and practice 22(1), 53–80 (2010)

    Google Scholar 

  103. Ebert, C., Paasivaara, M.: Scaling agile. Ieee Software 34(6), 98–103 (2017)

    Google Scholar 

  104. Egyed, A.: Automatically Detecting and Tracking Inconsistencies in Software Design Models. IEEE Transactions on Software Engineering 37(2), 188–204 (2010). DOI 10.1109/tse.2010.38

    Google Scholar 

  105. Ehrig, H., Ehrig, K., Hermann, F.: From Model Transformation to Model Integration based on the Algebraic Approach to Triple Graph Grammars. Electronic Communications of the EASST 10 (2008)

    Google Scholar 

  106. Eiffel protocol. https://github.com/eiffel-community/eiffel

  107. Eisenhardt, K.M.: Building theories from case study research. Academy of management review 14(4), 532–550 (1989)

    Google Scholar 

  108. Eklund, U., Olsson, H.H., Strøm, N.J.: Industrial challenges of scaling agile in mass-produced embedded systems. In: International Conference on Agile Software Development, pp. 30–42. Springer (2014)

    Google Scholar 

  109. Emanuelsson, P., Nilsson, U.: A comparative study of industrial static analysis tools. Electronic notes in theoretical computer science 217, 5–21 (2008)

    Google Scholar 

  110. Ernst, N.A., Bellomo, S., Ozkaya, I., Nord, R.L., Gorton, I.: Measure it? manage it? ignore it? software practitioners and technical debt. In: Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering, pp. 50–60 (2015)

    Google Scholar 

  111. Esling, P., Agon, C.: Time-series data mining. ACM Computing Surveys (CSUR) 45(1), 12 (2012)

    Google Scholar 

  112. ETSI: 3GPP Technical Specification Release 14 - ETSI TS 136 300. Tech. Rep. Release 14, ETSI, Valbonne, France (2017)

    Google Scholar 

  113. Evbota, F., Knauss, E., Sandberg, A.: Scaling up the planning game: Collaboration challenges in large-scale agile product development. In: International Conference on Agile Software Development, pp. 28–38. Springer, Cham (2016)

    Google Scholar 

  114. Fabijan, A., Dmitriev, P., McFarland, C., Vermeer, L., Holmström Olsson, H., Bosch, J.: Experimentation growth: Evolving trustworthy a/b testing capabilities in online software companies. Journal of Software: Evolution and Process 30(12), e2113 (2018)

    Google Scholar 

  115. Fabijan, A., Dmitriev, P., Olsson, H.H., Bosch, J.: The evolution of continuous experimentation in software product development: from data to a data-driven organization at scale. In: 2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE), pp. 770–780. IEEE (2017)

    Google Scholar 

  116. Fabijan, A., Olsson, H.H., Bosch, J.: Customer feedback and data collection techniques in software r&d: a literature review. In: International Conference of Software Business, pp. 139–153. Springer (2015)

    Google Scholar 

  117. Fabijan, A., Olsson, H.H., Bosch, J.: The lack of sharing of customer data in large software organizations: challenges and implications. In: International Conference on Agile Software Development, pp. 39–52. Springer (2016)

    Google Scholar 

  118. Fabijan, A., Olsson, H.H., Bosch, J.: Time to say’good bye’: Feature lifecycle. In: 2016 42th Euromicro Conference on Software Engineering and Advanced Applications (SEAA), pp. 9–16. IEEE (2016)

    Google Scholar 

  119. Fagerholm, F., Guinea, A.S., Mäenpää, H., Münch, J.: Building blocks for continuous experimentation. In: Proceedings of the 1st international workshop on rapid continuous software engineering, pp. 26–35 (2014)

    Google Scholar 

  120. Fagerholm, F., Guinea, A.S., Mäenpää, H., Münch, J.: The right model for continuous experimentation. Journal of Systems and Software 123, 292–305 (2017)

    Google Scholar 

  121. Fatima, A., Bibi, S., Hanif, R.: Comparative study on static code analysis tools for c/c++. In: Applied Sciences and Technology (IBCAST), 2018 15th International Bhurban Conference on, pp. 465–469. IEEE (2018)

    Google Scholar 

  122. Feldmann, S., Herzig, S., Kernschmidt, K., Wolfenstetter, T., Kammerl, D., Qamar, A., Lindemann, U., Krcmar, H., Paredis, C., Vogel-Heuser, B.: A Comparison of Inconsistency Management Approaches Using a Mechatronic Manufacturing System Design Case Study. In: 2015 IEEE International Conference on Automation Science and Engineering (CASE), pp. 158–165. IEEE (2015). DOI 10.1109/coase.2015.7294055

    Google Scholar 

  123. Feldmann, S., Wimmer, M., Kernschmidt, K., Vogel-Heuser, B.: A Comprehensive Approach for Managing Inter-Model Inconsistencies in Automated Production Systems Engineering. In: 2016 IEEE International Conference on Automation Science and Engineering (CASE), pp. 1120–1127. IEEE (2016). DOI 10.1109/coase.2016.7743530

    Google Scholar 

  124. Fenton, N., Bieman, J.: Software metrics: a rigorous and practical approach. CRC Press (2014)

    Google Scholar 

  125. Feyh, M., Petersen, K.: Lean software development measures and indicators-a systematic mapping study. In: Lean Enterprise Software and Systems, pp. 32–47. Springer (2013)

    Google Scholar 

  126. Fisher, R.A.: On the Interpretation of χ2 from Contingency Tables, and the Calculation of P. Journal of the Royal Statistical Society 85(1), 87 (1922). DOI 10.2307/2340521. URL http://www.jstor.org/stable/2340521?origin=crossref

    Article  Google Scholar 

  127. Fitzgerald, B., Stol, K.J.: Continuous software engineering: A roadmap and agenda. Journal of Systems and Software 123, 176–189 (2017)

    Google Scholar 

  128. Fitzgerald, B., Stol, K.J., O’Sullivan, R., O’Brien, D.: Scaling agile methods to regulated environments: An industry case study. In: 2013 35th International Conference on Software Engineering (ICSE), pp. 863–872. IEEE (2013)

    Google Scholar 

  129. Flick, U.: An introduction to qualitative research. Sage Publications Ltd (2009)

    Google Scholar 

  130. Flick, U.: Designing qualitative research. Sage (2018)

    Google Scholar 

  131. Fontana, F.A., Mäntylä, M.V., Zanoni, M., Marino, A.: Comparing and experimenting machine learning techniques for code smell detection. Empirical Software Engineering 21(3), 1143–1191 (2016)

    Google Scholar 

  132. Fontana, F.A., Roveda, R., Zanoni, M.: Tool support for evaluating architectural debt of an existing system: An experience report. In: Proceedings of the 31st Annual ACM Symposium on Applied Computing, pp. 1347–1349 (2016)

    Google Scholar 

  133. Fontana, F.A., Zanoni, M., Marino, A., Mantyla, M.V.: Code smell detection: Towards a machine learning-based approach. In: Software Maintenance (ICSM), 2013 29th IEEE International Conference on, pp. 396–399. IEEE (2013)

    Google Scholar 

  134. Fowler, K.: Mission-critical and safety-critical development. IEEE Instrumentation & Measurement Magazine 7(4), 52–59 (2004)

    Google Scholar 

  135. Fowler, M.: Continuous Integration. https://martinfowler.com/articles/continuousIntegration.html (2006). [Online; accessed 30-January-2013]

  136. Frajtak, K., Bures, M., Jelinek, I.: Model-based testing and exploratory testing: Is synergy possible? In: 2016 6th International Conference on IT Convergence and Security (ICITCS), pp. 1–6. IEEE (2016)

    Google Scholar 

  137. Frajtak, K., Bures, M., Jelinek, I.: Exploratory testing supported by automated reengineering of model of the system under test. Cluster Computing 20(1), 855–865 (2017)

    Google Scholar 

  138. Bernard Nicolau de França, B., Horta Travassos, G.: Simulation based studies in software engineering: A matter of validity. CLEI electronic journal 18(1), 5–5 (2015)

    Google Scholar 

  139. Freitas, A.A.: Comprehensible classification models: a position paper. ACM SIGKDD explorations newsletter 15(1), 1–10 (2014)

    Google Scholar 

  140. Fu, Q., Zhu, J., Hu, W., Lou, J.G., Ding, R., Lin, Q., Zhang, D., Xie, T.: Where do developers log? an empirical study on logging practices in industry. In: Companion Proceedings of the 36th International Conference on Software Engineering, pp. 24–33 (2014)

    Google Scholar 

  141. Fu, T.c.: A review on time series data mining. Engineering Applications of Artificial Intelligence 24(1), 164–181 (2011)

    Google Scholar 

  142. Fu, Y., Zhu, X., Li, B.: A survey on instance selection for active learning. Knowledge and information systems 35(2), 249–283 (2013)

    Google Scholar 

  143. Gatrell, M., Counsell, S., Hall, T.: Empirical support for two refactoring studies using commercial c# software. In: 13th International Conference on Evaluation and Assessment in Software Engineering (EASE), pp. 1–10 (2009)

    Google Scholar 

  144. Gebizli, C.S., Sözer, H.: Improving models for model-based testing based on exploratory testing. In: 2014 IEEE 38th International Computer Software and Applications Conference Workshops, pp. 656–661. IEEE (2014)

    Google Scholar 

  145. Gebizli, C.Ş., Sözer, H.: Automated refinement of models for model-based testing using exploratory testing. Software Quality Journal 25(3), 979–1005 (2017)

    Google Scholar 

  146. Gebizli, C.Ş., Sözer, H.: Impact of education and experience level on the effectiveness of exploratory testing: An industrial case study. In: 2017 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW), pp. 23–28. IEEE (2017)

    Google Scholar 

  147. Geels, F.W., Kemp, R.: Dynamics in socio-technical systems: Typology of change processes and contrasting case studies. Technology in Society 29(4), 441 – 455 (2007). DOI http://dx.doi.org/10.1016/j.techsoc.2007.08.009

    Google Scholar 

  148. Gestwicki, P.: The entity system architecture and its application in an undergraduate game development studio. In: Proceedings of the International Conference on the Foundations of Digital Games, pp. 73–80 (2012)

    Google Scholar 

  149. Ghazi, A.N., Garigapati, R.P., Petersen, K.: Checklists to support test charter design in exploratory testing. In: International Conference on Agile Software Development, pp. 251–258. Springer (2017)

    Google Scholar 

  150. Ghazi, A.N., Petersen, K., Bjarnason, E., Runeson, P.: Levels of exploration in exploratory testing: From freestyle to fully scripted. IEEE Access 6, 26416–26423 (2018)

    Google Scholar 

  151. Gibbs, G.R.: Analyzing qualitative data, vol. 6. Sage (2018)

    Google Scholar 

  152. Gilb, T.: Software metrics. Winthrop Publishers (1977)

    Google Scholar 

  153. Goldratt, E.M., Cox, J.: The goal: a process of ongoing improvement. Routledge (2016)

    Google Scholar 

  154. Goodman, D., Elbaz, M.: ”it’s not the pants, it’s the people in the pants” learnings from the gap agile transformation what worked, how we did it, and what still puzzles us. In: Agile 2008 Conference, pp. 112–115. IEEE (2008)

    Google Scholar 

  155. Goodman, L.A.: Snowball Sampling. The Annals of Mathematical Statistics 32(1), 148–170 (1961)

    MathSciNet  MATH  Google Scholar 

  156. Goodman, P.: Practical implementation of software metrics. International software quality assurance series. McGraw-Hill, London (1993). Lc92042989 Paul Goodman

    Google Scholar 

  157. Goodman, P.S., Bazerman, M., Conlon, E.: Institutionalization of planned organizational change. In: Research in Organizational Behavior, pp. 215–246. JAI Press,Greenwich (1980)

    Google Scholar 

  158. Goodman, P.S., Dean Jr, J.W.: Creating long-term organizational change. In: Change In Organizations. Carnegie-Mellon Univ Pittsburgh, PA, Graduate School of Industiral Administration (1982)

    Google Scholar 

  159. Goodman, R.M., Steckler, A.: A framework for assessing program institutionalization. Knowledge in Society 2(1), 57–71 (1989)

    Google Scholar 

  160. Gould, E., Marcus, A.: Company culture audit to improve development team’s collaboration, communication, and cooperation. In: Design, user experience, and usability. Theory, methods, tools and practice, pp. 415–424. Springer (2011)

    Google Scholar 

  161. Gregory, J., Crispin, L.: More agile testing: learning journeys for the whole team. Addison-Wesley Professional (2014)

    Google Scholar 

  162. Gryce, C., Finkelstein, A., Nentwich, C.: Lightweight Checking for UML Based Software Development. In: Workshop on Consistency Problems in UML-based Software Development., Dresden, Germany (2002)

    Google Scholar 

  163. Guinan, P.J., Cooprider, J.G., Faraj, S.: Enabling software development team performance during requirements definition: A behavioral versus technical approach. Information Systems Research 9(2), 101–125 (1998)

    Google Scholar 

  164. Guo, Y., Seaman, C., Gomes, R., Cavalcanti, A., Tonin, G., Da Silva, F.Q., Santos, A.L., Siebra, C.: Tracking technical debt—an exploratory case study. In: 2011 27th IEEE international conference on software maintenance (ICSM), pp. 528–531. IEEE (2011)

    Google Scholar 

  165. Guo, Y., Spínola, R.O., Seaman, C.: Exploring the costs of technical debt management–a case study. Empirical Software Engineering 21(1), 159–182 (2016)

    Google Scholar 

  166. Gyimothy, T., Ferenc, R., Siket, I.: Empirical validation of object-oriented metrics on open source software for fault prediction. Software Engineering, IEEE Transactions on 31(10), 897–910 (2005)

    Google Scholar 

  167. Hadar, E., Hassanzadeh, A.: Big data analytics on cyber attack graphs for prioritizing agile security requirements. In: 2019 IEEE 27th International Requirements Engineering Conference (RE), pp. 330–339 (2019). DOI 10.1109/RE.2019.00042

    Google Scholar 

  168. Hall, T., Beecham, S., Bowes, D., Gray, D., Counsell, S.: A systematic literature review on fault prediction performance in software engineering. Software Engineering, IEEE Transactions on 38(6), 1276–1304 (2012)

    Google Scholar 

  169. Hanssen, G.K., Haugset, B., Stålhane, T., Myklebust, T., Kulbrandstad, I.: Quality assurance in scrum applied to safety critical software. In: International Conference on Agile Software Development, pp. 92–103. Springer, Cham (2016)

    Google Scholar 

  170. Hartmann, D., Dymond, R.: Appropriate agile measurement: using metrics and diagnostics to deliver business value. In: Agile Conference, 2006, pp. 6–pp. IEEE (2006)

    Google Scholar 

  171. Hatcher, W.G., Yu, W.: A survey of deep learning: Platforms, applications and emerging research trends. IEEE Access 6, 24411–24432 (2018)

    Google Scholar 

  172. Hause, M.: The SysML Modelling Language. In: Fifteenth European Systems Engineering Conference, vol. 9, pp. 1–12. Citeseer (2006)

    Google Scholar 

  173. Heidenberg, J., Porres, I.: Metrics functions for kanban guards. In: Engineering of Computer Based Systems (ECBS), 2010 17th IEEE International Conference and Workshops on, pp. 306–310. IEEE (2010)

    Google Scholar 

  174. Heidenberg, J., Weijola, M., Mikkonen, K., Porres, I.: A metrics model to measure the impact of an agile transformation in large software development organizations. In: International Conference on Agile Software Development, pp. 165–179. Springer (2013)

    Google Scholar 

  175. Heikkilä, V.T., Damian, D., Lassenius, C., Paasivaara, M.: A mapping study on requirements engineering in agile software development. In: 2015 41st Euromicro conference on software engineering and advanced applications, pp. 199–207. IEEE (2015)

    Google Scholar 

  176. Heikkilä, V.T., Paasivaara, M., Lasssenius, C., Damian, D., Engblom, C.: Managing the requirements flow from strategy to release in large-scale agile development: a case study at ericsson. Empirical Software Engineering 22(6), 2892–2936 (2017)

    Google Scholar 

  177. Hellmann, T.D., Maurer, F.: Rule-based exploratory testing of graphical user interfaces. In: 2011 Agile Conference, pp. 107–116. IEEE (2011)

    Google Scholar 

  178. Hendrickson, E.: Explore it!: reduce risk and increase confidence with exploratory testing. Pragmatic Bookshelf (2013)

    Google Scholar 

  179. Herzig, S., Qamar, A., Paredis, C.: An approach to Identifying Inconsistencies in Model-Based Systems Engineering. Procedia Computer Science 28, 354–362 (2014). DOI 10.1016/j.procs.2014.03.044

    Google Scholar 

  180. Hetzel, B.: Making software measurement work: Building an effective measurement program. John Wiley & Sons, Inc. (1993)

    Google Scholar 

  181. Highsmith, J., Cockburn, A.: Agile software development: The business of innovation. Computer 34(9), 120–127 (2001)

    Google Scholar 

  182. Hill, J.H., Schmidt, D.C., Porter, A.A., Slaby, J.M.: Cicuts: combining system execution modeling tools with continuous integration environments. In: 15th Annual IEEE International Conference and Workshop on the Engineering of Computer Based Systems (ecbs 2008), pp. 66–75. IEEE (2008)

    Google Scholar 

  183. Hochstein, L., Basili, V.R., Zelkowitz, M.V., Hollingsworth, J.K., Carver, J.: Combining self-reported and automatic data to improve programming effort measurement. In: ACM SIGSOFT Software Engineering Notes, vol. 30, pp. 356–365. ACM (2005)

    Google Scholar 

  184. Hoda, R., Noble, J., Marshall, S.: Self-organizing roles on agile software development teams. IEEE Transactions on Software Engineering 39(3), 422–444 (2013). DOI 10.1109/TSE.2012.30

    Google Scholar 

  185. Hoffman, B., Cole, D., Vines, J.: Software process for rapid development of hpc software using cmake. In: 2009 DoD high performance computing modernization program users group conference, pp. 378–382. IEEE (2009)

    Google Scholar 

  186. Hohnhold, H., O’Brien, D., Tang, D.: Focusing on the long-term: It’s good for users and business. In: Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 1849–1858 (2015)

    Google Scholar 

  187. Holck, J., Jørgensen, N., et al.: Continuous integration and quality assurance: A case study of two open source projects. Australasian Journal of Information Systems 11(1) (2003)

    Google Scholar 

  188. Holmes, A., Kellogg, M.: Automating functional tests using selenium. In: AGILE 2006 (AGILE’06), pp. 6–pp. IEEE (2006)

    Google Scholar 

  189. Holmström Olsson, H., Alahyari, H., Bosch, J.: Climbing the “stairway to heaven”. In: Proceeding of the Euromicro Conference on Software Engineering and Advanced Applications. Cesme, Izmir, Turkey (2012)

    Google Scholar 

  190. Holvitie, J., Leppänen, V.: Debtflag: Technical debt management with a development environment integrated tool. In: 2013 4th International Workshop on Managing Technical Debt (MTD), pp. 20–27. IEEE (2013)

    Google Scholar 

  191. Holvitie, J., Leppänen, V., Hyrynsalmi, S.: Technical debt and the effect of agile software development practices on it-an industry practitioner survey. In: 2014 Sixth International Workshop on Managing Technical Debt, pp. 35–42. IEEE (2014)

    Google Scholar 

  192. Horkoff, J., Lindman, J., Hammouda, I., Knauss, E.: Experiences applying e3 value modeling in a cross-company study. In: International conference on conceptual modeling, pp. 610–625. Springer (2018)

    Google Scholar 

  193. Huang, H.Y., Liu, H.H., Li, Z.J., Zhu, J.: Surrogate: A simulation apparatus for continuous integration testing in service oriented architecture. In: 2008 IEEE International Conference on Services Computing, vol. 2, pp. 223–230. IEEE (2008)

    Google Scholar 

  194. Huang, Q., Shihab, E., Xia, X., Lo, D., Li, S.: Identifying self-admitted technical debt in open source projects using text mining. Empirical Software Engineering 23(1), 418–451 (2018)

    Google Scholar 

  195. Hudson, J., Denzinger, J.: Risk management for self-adapting self-organizing emergent multi-agent systems performing dynamic task fulfillment. Autonomous Agents and Multi-Agent Systems 29(5), 973–1022 (2015)

    Google Scholar 

  196. Humble, J., Farley, D.: Continuous delivery: reliable software releases through build, test, and deployment automation. Pearson Education (2010)

    Google Scholar 

  197. Humphrey, W.S., Chick, T.A., Nichols, W.R., Pomeroy-Huff, M.: Team software process(tsp) body of knowledge (bok). Tech. rep., Carnegie Mellon University (2010)

    Google Scholar 

  198. Huzar, Z., Kuzniarz, L., Reggio, G., Sourrouille, J.L.: Consistency Problems in UML-Based Software Development. In: UML Modeling Languages and Applications, pp. 1–12. Springer (2005). DOI 10.1007/978-3-540-31797-5_1

    Google Scholar 

  199. Idri, A., Abran, A.: Evaluating software project similarity by using linguistic quantifier guided aggregations. In: IFSA World Congress and 20th NAFIPS International Conference, 2001. Joint 9th, vol. 1, pp. 470–475. IEEE (2001)

    Google Scholar 

  200. Idri, A., azzahra Amazal, F., Abran, A.: Analogy-based software development effort estimation: A systematic mapping and review. Information and Software Technology 58, 206–230 (2015)

    Google Scholar 

  201. Idri, A., Zahi, A., Abran, A.: Software cost estimation by fuzzy analogy for web hypermedia applications. In: Proceedings of the International Conference on Software Process and Product Measurement, Cadiz, Spain, pp. 53–62 (2006)

    Google Scholar 

  202. IEEE Standard Glossary of Software Engineering Terminology (1990). IEEE Standards Board/American National Standards Institute, Std. 610.12-1990

    Google Scholar 

  203. Inayat, I., Salim, S.S., Marczak, S., Daneva, M., Shamshirband, S.: A systematic literature review on agile requirements engineering practices and challenges. Computers in human behavior 51, 915–929 (2015)

    Google Scholar 

  204. International vocabulary of basic and general terms in metrology (1993). International Organization for Standardization

    Google Scholar 

  205. Irwin, W., Churcher, N.: A generated parser of c++. NZ Journal of Computing 8(3), 26–37 (2001)

    Google Scholar 

  206. ISO: Iso 26262: 2018:“road vehicles—functional safety” (2018)

    Google Scholar 

  207. ISO/IEC/IEEE Systems and software engineering – Architecture description (2011). DOI 10.1109/IEEESTD.2011.6129467

    Google Scholar 

  208. ISO/IEC 15939: Systems and Software Engineering - Measurement Process (2007)

    Google Scholar 

  209. Itkonen, J., Mantyla, M.V., Lassenius, C.: How do testers do it? an exploratory study on manual testing practices. In: 2009 3rd International Symposium on Empirical Software Engineering and Measurement, pp. 494–497. IEEE (2009)

    Google Scholar 

  210. Itkonen, J., Mäntylä, M.V., Lassenius, C.: The role of the tester’s knowledge in exploratory software testing. IEEE Transactions on Software Engineering 39(5), 707–724 (2012)

    Google Scholar 

  211. Jacquet, J.P., Abran, A.: From software metrics to software measurement methods: a process model. In: Third IEEE International Software Engineering Standards Symposium and Forum – Emerging International Standards, ISESS, pp. 128–135. IEEE (1997)

    Google Scholar 

  212. Janus, A., Dumke, R., Schmietendorf, A., Jäger, J.: The 3c approach for agile quality assurance. In: 2012 3rd International Workshop on Emerging Trends in Software Metrics (WETSoM), pp. 9–13. IEEE (2012)

    Google Scholar 

  213. Jenkins. http://jenkins-ci.org. [Online; accessed 30-January-2013]

  214. John, M.M., Olsson, H.H., Bosch, J.: Ai on the edge: Architectural alternatives. In: 2020 46th Euromicro Conference on Software Engineering and Advanced Applications (SEAA), pp. 21–28. IEEE (2020)

    Google Scholar 

  215. John, M.M., Olsson, H.H., Bosch, J.: Developing ml/dl models: A design framework. In: Proceedings of the International Conference on Software and System Processes, pp. 1–10 (2020)

    Google Scholar 

  216. Johnson, D.E.: Crossover experiments. Wiley Interdisciplinary Reviews: Computational Statistics 2(5), 620–625 (2010)

    Google Scholar 

  217. Johnson, P.M.: Project hackystat: Accelerating adoption of empirically guided software development through non-disruptive, developer-centric, in-process data collection and analysis. Department of Information and Computer Sciences, University of Hawaii 22 (2001)

    Google Scholar 

  218. Johnson, P.M., Kou, H., Agustin, J., Chan, C., Moore, C., Miglani, J., Zhen, S., Doane, W.E.: Beyond the personal software process: Metrics collection and analysis for the differently disciplined. In: Proceedings of the 25th international Conference on Software Engineering, pp. 641–646. IEEE Computer Society (2003)

    Google Scholar 

  219. Johnson, T., Kerzhner, A., Paredis, C., Burkhart, R.: Integrating Models and Simulations of Continuous Dynamics into SysML. Journal of Computing and Information Science in Engineering 12 (2012). DOI 10.1115/1.4005452

    Google Scholar 

  220. Jorgensen, M.: Software quality measurement. Advances in Engineering Software 30(12), 907–912 (1999)

    Google Scholar 

  221. Jørgensen, M.: Do agile methods work for large software projects? In: International Conference on Agile Software Development, pp. 179–190. Springer (2018)

    Google Scholar 

  222. Jung, H.W., Kim, S.G., Chung, C.S.: Measuring software product quality: A survey of iso/iec 9126. IEEE software 21(5), 88–92 (2004)

    Google Scholar 

  223. Kahkonen, T.: Agile methods for large organizations-building communities of practice. In: Agile development conference, pp. 2–10. IEEE (2004)

    Google Scholar 

  224. Kai, G.: Virtual measurement system for muzzle velocity and firing frequency. In: 8th International Conference on Electronic Measurement and Instruments, pp. 176–179 (2001)

    Google Scholar 

  225. Kaisti, M., Mujunen, T., Mäkilä, T., Rantala, V., Lehtonen, T.: Agile principles in the embedded system development. In: Agile Processes in Software Engineering and Extreme Programming, Lecture Notes in Business Information Processing, vol. 179, pp. 16–31. Springer, Rome, Italy (2014). DOI 10.1007/978-3-319-06862-6_2

    Google Scholar 

  226. Kaner, C.: Testing computer software. TAB Books (1988)

    Google Scholar 

  227. Kaner, C., Bach, J., Pettichord, B.: Lessons learned in software testing. John Wiley & Sons (2001)

    Google Scholar 

  228. Kaplan, B., Maxwell, J.A.: Qualitative research methods for evaluating computer information systems. In: Evaluating the organizational impact of healthcare information systems, pp. 30–55. Springer (2005)

    Google Scholar 

  229. Kaplan, R.S., Norton, D.P.: Putting the balanced scorecard to work. Performance measurement, management, and appraisal sourcebook 66 (1995)

    Google Scholar 

  230. Kasauli, R., Knauss, E., Kanagwa, B., Nilsson, A., Calikli, G.: Safety-critical systems and agile development: A mapping study. In: 2018 44th Euromicro Conference on Software Engineering and Advanced Applications (SEAA), pp. 470–477 (2018). DOI 10.1109/SEAA.2018.00082

    Google Scholar 

  231. Kasauli, R., Knauss, E., Nilsson, A., Klug, S.: Adding value every sprint: A case study on large-scale continuous requirements engineering. In: REFSQ Workshops (2017)

    Google Scholar 

  232. Kasauli, R., Wohlrab, R., Knauss, E., Steghöfer, J.P., Horkoff, J., Maro, S.: Charting coordination needs in large-scale agile organisations with boundary objects and methodological islands. In: Proceedings of the International Conference on Software and System Processes, ICSSP ’20, p. 51–60. Association for Computing Machinery, New York, NY, USA (2020). DOI 10.1145/3379177.3388897. URL https://doi.org/10.1145/3379177.3388897

  233. Kazman, R., Cai, Y., Mo, R., Feng, Q., Xiao, L., Haziyev, S., Fedak, V., Shapochka, A.: A case study in locating the architectural roots of technical debt. In: 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering, vol. 2, pp. 179–188. IEEE (2015)

    Google Scholar 

  234. Keogh, E., Lin, J.: Clustering of time-series subsequences is meaningless: implications for previous and future research. Knowledge and information systems 8(2), 154–177 (2005)

    Google Scholar 

  235. Kerievsky, J.: Industrial XP: Making XP work in large organizations. Executive Report Vol. 6, No. 2, Cutter Consortium (2005). URL http://www.cutter.com/content-and-analysis/resource-centers/agile-project-management/sample-our-research/apmr0502.html

  236. Kettunen, P.: Adopting key lessons from agile manufacturing to agile software product development—a comparative study. Technovation 29(6), 408–422 (2009)

    Google Scholar 

  237. Kettunen, P., Laanti, M.: Combining agile software projects and large-scale organizational agility. Software Process: Improvement and Practice 13(2), 183–193 (2008). DOI 10.1002/spip.354. URL http://onlinelibrary.wiley.com/doi/10.1002/spip.354/abstract

    Article  Google Scholar 

  238. Khurum, M., Gorschek, T., Wilson, M.: The software value map—an exhaustive collection of value aspects for the development of software intensive products. Journal of software: Evolution and Process 25(7), 711–741 (2013)

    Google Scholar 

  239. Kilpi, T.: Implementing a software metrics program at nokia. IEEE Software 18(6), 72–77 (2001)

    Google Scholar 

  240. Kim, D.K., Lee, L.S.: Reverse engineering from exploratory testing to specification-based testing. International Journal of Software Engineering and Its Applications 8(11), 197–208 (2014)

    Google Scholar 

  241. Kim, E.H., Na, J.C., Ryoo, S.M.: Implementing an effective test automation framework. In: 2009 33rd Annual IEEE International Computer Software and Applications Conference, vol. 2, pp. 534–538. IEEE (2009)

    Google Scholar 

  242. Kim, E.H., Na, J.C., Ryoo, S.M.: Test automation framework for implementing continuous integration. In: 2009 Sixth International Conference on Information Technology: New Generations, pp. 784–789. IEEE (2009)

    Google Scholar 

  243. Kim, M., Zimmermann, T., DeLine, R., Begel, A.: The emerging role of data scientists on software development teams. In: 2016 IEEE/ACM 38th International Conference on Software Engineering (ICSE), pp. 96–107. IEEE (2016)

    Google Scholar 

  244. Kim, S., Park, S., Yun, J., Lee, Y.: Automated continuous integration of component-based software: An industrial experience. In: 2008 23rd IEEE/ACM International Conference on Automated Software Engineering, pp. 423–426. IEEE (2008)

    Google Scholar 

  245. Kitchenham, B.: Procedures for performing systematic reviews. Keele, UK, Keele University 33(2004), 1–26 (2004)

    Google Scholar 

  246. Kitchenham, B.: What’s up with software metrics?–a preliminary mapping study. Journal of systems and software 83(1), 37–51 (2010)

    Google Scholar 

  247. Klaine, P.V., Imran, M.A., Onireti, O., Souza, R.D.: A survey of machine learning techniques applied to self-organizing cellular networks. IEEE Communications Surveys & Tutorials 19(4), 2392–2431 (2017)

    Google Scholar 

  248. Knaster, R., Leffingwell, D.: SAFe 4.0 distilled: applying the Scaled Agile Framework for lean software and systems engineering. Addison-Wesley Professional (2017)

    Google Scholar 

  249. Knauss, E., Liebel, G., Horkoff, J., Wohlrab, R., Kasauli, R., Lange, F., Gildert, P.: T-reqs: Tool support for managing requirements in large-scale agile system development. In: 2018 IEEE 26th International Requirements Engineering Conference (RE), pp. 502–503. IEEE (2018)

    Google Scholar 

  250. Kohavi, R., Deng, A., Frasca, B., Walker, T., Xu, Y., Pohlmann, N.: Online controlled experiments at large scale. In: Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 1168–1176 (2013)

    Google Scholar 

  251. Kohavi, R., Deng, A., Longbotham, R., Xu, Y.: Seven rules of thumb for web site experimenters. In: Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 1857–1866 (2014)

    Google Scholar 

  252. Kohavi, R., Longbotham, R., Sommerfield, D., Henne, R.M.: Controlled experiments on the web: survey and practical guide. Data mining and knowledge discovery 18(1), 140–181 (2009)

    MathSciNet  Google Scholar 

  253. Kolovos, D., Paige, R., Polack, F.: The Epsilon Object Language (EOL). In: European Conference on Model Driven Architecture-Foundations and Applications, pp. 128–142. Springer (2006). DOI 10.1007/11787044_11

    Google Scholar 

  254. Kolovos, D., Paige, R., Polack, F.: Detecting and Repairing Inconsistencies Across Heterogeneous Models. In: 2008 1st International Conference on Software Testing, Verification, and Validation, pp. 356–364. IEEE (2008). DOI 10.1109/icst.2008.23

    Google Scholar 

  255. Kruchten, P., Nord, R.L., Ozkaya, I.: Technical debt: From metaphor to theory and practice. Ieee software 29(6), 18–21 (2012)

    Google Scholar 

  256. Kuhn, A.: On extracting unit tests from interactive live programming sessions. In: 2013 35th International Conference on Software Engineering (ICSE), pp. 1241–1244. IEEE (2013)

    Google Scholar 

  257. Kuhrmann, M., Diebold, P., Münch, J., Tell, P., Garousi, V., Felderer, M., Trektere, K., McCaffery, F., Linssen, O., Hanser, E., Prause, C.R.: Hybrid software and system development in practice: Waterfall, scrum, and beyond. In: Proceedings of the 2017 International Conference on Software and System Process, ICSSP 2017, p. 30–39. Association for Computing Machinery, New York, NY, USA (2017). DOI 10.1145/3084100.3084104. URL https://doi.org/10.1145/3084100.3084104

  258. Kumar, S., Wallace, C.: Guidance for exploratory testing through problem frames. In: 2013 26th International Conference on Software Engineering Education and Training (CSEE&T), pp. 284–288. IEEE (2013)

    Google Scholar 

  259. Kunz, R.F., Kasmala, G.F., Mahaffy, J.H., Murray, C.J.: On the automated assessment of nuclear reactor systems code accuracy. Nuclear Engineering and Design 211(2-3), 245–272 (2002). TY - JOUR

    Google Scholar 

  260. Laanti, M., Salo, O., Abrahamsson, P.: Agile methods rapidly replacing traditional methods at nokia: A survey of opinions on agile transformation. Information and Software Technology 53(3), 276–290 (2011)

    Google Scholar 

  261. Lacoste, F.J.: Killing the gatekeeper: Introducing a continuous integration system. In: 2009 agile conference, pp. 387–392. IEEE (2009)

    Google Scholar 

  262. Lagerberg, L., Skude, T., Emanuelsson, P., Sandahl, K., Ståhl, D.: The impact of agile principles and practices on large-scale software development projects: A multiple-case study of two projects at ericsson. In: 2013 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement, pp. 348–356. IEEE (2013)

    Google Scholar 

  263. Larman, C.: Scaling lean & agile development: thinking and organizational tools for large-scale Scrum. Pearson Education India (2008)

    Google Scholar 

  264. Larman, C., Vodde, B.: Large-scale scrum: More with LeSS. Addison-Wesley Professional (2016)

    Google Scholar 

  265. Lauesen, S.: Software requirements: styles and techniques. Pearson Education (2002)

    Google Scholar 

  266. Lauesen, S.: Guide to requirements SL-07. Lauesen Publishing (2017)

    Google Scholar 

  267. Layman, L., Williams, L., Cunningham, L.: Motivations and measurements in an agile case study. Journal of Systems Architecture 52(11), 654–667 (2006)

    Google Scholar 

  268. Lee, C.L., Yang, H.J.: Organization structure, competition and performance measurement systems and their joint effects on performance. Management Accounting Research 22(2), 84–104 (2011)

    Google Scholar 

  269. Leffingwell, D.: Agile software requirements: lean requirements practices for teams, programs, and the enterprise. Addison-Wesley Professional (2010)

    Google Scholar 

  270. Leffingwell, D., et al.: Scaled agile framework 3.0 (2014)

    Google Scholar 

  271. Li, Z., Avgeriou, P., Liang, P.: A systematic mapping study on technical debt and its management. Journal of Systems and Software 101, 193–220 (2015)

    Google Scholar 

  272. Lier, F., Wrede, S., Siepmann, F., Lütkebohle, I., Paul-Stueve, T., Wachsmuth, S.: Facilitating research cooperation through linking and sharing of heterogenous research artefacts: cross platform linking of semantically enriched research artefacts. In: Proceedings of the 8th International Conference on Semantic Systems, pp. 157–164 (2012)

    Google Scholar 

  273. Lin, J., Keogh, E., Lonardi, S., Chiu, B.: A symbolic representation of time series, with implications for streaming algorithms. In: Proceedings of the 8th ACM SIGMOD workshop on Research issues in data mining and knowledge discovery, pp. 2–11. ACM (2003)

    Google Scholar 

  274. Lin, J., Keogh, E., Lonardi, S., Patel, P.: Finding motifs in time series. In: In the 2nd Workshop on Temporal Data Mining, at the 8th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 53–68 (2002)

    Google Scholar 

  275. Lin, J., Keogh, E., Wei, L., Lonardi, S.: Experiencing sax: a novel symbolic representation of time series. Data Mining and knowledge discovery 15(2), 107–144 (2007)

    MathSciNet  Google Scholar 

  276. Lin, J., Kolcz, A.: Large-scale machine learning at twitter. In: Proceedings of the 2012 ACM SIGMOD International Conference on Management of Data, pp. 793–804 (2012)

    Google Scholar 

  277. Lindgren, E., Münch, J.: Raising the odds of success: the current state of experimentation in product development. Information and Software Technology 77, 80–91 (2016)

    Google Scholar 

  278. Lindman, J., Horkoff, J., Hammouda, I., Knauss, E.: Emerging perspectives of application programming interface strategy: A framework to respond to business concerns. IEEE Software 37(2), 52–59 (2020). DOI 10.1109/MS.2018.2875964

    Google Scholar 

  279. Lindvall, M., Muthig, D., Dagnino, A., Wallin, C., Stupperich, M., Kiefer, D., May, J., Kahkonen, T.: Agile software development in large organizations. Computer 37(12), 26–34 (2004)

    Google Scholar 

  280. Liu, H., Li, Z., Zhu, J., Tan, H., Huang, H.: A unified test framework for continuous integration testing of soa solutions. In: 2009 IEEE International Conference on Web Services, pp. 880–887. IEEE (2009)

    Google Scholar 

  281. Liu, S., Xiao, F., Ou, W., Si, L.: Cascade ranking for operational e-commerce search. In: Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 1557–1565 (2017)

    Google Scholar 

  282. Lokan, C., Mendes, E.: Cross-company and single-company effort models using the isbsg database: A further replicated study. In: Proceedings of the 2006 ACM/IEEE International Symposium on Empirical Software Engineering, ISESE ’06, pp. 75–84. ACM, New York, NY, USA (2006). DOI 10.1145/1159733.1159747. URL http://doi.acm.org/10.1145/1159733.1159747

  283. Lokan, C., Wright, T., Hill, P.R., Stringer, M.: Organizational benchmarking using the isbsg data repository. IEEE Software 18(5), 26–32 (2001)

    Google Scholar 

  284. Long, B.: Managing module dependencies to facilitate continuous testing. Information processing letters 108(3), 127–131 (2008)

    MathSciNet  MATH  Google Scholar 

  285. Lucas, F., Molina, F., Toval, A.: A systematic review of UML model consistency management. Information and Software Technology 51(12), 1631–1645 (2009). DOI 10.1016/j.infsof.2009.04.009

    Google Scholar 

  286. Lucassen, G., Dalpiaz, F., van der Werf, J.M.E., Brinkkemper, S.: Forging high-quality user stories: Towards a discipline for agile requirements. In: 2015 IEEE 23rd International Requirements Engineering Conference (RE), pp. 126–135 (2015). DOI 10.1109/RE.2015.7320415

    Google Scholar 

  287. Luckow, A., Cook, M., Ashcraft, N., Weill, E., Djerekarov, E., Vorster, B.: Deep learning in the automotive industry: Applications and tools. In: 2016 IEEE International Conference on Big Data (Big Data), pp. 3759–3768. IEEE (2016)

    Google Scholar 

  288. Lwakatare, L.E., Raj, A., Bosch, J., Olsson, H.H., Crnkovic, I.: A taxonomy of software engineering challenges for machine learning systems: An empirical investigation. In: International Conference on Agile Software Development, pp. 227–243. Springer, Cham (2019)

    Google Scholar 

  289. Maguire, M., Delahunt, B.: Doing a thematic analysis: A practical, step-by-step guide for learning and teaching scholars. All Ireland Journal of Higher Education 9(3) (2017)

    Google Scholar 

  290. van Manen, H., van Vliet, H.: Organization-wide agile expansion requires an organization-wide agile mindset. In: Product-Focused Software Process Improvement, Lecture Notes in Computer Science, pp. 48–62. Springer, Helsinki, Finland (2014). URL http://link.springer.com/chapter/10.1007/978-3-319-13835-0_4

  291. Mantere, M., Uusitalo, I., Roning, J.: Comparison of static code analysis tools. In: Emerging Security Information, Systems and Technologies, 2009. SECURWARE’09. Third International Conference on, pp. 15–22. IEEE (2009)

    Google Scholar 

  292. Manzi, J.: Uncontrolled: The surprising payoff of trial-and-error for business, politics, and society. Basic Books (AZ) (2012)

    Google Scholar 

  293. Mårtensson, T., Martini, A., Ståhl, D., Bosch, J.: Excellence in exploratory testing: Success factors in large-scale industry projects. In: International Conference on Product-Focused Software Process Improvement, pp. 299–314. Springer (2019)

    Google Scholar 

  294. Mårtensson, T., Ståhl, D., Bosch, J.: Exploratory testing of large-scale systems–testing in the continuous integration and delivery pipeline. In: International Conference on Product-Focused Software Process Improvement, pp. 368–384. Springer (2017)

    Google Scholar 

  295. Mårtensson, T., Ståhl, D., Bosch, J.: Enable more frequent integration of software in industry projects. Journal of Systems and Software 142, 223–236 (2018)

    Google Scholar 

  296. Mårtensson, T., Ståhl, D., Bosch, J.: Test activities in the continuous integration and delivery pipeline. Journal of Software: Evolution and Process 31(4), e2153 (2019)

    Google Scholar 

  297. Martin, R.C.: Agile software development: principles, patterns, and practices. Prentice Hall (2002)

    Google Scholar 

  298. Martini, A., Besker, T., Bosch, J.: The introduction of technical debt tracking in large companies. In: 2016 23rd Asia-Pacific Software Engineering Conference (APSEC), pp. 161–168. IEEE (2016)

    Google Scholar 

  299. Martini, A., Bosch, J.: The danger of architectural technical debt: Contagious debt and vicious circles. In: 2015 12th Working IEEE/IFIP Conference on Software Architecture, pp. 1–10. IEEE (2015)

    Google Scholar 

  300. Martini, A., Bosch, J.: An empirically developed method to aid decisions on architectural technical debt refactoring: Anacondebt. In: 2016 IEEE/ACM 38th International Conference on Software Engineering Companion (ICSE-C), pp. 31–40. IEEE (2016)

    Google Scholar 

  301. Martini, A., Bosch, J.: A multiple case study of continuous architecting in large agile companies: current gaps and the caffea framework. In: 2016 13th Working IEEE/IFIP Conference on Software Architecture (WICSA), pp. 1–10. IEEE (2016)

    Google Scholar 

  302. Martini, A., Bosch, J.: The magnificent seven: towards a systematic estimation of technical debt interest. In: Proceedings of the XP2017 Scientific Workshops, pp. 1–5 (2017)

    Google Scholar 

  303. Martini, A., Bosch, J., Chaudron, M.: Investigating architectural technical debt accumulation and refactoring over time: A multiple-case study. Information and Software Technology 67, 237–253 (2015)

    Google Scholar 

  304. Maruping, L.M., Zhang, X., Venkatesh, V.: Role of collective ownership and coding standards in coordinating expertise in software project teams. European Journal of Information Systems 18(4), 355–371 (2009)

    Google Scholar 

  305. Masters, J.: The history of action research. Action research electronic reader 22, 2005 (1995)

    Google Scholar 

  306. Masuda, S., Ono, K., Yasue, T., Hosokawa, N.: A survey of software quality for machine learning applications. In: 2018 IEEE International conference on software testing, verification and validation workshops (ICSTW), pp. 279–284. IEEE (2018)

    Google Scholar 

  307. Matsumoto, K., Kibe, S., Uehara, M., Mori, H.: Design of development as a service in the cloud. In: 2012 15th International Conference on Network-Based Information Systems, pp. 815–819. IEEE (2012)

    Google Scholar 

  308. Mattos, D.I., Bosch, J., Olsson, H.H.: Challenges and strategies for undertaking continuous experimentation to embedded systems: Industry and research perspectives. In: 19th International Conference on Agile Software Development (2018)

    Google Scholar 

  309. Maximilien, E.M., Williams, L.: Assessing test-driven development at ibm. In: Software Engineering, 2003. Proceedings. 25th International Conference on, pp. 564–569. IEEE (2003)

    Google Scholar 

  310. Maxwell, J.A.: Qualitative research design: An interactive approach, vol. 41. Sage publications (2012)

    Google Scholar 

  311. Maxwell, K.D., Forselius, P.: Benchmarking software development productivity. IEEE Software 17(1), 80–88 (2000). DOI 10.1109/52.820015

    Google Scholar 

  312. Mayring, P.: Qualitative content analysis–research instrument or mode of interpretation. The role of the researcher in qualitative psychology 2(139-148) (2002)

    Google Scholar 

  313. McConnell, S.: Managing technical debt presentation at icse 2013 (2013)

    Google Scholar 

  314. McGarry, J.: Practical software measurement: objective information for decision makers. Addison-Wesley Professional (2002)

    Google Scholar 

  315. McIntosh, S., Kamei, Y., Adams, B., Hassan, A.E.: The impact of code review coverage and code review participation on software quality: A case study of the qt, vtk, and itk projects. In: Proceedings of the 11th Working Conference on Mining Software Repositories, pp. 192–201. ACM (2014)

    Google Scholar 

  316. McMahon, P.: Extending agile methods: A distributed project and organizational improvement perspective. In: Systems and Software Technology Conference (2005)

    Google Scholar 

  317. Melão, N., Pidd, M.: A conceptual framework for understanding business processes and business process modelling. Information systems journal 10(2), 105–129 (2000)

    Google Scholar 

  318. Mellado, R.P., Montini, D.Á., Dias, L.A.V., da Cunha, A.M., et al.: Software product measurement and analysis in a continuous integration environment. In: 2010 Seventh International Conference on Information Technology: New Generations, pp. 1177–1182. IEEE (2010)

    Google Scholar 

  319. Mendes, E., Lokan, C., Harrison, R., Triggs, C.: A replicated comparison of cross-company and within-company effort estimation models using the isbsg database. In: 11th IEEE International Software Metrics Symposium (METRICS’05), pp. 10 pp.–36 (2005). DOI 10.1109/METRICS.2005.4

    Google Scholar 

  320. Menzies, T., Butcher, A., Cok, D., Marcus, A., Layman, L., Shull, F., Turhan, B., Zimmermann, T.: Local versus global lessons for defect prediction and effort estimation. IEEE Transactions on software engineering 39(6), 822–834 (2013)

    Google Scholar 

  321. Meyer, B.: The ugly, the hype and the good: an assessment of the agile approach. In: Agile!, pp. 149–154. Springer (2014)

    Google Scholar 

  322. Mi, Q., Keung, J., Xiao, Y., Mensah, S., Gao, Y.: Improving code readability classification using convolutional neural networks. Information and Software Technology 104, 60–71 (2018)

    Google Scholar 

  323. Micallef, M., Porter, C., Borg, A.: Do exploratory testers need formal training? an investigation using hci techniques. In: 2016 IEEE Ninth International Conference on Software Testing, Verification and Validation Workshops (ICSTW), pp. 305–314. IEEE (2016)

    Google Scholar 

  324. Mihindukulasooriya, N., Rizzo, G., Troncy, R., Corcho, O., García-Castro, R.: A two-fold quality assurance approach for dynamic knowledge bases: The 3cixty use case. In: (KNOW@ LOD/CoDeS)@ ESWC (2016)

    Google Scholar 

  325. Miles, M.B., Huberman, A.M.: Qualitative data analysis: An expanded sourcebook. sage (1994)

    Google Scholar 

  326. Miller, A.: A hundred days of continuous integration. In: Agile 2008 conference, pp. 289–293. IEEE (2008)

    Google Scholar 

  327. Moha, N., Gueheneuc, Y.G., Duchien, A.F., et al.: Decor: A method for the specification and detection of code and design smells. IEEE Transactions on Software Engineering (TSE) 36(1), 20–36 (2010)

    MATH  Google Scholar 

  328. Moitra, D.: Managing change for software process improvement initiatives: a practical experience-based approach. Software Process: Improvement and Practice 4(4), 199–207 (1998)

    Google Scholar 

  329. Mueen, A., Keogh, E., Zhu, Q., Cash, S., Westover, B.: Exact discovery of time series motifs. In: Proceedings of the 2009 SIAM international conference on data mining, pp. 473–484. SIAM (2009)

    Google Scholar 

  330. Mujtaba, S., Feldt, R., Petersen, K.: Waste and lead time reduction in a software product customization process with value stream maps. In: 2010 21st australian software engineering conference, pp. 139–148. IEEE (2010)

    Google Scholar 

  331. Müller, M., Sazama, F., Debou, C., Dudzic, P., Abowd, P.: Survey – State of Practice “Agile in Automotive”. Tech. rep., KUGLER MAAG CIE GmbH (2014). URL http://www.kuglermaag.com/improvement-concepts/agile-in-automotive/state-of-practice.html

  332. Munappy, A., Bosch, J., Olsson, H.H., Arpteg, A., Brinne, B.: Data management challenges for deep learning. In: 2019 45th Euromicro Conference on Software Engineering and Advanced Applications (SEAA), pp. 140–147. IEEE (2019)

    Google Scholar 

  333. Munappy, A.R., Mattos, D.I., Bosch, J., Olsson, H.H., Dakkak, A.: From ad-hoc data analytics to dataops. In: Proceedings of the International Conference on Software and System Processes, pp. 165–174 (2020)

    Google Scholar 

  334. Needleman, S.B., Wunsch, C.D.: A general method applicable to the search for similarities in the amino acid sequence of two proteins. Journal of molecular biology 48(3), 443–453 (1970)

    Google Scholar 

  335. Nentwich, C., Emmerich, W., Finkelstein, A., Ellmer, E.: Flexible Consistency Checking. ACM Transactions on Software Engineering and Methodology (TOSEM) 12(1), 28–63 (2003). DOI 10.1145/839268.839271

    Google Scholar 

  336. Niessink, F., van Vliet, H.: Measurements should generate value, rather than data. In: 6th International Software Metrics Symposium, pp. 31–38 (2000)

    Google Scholar 

  337. Niessink, F., van Vliet, H.: Measurement program success factors revisited. Information and Software Technology 43(10), 617–628 (2001). TY - JOUR

    Google Scholar 

  338. Nilsson, A., Bosch, J., Berger, C.: The civit model in a nutshell: Visualizing testing activities to support continuous integration. In: Continuous software engineering, pp. 97–106. Springer (2014)

    Google Scholar 

  339. Niven, P.R.: Balanced scorecard step-by-step: maximizing performance and maintaining results. John Wiley & Sons (2002)

    Google Scholar 

  340. Novak, J., Krajnc, A., ontar, R.: Taxonomy of static code analysis tools. In: MIPRO, 2010 Proceedings of the 33rd International Convention, pp. 418–422. IEEE (2010)

    Google Scholar 

  341. Ochodek, M., Staron, M., Bargowski, D., Meding, W., Hebig, R.: Using machine learning to design a flexible loc counter. In: Machine Learning Techniques for Software Quality Evaluation (MaLTeSQuE), IEEE Workshop on, pp. 14–20. IEEE (2017)

    Google Scholar 

  342. Offen, R.J., Jeffery, R.: Establishing software measurement programs. Software, IEEE 14(2), 45–53 (1997)

    Google Scholar 

  343. Olsson, H.H., Alahyari, H., Bosch, J.: Climbing the “stairway to heaven”–a mulitiple-case study exploring barriers in the transition from agile development towards continuous deployment of software. In: Software Engineering and Advanced Applications (SEAA), 2012 38th EUROMICRO Conference on, pp. 392–399. IEEE (2012)

    Google Scholar 

  344. Olsson, H.H., Bosch, J.: From opinions to data-driven software r&d: A multi-case study on how to close the ’open loop’ problem. In: 2014 40th EUROMICRO Conference on Software Engineering and Advanced Applications, pp. 9–16. IEEE (2014)

    Google Scholar 

  345. Olsson, H.H., Bosch, J.: The hypex model: from opinions to data-driven software development. In: Continuous software engineering, pp. 155–164. Springer (2014)

    Google Scholar 

  346. Olsson, H.H., Bosch, J.: Towards continuous customer validation: A conceptual model for combining qualitative customer feedback with quantitative customer observation. In: International Conference of Software Business, pp. 154–166. Springer (2015)

    Google Scholar 

  347. Olszewska, M., Heidenberg, J., Weijola, M., Mikkonen, K., Porres, I.: Quantitatively measuring a large-scale agile transformation. Journal of Systems and Software 117, 258 – 273 (2016). URL http://www.sciencedirect.com/science/article/pii/S016412121600087X

  348. Organization, I.S., Commission, I.E.: Software and systems engineering, software measurement process. Tech. rep., ISO/IEC (2007)

    Google Scholar 

  349. Paasivaara, M., Lassenius, C.: Challenges and success factors for large-scale agile transformations: A research proposal and a pilot study. In: Proceedings of the Scientific Workshop Proceedings of XP2016, pp. 1–5 (2016)

    Google Scholar 

  350. Paetsch, F., Eberlein, A., Maurer, F.: Requirements engineering and agile software development. In: WET ICE 2003. Proceedings. Twelfth IEEE International Workshops on Enabling Technologies: Infrastructure for Collaborative Enterprises, 2003., pp. 308–313. IEEE (2003)

    Google Scholar 

  351. Paige, R., Brooke, P., Ostroff, J.: Metamodel-Based Model Conformance and Multi-view Consistency Checking. ACM Transactions on Software Engineering and Methodology (TOSEM) 16(3), 11 (2007). DOI 10.1145/1243987.1243989

    Google Scholar 

  352. Pantazos, K., Shollo, A., Staron, M., Meding, W.: Presenting software metrics indicators-a case study. In: Proceedings of IWSM/Mensura conference (2010)

    Google Scholar 

  353. Patil, D.: Building data science teams. “ O’Reilly Media, Inc.” (2011)

    Google Scholar 

  354. Peach, R.W.: The ISO 9000 handbook. Irwin Professional Publishing (1995)

    Google Scholar 

  355. Pernstål, J., Magazinius, A., Gorschek, T.: A study investigating challenges in the interface between product development and manufacturing in the development of software-intensive automotive systems. International Journal of Software Engineering and Knowledge Engineering 22(07), 965–1004 (2012)

    Google Scholar 

  356. Persson, M., Torngren, M., Qamar, A., Westman, J., Biehl, M., Tripakis, S., Vangheluwe, H., Denil, J.: A Characterization of Integrated Multi-View Modeling in the Context of Embedded and Cyber-Physical Systems. In: Embedded Software (EMSOFT), 2013 Proceedings of the International Conference on, pp. 1–10. IEEE (2013). DOI 10.1109/emsoft.2013.6658588

    Google Scholar 

  357. Pesola, J.P., Tanner, H., Eskeli, J., Parviainen, P., Bendas, D.: Integrating early v&v support to a gse tool integration platform. In: 2011 IEEE Sixth International Conference on Global Software Engineering Workshop, pp. 95–101. IEEE (2011)

    Google Scholar 

  358. Petersen, K.: A palette of lean indicators to detect waste in software maintenance: A case study. In: Agile processes in software engineering and extreme programming, pp. 108–122. Springer (2012)

    Google Scholar 

  359. Petersen, K., Feldt, R., Mujtaba, S., Mattsson, M.: Systematic mapping studies in software engineering. In: 12th International Conference on Evaluation and Assessment in Software Engineering. sn (2008)

    Google Scholar 

  360. Petersen, K., Wohlin, C.: A comparison of issues and advantages in agile and incremental development between state of the art and an industrial case. Journal of Systems and Software 82(9), 1479–1490 (2009). DOI 10.1016/j.jss.2009.03.036

    Google Scholar 

  361. Pfahl, D., Yin, H., Mäntylä, M.V., Münch, J.: How is exploratory testing used? a state-of-the-practice survey. In: Proceedings of the 8th ACM/IEEE international symposium on empirical software engineering and measurement, pp. 1–10 (2014)

    Google Scholar 

  362. Pichler, J., Ramler, R.: How to test the intangible properties of graphical user interfaces? In: 2008 1st International Conference on Software Testing, Verification, and Validation, pp. 494–497. IEEE (2008)

    Google Scholar 

  363. Raappana, P., Saukkoriipi, S., Tervonen, I., Mäntylä, M.V.: The effect of team exploratory testing–experience report from f-secure. In: 2016 IEEE Ninth International Conference on Software Testing, Verification and Validation Workshops (ICSTW), pp. 295–304. IEEE (2016)

    Google Scholar 

  364. Radatz, J., Geraci, A., Katki, F.: Ieee standard glossary of software engineering terminology. IEEE Std 610121990(121990), 3 (1990)

    Google Scholar 

  365. Radjenović, D., Heričko, M., Torkar, R., Živkovič, A.: Software fault prediction metrics: A systematic literature review. Information and Software Technology 55(8), 1397–1418 (2013)

    Google Scholar 

  366. Raj, A., Bosch, J., Olsson, H.H., Wang, T.J.: Modelling data pipelines. In: 2020 46th Euromicro Conference on Software Engineering and Advanced Applications (SEAA), pp. 13–20. IEEE (2020)

    Google Scholar 

  367. Ramasubbu, N., Cataldo, M., Balan, R.K., Herbsleb, J.D.: Configuring global software teams: a multi-company analysis of project productivity, quality, and profits. In: Proceedings of the 33rd International Conference on Software Engineering, pp. 261–270. ACM (2011)

    Google Scholar 

  368. Ramesh, B., Cao, L., Baskerville, R.: Agile requirements engineering practices and challenges: an empirical study. Information Systems Journal 20(5), 449–480 (2010)

    Google Scholar 

  369. Rana, R., Staron, M., Berger, C., Hansson, J., Nilsson, M., Törner, F., Meding, W., Höglund, C.: Selecting software reliability growth models and improving their predictive accuracy using historical projects data. Journal of Systems and Software 98, 59–78 (2014)

    Google Scholar 

  370. Rashmi, N., Suma, V.: Defect detection efficiency of the combined approach. In: ICT and Critical Infrastructure: Proceedings of the 48th Annual Convention of Computer Society of India-Vol II, pp. 485–490. Springer (2014)

    Google Scholar 

  371. Rasmusson, J.: Long build trouble shooting guide. In: Conference on Extreme Programming and Agile Methods, pp. 13–21. Springer (2004)

    Google Scholar 

  372. Reis, J., Mota, A.: Aiding exploratory testing with pruned gui models. Information Processing Letters 133, 49–55 (2018)

    MathSciNet  MATH  Google Scholar 

  373. Ries, E.: The lean startup: How today’s entrepreneurs use continuous innovation to create radically successful businesses. Crown Business Publishing (2011)

    Google Scholar 

  374. Rissanen, O., Münch, J.: Continuous experimentation in the b2b domain: a case study. In: 2015 IEEE/ACM 2nd International Workshop on Rapid Continuous Software Engineering, pp. 12–18. IEEE (2015)

    Google Scholar 

  375. Roberts, M.: Enterprise continuous integration using binary dependencies. In: International Conference on Extreme Programming and Agile Processes in Software Engineering, pp. 194–201. Springer (2004)

    Google Scholar 

  376. Robson, C., McCartan, K.: Real world research. John Wiley & Sons (2016)

    Google Scholar 

  377. Rodden, K., Hutchinson, H., Fu, X.: Measuring the user experience on a large scale: user-centered metrics for web applications. In: Proceedings of the SIGCHI conference on human factors in computing systems, pp. 2395–2398 (2010)

    Google Scholar 

  378. Rodríguez, P., Haghighatkhah, A., Lwakatare, L.E., Teppola, S., Suomalainen, T., Eskeli, J., Karvonen, T., Kuvaja, P., Verner, J.M., Oivo, M.: Continuous deployment of software intensive products and services: A systematic mapping study. Journal of Systems and Software 123, 263–291 (2017)

    Google Scholar 

  379. Rogers, R.O.: Cruisecontrol. net: Continuous integration for. net. In: International Conference on Extreme Programming and Agile Processes in Software Engineering, pp. 114–122. Springer (2003)

    Google Scholar 

  380. Rogers, R.O.: Scaling continuous integration. In: International conference on extreme programming and agile processes in software engineering, pp. 68–76. Springer (2004)

    Google Scholar 

  381. Ruhe, G.: Software engineering decision support–a new paradigm for learning software organizations. In: Advances in Learning Software Organizations, pp. 104–113. Springer (2003)

    Google Scholar 

  382. Ruhe, G., Saliu, M.O.: The art and science of software release planning. Software, IEEE 22(6), 47–53 (2005)

    Google Scholar 

  383. Rumpe, B.: Agile modeling with the uml. In: M. Wirsing, A. Knapp, S. Balsamo (eds.) Radical Innovations of Software and Systems Engineering in the Future, pp. 297–309. Springer Berlin Heidelberg, Berlin, Heidelberg (2004)

    Google Scholar 

  384. Runeson, P., Höst, M.: Guidelines for conducting and reporting case study research in software engineering. Empirical software engineering 14(2), 131–164 (2009)

    Google Scholar 

  385. Runeson, P., Host, M., Rainer, A., Regnell, B.: Case study research in software engineering: Guidelines and examples. John Wiley & Sons (2012)

    Google Scholar 

  386. Runeson, P., Host, M., Rainer, A., Regnell, B.: Case study research in software engineering: Guidelines and examples. John Wiley & Sons (2012)

    Google Scholar 

  387. Salo, O., Abrahamsson, P.: Agile methods in european embedded software development organisations: a survey on the actual use and usefulness of extreme programming and scrum. IET software 2(1), 58–64 (2008)

    Google Scholar 

  388. Sandberg, A., Pareto, L., Arts, T.: Agile collaborative research: Action principles for industry-academia collaboration. Software, IEEE 28(4), 74–83 (2011)

    Google Scholar 

  389. Savolainen, J., Kuusela, J., Vilavaara, A.: Transition to agile development-rediscovery of important requirements engineering practices. In: 2010 18th IEEE International Requirements Engineering Conference, pp. 289–294. IEEE (2010)

    Google Scholar 

  390. Schaefer, C.J., Do, H.: Model-based exploratory testing: a controlled experiment. In: 2014 IEEE Seventh International Conference on Software Testing, Verification and Validation Workshops, pp. 284–293. IEEE (2014)

    Google Scholar 

  391. Schermann, G., Cito, J., Leitner, P., Zdun, U., Gall, H.C.: We’re doing it live: A multi-method empirical study on continuous experimentation. Information and Software Technology 99, 41–57 (2018)

    Google Scholar 

  392. Schmidt, D.C.: Model-driven engineering. IEEE Computer 39(2), 25 (2006)

    Google Scholar 

  393. Schuh, P.: Integrating agile development in the real world. Charles River Media Hingham (2005)

    Google Scholar 

  394. Sculley, D., Holt, G., Golovin, D., Davydov, E., Phillips, T., Ebner, D., Chaudhary, V., Young, M., Crespo, J.F., Dennison, D.: Hidden technical debt in machine learning systems. Advances in neural information processing systems 28, 2503–2511 (2015)

    Google Scholar 

  395. Seaman, C., Guo, Y., Zazworka, N., Shull, F., Izurieta, C., Cai, Y., Vetrò, A.: Using technical debt data in decision making: Potential decision approaches. In: 2012 Third International Workshop on Managing Technical Debt (MTD), pp. 45–48. IEEE (2012)

    Google Scholar 

  396. Sedano, T., Ralph, P., Praire, C.: The product backlog. In: 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE), pp. 200–211 (2019). DOI 10.1109/ICSE.2019.00036

    Google Scholar 

  397. Sehmi, A., Jones, N., Wang, S., Loudon, G.: Knowledge-based systems for neuroelectric signal processing. IEE Proceedings-Science, Measurement and Technology 141(3), 215–23 (2003)

    Google Scholar 

  398. Senin, P., Lin, J., Wang, X., Oates, T., Gandhi, S., Boedihardjo, A.P., Chen, C., Frankenstein, S., Lerner, M.: Grammarviz 2.0: a tool for grammar-based pattern discovery in time series. In: Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 468–472. Springer (2014)

    Google Scholar 

  399. Shadish, W.R., Cook, T.D., Campbell, D.T., et al.: Experimental and quasi-experimental designs for generalized causal inference/William R. Shedish, Thomas D. Cook, Donald T. Campbell. Boston: Houghton Mifflin, (2002)

    Google Scholar 

  400. Shah, A., Kerzhner, A., Schaefer, D., Paredis, C.: Multi-view Modeling to Support Embedded Systems Engineering in SysML. In: Graph transformations and model-driven engineering, pp. 580–601. Springer (2010). DOI 10.1007/978-3-642-17322-6_25

    Google Scholar 

  401. Shah, S.M.A., Gencel, C., Alvi, U.S., Petersen, K.: Towards a hybrid testing process unifying exploratory testing and scripted testing. Journal of software: Evolution and Process 26(2), 220–250 (2014)

    Google Scholar 

  402. Shah, S.M.A., Torchiano, M., Vetrò, A., Morisio, M.: Exploratory testing as a source of technical debt. IT Professional 16(3), 44–51 (2013)

    Google Scholar 

  403. Shahnewaz, S., Ruhe, G.: Relrea-an analytical approch for evaluating release readiness. In: SEKE (2014)

    Google Scholar 

  404. Shalloway, A., Beaver, G., Trott, J.R.: Lean-agile software development: achieving enterprise agility. Pearson Education (2009)

    Google Scholar 

  405. Shaukat, R., Shahoor, A., Urooj, A.: Probing into code analysis tools: A comparison of c# supporting static code analyzers. In: Applied Sciences and Technology (IBCAST), 2018 15th International Bhurban Conference on, pp. 455–464. IEEE (2018)

    Google Scholar 

  406. Shen, M., Yang, W., Rong, G., Shao, D.: Applying agile methods to embedded software development: A systematic review. In: Proceedings of the International Workshop on Software Engineering for Embedded Systems, pp. 30–36. IEEE (2012). DOI 10.1109/SEES.2012.6225488

    Google Scholar 

  407. Shoaib, L., Nadeem, A., Akbar, A.: An empirical evaluation of the influence of human personality on exploratory software testing. In: 2009 IEEE 13th International Multitopic Conference, pp. 1–6. IEEE (2009)

    Google Scholar 

  408. Shull, F., Singer, J., Sjøberg, D.I.K. (eds.): Guide to Advanced Empirical Software Engineering. Springer London, London (2008). DOI 10.1007/978-1-84800-044-5. URL http://www.springerlink.com/index/10.1007/978-1-84800-044-5

    Book  Google Scholar 

  409. Silhavy, P., Silhavy, R., Prokopova, Z.: Categorical variable segmentation model for software development effort estimation. IEEE Access 7, 9618–9626 (2019). DOI 10.1109/ACCESS.2019.2891878

    Google Scholar 

  410. Silhavy, R., Silhavy, P., Prokopova, Z.: Improving algorithmic optimisation method by spectral clustering. In: Computer Science On-line Conference, pp. 1–10. Springer (2017)

    Google Scholar 

  411. Silhavy, R., Silhavy, P., Prokopová, Z.: Evaluating subset selection methods for use case points estimation. Information and Software Technology 97, 1–9 (2018)

    Google Scholar 

  412. Singh, D., Sekar, V.R., Stolee, K.T., Johnson, B.: Evaluating how static analysis tools can reduce code review effort. In: Visual Languages and Human-Centric Computing (VL/HCC), 2017 IEEE Symposium on, pp. 101–105. IEEE (2017)

    Google Scholar 

  413. Sinnema, M., Deelstra, S., Nijhuis, J., Bosch, J.: Covamof: A framework for modeling variability in software product families. In: International Conference on Software Product Lines, pp. 197–213. Springer (2004)

    Google Scholar 

  414. Smit, M., Gergel, B., Hoover, H.J., Stroulia, E.: Maintainability and source code conventions: An analysis of open source projects. University of Alberta, Department of Computing Science, Tech. Rep. TR11-06 (2011)

    Google Scholar 

  415. Sommerville, I.: Software engineering. 6th. Ed., Harlow, UK.: Addison-Wesley (2001)

    Google Scholar 

  416. Sommerville, I.: Software Engineering, 10th edn. Pearson (2015)

    Google Scholar 

  417. Sorrell, S., et al.: Digitalisation of goods: a systematic review of the determinants and magnitude of the impacts on energy consumption. Environmental Research Letters 15(4), 043001 (2020)

    Google Scholar 

  418. Ståhl, D., Bosch, J.: Experienced benefits of continuous integration in industry software product development: A case study. In: The 12th iasted international conference on software engineering,(innsbruck, austria, 2013), pp. 736–743 (2013)

    Google Scholar 

  419. Ståhl, D., Bosch, J.: Continuous integration flows. In: Continuous software engineering, pp. 107–115. Springer (2014)

    Google Scholar 

  420. Ståhl, D., Bosch, J.: Modeling continuous integration practice differences in industry software development. Journal of Systems and Software 87, 48–59 (2014)

    Google Scholar 

  421. Ståhl, D., Bosch, J.: Industry application of continuous integration modeling: a multiple-case study. In: 2016 IEEE/ACM 38th International Conference on Software Engineering Companion (ICSE-C), pp. 270–279. IEEE (2016)

    Google Scholar 

  422. Ståhl, D., Bosch, J.: Cinders: The continuous integration and delivery architecture framework. Information and Software Technology 83, 76–93 (2017)

    Google Scholar 

  423. Ståhl, D., Hallén, K., Bosch, J.: Achieving traceability in large scale continuous integration and delivery deployment, usage and validation of the eiffel framework. Empirical Software Engineering 22(3), 967–995 (2017)

    Google Scholar 

  424. Stahl, D., Martensson, T., Bosch, J.: Continuous practices and devops: beyond the buzz, what does it all mean? In: 2017 43rd Euromicro Conference on Software Engineering and Advanced Applications (SEAA), pp. 440–448. IEEE (2017)

    Google Scholar 

  425. Staron, M.: Critical role of measures in decision processes: Managerial and technical measures in the context of large software development organizations. Information and Software Technology 54(8), 887–899 (2012)

    Google Scholar 

  426. Staron, M.: Software complexity metrics in general and in the context of ISO 26262 software verification requirements. In: Scandinavian Conference on Systems Safety. http://gup.ub.gu.se/records/fulltext/233026/233026.pdf(2016)

  427. Staron, M.: Action Research in Software Engineering. Springer (2020)

    Google Scholar 

  428. Staron, M., Hansson, J., Feldt, R., Henriksson, A., Meding, W., Nilsson, S., Hoglund, C.: Measuring and visualizing code stability–a case study at three companies. In: Software Measurement and the 2013 Eighth International Conference on Software Process and Product Measurement (IWSM-MENSURA), 2013 Joint Conference of the 23rd International Workshop on, pp. 191–200. IEEE (2013)

    Google Scholar 

  429. Staron, M., Meding, W.: Predicting short-term defect inflow in large software projects–an initial evaluation. 11th International Conference on Evaluation and Assessment in Software Engineering, EASE (2007)

    Google Scholar 

  430. Staron, M., Meding, W.: Predicting weekly defect inflow in large software projects based on project planning and test status. Information and Software Technology p. (available online) (2007)

    Google Scholar 

  431. Staron, M., Meding, W.: Ensuring reliability of information provided by measurement systems. In: Proceedings of the International Conferences on Software Process and Product Measurement. Springer Berlin / Heidelberg (2009)

    Google Scholar 

  432. Staron, M., Meding, W.: Factors determining long-term success of a measurement program: an industrial case study. e-Informatica Software Engineering Journal pp. 7–23 (2011)

    Google Scholar 

  433. Staron, M., Meding, W.: Software Development Measurement Programs: Development, Management and Evolution. Springer (2018)

    Google Scholar 

  434. Staron, M., Meding, W., Caiman, M.: Improving completeness of measurement systems for monitoring software development workflows. In: Software Quality. Increasing Value in Software and Systems Development, pp. 230–243. Springer (2013)

    Google Scholar 

  435. Staron, M., Meding, W., Hansson, J., Höglund, C., Niesel, K., Bergmann, V.: Dashboards for continuous monitoring of quality for software product under development. System Qualities and Software Architecture (SQSA) (2013)

    Google Scholar 

  436. Staron, M., Meding, W., Karlsson, G., Nilsson, C.: Developing measurement systems: an industrial case study. Journal of Software Maintenance and Evolution: Research and Practice 23(2), 89–107 (2011)

    Google Scholar 

  437. Staron, M., Meding, W., Nilsson, C.: A framework for developing measurement systems and its industrial evaluation. Information and Software Technology 51(4), 721–737 (2008)

    Google Scholar 

  438. Staron, M., Meding, W., Palm, K.: Release readiness indicator for mature agile and lean software development projects. In: Agile Processes in Software Engineering and Extreme Programming, pp. 93–107. Springer (2012)

    Google Scholar 

  439. Staron, M., Meding, W., Söderqvist, B.: A method for forecasting defect backlog in large streamline software development projects and its industrial evaluation. Information and Software Technology 52(10), 1069–1079 (2010)

    Google Scholar 

  440. Staron, M., Ochodek, M., Meding, W., Söder, O., Rosenberg, E.: Machine learning to support code reviews in continuous integration. In: Artificial Intelligence Methods For Software Engineering, pp. 141–167. World Scientific (2021)

    Google Scholar 

  441. Steghöfer, J.P., Knauss, E., Horkoff, J., Wohlrab, R.: Challenges of scaled agile for safety-critical systems. In: X. Franch, T. Männistö, S. Martínez-Fernández (eds.) Product-Focused Software Process Improvement, pp. 350–366. Springer International Publishing, Cham (2019)

    Google Scholar 

  442. Stolberg, S.: Enabling agile testing through continuous integration. In: 2009 agile conference, pp. 369–374 (2009)

    Google Scholar 

  443. Sturdevant, K.F.: Cruisin’and chillin’: Testing the java-based distributed ground data system” chill” with cruisecontrol system” chill” with cruisecontrol. In: 2007 IEEE Aerospace Conference, pp. 1–8. IEEE (2007)

    Google Scholar 

  444. Subramanyam, R., Krishnan, M.S.: Empirical analysis of ck metrics for object-oriented design complexity: Implications for software defects. Software Engineering, IEEE Transactions on 29(4), 297–310 (2003)

    Google Scholar 

  445. Sunindyo, W.D., Moser, T., Winkler, D., Biffl, S.: Foundations for event-based process analysis in heterogeneous software engineering environments. In: 2010 36th EUROMICRO Conference on Software Engineering and Advanced Applications, pp. 313–322. IEEE (2010)

    Google Scholar 

  446. Suryadevara, J., Tiwari, S.: Adopting MBSE in Construction Equipment Industry: An Experience Report. In: 25th Asia-Pacific Software Engineering Conference APSEC (2018). DOI 10.1109/apsec.2018.00066

    Google Scholar 

  447. Susman, G., Evered, R.: An Assessment of the Scientific Merits of Action Research. Journal of Administrative Science Quarterly 23(4), 582–603 (1978)

    Google Scholar 

  448. Susman, G.I.: Action research: a sociotechnical systems perspective. Beyond method: Strategies for social research pp. 95–113 (1983)

    Google Scholar 

  449. Susman, G.I., Evered, R.D.: An assessment of the scientific merits of action research. Administrative science quarterly pp. 582–603 (1978)

    Google Scholar 

  450. Sutherland, J., Frohman, R.: Hitting the wall: What to do when high performing scrum teams overwhelm operations and infrastructure. In: 2011 44th Hawaii International Conference on System Sciences, pp. 1–6. IEEE (2011)

    Google Scholar 

  451. Sviridova, T., Stakhova, D., Marikutsa, U.: Exploratory testing: Management solution. In: 2013 12th International Conference on the Experience of Designing and Application of CAD Systems in Microelectronics (CADSM), pp. 361–361. IEEE (2013)

    Google Scholar 

  452. Tamburri, D.A., Kruchten, P., Lago, P., van Vliet, H.: What is social debt in software engineering? In: 2013 6th International Workshop on Cooperative and Human Aspects of Software Engineering (CHASE), pp. 93–96. IEEE (2013)

    Google Scholar 

  453. Tang, D., Agarwal, A., O’Brien, D., Meyer, M.: Overlapping experiment infrastructure: More, better, faster experimentation. In: Proceedings of the 16th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 17–26 (2010)

    Google Scholar 

  454. Tingling, P., Saeed, A.: Extreme programming in action: a longitudinal case study. In: International Conference on Human-Computer Interaction, pp. 242–251. Springer (2007)

    Google Scholar 

  455. Tom, E., Aurum, A., Vidgen, R.: An exploration of technical debt. Journal of Systems and Software 86(6), 1498–1516 (2013)

    Google Scholar 

  456. Torunski, E., Shafiq, M.O., Whitehead, A.: Code style analytics for the automatic setting of formatting rules in ides: A solution to the tabs vs. spaces debate. In: Digital Information Management (ICDIM), 2017 Twelfth International Conference on, pp. 6–14. IEEE (2017)

    Google Scholar 

  457. Tosun, A., Turhan, B., Bener, A.: Practical considerations in deploying ai for defect prediction: a case study within the turkish telecommunication industry. In: Proceedings of the 5th International Conference on Predictor Models in Software Engineering, pp. 1–9 (2009)

    Google Scholar 

  458. Trist, E.: The evolution of socio-technical systems. Occasional paper 2, 1981 (1981)

    Google Scholar 

  459. Tsai, W., Heisler, K., Volovik, D., Zualkernan, I.: A critical look at the relationship between ai and software engineering. In: [Proceedings] 1988 IEEE Workshop on Languages for Automation@ m_Symbiotic and Intelligent Robotics, pp. 2–18. IEEE (1988)

    Google Scholar 

  460. Tuomikoski, J., Tervonen, I.: Absorbing software testing into the scrum method. In: International Conference on Product-Focused Software Process Improvement, pp. 199–215. Springer (2009)

    Google Scholar 

  461. Uludag, Ö., Kleehaus, M., Caprano, C., Matthes, F.: Identifying and structuring challenges in large-scale agile development based on a structured literature review. In: 2018 IEEE 22nd International Enterprise Distributed Object Computing Conference (EDOC), pp. 191–197. IEEE (2018)

    Google Scholar 

  462. Umarji, M., Emurian, H.: Acceptance issues in metrics program implementation. In: H. Emurian (ed.) 11th IEEE International Symposium Software Metrics, pp. 10–17 (2005)

    Google Scholar 

  463. Unterkalmsteiner, M., Gorschek, T., Islam, A., Cheng, C.K., Permadi, R.B., Feldt, R.: A conceptual framework for spi evaluation. Journal of Software: Evolution and Process 26(2), 251–279 (2014)

    Google Scholar 

  464. Unterkalmsteiner, M., Gorschek, T., Islam, A.M., Cheng, C.K., Permadi, R.B., Feldt, R.: Evaluation and measurement of software process improvement—a systematic literature review. Software Engineering, IEEE Transactions on 38(2), 398–424 (2012)

    Google Scholar 

  465. Van Der Linden, F., Bosch, J., Kamsties, E., Känsälä, K., Obbink, H.: Software product family evaluation. In: International Conference on Software Product Lines, pp. 110–129. Springer (2004)

    Google Scholar 

  466. Van Der Storm, T.: Continuous release and upgrade of component-based software. In: Proceedings of the 12th international workshop on Software configuration management, pp. 43–57 (2005)

    Google Scholar 

  467. Van Der Storm, T.: The sisyphus continuous integration system. In: 11th European Conference on Software Maintenance and Reengineering (CSMR’07), pp. 335–336. IEEE (2007)

    Google Scholar 

  468. Van Der Storm, T.: Backtracking incremental continuous integration. In: 2008 12th European Conference on Software Maintenance and Reengineering, pp. 233–242. IEEE (2008)

    Google Scholar 

  469. Van Nostrand, R.C.: Design of experiments using the taguchi approach: 16 steps to product and process improvement (2002)

    Google Scholar 

  470. Vidgen, R., Wang, X.: Coevolving systems and the organization of agile software development. Information Systems Research 20(3), 355–376 (2009)

    Google Scholar 

  471. van Waardenburg, G., van Vliet, H.: When agile meets the enterprise. Information and Software Technology 55(12), 2154–2171 (2013). DOI 10.1016/j.infsof.2013.07.012. URL http://www.sciencedirect.com/science/article/pii/S0950584913001584

    Article  Google Scholar 

  472. Walsham, G.: Interpretive case studies in is research: nature and method. European Journal of information systems 4(2), 74–81 (1995)

    Google Scholar 

  473. Watanabe, W.M., Fortes, R.P., Dias, A.L.: Using acceptance tests to validate accessibility requirements in ria. In: Proceedings of the International Cross-Disciplinary Conference on Web Accessibility, pp. 1–10 (2012)

    Google Scholar 

  474. Weippl, E.R.: Security in data warehouses. IGI Global, Data Ware-housing Design and Advanced Engineering Applications (2010)

    Google Scholar 

  475. Westerman, G., Tannou, M., Bonnet, D., Ferraris, P., McAfee, A.: The digital advantage: How digital leaders outperform their peers in every industry. MITSloan Management and Capgemini Consulting, MA 2, 2–23 (2012)

    Google Scholar 

  476. Weyuker, E.J.: Evaluating software complexity measures. Software Engineering, IEEE Transactions on 14(9), 1357–1365 (1988)

    MathSciNet  Google Scholar 

  477. Whittaker, J.A.: Exploratory software testing: tips, tricks, tours, and techniques to guide test design. Pearson Education (2009)

    Google Scholar 

  478. Wieringa, R., Daneva, M.: Six strategies for generalizing software engineering theories. Science of computer programming 101, 136–152 (2015)

    Google Scholar 

  479. Wiklund, K., Sundmark, D., Eldh, S., Lundqvist, K.: Impediments in agile software development: An empirical investigation. In: International Conference on Product Focused Software Process Improvement, pp. 35–49. Springer (2013)

    Google Scholar 

  480. Williams, L., Cockburn, A.: Agile software development: it’s about feedback and change. IEEE computer 36(6), 39–43 (2003)

    Google Scholar 

  481. Wisell, D., Stenvard, P., Hansebacke, A., Keskitalo, N.: Considerations when designing and using virtual instruments as building blocks in flexible measurement system solutions. In: P. Stenvard (ed.) IEEE Instrumentation and Measurement Technology Conference, pp. 1–5 (2007)

    Google Scholar 

  482. Wohlin, C., Aurum, A., Angelis, L., Phillips, L., Dittrich, Y., Gorschek, T., Grahn, H., Henningsson, K., Kagstrom, S., Low, G., et al.: The success factors powering industry-academia collaboration. IEEE software 29(2), 67–73 (2012)

    Google Scholar 

  483. Wohlin, C., Runeson, P., Host, M., Ohlsson, M.C., Regnell, B., Wessln, A.: Experimentation in Software Engineering: An Introduction. Kluwer Academic Publisher, Boston MA (2000)

    MATH  Google Scholar 

  484. Wohlrab, R., Knauss, E., Pelliccione, P.: Why and how to balance alignment and diversity of requirements engineering practices in automotive. Journal of Systems and Software 162, 110516 (2020). DOI https://doi.org/10.1016/j.jss.2019.110516. URL https://www.sciencedirect.com/science/article/pii/S0164121219302900

  485. Wohlrab, R., Pelliccione, P., Knauss, E., Larsson, M.: Boundary objects and their use in agile systems engineering. J. Softw. Evol. Process. 31(5) (2019)

    Google Scholar 

  486. Wood, W., Tam, L., Witt, M.G.: Changing circumstances, disrupting habits. Journal of personality and social psychology 88(6), 918 (2005)

    Google Scholar 

  487. Woskowski, C.: Applying industrial-strength testing techniques to critical care medical equipment. In: International Conference on Computer Safety, Reliability, and Security, pp. 62–73. Springer (2012)

    Google Scholar 

  488. Xenos, M., Christodoulakis, D.: Measuring perceived software quality. Information and software technology 39(6), 417–424 (1997)

    Google Scholar 

  489. Yaman, S.G., Fagerholm, F., Munezero, M., Münch, J., Aaltola, M., Palmu, C., Männistö, T.: Transitioning towards continuous experimentation in a large software product and service development organisation–a case study. In: International Conference on Product-Focused Software Process Improvement, pp. 344–359. Springer (2016)

    Google Scholar 

  490. Yaman, S.G., Munezero, M., Münch, J., Fagerholm, F., Syd, O., Aaltola, M., Palmu, C., Männistö, T.: Introducing continuous experimentation in large software-intensive product and service organisations. Journal of Systems and Software 133, 195–211 (2017)

    Google Scholar 

  491. Yin, R.K.: Case study research design and methods third edition. Applied social research methods series 5 (2003)

    Google Scholar 

  492. Yin, R.K.: Case study research and applications: Design and methods. Sage publications (2017)

    Google Scholar 

  493. Yli-Huumo, J., Maglyas, A., Smolander, K.: How do software development teams manage technical debt?–an empirical study. Journal of Systems and Software 120, 195–218 (2016)

    Google Scholar 

  494. Yuan, D., Park, S., Zhou, Y.: Characterizing logging practices in open-source software. In: 2012 34th International Conference on Software Engineering (ICSE), pp. 102–112. IEEE (2012)

    Google Scholar 

  495. Yuksel, H.M., Tuzun, E., Gelirli, E., Biyikli, E., Baykal, B.: Using continuous integration and automated test techniques for a robust c4isr system. In: 2009 24th International Symposium on Computer and Information Sciences, pp. 743–748. IEEE (2009)

    Google Scholar 

  496. Zaborovsky, A.N., Danilov, D.O., Leonov, G.V., Mescheriakov, R.V.: Software and hardware for measurements systems. In: D.O. Danilov (ed.) The IEEE-Siberian Conference on Electron Devices and Materials, pp. 53–57. IEEE (2007)

    Google Scholar 

  497. Zazworka, N., Spínola, R.O., Vetro’, A., Shull, F., Seaman, C.: A case study on effectively identifying technical debt. In: Proceedings of the 17th International Conference on Evaluation and Assessment in Software Engineering, pp. 42–47 (2013)

    Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Editor information

Editors and Affiliations

Rights and permissions

Reprints and permissions

Copyright information

© 2017 IEEE

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Fabijan, A., Dmitriev, P., Bosch, J., Holmström Olsson, H. (2017). Chapter 12 The Evolution of Continuous Experimentation in Software Product Development: From Data to a Data-Driven Organization at Scale. In: Bosch, J., Carlson, J., Holmström Olsson, H., Sandahl, K., Staron, M. (eds) Accelerating Digital Transformation. Springer, Cham. https://doi.org/10.1007/978-3-031-10873-0_16

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-10873-0_16

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-10872-3

  • Online ISBN: 978-3-031-10873-0

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics