Abstract
The users or deployment engineers of a software system can adapt such a system to a wide range of deployment and usage scenarios by changing the value of configuration options, for example by disabling unnecessary features, tweaking performance-related parameters or specifying GUI preferences. However, the literature agrees that the flexibility of such options comes at a price: misconfigured options can lead a software system to crash in the production environment, while even in the absence of such configuration errors, a large number of configuration options makes a software system more complicated to deploy and use. In earlier work, we also found that developers who intend to make their application configurable face 22 challenges that impact their configuration engineering activities, ranging from technical to management-related or even inherent to the domain of configuration engineering. In this paper, we use a prototyping approach to derive and empirically evaluate requirements for tool support able to deal with 13 (primarily technical) configuration engineering challenges. In particular, via a set of interviews with domain experts, we identify four requirements by soliciting feedback on an incrementally evolving prototype. The resulting “Config2Code” prototype, which implements the four requirements, is then empirically evaluated via a user study involving 55 participants that comprises 10 typical configuration engineering tasks, ranging from the creation, comprehension, refactoring, and reviewing of configuration options to the quality assurance of options and debugging of configuration failures. A configuration framework satisfying the four requirements enables developers to perform more accurately and more swiftly in 70% and 60% (respectively) of the configuration engineering tasks than a state-of-the-practice framework not satisfying the requirements. Furthermore, such a framework allows to reduce the time taken for these tasks by up to 94.62%, being slower for only one task.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Ansible (2018) Core java preferences api, http://www.ansible.com/. Accessed March 08, 2018
Apel S, Batory D, Kästner C, Saake G (2013) Feature-oriented software product lines: Concepts and implementation. Springer, Berlin/Heidelberg, p 308. ISBN 978-3-642-37520-0. http://www.springer.com/computer/swe/book/978-3-642-37520-0, Tech Rep
Bass L, Weber I, Zhu L (2015) Devops: a software architect’s perspective, 1st edn. Addison-Wesley Professional, Reading
Behrang F, Cohen MB, Orso A (2015) Users beware: Preference inconsistencies ahead. In: Proceedings of the 10th joint meeting on foundations of software engineering, ser. ESEC/FSE’15, New York, NY, USA, pp 295–306
Buse RP, Sadowski C, Weimer W (2011) Benefits and barriers of user evaluation in software engineering research. In: Proceedings of the International Conference on Object Oriented Programming Systems Languages and Applications, ser OOPSLA’11, pp 643–656
Cassoli J (2016) Web application with spring annotation-driven configuration: Rapidly develop lightweight Java web applications using Spring with annotations, 1st edn. CreateSpace Independent Publishing Platform, Scotts Valley
Dong Z, Andrzejak A, Shao K (2015) Practical and accurate pinpointing of configuration errors using static analysis, Bremen, Germany, automated debugging;Configuration errors;Configuration options;Correlation degree;Misconfigurations;Run-time information;Software program;Testing oracles;. Online. Available https://doi.org/10.1109/ICSM.2015.7332463
Dong Z, Andrzejak A, Lo D, Costa D (2016) Orplocator: Identifying read points of configuration options via static analysis. In: Proceedings of the 27th international symposium on software reliability engineering, ser. ISSRE’16, pp 185–195
Huang P, Bolosky WJ, Singh A, Zhou Y (2015) Confvalley: a systematic configuration validation framework for cloud services. In: Proceedings of the tenth European conference on computer systems, ser. Eurosys ’15, NY, USA, pp 1–16
Hubaux A, Xiong Y, Czarnecki K (2012) A user survey of configuration challenges in linux and ecos. In: Proceedings of the 6th Int’l workshop on variability modeling of software-intensive systems, pp 149–155
Humble J, Farley D (2010) Continuous delivery: reliable software releases through build, test, and deployment automation, 1st edn. Addison-Wesley Professional, Reading
Jiang Y, Adams B (2015) Co-evolution of infrastructure and source code: an empirical study. In: Proceedings of the 12th Conference on Mining Software Repositories, ser. MSR’15, pp 45–55
Jin D, Qu X, Cohen MB, Robinson B (2014) Configurations everywhere: Implications for testing and debugging in practice. In: Proceedings of the 36th international conference on soft. eng., ser. ICSE’14, pp 215–224
Jin D, Cohen MB, Qu X, Robinson B (2014) Preffinder: Getting the right preference in configurable software systems. In: Proceedings of the 29th international conference on automated software engineering, ser. ASE ’14, pp 151–162
Khan M, Huang Z, Li M, Taylor GA, Khan M (2017) Optimizing hadoop parameter settings with gene expression programming guided pso. Concurrency and Computation: Practice and Experience 29(3):186–197
Ko AJ, LaToza T, Burnett M (2015) A practical guide to controlled experiments of software engineering tools with human participants. Empir Softw Eng 20(1):110–141
Krum S, Van Hevelingen W, Kero B, Turnbull J, McCune J (2013) Pro Puppet, 2nd edn. Apress, New York
Li W, Li S, Liao X, Xu X, Zhou S, Jia Z (2017) Conftest: Generating comprehensive misconfiguration for system reaction ability evaluation. In: Proceedings of the 21st international conference on evaluation and assessment in software engineering, ser. EASE’17, NY, USA, pp 88–97
van der Linden F, Schmid K, Rommes E (2007) Software product lines in action: the best industrial practice in product line engineering. Springer, Berlin. Online. Available: https://books.google.ca/books?id=PC4LyoSNNAkC
Lwakatare LE, Kuvaja P, Oivo M (2016) Relationship of devops to agile, lean and continuous deployment. In: Abrahamsson P., Jedlitschka A., Nguyen Duc A., Felderer M., Amasaki S., Mikkonen T. (eds) Product-focused software process improvement. Springer International Publishing, Cham, pp 399–415
Medeiros F, Kästner C, Ribeiro M, Gheyi R, Apel S (2016) A comparison of 10 sampling algorithms for configurable systems. In: Proceedings of the 38th international conference on software engineering, ser. ICSE ’16. Online. Available: https://doi.org/10.1145/2884781.2884793. ACM, New York, pp 643–654
Meng X, Foong PS, Perrault S, Zhao S (2016) 5-step approach to designing controlled experiments. In: Proceedings of the international working conference on advanced visual interfaces, pp 358–359
Morris K (2016) Infrastructure as code: managing servers in the cloud, 1st edn. O’Reilly Media, Sebastopol
Nadi S, Berger T, Kästner C, Czarnecki K (2014) Mining configuration constraints: Static analyses and empirical results. In: Proceedings of the 36th international conference on Soft. Eng., ser. ICSE’14, pp 140–151
Nadi S, Berger T, Kästner C, Czarnecki K (2015) Where do configuration constraints stem from? an extraction approach and an empirical study. IEEE Trans Softw Eng 41(8):820–841
Pohl K, Böckle G, van der Linden F (2005) Software product line engineering: foundations, principles and techniques. Springer, Berlin. Online. Available: https://books.google.ca/books?id=lsX8_O_TRkEC
Qu X, Acharya M, Robinson B (2011) Impact analysis of configuration changes for test case selection, ser. ISSRE’11, Hiroshima, Japan, pp 140–149
Rabkin A, Katz R (2011) Precomputing possible configuration error diagnoses, ser. ASE’11, Lawrence, KS, United States, pp 193–202
Rabiser R, Dhungana D (2007) Integrated support for product configuration and requirements engineering in product derivation, ser. EUROMICRO’07, Piscataway, USA, pp 193–200
Rahman MT, Querel L-P, Rigby PC, Adams B (2016) Feature toggles: a case study and survey. In: Proceedings of the 13th IEEE working conference on mining software repositories (MSR), Austin, TX, pp 201–211
Sarma A, Bortis G, van der Hoek A (2007) Towards supporting awareness of indirect conflicts across software configuration management workspaces. In: Proceedings of the 22 international conference on automated software engineering, ser. ASE’07, pp 94–103
Sayagh M, Adams B (2018) Videos, http://mcis.polymtl.ca/msayagh/userStudy/Config2Code/
Sayagh M, Kerzazi N, Adams B (2017a) On cross-stack configuration errors. In: Proceedings of the 39th international conference on software engineering, ser. ICSE ’17. IEEE Press, Piscataway, pp 255–265
Sayagh M, Dong Z, Andrzejak A, Adams B (2017b) Does the choice of configuration framework matter for developers? empirical study on 11 java configuration frameworks. In: Proceedings of the 17th international working conference on source code analysis and manipulation, ser. SCAM’17, pp 41–50
Sayagh M, Kerzazi N, Adams B, Petrillo F (2018) Software configuration engineering in practice - interviews, survey, and systematic literature review. Transactions on Software Engineering (TSE). Online. Available: mcis.polymtl.ca/publications/2018/tse_mohammed.pdf
Sedlmair M, Meyer M, Munzner T (2012) Design study methodology: Reflections from the trenches and the stacks. IEEE Trans Vis Comput Graphics 18 (12):2431–2440. Online. Available:
Sharma T, Fragkoulis M, Spinellis D (2016) Does your configuration code smell?. In: Proceedings of the 13th international conference on mining software repositories, ser. MSR’16, pp 189–200
Sommerville I (2010) Software engineering, 9th edn. Addison-Wesley Publishing Company, USA
Wagner S, Fernández DM (2015) Analysing text in software projects. arXiv:1612.00164
Wettel R, Lanza M, Robbes R (2011) Software systems as cities: a controlled experiment. In: 2011 33rd international conference on software engineering (ICSE), pp 551–560
Wohlin C, Runeson P, Höst M, Ohlsson MC, Regnell B, Wesslén A (2000) Experimentation in software engineering: an introduction. Kluwer Academic Publishers, Norwell
Xu T, Zhou Y (2015) Systems approaches to tackling configuration errors: a survey. ACM Computing Surveys 47(4):70:1–70:41
Xu T, Jin L, Fan X, Zhou Y, Pasupathy S, Talwadker R (2015) Hey, you have given me too many knobs!: Understanding and dealing with over-designed configuration in system software. In: Proceedings of the 10th joint meeting on foundations of software engineering, ser. ESEC/FSE’2015, NY, USA, pp 307–319
Xu T, Zhang J, Huang P, Zheng J, Sheng T, Yuan D, Zhou Y, Pasupathy S (2013) Do not blame users for misconfigurations. In: Proceedings of the 24 symposium on operating systems principles, pp 244–259
Yin Z, Ma X, Zheng J, Zhou Y, Bairavasundaram LN, Pasupathy S (2011) An empirical study on configuration errors in commercial and open source systems. In: Proceedings of the twenty-third symposium on operating systems principles, ser. SOSP’11, pp 159–172
Zhang S, Ernst M (2013) Automated diagnosis of software configuration errors. Piscataway, NJ, USA, pp 312–21, software configuration error diagnosis;ConfDiagnoser;configuration error identification;static analysis;dynamic profiling;statistical analysis;noncrashing configuration errors;configurable software system;Java;. Online. Available: https://doi.org/10.1109/ICSE.2013.6606577
Zhang S, Ernst M (2014) Which configuration option should i change?. In: Proceedings of the 36th international conference on software engineering, ser. ICSE’14, NY, USA, pp 152–163
Author information
Authors and Affiliations
Corresponding author
Additional information
Communicated by: Romain Robbes
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Sayagh, M., Kerzazi, N., Petrillo, F. et al. What should your run-time configuration framework do to help developers?. Empir Software Eng 25, 1259–1293 (2020). https://doi.org/10.1007/s10664-019-09790-x
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10664-019-09790-x