More Web Proxy on the site http://driver.im/

research-article

Model-based testing in practice: : An experience report from the web applications domain

Authors:

Alper Buğra Keleş,

Zeynep Özdemir Güler,

Andrea ArcuriAuthors Info & Claims

Volume 180, Issue C

https://doi.org/10.1016/j.jss.2021.111032

Published: 01 October 2021 Publication History

Abstract

In the context of a software testing company, we have deployed the model-based testing (MBT) approach to take the company’s test automation practices to higher levels of maturity and capability. We have chosen, from a set of open-source/commercial MBT tools, an open-source tool named GraphWalker, and have pragmatically used MBT for end-to-end test automation of several large web and mobile applications under test. The MBT approach has provided, so far in our project, various tangible and intangible benefits in terms of improved test coverage (number of paths tested), improved test-design practices, and also improved real-fault detection effectiveness. The goal of this experience report (applied research report), done based on “action research”, is to share our experience of applying and evaluating MBT as a software technology (technique and tool) in a real industrial setting. We aim at contributing to the body of empirical evidence in industrial application of MBT by sharing our industry-academia project on applying MBT in practice, the insights that we have gained, and the challenges and questions that we have faced and tackled so far. We discuss an overview of the industrial setting, provide motivation, explain the events leading to the outcomes, discuss the challenges faced, summarize the outcomes, and conclude with lessons learned, take-away messages, and practical advices based on the described experience. By learning from the best practices in this paper, other test engineers could conduct more mature MBT in their test projects.

Highlights

•

We deployed model-based testing (MBT) to increase the maturity of test practices.

•

MBT has provided various tangible and intangible benefits.

•

Including MBT in the Continuous Integration (CI) pipeline was a good practice.

•

This work contributes to the empirical evidence body in application of MBT.

References

[1]

Arcuri A., An experience report on applying software testing academic results in industry: we need usable automated test generation, Empir. Softw. Eng. (2017),.

Digital Library

[2]

Artho C.V., et al., Modbat: A model-based API tester for event-driven systems, in: Haifa Verification Conference, Springer, 2013, pp. 112–128,.

[3]

Belinfante A., Jtorx: A tool for on-line model-driven test derivation and execution, in: International Conference on tools and Algorithms for the Construction and Analysis of Systems, Springer, 2010, pp. 266–270,.

Digital Library

[4]

Biffl S., Aurum A., Boehm B., Erdogmus H., Grünbacher P., Value-Based Software Engineering, Springer, 2006,.

[5]

Blackburn M.R., Busser R.D., Nauman A.M., Morgan T.R., Model-based testing in practice, in: INFORMATIK 2006–Informatik Für Menschen–Band 2, in: Beiträge der 36, Jahrestagung der Gesellschaft für Informatik eV (GI), 2006,.

Digital Library

[6]

Blome A., Ochoa M., Li K., Peroli M., Dashti M.T., Vera: A flexible model-based vulnerability testing tool, in: IEEE International Conference on Software Testing, Verification and Validation, IEEE, 2013, pp. 471–478,.

Digital Library

[7]

Borjesson E., Feldt R., Automated system testing using visual gui testing tools: A comparative study in industry, in: IEEE International Conference on Software Testing, Verification and Validation, IEEE, 2012, pp. 350–359,.

Digital Library

[8]

Clarke P., O’Connor R.V., Leavy B., A complexity theory viewpoint on the software development process and situational context, in: Proceedings of the International Conference on Software and Systems Process, 2016, pp. 86–90,.

Digital Library

[9]

De Nicola R., Hennessy M.C., Testing equivalences for processes, Theoret. Comput. Sci. 34 (1–2) (1984) 83–133,.

[10]

Dias Neto A.C., Subramanyan R., Vieira M., Travassos G.H., A survey on model-based testing approaches: a systematic review, in: Proceedings of International Workshop on Empirical Assessment of Software Engineering Languages and Technologies, 2007, pp. 31–36,.

Digital Library

[11]

Eldh S., On Test Design, (Ph.D. thesis) Mälardalen University, 2011, doi: none.

[12]

Elmendorf W., Automated Design of Program Test Libraries, 1970, https://benderrbt.com/Automated%20Design%20of%20Program%20Test%20Libraries%20-%201970.pdf.

[13]

Elodie B., Fabrice A., Bruno L., Arnaud B., Lightweight model-based testing for enterprise IT, in: IEEE International Conference on Software Testing, Verification and Validation, IEEE, 2018, pp. 224–230,.

[14]

Ernits J., Roo R., Jacky J., Veanes M., Model-based testing of web applications using nmodel, in: Testing of Software and Communication Systems, Springer, 2009, pp. 211–216,.

Digital Library

[15]

Garousi V., Traffic-Aware Stress Testing of Distributed Real-Time Systems Based on UML Models using Genetic Algorithms, (Ph.D. thesis) Department of Systems and Computer Engineering, Carleton University, 2006, doi: none.

[16]

Garousi V., Borg M., Oivo M., Practical relevance of software engineering research: synthesizing the community’s voice, Empir. Softw. Eng. 25 (2020) 1687–1754,.

Digital Library

[17]

Garousi V., Elberzhager F., Test automation: not just for test execution, IEEE Softw. 34 (2) (2017) 90–96,.

Digital Library

[18]

Garousi V., Eskandar M.M., Herkiloğlu K., Industry-academia collaborations in software testing: experience and success stories from Canada and Turkey, Softw. Qual. J. 25 (4) (2017) 1091–1143,.

Digital Library

[19]

Garousi V., Felderer M., Developing, verifying and maintaining high-quality automated test scripts, IEEE Softw. 33 (3) (2016) 68–75,.

Digital Library

[20]

Garousi V., Felderer M., Hacaloğlu T., Software test maturity assessment and test process improvement: A multivocal literature review, Inf. Softw. Technol. 85 (2017) 16–42,.

Digital Library

[21]

Garousi V., Felderer M., Hacaloğlu T., What we know about software test maturity and test process improvement, IEEE Softw. 35 (1) (2018) 84–92,.

[22]

Garousi V., Felderer M., Ç. M. Karapıçak T., Yılmaz U., What we know about testing embedded software, IEEE Softw. 35 (4) (2018) 62–69,.

Digital Library

[23]

Garousi V., Keleş A.B., Balaman Y., Güler Z.Ö., Arcuri A., Data set: MBT of testinium-test suite and demo video run of the test suite, 2021, https://doi.org/10.5281/zenodo.4642018 (Last accessed: April 2021).

[24]

Garousi V., Keles A.B., Next level of test automation with model-based testing, in: A Talk in the Northern Ireland Developer Conference (NIDevConf), 2020, https://youtu.be/zaMpJ1aoj30.

[25]

Garousi V., Keleş A.B., Balaman Y., Güler Z.Ö., Test automation with the gauge framework: Experience and best practices, in: International Conference on Computational Science and Its Applications, 2020, pp. 458–470,.

Digital Library

[26]

Garousi V., Keleş A.B., Güler Z.Ö., Balaman Y., Executable natural-language test specifications: A test-automation experience report, in: Proceedings of the Turkish National Software Engineering Symposium (UYMS), 2019, doi: none.

[27]

Garousi V., Küçük B., Smells in software test code: A survey of knowledge in industry and academia, J. Syst. Softw. 138 (2018) 52–81,.

[28]

Garousi V., Shepherd D.C., Herkiloğlu K., Successful engagement of practitioners and software engineering researchers: Evidence from 26 international industry-academia collaborative projects, IEEE Softw. (2020),. (in press).

[29]

Garousi V., van Veenendaal E., Test maturity model integration (TMMi): Trends of worldwide test maturity and certifications, IEEE Softw. (2021),. (in press).

[30]

Garousi V., et al., Comparing automated visual GUI testing tools: an industrial case study, in: Proceedings of ACM SIGSOFT International Workshop on Automated Software Testing, 2017, pp. 21–28,.

Digital Library

[31]

Gorschek T., Garre P., Larsson S., Wohlin C., A model for technology transfer in practice, IEEE Softw. 23 (6) (2006) 88–95,.

Digital Library

[32]

Graham D., Fewster M., Experiences of Test Automation: Case Studies of Software Test Automation, Addison-Wesley Professional, 2012, doi: none.

[33]

Grieskamp W., Microsoft’s protocol documentation program: A success story for model-based testing, in: International Academic and Industrial Conference on Practice and Research Techniques, Springer, 2010, p. 7,.

[34]

Grieskamp W., Kicillof N., Stobie K., Braberman V., Model-based quality assurance of protocol documentation: tools and methodology, Softw. Test. Verif. Reliab. 21 (1) (2011) 55–71,.

Digital Library

[35]

Groz R., Simao A., Petrenko A., Oriat C., Inferring finite state machines without reset using state identification sequences, in: IFIP International Conference on Testing Software and Systems, Springer, 2015, pp. 161–177,.

Digital Library

[36]

Harman M., Jones B.F., Search-based software engineering, Inf. Softw. Technol. 43 (14) (2001) 833–839,.

[37]

Iivari J., Venable J., Action research and design science research: seemingly similar but decisively dissimilar, in: European Conference on Information Systems, 2009, doi: none.

[38]

Janicki M., Katara M., Pääkkönen T., Obstacles and opportunities in deploying model-based GUI testing of mobile software: a survey, Softw. Test. Verif. Reliab. 22 (5) (2012) 313–341,.

Digital Library

[39]

Kramer A., Legeard B., Model-Based Testing Essentials-Guide to the ISTQB Certified Model-Based Tester, John Wiley & Sons, ISBN 978-1-119-13001-7, 2016.

[40]

Krenn W., Schlick R., Tiran S., Aichernig B., Jobstl E., Brandl H., Momut:: UML model-based mutation testing for UML, in: IEEE International Conference on Software Testing, Verification and Validation, IEEE, 2015, pp. 1–8,.

[41]

Lackner H., Svacina J., Weißleder S., Aigner M., Kresse M., Introducing model-based testing in industrial context–an experience report, Model-Based Test. Pract. 11 (2010) doi: none.

[42]

Li W., Le Gall F., Spaseski N., A survey on model-based testing tools for test case generation, in: International Conference on tools and Methods for Program Analysis, Springer, 2017, pp. 77–89,.

[43]

Meinke K., Walkinshaw N., Model-based testing and model inference, in: International Symposium on Leveraging Applications of Formal Methods, Verification and Validation, Springer, 2012, pp. 440–443,.

Digital Library

[44]

Memon A., Banerjee I., Nagarajan A., GUI Ripping: Reverse engineering of graphical user interfaces for testing, in: 10th Working Conference on Reverse Engineering, 2003. WCRE 2003. Proceedings, Citeseer, 2003, pp. 260–269. doi: none.

[45]

Memon A., Banerjee I., Nagarajan A., GUI Ripping: Reverse engineering of graphical user interfaces for testing, in: Proceedings of Working Conference on Reverse Engineering, 2003, pp. 260–269. doi: none.

[46]

Mesbah A., Bozdag E., Van Deursen A., Crawling ajax by inferring user interface state changes, in: International Conference on Web Engineering, 2008, pp. 122–134,.

Digital Library

[47]

Musa J.D., Operational profiles in software-reliability engineering, IEEE Softw. 10 (2) (1993) 14–32,.

Digital Library

[48]

Neto A.D., Subramanyan R., Vieira M., Travassos G.H., Shull F., Improving evidence about software technologies: A look at model-based testing, IEEE Softw. 25 (3) (2008) 10–13,.

Digital Library

[49]

Peleska J., Industrial-strength model-based testing-state of the art and current challenges, 2013, arXiv preprint arXiv:1303.1006 doi: none.

[50]

Petersen K., Gencel C., Asghari N., Baca D., Betz S., Action research as a model for industry-academia collaboration in the software engineering context, in: Proceedings of International Workshop on Long-Term Industrial Collaboration on Software Engineering, 2014,. pp. 55-62.

Digital Library

[51]

Petrenko A., Simao A., Maldonado J.C., Model-Based Testing of Software and Systems: Recent Advances and Challenges, Springer, ed, 2012,.

[52]

Plays-in-Business.com, Maturity and capability — What is it?, 2021, https://www.plays-in-business.com/maturity-and-capability-what-is-it/ (Last accessed: April 2021).

[53]

Polo M., Reales P., Piattini M., Ebert C., Test automation, IEEE Softw. 30 (1) (2013) 84–89,.

[54]

Pree W., Design Patterns for Object-Oriented Software Development, Addison-Wesley, 1995, doi: none.

[55]

Raulamo P., Mäntylä M.V., Garousi V., Choosing the right test automation tool: a grey literature review, in: International Conference on Evaluation and Assessment in Software Engineering, 2017, pp. 21–30,.

Digital Library

[56]

Robinson H., Obstacles and opportunities for model-based testing in an industrial software environment, in: Proceedings of the European Conference on Model-Driven Software Engineering, 2003, pp. 118–127. doi: none.

[57]

d. Santos P.S.M., Travassos G.H., Action-research use in software engineering: An initial survey, in: Proceedings of the International Symposium on Empirical Software Engineering and Measurement, 2009, pp. 414–417,.

Digital Library

[58]

Schieferdecker I., Model-based testing, IEEE Softw. 29 (1) (2012) 14,.

Digital Library

[59]

Shafique M., Labiche Y., A Systematic Review of Model Based Testing tool Support, Carleton University, 2010, doi: none.

[60]

Stringer E.T., Action Research, SAGE Publications, 2013, doi: none.

[61]

Sunyé G., Pollet D., Le Traon Y., Jézéquel J.-M., models Refactoring.UML., Refactoring UML models, in: International Conference on the Unified Modeling Language, Springer, 2001, pp. 134–148,.

[62]

toye F., ’not everything that can be counted counts and not everything that counts can be counted’ (attributed to albert Einstein) (in eng), Br. J. Pain 9 (1) (2015) 7,.

[63]

Tretmans J., Model based testing with labelled transition systems, in: Formal Methods and Testing, Springer, 2008, pp. 1–38,.

[64]

Tretmans G., van de Laar P., Model-based testing with torxakis: The mysteries of dropbox revisited, in: Proceedings of the Central European Conference on Information and Intelligent Systems, 2019, doi: none.

[65]

Utting M., Legeard B., Practical Model-Based Testing: A tools Approach, Elsevier, 2010.

Digital Library

[66]

Utting M., Pretschner A., Legeard B., A taxonomy of model-based testing approaches, Softw. Test. Verif. Reliab. 22 (5) (2012) 297–312,.

Digital Library

[67]

Veanes M., Campbell C., Grieskamp W., Schulte W., Tillmann N., Nachmanson L., Model-based testing of object-oriented reactive systems with spec explorer, in: Formal Methods and Testing, Springer, 2008, pp. 39–76,.

[68]

Veenendaal E.v., Wells B., Test Maturity Model Integration (TMMi): Guidelines for Test Process Improvement, Uitgeverij Tutein Nolthenius, 2012.

[69]

Vegas S., Dieste Ó., Juristo N., Difficulties in running experiments in the software industry: experiences from the trenches, in: International Workshop on Conducting Empirical Studies in Industry, 2015, pp. 3–9,.

Digital Library

[70]

Vos T.E., Kruse P.M., Condori-Fernández N., Bauersfeld S., Wegener J., Testar: Tool support for test automation at the user interface level, Int. J. Inf. Syst. Model. Des. (IJISMD) 6 (3) (2015) 46–83,.

Digital Library

[71]

Walkinshaw N., Derrick J., Guo Q., Iterative refinement of reverse-engineered models by model-based testing, in: International Symposium on Formal Methods, Springer, 2009, pp. 305–320,.

Digital Library

[72]

West D.B., Introduction to Graph Theory, Prentice hall, 1996, doi: none.

Cited By

Cao SChen RPan MYang WLi XFilkov VRay BZhou M(2024)Beyond Manual Modeling: Automating GUI Model Generation Using Design DocumentsProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695032(91-103)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3695032
Lonetti FBertolino ADi Giandomenico F(2024)Model-based security testing in IoT systemsInformation and Software Technology10.1016/j.infsof.2023.107326164:COnline publication date: 10-Jan-2024
https://dl.acm.org/doi/10.1016/j.infsof.2023.107326

Index Terms

Model-based testing in practice: An experience report from the web applications domain

Index terms have been assigned to the content through auto-classification.

Recommendations

Model-based testing in industry: a case study with two MBT tools
AST '10: Proceedings of the 5th Workshop on Automation of Software Test

Traditional testing techniques may not always be suitable for adequate, thorough, and extensible testing of critical and complex software in a resource and time constrained software development environment. Model-based testing (MBT) is an evolving ...
An orchestrated survey of methodologies for automated software test case generation

Test case generation is among the most labour-intensive tasks in software testing. It also has a strong impact on the effectiveness and efficiency of software testing. For these reasons, it has been one of the most active research topics in software ...
Model-Based Testing in Practice: An Industrial Case Study using GraphWalker
ISEC '21: Proceedings of the 14th Innovations in Software Engineering Conference (formerly known as India Software Engineering Conference)

Model-based testing (MBT) is a test design technique that supports the automation of software testing processes and generates test artefacts based on a system model representing behavioural aspects of the system under test (SUT). Previous research has ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Journal of Systems and Software

Journal of Systems and Software Volume 180, Issue C

Oct 2021

385 pages

ISSN:0164-1212

Issue’s Table of Contents

Elsevier Inc.

Publisher

Elsevier Science Inc.

United States

Publication History

Published: 01 October 2021

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 12 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Cao SChen RPan MYang WLi XFilkov VRay BZhou M(2024)Beyond Manual Modeling: Automating GUI Model Generation Using Design DocumentsProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695032(91-103)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3695032
Lonetti FBertolino ADi Giandomenico F(2024)Model-based security testing in IoT systemsInformation and Software Technology10.1016/j.infsof.2023.107326164:COnline publication date: 10-Jan-2024
https://dl.acm.org/doi/10.1016/j.infsof.2023.107326

View Options

View options

Media

Figures

Other

Tables

View Issue’s Table of Contents