More Web Proxy on the site http://driver.im/

research-article

A Self-assessment Instrument for Assessing Test Automation Maturity

Authors:

Mika Mäntylä,

Jouni Markkula,

Kristian Wiklund,

Päivi Raulamo-Jurvanen,

Antti HaukinenAuthors Info & Claims

EASE '19: Proceedings of the 23rd International Conference on Evaluation and Assessment in Software Engineering

Pages 145 - 154

https://doi.org/10.1145/3319008.3319020

Published: 15 April 2019 Publication History

Abstract

Test automation is important in the software industry but self-assessment instruments for assessing its maturity are not sufficient. The two objectives of this study are to synthesize what an organization should focus to assess its test automation; develop a self-assessment instrument (a survey) for assessing test automation maturity and scientifically evaluate it. We carried out the study in four stages. First, a literature review of 25 sources was conducted. Second, the initial instrument was developed. Third, seven experts from five companies evaluated the initial instrument. Content Validity Index and Cognitive Interview methods were used. Fourth, we revised the developed instrument. Our contributions are as follows: (a) we collected practices mapped into 15 key areas that indicate where an organization should focus to assess its test automation; (b) we developed and evaluated a self-assessment instrument for assessing test automation maturity; (c) we discuss important topics such as response bias that threatens self-assessment instruments. Our results help companies and researchers to understand and improve test automation practices and processes.

References

[1]

{n. d.}. Tmap Checklists. http://tmap.net/checklists-and-templates

[2]

ISO/TC 176/SC 2. 2015. ISO 9001:2015. (Sep 2015). https://www.isG.org/standard/62085.html

[3]

Pekka Abrahamsson, Juhani Warsta, Mikko T Siponen, and Jussi Ronkainen. 2003. New directions on agile methods: a comparative analysis. In Software Engineering, 2003. Proceedings. 25th International Conference on (ICSE '03). IEEE Computer Society, 244--254.

Digital Library

[4]

Wasif Afzal, Snehal Alone, Kerstin Glocksien, and Richard Torkar. 2016. Software test process improvement approaches: A systematic literature review and an industrial case study. Journal of Systems and Software 111 (2016), 1--33.

Digital Library

[5]

A. Anastasi and S. Urbina. 1997. Psychological Testing. Prentice Hall. https://books.google.fi/books?id=lfFGAAAAMAAJ

[6]

Jari Andersin. 2004. TPI-a model for Test Process Improvement. Department of Computer Science, University of Helsinki (http://www.cs.helsinki.fi/u/paakki/Andersin.pdf), Helsinki (2004).

[7]

Manuel Barrera, Irwin N Sandler, and Thomas B Ramsay. 1981. Preliminary development of a scale of social support: Studies on college students. American Journal of Community Psychology 9, 4 (1981), 435--447.

[8]

Kent Beck, Mike Beedle, Arie Van Bennekum, Alistair Cockburn, Ward Cunningham, Martin Fowler, James Grenning, Jim Highsmith, Andrew Hunt, Ron Jeffries, et al. 2001. Manifesto for agile software development. (2001).

[9]

Jan Bosch. 2016. Speed, data, and ecosystems: The future of software engineering. IEEE Software 33, 1 (2016), 82--88.

Digital Library

[10]

Ilene Burnstein, Taratip Suwanassart, and Robert Carlson. 1996. Developing a testing maturity model for software test process evaluation and improvement. In Test Conference, 1996. Proceedings., International. IEEE, 581--589.

Digital Library

[11]

Ilene Burnstein, Taratip Suwanassart, and Robert Carlson. 1996. Developing a testing maturity model for software test process evaluation and improvement. In Test Conference, 1996. Proceedings., International. IEEE, 581--589.

Digital Library

[12]

Daniela S. Cruzes and Tore Dyba. 2011. Recommended steps for thematic synthesis in software engineering. In Empirical Software Engineering and Measurement (ESEM), 2011 International Symposium on. IEEE, 275--284.

Digital Library

[13]

Paul M Duvall, Steve Matyas, and Andrew Glover. 2007. Continuous integration: improving software quality and reducing risk. Pearson Education.

Digital Library

[14]

Tore Dybå. 2000. An instrument for measuring the key factors of success in software process improvement. Empirical software engineering 5, 4(2000), 357--390.

Digital Library

[15]

Tore Dybå and Torgeir Dingsøyr. 2008. Empirical studies of agile software development: A systematic review. Information and software technology 50, 9--10 (2008), 833--859.

Digital Library

[16]

Steve Easterbrook, Janice Singer, Margaret-Anne Storey, and Daniela Damian. 2008. Selecting empirical methods for software engineering research. In Guide to advanced empirical software engineering. Springer, 285--311.

[17]

Khaled El Emam and Andreas Birk. 2000. Validating the ISO/IEC 15504 measure of software requirements analysis process capability. IEEE transactions on Software Engineering 26, 6 (2000), 541--566.

Digital Library

[18]

Khaled El Emam and Nazim H Madhavji. 1995. The reliability of measuring organizational maturity. Software Process Improvement and Practice 1 (1995), 3--26.

[19]

Sigrid Eldh, Kenneth Andersson, Andreas Ermedahl, and Kristian Wiklund. 2014. Towards a Test Automation Improvement Model (TAIM). Proceedings - IEEE 7th International Conference on Software Testing, Verification and Validation Workshops, ICSTW2014 (2014), 337.

Digital Library

[20]

A v Ewijk, B Linker, M v Oosterwijk, and B Visser. 2013. TPI next: business driven test process improvement. Kleine Uil (2013).

[21]

Ann E Fairhurst, Linda K Good, and James W Gentry. 1989. Fashion involvement: An instrument validation procedure. Clothing and textiles research journal 7, 3 (1989), 10--14.

[22]

Mark Fewster and Dorothy Graham. 1999. Software test automation. ACM Press/Addison-Wesley Publishing Co.

Digital Library

[23]

Ronald P Fisher and R Edward Geiselman. 1992. Memory enhancing techniques for investigative interviewing: The cognitive interview. Charles C Thomas Publisher.

[24]

Micro Focus, Capgemini, and Sogeti. 2018. World Quality Report 2017--18. Technical Report.

[25]

TMM foundation. 2012. Test Maturity Model integration. http://www.tmmi.org/pdf/TMMi.Framework.pdf

Digital Library

[26]

Adrian Furnham. 1986. Response bias, social desirability and dissimulation. Personality and individual differences 7, 3 (1986), 385--400.

[27]

Vahid Garousi, Michael Felderer, and Tuna Hacaloğlu. 2017. Software test maturity assessment and test process improvement: A multivocal literature review. Information and Software Technology 85 (May 2017), 16--42.

Digital Library

[28]

William E Hefley and William Curtis. 1998. People CMM-Based Assessment Method Description. (1998).

[29]

Katarína Hrabovská, Bruno Rossi, and Tomas Pitner. 2019. Software Testing Process Models Benefits & Drawbacks: a Systematic Literature Review. arXiv preprint arXiv:1901.01450 (2019).

[30]

Jez Humble and Joanne Molesky. 2011. Why enterprises must adopt devops to enable continuous delivery. Cutter IT Journal 24, 8 (2011), 6.

[31]

Carole L Kimberlin and Almut G Winterstein. 2008. Validity and reliability of measurement instruments used in research. American Journal of Health-System Pharmacy 65, 23 (2008), 2276--2284.

[32]

Barbara A Kitchenham and Shari Lawrence Pfleeger. 2002. Principles of survey research: part 3: constructing a survey instrument. ACM SIGSOFT Software Engineering Notes 27, 2 (2002), 20--24.

Digital Library

[33]

Tim Koomen, Bart Broekman, Leo van der Aalst, and Michiel Vroon. 2013. TMap next: for result-driven testing. Uitgeverij kleine Uil.

Digital Library

[34]

Tim Koomen and Martin Pol. 1999. Test process improvement: a practical step-by-step guide to structured testing. Addison-Wesley Longman Publishing Co., Inc.

Digital Library

[35]

Jon A Krosnick, Sowmya Narayan, and Wendy R Smith. 1996. Satisficing in surveys: Initial evidence. New directions for evaluation 1996, 70 (1996), 29--44.

[36]

G.C. Limited. 2006. Test Organization Maturity Questionnaire. http://cm.techwell.com/sites/default/files/articles/XML0161_0.pdf

[37]

QSR International Pty Ltd. 2018. NVIVO. https://www.qsrinternational.com/nvivo/nvivo-products

[38]

Mary Lynn. 1986. Determination and Quantification Of Content Validity. Nursing Research 35, 6 (Nov 1986), 382--386.

[39]

Victor R Martuza. 1977. Applying norm-referenced and criterion-referenced measurement in education. Allyn & Bacon, Incorporated.

[40]

Delroy L Paulhus. 1991. Measurement and control of response bias. (1991).

[41]

DF Polit, T Beck, and SV Owen. 2007. Focus on research methods is the CVI an acceptable indicator of content validity. Res Nurs Health 30 (2007), 459--467.

[42]

Denise F Polit and Cheryl Tatano Beck. 2006. The content validity index: are you sure you know what's being reported? Critique and recommendations. Research in nursing & health 29, 5 (2006), 489--497.

[43]

Stuart Reid. 2006. Personal Test Maturity Matrix. In CAST 2006: Influencing the Practice June 5th-7th, 2006-Indianapolis. 133.

[44]

Doris McGartland Rubio, Marla Berg-Weger, Susan S Tebb, E Suzanne Lee, and Shannon Rauch. 2003. Objectifying content validity: Conducting a content validity study in social work research. Social work research 27, 2 (2003), 94--104.

[45]

Dag IK Sjoberg, Tore Dyba, and Magne Jorgensen. 2007. The future of empirical methods in software engineering research. In 2007 Future of Software Engineering. IEEE Computer Society, 358--378.

Digital Library

[46]

Ron Swinkels. 2000. A comparison of TMM and other test process improvement models. MB-TMM Project Report (2000), 12--4.

[47]

SIG TestSPICE. 2014. TestSPICE 3.0. http://www.intacs.info/index.php/testspice

[48]

Carolyn F Waltz, Ora Lea Strickland, and Elizabeth R Lenz. 2010. Measurement in nursing and health research. Springer publishing company.

[49]

Yuqing Wang. 2018. Test automation maturity assessment. In Software Testing, Verification and Validation (ICST), 2018 IEEE 11th International Conference on. IEEE, 424--425.

[50]

Kristian Wiklund. 2015. Impediments for Automated Software Test Execution. Ph.D. Dissertation. Mälardalen University.

[51]

Kristian Wiklund, Sigrid Eldh, Daniel Sundmark, and Kristina Lundqvist. 2017. Impediments for software test automation: A systematic literature review. Software Testing, Verification and Reliability 27, 8 (Dec 2017).

[52]

David Zubrow, William Hayes, Jane Siegel, and Dennis Goldenson. 1994. Maturity questionnaire. Technical Report. Carnegie Mellon University Software Engineering Institute.

[53]

{S1} Prabu Chelladurai. 2016. Watch Your STEP. https://www.uploads.pnsqc.org/2011/papers/T- 56Chelladuraipaper.pdf

[54]

{S2} Codenomicon. 2014. Fuzzy Testing Maturity Mode. http://www.codenomicon.com/resources/white-paper/2013/11/01/fuzz-maturity-model.html

[55]

{S3} F. I. Duncan and A. G. Smeaton. 2012. Assessing and improving software quality in safety critical systems by the application of a software test maturity model. IET, Stevenage, UK, 2B3. http://digital-library.theiet.org/content/conferences/10.1049/cp.2012.1509

[56]

{S4} Sigrid Eldh, Kenneth Andersson, Andreas Ermedahl, and Kristian Wiklund. 2014. Towards a test automation improvement model (TAIM). In Software Testing, Verification and Validation Workshops (ICSTW), 2014 IEEE Seventh International Conference on. IEEE, 337--342.

Digital Library

[57]

{S5} Mark Fewster and Dorothy Graham. 1999. Software test automation. ACM Press/Addison-Wesley Publishing Co.

Digital Library

[58]

{S6} TMM foundation. 2012. Test Maturity Model integration. http://www.tmmi.org/pdf/TMMi.Framework.pdf

[59]

{S7} Ana Paula Carvalho Cavalcanti Furtado, Marcos André Wanderley Gomes, Ermeson Carneiro Andrade, and Ivaldir Honorio de Farias Junior. 2012. MPT.BR: A Brazilian Maturity Model for Testing. IEEE. https://ieeexplore.ieee.org/document/6319253

[60]

{S8} Vahid Garousi and Mika Mäntylä. 2016. When and what to automate in software testing? A multi-vocal literature review. Information and Software Technology 76 (2016), 92--117.

Digital Library

[61]

{S9} S. Ronen Harel. 2010. ATMM Agile Testing Maturity. https://www.slideshare.net/AgileSparks/atmm-practical-view

[62]

{S10} Henri Heiskanen, Mika Maunumaa, and Mika Katara. 2012. A Test Process Improvement Model for Automated Test Generation. Product-Focused Software Process Improvement, Vol. 7343. Springer Berlin Heidelberg, Berlin, Heidelberg, 17--31.

Digital Library

[63]

{S11} HP, Sogetti, and capgemini. 2016. World quality report 2015--2016. Technical Report. https://www.capgemini.com/resources/world-quality-report-2015-16/

[64]

{S12} QA Intelligence. 2018. State of testing report 2017. Technical Report.

[65]

{S13} ISTQB. 2016. ISTQBB worldwide software testing practices report 2015--16. Technical Report. ISTQBB International software testing qualifications board. https://www.istqb.org/references/surveys/istqb-worldwide-software-testing-practices-report.html

[66]

{S14} Eun Jung. 2009. A Test Process Improvement Model for Embedded Software Developments. 432--437. https://ieeexplore.ieee.org/document/5381375

Digital Library

[67]

{S15} M. D. Karr. 2013. The Unit Test Maturity Model. http://davidmichaelkarr.blogspot.com.tr/2013/01/the-unit-test-maturity-model.html

[68]

{S16} Tim Koomen, Bart Broekman, Leo van der Aalst, and Michiel Vroon. 2013. TMap next: for result-driven testing. Uitgeverij kleine Uil.

Digital Library

[69]

{S17} Tim Koomen and Martin Pol. 1999. Test process improvement: a practical step-by-step guide to structured testing. Addison-Wesley Longman Publishing Co., Inc.

Digital Library

[70]

{S18} Chongwon Lee. Nov 24, 2009. Adapting and adjusting test process reflecting characteristics of embedded software and industrial properties based on referential models (ICIS '09). ACM, 1372--1377.

Digital Library

[71]

{S19} G.C. Limited. 2006. Test Organization Maturity Questionnaire. http://cm.techwell.com/sites/default/files/articles/XML0161_0.pdf

[72]

{S20} Sandro Morasca, Davide Taibi, and Davide Tosi. 2011. OSS-TMM: Guidelines for Improving the Testing Process of Open Source Software. International Journal of Open Source Software and Processes (IJOSSP) 3, 2 (Apr 1, 2011), 1--22. http://services.igi-global.com/resolvedoi/resolve.aspx?doi=10.4018/jossp.2011040101

Digital Library

[73]

{S21} Rudolf Ramler and Johannes Gmeiner. 2014. Practical Challenges in Test Environment Management. In Software Testing, Verification and Validation Workshops (ICSTW), 2014 IEEE Seventh International Conference on. IEEE, 358--359.

Digital Library

[74]

{S22} Stuart Reid. 2006. personal Test Maturity Matrix. In CAST 2006: Influencing the Practice June 5th-7th, 2006-Indianapolis. 133.

[75]

{S23} J Saldaña-Ramos, Ana Sanz-Esteban, J García-Guzmán, and A Amescua. 2012. Design of a competence model for testing teams. IET Software 6, 5 (Oct 1, 2012), 405. https://search.proquest.com/docview/1638872885

[76]

{S24} SIG TestSPICE. 2014. TestSPICE 3.0. http://www.intacs.info/index.php/testspice

[77]

{S25} Kristian Wiklund, Sigrid Eldh, Daniel Sundmark, and Kristina Lundqvist. 2017. Impediments for software test automation: A systematic literature review. Software Testing, Verification and Reliability 27, 8 (Dec 2017).

Cited By

Moradi MVan Acker BDenil J(2023)Failure Identification Using Model-Implemented Fault Injection with Domain Knowledge-Guided Reinforcement LearningSensors10.3390/s2304216623:4(2166)Online publication date: 14-Feb-2023
https://doi.org/10.3390/s23042166
Wang YMäntylä MLiu ZMarkkula J(2022)Test automation maturity improves product quality—Quantitative study of open source projects using continuous integrationJournal of Systems and Software10.1016/j.jss.2022.111259188:COnline publication date: 1-Jun-2022
https://dl.acm.org/doi/10.1016/j.jss.2022.111259
Gomes MPereira RSilva Mde Vasconcelos JRocha Á(2022)KPI’s for Evaluation of DevOps TeamsInformation Systems and Technologies10.1007/978-3-031-04829-6_13(142-156)Online publication date: 11-May-2022
https://doi.org/10.1007/978-3-031-04829-6_13
Show More Cited By

Index Terms

A Self-assessment Instrument for Assessing Test Automation Maturity
1. Software and its engineering
  1. Software creation and management
    1. Software development process management
    2. Software verification and validation
      1. Software defect analysis
        Software testing and debugging

Recommendations

Group development and group maturity when building agile teams

Overlap between what is meant by an agile team and a mature group in psychology.Group developmental issues are dealt with in practice in agile groups.Quantitative survey data support the connections.Focusing on group development is a key success factor ...
The State of Test Automation in DevOps: A Systematic Literature Review
IC3-2022: Proceedings of the 2022 Fourteenth International Conference on Contemporary Computing

DevOps is becoming increasingly popular in the software industry to deliver quality software in less time. However, building a DevOps pipeline is not easy. Many factors need to be considered in the process, and many decisions need to be made. Making the ...
Software test automation practices in agile development environment: an industry experience report
AST '12: Proceedings of the 7th International Workshop on Automation of Software Test

The increased importance of Test Automation in software engineering is very evident considering the number of companies investing in automated testing tools nowadays, with the main aim of preventing defects during the development process. Test ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Other conferences

EASE '19: Proceedings of the 23rd International Conference on Evaluation and Assessment in Software Engineering

April 2019

345 pages

ISBN:9781450371452

DOI:10.1145/3319008

Program Chairs:
Shaukat Ali,
Vahid Garousi

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

In-Cooperation

IT University of Copenhagen

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 15 April 2019

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

EASE '19

EASE '19: Evaluation and Assessment in Software Engineering

April 15 - 17, 2019

Copenhagen, Denmark

Acceptance Rates

EASE '19 Paper Acceptance Rate 20 of 73 submissions, 27%;

Overall Acceptance Rate 71 of 232 submissions, 31%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

6
Total Citations
View Citations
304
Total Downloads

Downloads (Last 12 months)27
Downloads (Last 6 weeks)4

Reflects downloads up to 15 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Moradi MVan Acker BDenil J(2023)Failure Identification Using Model-Implemented Fault Injection with Domain Knowledge-Guided Reinforcement LearningSensors10.3390/s2304216623:4(2166)Online publication date: 14-Feb-2023
https://doi.org/10.3390/s23042166
Wang YMäntylä MLiu ZMarkkula J(2022)Test automation maturity improves product quality—Quantitative study of open source projects using continuous integrationJournal of Systems and Software10.1016/j.jss.2022.111259188:COnline publication date: 1-Jun-2022
https://dl.acm.org/doi/10.1016/j.jss.2022.111259
Gomes MPereira RSilva Mde Vasconcelos JRocha Á(2022)KPI’s for Evaluation of DevOps TeamsInformation Systems and Technologies10.1007/978-3-031-04829-6_13(142-156)Online publication date: 11-May-2022
https://doi.org/10.1007/978-3-031-04829-6_13
Wang YMäntylä MLiu ZMarkkula JRaulamo‐jurvanen P(2022)Improving test automation maturity: A multivocal literature reviewSoftware Testing, Verification and Reliability10.1002/stvr.180432:3Online publication date: 15-Feb-2022
https://doi.org/10.1002/stvr.1804
Eldh S(2020)Test Automation Improvement Model - TAIM 2.02020 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)10.1109/ICSTW50294.2020.00060(334-337)Online publication date: Oct-2020
https://doi.org/10.1109/ICSTW50294.2020.00060
Wang YPyhajarvi MMantyla M(2020)Test Automation Process Improvement in a DevOps Team: Experience Report2020 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)10.1109/ICSTW50294.2020.00057(314-321)Online publication date: Oct-2020
https://doi.org/10.1109/ICSTW50294.2020.00057

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents