More Web Proxy on the site http://driver.im/

research-article

Practitioner Evaluations on Software Testing Tools

Authors:

Päivi Raulamo-Jurvanen,

Mika V. MäntyläAuthors Info & Claims

EASE '19: Proceedings of the 23rd International Conference on Evaluation and Assessment in Software Engineering

Pages 57 - 66

https://doi.org/10.1145/3319008.3319018

Published: 15 April 2019 Publication History

Abstract

In software engineering practice, evaluating and selecting the software testing tools that best fit the project at hand is an important and challenging task. In scientific studies of software engineering, practitioner evaluations and beliefs have recently gained interest, and some studies suggest that practitioners find beliefs of peers more credible than empirical evidence. To study how software practitioners evaluate testing tools, we applied online opinion surveys (n=89). We analyzed the reliability of the opinions utilizing Krippendorff's alpha, intra-class correlation coefficient (ICC), and coefficients of variation (CV). Negative binomial regression was used to evaluate the effect of demographics. We find that opinions towards a specific tool can be conflicting. We show how increasing the number of respondents improves the reliability of the estimates measured with ICC. Our results indicate that on average, opinions from seven experts provide a moderate level of reliability. From demographics, we find that technical seniority leads to more negative evaluations. To improve the understanding, robustness, and impact of the findings, we need to conduct further studies by utilizing diverse sources and complementary methods.

References

[1]

Mohsen Anvaari, Carl-Fredrik Sørensen, and Olaf Zimmermann. 2016. Associating Architectural Issues with Quality Attributes: A Survey on Expert Agreement. In Proceedings of the 10th European Conference on Software Architecture Workshops (ECSAW '16). ACM, New York, NY, USA, Article 11, 7 pages.

Digital Library

[2]

Gayane Azizyan, Miganoush K. Magarian, and Mira Kajko-Matsson. 2011. Survey of Agile Tool Usage and Needs. In 2011 Agile Conference. IEEE, Salt Lake City, UT, USA, 29--38.

Digital Library

[3]

Bajaj Harsh, Infosys. 2015. Choosing the right automation tool. https://www.infosys.com/it-services/validation-solutions/white-papers/documents/choosing-right-automation-tool.pdf.

[4]

Markus Borg, Iben Lennerstad, Rasmus Ros, and Elizabeth Bjarnason. 2017. On Using Active Learning and Self-training when Mining Performance Discussions on Stack Overflow. In Proceedings of the 21st International Conference on Evaluation and Assessment in Software Engineering, EASE'17. ACM, New York, NY, USA, 308--313.

Digital Library

[5]

Anne Bruton, Joy H Conway, and Stephen T Holgate. 2000. Reliability: What is it, and how is it measured. Physiotherapy 86, 2 (2000), 94--99.

[6]

Capgemini Consulting. 2015. World Quality Report 2014--2015, Sixth Edition, 2014. https://www.sogeti.com/explore/reports/world-quality-report-2014-2015/.

[7]

Capgemini Consulting. 2016. World Quality Report 2015--2016, 2015. https://www.sogeti.com/explore/reports/world-quality-report-2015-2016/.

[8]

Adnan Causevic, Daniel Sundmark, and Sasikumar Punnekkat. 2010. An Industrial Survey on Contemporary Aspects of Software Testing. In 2010 3rd International Conference on Software Testing, Verification and Validation, Vol. 1. IEEE, USA, 393--401.

Digital Library

[9]

Varun Chandola, Arindam Banerjee, and Vipin Kumar. 2009. Anomaly Detection: A Survey. ACM Comput. Surv. 41, 3, Article 15 (July 2009), 58 pages.

Digital Library

[10]

Prem Devanbu, Thomas Zimmermann, and Christian Bird. 2016. Belief & Evidence in Empirical Software Engineering. In Proceedings of the 38th International Conference on Software Engineering (ICSE '16). ACM, New York, NY, USA, 108--119.

Digital Library

[11]

Arilo Claudio Dias-Neto, Santiago Matalonga, Martín Solari, Gabriela Robiolo, and Guilherme Horta Travassos. 2017. Toward the characterization of software testing practices in South America: looking at Brazil and Uruguay. Software Quality Journal 25, 4 (01 Dec 2017), 1145--1183.

Digital Library

[12]

Tore Dybå, Dag I.K. Sjøberg, and Daniela S. Cruzes. 2012. What Works for Whom, Where, when, and Why?: On the Role of Context in Empirical Software Engineering. In Proceedings of the ACM-IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM '12). ACM, New York, NY, USA, 19--28.

Digital Library

[13]

Emelie Engström and Per Runeson. 2010. A qualitative survey of regression testing practices. In Product-Focused Software Process Improvement/Lecture Notes in Computer Science, M. A. Babar, M. Vierimaa, and M. Oivo (Eds.) (Eds.), Vol. 6156. Springer, Berlin, Heidelberg, 3--16.

Digital Library

[14]

Arlene Fink. 1995. The Survey Handbook. Sage publications, CA, USA.

[15]

Vahid Garousi and Mika V. Mäntylä". 2016. When and what to automate in software testing? A multi-vocal literature review. Information and Software Technology 76 (2016), 92--117.

Digital Library

[16]

Vahid Garousi and Junji Zhi. 2013. A survey of software testing practices in Canada. Journal of Systems and Software 86, 5 (2013), 1354--1376.

Digital Library

[17]

Adam M. Geras, Michael R. Smith, and James Miller. 2004. A survey of software testing practices in alberta. Canadian Journal of Electrical and Computer Engineering 29, 3 (July 2004), 183--191.

[18]

Jorge Goncalves, Simo Hosio, and Vassilis Kostakos. 2017. Eliciting Structured Knowledge from Situated Crowd Markets. ACM Trans. Internet Technol. 17, 2, Article 14 (mar 2017), 21 pages.

Digital Library

[19]

Kevin A. Hallgren. 2012. Computing Inter-Rater Reliability for Observational Data: An Overview and Tutorial. Tutorials in quantitative methods for psychology 8, 1 (2012), 23--34. http://europepmc.org/articles/PMC3402032

[20]

Helppi, Ville-Veikko. Bitbar Technologies. 2015. The Basics Of Test Automation For Apps, Games, Mobile Web. https://www.smashingmagazine.com/2015/01/basic-test-automation-for-apps-games-and-mobile-web/.

[21]

Simo Hosio, Jorge Goncalves, Theodoros Anagnostopoulos, and Vassilis Kostakos. 2016. Leveraging Wisdom of the Crowd for Decision Support. In Proceedings of the 30th International BCS Human Computer Interaction Conference: Fusion! (HCI '16). BCS Learning & Development Ltd., Swindon, UK, Article 38, 12 pages.

Digital Library

[22]

Simo Hosio, Jorge Goncalves, Vassilis Kostakos, and Jukka Riekki. 2015. Crowd-sourcing Public Opinion Using Urban Pervasive Technologies: Lessons From Real-Life Experiments in Oulu. Policy & Internet 7, 2 (2015), 203--222.

[23]

ISTQB (International Software Testing Qualifications Board). 2016. Worldwide Software Testing Practices Report 2015--2016. https://www.istqb.org/documents/ISTQB_Worldwide_Software_Testing_Practices_Report.pdf.

[24]

Tanjila Kanij, Robert Merkel, and John Grundy. 2013. Lessons learned from conducting industry surveys in software testing. In 2013 1st International Workshop on Conducting Empirical Studies in Industry (CESI). IEEE, NJ, USA, 63--66.

Digital Library

[25]

Barbara A. Kitchenham and Shari L. Pfleeger. 2008. Personal Opinion Surveys. Springer London, London, 63--92.

[26]

Barbara A. Kitchenham, Shari L. Pfleeger, Lesley M. Pickard, W. Jones, Peter, David C. Hoaglin, Khaled El Emam, and Jarrett Rosenberg. 2002. Preliminary guidelines for empirical research in software engineering. IEEE Transactions on Software Engineering 28, 8 (Aug 2002), 721--734.

Digital Library

[27]

Barbara Ann Kitchenham, Dag I.K. Sjøberg, Tore Dybå, Dietmar Pfahl, Pearl Brereton, David Budgen, Martin Höst, and Per Runeson. 2012. Three empirical studies on the agreement of reviewers about the quality of software engineering experiments. Information and Software Technology 54, 8 (2012), 804--819. Special Issue: Voice of the Editorial Board.

Digital Library

[28]

Terry K. Koo and Mae Y. Li. 2016. A Guideline of Selecting and Reporting Intraclass Correlation Coefficients for Reliability Research. Journal of Chiropractic Medicine 15, 2 (2016), 155-163.

[29]

Klaus Krippendorff. 2004. Content Analysis: An Introduction to Its Methodology. Sage, Thousand Oaks, CA, USA.

[30]

Klaus Krippendorff. 2011. Computing Krippendorff's Alpha Reliability. Retrieved May 18, 2018 from https://repository.upenn.edu/asc_papers/43

[31]

Tarald O. Kvålseth. 1995. Coefficients of Variation for Nominal and Ordinal Categorical Data. Perceptual and Motor Skills 80, 3 (1995), 843--847.

[32]

J. Lee, S. Kang, and D. Lee. 2012. Survey on software testing practices. IET Software 6, 3 (2012), 275--282.

[33]

Robert Libby and Roger K. Blashfield. 1978. Performance of a composite as a function of the number of judges. Organizational Behavior and Human Performance 21, 2 (1978), 121--129.

[34]

A.E. Mannes, Jack B. Soll, and Richard P. Larrick. 2014. The wisdom of select crowds. Personality and Social Psychology 107, 2 (2014), 276--299.

[35]

Jason W. Osborne and Amy Overbay. 2004. The power of outliers (and why researchers should ALWAYS check for them). Practical Assessment, Research & Evaluation 9, 6 (2004), 1--8. https://www.ingentaconnect.com/content/doaj/15317714/2004/00000009/00000006/art00001

[36]

Amantia Pano, Daniel Graziotin, and Pekka Abrahamsson. 2018. Factors and actors leading to the adoption of a JavaScript framework. "Article in Press".

Digital Library

[37]

Carol Passos, Ana P. Braun, Daniela S. Cruzes, and Manoel Mendonca. 2011. Analyzing the Impact of Beliefs in Software Project Practices. In 2011 International Symposium on Empirical Software Engineering and Measurement. IEEE, Banff, AB, Canada, 444--452.

Digital Library

[38]

Michael Quinn Patton. 2002. Qualitative Research and Evaluation Methods, 3rd. ed. Sage publications, Thousand Oaks, CA, USA.

[39]

Robert M. Poston and Michael P. Sexton. 1992. Evaluating and selecting testing tools. In {1992} Proceedings of the Second Symposium on Assessment of Quality Software Development Tools. IEEE, New Orleans, LA, USA, 55--64.

[40]

QASource. 2016. A Guide to Selecting The Best Test Automation Tool. https://drive.google.com/open?id=0B6dKdxaNjBENY3pxMkgwb3BneVk.

[41]

Dudekula Mohammad Rafi, Katam Reddy Kiran Moses, Kai Petersen, and Mika V. Mäntylä. 2012. Benefits and Limitations of Automated Software Testing: Systematic Literature Review and Practitioner Survey. In Proceedings of the 7th International Workshop on Automation of Software Test (AST '12). IEEE Press, Piscataway, NJ, USA, Article 6, 7 pages. http://dl.acm.org/citation.cfm?id=2663608.2663616

Digital Library

[42]

Pär J. Ågerfalk and Brian Fitzgerald. 2008. Outsourcing to an Unknown Workforce: Exploring Opensourcing as a Global Sourcing Strategy. MIS Quarterly 32, 2 (2008), 385--409. http://www.jstor.org/stable/25148845

Digital Library

[43]

Austen Rainer, Timothy Hall, and Nathan Baddoo. 2003. Persuading developers to "buy into" software process improvement: a local opinion and empirical evidence. In 2003 International Symposium on Empirical Software Engineering, 2003. ISESE 2003. Proceedings. IEEE, Rome, Italy, 326--335.

Digital Library

[44]

Päivi Raulamo-Jurvanen, Kari Kakkonen, and Mika Mäntylä. 2016. Using Surveys and Web-Scraping to Select Tools for Software Testing Consultancy. Springer International Publishing, Cham, 285--300.

[45]

Päivi Raulamo-Jurvanen, Mika Mäntylä, and Vahid Garousi. 2017. Choosing the Right Test Automation Tool: A Grey Literature Review of Practitioner Sources. In Proceedings of the 21st International Conference on Evaluation and Assessment in Software Engineering (EASE'17). ACM, New York, NY, USA, Article 3, 10 pages.

Digital Library

[46]

Patrick E. Shrout and Joseph L. Fleiss. 1979. Intraclass Correlations: Uses in Assessing Rater Reliability. Psychological Bulletin 86, 2 (July 1979), 420--428.

[47]

Steven E. Stemler. 2004. A Comparison of Consensus, Consistency, and Measurement Approaches to Estimating Interrater Reliability. Practical Assessment, Research & Evaluation 9, 4 (2004), 1--11. https://www.ingentaconnect.com/content/doaj/15317714/2004/00000009/00000004/art00001

[48]

James Surowiecki. 2005. The Wisdom of Crowds. Anchor, New York, USA.

Digital Library

[49]

John W. Tukey. 1977. Exploratory data analysis. Addison-Wesley, Reading, Mass.

[50]

Gerard Wagenaar, Sietse Overbeek, and Remko Helms. 2017. Describing Criteria for Selecting a Scrum Tool Using the Technology Acceptance Model. In Intelligent Information and Database Systems, Ngoc Thanh Nguyen, Satoshi Tojo, Le Minh Nguyen, and Bogdan Trawiński (Eds.). Springer International Publishing, Cham, 811--821.

[51]

Claes Wohlin, Martin Höst, and Kennet Henningsson. 2003. Empirical Research Methods in Software Engineering. Springer, Berlin, Heidelberg, 7--23.

[52]

Claes Wohlin, Martin Höst, and Kennet Henningsson. 2006. Empirical Research Methods in Web and Software Engineering. In Web Engineering, Emilia Mendes and Nile Mosley (Eds.). Springer, Berlin, Heidelberg, 409--430.

Cited By

Nguyen TBaldassarre Mde Lima Lde Souza Santos R(2024)From Literature to Practice: Exploring Fairness Testing Tools for the Software Industry AdoptionProceedings of the 18th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement10.1145/3674805.3695404(549-555)Online publication date: 24-Oct-2024
https://dl.acm.org/doi/10.1145/3674805.3695404
Aburas A(2024)Choosing the Right Automated Software Testing Tools2024 IEEE 4th International Maghreb Meeting of the Conference on Sciences and Techniques of Automatic Control and Computer Engineering (MI-STA)10.1109/MI-STA61267.2024.10599723(31-35)Online publication date: 19-May-2024
https://doi.org/10.1109/MI-STA61267.2024.10599723
Mahmud TChe MNgu AYang G(2024)An Empirical Investigation on Android App Testing Practices2024 IEEE 35th International Symposium on Software Reliability Engineering (ISSRE)10.1109/ISSRE62328.2024.00042(355-366)Online publication date: 28-Oct-2024
https://doi.org/10.1109/ISSRE62328.2024.00042
Show More Cited By

Index Terms

Practitioner Evaluations on Software Testing Tools
1. Social and professional topics
  1. Professional topics
    1. Computing and business
      1. Automation
2. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Software defect analysis
        Software testing and debugging
  2. Software notations and tools
    1. Software maintenance tools

Recommendations

What industry wants from academia in software testing?: Hearing practitioners' opinions
EASE '17: Proceedings of the 21st International Conference on Evaluation and Assessment in Software Engineering

The level of industry-academia collaboration (IAC) in software engineering in general and in software testing in particular is quite low. Many researchers and practitioners are not collaborating with the "other side" to solve industrial problems. To ...
Static analysis summit II
SIGAda '07: Proceedings of the 2007 ACM international conference on SIGAda annual international conference

Funded by the Department of Homeland Security (DHS), the National Institute of Standards and Technology (NIST) started a long-term, ambitious project to identify, enhance and develop software assurance tools. The Software Assurance Metrics And Tool ...
Exploring the industry's challenges in software testing: An empirical study
Abstract Context
Software testing is an important and costly software engineering activity in the industry. Despite the efforts of the software testing research community in the last several decades, various studies show that still many practitioners in the ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Other conferences

EASE '19: Proceedings of the 23rd International Conference on Evaluation and Assessment in Software Engineering

April 2019

345 pages

ISBN:9781450371452

DOI:10.1145/3319008

Program Chairs:
Shaukat Ali,
Vahid Garousi

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

In-Cooperation

IT University of Copenhagen

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 15 April 2019

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Funding Sources

Business Finland
Academy of Finland

Conference

EASE '19

EASE '19: Evaluation and Assessment in Software Engineering

April 15 - 17, 2019

Copenhagen, Denmark

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

10
Total Citations
View Citations
449
Total Downloads

Downloads (Last 12 months)37
Downloads (Last 6 weeks)2

Reflects downloads up to 15 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Nguyen TBaldassarre Mde Lima Lde Souza Santos R(2024)From Literature to Practice: Exploring Fairness Testing Tools for the Software Industry AdoptionProceedings of the 18th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement10.1145/3674805.3695404(549-555)Online publication date: 24-Oct-2024
https://dl.acm.org/doi/10.1145/3674805.3695404
Aburas A(2024)Choosing the Right Automated Software Testing Tools2024 IEEE 4th International Maghreb Meeting of the Conference on Sciences and Techniques of Automatic Control and Computer Engineering (MI-STA)10.1109/MI-STA61267.2024.10599723(31-35)Online publication date: 19-May-2024
https://doi.org/10.1109/MI-STA61267.2024.10599723
Mahmud TChe MNgu AYang G(2024)An Empirical Investigation on Android App Testing Practices2024 IEEE 35th International Symposium on Software Reliability Engineering (ISSRE)10.1109/ISSRE62328.2024.00042(355-366)Online publication date: 28-Oct-2024
https://doi.org/10.1109/ISSRE62328.2024.00042
Ren HLi YChen LZhou YNie C(2024)Why and how bug blocking relations are breakableInformation and Software Technology10.1016/j.infsof.2023.107354166:COnline publication date: 1-Feb-2024
https://dl.acm.org/doi/10.1016/j.infsof.2023.107354
Rahman AShamim SBose DPandita R(2023)Security Misconfigurations in Open Source Kubernetes Manifests: An Empirical StudyACM Transactions on Software Engineering and Methodology10.1145/357963932:4(1-36)Online publication date: 26-May-2023
https://dl.acm.org/doi/10.1145/3579639
Rahman ABose DShakya RPandita R(2023)Come for syntax, stay for speed, understand defects: an empirical study of defects in Julia programsEmpirical Software Engineering10.1007/s10664-023-10328-528:4Online publication date: 14-Jun-2023
https://doi.org/10.1007/s10664-023-10328-5
Salari MPaul Enoiu EAfzal WSeceleanu C(2022)Choosing a Test Automation Framework for Programmable Logic Controllers in CODESYS Development Environment2022 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)10.1109/ICSTW55395.2022.00055(277-284)Online publication date: Apr-2022
https://doi.org/10.1109/ICSTW55395.2022.00055
Wang YMäntylä MLiu ZMarkkula JRaulamo‐jurvanen P(2022)Improving test automation maturity: A multivocal literature reviewSoftware Testing, Verification and Reliability10.1002/stvr.180432:3Online publication date: 15-Feb-2022
https://doi.org/10.1002/stvr.1804
Sivaji ARazak RMohamad NSazali NMusa ABajuri NHashim AAbdullah MJoha NAzis NKuppusamy ADeniel AChuan NClemmensen T(2020)Software Testing Automation: A Comparative Study on Productivity Rate of Open Source Automated Software Testing Tools For Smart Manufacturing2020 IEEE Conference on Open Systems (ICOS)10.1109/ICOS50156.2020.9293650(7-12)Online publication date: 17-Nov-2020
https://doi.org/10.1109/ICOS50156.2020.9293650
Raulamo-Jurvanen PHosio SMäntylä M(2019)Applying Surveys and Interviews in Software Test Tool EvaluationProduct-Focused Software Process Improvement10.1007/978-3-030-35333-9_2(20-36)Online publication date: 18-Nov-2019
https://doi.org/10.1007/978-3-030-35333-9_2

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents