More Web Proxy on the site http://driver.im/

research-article

What Makes Agile Test Artifacts Useful?: An Activity-Based Quality Model from a Practitioners' Perspective

Authors:

Jannik Fischbach,

Henning Femmer,

Andreas VogelsangAuthors Info & Claims

ESEM '20: Proceedings of the 14th ACM / IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM)

Article No.: 41, Pages 1 - 10

https://doi.org/10.1145/3382494.3421462

Published: 23 October 2020 Publication History

Abstract

Background: The artifacts used in Agile software testing and the reasons why these artifacts are used are fairly well-understood. However, empirical research on how Agile test artifacts are eventually designed in practice and which quality factors make them useful for software testing remains sparse. Aims: Our objective is two-fold. First, we identify current challenges in using test artifacts to understand why certain quality factors are considered good or bad. Second, we build an Activity-Based Artifact Quality Model that describes what Agile test artifacts should look like. Method: We conduct an industrial survey with 18 practitioners from 12 companies operating in seven different domains. Results: Our analysis reveals nine challenges and 16 factors describing the quality of six test artifacts from the perspective of Agile testers. Interestingly, we observed mostly challenges regarding language and traceability, which are well-known to occur in non-Agile projects. Conclusions: Although Agile software testing is becoming the norm, we still have little confidence about general do's and don'ts going beyond conventional wisdom. This study is the first to distill a list of quality factors deemed important to what can be considered as useful test artifacts.

References

[1]

V. R. Basili, G. Caldiera, and D. H. Rombach. 1994. The Goal Question Metric Approach. Encyclopedia of Software Engineering 1 (1994).

[2]

J. Bass. 2016. Artefacts and agile method tailoring in large-scale offshore software development programmes. Information and Software Technology 75 (2016).

[3]

K. Beck, M. Beedle, A. van Bennekum, A. Cockburn, W. Cunningham, J. Fowler, M. and Grenning, J. Highsmith, A. Hunt, R. Jeffries, J. Kern, B. Marick, R. C. Martin, S. Mellor, K. Schwaber, J. Sutherland, and D. Thomas. 2001. Manifesto for Agile Software Development.

[4]

D. M. Berry and E. Kamsties. 2004. Perspectives on Software Requirements. Chapter Ambiguity in Requirements Specification.

[5]

M. Bovee, R. P. Srivastava, and B. Mak. 2001. A conceptual framework and belief-function approach to assessing overall information quality. International Journal of Intelligent Systems 18 (2001).

[6]

T. Catarci and M. Scannapieco. 2003. Data quality under the computer science perspective. Archivi & Computer 2 (2003).

[7]

M. Ciolkowski, O. Laitenberger, S. Vegas, and S. Biffl. 2003. Practical Experiences in the Design and Conduct of Surveys in Empirical Software Engineering.

[8]

D. A. Dillman, J. D. Smyth, and L. M. Christian. 2014. Internet, Phone, Mail, and Mixed-Mode Surveys: The Tailored Design Method.

[9]

H. Femmer, D. M. Fernández, S. Wagner, and S. Eder. 2017. Rapid quality assurance with requirements smells. Journal of Systems and Software 123 (2017).

[10]

H. Femmer and A. Vogelsang. 2019. Requirements Quality Is Quality in Use. IEEE Software 36 (2019).

[11]

J. Fischbach, A. Vogelsang, D. Spies, A. Wehrle, M. Junker, and D. Freudenstein. 2020. SPECMATE: Automated Creation of Test Cases from Acceptance Criteria. In ICST.

[12]

V. Gervasi, A. Ferrari, D. Zowghi, and P. Spoletini. 2019. From Software Engineering to Formal Methods and Tools, and Back: Essays Dedicated to Stefania Gnesi on the Occasion of Her 65th Birthday. Chapter Ambiguity in Requirements Engineering: Towards a Unifying Framework.

[13]

S. Hotomski, E. B. Charrada, and M. Glinz. 2016. An Exploratory Study on Handling Requirements and Acceptance Test Documentation in Industry. In RE.

[14]

International Software Testing Qualifications Board. 2019. Certified Tester Specialist Syllabus. https://www.istqb.org/downloads/send/62-acceptance-testing/257-acceptance-testing-specialist-syllabus.html

[15]

ISO 29119 2013. Software and systems engineering - Software testing. Standard. International Organization for Standardization.

[16]

O.Liskin.2015.HowArtifactsSupportandImpedeRequirementsCommunication. In REFSQ, S. A. Fricker and K. Schneider (Eds.).

[17]

D. M. Fernández and S. Wagner. 2015. Naming the pain in requirements engineering: A design for a global family of surveys and first results from Germany. Information and Software Technology 57 (2015).

[18]

P. Mayring. 2014. Qualitative content analysis: theoretical foundation, basic procedures and software solution.

[19]

M. Q. Patton. 1990. Qualitative evaluation and research methods.

[20]

C. Robson. 2002. Real World Research - A Resource for Social Scientists and Practitioner-Researchers.

[21]

K. Schwaber. 1995. SCRUM Development Process. In OOPSLA.

[22]

C. J. Stettina and W. Heijstek. 2011. Necessary and Neglected?: An Empirical Study of Internal Documentation in Agile Software Development Teams. In SIGDOC.

[23]

G. Wagenaar, R. Helms, D. Damian, and S. Brinkkemper. 2015. Artefacts in Agile Software Development. In PROFES.

[24]

G. Wagenaar, S. Overbeek, G. Lucassen, S. Brinkkemper, and K. Schneider. 2018. Working software over comprehensive documentation - Rationales of agile teams for artefacts usage. Journal of Software Engineering Research and Development 6 (2018).

[25]

Y. Wand and R. Y. Wang. 1996. Anchoring Data Quality Dimensions in Ontological Foundations. Commun. ACM 39 (1996).

[26]

R. Y. Wang and D. M. Strong. 1996. Beyond Accuracy: What Data Quality Means to Data Consumers. J. Manage. Inf. Syst. 12 (1996).

Cited By

Gaborov MStojanov ZPopov SStojanov JKavalić MKovač DVecštejn I(2024)Development and Validation of a Theoretical Model for Addressing Problems in Agile Meetings: A Systematic Literature Review and a Qualitative StudyApplied Sciences10.3390/app1421968914:21(9689)Online publication date: 23-Oct-2024
https://doi.org/10.3390/app14219689
Melegati JConboy KGraziotin D(2024)Qualitative Surveys in Software Engineering Research: Definition, Critical Review, and GuidelinesIEEE Transactions on Software Engineering10.1109/TSE.2024.3474173(1-16)Online publication date: 2024
https://doi.org/10.1109/TSE.2024.3474173
Fischbach JFrattini JVogelsang AMendez DUnterkalmsteiner MWehrle AHenao PYousefi PJuricic TRadduenz JWiecher C(2023)Automatic creation of acceptance tests by extracting conditionals from requirementsJournal of Systems and Software10.1016/j.jss.2022.111549197:COnline publication date: 1-Mar-2023
https://dl.acm.org/doi/10.1016/j.jss.2022.111549
Show More Cited By

Index Terms

What Makes Agile Test Artifacts Useful?: An Activity-Based Quality Model from a Practitioners' Perspective
1. General and reference
  1. Cross-computing tools and techniques
    1. Empirical studies
2. Software and its engineering
  1. Software creation and management
    1. Software development process management
      1. Software development methods
        Agile software development

Recommendations

Software test automation practices in agile development environment: an industry experience report
AST '12: Proceedings of the 7th International Workshop on Automation of Software Test

The increased importance of Test Automation in software engineering is very evident considering the number of companies investing in automated testing tools nowadays, with the main aim of preventing defects during the development process. Test ...
Towards Agile Testing for Railway Safety-critical Software
XP '16 Workshops: Proceedings of the Scientific Workshop Proceedings of XP2016

EN 50128 gives a general guidance for testing railway safety-critical software. However, it does not specify how to test safety-critical software in an agile development environment. Based on our observation, agile software development has not been ...
Enabling Agile Testing through Continuous Integration
AGILE '09: Proceedings of the 2009 Agile Conference

A Continuous Integration system is often considered one of the key elements involved in supporting an agile software development and testing environment. As a traditional software tester transitioning to an agile development environment it became clear ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

ESEM '20: Proceedings of the 14th ACM / IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM)

October 2020

412 pages

ISBN:9781450375801

DOI:10.1145/3382494

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 23 October 2020

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

ESEM '20

Sponsor:

SIGSOFT

ESEM '20: ACM / IEEE International Symposium on Empirical Software Engineering and Measurement

October 5 - 9, 2020

Bari, Italy

Acceptance Rates

ESEM '20 Paper Acceptance Rate 26 of 123 submissions, 21%;

Overall Acceptance Rate 130 of 594 submissions, 22%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

8
Total Citations
View Citations
190
Total Downloads

Downloads (Last 12 months)18
Downloads (Last 6 weeks)0

Reflects downloads up to 12 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Gaborov MStojanov ZPopov SStojanov JKavalić MKovač DVecštejn I(2024)Development and Validation of a Theoretical Model for Addressing Problems in Agile Meetings: A Systematic Literature Review and a Qualitative StudyApplied Sciences10.3390/app1421968914:21(9689)Online publication date: 23-Oct-2024
https://doi.org/10.3390/app14219689
Melegati JConboy KGraziotin D(2024)Qualitative Surveys in Software Engineering Research: Definition, Critical Review, and GuidelinesIEEE Transactions on Software Engineering10.1109/TSE.2024.3474173(1-16)Online publication date: 2024
https://doi.org/10.1109/TSE.2024.3474173
Fischbach JFrattini JVogelsang AMendez DUnterkalmsteiner MWehrle AHenao PYousefi PJuricic TRadduenz JWiecher C(2023)Automatic creation of acceptance tests by extracting conditionals from requirementsJournal of Systems and Software10.1016/j.jss.2022.111549197:COnline publication date: 1-Mar-2023
https://dl.acm.org/doi/10.1016/j.jss.2022.111549
Chugh MChugh N(2023)A Deep Drive into Software Development Agile Methodologies for Software Quality AssuranceAgile Software Development10.1002/9781119896838.ch12(235-255)Online publication date: 8-Feb-2023
https://doi.org/10.1002/9781119896838.ch12
Budiman RRaharjo TSuhanto A(2022)Scrum Project Management Challenges and Solutions: Systematic Literature Review2022 IEEE 8th International Conference on Computing, Engineering and Design (ICCED)10.1109/ICCED56140.2022.10010471(1-6)Online publication date: 28-Jul-2022
https://doi.org/10.1109/ICCED56140.2022.10010471
Barraood SMohd HBaharom F(2022)An initial investigation of the effect of quality factors on Agile test case quality through experts’ reviewCogent Engineering10.1080/23311916.2022.20821219:1Online publication date: 6-Jun-2022
https://doi.org/10.1080/23311916.2022.2082121
Sydorov N(2021)Programming Style as an Artefact of a Software Artefacts EcosystemAdvances in Computer Science for Engineering and Education IV10.1007/978-3-030-80472-5_21(244-255)Online publication date: 21-Jul-2021
https://doi.org/10.1007/978-3-030-80472-5_21
Sydorov N(2020)Toward software artifacts ecosystemPROBLEMS IN PROGRAMMING10.15407/pp2020.04.110(110-120)Online publication date: Dec-2020
https://doi.org/10.15407/pp2020.04.110

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents