More Web Proxy on the site http://driver.im/

research-article

Testability and software performance: a systematic mapping study

Authors:

Mohammad Mahdi Hassan,

Birgitta Lindström,

Syed Muhammad Ali Shah,

Sten F. Andler,

Martin BlomAuthors Info & Claims

SAC '16: Proceedings of the 31st Annual ACM Symposium on Applied Computing

Pages 1566 - 1569

https://doi.org/10.1145/2851613.2851978

Published: 04 April 2016 Publication History

Abstract

In most of the research on software testability, functional correctness of the software has been the focus while the evidence regarding testability and non-functional properties such as performance is sporadic. The objective of this study is to present the current state-of-the-art related to issues of importance, types and domains of software under test, types of research, contribution types and design evaluation methods concerning testability and software performance. We find that observability, controllability and testing effort are the main testability issues while timeliness and response time (i.e., time constraints) are the main performance issues in focus. The primary studies in the area use diverse types of software under test within different domains, with realtime systems as being a dominant domain. The researchers have proposed many different methods in the area, however these methods lack implementation in practice.

References

[1]

R. Bache and M. Müllerburg. Measures of testability as a basis for quality assurance. Software Engineering Journal, 5(2):86--92, 1990.

Digital Library

[2]

A. Beer and M. Heindl. Issues in testing dependable event-based systems at a systems integration company. In Proc. of the 2nd Intl. Conf. on Availability, Reliability and Security, 2007.

Digital Library

[3]

R. Birgisson, J. Mellin, and S. F. Andler. Bounds on test effort for event-triggered real-time systems. In Proc. of the 6th Intl. Conf. on Real-Time Computing Systems and Applications. IEEE, 1999.

Digital Library

[4]

M. Bozzano, A. Cimatti, J.-P. Katoen, V. Nguyen, T. Noll, and M. Roveri. The COMPASS Approach: Correctness, modelling and performability of aerospace systems. In B. Buth, G. Rabe, and T. Seyfarth, editors, Computer Safety, Reliability, and Security, volume 5775 of Lecture Notes in Computer Science, pages 173--186. Springer, 2009.

Digital Library

[5]

S. Chanson, A. Loureiro, and S. Vuong. On the design for testability of communication software. In Proc. of the 1993 Int. Test Conf., 1993.

Digital Library

[6]

O. Dias, I. Teixeira, J. Teixeira, L. Becker, and C. Pereira. On identifying and evaluating object architectures for real-time applications. Control Engineering Practice, 9(4):403--409, 2001.

[7]

J. Etkin and J. Zinky. Distributed debugging: Network analysis tools. Microprocessing and Microprogramming, 25(1--5):307--312, 1989.

Digital Library

[8]

R. S. Freedman. Testability of software components. IEEE Tran. on Soft. Eng., 17(6):553--564, 1991.

Digital Library

[9]

A. Groce, G. Holzmann, and R. Joshi. Randomized differential testing As a prelude to formal verification. In Proc. of the 29th Intl. Conf. on Software Engineering. IEEE, 2007.

Digital Library

[10]

H.-G. Groß. A prediction system for evolutionary testability applied to dynamic execution time analysis. Inf. & Software Tech., 43(14):855--862, 2001.

[11]

K. Haller. Mobile testing. SIGSOFT Software Engineering Notes, 38(6):1--8, 2013.

Digital Library

[12]

M. Hassan, W. Afzal, M. Blom, B. Lindström, S. Andler, and S. Eldh. Testability and software robustness: A systematic literature review. In Proc. of 41st Euromicro Conf. on Software Engineering and Advanced Applications (SEAA'15), 2015.

Digital Library

[13]

A. R. Hevner, S. T. March, J. Park, and S. Ram. Design science in information systems research. MIS Quarterly, 28(1):75--105, 2004.

Digital Library

[14]

R. M. Hierons, T.-H. Kim, and H. Ural. On the testability of SDL specifications. Computer Networks, 44(5):681--700, 2004.

Digital Library

[15]

V. Izosimov, G. Guglielmo, M. Lora, G. Pravadelli, F. Fummi, Z. Peng, and M. Fujita. Time-constraint-aware optimization of assertions in embedded software. Journal of Electronic Testing: Theory and Applications, 28(4):469--486, 2012.

Digital Library

[16]

M. Jevtic and M. Damnjanovic. An approach to design for testability in hard real-time systems. In Proc. of the Int. Conf. on Microelectronics, 1997.

[17]

A. Keshk and A. Ibrahim. Ensuring the quality testing of web using a new methodology. In Proc. of the 2007 IEEE Intl. Symposium on Signal Processing and Information Technology, 2007.

[18]

T. M. King, A. A. Allen, Y. Wu, P. J. Clarke, and A. E. Ramirez. A comparative case study on the engineering of self-testable autonomic software. In Proc. of the 8th IEEE Int. Conf. and WS on Eng. of Autonomic and Autonomous Sys., 2011.

Digital Library

[19]

H. Kopetz. Event-triggered versus time-triggered real-time systems. In A. Karshmer and J. Nehmer, editors, Operating Systems of the 90s and Beyond, volume 563 of Lecture Notes in Computer Science, pages 86--101. Springer, 1991.

Digital Library

[20]

H. Kopetz, R. Zainlinger, G. Fohler, H. Kantz, P. Puschner, and W. Schütz. An engineering approach to hard real-time system design, publisher=Springer. In A. van Lamsweerde and A. Fugetta, editors, 3rd European Software Engineering Conf. (ESEC'91), volume 550 of Lecture Notes in Computer Science, pages 166--188. 1991.

Digital Library

[21]

N. Kranitis, G. Xenoulis, D. Gizopoulos, A. Paschalis, and Y. Zorian. Low-cost software-based self-testing of RISC processor cores. In Proc. of the Conf. on Design, Automation and Test in Europe - Volume 1. IEEE, 2003.

Digital Library

[22]

S. Limsoonthrakul, M. N. Dailey, M. Srisupundit, S. Tongphu, and M. Parnichkun. A modular system architecture for autonomous robots based on blackboard and publish-subscribe mechanisms. In Proc. of the IEEE Intl. Conf. on Robotics and Biomimetics, 2009.

Digital Library

[23]

B. Lindström, J. Offutt, and S. F. Andler. Testability of dynamic real-time systems: An empirical study of constrained execution environment implications. In Proc. of the 2008 Intl. Conf. on Software Testing, Verification, and Validation, 2008.

Digital Library

[24]

M. Merdes, R. Malaka, D. Suliman, B. Paech, D. Brenner, and C. Atkinson. Ubiquitous RATs: How resource-aware run-time tests can improve ubiquitous software systems. In Proc. of the 6th Intl. Workshop on Software Engineering and Middleware, New York, NY, USA, 2006. ACM.

Digital Library

[25]

K. Petersen, R. Feldt, S. Mujtaba, and M. Mattsson. Systematic mapping studies in software engineering. In Proc. of the 12th Intl. Conf. on Evaluation and Assessment in Software Engineering, Swinton, UK, UK, 2008. British Computer Society.

Digital Library

[26]

S. Salva and H. Fouchal. Some parameters for timed system testability. In ACS/IEEE Intl. Conf. on Computer Systems and Applications, 2001.

Digital Library

[27]

W. Schütz. On the testability of distributed real-time systems. In Proc. of the 10th Symposium on Reliable Distributed Systems (RDS'91), 1991.

[28]

W. Schütz. Fundamental issues in testing distributed real-time systems. Real-Time Systems, 7(2):129--157, 1994.

Digital Library

[29]

Standards Coordinating Committee of the Computer Society of the IEEE. IEEE Standard Glossary of Software Engineering Terminology, 1990.

[30]

H. Thane and H. Hansson. Testing distributed real-time systems. Microprocessors and Microsystems, 24(9):463--478, 2001.

[31]

J. Vincent, G. King, P. Lay, and J. Kinghorn. Principles of built-in-test for run-time-testability in component-based software systems. Software Quality Journal, 10(2):115--133, 2002.

Digital Library

[32]

J. M. Voas and K. W. Miller. Software testability: The new verification. IEEE Software, 12(3):17--28, 1995.

Digital Library

[33]

R. Wieringa, N. Maiden, N. Mead, and C. Rolland. Requirements engineering paper classification and evaluation criteria: A proposal and a discussion. Requirements Engineering, 11(1):102--107, 2005.

Digital Library

[34]

X. Yingshi, L. Bin, R. Lian, and X. Ping. A study on software architecture of testability experiment verification environment. In Proc. of the 1st Int. Conf. on Maintenance Engineering, 2006.

Cited By

Purohit SSingh SAgarwal MVerma N(2023)A Qualitative and Comprehensive Analysis of Software Testability Metrics and their Trends2023 Second International Conference on Electronics and Renewable Systems (ICEARS)10.1109/ICEARS56392.2023.10085333(1545-1552)Online publication date: 2-Mar-2023
https://doi.org/10.1109/ICEARS56392.2023.10085333
Han XYu TYan G(2023)A systematic mapping study of software performance researchSoftware: Practice and Experience10.1002/spe.318553:5(1249-1270)Online publication date: 2-Jan-2023
https://doi.org/10.1002/spe.3185
Mona J(2022)Software Testability (Its Benefits, Limitations, and Facilitation)Next Generation of Internet of Things10.1007/978-981-19-1412-6_23(287-298)Online publication date: 27-Sep-2022
https://doi.org/10.1007/978-981-19-1412-6_23
Show More Cited By

Index Terms

Testability and software performance: a systematic mapping study
1. Software and its engineering
  1. Software organization and properties
    1. Extra-functional properties

Recommendations

Correlations among Software Testability Metrics
SBQS '20: Proceedings of the XIX Brazilian Symposium on Software Quality

Testability is a relevant characteristic to evaluate software quality and it is addressed in most quality models. This characteristic is directly related to software testing activities and defines how easily a program can be tested. However, ...
Request-Oriented Method of Software Testability Measurement
ITCS '09: Proceedings of the 2009 International Conference on Information Technology and Computer Science - Volume 02

As the quantity of the software testability measurement method is finite there is maybe no appropriate method existed for some measurement requests. To solve this problem a request-oriented method of software testability measurement is proposed. Firstly,...
An aspect-oriented testability framework
RACS '12: Proceedings of the 2012 ACM Research in Applied Computation Symposium

We can specify design constraints of a software system using a formal language such as Object Constraint Language (OCL). However, OCL is not executable and cannot be explicitly used as a testing method. So runtime constraint checking is needed to ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

SAC '16: Proceedings of the 31st Annual ACM Symposium on Applied Computing

April 2016

2360 pages

ISBN:9781450337397

DOI:10.1145/2851613

Conference Chair:
Sascha Ossowski
University Rey Juan Carlos, Spain

Copyright © 2016 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGAPP: ACM Special Interest Group on Applied Computing

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 04 April 2016

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

SAC 2016

Sponsor:

SIGAPP

SAC 2016: Symposium on Applied Computing

April 4 - 8, 2016

Pisa, Italy

Acceptance Rates

SAC '16 Paper Acceptance Rate 252 of 1,047 submissions, 24%;

Overall Acceptance Rate 1,650 of 6,669 submissions, 25%

Upcoming Conference

SAC '25

Sponsor:
sigapp

The 40th ACM/SIGAPP Symposium on Applied Computing

March 31 - April 4, 2025

Catania , Italy

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

11
Total Citations
View Citations
299
Total Downloads

Downloads (Last 12 months)20
Downloads (Last 6 weeks)1

Reflects downloads up to 13 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Purohit SSingh SAgarwal MVerma N(2023)A Qualitative and Comprehensive Analysis of Software Testability Metrics and their Trends2023 Second International Conference on Electronics and Renewable Systems (ICEARS)10.1109/ICEARS56392.2023.10085333(1545-1552)Online publication date: 2-Mar-2023
https://doi.org/10.1109/ICEARS56392.2023.10085333
Han XYu TYan G(2023)A systematic mapping study of software performance researchSoftware: Practice and Experience10.1002/spe.318553:5(1249-1270)Online publication date: 2-Jan-2023
https://doi.org/10.1002/spe.3185
Mona J(2022)Software Testability (Its Benefits, Limitations, and Facilitation)Next Generation of Internet of Things10.1007/978-981-19-1412-6_23(287-298)Online publication date: 27-Sep-2022
https://doi.org/10.1007/978-981-19-1412-6_23
Bluemke IMalanowska A(2021)Software Testing Effort Estimation and Related ProblemsACM Computing Surveys10.1145/344269454:3(1-38)Online publication date: 17-Apr-2021
https://dl.acm.org/doi/10.1145/3442694
Sanchez ADelgado-Perez PMedina-Bulo ISegura S(2020)TANDEM: A Taxonomy and a Dataset of Real-World Performance BugsIEEE Access10.1109/ACCESS.2020.30009288(107214-107228)Online publication date: 2020
https://doi.org/10.1109/ACCESS.2020.3000928
Yasin AFatima RWen LAfzal WAzhar MTorkar R(2020)On Using Grey Literature and Google Scholar in Systematic Literature Reviews in Software EngineeringIEEE Access10.1109/ACCESS.2020.29717128(36226-36243)Online publication date: 2020
https://doi.org/10.1109/ACCESS.2020.2971712
Riboni AGuglielmo LOrru MBraione PDenaro G(2019)Design for Testability of ERMTS Applications2019 IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW)10.1109/ISSREW.2019.00062(128-136)Online publication date: Oct-2019
https://doi.org/10.1109/ISSREW.2019.00062
Bechennec JFaucou SRoux OBrun MGivel L(2018)Testing Real-Time Systems With Runtime EnforcementIEEE Design & Test10.1109/MDAT.2018.279180135:4(31-37)Online publication date: Aug-2018
https://doi.org/10.1109/MDAT.2018.2791801
Felderer MMarculescu BGomes de Oliveira Neto FFeldt RTorkar R(2018)A Testability Analysis Framework for Non-functional Properties2018 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)10.1109/ICSTW.2018.00028(54-58)Online publication date: Apr-2018
https://doi.org/10.1109/ICSTW.2018.00028
Sharma RSaha A(2018)A Systematic Review of Software Testability Measurement Techniques2018 International Conference on Computing, Power and Communication Technologies (GUCON)10.1109/GUCON.2018.8675006(299-303)Online publication date: Sep-2018
https://doi.org/10.1109/GUCON.2018.8675006
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents