Abstract
Integration testing is an integral part of software testing. Prior studies have focused on reducing test cost in integration test order generation. However, there are no studies concerning the testing priorities of critical classes when generating integration test orders. Such priorities greatly affect testing efficiency. In this study, we propose an effective strategy that considers both test cost and efficiency when generating test orders. According to a series of dynamic execution scenarios, the software is mapped into a multi-layer dynamic execution network (MDEN) model. By analyzing the dynamic structural complexity, an evaluation scheme is proposed to quantify the class testing priority with the defined class risk index. Cost—benefit analysis is used to perform cycle-breaking operations, satisfying two principles: assigning higher priorities to higher-risk classes and minimizing the total complexity of test stubs. We also present a strategy to evaluate the effectiveness of integration test order algorithms by calculating the reduction of software risk during their testing process. Experiment results show that our approach performs better across software of different scales, in comparison with the existing algorithms that aim only to minimize test cost. Finally, we implement a tool, ITOsolution, to help practitioners automatically generate test orders.
摘要
集成测试是面向对象软件测试的重要组成部分。传统的类级集成测试顺序研究策略大多围绕如何降低测试成本开展工作, 并未考虑赋予可靠性风险较大的节点较高测试优先级, 从而影响软件测试效率。本文提出一种兼顾测试成本与测试效率的方法生成集成测试序列。根据软件在不同场景下的运行状态, 将其映射成多层动态执行网络 (multi-layer dynamic execution network, MDEN) 。借助该网络模型与概率风险评估方法为软件中每一个类赋予风险权重。利用成本收益分析方法, 在生成测试用例的过程中保证两条原则: 为高风险的类赋予较高权重, 同时最小化测试桩复杂度。在此基础上, 分析测试序列对软件系统总体运行风险的影响, 从而提出评估测试序列优劣的度量方案。通过与现有算法的实验对比分析, 证明所提算法生成的类级集成测试序列能有效降低测试代价。最后, 将所提算法实现为自动生成集成测试序列的开源工具ITOsolution。
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Abdurazik A, Offutt J, 2006. Coupling-based class integration and test order. Proc Int Workshop on Automation of Software Test, p.50–56. https://doi.org/10.1145/1138929.1138940
Al Mannai WI, Lewis TG, 2008. A general defender-attacker risk model for networks. J Risk Finance, 9(3):244–261. https://doi.org/10.1108/15265940810875577
Amland S, 2000. Risk-based testing: risk analysis fundamentals and metrics for software testing including a financial application case study. J Syst Softw, 53(3):287–295. https://doi.org/10.1016/S0164-1212(00)00019-4
Assunção WKG, Colanzi TE, Vergilio SR, et al., 2014. A multi-objective optimization approach for the integration and test order problem. Inform Sci, 267:119–139. https://doi.org/10.1016/j.ins.2013.12.040
Bang L, Aydin A, Bultan T, 2015. Automatically computing path complexity of programs. Proc 10th Joint Meeting on Foundations of Software Engineering, p.61–72. https://doi.org/10.1145/2786805.2786863
Bansal P, Sabharwal S, Sidhu P, 2009. An investigation of strategies for finding test order during integration testing of object oriented applications. Proc Int Conf on Methods and Models in Computer Science, p.1–8. https://doi.org/10.1109/ICM2CS.2009.5397936
Binder RV, 1996. Testing object-oriented software: a survey. Softw Test Verif Reliab, 6(3–4):125–252. https://doi.org/10.1002/(SICI)1099-1689(199609/12)6:3/4<125::AID-STVR121>3.0.CO;2-X
Bowring JF, Rehg JM, Harrold MJ, 2004. Active learning for automatic classification of software behavior. ACM SIGSOFT Softw Eng Notes, 29(4):195–205. https://doi.org/10.1145/1013886.1007539
Briand LC, Jie F, Labiche Y, 2002. Experimenting with genetic algorithms to devise optimal integration test orders. Technical Report, No. SCE-02-03, Carleton University, Ottawa, Canada.
Briand LC, Labiche Y, Wang YH, 2003. An investigation of graph-based class integration test order strategies. IEEE Trans Softw Eng, 29(7):594–607. https://doi.org/10.1109/TSE.2003.1214324
Cai KY, Yin BB, 2009. Software execution processes as an evolving complex network. Inform Sci, 179(12):1903–1928. https://doi.org/10.1016/j.ins.2009.01.011
da Veiga Cabral R, Pozo A, Vergilio SR, 2010. A Pareto ant colony algorithm applied to the class integration and test order problem. Proc 22nd IFIP WG 6.1 Int Conf on Testing Software and Systems, p.16–29. https://doi.org/10.1007/978-3-642-16573-3_3
Deb K, Pratap A, Agarwal S, et al., 2002. A fast and elitist multiobjective genetic algorithm: NSGA-II. IEEE Trans Evol Comput, 6(2):182–197. https://doi.org/10.1109/4235.996017
Floyd RW, 1962. Algorithm 97: shortest path. Commun ACM, 5(6):345. https://doi.org/10.1145/367766.368168
Frankl PG, Weyuker EJ, 2000. Testing software to detect and reduce risk. J Syst Softw, 53(3):275–286. https://doi.org/10.1016/S0164-1212(00)00018-2
Goseva-Popstojanova K, Hassan A, Guedem A, et al., 2003. Architectural-level risk analysis using UML. IEEE Trans Softw Eng, 29(10):946–960. https://doi.org/10.1109/TSE.2003.1237174
Hao D, Zhang L, Zang L, et al., 2016. To be optimal or not in test-case prioritization. IEEE Trans Softw Eng, 42(5):490–505. https://doi.org/10.1109/TSE.2015.2496939
Henry S, Kafura D, 1981. Software structure metrics based on information flow. IEEE Trans Softw Eng, 7(5):510–518. https://doi.org/10.1109/TSE.1981.231113
Huang CY, Lyu MR, 2005. Optimal testing resource allocation, and sensitivity analysis in software development. IEEE Trans Reliab, 54(4):592–603. https://doi.org/10.1109/TR.2005.858099
Jiang SJ, Zhang YM, Li HY, et al., 2011. An approach for inter-class integration test order determination based on coupling measures. Chin J Comput, 34(6):1062–1074. https://doi.org/10.3724/SP.J.1016.2011.01062
Jiang SJ, Zhang M, Zhang YM, et al., 2021. An integration test order strategy to consider control coupling. IEEE Trans Softw Eng, 47(7):1350–1367. https://doi.org/10.1109/TSE.2019.2921965
Jorgensen PC, Erickson C, 1994. Object-oriented integration testing. Commun ACM, 37(9):30–38. https://doi.org/10.1145/182987.182989
Knowles JD, Corne DW, 2000. Approximating the nondominated front using the Pareto archived evolution strategy. Evol Comput, 8(2):149–172. https://doi.org/10.1162/106365600568167
Kung D, Gao J, Hsia P, et al., 1995. A test strategy for object-oriented programs. Proc 19th Annual Int Computer Software and Applications Conf, p.239–244. https://doi.org/10.1109/CMPSAC.1995.524786
Le Traon Y, Jeron T, Jezequel JM, et al., 2000. Efficient object-oriented integration and regression testing. IEEE Trans Reliab, 49(1):12–25. https://doi.org/10.1109/24.855533
Lipow M, 1982. Number of faults per line of code. IEEE Trans Softw Eng, 8(4):437–439. https://doi.org/10.1109/TSE.1982.235579
McCabe TJ, 1976. A complexity measure. IEEE Trans Softw Eng, 2(4):308–320. https://doi.org/10.1109/TSE.1976.233837
Monden A, Hayashi T, Shinoda S, et al., 2013. Assessing the cost effectiveness of fault prediction in acceptance testing. IEEE Trans Reliab, 39(10):1345–1357. https://doi.org/10.1109/TSE.2013.21
Myers CR, 2003. Software systems as complex networks: structure, function, and evolvability of software collaboration graphs. Phys Rev E, 68(2):046116. https://doi.org/10.1103/PhysRevE.68.046116
NASA, 1999. Pyroshock Test Crite. Technical Report, No. NASA-STD-7003A. NASA, Washington, USA.
Sharma C, Sibal R, 2013. Application of different metaheuristic techniques for finding optimal test order during integration testing of object oriented systems and their comparative study. Int J Soft Comput Eng, 3(12):1–19.
Tai KC, Daniels JF, 1999. Interclass test order for object-oriented software. J Obj-Orient Progr, 12(4):18–25.
Vergilio SR, Pozo A, Árias JCG, et al., 2012. Multi-objective optimization algorithms applied to the class integration and test order problem. Int J Softw Tools Technol Transf, 14(4):461–475. https://doi.org/10.1007/s10009-012-0226-1
Walters C, Ludwig D, 1994. Calculation of Bayes posterior probability distributions for key population parameters. Can J Fish Aquat Sci, 51(3):713–722. https://doi.org/10.1139/f94-071
Wang Y, Zhu ZL, Yang B, et al., 2018a. Using reliability risk analysis to prioritize test cases. J Syst Softw, 139:14–31. https://doi.org/10.1016/j.jss.2018.01.033
Wang Y, Zhu ZL, Yu H, et al., 2018b. Risk analysis on multi-granular flow network for software integration testing. IEEE Trans Circ Syst II Expr Briefs, 65(8):1059–1063. https://doi.org/10.1109/TCSII.2017.2775442
Wang ZS, Li BX, Wang LL, et al., 2011. A brief survey on automatic integration test order generation. Proc 23rd Int Conf on Software Engineering & Knowledge Engineering, p.254–257.
Weyuker EJ, 1988. Evaluating software complexity measures. IEEE Trans Softw Eng, 14(9):1357–1365. https://doi.org/10.1109/32.6178
Xu C, Qin Y, Yu P, et al., 2020. Theories and techniques for growing software: paradigm and beyond. Sci Sin Inform, 50(11):1595–1611. https://doi.org/10.1360/SSI-2020-0079
Author information
Authors and Affiliations
Contributions
Fanyi MENG and Ying WANG designed the research and processed the data. Fanyi MENG drafted the paper. Ying WANG helped organize the paper. Hai YU and Zhiliang ZHU revised and finalized the paper.
Corresponding author
Additional information
Compliance with ethics guidelines
Fanyi MENG, Ying WANG, Hai YU, and Zhiliang ZHU declare that they have no conflict of interest.
Project supported by the National Natural Science Foundation of China (Nos. 61902056, 61977014, and 61603082), the Shenyang Young and Middle-Aged Talent Support Program, China (No. ZX20200272), the Fundamental Research Funds for the Central Universities, China (No. N2017011), and the Open Fund of State Key Lab for Novel Software Technology, Nanjing University, China (No. KFKT2021B01)
Supplementary materials for
Rights and permissions
About this article
Cite this article
Meng, F., Wang, Y., Yu, H. et al. Devising optimal integration test orders using cost–benefit analysis. Front Inform Technol Electron Eng 23, 692–714 (2022). https://doi.org/10.1631/FITEE.2100466
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1631/FITEE.2100466