Practical, Automated Scenario-Based Mobile App Testing
Authors: 
Shengcheng Yu, Chunrong Fang, Mingzhe Du, Zimin Ding, Zhenyu Chen, 
Zhendong SuAuthors Info & Claims
Pages 1949 - 1966
Abstract
The importance of mobile application (app) quality assurance is increasing with the rapid development of the mobile Internet. Automated test generation approaches, as a dominant direction of app quality assurance, follow specific models or strategies, targeting at optimizing the code coverage. Such approaches lead to a huge gap between testing execution and app business logic. Test scripts developed by human testers consider business logic by focusing on testing scenarios. Due to the GUI-intensive feature of mobile apps, human testers always understand app GUI to organize test scripts for scenarios. This inspires us to utilize domain knowledge from app GUI understanding for scenario-based test generation. In this paper, we propose a novel approach, <sc>ScenTest</sc>, for scenario-based mobile app testing with event knowledge graph (EKG) via GUI image understanding. <sc>ScenTest</sc> tries to start automated testing by imitating human practices and integrating domain knowledge into scenario-based mobile app testing, realizing fully automated testing on target testing scenarios for the first time. <sc>ScenTest</sc> extracts four kinds of entities and five kinds of corresponding relationships from crowdsourced test reports, where the test events and app GUI information are presented, and constructs the EKGs for specific scenarios. Then, <sc>ScenTest</sc> conducts test generation for specific scenarios on different apps with the guidance of EKG with the combination consideration of app current state and testing context. We conduct an evaluation on <sc>ScenTest</sc> on different aspects. The results show that the test generation of <sc>ScenTest</sc> on the basis of EKG is effective, and <sc>ScenTest</sc> reveals 150+ distinct real-world bugs in specific scenarios compared with representative baselines.
References
[1]
F. Behrang and A. Orso, “Test migration between mobile apps with similar functionality,” in Proc. 34th IEEE/ACM Int. Conf. Automated Softw. Eng. (ASE), Piscataway, NJ, USA: IEEE Press, 2019, pp. 54–65.
[2]
S. Talebipour, Y. Zhao, L. Dojcilović, C. Li, and N. Medvidović, “UI test migration across mobile platforms,” in Proc. 36th IEEE/ACM Int. Conf. Automated Softw. Eng. (ASE), Piscataway, NJ, USA: IEEE Press, 2021, pp. 756–767.
[3]
T. Xu et al., “GUIDER: GUI structure and vision co-guided test script repair for android apps,” in Proc. 30th ACM SIGSOFT Int. Symp. Softw. Testing Anal., 2021, pp. 191–203.
[4]
S. Yu, C. Fang, Y. Yun, and Y. Feng, “Layout and image recognition driving cross-platform automated mobile testing,” in Proc. IEEE/ACM 43rd Int. Conf. Softw. Eng. (ICSE), Piscataway, NJ, USA: IEEE Press, 2021, pp. 1561–1571.
[5]
Y.-M. Baek and D.-H. Bae, “Automated model-based android GUI testing using multi-level GUI comparison criteria,” in Proc. 31st IEEE/ACM Int. Conf. Automated Softw. Eng., 2016, pp. 238–249.
[6]
“UI/Application exerciser monkey.” Google. Accessed: Jun. 2024. [Online]. Available: https://developer.android.com/studio/test/other-testing-tools/monkey
[7]
A. Machiry, R. Tahiliani, and M. Naik, “Dynodroid: An input generation system for android apps,” in Proc. 9th Joint Meeting Found. Softw. Eng., 2013, pp. 224–234.
[8]
K. Mao, M. Harman, and Y. Jia, “Sapienz: Multi-objective automated testing for android applications,” in Proc. 25th Int. Symp. Softw. Testing Anal., 2016, pp. 94–105.
[9]
T. Su et al., “Guided, stochastic model-based GUI testing of android apps,” in Proc. 11th Joint Meeting Found. Softw. Eng., 2017, pp. 245–256.
[10]
D. Adamo, M. K. Khan, S. Koppula, and R. Bryce, “Reinforcement learning for android GUI testing,” in Proc. 9th ACM SIGSOFT Int. Workshop Automating TEST Case Des., Selection, Eval., 2018, pp. 2–8.
[11]
M. Pan, A. Huang, G. Wang, T. Zhang, and X. Li, “Reinforcement learning based curiosity-driven testing of android applications,” in Proc. 29th ACM SIGSOFT Int. Symp. Softw. Testing Anal., 2020, pp. 153–164.
[12]
A. Romdhana, A. Merlo, M. Ceccato, and P. Tonella, “Deep reinforcement learning for black-box testing of android apps,” ACM Trans. Softw. Eng. Methodol., Nov. 2021, to be published.
[13]
S. R. Choudhary, A. Gorla, and A. Orso, “Automated test input generation for android: Are we there yet?(e),” in Proc. 30th IEEE/ACM Int. Conf. Automated Softw. Eng. (ASE), Piscataway, NJ, USA: IEEE Press, 2015, pp. 429–440.
[14]
X. Chen, S. Jia, and Y. Xiang, “A review: Knowledge reasoning over knowledge graph,” Expert Syst. Appl., vol. 141, 2020, Art. no.
[15]
S. Yu, C. Fang, Z. Cao, X. Wang, T. Li, and Z. Chen, “Prioritize crowdsourced test reports via deep screenshot understanding,” in Proc. IEEE/ACM 43rd Int. Conf. Softw. Eng. (ICSE), Piscataway, NJ, USA: IEEE Press, 2021, pp. 946–956.
[16]
Y. Tian, S. Yu, C. Fang, and P. Li, “FuRong: Fusing report of automated android testing on multi-devices,” in Proc. ACM/IEEE 42nd Int. Conf. Softw. Eng.: Companion Proc., 2020, pp. 49–52.
[17]
Z. Zhang et al., “UniRLTest: Universal platform-independent testing with reinforcement learning via image understanding,” in Proc. 31th ACM SIGSOFT Int. Symp. Softw. Testing Anal., 2022, pp. 805–808.
[18]
X. Zhao, H. Chen, Z. Xing, and C. Miao, “Brain-inspired search engine assistant based on knowledge graph,” IEEE Trans. Neural Netw. Learn. Syst., vol. 34, no. 8, pp. 4386–4400, Aug. 2023.
[19]
C. Guo, T. He, W. Yuan, Y. Guo, and R. Hao, “Crowdsourced requirements generation for automatic testing via knowledge graph,” in Proc. 29th ACM SIGSOFT Int. Symp. Softw. Testing Anal., 2020, pp. 545–548.
[20]
R. Gao, Y. Wang, Y. Feng, Z. Chen, and W. Eric Wong, “Successes, challenges, and rethinking–an industrial investigation on crowdsourced mobile application testing,” Empirical Softw. Eng., vol. 24, no. 2, pp. 537–561, 2019.
[21]
S. Zhang, L. Wang, K. Sun, and X. Xiao, “A practical Chinese dependency parser based on a large-scale dataset,” 2020,.
[22]
M. Zhu, Y. Zhang, W. Chen, M. Zhang, and J. Zhu, “Fast and accurate shift-reduce constituent parsing,” in Proc. 51st Annu. Meeting Assoc. Comput. Linguistics (vol. 1 Long Papers), 2013, pp. 434–443.
[23]
J. Chen et al., “Object detection for graphical user interface: Old fashioned or deep learning or a combination?” in Proc. 28th ACM Joint Meeting Eur. Softw. Eng. Conf. Symp. Found. Softw. Eng., 2020, pp. 1202–1214.
[24]
P. Liu, L. Li, Y. Zhao, X. Sun, and J. Grundy, “AndroZooOpen: Collecting large-scale open source android apps for the research community,” in Proc. 17th Int. Conf. Mining Softw. Repositories, 2020, pp. 548–552.
[25]
S. Guan et al., “What is event knowledge graph: A survey,” IEEE Trans. Knowl. Data Eng., vol. 35, no. 7, pp. 7569–7589, Jul. 2023.
[26]
A. Nayak, V. Kesri, and R. K. Dubey, “Knowledge graph based automated generation of test cases in software engineering,” in Proc. 7th ACM IKDD CoDS 25th COMAD, 2020, pp. 289–295.
[27]
Y. Zhang and X. Li, “The scheme of security requirement acquisition based on knowledge graph,” in Proc. Int. Symp. Theor. Aspects Softw. Eng. (TASE), Piscataway, NJ, USA: IEEE Press, 2020, pp. 225–231.
[28]
Z. Jiang, C. Chi, and Y. Zhan, “Research on medical question answering system based on knowledge graph,” IEEE Access, vol. 9, pp. 21094–21101, 2021.
[29]
L. Huang, Y. Duan, X. Sun, Z. Lin, and C. Zhu, “Enhancing UML class diagram abstraction with knowledge graph,” in Proc. Int. Conf. Intell. Data Eng. Automated Learn., Berlin, Germany: Springer, 2016, pp. 606–616.
[30]
Y. Zhang, J. Kang, and W. Dai, “Non-functional requirements elicitation based on domain knowledge graph for automatic code generation of industrial cyber-physical systems,” in Proc. 47th Annu. Conf. IEEE Ind. Electron. Soc., Piscataway, NJ, USA: IEEE Press, 2021, pp. 1–6.
[31]
W. Cheng, X. Zhu, and W. Hu, “Conflict-aware inference of Python compatible runtime environments with domain knowledge graph,” 2022,.
[32]
L. Wang, X. Sun, J. Wang, Y. Duan, and B. Li, “Construct bug knowledge graph for bug resolution,” in Proc. IEEE/ACM 39th Int. Conf. Softw. Eng. Companion (ICSE-C), Piscataway, NJ, USA: IEEE Press, 2017, pp. 189–191.
[33]
C. Zhou, “Intelligent bug fixing with software bug knowledge graph,” in Proc. 26th ACM Joint Meeting Eur. Softw. Eng. Conf. Symp. Found. Softw. Eng., 2018, pp. 944–947.
[34]
W. Yang, F. Deng, S. Ma, L. Wu, Z. Sun, and C. Hu, “Test case reuse based on software testing knowledge graph and collaborative filtering recommendation algorithm,” in Proc. IEEE 21st Int. Conf. Softw. Qual., Rel. Secur. Companion (QRS-C), Piscataway, NJ, USA: IEEE Press, 2021, pp. 67–76.
[35]
Y. Liu, M. Liu, X. Peng, C. Treude, Z. Xing, and X. Zhang, “Generating concept based API element comparison using a knowledge graph,” in Proc. 35th IEEE/ACM Int. Conf. Automated Softw. Eng., 2020, pp. 834–845.
[36]
H. Yin, Y. Zheng, Y. Sun, and G. Huang, “An API learning service for inexperienced developers based on API knowledge graph,” in Proc. IEEE Int. Conf. Web Services (ICWS), Piscataway, NJ, USA: IEEE Press, 2021, pp. 251–261.
[37]
W. Ke, C. Wu, X. Fu, C. Gao, and Y. Song, “Interpretable test case recommendation based on knowledge graph,” in Proc. IEEE 20th Int. Conf. Softw. Qual., Rel. Secur. (QRS), Piscataway, NJ, USA: IEEE Press, 2020, pp. 489–496.
[38]
B. Kwapong and K. Fletcher, “A knowledge graph based framework for web API recommendation,” in Proc. IEEE World Congress Services, vol. 2642, Piscataway, NJ, USA: IEEE Press, 2019, pp. 115–120.
[39]
X. Li, S. Yu, L. Sun, Y. Liu, and C. Fang, “Towards effective bug reproduction for mobile applications,” in Proc. 10th Int. Conf. Dependable Syst. Their Appl. (DSA), Piscataway, NJ, USA: IEEE Press, 2023, pp. 114–125.
[40]
Y. Zhao et al., “ReCDroid: Automatically reproducing android application crashes from bug reports,” in Proc. IEEE/ACM 41st Int. Conf. Softw. Eng. (ICSE), Piscataway, NJ, USA: IEEE Press, 2019, pp. 128–139.
[41]
Y. Zhao et al., “ReCDroid+: Automated end-to-end crash reproduction from bug reports for android apps,” ACM Trans. Softw. Eng. Methodol., vol. 31, no. 3, pp. 1–33, 2022.
[42]
M. Fazzini, M. Prammer, M. d’Amorim, and A. Orso, “Automatically translating bug reports into test cases for mobile apps,” in Proc. 27th ACM SIGSOFT Int. Symp. Softw. Testing Anal., 2018, pp. 141–152.
[43]
S. Feng and C. Chen, “GIFdroid: Automated replay of visual bug reports for android apps,” in Proc. 44th Int. Conf. Softw. Eng., 2022, pp. 1045–1057.
[44]
O. El Ariss, D. Xu, S. Dandey, B. Vender, P. McClean, and B. Slator, “A systematic capture and replay strategy for testing complex GUI based Java applications,” in Proc. 7th Int. Conf. Inf. Technol.: New Gener., 2010, pp. 1038–1043.
[45]
L. Gomez, I. Neamtiu, T. Azim, and T. Millstein, “RERAN: Timing-and touch-sensitive record and replay for android,” in Proc. 35th Int. Conf. Softw. Eng., 2013, pp. 72–81.
[46]
W. Lam et al., “Record and replay for android: Are we there yet in industrial cases?” in Proc. 11th joint Meeting Found. Softw. Eng., 2017, pp. 854–859.
[47]
A. Rau, J. Hotzkow, and A. Zeller, “Poster: Efficient GUI test generation by learning from tests of other apps,” in Proc. IEEE/ACM 40th Int. Conf. Softw. Eng.: Companion (ICSE-Companion), Piscataway, NJ, USA: IEEE Press, 2018, pp. 370–371.
[48]
J. Liang, S. Wang, X. Deng, and Y. Liu, “RIDA: Cross-app record and replay for android,” in Proc. IEEE Conf. Softw. Testing, Verification Validation, 2023, pp. 246–257.
[49]
F. Behrang and A. Orso, “Automated test migration for mobile apps,” in Proc. 40th Int. Conf. Softw. Eng.: Companion Proc., 2018, pp. 384–385.
[50]
X. Xiao, X. Wang, Z. Cao, H. Wang, and P. Gao, “ICONINTENT: Automatic identification of sensitive UI widgets based on icon classification for android apps,” in Proc. IEEE/ACM 41st Int. Conf. Softw. Eng. (ICSE), Piscataway, NJ, USA: IEEE Press, 2019, pp. 257–268.
[51]
Z. Liu, C. Chen, J. Wang, Y. Huang, J. Hu, and Q. Wang, “Owl eyes: Spotting UI display issues via visual understanding,” in Proc. 35th IEEE/ACM Int. Conf. Automated Softw. Eng. (ASE), Piscataway, NJ, USA: IEEE Press, 2020, pp. 398–409.
[52]
T. A. Nguyen and C. Csallner, “Reverse engineering mobile application user interfaces with REMAUI (t),” in Proc. 30th IEEE/ACM Int. Conf. Automated Softw. Eng. (ASE), Piscataway, NJ, USA: IEEE Press, 2015, pp. 248–259.
[53]
J. Chen et al., “Unblind your apps: Predicting natural-language labels for mobile GUI components by deep learning,” in Proc. IEEE/ACM 42nd Int. Conf. Softw. Eng. (ICSE), Piscataway, NJ, USA: IEEE Press, 2020, pp. 322–334.
[54]
T.-H. Chang, T. Yeh, and R. C. Miller, “GUI testing using computer vision,” in Proc. SIGCHI Conf. Human Factors Comput. Syst., 2010, pp. 1535–1544.
[55]
S. Chen, L. Fan, T. Su, L. Ma, Y. Liu, and L. Xu, “Automated cross-platform GUI code generation for mobile apps,” in Proc. IEEE 1st Int. Workshop Artif. Intell. Mobile (AI4Mobile), Piscataway, NJ, USA: IEEE Press, 2019, pp. 13–16.
[56]
T. Beltramelli, “pix2code: Generating code from a graphical user interface screenshot,” in Proc. ACM SIGCHI Symp. Eng. Interactive Comput. Syst., 2018, pp. 1–6.
[57]
K. Moran, C. Bernal-Cárdenas, M. Curcio, R. Bonett, and D. Poshyvanyk, “Machine learning-based prototyping of graphical user interfaces for mobile apps,” IEEE Trans. Softw. Eng., vol. 46, no. 2, pp. 196–221, 2018.
[58]
M. Pan, T. Xu, Y. Pei, Z. Li, T. Zhang, and X. Li, “GUI-guided test script repair for mobile apps,” IEEE Trans. Softw. Eng., vol. 48, no. 3, pp. 910–929, Mar. 2022.
[59]
Y. Li, Z. Yang, Y. Guo, and X. Chen, “Humanoid: A deep learning-based approach to automated black-box android app testing,” in Proc. 34th IEEE/ACM Int. Conf. Automated Softw. Eng. (ASE), Piscataway, NJ, USA: IEEE Press, 2019, pp. 1070–1073.
[60]
C. Chen, T. Su, G. Meng, Z. Xing, and Y. Liu, “From UI design image to GUI skeleton: A neural machine translator to bootstrap mobile GUI implementation,” in Proc. 40th Int. Conf. Softw. Eng., 2018, pp. 665–676.
[61]
J. Chen et al., “Wireframe-based UI design search through image autoencoder,” ACM Trans. Softw. Eng. Methodol., vol. 29, no. 3, pp. 1–31, 2020.
Recommendations
Testing cross-platform mobile app development frameworks
ASE '15: Proceedings of the 30th IEEE/ACM International Conference on Automated Software EngineeringMobile app developers often wish to make their apps available on a wide variety of platforms, e.g., Android, iOS, and Windows devices. Each of these platforms uses a different programming environment, each with its own language and APIs for app ...
Learning-Replay Based Automated Robotic Testing for Mobile App
Record-replay testing is widely used in mobile app testing as an automated testing method. However, the current record-replay methods are closely dependent on the internal information of the device or app under test. Due to the diversity of mobile devices ...
Survey on Crowd-based Mobile App Testing
ICMLC '19: Proceedings of the 2019 11th International Conference on Machine Learning and ComputingIn recent years, mobile applications market develop rapidly, especially in China. A large number of mobile applications are developed and applied in various fields, forming a huge market. The crowd-based mobile app testing provides a new method for ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
0098-5589 © 2024 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See https://www.ieee.org/publications/rights/index.html for more information.
Publisher
IEEE Press
Publication History
Published: 14 June 2024
Qualifiers
- Research-article
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 0Total Downloads
- Downloads (Last 12 months)0
- Downloads (Last 6 weeks)0
Reflects downloads up to 01 Jan 2025