Towards a Robust Waiting Strategy for Web GUI Testing for an Industrial Software System
Authors: 
Haonan Zhang, Lizhi Liao, Zishuo Ding, Weiyi Shang, Nidhi Narula, Catalin Sporea, Andrei Toma, Sarah SajediAuthors Info & Claims
Abstract
Automated web GUI testing has been widely adopted since manual testing is time-consuming and tedious. Waiting strategy plays a vital role in automated web GUI testing since it significantly impacts the testing performance. Though important, little focus has been set on the waiting strategies in web GUI testing. Existing waiting strategies either wait for a predetermined time, which is not reliable in a dynamic environment, or only wait for a specific condition to be verified, which is often not robust enough to handle the complicated testing scenarios. In this work, we introduce a robust waiting strategy. Instead of waiting for a predetermined time or waiting for the availability of a particular element, our approach waits for a desired state to reach. This is achieved by capturing the Document Object Models (DOM) at the desired point, followed by an offline analysis to identify the differences between the DOMs associated with every two consecutive test actions. Such differences are used to determine the appropriate waiting time when automatically generating tests. Evaluation results with an industrial web application indicate that our approach produces more robust tests than the conventional waiting strategies used in web GUI testing. Furthermore, our generated tests are more representative of the recorded usage scenarios and are efficient with low overhead in test execution time.
References
[1]
U.S. Environmental Protection Agency. 2024. Toxics Release Inventory (TRI) Program. Retrieved January 16, 2024 from https://www.epa.gov/toxics-release-inventory-tri-program
[2]
Emil Alégroth and Robert Feldt. 2017. On the Long-Term Use of Visual Gui Testing in Industrial Practice: A Case Study. Empirical Softw. Engg. 22, 6 (dec 2017), 2937--2971.
[3]
Sacha Brisset, Romain Rouvoy, Lionel Seinturier, and Renaud Pawlak. 2022. Erratum: Leveraging Flexible Tree Matching to repair broken locators in web automation scripts. Inf. Softw. Technol. 144 (2022), 106754.
[4]
Sacha Brisset, Romain Rouvoy, Lionel Seinturier, and Renaud Pawlak. 2023. SFTM: Fast matching of web pages using Similarity-based Flexible Tree Matching. Information Systems 112 (2023), 102126.
[5]
Michael Butkiewicz, Daimeng Wang, Zhe Wu, Harsha V. Madhyastha, and Vyas Sekar. 2015. Klotski: Reprioritizing Web Content to Improve User Experience on Mobile Devices. In 12th USENIX Symposium on Networked Systems Design and Implementation (NSDI 15). USENIX Association, Oakland, CA, 439--453.
[6]
Shauvik Roy Choudhary, Dan Zhao, Husayn Versee, and Alessandro Orso. 2011. WATER: Web Application TEst Repair. In Proceedings of the First International Workshop on End-to-End Test Script Engineering (Toronto, Ontario, Canada) (ETSE '11). Association for Computing Machinery, New York, NY, USA, 24--29.
[7]
Laurent Christophe, Reinout Stevens, Coen De Roover, and Wolfgang De Meuter. 2014. Prevalence and Maintenance of Automated Functional Tests for Web Applications. In 2014 IEEE International Conference on Software Maintenance and Evolution. 141--150.
[8]
N. Cliff. 1996. Ordinal Methods for Behavioral Data Analysis. Erlbaum.
[9]
Biplab Deka, Zifeng Huang, Chad Franzen, Joshua Hibschman, Daniel Afergan, Yang Li, Jeffrey Nichols, and Ranjitha Kumar. 2017. Rico: A Mobile App Dataset for Building Data-Driven Design Applications. In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology (Québec City, QC, Canada) (UIST '17). Association for Computing Machinery, New York, NY, USA, 845--854.
[10]
Felix Dobslaw, Robert Feldt, David Michaëlsson, Patrik Haar, Francisco Gomes de Oliveira Neto, and Richard Torkar. 2019. Estimating Return on Investment for GUI Test Automation Frameworks. In 2019 IEEE 30th International Symposium on Software Reliability Engineering (ISSRE). 271--282.
[11]
Selenium Project Documentation. 2023. No Such Element Exception. Retrieved August 21, 2023 from https://www.selenium.dev/documentation/webdriver/troubleshooting/errors/#no-such-element-exception
[12]
Moritz Eck, Fabio Palomba, Marco Castelluccio, and Alberto Bacchelli. 2019. Understanding Flaky Tests: The Developer's Perspective. In Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (Tallinn, Estonia) (ESEC/FSE 2019). Association for Computing Machinery, New York, NY, USA, 830--840.
[13]
Environment and Natural Resources. 2024. National Pollutant Release Inventory. Retrieved January 16, 2024 from https://www.canada.ca/en/services/environment/pollution-waste-management/national-pollutant-release-inventory.html
[14]
Sidong Feng, Haochuan Lu, Ting Xiong, Yuetang Deng, and Chunyang Chen. 2023. Towards Efficient Record and Replay: A Case Study in WeChat. In Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (San Francisco, CA, USA) (ESEC/FSE 2023). Association for Computing Machinery, New York, NY, USA, 1681--1692.
[15]
Sidong Feng, Mulong Xie, and Chunyang Chen. 2023. Efficiency Matters: Speeding Up Automated Testing with GUI Rendering Inference. In Proceedings of the 45th International Conference on Software Engineering (Melbourne, Victoria, Australia) (ICSE '23). IEEE Press, 906--918.
[16]
Apache Software Foundation. 2023. Apache FreeMarker is a template engine to generate text output based on templates and changing data. Retrieved August 21, 2023 from https://freemarker.apache.org/index.html
[17]
Mark Grechanik, Qing Xie, and Chen Fu. 2009. Maintaining and evolving GUI-directed test scripts. In 2009 IEEE 31st International Conference on Software Engineering. 408--418.
[18]
Tianxiao Gu, Chengnian Sun, Xiaoxing Ma, Chun Cao, Chang Xu, Yuan Yao, Qirun Zhang, Jian Lu, and Zhendong Su. 2019. Practical GUI Testing of Android Applications Via Model Abstraction and Refinement. In 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE). 269--280.
[19]
Mouna Hammoudi, Gregg Rothermel, and Andrea Stocco. 2016. WATERFALL: an incremental approach for repairing record-replay tests of web applications. In Proceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering (Seattle, WA, USA) (FSE 2016). Association for Computing Machinery, New York, NY, USA, 751--762.
[20]
Mouna Hammoudi, Gregg Rothermel, and Paolo Tonella. 2016. Why do Record/Replay Tests of Web Applications Break?. In 2016 IEEE International Conference on Software Testing, Verification and Validation (ICST). 180--190.
[21]
Jonathan Hedley. 2023. JSoup is a Java library that simplifies working with real-world HTML and XML. Retrieved August 21, 2023 from https://jsoup.org/
[22]
Selenium IDE. 2023. Open source record and playback test automation for the web. Retrieved August 21, 2023 from https://www.selenium.dev/selenium-ide/
[23]
Javaria Imtiaz, Muhammad Zohaib Iqbal, and Muhammad Uzair Khan. 2021. An automated model-based approach to repair test suites of evolving web applications. J. Syst. Softw. 171 (2021), 110841.
[24]
Ranjitha Kumar, Jerry O. Talton, Salman Ahmad, Tim Roughgarden, and Scott R. Klemmer. 2011. Flexible Tree Matching. In Proceedings of the Twenty-Second International Joint Conference on Artificial Intelligence - Volume Volume Three (Barcelona, Catalonia, Spain) (IJCAI'11). AAAI Press, 2674--2679.
[25]
Christoph Laaber, Joel Scheuner, and Philipp Leitner. 2019. Software Microbench-marking in the Cloud. How Bad is It Really? Empirical Softw. Engg. 24, 4 (Aug. 2019), 2469--2508.
[26]
Maurizio Leotta, Diego Clerissi, Filippo Ricca, and Paolo Tonella. 2013. Capture-replay vs. programmable web testing: An empirical assessment during test case evolution. In 2013 20th Working Conference on Reverse Engineering (WCRE). 272--281.
[27]
Maurizio Leotta, Boni García, Filippo Ricca, and Jim Whitehead. 2023. Challenges of End-to-End Testing with Selenium WebDriver and How to Face Them: A Survey. In 2023 IEEE Conference on Software Testing, Verification and Validation (ICST). 339--350.
[28]
Maurizio Leotta, Filippo Ricca, and Paolo Tonella. 2021. Sidereal: Statistical adaptive generation of robust locators for web testing. Softw. Test. Verification Reliab. 31, 3 (2021).
[29]
Maurizio Leotta, Andrea Stocco, Filippo Ricca, and Paolo Tonella. 2015. Using Multi-Locators to Increase the Robustness of Web Test Cases. In 8th IEEE International Conference on Software Testing, Verification and Validation, ICST 2015, Graz, Austria, April 13--17, 2015. IEEE Computer Society, 1--10.
[30]
Maurizio Leotta, Andrea Stocco, Filippo Ricca, and Paolo Tonella. 2016. Robula+: an algorithm for generating robust XPath locators for web testing. J. Softw. Evol. Process. 28, 3 (2016), 177--204.
[31]
Yuanchun Li, Ziyue Yang, Yao Guo, and Xiangqun Chen. 2017. DroidBot: a lightweight UI-Guided test input generator for android. In 2017 IEEE/ACM 39th International Conference on Software Engineering Companion (ICSE-C). 23--26.
[32]
Lizhi Liao, Heng Li, Weiyi Shang, Catalin Sporea, Andrei Toma, and Sarah Sajedi. 2023. Adapting Performance Analytic Techniques in a Real-World Database-Centric System: An Industrial Experience Report. In Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (San Francisco, CA, USA) (ESEC/FSE 2023). Association for Computing Machinery, New York, NY, USA, 1855--1866.
[33]
Jun-Wei Lin and Sam Malek. 2022. GUI Test Transfer from Web to Android. In 2022 IEEE Conference on Software Testing, Verification and Validation (ICST). 1--11.
[34]
Yuanzhang Lin, Guoyao Wen, and Xiang Gao. 2023. Automated Fixing of Web UI Tests via Iterative Element Matching. In 38th IEEE/ACM International Conference on Automated Software Engineering, ASE 2023, Luxembourg, September 11--15, 2023. IEEE, 1188--1199.
[35]
Xinyue Liu, Zihe Song, Weike Fang, Wei Yang, and Weihang Wang. 2024. WEFix: Intelligent Automatic Generation of Explicit Waits for Efficient Web End-to-End Flaky Tests. In Proceedings of the ACM on Web Conference 2024 (Singapore, Singapore) (WWW '24). Association for Computing Machinery, New York, NY, USA, 3043--3052.
[36]
Zhenyue Long, Guoquan Wu, Xiaojiang Chen, Wei Chen, and Jun Wei. 2020. WebRR: Self-Replay Enhanced Robust Record/Replay for Web Application Testing. In Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (Virtual Event, USA) (ESEC/FSE 2020). Association for Computing Machinery, New York, NY, USA, 1498--1508.
[37]
Qingzhou Luo, Farah Hariri, Lamyaa Eloussi, and Darko Marinov. 2014. An Empirical Analysis of Flaky Tests. In Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering (Hong Kong, China) (FSE 2014). Association for Computing Machinery, New York, NY, USA, 643--653.
[38]
OpenSTF Minicap. 2023. Stream real-time screen capture data out of Android devices. Retrieved August 21, 2023 from https://github.com/openstf/minicap
[39]
UI/Application Exerciser Monkey. 2023. UI/Application Exerciser Monkey. Retrieved August 21, 2023 from https://developer.android.com/studio/test/other-testing-tools/monkey
[40]
Nadim Nachar. 2008. The Mann-Whitney U: A Test for Assessing Whether Two Independent Samples Come from the Same Distribution. Tutorials in Quantitative Methods for Psychology 4 (03 2008).
[41]
Michel Nass, Emil Alégroth, Robert Feldt, Maurizio Leotta, and Filippo Ricca. 2023. Similarity-based Web Element Localization for Robust Test Automation. ACM Trans. Softw. Eng. Methodol. 32, 3, Article 75 (apr 2023), 30 pages.
[42]
Michel Nass, Emil Alégroth, and Robert Feldt. 2021. Why many challenges with GUI test automation (will) remain. Information and Software Technology 138 (2021), 106625.
[43]
Dario Olianas, Maurizio Leotta, and Filippo Ricca. 2022. SleepReplacer: a novel tool-based approach for replacing thread sleeps in selenium WebDriver test code. Softw. Qual. J. 30, 4 (2022), 1089--1121.
[44]
Mateusz Pawlik and Nikolaus Augsten. 2015. Efficient Computation of the Tree Edit Distance. ACM Trans. Database Syst. 40, 1, Article 3 (mar 2015), 40 pages.
[45]
Kai Presler-Marshall, Eric Horton, Sarah Heckman, and Kathryn Stolee. 2019. Wait, Wait. No, Tell Me. Analyzing Selenium Configuration Effects on Test Flakiness. In 2019 IEEE/ACM 14th International Workshop on Automation of Software Test (AST). 7--13.
[46]
Xiaofang Qi, Xiang Qian, and Yanhui Li. 2023. Semantic Test Repair for Web Applications. In Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (San Francisco, CA, USA) (ESEC/FSE 2023). Association for Computing Machinery, New York, NY, USA, 1190--1202.
[47]
Filippo Ricca, Maurizio Leotta, and Andrea Stocco. 2019. Chapter Three - Three Open Problems in the Context of E2E Web Testing and a Vision: NEONATE. Adv. Comput. 113 (2019), 89--133.
[48]
Alan Romano, Zihe Song, Sampath Grandhi, Wei Yang, and Weihang Wang. 2021. An Empirical Analysis of UI-Based Flaky Tests. In Proceedings of the 43rd International Conference on Software Engineering (Madrid, Spain) (ICSE '21). IEEE Press, 1585--1597.
[49]
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. 2018. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition. 4510--4520.
[50]
Selenium. 2023. Selenium. Retrieved August 21, 2023 from https://www.selenium.dev
[51]
Thread Sleep. 2023. Thread Sleep in Java. Retrieved August 21, 2023 from https://docs.oracle.com/javase/8/docs/api/java/lang/Thread.html#sleep-long-int-
[52]
Andrea Stocco, Rahulkrishna Yandrapally, and Ali Mesbah. 2018. Visual web test repair. In Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (Lake Buena Vista, FL, USA) (ESEC/FSE 2018). Association for Computing Machinery, New York, NY, USA, 503--514.
[53]
Explicit Waits. 2023. Selenium Explicit Waits. Retrieved August 21, 2023 from https://www.selenium.dev/documentation/webdriver/waits/#explicit-waits
[54]
Selenium WebDriver. 2023. Selenium WebDriver. Retrieved August 21, 2023 from https://www.selenium.dev/documentation/webdriver/
[55]
Zhuolin Xu, Yuanzhang Lin, Qiushi Li, and Shin Hwei Tan. 2023. Guiding ChatGPT to Fix Web UI Tests via Explanation-Consistency Checking. CoRR abs/2312.05778 (2023). arXiv:2312.05778
[56]
Rahulkrishna Yandrapally, Andrea Stocco, and Ali Mesbah. 2020. Near-Duplicate Detection in Web App Model Inference. In Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering (Seoul, South Korea) (ICSE '20). Association for Computing Machinery, New York, NY, USA, 186--197.
Index Terms
- Towards a Robust Waiting Strategy for Web GUI Testing for an Industrial Software System
Recommendations
GUI testing assisted by human knowledge
HighlightsAn empirical study was conducted to compare manual random testing and manual functional testing.The complementary of random testing and functional testing is studied.The influence of test case length on the effectiveness of testing is ...
Enhancing the Automation of GUI Testing
ICSIE '19: Proceedings of the 8th International Conference on Software and Information EngineeringGUI testing is one of the most important and significant testing approaches among all different software testing techniques. Most software errors are captured and detected through the software GUI layer. Manual testing for GUIs has its problems. It ...
Efficiency Matters: Speeding Up Automated Testing with GUI Rendering Inference
ICSE '23: Proceedings of the 45th International Conference on Software EngineeringDue to the importance of Android app quality assurance, many automated GUI testing tools have been developed. Although the test algorithms have been improved, the impact of GUI rendering has been overlooked. On the one hand, setting a long waiting ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
October 2024
2587 pages
ISBN:9798400712487
DOI:10.1145/3691620
- General Chair:
- Vladimir Filkov,
- Program Co-chairs:
- Baishakhi Ray,
- Minghui Zhou
Copyright © 2024 Copyright is held by the owner/author(s). Publication rights licensed to ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 27 October 2024
Check for updates
Author Tags
Qualifiers
- Research-article
Conference
ASE '24: 39th IEEE/ACM International Conference on Automated Software Engineering
October 27 - November 1, 2024
CA, Sacramento, USA
Acceptance Rates
Overall Acceptance Rate 82 of 337 submissions, 24%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 67Total Downloads
- Downloads (Last 12 months)67
- Downloads (Last 6 weeks)19
Reflects downloads up to 03 Mar 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in