Can Cooperative Multi-Agent Reinforcement Learning Boost Automatic Web Testing? An Exploratory Study
Abstract
Reinforcement learning (RL)-based web GUI testing techniques have attracted significant attention in both academia and industry due to their ability to facilitate automatic and intelligent exploration of websites under test. Yet, the existing approaches that leverage a single RL agent often struggle to comprehensively explore the vast state space of large-scale websites with complex structures and dynamic content. Observing this phenomenon and recognizing the benefit of multiple agents, we explore the use of Multi-Agent RL (MARL) algorithms for automatic web GUI testing, aiming to improve test efficiency and coverage. However, how to share information among different agents to avoid redundant actions and achieve effective cooperation is a non-trivial problem. To address the challenge, we propose the first MARL-based web GUI testing system, MARG, which coordinates multiple testing agents to efficiently explore a website under test. To share testing experience among different agents, we have designed two data sharing schemes: one centralized scheme with a shared Q-table to facilitate efficient communication, and another distributed scheme with data exchange to decrease the overhead of maintaining Q-tables. We have evaluated MARG on nine popular real-world websites. When configuring with five agents, MARG achieves an average increase of 4.34 and 3.89 times in the number of explored states, as well as a corresponding increase of 4.03 and 3.76 times in the number of detected failures, respectively, when compared to two state-of-the-art approaches. Additionally, compared to independently running the same number of agents, MARG can explore 36.42% more unique web states. These results demonstrate the usefulness of MARL in enhancing the efficiency and performance of web GUI testing tasks.
References
[1]
[n.d.]. Flask Documentation --- flask.palletsprojects.com. https://flask.palletsprojects.com/. [Accessed 27-03-2024].
[2]
[n. d.]. SeleniumHQ/selenium: A browser automation framework and ecosystem. --- github.com. https://github.com/SeleniumHQ/selenium/. [Accessed 27-03-2024].
[3]
[n. d.]. Top Websites in the World - Most Visited & Popular Rankings - Semrush --- semrush.com. https://semrush.com/website/top/. [Accessed 01-04-2024].
[4]
Jeffrey L Adler and Victor J Blue. 2002. A cooperative multi-agent transportation management and route guidance system. Transportation Research Part C: Emerging Technologies 10, 5--6 (2002), 433--454.
[5]
Kai Arulkumaran, Marc Peter Deisenroth, Miles Brundage, and Anil Anthony Bharath. 2017. A brief survey of deep reinforcement learning. arXiv preprint arXiv:1708.05866 (2017).
[6]
Xiaoying Bai, Guilan Dai, Dezheng Xu, and Wei-Tek Tsai. 2006. A multi-agent based framework for collaborative testing on web services. In The Fourth IEEE Workshop on Software Technologies for Future Embedded and Ubiquitous Systems, and the Second International Workshop on Collaborative Computing, Integration, and Assurance (SEUS-WCCIA'06). IEEE, 6--pp.
[7]
Sebastian Bauersfeld and Tanja Vos. 2012. A reinforcement learning approach to automated gui robustness testing. In Fast abstracts of the 4th symposium on search-based software engineering (SSBSE 2012). 7--12.
[8]
Tianqin Cai, Zhao Zhang, and Ping Yang. 2020. Fastbot: A Multi-Agent Model-Based Test Generation System Beijing Bytedance Network Technology Co., Ltd. In Proceedings of the IEEE/ACM 1st International Conference on Automation of Software Test. 93--96.
[9]
Peter Corke, Ron Peterson, and Daniela Rus. 2005. Networked robots: Flying robot navigation using a sensor net. In Robotics Research. The Eleventh International Symposium: With 303 Figures. Springer, 234--243.
[10]
Anna I Esparcia-Alcázar, Francisco Almenar, Mirella Martínez, Urko Rueda, and T Vos. 2016. Q-learning strategies for action selection in the TESTAR automated testing tool. 6th International Conferenrence on Metaheuristics and nature inspired computing (META 2016) (2016), 130--137.
[11]
Yujia Fan, Siyi Wang, Sinan Wang, Yepang Liu, Guoyao Wen, and Qi Rong. 2023. A Comprehensive Evaluation of Q-Learning Based Automatic Web GUI Testing. In 2023 10th International Conference on Dependable Systems and Their Applications (DSA). IEEE, 12--23.
[12]
Roy Fielding, Jim Gettys, Jeffrey Mogul, Henrik Frystyk, Larry Masinter, Paul Leach, and Tim Berners-Lee. 1999. Hypertext transfer protocol-HTTP/1.1. Technical Report.
[13]
Jakob Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. 2018. Counterfactual multi-agent policy gradients. In Proceedings of the AAAI conference on artificial intelligence, Vol. 32.
[14]
Jakob Foerster, Nantas Nardelli, Gregory Farquhar, Triantafyllos Afouras, Philip HS Torr, Pushmeet Kohli, and Shimon Whiteson. 2017. Stabilising experience replay for deep multi-agent reinforcement learning. In International conference on machine learning. PMLR, 1146--1155.
[15]
Jayesh K Gupta, Maxim Egorov, and Mykel Kochenderfer. 2017. Cooperative multi-agent control using deep reinforcement learning. In Autonomous Agents and Multiagent Systems: AAMAS 2017 Workshops, Best Papers, São Paulo, Brazil, May 8--12, 2017, Revised Selected Papers 16. Springer, 66--83.
[16]
Eric A Hansen, Daniel S Bernstein, and Shlomo Zilberstein. 2004. Dynamic programming for partially observable stochastic games. In AAAI, Vol. 4. 709--715.
[17]
Hado Hasselt. 2010. Double Q-learning. Advances in neural information processing systems 23 (2010).
[18]
Qingning Huo, Hong Zhu, and Sue Greenwood. 2003. A multi-agent software engineering environment for testing Web-based applications. In Proceedings 27th Annual International Computer Software and Applications Conference. COMPAC 2003. IEEE, 210--215.
[19]
Yavuz Koroglu, Alper Sen, Ozlem Muslu, Yunus Mete, Ceyda Ulker, Tolga Tanriverdi, and Yunus Donmez. 2018. QBE: QLearning-based exploration of android applications. In 2018 IEEE 11th International Conference on Software Testing, Verification and Validation (ICST). IEEE, 105--115.
[20]
Yuanhong Lan, Yifei Lu, Zhong Li, Minxue Pan, Wenhua Yang, Tian Zhang, and Xuandong Li. 2024. Deeply Reinforcing Android GUI Testing with Deep Reinforcement Learning. In Proceedings of the 46th IEEE/ACM International Conference on Software Engineering. 1--13.
[21]
Scott M Lewandowski. 1998. Frameworks for component-based client/server computing. ACM Computing Surveys (CSUR) 30, 1 (1998), 3--27.
[22]
Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. 2015. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971 (2015).
[23]
Michael L Littman. 1994. Markov games as a framework for multi-agent reinforcement learning. In Machine learning proceedings 1994. Elsevier, 157--163.
[24]
Sergio Valcarcel Macua, Jianshu Chen, Santiago Zazo, and Ali H Sayed. 2014. Distributed policy evaluation under multiple behavior strategies. IEEE Trans. Automat. Control 60, 5 (2014), 1260--1274.
[25]
Leonardo Mariani, Mauro Pezzè, Oliviero Riganelli, and Mauro Santoro. 2011. AutoBlackTest: a tool for automatic black-box testing. In Proceedings of the 33rd international conference on software engineering. 1013--1015.
[26]
Leonardo Mariani, Mauro Pezzè, Oliviero Riganelli, and Mauro Santoro. 2014. Automatic testing of GUI-based applications. Software Testing, Verification and Reliability 24, 5 (2014), 341--366.
[27]
Laetitia Matignon, Guilláume J Laurent, and Nadine Le Fort-Piat. 2012. Independent reinforcement learners in cooperative markov games: a survey regarding coordination problems. The Knowledge Engineering Review 27, 1 (2012), 1--31.
[28]
Francisco S Melo and Manuela Veloso. 2009. Learning of coordination: Exploiting sparse interactions in multiagent systems. In Proceedings of The 8th International Conference on Autonomous Agents and Multiagent Systems-Volume 2. Citeseer, 773--780.
[29]
Ali Mesbah, Engin Bozdag, and Arie Van Deursen. 2008. Crawling Ajax by inferring user interface state changes. In 2008 eighth international conference on web engineering. IEEE, 122--134.
[30]
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. 2016. Asynchronous methods for deep reinforcement learning. In International conference on machine learning. PMLR, 1928--1937.
[31]
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. 2015. Human-level control through deep reinforcement learning. nature 518, 7540 (2015), 529--533.
[32]
Marco Mobilio, Diego Clerissi, Giovanni Denaro, and Leonardo Mariani. 2023. GUI Testing to the Power of Parallel Q-Learning. In 2023 IEEE/ACM 45th International Conference on Software Engineering: Companion Proceedings (ICSE-Companion). IEEE, 55--59.
[33]
Miguel Morales. 2020. Grokking deep reinforcement learning. Manning Publications.
[34]
Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. 2016. Deep exploration via bootstrapped DQN. Advances in neural information processing systems 29 (2016).
[35]
Minxue Pan, An Huang, Guoxin Wang, Tian Zhang, and Xuandong Li. 2020. Reinforcement learning based curiosity-driven testing of Android applications. In Proceedings of the 29th ACM SIGSOFT International Symposium on Software Testing and Analysis. 153--164.
[36]
Huy Xuan Pham, Hung Manh La, David Feil-Seifer, and Aria Nefian. 2018. Co-operative and distributed reinforcement learning of drones for field coverage. arXiv preprint arXiv:1803.07250 (2018).
[37]
Tabish Rashid, Mikayel Samvelyan, Christian Schroeder De Witt, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. 2020. Monotonic value function factorisation for deep multi-agent reinforcement learning. Journal of Machine Learning Research 21, 178 (2020), 1--51.
[38]
Salman Sherin, Asmar Muqeet, Muhammad Uzair Khan, and Muhammad Zohaib Iqbal. 2023. QExplore: An exploration strategy for dynamic web applications using guided search. Journal of Systems and Software 195 (2023), 111512.
[39]
David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. 2016. Mastering the game of Go with deep neural networks and tree search. nature 529, 7587 (2016), 484--489.
[40]
David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. 2017. Mastering the game of go without human knowledge. nature 550, 7676 (2017), 354--359.
[41]
Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojciech Marian Czarnecki, Vinicius Zambaldi, Max Jaderberg, Marc Lanctot, Nicolas Sonnerat, Joel Z Leibo, Karl Tuyls, et al. 2017. Value-decomposition networks for cooperative multi-agent learning. arXiv preprint arXiv:1706.05296 (2017).
[42]
Richard S Sutton and Andrew G Barto. 2018. Reinforcement learning: An introduction. MIT press.
[43]
Ming Tan. 1993. Multi-agent reinforcement learning: Independent vs. cooperative agents. In Proceedings of the tenth international conference on machine learning. 330--337.
[44]
Thi Anh Tuyet Vuong and Shingo Takada. 2018. A reinforcement learning based approach to automated testing of android applications. In Proceedings of the 9th ACM SIGSOFT International Workshop on Automating TEST Case Design, Selection, and Evaluation. 31--37.
[45]
Thi Anh Tuyet Vuong and Shingo Takada. 2019. Semantic Analysis for Deep Q-Network in Android GUI Testing. In SEKE. 123--170.
[46]
Hoi-To Wai, Zhuoran Yang, Zhaoran Wang, and Mingyi Hong. 2018. Multi-agent reinforcement learning via double averaging primal-dual optimization. Advances in Neural Information Processing Systems 31 (2018).
[47]
Christopher JCH Watkins and Peter Dayan. 1992. Q-learning. Machine learning 8 (1992), 279--292.
[48]
Yaodong Yang and Jun Wang. 2020. An overview of multi-agent reinforcement learning from game theoretical perspective. arXiv preprint arXiv:2011.00583 (2020).
[49]
Shengcheng Yu, Chunrong Fang, Yexiao Yun, and Yang Feng. 2021. Layout and Image Recognition Driving Cross-Platform Automated Mobile Testing. In 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE). 1561--1571.
[50]
Kaiqing Zhang, Zhuoran Yang, and Tamer Başar. 2021. Multi-agent reinforcement learning: A selective overview of theories and algorithms. Handbook of reinforcement learning and control (2021), 321--384.
[51]
Kaiqing Zhang, Zhuoran Yang, Han Liu, Tong Zhang, and Tamer Basar. 2018. Fully decentralized multi-agent reinforcement learning with networked agents. In International Conference on Machine Learning. PMLR, 5872--5881.
[52]
Yan Zheng, Yi Liu, Xiaofei Xie, Yepang Liu, Lei Ma, Jianye Hao, and Yang Liu. 2021. Automatic web testing using curiosity-driven reinforcement learning. In 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE). IEEE, 423--435.
Index Terms
- Can Cooperative Multi-Agent Reinforcement Learning Boost Automatic Web Testing? An Exploratory Study
Index terms have been assigned to the content through auto-classification.
Recommendations
Mediated Multi-Agent Reinforcement Learning
AAMAS '23: Proceedings of the 2023 International Conference on Autonomous Agents and Multiagent SystemsThe majority of Multi-Agent Reinforcement Learning (MARL) literature equates the cooperation of self-interested agents in mixed environments to the problem of social welfare maximization, allowing agents to arbitrarily share rewards and private ...
Training Cooperative Agents for Multi-Agent Reinforcement Learning
AAMAS '19: Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent SystemsDeep Learning and back-propagation has been successfully used to perform centralized training with communication protocols among multiple agents in a cooperative environment. In this paper we present techniques for centralized training of Multi-Agent (...
Levels of realism for cooperative multi-agent reinforcement learning
ICSI'12: Proceedings of the Third international conference on Advances in Swarm Intelligence - Volume Part ITraining agents in a virtual crowd to achieve a task can be accomplished by allowing the agents to learn by trial-and-error and by sharing information with other agents. Since sharing enables agents to potentially reach optimal behavior more quickly, ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
October 2024
2587 pages
ISBN:9798400712487
DOI:10.1145/3691620
- General Chair:
- Vladimir Filkov,
- Program Co-chairs:
- Baishakhi Ray,
- Minghui Zhou
Copyright © 2024 Copyright is held by the owner/author(s). Publication rights licensed to ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 27 October 2024
Check for updates
Author Tags
Qualifiers
- Research-article
Conference
ASE '24: 39th IEEE/ACM International Conference on Automated Software Engineering
October 27 - November 1, 2024
CA, Sacramento, USA
Acceptance Rates
Overall Acceptance Rate 82 of 337 submissions, 24%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 138Total Downloads
- Downloads (Last 12 months)138
- Downloads (Last 6 weeks)31
Reflects downloads up to 28 Jan 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in