More Web Proxy on the site http://driver.im/

research-article

Improving random GUI testing with image-based widget detection

Authors:

Thomas D. White,

Guy J. BrownAuthors Info & Claims

ISSTA 2019: Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis

Pages 307 - 317

https://doi.org/10.1145/3293882.3330551

Published: 10 July 2019 Publication History

Abstract

Graphical User Interfaces (GUIs) are amongst the most common user interfaces, enabling interactions with applications through mouse movements and key presses. Tools for automated testing of programs through their GUI exist, however they usually rely on operating system or framework specific knowledge to interact with an application. Due to frequent operating system updates, which can remove required information, and a large variety of different GUI frameworks using unique underlying data structures, such tools rapidly become obsolete, Consequently, for an automated GUI test generation tool, supporting many frameworks and operating systems is impractical. We propose a technique for improving GUI testing by automatically identifying GUI widgets in screen shots using machine learning techniques. As training data, we generate randomized GUIs to automatically extract widget information. The resulting model provides guidance to GUI testing tools in environments not currently supported by deriving GUI widget information from screen shots only. In our experiments, we found that identifying GUI widgets in screen shots and using this information to guide random testing achieved a significantly higher branch coverage in 18 of 20 applications, with an average increase of 42.5% when compared to conventional random testing.

References

[1]

Android Developers. 2015. Monkeyrunner.

[2]

M. Bajammal and A. Mesbah. 2018. Web Canvas Testing Through Visual Inference. In 2018 IEEE 11th International Conference on Software Testing, Verification and Validation (ICST). 193–203.

[3]

S. Bauersfeld and T. E. J. Vos. 2012. GUITest: A Java Library for Fully Automated GUI Robustness Testing. In 2012 Proceedings of the 27th IEEE/ACM International Conference on Automated Software Engineering. 330–333. 2351676.2351739

Digital Library

[4]

Giovanni Becce, Leonardo Mariani, Oliviero Riganelli, and Mauro Santoro. 2012. Extracting Widget Descriptions from GUIs. In Fundamental Approaches to Software Engineering, Juan de Lara and Andrea Zisman (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 347–361.

Digital Library

[5]

Nataniel P. Borges, Jr., Maria Gómez, and Andreas Zeller. 2018. Guiding App Testing with Mined Interaction Models. In Proceedings of the 5th International Conference on Mobile Software Engineering and Systems (MOBILESoft ’18). ACM, New York, NY, USA, 133–143.

Digital Library

[6]

Hatim Chahim, Mehmet Duran, and Tanja EJ Vos. 2018. Challenging TESTAR in an industrial setting: the rail sector. (2018).

[7]

Shauvik Roy Choudhary, Alessandra Gorla, and Alessandro Orso. 2015. Automated Test Input Generation for Android:Are We There Yet? (E). In 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 429–440.

Digital Library

[8]

J. Ding, B. Chen, H. Liu, and M. Huang. 2016. Convolutional Neural Network With Data Augmentation for SAR Target Recognition. IEEE Geoscience and Remote Sensing Letters 13, 3 (March 2016), 364–368.

[9]

2513754

[10]

Justin E Forrester and Barton P Miller. 2000. An Empirical Study of the Robustness of Windows NT Applications Using Random Testing. In Proceedings of the 4th USENIX Windows System Symposium. Seattle, 59–68.

Digital Library

[11]

Gordon Fraser and Andrea Arcuri. 2014. A Large-Scale Evaluation of Automated Unit Test Generation Using EvoSuite. ACM Transactions on Software Engineering and Methodology (TOSEM) 24, 2, Article 8 (Dec. 2014), 42 pages.

Digital Library

[12]

Zebao Gao, Yalan Liang, Myra B Cohen, Atif M Memon, and Zhen Wang. 2015. Making system user interactive tests repeatable: When and what should we control?. In 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering, Vol. 1. IEEE, 55–65.

Digital Library

[13]

Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. 2014. Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation. In Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition (CVPR ’14). IEEE Computer Society, Washington, DC, USA, 580–587.

Digital Library

[14]

R. Lo, R. Webby, and R. Jeffery. 1996. Sizing and Estimating the Coding and Unit Testing Effort for GUI Systems. In Proceedings of the 3rd International Software Metrics Symposium. 166–173.

Digital Library

[15]

Ke Mao, Mark Harman, and Yue Jia. 2016. Sapienz: Multi-objective Automated Testing for Android Applications. In Proceedings of the 25th International Symposium on Software Testing and Analysis. ACM, 94–105.

Digital Library

[16]

L. Mariani, M. PezzÃĺ, O. Riganelli, and M. Santoro. 2011. AutoBlackTest: A Tool for Automatic Black-Box Testing. (May 2011), 1013–1015.

Digital Library

[17]

1145/1985793.1985979

[18]

A. Memon, I. Banerjee, and A. Nagarajan. 2003. GUI Ripping: Reverse Engineering of Graphical User Interfaces for Testing. In 10th Working Conference on Reverse Engineering, 2003. WCRE 2003. Proceedings. 260–269. WCRE.2003.1287256

Digital Library

[19]

Bao Nguyen, Bryan Robbins, Ishan Banerjee, and Atif Memon. 2014. GUITAR: An innovative tool for automated testing of GUI-driven software. Automated Software Engineering 21 (03 2014).

Digital Library

[20]

Joseph Redmon, Santosh Kumar Divvala, Ross B. Girshick, and Ali Farhadi. 2015. You Only Look Once: Unified, Real-Time Object Detection. CoRR abs/1506.02640 (2015). arXiv: 1506.02640 http://arxiv.org/abs/1506.02640

[21]

Joseph Redmon and Ali Farhadi. 2016. YOLO9000: Better, Faster, Stronger. CoRR abs/1612.08242 (2016). arXiv: 1612.08242 http://arxiv.org/abs/1612.08242

[22]

Joseph Redmon and Ali Farhadi. 2018. YOLOv3: An Incremental Improvement. CoRR abs/1804.02767 (2018). arXiv: 1804.02767 http://arxiv.org/abs/1804.02767

[23]

O. Russakovsky, L. Li, and L. Fei-Fei. 2015. Best of both worlds: Human-machine collaboration for object annotation. In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2121–2131.

[24]

7298824

[25]

J. Salamon and J. P. Bello. 2017. Deep Convolutional Neural Networks and Data Augmentation for Environmental Sound Classification. IEEE Signal Processing Letters 24, 3 (March 2017), 279–283.

[26]

Ting Su, Guozhu Meng, Yuting Chen, Ke Wu, Weiming Yang, Yao Yao, Geguang Pu, Yang Liu, and Zhendong Su. 2017. Guided, Stochastic Model-Based GUI Testing of Android Apps. In Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering. ACM, 245–256.

Digital Library

Cited By

Huang HQian JJia WJin Y(2025)cvripJournal of Software: Evolution and Process10.1002/smr.7000937:2Online publication date: 6-Feb-2025
https://dl.acm.org/doi/10.1002/smr.70009
Pan XHuan ZLi YCao Y(2024)Enhancement of GUI Display Error Detection Using Improved Faster R-CNN and Multi-Scale Attention MechanismApplied Sciences10.3390/app1403114414:3(1144)Online publication date: 30-Jan-2024
https://doi.org/10.3390/app14031144
Kumar SNitin Yadav M(2024)An Effective GDP-LSTM and SDQL-Based Finite State Testing of GUIApplied Sciences10.3390/app1402054914:2(549)Online publication date: 8-Jan-2024
https://doi.org/10.3390/app14020549
Show More Cited By

Index Terms

Improving random GUI testing with image-based widget detection
1. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Empirical software validation
      2. Software defect analysis
        Software testing and debugging

Recommendations

Towards a pattern language for model-based GUI testing
EuroPLoP '14: Proceedings of the 19th European Conference on Pattern Languages of Programs

Graphical user interfaces (GUIs) have become popular as they appear in everydays' software. GUIs have become an ideal way of interacting with computer programs, making the software friendlier to its users. GUIs have grown, and so has the usage of UI ...
Coverage criteria for GUI testing

A widespread recognition of the usefulness of graphical user interfaces (GUIs) has established their importance as critical components of today's software. GUIs have characteristics different from traditional software, and conventional testing ...
Coverage criteria for GUI testing
ESEC/FSE-9: Proceedings of the 8th European software engineering conference held jointly with 9th ACM SIGSOFT international symposium on Foundations of software engineering

A widespread recognition of the usefulness of graphical user interfaces (GUIs) has established their importance as critical components of today's software. GUIs have characteristics different from traditional software, and conventional testing ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

ISSTA 2019: Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis

July 2019

451 pages

ISBN:9781450362245

DOI:10.1145/3293882

General Chair:
Dongmei Zhang
Microsoft Research, China
,
Program Chair:
Anders Møller
Aarhus University, Denmark

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGSOFT: ACM Special Interest Group on Software Engineering

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 10 July 2019

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Engineering and Physical Sciences Research Council

Conference

ISSTA '19

Sponsor:

SIGSOFT

ISSTA '19: 28th ACM SIGSOFT International Symposium on Software Testing and Analysis

July 15 - 19, 2019

Beijing, China

Acceptance Rates

Overall Acceptance Rate 58 of 213 submissions, 27%

Upcoming Conference

ISSTA '25

Sponsor:
sigsoft

34th ACM SIGSOFT International Symposium on Software Testing and Analysis

June 25 - 28, 2025

Trondheim , Norway

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

61
Total Citations
View Citations
858
Total Downloads

Downloads (Last 12 months)95
Downloads (Last 6 weeks)4

Reflects downloads up to 02 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Huang HQian JJia WJin Y(2025)cvripJournal of Software: Evolution and Process10.1002/smr.7000937:2Online publication date: 6-Feb-2025
https://dl.acm.org/doi/10.1002/smr.70009
Pan XHuan ZLi YCao Y(2024)Enhancement of GUI Display Error Detection Using Improved Faster R-CNN and Multi-Scale Attention MechanismApplied Sciences10.3390/app1403114414:3(1144)Online publication date: 30-Jan-2024
https://doi.org/10.3390/app14031144
Kumar SNitin Yadav M(2024)An Effective GDP-LSTM and SDQL-Based Finite State Testing of GUIApplied Sciences10.3390/app1402054914:2(549)Online publication date: 8-Jan-2024
https://doi.org/10.3390/app14020549
Cao SChen RPan MYang WLi XFilkov VRay BZhou M(2024)Beyond Manual Modeling: Automating GUI Model Generation Using Design DocumentsProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695032(91-103)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3695032
Yu SFang CLi XLing YChen ZSu Z(2024)Effective, Platform-Independent GUI Testing via Image Embedding and Reinforcement LearningACM Transactions on Software Engineering and Methodology10.1145/367472833:7(1-27)Online publication date: 21-Jun-2024
https://dl.acm.org/doi/10.1145/3674728
Lu YKnearem TDutta SBlass JKliman-Silver CBentley F(2024)AI Is Not Enough: A Hybrid Technical Approach to AI Adoption in UI Linting With HeuristicsExtended Abstracts of the CHI Conference on Human Factors in Computing Systems10.1145/3613905.3637135(1-7)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613905.3637135
Su YLiao DXing ZHuang QXie MLu QXu XRoychoudhury APaiva AAbreu RStorey M(2024)Enhancing Exploratory Testing by Large Language Model and Knowledge GraphProceedings of the IEEE/ACM 46th International Conference on Software Engineering10.1145/3597503.3639157(1-12)Online publication date: 20-May-2024
https://dl.acm.org/doi/10.1145/3597503.3639157
Wang YZhao YWang XTang WZhang JWang SWang PHu J(2024)Robotic Process Automation Efficiency for Mobile App Testing: An Empirical InvestigationInternational Journal of Software Engineering and Knowledge Engineering10.1142/S021819402450011634:07(1025-1046)Online publication date: 14-May-2024
https://doi.org/10.1142/S0218194024500116
Jose SSamuel PIdicula S(2024)GUI Element Detection via YOLOV8: A Deep Learning Approach for Widget Identification2024 1st International Conference on Trends in Engineering Systems and Technologies (ICTEST)10.1109/ICTEST60614.2024.10576164(1-6)Online publication date: 11-Apr-2024
https://doi.org/10.1109/ICTEST60614.2024.10576164
Chowdhury PSakib K(2024)NeuroUI: A Metamorphic Testing Strategy to Make UI Component Detection Models Robust2024 IEEE International Conference on Software Maintenance and Evolution (ICSME)10.1109/ICSME58944.2024.00065(636-641)Online publication date: 6-Oct-2024
https://doi.org/10.1109/ICSME58944.2024.00065
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten