research-article

UI X-Ray: Interactive Mobile UI Testing Based on Computer Vision

Authors:

Chun-Fu (Richard) Chen,

Yong WangAuthors Info & Claims

IUI '17: Proceedings of the 22nd International Conference on Intelligent User Interfaces

Pages 245 - 255

https://doi.org/10.1145/3025171.3025190

Published: 07 March 2017 Publication History

Get Access

Abstract

User Interface/eXperience (UI/UX) significantly affects the lifetime of any software program, particularly mobile apps. A bad UX can undermine the success of a mobile app even if that app enables sophisticated capabilities. A good UX, however, needs to be supported of a highly functional and user friendly UI design. In spite of the importance of building mobile apps based on solid UI designs, UI discrepancies---inconsistencies between UI design and implementation---are among the most numerous and expensive defects encountered during testing. This paper presents UI X-Ray, an interactive UI testing system that integrates computer-vision methods to facilitate the correction of UI discrepancies---such as inconsistent positions, sizes and colors of objects and fonts. Using UI X-Ray does not require any programming experience; therefore, UI X-Ray can be used even by non-programmers---particularly designers---which significantly reduces the overhead involved in writing tests. With the feature of interactive interface, UI testers can quickly generate defect reports and revision instructions---which would otherwise be done manually. We verified our UI X-Ray on 4 developed mobile apps of which the entire development history was saved. UI X-Ray achieved a 99.03% true-positive rate, which significantly surpassed the 20.92% true-positive rate obtained via manual analysis. Furthermore, evaluating the results of our automated analysis can be completed quickly (< 1 minute per view on average) compared to hours of manual work required by UI testers. On the other hand, UI X-Ray received the appreciations from skilled designers and UI X-Ray improves their current work flow to generate UI defect reports and revision instructions. The proposed system, UI X-Ray, presented in this paper has recently become part of a commercial product.

References

[1]

Chang, T.-H., Yeh, T., and Miller, R. C. GUI Testing Using Computer Vision. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, ACM (NY, NY, USA, 2010), 1535--1544.

Digital Library

Google Scholar

[2]

Espresso Test Recorder. http://android-developers.blogspot.nl/2016/05/android-studio-22-preview-new-ui.html, 2016.

Google Scholar

[3]

FBSnapshotTestCase. https://github.com/facebook/ios-snapshot-test-case, 2016.

Google Scholar

[4]

Fighting Layout Bugs. https://code.google.com/archive/p/fighting-layout-bugs/.

Google Scholar

[5]

Lewis, J. P. Fast template matching. In Vision interface, vol. 95 (1995), 15--19.

Google Scholar

[6]

Ligman, J., Pistoia, M., Tripp, O., and Thomas, G. Improving Design Validation of Mobile Application User Interface Implementation. In Proceedings of the International Conference on Mobile Software Engineering and Systems, ACM (NY, NY, USA, 2016), 277--278.

Digital Library

Google Scholar

[7]

Lloyd, S. Least squares quantization in pcm. IEEE Trans. Inf. Theor. 28, 2 (Sept. 2006), 129--137.

Digital Library

Google Scholar

[8]

Mahajan, S., and Halfond, W. G. J. Detection and Localization of HTML Presentation Failures Using Computer Vision-Based Techniques. In 2015 IEEE 8th International Conference on Software Testing, Verification and Validation (ICST), IEEE (Apr. 2015), 1--10.

Google Scholar

[9]

Mahajan, S., Li, B., Behnamghader, P., and Halfond, W. G. J. Using Visual Symptoms for Debugging Presentation Failures in Web Applications. In 2016 IEEE International Conference on Software Testing, Verification and Validation (ICST), IEEE (Apr. 2016), 191--201.

Google Scholar

[10]

Open Source Computer Vision Library. http://opencv.org, 2016.

Google Scholar

[11]

Screenshot Tests for Android. https://github.com/facebook/screenshot-tests-for-android, 2016.

Google Scholar

[12]

Selenium. http://docs.seleniumhq.org/.

Google Scholar

[13]

User Interface Testing. https://developer.apple.com/videos/play/wwdc2015/406/, 2015.

Google Scholar

Cited By

View all

Azimi MYilmaz C(2025)Model-based test execution from high-level natural language instructions using GPT-4Software Quality Journal10.1007/s11219-025-09712-933:1Online publication date: 1-Mar-2025
https://dl.acm.org/doi/10.1007/s11219-025-09712-9
Duan PWarner JLi YHartmann B(2024)Generating Automatic Feedback on UI Mockups with Large Language ModelsProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642782(1-20)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642782
Lee LYau YHui P(2024)Perceived User Reachability in Mobile UIs Using Data Analytics and Machine LearningInternational Journal of Human–Computer Interaction10.1080/10447318.2024.232719941:4(2703-2726)Online publication date: 25-Mar-2024
https://doi.org/10.1080/10447318.2024.2327199
Show More Cited By

Index Terms

UI X-Ray: Interactive Mobile UI Testing Based on Computer Vision
1. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Software defect analysis
        Software testing and debugging

Recommendations

Akin: Generating UI Wireframes From UI Design Patterns Using Deep Learning
IUI '21 Companion: Companion Proceedings of the 26th International Conference on Intelligent User Interfaces

During the User interface (UI) design process, designers use UI design patterns for conceptualizing different UI wireframes for an application. This paper introduces Akin, a UI wireframe generator that allows designers to chose a UI design pattern and ...
Data flow oriented UI testing: exploiting data flows and UI elements to test Android applications
ISSTA 2017: Proceedings of the 26th ACM SIGSOFT International Symposium on Software Testing and Analysis

Testing user interfaces (UIs) is a challenging task. Ideally, every sequence of UI elements should be tested to guarantee that the application works correctly. This is, however, unfeasible due to the number of UI elements in an application. A better ...
A Formal Model for Generating Integrated Functional and User Interface Test Cases
ICST '10: Proceedings of the 2010 Third International Conference on Software Testing, Verification and Validation

Black box testing focuses on the core functionality of the system, while user interface testing is concerned with details of user interactions. Functional and user interface test cases are usually generated from two distinct system models, one for the ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

IUI '17: Proceedings of the 22nd International Conference on Intelligent User Interfaces

March 2017

654 pages

ISBN:9781450343480

DOI:10.1145/3025171

General Chairs:
George A. Papadopoulos
The University of Cyprus, Cyprus
,
Tsvi Kuflik
The University of Haifa, Israel
,
Program Chairs:
Fang Chen
Data61, CSIRO Australia
,
Carlos Duarte
The University of Lisbon, Portugal
,
Wai-Tat Fu
The University of Illinois at Urbana-Champaign, USA

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 07 March 2017

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

IUI'17

Sponsor:

IUI'17: 22nd International Conference on Intelligent User Interfaces

March 13 - 16, 2017

Limassol, Cyprus

Acceptance Rates

IUI '17 Paper Acceptance Rate 63 of 272 submissions, 23%;

Overall Acceptance Rate 746 of 2,811 submissions, 27%

Upcoming Conference

IUI '25

Sponsor:
sigai
sigai

30th International Conference on Intelligent User Interfaces

March 24 - 27, 2025

Cagliari , Italy

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

10
Total Citations
View Citations
800
Total Downloads

Downloads (Last 12 months)50
Downloads (Last 6 weeks)2

Reflects downloads up to 05 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

View all

Azimi MYilmaz C(2025)Model-based test execution from high-level natural language instructions using GPT-4Software Quality Journal10.1007/s11219-025-09712-933:1Online publication date: 1-Mar-2025
https://dl.acm.org/doi/10.1007/s11219-025-09712-9
Duan PWarner JLi YHartmann B(2024)Generating Automatic Feedback on UI Mockups with Large Language ModelsProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642782(1-20)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642782
Lee LYau YHui P(2024)Perceived User Reachability in Mobile UIs Using Data Analytics and Machine LearningInternational Journal of Human–Computer Interaction10.1080/10447318.2024.232719941:4(2703-2726)Online publication date: 25-Mar-2024
https://doi.org/10.1080/10447318.2024.2327199
Duan PWarner JHartmann B(2023)Towards Generating UI Design Feedback with LLMsAdjunct Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology10.1145/3586182.3615810(1-3)Online publication date: 29-Oct-2023
https://dl.acm.org/doi/10.1145/3586182.3615810
Ang GLim E(2023)Learning and Understanding User Interface Semantics from Heterogeneous Networks with Multimodal and Positional AttributesACM Transactions on Interactive Intelligent Systems10.1145/357852213:3(1-31)Online publication date: 11-Sep-2023
https://dl.acm.org/doi/10.1145/3578522
Schoop EZhou XLi GChen ZHartmann BLi Y(2022)Predicting and Explaining Mobile UI Tappability with Vision Modeling and Saliency AnalysisProceedings of the 2022 CHI Conference on Human Factors in Computing Systems10.1145/3491102.3517497(1-21)Online publication date: 29-Apr-2022
https://dl.acm.org/doi/10.1145/3491102.3517497
Bajammal MStocco AMazinanian DMesbah A(2022)A Survey on the Use of Computer Vision to Improve Software Engineering TasksIEEE Transactions on Software Engineering10.1109/TSE.2020.303298648:5(1722-1742)Online publication date: 1-May-2022
https://doi.org/10.1109/TSE.2020.3032986
Ren YGu YMa ZZhu HYin F(2022)Cross-Device Difference Detector for Mobile Application GUI Compatibility Testing2022 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)10.1109/ICSTW55395.2022.00052(253-260)Online publication date: Apr-2022
https://doi.org/10.1109/ICSTW55395.2022.00052
Packevičius ŠRudžionienė GBareiša E(2021)Automated Visual Testing of Application User Interfaces Using Static Analysis of ScreenshotsInternational Journal of Software Engineering and Knowledge Engineering10.1142/S021819402150004231:02(167-191)Online publication date: 2-Mar-2021
https://doi.org/10.1142/S0218194021500042
Pan MXu TPei YLi ZZhang TLi X(2020)GUI-Guided Test Script Repair for Mobile AppsIEEE Transactions on Software Engineering10.1109/TSE.2020.3007664(1-1)Online publication date: 2020
https://doi.org/10.1109/TSE.2020.3007664

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Abstract

References

Cited By

Index Terms

Recommendations

Akin: Generating UI Wireframes From UI Design Patterns Using Deep Learning

Data flow oriented UI testing: exploiting data flows and UI elements to test Android applications

A Formal Model for Generating Integrated Functional and User Interface Test Cases

Comments

Information

Published In

Sponsors

Publisher

Publication History

Permissions

Check for updates

Author Tags

Qualifiers

Conference

Acceptance Rates

Upcoming Conference

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Cited By

Login options

Full Access

View options

PDF

eReader

Share

Share this Publication link

Share on social media

Affiliations