More Web Proxy on the site http://driver.im/

research-article

An empirical comparison between monkey testing and human testing (WIP paper)

Authors:

Mostafa Mohammed,

Na MengAuthors Info & Claims

LCTES 2019: Proceedings of the 20th ACM SIGPLAN/SIGBED International Conference on Languages, Compilers, and Tools for Embedded Systems

Pages 188 - 192

https://doi.org/10.1145/3316482.3326342

Published: 23 June 2019 Publication History

Abstract

Android app testing is challenging and time-consuming because fully testing all feasible execution paths is difficult. Nowadays apps are usually tested in two ways: human testing or automated testing. Prior work compared different automated tools. However, some fundamental questions are still unexplored, including (1) how automated testing behaves differently from human testing, and (2) whether automated testing can fully or partially substitute human testing.

This paper presents our study to explore the open questions. Monkey has been considered one of the best automated testing tools due to its usability, reliability, and competitive coverage metrics, so we applied Monkey to five Android apps and collected their dynamic event traces. Meanwhile, we recruited eight users to manually test the same apps and gathered the traces. By comparing the collected data, we revealed that i.) on average, the two methods generated similar numbers of unique events; ii.) Monkey created more system events while humans created more UI events; iii.) Monkey could mimic human behaviors when apps have UIs full of clickable widgets to trigger logically independent events; and iv.) Monkey was insufficient to test apps that require information comprehension and problem-solving skills. Our research sheds light on future research that combines human expertise with the agility of Monkey testing.

References

[1]

2017. App Stores Start to Mature - 2016 Year in Review. https://blog.appfigures.com/app-stores-start-to-mature-2016-year-in-review/. (2017).

[2]

2018. Amazon Shopping APK. https://apkpure.com/cn/amazon-shopping/com.amazon.mShop.android.shopping. (2018).

[3]

2018. Candy Crush APK. https://apkpure.com/cn/candy-crush-saga/com.king.candycrushsaga. (2018).

[4]

2018. Google. Android Monkey. http://developer.android.com/tools/help/monkey.html. (2018).

[5]

2018. Intent fuzzer. https://www.nccgroup.trust/us/about-us/resources/intent-fuzzer/. (2018).

[6]

2018. Run Apps on the Android Emulator. https://developer.android.com/studio/run/emulator.html. (2018).

[7]

2018. Spotify Music APK. https://apkpure.com/cn/spotify-premium-music/com.spotify.music. (2018).

[8]

2018. Twitter APK. https://apkpure.com/cn/twitter/com.twitter.android. (2018).

[9]

2018. Viber Messenger APK. https://apkpure.com/cn/viber-messenger/com.viber.voip. (2018).

[10]

Domenico Amalfitano, Anna Rita Fasolino, Porfirio Tramontana, Salvatore De Carmine, and Atif M. Memon. 2012. Using GUI Ripping for Automated Testing of Android Applications. In Proceedings of the 27th IEEE/ACM International Conference on Automated Software Engineering (ASE 2012). ACM, New York, NY, USA, 258-261.

Digital Library

[11]

Saswat Anand, Mayur Naik, Mary Jean Harrold, and Hongseok Yang. 2012. Automated Concolic Testing of Smartphone Apps. In Proceedings of the ACM SIGSOFT 20th International Symposium on the Foundations of Software Engineering (FSE'12). ACM, New York, NY, USA, Article 59, 11 pages.

Digital Library

[12]

Haipeng Cai and Barbara Ryder. 2017. DroidFax: A Toolkit for Systematic Characterization of Android Applications. In International Conference on Software Maintenance and Evolution (ICSME). 643-647.

[13]

Shauvik Roy Choudhary, Alessandra Gorla, and Alessandro Orso. 2015. Automated Test Input Generation for Android: Are We There Yet? (E). In Proceedings of the 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE) (ASE'15). IEEE Computer Society, Washington, DC, USA, 429-440.

Digital Library

[14]

Shuai Hao, Bin Liu, Suman Nath, William G.J. Halfond, and Ramesh Govindan. 2014. PUMA: Programmable UI-automation for Largescale Dynamic Analysis of Mobile Apps. In Proceedings of the 12th Annual International Conference on Mobile Systems, Applications, and Services (MobiSys'14). ACM, New York, NY, USA, 204-217.

Digital Library

[15]

Winston Haynes. 2013. Student's t-Test. Springer New York, New York, NY, 2023-2025.

[16]

Aravind Machiry, Rohan Tahiliani, and Mayur Naik. 2013. Dynodroid: An Input Generation System for Android Apps. In Proceedings of the 2013 9th Joint Meeting on Foundations of Software Engineering (ESEC/FSE 2013). ACM, New York, NY, USA, 224-234.

Digital Library

[17]

Riyadh Mahmood, Nariman Mirzaei, and Sam Malek. 2014. EvoDroid: Segmented Evolutionary Testing of Android Apps. In Proceedings of the 22Nd ACM SIGSOFT International Symposium on Foundations of Software Engineering (FSE 2014). ACM, New York, NY, USA, 599-609.

Digital Library

[18]

Ke Mao, Mark Harman, and Yue Jia. 2016. Sapienz: Multi-objective automated testing for android applications. In Proceedings of the 25th International Symposium on Software Testing and Analysis. 94-105.

Digital Library

[19]

Priyam Patel, Gokul Srinivasan, Sydur Rahaman, and Iulian Neamtiu. 2018. On the effectiveness of random testing for Android: or how i learned to stop worrying and love the monkey. In Proceedings of the 13th International Workshop on Automation of Software Test. ACM, 34-37.

Digital Library

[20]

Ting Su, Guozhu Meng, Yuting Chen, Ke Wu, Weiming Yang, Yao Yao, Geguang Pu, Yang Liu, and Zhendong Su. 2017. Guided, stochastic model-based GUI testing of Android apps. In Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering. 245-256.

Digital Library

[21]

C. Sun, Z. Zhang, B. Jiang, and W. K. Chan. 2016. Facilitating Monkey Test by Detecting Operable Regions in Rendered GUI of Mobile Game Apps. In 2016 IEEE International Conference on Software Quality, Reliability and Security (QRS). 298-306.

[22]

Heila van der Merwe, Brink van der Merwe, and Willem Visser. 2014. Execution and Property Specifications for JPF-android. SIGSOFT Softw. Eng. Notes 39, 1 (Feb. 2014), 1-5.

Digital Library

[23]

Wei Yang, Mukul R. Prasad, and Tao Xie. 2013. A Grey-box Approach for Automated GUI-model Generation of Mobile Applications. In Proceedings of the 16th International Conference on Fundamental Approaches to Software Engineering (FASE'13). Springer-Verlag, Berlin, Heidelberg, 250-265.

Digital Library

Cited By

da Costa LSantos ICosta IAndrade R(2022)X-Monkey: a library to extend the monkey testing2022 17th Iberian Conference on Information Systems and Technologies (CISTI)10.23919/CISTI54924.2022.9820543(1-6)Online publication date: 22-Jun-2022
https://doi.org/10.23919/CISTI54924.2022.9820543
Zhao YTang ECai HGuo XWang XMeng N(2022)A Lightweight Approach of Human-Like Playtest for Android Apps2022 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER)10.1109/SANER53432.2022.00047(309-320)Online publication date: Mar-2022
https://doi.org/10.1109/SANER53432.2022.00047
Ali SIbrahim HNouh KFayek M(2022)An Improved Framework for Monkey GUI Testing for EDA Desktop Applications2022 12th International Conference on Software Technology and Engineering (ICSTE)10.1109/ICSTE57415.2022.00009(8-13)Online publication date: Oct-2022
https://doi.org/10.1109/ICSTE57415.2022.00009
Show More Cited By

Index Terms

An empirical comparison between monkey testing and human testing (WIP paper)
1. General and reference
  1. Cross-computing tools and techniques
    1. Empirical studies

Recommendations

Making Android apps monkey-friendly
MOBILESoft '20: Proceedings of the IEEE/ACM 7th International Conference on Mobile Software Engineering and Systems

Monkey testing is a random testing technique in which a stream of pseudo-random events are automatically fired on the GUI of the application under test, usually with the purpose of robustness testing or responsiveness analysis. A line of research is ...
Automated System Testing Using Visual GUI Testing Tools: A Comparative Study in Industry
ICST '12: Proceedings of the 2012 IEEE Fifth International Conference on Software Testing, Verification and Validation

Software companies are under continuous pressure to shorten time to market, raise quality and lower costs. More automated system testing could be instrumental in achieving these goals and in recent years testing tools have been developed to automate the ...
A comparison of paper-and-pencil, computer-administered, computerized feedback, and computerized adaptive testing methods for classroom achievement testing

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

LCTES 2019: Proceedings of the 20th ACM SIGPLAN/SIGBED International Conference on Languages, Compilers, and Tools for Embedded Systems

June 2019

218 pages

ISBN:9781450367240

DOI:10.1145/3316482

General Chair:
Jian-Jia Chen
TU Dortmund, Germany
,
Program Chair:
Aviral Shrivastava
Arizona State University, USA

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 23 June 2019

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

LCTES '19

Sponsor:

LCTES '19: 20th ACM SIGPLAN/SIGBED Conference on Languages, Compilers, and Tools for Embedded Systems

June 23, 2019

AZ, Phoenix, USA

Acceptance Rates

Overall Acceptance Rate 116 of 438 submissions, 26%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
155
Total Downloads

Downloads (Last 12 months)13
Downloads (Last 6 weeks)1

Reflects downloads up to 13 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

da Costa LSantos ICosta IAndrade R(2022)X-Monkey: a library to extend the monkey testing2022 17th Iberian Conference on Information Systems and Technologies (CISTI)10.23919/CISTI54924.2022.9820543(1-6)Online publication date: 22-Jun-2022
https://doi.org/10.23919/CISTI54924.2022.9820543
Zhao YTang ECai HGuo XWang XMeng N(2022)A Lightweight Approach of Human-Like Playtest for Android Apps2022 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER)10.1109/SANER53432.2022.00047(309-320)Online publication date: Mar-2022
https://doi.org/10.1109/SANER53432.2022.00047
Ali SIbrahim HNouh KFayek M(2022)An Improved Framework for Monkey GUI Testing for EDA Desktop Applications2022 12th International Conference on Software Technology and Engineering (ICSTE)10.1109/ICSTE57415.2022.00009(8-13)Online publication date: Oct-2022
https://doi.org/10.1109/ICSTE57415.2022.00009
Di Martino SFasolino AStarace LTramontana P(2020)Comparing the effectiveness of capture and replay against automatic input generation for Android graphical user interface testingSoftware Testing, Verification and Reliability10.1002/stvr.175431:3Online publication date: 16-Oct-2020
https://doi.org/10.1002/stvr.1754

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents