Automated detection, categorisation and developers’ experience with the violations of honesty in mobile apps
Authors: 
Humphrey O. Obie, Hung Du, 
Kashumi Madampe, 
Mojtaba Shahin, Idowu Ilekura, 
John Grundy, Li Li, Jon Whittle, 
Burak Turhan, Hourieh KhalajzadehAuthors Info & Claims
Abstract
Human values such as honesty, social responsibility, fairness, privacy, and the like are things considered important by individuals and society. Software systems, including mobile software applications (apps), may ignore or violate such values, leading to negative effects in various ways for individuals and society. While some works have investigated different aspects of human values in software engineering, this mixed-methods study focuses on honesty as a critical human value. In particular, we studied (i) how to detect honesty violations in mobile apps, (ii) the types of honesty violations in mobile apps, and (iii) the perspectives of app developers on these detected honesty violations. We first develop and evaluate 7 machine learning (ML) models to automatically detect violations of the value of honesty in app reviews from an end-user perspective. The most promising was a Deep Neural Network model with F1 score of 0.921. We then conducted a manual analysis of 401 reviews containing honesty violations and characterised honesty violations in mobile apps into 10 categories: unfair cancellation and refund policies; false advertisements; delusive subscriptions; cheating systems; inaccurate information; unfair fees; no service; deletion of reviews; impersonation; and fraudulent-looking apps. A developer survey and interview study with mobile developers then identified 7 key causes behind honesty violations in mobile apps and 8 strategies to avoid or fix such violations. The findings of our developer study also articulate the negative consequences that honesty violations might bring for businesses, developers, and users. Finally, the app developers’ feedback shows that our prototype ML-based models can have promising benefits in practice.
References
[1]
Cheating, Corruption, and Concealment (2016) pp 1–12. Cambridge University Press
[2]
Aggarwal CC, Zhai C (2012) A survey of text classification algorithms. In: Mining text data, pp 163–222. Springer
[3]
Agre PE Social science, technical systems and cooperative work: beyond the great divide, chap 1997 toward a critical technical practice Lessons learned in trying to reform AI. Erlbaum
[4]
Aldewereld H, Dignum V, Tan Yh (2015) Design for values information and communication technologies in software development, pp 831–845. Springer Netherlands, Dordrecht
[5]
AlOmar EA, Aljedaani W, Tamjeed M, Mkaouer MW, El-Glaly YN (2021) Finding the needle in a haystack: on the automatic identification of accessibility user reviews. In: Proceedings of the 2021 CHI conference on human factors in computing systems, CHI ’21. Association for computing machinery, New York, NY, USA.
[6]
Barr K (2022) Pervasive ‘dark patterns’ are fooling people into signing up for services they don’t want. https://gizmodo.com/dark-patterns-ui-cancel-subscription-1849542166
[7]
Bowman E (2021) After data breach exposes 530 million, facebook says it will not notify users. https://www.npr.org/2021/04/09/986005820/after-data-breach-exposes-530-million-facebook-says-it-will-not-notify-users
[8]
Campbell JL, Quincy C, Osserman J, and Pedersen OK Coding in-depth semistructured interviews: problems of unitization and intercoder reliability and agreement Sociological Methods & Research 2013 42 3 294-320
[9]
Carreño LVG, Winbladh K (2013) Analysis of user comments: an approach for software requirements evolution. In: 2013 35th International conference on software engineering (ICSE), pp 582–591.
[10]
Chee FM, Taylor NT, and de Castell S Re-mediating research ethics: end-user license agreements in online games Bull Sci Technol Soc 2012 32 6 497-506
[11]
Cheng AS, Fleischmann KR (2010) Developing a meta-inventory of human values. In: Proceedings of the 73rd ASIS &T annual meeting on navigating streams in an information ecosystem, vol 47. American society for information science
[12]
Ciurumelea A, Schaufelbühl A, Panichella S, Gall HC (2017) Analyzing reviews and code of mobile apps for better release planning. In: 2017 IEEE 24th international conference on software analysis, evolution and reengineering (SANER), pp 91–102.
[13]
Devlin J, Chang MW, Lee K, Toutanova K (2019) Bert: pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, vol 1 (long and short papers), pp 4171–4186
[14]
Di Sorbo A, Panichella S, Alexandru CV, Shimagaki J, Visaggio CA, Canfora G, Gall HC (2016) What would users change in my app? summarizing app reviews for recommending software changes. In: Proceedings of the 2016 24th ACM SIGSOFT international symposium on foundations of software engineering, FSE 2016, pp 499–510. Association for computing machinery, New York, NY, USA.
[15]
Dictionary C (2021) Definition of ‘impersonate’. https://www.collinsdictionary.com/dictionary/english/impersonate
[16]
Dictionary C (2021) Definition of ‘honesty’. https://www.collinsdictionary.com/dictionary/english/honesty
[17]
Dong F, Wang H, Li L, Guo Y, Bissyandé TF, Liu T, Xu G, Klein J (2018) Frauddroid: automated ad fraud detection for android apps. In: Proceedings of the 2018 26th ACM joint meeting on European software engineering conference and symposium on the foundations of software engineering, pp 257–268
[18]
Eler MM, Orlandin L, Oliveira ADA (2019) Do android app users care about accessibility? An analysis of user reviews on the google play store. In: Proceedings of the 18th Brazilian symposium on human factors in computing systems, IHC ’19. Association for computing machinery, New York, NY, USA
[19]
Fochmann M, Fochmann N, Kocher MG, Müller N (2021) Dishonesty and risk-taking: Compliance decisions of individuals and groups. J Econ Behav Organ 185:250–286., https://www.sciencedirect.com/science/article/pii/S0167268121%000822
[20]
Gao Y, Xu G, Li L, Luo X, Wang C, Sui Y (2022) Demystifying the underground ecosystem of account registration bots. In: ACM joint European software engineering conference and symposium on the foundations of software engineering (ESEC/FSE 2022)
[21]
Glaser BG, Strauss AL, and Strutzel E The discovery of grounded theory; strategies for qualitative research Nursing Research 1968 17 4 364
[22]
Goodfellow I, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, Bengio Y (2014) Generative adversarial nets. In: Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, K. Weinberger (eds.) Advances in neural information processing systems, vol 27. Curran Associates, Inc. https://proceedings.neurips.cc/paper/2014/file/5ca3e9b122f61f%8f06494c97b1afccf3-Paper.pdf
[23]
Gotterbarn D, Bruckman A, Flick C, Miller K, Wolf MJ (2017) Acm code of ethics: a guide for positive action. Commun ACM 61(1):121–128.
[24]
Guzman E, Maalej W (2014) How do users like this feature? A fine grained sentiment analysis of app reviews. In: 2014 IEEE 22nd international requirements engineering conference (RE), pp 153–162.
[25]
van Haasteren A, Gille F, Fadda M, and Vayena E Development of the mhealth app trustworthiness checklist Digit Health 2019 5 2055207619886463
[26]
Haggag O, Grundy J, Abdelrazek M, and Haggag S A large scale analysis of mhealth app user reviews Empir Softw Eng 2022 27 7 1-53
[27]
Henderson JG (2022) FTC report shows rise in sophisticated dark patterns designed to trick and trap consumers. https://www.ftc.gov/news-events/news/press-releases/2022/09/ftc-report-shows-rise-sophisticated-dark-patterns-designed-trick-trap-consumers
[28]
Hu Y, Wang H, Zhou Y, Guo Y, Li L, Luo B, Xu F (2019) Dating with scambots: understanding the ecosystem of fraudulent dating applications. IEEE Transactions on Dependable and Secure Computing
[29]
Hussain W, Perera H, Whittle J, Nurwidyantoro A, Hoda R, Shams RA, Oliver G (2020) Human values in software engineering: contrasting case studies of practice. IEEE Transactions on Software Engineering. pp 1–15
[30]
Iacob C, Harrison R: Retrieving and analyzing mobile apps feature requests from online reviews. In: 2013 10th working conference on mining software repositories (MSR), pp 41–44 (2013).
[31]
Jacquemet N, James AG, Luchini S, Murphy JJ, and Shogren JF Do truth-telling oaths improve honesty in crowd-working? PloS one 2021 16 1 1-18
[32]
Joerges B Do politics have artefacts? Soc Stud Sci 1999 29 3 411-431
[33]
Keyes R (2004) The post-truth era: dishonesty and deception in contemporary life, 1st, ed. St. Martin’s Press, New York
[34]
Khalajzadeh H, Shahin M, Obie HO, Agrawal P, Grundy J (2022) Supporting developers in addressing human-centric issues in mobile apps. IEEE Transactions on software engineering (TSE), arXiv:2203.12212
[35]
King C Forcing players to walk the plank: why end user license agreements improperly control players’ rights regarding microtransactions in video games William and Mary Law Review 2017 58 4 1365
[36]
Kohavi R et al. A study of cross-validation and bootstrap for accuracy estimation and model selection Ijcai 1995 Canada Montreal 1137-1145
[37]
Lang J (2013) Cheating Lessons: Learning from Academic Dishonesty. Harvard University Press.https://books.google.fm/books?id=hTeImwEACAAJ
[38]
Lennox J (2020) 2084: artificial intelligence, the future of humanity, and the god question. Zondervan
[39]
Levin S, Yehudai A (2017) Boosting automatic commit classification into maintenance activities by utilizing source code changes. In: Proceedings of the 13th international conference on predictive models and data analytics in software engineering, PROMISE, pp 97–106. Association for Computing Machinery, New York, NY, USA
[40]
Levin S, Yehudai A (2019) Towards software analytics: modeling maintenance activities. CoRR abs/1903.04909. arXiv:1903.04909
[41]
Li C, Obie HO, Khalajzadeh H (2021) A first step towards detecting values-violating defects in android apis
[42]
Li H, Zhang L, Zhang L, Shen J (2010) A user satisfaction analysis approach for software evolution. In: 2010 IEEE international conference on progress in informatics and computing, vol 2, pp 1093–1097.
[43]
Li L, Allix K, Li D, Bartel A, Bissyandé TF, Klein J (2015) Potential component leaks in android apps: An investigation into a new feature set for malware detection. In: 2015 IEEE international conference on software quality, reliability and security, pp 195–200. IEEE
[44]
Li L, Li D, Bissyandé TF, Klein J, Cai H, Lo D, Le Traon Y (2017) Automatically locating malicious packages in piggybacked android apps. In: 2017 IEEE/ACM 4th international conference on mobile software engineering and systems (MOBILESoft), pp 170–174. IEEE
[45]
Li X, Zhang Z, Stefanidis K (2018) Mobile app evolution analysis based on user reviews. In: SoMeT
[46]
Maldonado EdS, Shihab E, and Tsantalis N Using natural language processing to automatically detect self-admitted technical debt IEEE Trans Softw Eng 2017 43 11 1044-1062
[47]
Mathews C, Ye K, Grozdanovski J, Marinelli M, Zhong K, Khalajzadeh H, Obie HO, Grundy J (2021) Ah-cid: a tool to automatically detect human-centric issues in app reviews. In: ICSOFT, pp 386–397
[48]
Mazar N, Ariely D (2006) Dishonesty in everyday life and its policy implications. J Pub Pol Market 25(1):117–126. http://www.jstor.org/stable/30000530
[49]
Miller CB Honesty: the philosophy and psychology of a neglected virtue 2021 Oxford Oxford University Press USA - OSO
[50]
Morrissey ER Sources of error in the coding of questionnaire data Soc Methods Res 1974 3 2 209-232
[51]
Mougouei D (2020) Engineering human values in software through value programming. Proceedings of the IEEE/ACM 42nd international conference on software engineering workshops. pp 133–136
[52]
Mougouei D, Perera H, Hussain W, Shams R, Whittle J (2018) Operationalizing human values in software: a research roadmap. ESEC/FSE 2018, pp 780–784. Association for Computing Machinery, New York, NY, USA.
[53]
Nakamoto S (2009) Bitcoin: a peer-to-peer electronic cash system. http://www.bitcoin.org/bitcoin.pdf
[54]
Noble WS What is a support vector machine? Nat Biotechnol 2006 24 12 1565-1567
[55]
Nurwidyantoro A, Shahin M, Chaudron M, Hussain W, Perera H, Shams RA, Whittle J (2021) Towards a human values dashboard for software development: an exploratory study. In: Proceedings of the 15th ACM / IEEE international symposium on empirical software engineering and measurement (ESEM), ESEM ’21. Association for Computing Machinery, New York, NY, USA
[56]
Obie HO, Hussain W, Xia X, Grundy J, Li L, Turhan B, Whittle J, Shahin M (2021) A first look at human values-violation in app reviews. In: 2021 IEEE/ACM 43rd international conference on software engineering: software engineering in society (ICSE-SEIS), pp 29–38
[57]
Obie HO, Ilekura I, Du H, Shahin M, Grundy J, Li L, Whittle J, Turhan B (2022) On the violation of honesty in mobile apps: automated detection and categories. In: 2022 IEEE/ACM 19th international conference on mining software repositories (MSR), pp 321–332.
[58]
Obie HO, Ilekura I, Du H, Shahin M, Grundy J, Li L, Whittle J, Turhan B (2022) The replication repository of this manuscript. https://anonymous.4open.science/r/ml_app_reviews-3ED6/README. md
[59]
Obie HO, Shahin M, Grundy J, Turhan B, Li L, Hussain W, Whittle J (2021) Does domain change the opinion of individuals on human values? A preliminary investigation on ehealth apps end-users
[60]
O’Neill O (2002) Trust is the first casualty of the cult of transparency.https://www.telegraph.co.uk/comment/personal-view/3575750/Trust-is-the-first-casualty-of-the-cult-of-transparency.html
[61]
Palomba F, Linares-Vásquez M, Bavota G, Oliveto R, Di Penta M, Poshyvanyk D, De Lucia A (2015) User reviews matter! Tracking crowdsourced reviews to support evolution of successful apps. In: 2015 IEEE international conference on software maintenance and evolution (ICSME), pp 291–300.
[62]
Panichella S, Di Sorbo A, Guzman E, Visaggio CA, Canfora G, Gall HC (2015) How can i improve my app? Classifying user reviews for software maintenance and evolution. In: 2015 IEEE international conference on software maintenance and evolution (ICSME), pp 281–290.
[63]
Pelloni L, Grano G, Ciurumelea A, Panichella S, Palomba F, Gall HC (2018) Becloma: augmenting stack traces with user review information. In: 2018 IEEE 25th international conference on software analysis, evolution and reengineering (SANER), pp 522–526.
[64]
Perera H, Hussain W, Mougouei D, Shams RA, Nurwidyantoro A, Whittle J (2019) Towards integrating human values into software: mapping principles and rights of gdpr to values. In: 2019 IEEE 27th international requirements engineering conference (RE), pp 404–409
[65]
Perera H, Hussain W, Whittle J, Nurwidyantoro A, Mougouei D, Shams RA, Oliver G (2020) A study on the prevalence of human values in software engineering publications, 2015 – 2018. In: Proceedings of the ACM/IEEE 42nd international conference on software engineering, ICSE ’20, pp 409–420. Association for Computing Machinery, New York, NY, USA.
[66]
Perez S (2019) Dating app maker match sued by ftc for fraud. https://techcrunch.com/2019/09/26/dating-app-maker-match-sued-by-ftc-for-fraud/
[67]
Peters F, Tun TT, Yu Y, and Nuseibeh B Text filtering and ranking for security bug report prediction IEEE Trans Softw Eng 2017 45 6 615-631
[68]
Phong MV, Nguyen TT, Pham HV, Nguyen TT (2015) Mining user opinions in mobile app reviews: a keyword-based approach. In: 2015 30th IEEE/ACM international conference on automated software engineering (ASE), pp 749–759.
[69]
Rokeach M The Nature of Human Values 1973 Free Press
[70]
Samhi J, Li L, Bissyandé TF, Klein J (2022) Difuzer: uncovering suspicious hidden sensitive operations in android apps. In: The 44th international conference on software engineering (ICSE 2022)
[71]
Schwartz S (1992) Universals in the content and structure of values: theoretical advances and empirical tests in 20 countries. Adv Exp Soc Psychol 25
[72]
Schwartz S (2012) An overview of the schwartz theory of basic values. Online Readings in Psychology and Culture 2
[73]
Seyff N, Graf F, Maiden N (2010) Using mobile re tools to give end-users their own voice. In: 2010 18th IEEE international requirements engineering Conference, pp 37–46.
[74]
Shaffery P (2021) Cyber security: When the cover up is worse than the crime: uber & the consequences of hiding a data breach. https://www.pooleshaffery.com/news/2017/december/cyber-security-when-the-cover-up-is-worse-than-t/
[75]
Shams RA, Hussain W, Oliver G, Nurwidyantoro A, Perera H, Whittle J (2020) Society-oriented applications development: investigating users’ values from bangladeshi agriculture mobile applications. In: 2020 IEEE/ACM 42nd international conference on software engineering: software engineering in society (ICSE-SEIS), pp 53–62. IEEE
[76]
Shams RA, Shahin M, Oliver G, Hussain W, Perera H, Nurwidyantoro A, Whittle J (2021) Measuring bangladeshi female farmers’ values for agriculture mobile applications development. In: 54th Hawaii international conference on system sciences, HICSS’21, pp 1–10
[77]
Sullins J (2018) Information technology and moral values.https://plato.stanford.edu/entries/it-moral-values/
[78]
Whittle J Is your software valueless? IEEE Software 2019 36 3 112-115
[79]
Whittle J, Ferrario MA, Simm W, and Hussain W A case for human values in software engineering IEEE Software 2021 38 1 106-113
[80]
Winner L (1980) Do artifacts have politics? Daedalus 109(1):121–136. http://www.jstor.org/stable/20024652
[81]
Winter E, Forshaw S, Ferrario MA (2018) Measuring human values in software engineering. In: 2018 ACM/IEEE 12th international symposium on empirical software engineering and measurement, pp 1–4
[82]
Yiacoumi R (2021) Online educator shaw academy to refund students: ‘free trial’ charged students even when they cancelled. https://ia.acs.org.au/article/2021/online-educator-shaw-academy-to-refund-students.html
[83]
Zhu L, Xu X, Lu Q, Governatori G, Whittle J (2021) Ai and ethics – operationalising responsible ai
[84]
Zhu Y, Kiros R, Zemel R, Salakhutdinov R, Urtasun R, Torralba A, Fidler S (2015) Aligning books and movies: towards story-like visual explanations by watching movies and reading books. In: Proceedings of the IEEE international conference on computer vision, pp 19–27
Index Terms
- Automated detection, categorisation and developers’ experience with the violations of honesty in mobile apps
Index terms have been assigned to the content through auto-classification.
Recommendations
An Explorative Study of the Mobile App Ecosystem from App Developers' Perspective
WWW '17: Proceedings of the 26th International Conference on World Wide WebWith the prevalence of smartphones, app markets such as Apple App Store and Google Play has become the center stage in the mobile app ecosystem, with millions of apps developed by tens of thousands of app developers in each major market. This paper ...
Spam Mobile Apps: Characteristics, Detection, and in the Wild Analysis
The increased popularity of smartphones has attracted a large number of developers to offer various applications for the different smartphone platforms via the respective app markets. One consequence of this popularity is that the app markets are also ...
Early Detection of Spam Mobile Apps
WWW '15: Proceedings of the 24th International Conference on World Wide WebIncreased popularity of smartphones has attracted a large number of developers to various smartphone platforms. As a result, app markets are also populated with spam apps, which reduce the users' quality of experience and increase the workload of app ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
© The Author(s) 2023.
Publisher
Kluwer Academic Publishers
United States
Publication History
Published: 27 September 2023
Accepted: 12 June 2023
Author Tags
Qualifiers
- Research-article
Funding Sources
- Monash University
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 0Total Downloads
- Downloads (Last 12 months)0
- Downloads (Last 6 weeks)0
Reflects downloads up to 21 Dec 2024