“Not quite there yet”: On Users Perception of Popular Healthcare Chatbot Apps for Personal Health Management
Pages 191 - 197
Abstract
Many individuals rely on digital resources for advice related to their health management such as passive information on web or more active resources such as chatbots. Chatbot technology has made rapid technical advances in recent years and holds potential for making health information accessible to a wider range of individuals including sparsely populated and rural areas. One challenge with new technology is the gap that can occur with the functionality offered by the technology and the actual needs of users. Not only is it important that health related information and advice is accurate and correct as it directly can affect individuals’ health-related decisions, but the user experience must be positive, and users must trust the technology. This study explores how users perceive four popular health related chatbot apps by analyzing 708 reviews. The results confirm that there is a gap between users’ needs and the user experience provided by the chatbots. Suggestions for chatbot developers are provided which could help reduce the gap between the available functionalities and users’ needs.
References
[1]
Alaa Abd-Alrazaq, Zeineb Safi, Mohannad Alajlani, Jim Warren, Mowafa Househ, and Kerstin Denecke. 2020. Technical metrics used to evaluate health care chatbots: scoping review. Journal of medical Internet research, 22(6), e18301.
[2]
Eleni Adamopoulou and Lefteris Moussiades Adamopoulou. 2020. Chatbots: History, technology, and applications. Machine Learning with Applications, 2(100006). https://doi.org/https://doi.org/10.1016/j.mlwa.2020.100006.
[3]
Flora Amato, Stefano Marrone, Vincenzo Moscato, Gabriele Piantadosi, Antonio Picariello, and Carlo Sansone. 2017. Chatbots Meet eHealth: Automatizing Healthcare. In WAIAH@ AI* IA (pp. 40-49).
[4]
Tae Hyun Baek and Minseong Kim. 2023. Is ChatGPT scary good? How user motivations affect creepiness and trust in generative artificial intelligence. Telematics and Informatics, 83, 102030.
[5]
Mary Bates. 2019. Health care chatbots are here to help. IEEE pulse, 10(3), 12-14.
[6]
Gerd Berget and Frode Eika Sandnes. 2019. Why textual search interfaces fail: a study of cognitive skills needed to construct successful queries. Information Research: An International Electronic Journal, 24(1), n1.
[7]
Nivedita Bhirud, Subhash Tataale, Sayali Randive, and Shubham Nahar. 2019. A literature review on chatbots in healthcare domain. International journal of scientific & technology research, 8(7), 225-231.
[8]
Svetlana Bialkova. 2023. How to Optimise Interaction with Chatbots? Key Parameters Emerging from Actual Application. International Journal of Human–Computer Interaction, 1-10.
[9]
Marouane Birjali, Mohammed Kasri, and Abderrahim Beni-Hssane. 2021. A comprehensive survey on sentiment analysis: Approaches, challenges and trends. Knowledge-Based Systems. 226(107134). https://doi.org/https://doi.org/10.1016/j.knosys.2021.107134.
[10]
Gillian Cameron, David Cameron, Gavin Megaw, R. R. Bond, Maurice Mulvenna, Siobhan O'Neill, C Armour, and Michael McTear. 2018. Best practices for designing chatbots in mental healthcare–A case study on iHelpr. In: British HCI Conference 2018. BCS Learning & Development Ltd.
[11]
Jiahao Chen, Fu Guo, Zenggen Ren, Mingming Li, andJaap Ham. 2023. Effects of anthropomorphic design cues of chatbots on users’ perception and visual behaviors. International Journal of Human–Computer Interaction, 1-19.
[12]
Qiuyuan Chen, Chunyang Chen, Safwat Hassan, Zhengchang Xing, Xin Xia, and Ahmed E. Hassan. 2021. How Should I Improve the UI of My App? A Study of User Reviews of Popular Apps in the Google Play. ACM Trans. Softw. Eng. Methodol. 30, 3, Article 37 (July 2021), 38 pages. https://doi.org/10.1145/3447808
[13]
Yuyan Chen, Qiang Fu, Yichen Yuan, Zhihao Wen, Ge Fan, Dayiheng Liu, Dongmei Zhang, Zhixu Li, and Yanghua Xiao. 2023. Hallucination Detection: Robustly Discerning Reliable Answers in Large Language Models. In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management (CIKM '23). Association for Computing Machinery, New York, NY, USA, 245–255. https://doi.org/10.1145/3583780.3614905
[14]
Jayati Dev and L. Jean Camp. 2020. User Engagement with Chatbots: A Discursive Psychology Approach. In Proceedings of the 2nd Conference on Conversational User Interfaces (CUI '20). Association for Computing Machinery, New York, NY, USA, Article 52, 1–4. https://doi.org/10.1145/3405755.3406165
[15]
Hanyu Duan, Yi Yang, and Kar Yan Tam. 2024. Do LLMs Know about Hallucination? An Empirical Investigation of LLM's Hidden States. arXiv preprint arXiv:2402.09733. https://doi.org/10.48550/arXiv.2402.09733
[16]
Faruk Lawal Ibrahim Dutsinma, Debajyoti Pal, Suree Funilkul, and Jonathan H. Chan. 2022. A systematic review of voice assistant usability: An iso 9241–11 approach. SN Computer Science, 3(4), 267.
[17]
Amir Fayazi, Kyumin Lee, James Caverlee, and Anna Squicciarini. 2015. Uncovering Crowdsourced Manipulation of Online Reviews. In Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR '15). Association for Computing Machinery, New York, NY, USA, 233–242. https://doi.org/10.1145/2766462.2767742
[18]
Fotos Frangoudes, Marios Hadjiaros, Eirini C. Schiza, Maria Matsangidou, Olia Tsivitanidou, and Kleanthis Neokleous. 2021. An overview of the use of chatbots in medical and healthcare education. In International Conference on Human-Computer Interaction (pp. 170-184). Cham: Springer International Publishing.
[19]
Emitza Guzman and Walid Maalej. 2014. How do users like this feature? a fine grained sentiment analysis of app reviews. 2014 IEEE 22nd international requirements engineering conference. IEEE, 153-162.
[20]
Samuel Holmes, Anne Moorhead, Raymond Bond, Huiru Zheng, Vivien Coates, and Michael Mctear. 2019. Usability testing of a healthcare chatbot: Can we use conventional methods to assess conversational user interfaces? In Proceedings of the 31st European Conference on Cognitive Ergonomics (ECCE '19). Association for Computing Machinery, New York, NY, USA, 207–214. https://doi.org/10.1145/3335082.3335094
[21]
Mlađan Jovanović, Marcos Baez, and Fabio Casati. 2020. Chatbots as conversational healthcare services. IEEE Internet Computing, 25(3), 44-51.
[22]
Tobias Kowatsch, Marcia Nißen, Chen-Hsuan I.Shih, Dominik Rüegger, Dirk Volland, Andreas Filler, Florian Künzler, Filipe Barata, Sandy Hung, Dirk Büchter, Björn Brogle, Katrin Heldt, Pauline Gindrat, Nathalie Farpour-Lambert. 2017. Text-based healthcare chatbots supporting patient and health professional teams: preliminary results of a randomized controlled trial on childhood obesity. Persuasive Embodied Agents for Behavior Change (PEACH2017).
[23]
Shanshan Li, James Caverlee, Wei Niu, and Parisa Kaghazgaran. 2017. Crowdsourced App Review Manipulation. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR '17). Association for Computing Machinery, New York, NY, USA, 1137–1140. https://doi.org/10.1145/3077136.3080741
[24]
Maryam Lotfigolian, Christos Papanikolaou, Samaneh Taghizadeh, and Frode Eika Sandnes. 2023. Human Experts’ Perceptions of Auto-Generated Summarization Quality. In Proceedings of the 16th International Conference on PErvasive Technologies Related to Assistive Environments (PETRA '23). Association for Computing Machinery, New York, NY, USA, 95–98. https://doi.org/10.1145/3594806.3594828
[25]
Richard May and Kerstin Denecke. 2022. Security, privacy, and healthcare-related conversational agents: a scoping review. Informatics for Health and Social Care, 47(2), 194-210.
[26]
Quim Motger, Xavier Franch, and Jordi Marco. 2022. Software-Based Dialogue Systems: Survey, Taxonomy, and Challenges. ACM Comput. Surv. 55, 5, Article 91 (May 2023), 42 pages. https://doi.org/10.1145/3527450t
[27]
Quynh N. Nguyen, Anna Sidorova, and Russell Torres. 2022. User interactions with chatbot interfaces vs. Menu-based interfaces: An empirical study. Computers in Human Behavior, 128(107093). https://doi.org/https://doi.org/10.1016/j.chb.2021.107093
[28]
Adam Palanica, Peter Flaschner, Anirudh Thommandram, Michael Li, Yan Fossat. 2019. Physicians’ perceptions of chatbots in health care: cross-sectional web-based survey. Journal of medical Internet research, 21(4), e12887.
[29]
Zeineb Safi, Alaa Abd-Alrazaq, Mohamed Khalifa, Mowafa Househ. 2020. Technical aspects of developing chatbots for medical applications: scoping review. Journal of medical Internet research, 22(12), e19127.
[30]
Gabriel Zúñiga Salazar, Diego Zúñiga, Carlos L. Vindel, Ana M. Yoong, Sofia Hincapie, Ana B. Zúñiga, Paula Zúñiga, Erin Salazar, Byron Zúñiga. 2023. Efficacy of AI Chats to Determine an Emergency: A Comparison Between OpenAI's ChatGPT, Google Bard, and Microsoft Bing AI Chat. Cureus, 15(9), e45473. https://doi.org/10.7759/cureus.45473
[31]
Frode Eika Sandnes and Hua-Li Jian. 2004. Pair-wise variability index: Evaluating the cognitive difficulty of using mobile text entry systems. In International Conference on Mobile Human-Computer Interaction (pp. 347-350). Berlin, Heidelberg: Springer Berlin Heidelberg.
[32]
Nima Shahbazi, Yin Lin, Abolfazl Asudeh, and H. V. Jagadish. 2023. Representation Bias in Data: A Survey on Identification and Resolution Techniques. ACM Comput. Surv. 55, 13s, Article 293 (December 2023), 39 pages. https://doi.org/10.1145/3588433
[33]
Yi Shan, Meng Ji, Wenxiu Xie, Xiaobo Qian, Rongying Li, Xiaomin Zhang, Tianyong Hao. 2022. Language Use in Conversational Agent–Based Health Communication: Systematic Review. Journal of Medical Internet Research, 24(7), e37403.
[34]
Supawadee Suppadungsuk, Charat Thongprayoon, JingMiao, Pajaree Krisanapan, Fawad Qureshi, Kianoush Kashani, and Wisit Cheungpasitporn. 2023. Exploring the Potential of Chatbots in Critical Care Nephrology. Medicines (Basel), 10(10). https://doi.org/10.3390/medicines10100058
[35]
Colm Sweeney, Courtney Potts, Edel Ennis, Raymond Bond, Maurice D. Mulvenna, Siobhan O'neill, Martin Malcolm, Lauri Kuosmanen, Catrine Kostenius, Alex Vakaloudis, Gavin Mcconvey, Robin Turkington, David Hanna, Heidi Nieminen, Anna-Kaisa Vartiainen, Alison Robertson, and Michael F. Mctear. 2021. Can Chatbots Help Support a Person's Mental Health? Perceptions and Views from Mental Healthcare Professionals and Experts. ACM Trans. Comput. Healthcare 2, 3, Article 25 (July 2021), 15 pages. https://doi.org/10.1145/3453175
[36]
Chenxing Xie, Yanding Wang, and Yang Cheng. 2024. Does artificial intelligence satisfy you? A meta-analysis of user gratification and user satisfaction with AI-powered chatbots. International Journal of Human–Computer Interaction, 40(3), 613-623.
[37]
Thea Bratteberg Ytterland, Siri Fagernes, and Frode Eika Sandnes. 2022. Perceptions of Digital Nudging for Cervical Testing: A Comparison Four Nudge Types. In International Conference on Human-Computer Interaction (pp. 212-228). Cham: Springer International Publishing.
[38]
Jia-Yu Yao, Kun-Peng Ning, Zhen-Hui Liu, Mu-Nan Ning, and Li Yuan. 2023. Llm lies: Hallucinations are not bugs, but features as adversarial examples. arXiv preprint arXiv:2310.01469. https://doi.org/10.48550/arXiv.2310.01469%3c/bib>
Index Terms
- “Not quite there yet”: On Users Perception of Popular Healthcare Chatbot Apps for Personal Health Management
Recommendations
"This Chatbot Would Never...": Perceived Moral Agency of Mental Health Chatbots
CSCWDespite repeated reports of socially inappropriate and dangerous chatbot behaviour, chatbots are increasingly used as mental health services in providing support for young people. In sensitive settings as such, the notion of perceived moral agency (PMA) ...
Measuring the Effect of Mental Health Chatbot Personality on User Engagement
MUM '22: Proceedings of the 21st International Conference on Mobile and Ubiquitous MultimediaArtificial Intelligence is seen as humanity’s current best bet to solve the looming crisis in healthcare. Conversational Agents, or chatbots, rely on advances in AI and are increasingly investigated in the context of digital mental health care. Given ...
User Perception of Text-Based Chatbot Personality
Chatbot Research and DesignAbstractThis work explores the effect of chatbot personality on user experience and investigates how users perceive agent personality when conveyed through text. Building on previous work in the field of human-computer interaction on designing chatbot ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
June 2024
708 pages
ISBN:9798400717604
DOI:10.1145/3652037
Copyright © 2024 Owner/Author.
This work is licensed under a Creative Commons Attribution International 4.0 License.
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 26 June 2024
Check for updates
Author Tags
Qualifiers
- Research-article
- Research
- Refereed limited
Conference
PETRA '24
PETRA '24: The PErvasive Technologies Related to Assistive Environments Conference
June 26 - 28, 2024
Crete, Greece
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 360Total Downloads
- Downloads (Last 12 months)360
- Downloads (Last 6 weeks)77
Reflects downloads up to 04 Jan 2025
Other Metrics
Citations
View Options
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderHTML Format
View this article in HTML Format.
HTML FormatLogin options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in