Trust by Interface: How Different User Interfaces Shape Human Trust in Health Information from Large Language Models
Article No.: 344, Pages 1 - 7
Abstract
The integration of Large Language Models (LLMs) with Conversational User Interfaces (CUIs) has significantly transformed health information seeking, offering interactive access to health resources. Despite the importance of trust in adopting health advice, the impact of user interfaces on trust perception in LLM-provided information remains unclear. Our mixed-methods study investigated how different CUIs (text-based, speech-based, and embodied) influence trust when using an identical LLM source. Key findings include (a) higher trust levels in information delivered via text-based interface compared to others; (b) a significant correlation between trust in the interface and the information provided; (c) participant’s prior experience, processing approach for information with different modalities and presentation styles, and usability level were key determinants of trust in health-related information. Our study sheds light on trust perceptions in health information from LLMs and its dissemination, underscoring the importance of user interface in trustworthy and effective health information seeking with LLM-powered CUIs.
Supplemental Material
- Download
- 48.02 MB
- Transcript
References
[1]
adaTakeCare. 2024. Take care of yourself with Ada. More than a symptom checker. — ada.com. https://ada.com/app/.
[2]
Babylon. 2024. Symptom Checker | eMed UK — babylonhealth.com. https://www.babylonhealth.com/en-us/what-we-offer/chatbot.
[3]
Asma Ben Abacha and Dina Demner-Fushman. 2019. A Question-Entailment Approach to Question Answering. BMC Bioinform. 20, 1 (2019), 511:1–511:23. https://bmcbioinformatics.biomedcentral.com/articles/10.1186/s12859-019-3119-4
[4]
Som S. Biswas. 2023. Role of ChatGPT in Public Health. Annals of Biomedical Engineering 51 (2023), 868–869. https://doi.org/10.1007/s10439-023-03172-7
[5]
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language Models are Few-Shot Learners. In Advances in Neural Information Processing Systems, H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (Eds.). Vol. 33. Curran Associates, Inc., 1877–1901. https://proceedings.neurips.cc/paper_files/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf
[6]
Astrid Carolus, Martin J. Koch, Samantha Straka, Marc Erich Latoschik, and Carolin Wienrich. 2023. MAILS - Meta AI literacy scale: Development and testing of an AI literacy questionnaire based on well-founded competency models and psychological change- and meta-competencies. Computers in Human Behavior: Artificial Humans 1, 2 (2023), 100014. https://doi.org/10.1016/j.chbah.2023.100014
[7]
Astrid Carolus and Carolin Wienrich. 2022. “Imagine this smart speaker to have a body”: An analysis of the external appearances and the characteristics that people associate with voice assistants. Frontiers in Computer Science 4 (2022). https://doi.org/10.3389/fcomp.2022.981435
[8]
K. R. Chowdhary. 2020. Natural Language Processing. Springer India, New Delhi, 603–649. https://doi.org/10.1007/978-81-322-3972-7_19
[9]
R. J. W. Cline and K. M. Haynes. 2001. Consumer health information seeking on the Internet: the state of the art. Health Education Research 16, 6 (12 2001), 671–692. https://doi.org/10.1093/her/16.6.671 arXiv:https://academic.oup.com/her/article-pdf/16/6/671/9809432/160671.pdf
[10]
Cynthia Corritore, Susan Wiedenbeck, Beverly Kracher, and Robert Marble. 2007. Online Trust and Health Information Websites. International Journal of Technology and Human Interaction 8 (01 2007). https://doi.org/10.4018/jthi.2012100106
[11]
Jeffrey Dalton, Sophie Fischer, Paul Owoicho, Filip Radlinski, Federico Rossetto, Johanne R. Trippas, and Hamed Zamani. 2022. Conversational Information Seeking: Theory and Application. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval (Madrid, Spain) (SIGIR ’22). Association for Computing Machinery, New York, NY, USA, 3455–3458. https://doi.org/10.1145/3477495.3532678
[12]
Caroline de Cock, Madison Milne-Ives, Michelle Helena van Velthoven, Abrar Alturkistani, Ching Lam, and Edward Meinert. 2020. Effectiveness of conversational agents (virtual assistants) in health care: Protocol for a systematic review. JMIR Res. Protoc. 9, 3 (March 2020), e16934.
[13]
Franz Faul, Edgar Erdfelder, Albert-Georg Lang, and Axel Buchner. 2007. G*Power 3: a flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behav. Res. Methods 39, 2 (May 2007), 175–191.
[14]
Asbjørn Følstad, Cecilie Bertinussen Nordheim, and Cato Alexander Bjørkli. 2018. What Makes Users Trust a Chatbot for Customer Service? An Exploratory Interview Study. In Internet Science, Svetlana S. Bodrunova (Ed.). Springer International Publishing, Cham, 194–208.
[15]
Ann L Fruhling and Sang M Lee. 2006. The influence of user interface usability on rural consumers’ trust of e-health services. International journal of electronic healthcare 2, 4 (2006), 305–321.
[16]
Luciano Gamberini, Patrik Pluchino, Luigi Porto, Filippo Zordan, Alessandro Monterosso, Kumara Palanivel, and Adriano Fontanari. 2021. PatchAi: An e-Health Application Powered by an AI Virtual Assistant to Support Patients in Their Clinical Trials. 315–322. https://doi.org/10.1007/978-3-030-78645-8_39
[17]
Oscar A. Garcia Valencia, Supawadee Suppadungsuk, Charat Thongprayoon, Jing Miao, Supawit Tangpanithandee, Iasmina M. Craici, and Wisit Cheungpasitporn. 2023. Ethical Implications of Chatbot Utilization in Nephrology. Journal of Personalized Medicine 13, 9 (2023). https://doi.org/10.3390/jpm13091363
[18]
Ravindra Kumar Garg, Vijeth L Urs, Akshya Anand Agrawal, Sarvesh Kumar Chaudhary, Vimal Paliwal, and Sujita Kumar Kar. 2023. Exploring the Role of ChatGPT in patient care (diagnosis and Treatment) and medical research: A Systematic Review. medRxiv (2023). https://doi.org/10.1101/2023.06.13.23291311
[19]
Zhongxuan He. 2023. From ELIZA to ChatGPT: The Evolution of Chatbots in Public Health. XRDS 29, 3 (apr 2023), 59. https://doi.org/10.1145/3589658
[20]
Sarah Jessup, Tamera Schneider, Gene Alarcon, Tyler Ryan, and August Capiola. 2019. The Measurement of the Propensity to Trust Automation. 476–489. https://doi.org/10.1007/978-3-030-21565-1_32
[21]
Frances C. Johnson, Jennifer E. Rowley, and Laura Sbaffi. 2015. Modelling trust formation in health information contexts. Journal of Information Science 41 (2015), 415 – 429. https://api.semanticscholar.org/CorpusID:206454953
[22]
Haruna Kibirige and Lila Ghemri. 2014. Trust Ranking of Medical Websites. In Proceedings of the 4th ACM Conference on Data and Application Security and Privacy (San Antonio, Texas, USA) (CODASPY ’14). Association for Computing Machinery, New York, NY, USA, 151–154. https://doi.org/10.1145/2557547.2557584
[23]
Michael R King. 2023. The Future of AI in Medicine: A Perspective from a Chatbot. Annals of Biomedical Engineering 51, 2 (Feb. 2023), 291–295.
[24]
Liliana Laranjo, Adam G Dunn, Huong Ly Tong, Ahmet Baki Kocaballi, Jessica Chen, Rabia Bashir, Didi Surian, Blanca Gallego, Farah Magrabi, Annie Y S Lau, and Enrico Coiera. 2018. Conversational agents in healthcare: a systematic review. Journal of the American Medical Informatics Association 25, 9 (07 2018), 1248–1258. https://doi.org/10.1093/jamia/ocy072 arXiv:https://academic.oup.com/jamia/article-pdf/25/9/1248/34150600/ocy072.pdf
[25]
John D Lee and Katrina A See. 2004. Trust in automation: designing for appropriate reliance. Hum. Factors 46, 1 (2004), 50–80.
[26]
Peter Lee, Sebastien Bubeck, and Joseph Petro. 2023. Benefits, Limits, and Risks of GPT-4 as an AI Chatbot for Medicine. New England Journal of Medicine 388, 13 (2023), 1233–1239. https://doi.org/10.1056/NEJMsr2214184 arXiv:https://doi.org/10.1056/NEJMsr2214184PMID: 36988602.
[27]
Florian Leiser, Sven Eckhardt, Merlin Knaeble, Alexander Maedche, Gerhard Schwabe, and Ali Sunyaev. 2023. From ChatGPT to FactGPT: A Participatory Design Study to Mitigate the Effects of Large Language Model Hallucinations on Users. In Proceedings of Mensch Und Computer 2023 (Rapperswil, Switzerland) (MuC ’23). Association for Computing Machinery, New York, NY, USA, 81–90. https://doi.org/10.1145/3603555.3603565
[28]
James R. Lewis, Brian S. Utesch, and Deborah E. Maher. 2013. UMUX-LITE: when there’s no time for the SUS. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Paris, France) (CHI ’13). Association for Computing Machinery, New York, NY, USA, 2099–2102. https://doi.org/10.1145/2470654.2481287
[29]
Ewa Luger and Abigail Sellen. 2016. "Like Having a Really Bad PA": The Gulf between User Expectation and Experience of Conversational Agents. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (San Jose, California, USA) (CHI ’16). Association for Computing Machinery, New York, NY, USA, 5286–5297. https://doi.org/10.1145/2858036.2858288
[30]
Michal Luria, Guy Hoffman, and Oren Zuckerman. 2017. Comparing Social Robot, Screen and Voice Interfaces for Smart-Home Control. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (Denver, Colorado, USA) (CHI ’17). Association for Computing Machinery, New York, NY, USA, 580–628. https://doi.org/10.1145/3025453.3025786
[31]
MAYO CLINIC. 2023. MAYO CLINIC. https://www.mayoclinic.org/
[32]
Joao Luis Zeni Montenegro, Cristiano André da Costa, and Rodrigo da Rosa Righi. 2019. Survey of conversational agents in health. Expert Systems with Applications 129 (2019), 56–67. https://doi.org/10.1016/j.eswa.2019.03.054
[33]
Cameron D Norman and Harvey A Skinner. 2006. EHEALS: The eHealth Literacy Scale. J. Med. Internet Res. 8, 4 (Nov. 2006), e27.
[34]
OpenAI. 2023. GPT-4 Technical Report. arxiv:2303.08774 [cs.CL]
[35]
Adam Palanica, Peter Flaschner, Anirudh Thommandram, Michael Li, and Yan Fossat. 2018. Physicians’ Perceptions of Chatbots in Health Care: Cross-Sectional Web-Based Survey. Journal of Medical Internet Research 21 (11 2018). https://doi.org/10.2196/12887
[36]
Amit Kumar Pandey and Rodolphe Gelin. 2018. A Mass-Produced Sociable Humanoid Robot: Pepper: The First Machine of Its Kind. IEEE Robotics & Automation Magazine PP (07 2018), 1–1. https://doi.org/10.1109/MRA.2018.2833157
[37]
Abhishek Vaish Raju Vaishya, Anoop Misra. 2023. ChatGPT: Is this version good for healthcare and research?Diabetes and Metabolic Syndrome: Clinical Research and Reviews 17 (2023), 1871–4021. https://doi.org/10.1016/j.dsx.2023.102744
[38]
Minjin Rheu, Ji Youn Shin, Wei Peng, and Jina Huh-Yoo. 2020. Systematic Review: Trust-Building Factors and Implications for Conversational Agent Design. International Journal of Human-Computer Interaction 37 (09 2020), 1–16. https://doi.org/10.1080/10447318.2020.1807710
[39]
Jennifer E. Rowley, Frances C. Johnson, and Laura Sbaffi. 2015. Students’ trust judgements in online health information seeking. Health Informatics Journal 21 (2015), 316 – 327. https://api.semanticscholar.org/CorpusID:21888204
[40]
James Schaffer, John O’Donovan, James Michaelis, Adrienne Raglin, and Tobias Höllerer. 2019. I Can Do Better than Your AI: Expertise and Explanations. In Proceedings of the 24th International Conference on Intelligent User Interfaces (Marina del Ray, California) (IUI ’19). Association for Computing Machinery, New York, NY, USA, 240–251. https://doi.org/10.1145/3301275.3302308
[41]
Elizabeth Sillence, Pam Briggs, Peter Richard Harris, and Lesley Fishwick. 2007. How do patients evaluate and make use of online health information?Social Science and Medicine 64, 9 (2007), 1853–1862. https://doi.org/10.1016/j.socscimed.2007.01.012
[42]
Matthias Söllner, Axel Hoffmann, Maximilian Eike, Hirdes, Liudmila Rudakova, Stefanie Leimeister, and Jan Marco Leimeister. 2012. Towards a Formative Measurement Model for Trust. (10 2012).
[43]
Kristian Torning and Harri Oinas-Kukkonen. 2009. Persuasive system design : State of the art and future directions, Vol. 350. 30. https://doi.org/10.1145/1541948.1541989
[44]
Liesbet Van Bulck and Philip Moons. 2023. What if your patient switches from Dr. Google to Dr. ChatGPT? A vignette-based survey of the trustworthiness, value, and danger of ChatGPT-generated responses to health questions. European Journal of Cardiovascular Nursing (04 2023), zvad038. https://doi.org/10.1093/eurjcn/zvad038
[45]
Wikipedia. 2024. Optimus (robot) — Wikipedia, The Free Encyclopedia. http://en.wikipedia.org/w/index.php?title=Optimus%20(robot)&oldid=1195180995.
[46]
Tianyu Wu, Shizhu He, Jingping Liu, Siqi Sun, Kang Liu, Qing-Long Han, and Yang Tang. 2023. A Brief Overview of ChatGPT: The History, Status Quo and Potential Future Development. IEEE/CAA Journal of Automatica Sinica 10, 5 (2023), 1122–1136. https://doi.org/10.1109/JAS.2023.123618
[47]
Yue You, Chun-Hua Tsai, Yao Li, Fenglong Ma, Christopher Heron, and Xinning Gui. 2023. Beyond Self-Diagnosis: How a Chatbot-Based Symptom Checker Should Respond. ACM Trans. Comput.-Hum. Interact. 30, 4, Article 64 (sep 2023), 44 pages. https://doi.org/10.1145/3589959
[48]
Lingxuan Zhu, Weiming Mou, and Rui Chen. 2023. Can the ChatGPT and other large language models with internet-connected database solve the questions and concerns of patient with prostate cancer and help democratize medical knowledge?Journal of Translational Medicine 21, 1 (19 Apr 2023), 269. https://doi.org/10.1186/s12967-023-04123-5
Index Terms
- Trust by Interface: How Different User Interfaces Shape Human Trust in Health Information from Large Language Models
Recommendations
To Trust or Not To Trust: How a Conversational Interface Affects Trust in a Decision Support System
WWW '22: Proceedings of the ACM Web Conference 2022Trust is an important component of human-AI relationships and plays a major role in shaping the reliance of users on online algorithmic decision support systems. With recent advances in natural language processing, text and voice-based conversational ...
Online Trust and Health Information Websites
Health care websites are important to people managing their health. But the quality of the information on health care websites varies, so it's hard for users to decide whether to trust their information. This study develops and tests a model of the ...
Examining Mobile Banking User Trust: A Tripartite Perspective
Building users' trust is crucial to alleviating their perceived risk and facilitating their usage of mobile banking. Drawing on a tripartite perspective of transference-based, personality-based and self-perception-based determinants, this research ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
May 2024
4761 pages
ISBN:9798400703317
DOI:10.1145/3613905
Copyright © 2024 Owner/Author.
Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 11 May 2024
Check for updates
Author Tags
Qualifiers
- Work in progress
- Research
- Refereed limited
Funding Sources
Conference
Acceptance Rates
Overall Acceptance Rate 6,164 of 23,696 submissions, 26%
Upcoming Conference
CHI 2025
- Sponsor:
- sigchi
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 591Total Downloads
- Downloads (Last 12 months)591
- Downloads (Last 6 weeks)106
Reflects downloads up to 11 Dec 2024
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign inFull Access
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderFull Text
View this article in Full Text.
Full TextHTML Format
View this article in HTML Format.
HTML Format