More Web Proxy on the site http://driver.im/

research-article

Key Qualities of Conversational Recommender Systems: From Users’ Perspective

Authors:

Pearl PuAuthors Info & Claims

HAI '21: Proceedings of the 9th International Conference on Human-Agent Interaction

Pages 93 - 102

https://doi.org/10.1145/3472307.3484164

Published: 09 November 2021 Publication History

Abstract

An increasing number of recommender systems enable conversational interaction to enhance the system’s overall user experience (UX). However, it is unclear what qualities of a conversational recommender system (CRS) are essential to determine the success of a CRS. This paper presents a model to capture the key qualities of conversational recommender systems and their related user experience aspects. Our model incorporates the characteristics of conversations (such as adaptability, understanding, response quality, rapport, humanness, etc.) in four major user experience dimensions of the recommender system: User Perceived Qualities, User Belief, User Attitudes, and Behavioral Intentions. Following the psychometric modeling method, we validate the combined metrics using the data collected from an online user study of a conversational music recommender system. The user study results 1) support the consistency, validity, and reliability of the model that identifies seven key qualities of a CRS; and 2) reveal how conversation constructs interact with recommendation constructs to influence the overall user experience of a CRS. We believe that the key qualities identified in the model help practitioners design and evaluate conversational recommender systems.

References

[1]

[1] Y. Jin et al. Musicbot: Evaluating critiquing-based music recommenders with conversational interaction. In Proc. of CIKM’19, pp. 951–960, 2019.

[2]

[2] W. Cai et al. Critiquing for music exploration in conversational recommender systems. In Proc. of IUI’21, pp. 480–490, 2021.

[3]

[3] S. M. McNee et al. Being accurate is not enough: how accuracy metrics have hurt recommender systems. In Proc. of CHI’06 EA, pp. 1097–1101, 2006.

[4]

[4] J. A. Konstan and J. Riedl. Recommender systems: from algorithms to user experience. UMUAI, 22(1):101–123, 2012.

Digital Library

[5]

[5] M. Ge et al. Beyond accuracy: evaluating recommender systems by coverage and serendipity. In Proc. of RecSys’10, pp. 257–260, 2010.

[6]

[6] P. Pu et al. A user-centric evaluation framework for recommender systems. In Proc. of RecSys’11, pp. 157–164, 2011.

[7]

[7] B. P. Knijnenburg et al. Explaining the user experience of recommender systems. UMUAI, 22(4):441–504, 2012.

[8]

[8] F. Pecune et al. A model of social explanations for a conversational movie recommendation system. In Proc. of HAI’19, pp. 135–143, 2019.

[9]

[9] D. Jannach et al. A survey on conversational recommender systems. arXiv preprint arXiv:2004.00646, 2020.

[10]

[10] F. Ricci et al. Recommender Systems Handbook. Springer-Verlag, 2nd edition, 2015.

[11]

[11] K. Christakopoulou et al. Towards conversational recommender systems. In Proc. of KDD’16, pp. 815–824, 2016.

[12]

[12] L. Chen and P. Pu. Critiquing-based recommenders: Survey and emerging trends. UMUAI, 22(1-2):125–150, 2012.

Digital Library

[13]

[13] J. Kang et al. Understanding how people use natural language to ask for recommendations. In Proc. of RecSys’17, pp. 229–237, 2017.

[14]

[14] W. Cai and L. Chen. Predicting user intents and satisfaction with dialogue-based conversational recommendations. In Proc. of UMAP’20, pp. 33–42. ACM, 2020.

[15]

[15] Y. Sun and Y. Zhang. Conversational recommender system. In Proc. of SIGIR’18, pp. 235–244, 2018.

Digital Library

[16]

[16] K. Papineni et al. Bleu: a method for automatic evaluation of machine translation. In Proc of ACL’02, pp. 311–318, 2002.

[17]

[17] S. Fazeli et al. User-centric evaluation of recommender systems in social learning platforms: accuracy is just the tip of the iceberg. IEEE Transactions on Learning Technologies, 11(3):294–306, 2017.

[18]

[18] A. Said et al. User-centric evaluation of a k-furthest neighbor collaborative filtering recommender algorithm. In Proc. of CSCW’13, pp. 1399–1408, 2013.

[19]

[19] Y. Jin et al. Go with the flow: effects of transparency and user control on targeted advertising using flow charts. In Proc. of AVI’16, pp. 68–75, 2016.

[20]

[20] N. Tintarev and J. Masthoff. Evaluating the effectiveness of explanations for recommender systems. UMUAI, 22(4-5):399–439, 2012.

Digital Library

[21]

[21] P. Pu and L. Chen. Trust building with explanation interfaces. In Proc. of IUI’06, pp. 93–100, 2006.

Digital Library

[22]

[22] B. P. Knijnenburg et al. Inspectability and control in social recommenders. In Proc. of RecSys’12, pp. 43–50, 2012.

[23]

[23] D. Braun et al. Evaluating natural language understanding services for conversational question answering systems. In Proc. of SIGdial’17, pp. 174–185, 2017.

[24]

[24] R. Dale and C. Mellish. Towards evaluation in natural language generation. In Proc. of LREC’98, 1998.

[25]

[25] A. Ghandeharioun et al. Approximating interactive human evaluation with self-play for open-domain dialog systems. 32:13658–13669, 2019.

[26]

[26] M. Walker et al. Paradise: A framework for evaluating spoken dialogue agents. In Proc. of ACL’97, pp. 271–280, 1997.

[27]

[27] M. Turunen et al. Evaluation of a spoken dialogue system with usability tests and long-term pilot studies: Similarities and differences. In Proc. of ICSLP’06, 2006.

[28]

[28] Z. Ruttkay et al. Embodied conversational agents on a common ground. In From brows to trust, pp. 27–66. Springer, 2004.

[29]

[29] K. Kuligowska. Commercial chatbot: performance evaluation, usability metrics and quality standards of embodied conversational agents. Professionals Center for Business Research, 2, 2015.

[30]

[30] N. Radziwill and M. Benton. Evaluating quality of chatbots and intelligent conversational agents. Software Quality Professional, 19(3):25, 2017.

[31]

[31] M. Guerini et al. A methodology for evaluating interaction strategies of task-oriented conversational agents. In Proc. of EMNLP Workshop SCAI’18, pp. 24–32, 2018.

[32]

[32] R. Zhao et al. Sogo: a social intelligent negotiation dialogue system. In Proc. of IVA’18, pp. 239–246, 2018.

[33]

[33] E. Svikhuushina and P. Pu. Key qualities of conversational chatbots – the peace model. In Proc. of IUI’21, pp. 520–530, 2021.

[34]

[34] M. Walker et al. Towards developing general models of usability with paradise. Natural Language Engineering, 6(3 & 4):363–377, 2000.

[35]

[35] P. Cremonesi et al. Looking for “good” recommendations: A comparative evaluation of recommender systems. In INTERACT’11, pp. 152–168. Springer, 2011.

[36]

[36] N. Tintarev. Explanations of recommendations. In Proc. of RecSys’07, pp. 203–206, 2007.

Digital Library

[37]

[37] N. Tintarev and J. Masthoff. Designing and evaluating explanations for recommender systems. In Recommender systems handbook, pp. 479–510. Springer, 2011.

[38]

[38] F. Gedikli et al. How should i explain? a comparison of different explanation types for recommender systems. IJHCS, 72(4):367–382, 2014.

Digital Library

[39]

[39] H. Cramer et al. The effects of transparency on trust in and acceptance of a content-based art recommender. UMUAI, 18(5):455, 2008.

Digital Library

[40]

[40] J. L. Herlocker et al. Evaluating collaborative filtering recommender systems. ACM TOIS, 22(1):5–53, 2004.

Digital Library

[41]

[41] M. Kaminskas and D. Bridge. Diversity, serendipity, novelty, and coverage: a survey and empirical analysis of beyond-accuracy objectives in recommender systems. ACM TiiS, 7(1):1–42, 2016.

[42]

[42] F. Narducci et al. Improving the user experience with a conversational recommender system. In Proc. of AIxIA’18, pp. 528–538. Springer, 2018.

[43]

[43] B. Priyogi. Preference elicitation strategy for conversational recommender system. In Proc. of WSDM’19, pp. 824–825, 2019.

Digital Library

[44]

[44] L. Chen and P. Pu. Interaction design guidelines on critiquing-based recommender systems. UMUAI, 19(3):167, 2009.

Digital Library

[45]

[45] L. Tickle-Degnen and R. Rosenthal. The nature of rapport and its nonverbal correlates. Psychological inquiry, 1(4):285–293, 1990.

[46]

[46] A. Abdellatif et al. A comparison of natural language understanding platforms for chatbots in software engineering. arXiv preprint arXiv:2012.02640, 2020.

[47]

[47] P. Kataria et al. User adaptive chatbot for mitigating depression. International Journal of Pure and Applied Mathematics, 118(16):349–361, 2018.

[48]

[48] J. Seering et al. It takes a village: Integrating an adaptive chatbot into an online gaming community. In Proc. of CHI’20, pp. 1–13, 2020.

[49]

[49] D. Wang and H. Fang. Length adaptive regularization for retrieval-based chatbot models. In Proc. of SIGIR’20, pp. 113–120, 2020.

Digital Library

[50]

[50] J. Jiang and N. Ahuja. Response quality in human-chatbot collaborative systems. In Proc. of SIGIR’20, pp. 1545–1548, 2020.

Digital Library

[51]

[51] A. See et al. What makes a good conversation? how controllable attributes affect human judgments. In NAACL-HLT (1), 2019.

[52]

[52] Y. Jin et al. Contextplay: Evaluating user control for context-aware music recommendation. In Proc. of UMAP’19, pp. 294–302, 2019.

[53]

[53] Y. Hijikata et al. A study of user intervention and user satisfaction in recommender systems. Journal of information processing, 22(4):669–678, 2014.

[54]

[54] D. Jannach et al. User control in recommender systems: Overview and interaction challenges. In Proc. of EC-Web’16, pp. 21–33. Springer, 2016.

[55]

[55] B. P. Knijnenburg et al. Each to his own: how different users call for different interaction methods in recommender systems. In Proc. of RecSys’11, pp. 141–148, 2011.

[56]

[56] Y. Jin et al. Effects of personal characteristics in control-oriented user interfaces for music recommender systems. UMUAI, 30(2):199–249, 2020.

Digital Library

[57]

[57] D. Mican et al. Perceived usefulness: A silver bullet to assure user data availability for online recommendation systems. Decision Support Systems, 139:113420, 2020.

[58]

[58] S. G. Hart and L. E. Staveland. Development of nasa-tlx (task load index): Results of empirical and theoretical research. In Advances in psychology, volume 52, pp. 139–183. Elsevier, 1988.

[59]

[59] I. Simonson. Determinants of customers’ responses to customized offers: Conceptual framework and research propositions. Journal of marketing, 69(1):32–45, 2005.

[60]

[60] K. Swearingen and R. Sinha. Interaction design for recommender systems. In Proc. of DIS’02, volume 6, pp. 312–334. Citeseer, 2002.

[61]

[61] R. F. Kizilcec. How much information? effects of transparency on trust in an algorithmic interface. In Proc. of CHI’16, pp. 2390–2395, 2016.

[62]

[62] R. Sinha and K. Swearingen. The role of transparency in recommender systems. In Proc. of CHI’02 EA, pp. 830–831, 2002.

Digital Library

[63]

[63] D. Novick and I. Gris. Building rapport between human and eca: A pilot study. In Proc. of HCI International’14, pp. 472–480. Springer, 2014.

[64]

[64] L. D. Riek et al. When my robot smiles at me: Enabling human-robot rapport via real-time head gesture mimicry. JMUI, 3(1):99–108, 2010.

[65]

[65] M. Lalmas et al. Measuring user engagement. Synthesis lectures on information concepts, retrieval, and services, 6(4):1–132, 2014.

[66]

[66] H. L. O’Brien and E. G. Toms. What is user engagement? a conceptual framework for defining user engagement with technology. JASIST, 59(6):938–955, 2008.

[67]

[67] D. Adiwardana et al. Towards a human-like open-domain chatbot. arXiv preprint arXiv:2001.09977, 2020.

[68]

[68] E. Go and S. S. Sundar. Humanizing chatbots: The effects of visual, identity and conversational cues on humanness perceptions. Computers in Human Behavior, 97:304–316, 2019.

Digital Library

[69]

[69] D. Westerman et al. I believe in a thing called bot: Perceptions of the humanness of “chatbots”. Communication Studies, 70(3):295–312, 2019.

[70]

[70] H. Candello et al. Typefaces and the perception of humanness in natural language chatbots. In Proc. of CHI’17, pp. 3476–3487, 2017.

[71]

[71] R. M. Schuetzler et al. The impact of chatbot conversational skill on engagement and perceived humanness. JMIS, 37(3):875–900, 2020.

[72]

[72] N. Svenningsson and M. Faraon. Artificial intelligence in conversational agents: A study of factors related to perceived humanness in chatbots. In Proc. of AICCC’19, pp. 151–161, 2019.

Digital Library

[73]

[73] J. O’Donovan and B. Smyth. Trust in recommender systems. In Proc. of IUI’05, pp. 167–174, 2005.

Digital Library

[74]

[74] P. Massa and P. Avesani. Trust-aware recommender systems. In Proc. of RecSys’07, pp. 17–24, 2007.

Digital Library

[75]

[75] J. Kunkel et al. Let me explain: impact of personal and impersonal explanations on trust in recommender systems. In Proc. of CHI’19, pp. 1–12, 2019.

[76]

[76] L. Chen and P. Pu. A cross-cultural user evaluation of product recommender interfaces. In Proc. of RecSys’08, pp. 75–82, 2008.

Digital Library

[77]

[77] L. Wang et al. When in rome: the role of culture & context in adherence to robot recommendations. In Proc. of HRI’10, pp. 359–366. IEEE, 2010.

[78]

[78] S. Berkovsky et al. How to recommend? user trust factors in movie recommender systems. In Proc. of CHI’17, pp. 287–300, 2017.

[79]

[79] E. J. De Visser et al. Almost human: Anthropomorphism increases trust resilience in cognitive agents. Journal of Experimental Psychology: Applied, 22(3):331, 2016.

[80]

[80] A. Przegalinska et al. In bot we trust: A new methodology of chatbot performance measures. Business Horizons, 62(6):785–797, 2019.

[81]

[81] J. Edwards and E. Sanoubari. A need for trust in conversational interface research. In Proc. of CUI’19, pp. 1–3, 2019.

Digital Library

[82]

[82] J. A. Hoxmeier et al. The impact of gender and experience on user confidence in electronic mail. JOEUC, 12(4):11–20, 2000.

[83]

[83] S. Z. Arshad et al. Investigating user confidence for uncertainty presentation in predictive decision making. In Proceedings of OzCHI’15, pp. 352–360, 2015.

[84]

[84] T. T. Nguyen et al. User personality and user satisfaction with recommender systems. Information Systems Frontiers, 20(6):1173–1189, 2018.

Digital Library

[85]

[85] B. P. Knijnenburg and M. C. Willemsen. Understanding the effect of adaptive preference elicitation methods on user satisfaction of a recommender system. In Proc. of RecSys’09, pp. 381–384, 2009.

Digital Library

[86]

[86] L. Chen et al. How serendipity improves user satisfaction with recommendations? a large-scale user evaluation. In Proc. of WWW’19, pp. 240–250, 2019.

[87]

[87] Y.-Y. Wang et al. Understanding the moderating roles of types of recommender systems and products on customer behavioral intention to use recommender systems. ISeB, 13(4):769–799, 2015.

Digital Library

[88]

[88] D. Shin. How do users interact with algorithm recommender systems? the interaction of users, algorithms, and performance. Computers in Human Behavior, 109:106344, 2020.

[89]

[89] J. Wang and X. Wang. Structural equation modeling: Applications using Mplus: Chapter 7.1. John Wiley & Sons, 2019.

[90]

[90] T. A. Brown. Confirmatory factor analysis for applied research: 3. Introduction to CFA. Guilford publications, 2015.

[91]

[91] M. C. Willemsen et al. Understanding the role of latent feature diversification on choice difficulty and satisfaction. UMUAI, 26(4):347–389, 2016.

[92]

[92] R. A. Peterson. A meta-analysis of cronbach’s coefficient alpha. Journal of consumer research, 21(2):381–391, 1994.

[93]

[93] R. B. Kline and D. A. Santor. Principles & practice of structural equation modelling. Canadian Psychology, 40(4):381, 1999.

[94]

[94] D. Hooper et al. Structural equation modelling: Guidelines for determining model fit. EJBRM, 6(1):53–60, 2008.

[95]

[95] J. Pearl. The causal foundations of structural equation modeling. Technical report, UCLA DEPT OF COMPUTER SCIENCE, 2012.

[96]

[96] L. Qiu and I. Benbasat. Evaluating anthropomorphic product recommendation agents: A social relationship perspective to designing information systems. JMIS, 25(4):145–182, 2009.

Digital Library

[97]

[97] M. Heerink et al. Enjoyment intention to use and actual use of a conversational robot by elderly people. In Proc. of HRI’08, pp. 113–120, 2008.

Cited By

Jin YChen LCai WZhao X(2024)CRS-Que: A User-centric Evaluation Framework for Conversational Recommender SystemsACM Transactions on Recommender Systems10.1145/36315342:1(1-34)Online publication date: 7-Mar-2024
https://dl.acm.org/doi/10.1145/3631534
Bauer CZangerle ESaid A(2024)Exploring the Landscape of Recommender Systems Evaluation: Practices and PerspectivesACM Transactions on Recommender Systems10.1145/36291702:1(1-31)Online publication date: 7-Mar-2024
https://dl.acm.org/doi/10.1145/3629170
Alves GJannach DFerrari De Souza RManzato M(2024)User Perception of Fairness-Calibrated RecommendationsProceedings of the 32nd ACM Conference on User Modeling, Adaptation and Personalization10.1145/3627043.3659558(78-88)Online publication date: 22-Jun-2024
https://dl.acm.org/doi/10.1145/3627043.3659558
Show More Cited By

Index Terms

Key Qualities of Conversational Recommender Systems: From Users’ Perspective
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. HCI design and evaluation methods
      1. User models
2. Information systems
  1. Information retrieval
    1. Retrieval tasks and goals
      1. Recommender systems

Index terms have been assigned to the content through auto-classification.

Recommendations

CRS-Que: A User-centric Evaluation Framework for Conversational Recommender Systems
An increasing number of recommendation systems try to enhance the overall user experience by incorporating conversational interaction. However, evaluating conversational recommender systems (CRSs) from the user’s perspective remains elusive. The GUI-based ...
Impacts of Personal Characteristics on User Trust in Conversational Recommender Systems
CHI '22: Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems

Conversational recommender systems (CRSs) imitate human advisors to assist users in finding items through conversations and have recently gained increasing attention in domains such as media and e-commerce. Like in human communication, building trust in ...
Unifying Recommender Systems and Conversational User Interfaces
CUI '22: Proceedings of the 4th Conference on Conversational User Interfaces

This paper considers unifying research on conversational user interfaces and recommender systems. Studies on conversational user interfaces (CUIs) typically examine how conversations can be facilitated (i.e., optimizing the means). Recommender systems ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

HAI '21: Proceedings of the 9th International Conference on Human-Agent Interaction

November 2021

447 pages

ISBN:9781450386203

DOI:10.1145/3472307

Editors:
Kohei Ogawa
Nagoya University, Japan
,
Tomoko Yonezawa
Kansai University, Japan
,
Gale M. Lucas
USC Institute for Creative Technologies, United States
,
Hirotaka Osawa
University of Tsukuba, Japan
,
Wafa Johal
University of New South Wales, Australia
,
Masahiro Shiomi,
Advanced Telecommunications Research Institute International, Japan

Copyright © 2021 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 09 November 2021

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Funding Sources

Research Grants Council (RGC) of Hong Kong
HKBU IRCMS

Conference

HAI '21

Sponsor:

SIGCHI

HAI '21: International Conference on Human-Agent Interaction

November 9 - 11, 2021

Virtual Event, Japan

Acceptance Rates

Overall Acceptance Rate 121 of 404 submissions, 30%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

9
Total Citations
View Citations
503
Total Downloads

Downloads (Last 12 months)196
Downloads (Last 6 weeks)6

Reflects downloads up to 13 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Jin YChen LCai WZhao X(2024)CRS-Que: A User-centric Evaluation Framework for Conversational Recommender SystemsACM Transactions on Recommender Systems10.1145/36315342:1(1-34)Online publication date: 7-Mar-2024
https://dl.acm.org/doi/10.1145/3631534
Bauer CZangerle ESaid A(2024)Exploring the Landscape of Recommender Systems Evaluation: Practices and PerspectivesACM Transactions on Recommender Systems10.1145/36291702:1(1-31)Online publication date: 7-Mar-2024
https://dl.acm.org/doi/10.1145/3629170
Alves GJannach DFerrari De Souza RManzato M(2024)User Perception of Fairness-Calibrated RecommendationsProceedings of the 32nd ACM Conference on User Modeling, Adaptation and Personalization10.1145/3627043.3659558(78-88)Online publication date: 22-Jun-2024
https://dl.acm.org/doi/10.1145/3627043.3659558
Ziegler JLoepp B(2024)Interactive Recommendation SystemsHandbook of Human Computer Interaction10.1007/978-3-319-27648-9_54-1(1-29)Online publication date: 11-Feb-2024
https://doi.org/10.1007/978-3-319-27648-9_54-1
Siro CAliannejadi MDe Rijke M(2023)Understanding and Predicting User Satisfaction with Conversational Recommender SystemsACM Transactions on Information Systems10.1145/362498942:2(1-37)Online publication date: 8-Nov-2023
https://dl.acm.org/doi/10.1145/3624989
Radensky MSéguin JLim JOlson KGeiger R(2023)“I Think You Might Like This”: Exploring Effects of Confidence Signal Patterns on Trust in and Reliance on Conversational Recommender SystemsProceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency10.1145/3593013.3594043(792-804)Online publication date: 12-Jun-2023
https://dl.acm.org/doi/10.1145/3593013.3594043
Cai WJin YZhao XChen L(2023)“Listen to Music, Listen to Yourself”: Design of a Conversational Agent to Support Self-Awareness While Listening to MusicProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3581427(1-19)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3581427
Rossouw ASmuts H(2023)Key Principles Pertinent to User Experience Design for Conversational User Interfaces: A Conceptual Learning ModelInnovative Technologies and Learning10.1007/978-3-031-40113-8_17(174-186)Online publication date: 28-Aug-2023
https://dl.acm.org/doi/10.1007/978-3-031-40113-8_17
Cai WJin YChen L(2022)Impacts of Personal Characteristics on User Trust in Conversational Recommender SystemsProceedings of the 2022 CHI Conference on Human Factors in Computing Systems10.1145/3491102.3517471(1-14)Online publication date: 29-Apr-2022
https://dl.acm.org/doi/10.1145/3491102.3517471
Jannach DChen L(2022)Conversational recommendationAI Magazine10.1002/aaai.1205943:2(151-163)Online publication date: 23-Jun-2022
https://dl.acm.org/doi/10.1002/aaai.12059

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Table of Contents