tutorial

Conducting user experiments in recommender systems

Author:

Bart P. KnijnenburgAuthors Info & Claims

RecSys '12: Proceedings of the sixth ACM conference on Recommender systems

Pages 3 - 4

https://doi.org/10.1145/2365952.2365956

Published: 09 September 2012 Publication History

Get Access

Abstract

There is an increasing consensus in the field of recommender systems that we should move beyond the offline evaluation of algorithms towards a more user-centric approach. This tutorial teaches the essential skills involved in conducting user experiments, the scientific approach to user-centric evaluation. Such experiments are essential in uncovering how and why the user experience of recommender systems comes about.

References

[1]

Adomavicius, G. and Tuzhilin, A. 2011. Context-Aware Recommender Systems. Recommender Systems Handbook. F. Ricci, L. Rokach, B. Shapira, and P.B. Kantor, eds. Springer US. 217--253.

Digital Library

Google Scholar

[2]

Bollen, D., Knijnenburg, B.P., Willemsen, M.C. and Graus, M. 2010. Understanding choice overload in recommender systems. Proceedings of the fourth ACM conference on Recommender systems (Barcelona, Spain, 2010), 63--70.

Digital Library

Google Scholar

[3]

Knijnenburg, B.P., Reijmer, N.J.M. and Willemsen, M.C. 2011. Each to his own: how different users call for different interaction methods in recommender systems. Proceedings of the fifth ACM conference on Recommender systems (Chicago, IL, 2011), 141--148.

Digital Library

Google Scholar

[4]

Knijnenburg, B.P., Schmidt-Thieme, L. and Bollen, D.G.F.M. 2010. Workshop on user-centric evaluation of recommender systems and their interfaces. Proceedings of the fourth ACM conference on Recommender systems (New York, NY, USA, 2010), 383--384.

Digital Library

Google Scholar

[5]

Knijnenburg, B.P. and Willemsen, M.C. 2009. Understanding the effect of adaptive preference elicitation methods on user satisfaction of a recommender system. Proceedings of the third ACM conference on Recommender systems (New York, NY, 2009), 381--384.

Digital Library

Google Scholar

[6]

Knijnenburg, B.P., Willemsen, M.C., Gantner, Z., Soncu, H. and Newell, C. 2012. Explaining the user experience of recommender systems. User Modeling and User-Adapted Interaction. 22, 4--5 (2012), 441--504.

Digital Library

Google Scholar

[7]

Knijnenburg, B.P., Willemsen, M.C. and Kobsa, A. 2011. A pragmatic procedure to support the user-centric evaluation of recommender systems. Proceedings of the fifth ACM conference on Recommender systems (New York, NY, USA, 2011), 321--324.

Digital Library

Google Scholar

[8]

Konstan, J. and Riedl, J. 2012. Recommender systems: from algorithms to user experience. User Modeling and User-Adapted Interaction. 22, 1 (2012), 101--123.

Digital Library

Google Scholar

[9]

McNee, S.M., Riedl, J. and Konstan, J.A. 2006. Being accurate is not enough. CHI '06 extended abstracts on Human factors in computing systems (Montreal, Quebec, Canada, 2006), 1097--1101.

Digital Library

Google Scholar

[10]

Pu, P., Chen, L. and Hu, R. 2012. Evaluating recommender systems from the user's perspective: survey of the state of the art. User Modeling and User-Adapted Interaction. 22, 4 (2012), 317--355.

Digital Library

Google Scholar

[11]

Willemsen, M., Bollen, D. and Ekstrand, M. 2011. UCERSTI 2: second workshop on user-centric evaluation of recom-mender systems and their interfaces. Proceedings of the fifth ACM conference on Recommender systems (New York, NY, USA, 2011), 395--396.

Digital Library

Google Scholar

Cited By

View all

Alzughbi HCaine KIqbal MKnijnenburg BLee HMcGregor SSidnam-Mauch E(2024)EPIC: Enhanced Privacy and Integrity Considerations for Research (Tutorial)Companion Proceedings of the 29th International Conference on Intelligent User Interfaces10.1145/3640544.3645249(166-168)Online publication date: 18-Mar-2024
https://dl.acm.org/doi/10.1145/3640544.3645249
Knijnenburg BMalthouse E(2024)Conducting User Experiments in Recommender SystemsProceedings of the 18th ACM Conference on Recommender Systems10.1145/3640457.3687090(1272-1273)Online publication date: 8-Oct-2024
https://dl.acm.org/doi/10.1145/3640457.3687090
Eslami SHassanein K(2022)Understanding Data Analytics Recommendation Execution: The Role of Recommendation QualityJournal of Computer Information Systems10.1080/08874417.2021.201015062:6(1283-1296)Online publication date: 20-Jan-2022
https://doi.org/10.1080/08874417.2021.2010150
Show More Cited By

Index Terms

Conducting user experiments in recommender systems

Recommendations

A pragmatic procedure to support the user-centric evaluation of recommender systems
RecSys '11: Proceedings of the fifth ACM conference on Recommender systems

As recommender systems are increasingly deployed in the real world, they are not merely tested offline for precision and coverage, but also "online" with test users to ensure good user experience. The user evaluation of recommenders is however complex ...
Evaluating Intelligent User Interfaces with User Experiments
IUI '16 Companion: Companion Publication of the 21st International Conference on Intelligent User Interfaces

User experiments are an essential tool to evaluate the user experience of intelligent user interfaces. This tutorial teaches the practical aspects of designing and setting up user experiments, as well as state-of-the-art methods to statistically ...
CRS-Que: A User-centric Evaluation Framework for Conversational Recommender Systems
An increasing number of recommendation systems try to enhance the overall user experience by incorporating conversational interaction. However, evaluating conversational recommender systems (CRSs) from the user’s perspective remains elusive. The GUI-based ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

RecSys '12: Proceedings of the sixth ACM conference on Recommender systems

September 2012

376 pages

ISBN:9781450312707

DOI:10.1145/2365952

General Chairs:
Pádraig Cunningham
University College Dublin, Ireland
,
Neil Hurley
University College Dublin, Ireland
,
Program Chairs:
Ido Guy
IBM Haifa Research Laboratory, Israel
,
Sarabjot Singh Anand
University of Warwick, England

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

In-Cooperation

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 09 September 2012

Check for updates

Author Tags

Qualifiers

Tutorial

Conference

RecSys '12

Sponsor:

SIGCHI

RecSys '12: Sixth ACM Conference on Recommender Systems

September 9 - 13, 2012

Dublin, Ireland

Acceptance Rates

RecSys '12 Paper Acceptance Rate 24 of 119 submissions, 20%;

Overall Acceptance Rate 254 of 1,295 submissions, 20%

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

13
Total Citations
View Citations
717
Total Downloads

Downloads (Last 12 months)13
Downloads (Last 6 weeks)2

Reflects downloads up to 20 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

View all

Alzughbi HCaine KIqbal MKnijnenburg BLee HMcGregor SSidnam-Mauch E(2024)EPIC: Enhanced Privacy and Integrity Considerations for Research (Tutorial)Companion Proceedings of the 29th International Conference on Intelligent User Interfaces10.1145/3640544.3645249(166-168)Online publication date: 18-Mar-2024
https://dl.acm.org/doi/10.1145/3640544.3645249
Knijnenburg BMalthouse E(2024)Conducting User Experiments in Recommender SystemsProceedings of the 18th ACM Conference on Recommender Systems10.1145/3640457.3687090(1272-1273)Online publication date: 8-Oct-2024
https://dl.acm.org/doi/10.1145/3640457.3687090
Eslami SHassanein K(2022)Understanding Data Analytics Recommendation Execution: The Role of Recommendation QualityJournal of Computer Information Systems10.1080/08874417.2021.201015062:6(1283-1296)Online publication date: 20-Jan-2022
https://doi.org/10.1080/08874417.2021.2010150
Losada DElsweiler DHarvey MTrattner C(2022)A day at the racesApplied Intelligence10.1007/s10489-021-02719-252:5(5617-5632)Online publication date: 1-Mar-2022
https://dl.acm.org/doi/10.1007/s10489-021-02719-2
Esteller-Cucala MFernandez VVilluendas D(2020)Evaluating Personalization: The AB Testing Pitfalls Companies Might Not Be Aware of—A Spotlight on the Automotive Sector WebsitesFrontiers in Artificial Intelligence10.3389/frai.2020.000203Online publication date: 9-Apr-2020
https://doi.org/10.3389/frai.2020.00020
Schaer PSchaible JGarcia Castro L(2020)Overview of LiLAS 2020 – Living Labs for Academic SearchExperimental IR Meets Multilinguality, Multimodality, and Interaction10.1007/978-3-030-58219-7_24(364-371)Online publication date: 22-Sep-2020
https://dl.acm.org/doi/10.1007/978-3-030-58219-7_24
Esteller-Cucala MFernandez VVilluendas DPapadopoulos GSamaras GWeibelzahl SJannach DSantos O(2019)Experimentation Pitfalls to Avoid in A/B Testing for Online PersonalizationAdjunct Publication of the 27th Conference on User Modeling, Adaptation and Personalization10.1145/3314183.3323853(153-159)Online publication date: 6-Jun-2019
https://dl.acm.org/doi/10.1145/3314183.3323853
Garcia-Gathright JHosey CThomas BCarterette BDiaz FPera SEkstrand MAmatriain XO'Donovan J(2018)Mixed methods for evaluating user satisfactionProceedings of the 12th ACM Conference on Recommender Systems10.1145/3240323.3241622(541-542)Online publication date: 27-Sep-2018
https://dl.acm.org/doi/10.1145/3240323.3241622
Kavu TDube KRaeth PHapanyengwi G(2017)A Characterisation and Framework for User-Centric Factors in Evaluation Methods for Recommender SystemsInternational Journal of ICT Research in Africa and the Middle East10.4018/IJICTRAME.20170101016:1(1-16)Online publication date: Jan-2017
https://doi.org/10.4018/IJICTRAME.2017010101
Panniello UGorgoglione MTuzhilin A(2016)Research Note—In CARSs We Trust: How Context-Aware Recommendations Affect Customers’ Trust and Other Business Performance Measures of Recommender SystemsInformation Systems Research10.1287/isre.2015.061027:1(182-196)Online publication date: Mar-2016
https://doi.org/10.1287/isre.2015.0610
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Index Terms

Recommendations

A pragmatic procedure to support the user-centric evaluation of recommender systems

Evaluating Intelligent User Interfaces with User Experiments

CRS-Que: A User-centric Evaluation Framework for Conversational Recommender Systems