More Web Proxy on the site http://driver.im/

tutorial

REDD 2014 -- international workshop on recommender systems evaluation: dimensions and design

Authors:

Panagiotis Adamopoulos,

Alejandro Bellogín,

Pablo Castells,

Paolo Cremonesi,

Harald SteckAuthors Info & Claims

RecSys '14: Proceedings of the 8th ACM Conference on Recommender systems

Pages 393 - 394

https://doi.org/10.1145/2645710.2645780

Published: 06 October 2014 Publication History

Abstract

Evaluation is a cardinal issue in recommender systems; as in any technical discipline, it highlights to a large extent the problems that need to be solved by the field and, hence, leads the way for algorithmic research and development in the community. Yet, in the field of recommender systems, there still exists considerable disparity in evaluation methods, metrics and experimental designs, as well as a significant mismatch between evaluation methods in the lab and what constitutes an effective recommendation for real users and businesses. Even after the relevant quality dimensions have been defined, a clear evaluation protocol should be specified in detail and agreed upon, allowing for the comparison of results and experiments conducted by different authors. This would enable any contribution to the same problem to be incremental and add up on top of previous work, rather than grow sideways. The REDD 2014 workshop seeks to provide an informal forum to tackle such issues and to move towards better understood and shared evaluation methodologies, allowing one to leverage the efforts and the workforce of the academic community towards meaningful and relevant directions in real-world developments.

References

[1]

Adamopoulos, P. and Tuzhilin, A. On unexpectedness in recommender systems: Or how to better expect the unexpected. ACM TIST.

[2]

Adomavicius, G. and Kwon, Y. Improving aggregate recommendation diversity using ranking-based techniques. IEEE TKDE, 24(5):896 --911, 2012.

Digital Library

[3]

Breese, J. S., Heckerman, D., and Kadie, C. Empirical analysis of predictive algorithms for collaborative filtering. 14th Conf. on Uncertainty in Artificial Intelligence (UAI 1998), 43--52, 1998.

Digital Library

[4]

Celma, O. and Herrera, P. A New Approach to Evaluating Novel Recommendations. 2nd ACM International Conference on Recommender Systems (RecSys 2008), 179--186, 2008.

Digital Library

[5]

Cremonesi, P., Garzotto, F., Negro, S., Papadopoulos, A. V., and Turrin, R. Comparative evaluation of recommender system quality. ACM Conference on Human Factors in Computing Systems (CHI 2011), 1927--1932, 2011.

Digital Library

[6]

Cremonesi, P., Koren, Y., and Turrin, R. Performance of Recommender Algorithms on Top-N Recommendation Tasks. 4th ACM International Conference on Recommender Systems (RecSys 2010), 39--46, 2010.

Digital Library

[7]

Fleder, D. M. and Hosanagar, K. Blockbuster Culture's Next Rise or Fall: The Impact of Recommender Systems on Sales Diversity. Management Science, 55(5):697--712, 2009.

Digital Library

[8]

Ge, M., Delgado-Battenfeld, C. and Jannach, D. Beyond accuracy: Evaluating recommender systems by coverage and serendipity. 4th ACM International Conference on Recommender Systems (RecSys 2010), 257--260, 2010.

Digital Library

[9]

Herlocker, J. L., Konstan, J. A., Terveen, L. G., and Riedl, J. T. Evaluating collaborative filtering recommender systems. ACM Trans. on Information Systems, 22(1):5--53, 2004.

Digital Library

[10]

Lathia, N., Hailes, S., Capra, L., and Amatriain, X. Temporal Diversity in Recommender Systems. 33rd International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2010), 210--217, 2010.

Digital Library

[11]

McNee, S. M., Riedl, J., and Konstan, J. A. Being Accurate is Not Enough: How Accuracy Metrics have hurt Recommender Systems. ACM Conf. on Human Factors in Computing Systems (CHI 2006), 1097--1101, 2006.

Digital Library

[12]

Shani, G. and Gunawardana, A. Evaluating Recommendation Systems. In Ricci, F. et al (Eds.), Recommender Systems Handbook, pages 257--297. Springer, 2011.

[13]

Steck, H. Training and Testing of Recommender Systems on Data Missing Not At Random. 16th ACM International Conference on Knowledge Discovery and Data Mining (KDD 2010), 713--722, 2010.

Digital Library

[14]

Vargas, S. and Castells, P. Rank and Relevance in Novelty and Diversity Metrics for Recommender Systems. 5th ACM International Conference on Recommender Systems (RecSys 2011), 109--116, 2011.

Digital Library

[15]

Voorhees, E. M. and Harman, D. K. TREC: Experiment and Evaluation in Information Retrieval. The MIT Press, 2005.

Digital Library

Cited By

Kar PRoy MDatta SKar PRoy MDatta S(2024)Introduction to Recommendation SystemsRecommender Systems: Algorithms and their Applications10.1007/978-981-97-0538-2_1(1-10)Online publication date: 12-Jun-2024
https://doi.org/10.1007/978-981-97-0538-2_1
Frigó EKocsis LBogers TSaid ABrusilovsky PTikk D(2019)Online ranking combinationProceedings of the 13th ACM Conference on Recommender Systems10.1145/3298689.3346993(12-19)Online publication date: 10-Sep-2019
https://dl.acm.org/doi/10.1145/3298689.3346993
Said ABellogín AWerthner HZanker MGolbeck JSemeraro G(2015)Replicable Evaluation of Recommender SystemsProceedings of the 9th ACM Conference on Recommender Systems10.1145/2792838.2792841(363-364)Online publication date: 16-Sep-2015
https://dl.acm.org/doi/10.1145/2792838.2792841

Index Terms

REDD 2014 -- international workshop on recommender systems evaluation: dimensions and design
1. Information systems
  1. Information retrieval
    1. Retrieval tasks and goals
      1. Document filtering
      2. Information extraction

Recommendations

Workshop on recommendation utility evaluation: beyond RMSE -- RUE 2012
RecSys '12: Proceedings of the sixth ACM conference on Recommender systems

Measuring the error in rating prediction has been by far the dominant evaluation methodology in the Recommender Systems literature. Yet there seems to be a general consensus that this criterion alone is far from being enough to assess the practical ...
A Novel Framework for Improving Recommender Diversity
International Workshop on Behavior and Social Informatics on Behavior and Social Computing - Volume 8178

Recommender systems are being used to assist users in finding relevant items from a large set of alternatives in many online applications. However, while most research up to this point has focused on improving the accuracy of recommender systems, other ...
Temporal diversity in recommender systems
SIGIR '10: Proceedings of the 33rd international ACM SIGIR conference on Research and development in information retrieval

Collaborative Filtering (CF) algorithms, used to build web-based recommender systems, are often evaluated in terms of how accurately they predict user ratings. However, current evaluation techniques disregard the fact that users continue to rate items ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

RecSys '14: Proceedings of the 8th ACM Conference on Recommender systems

October 2014

458 pages

ISBN:9781450326681

DOI:10.1145/2645710

General Chairs:
Alfred Kobsa
University of California, Irvine
,
Michelle Zhou
IBM
,
Program Chairs:
Martin Ester
Simon Fraser University
,
Yehuda Koren
Google

Copyright © 2014 Owner/Author.

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

In-Cooperation

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 06 October 2014

Check for updates

Author Tags

Qualifiers

Tutorial

Conference

RecSys'14

Sponsor:

SIGCHI

RecSys'14: Eighth ACM Conference on Recommender Systems

October 6 - 10, 2014

California, Foster City, Silicon Valley, USA

Acceptance Rates

RecSys '14 Paper Acceptance Rate 35 of 234 submissions, 15%;

Overall Acceptance Rate 254 of 1,295 submissions, 20%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
267
Total Downloads

Downloads (Last 12 months)3
Downloads (Last 6 weeks)0

Reflects downloads up to 12 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Kar PRoy MDatta SKar PRoy MDatta S(2024)Introduction to Recommendation SystemsRecommender Systems: Algorithms and their Applications10.1007/978-981-97-0538-2_1(1-10)Online publication date: 12-Jun-2024
https://doi.org/10.1007/978-981-97-0538-2_1
Frigó EKocsis LBogers TSaid ABrusilovsky PTikk D(2019)Online ranking combinationProceedings of the 13th ACM Conference on Recommender Systems10.1145/3298689.3346993(12-19)Online publication date: 10-Sep-2019
https://dl.acm.org/doi/10.1145/3298689.3346993
Said ABellogín AWerthner HZanker MGolbeck JSemeraro G(2015)Replicable Evaluation of Recommender SystemsProceedings of the 9th ACM Conference on Recommender Systems10.1145/2792838.2792841(363-364)Online publication date: 16-Sep-2015
https://dl.acm.org/doi/10.1145/2792838.2792841

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents