More Web Proxy on the site http://driver.im/

research-article

Interactive Modeling of Concept Drift and Errors in Relevance Feedback

Authors:

Antti Kangasrääsiö,

Dorota Głowacka,

Samuel KaskiAuthors Info & Claims

UMAP '16: Proceedings of the 2016 Conference on User Modeling Adaptation and Personalization

Pages 185 - 193

https://doi.org/10.1145/2930238.2930243

Published: 13 July 2016 Publication History

Abstract

In exploratory search tasks, users usually start with considerable uncertainty about their search goals, and so the search intent of the user may be volatile as the user is constantly learning and reformulating her search hypothesis during the search. This may lead to a noticeable concept drift in the relevance feedback given by the user. We formulate a Bayesian regression model for predicting the accuracy of each individual user feedback and thus find outliers in the feedback data set. To accompany this model, we introduce a timeline interface that visualizes the feedback history to the user and gives her suggestions on which past feedback is likely in need of adjustment. This interface also allows the user to adjust the feedback accuracy inferences made by the model. Simulation experiments demonstrate that the performance of the new user model outperforms a simpler baseline and that the performance approaches that of an oracle, given a small amount of additional user interaction. A user study shows that the proposed modeling technique, combined with the timeline interface, made it easier for the users to notice and correct mistakes in their feedback, resulted in better and more diverse recommendations, allowed users to easier find items they liked, and was more understandable.

References

[1]

J. Ahn and P. Brusilovsky. Adaptive visualization of search results: Bringing user models to visual analytics. Information Visualization, 8(3):167--179, 2009.

Digital Library

[2]

J. Ahn, P. Brusilovsky, J. Grady, D. He, and S. Y. Syn. Open user profiles for adaptive news systems: Help or harm? In Proc. of the 16th International Conference on World Wide Web, WWW '07, pages 11--20. ACM, 2007.

Digital Library

[3]

H. Attias. Inferring parameters and structure of latent variable models by variational Bayes. In Proc. of the Fifteenth Conference on Uncertainty in Artificial Intelligence, UAI'99, pages 21--30. Morgan Kaufmann Publishers Inc., 1999.

Digital Library

[4]

F. Bakalov, M.-J. Meurs, B. König-Ries, B. Sateli, R. Witte, G. Butler, and A. Tsang. An approach to controlling user models and personalization effects in recommender systems. In Proc. of the 2013 International Conference on Intelligent User Interfaces, IUI '13, pages 49--56. ACM, 2013.

Digital Library

[5]

J. Brooke. SUS--A quick and dirty usability scale. Usability Evaluation in Industry, 189:194, 1996.

[6]

S. Bull. Supporting learning with open learner models. In Proc. of 4th Hellenic Conference on Information and Communication Technologies in Education, pages 47--61, 2004. Keynote.

[7]

M. Cakmak, C. Chao, and A. L. Thomaz. Designing interactions for robot active learners. Autonomous Mental Development, IEEE Transactions on, 2(2):108--118, 2010.

Digital Library

[8]

S. Chowdhury, F. Gibb, and M. Landoni. Uncertainty in information seeking and retrieval: A study in an academic environment. Information Processing & Management, 47(2):157--175, 2011.

Digital Library

[9]

V. Dimitrova, J. Self, and P. Brna. Applying interactive open learner models to learning technical terminology. In Proc. of User Modeling, pages 148--157. Springer, 2001.

Digital Library

[10]

J. A. Fails and D. R. Olsen Jr. Interactive machine learning. In Proc. of the 8th international conference on Intelligent user interfaces, pages 39--45. ACM, 2003.

Digital Library

[11]

J. Fogarty, D. Tan, A. Kapoor, and S. Winder. CueFlik: Interactive concept learning in image search. In Proc. of the SIGCHI Conference on Human Factors in Computing Systems, pages 29--38. ACM, 2008.

Digital Library

[12]

J. Gama, I.vZliobait\.e, A. Bifet, M. Pechenizkiy, and A. Bouchachia. A survey on concept drift adaptation. ACM Computing Surveys, 46(4):1--37, 2014.

Digital Library

[13]

D. Glowacka, T. Ruotsalo, K. Konuyshkova, K. Athukorala, S. Kaski, and G. Jacucci. Directing exploratory search: Reinforcement learning from user interactions with keywords. In Proc. of the 2013 International Conference on Intelligent User Interfaces, IUI '13, pages 117--128. ACM, 2013.

Digital Library

[14]

A. Kangasraasiö, Y. Chen, D. Glowacka, and S. Kaski. Dealing with concept drift in exploratory search: An interactive Bayesian approach. In Companion Publication of the 21st International Conference on Intelligent User Interfaces, IUI '16 Companion, pages 62--66. ACM, 2016.

Digital Library

[15]

A. Kangasraasiö, D. Glowacka, and S. Kaski. Improving controllability and predictability of interactive recommendation interfaces for exploratory search. In Proc. of the 20th International Conference on Intelligent User Interfaces, IUI '15, pages 247--251. ACM, 2015.

Digital Library

[16]

A. Kapoor, B. Lee, D. Tan, and E. Horvitz. Interactive optimization for steering machine classification. In Proc. of the SIGCHI Conference on Human Factors in Computing Systems, pages 1343--1352. ACM, 2010.

Digital Library

[17]

W. B. Knox and P. Stone. Reinforcement learning from human reward: Discounting in episodic tasks. In RO-MAN, 2012 IEEE, pages 878--885. IEEE, 2012.

[18]

T. Kulesza, M. Burnett, W.-K. Wong, and S. Stumpf. Principles of explanatory debugging to personalize interactive machine learning. In Proc. of the 20th International Conference on Intelligent User Interfaces, pages 126--137. ACM, 2015.

Digital Library

[19]

S. Lehmann, U. Schwanecke, and R. Dörner. Interactive visualization for opportunistic exploration of large document collections. Information Systems, 35(2):260--269, 2010.

Digital Library

[20]

D. J. C. MacKay. Bayesian nonlinear modeling for the prediction competition. ASHRAE transactions, 100(2):1053--1062, 1994.

[21]

G. Marchionini. Exploratory search: From finding to understanding. Communications of the ACM, 49(4):41--46, 2006.

Digital Library

[22]

P. Pu, L. Chen, and R. Hu. A user-centric evaluation framework for recommender systems. In Proc. of the fifth ACM conference on Recommender systems, pages 157--164. ACM, 2011.

Digital Library

[23]

J. Rennie and K. Lang. The 20 Newsgroups dataset, January 2008. http://qwone.com/ jason/20Newsgroups/.

[24]

S. Stumpf, V. Rajaram, L. Li, W.-K. Wong, M. Burnett, T. Dietterich, E. Sullivan, and J. Herlocker. Interacting meaningfully with machine learning systems: Three experiments. International Journal of Human-Computer Studies, 67(8):639--662, 2009.

Digital Library

[25]

J. Ting, A. D'Souza, and S. Schaal. Automatic outlier detection: A Bayesian approach. In Robotics and Automation, IEEE International Conference on, pages 2489--2494. IEEE, 2007.

[26]

M. Tipping and N. Lawrence. A variational approach to robust Bayesian interpolation. In Neural Networks for Signal Processing, IEEE 13th Workshop on, NNSP '03, pages 229--238. IEEE, 2003.

[27]

A. Tsymbal. The problem of concept drift: Definitions and related work. Technical Report TCD-CS-2004--15, Computer Science Department, Trinity College Dublin, 2004.

[28]

R. W. White, B. Kules, S. M. Drucker, and M. C. Schraefel. Supporting exploratory search. Communications of the ACM, 49(4):36--39, 2006.

Digital Library

Cited By

Yanez FConati COttley ANobre C(2024)The State of the Art in User‐Adaptive VisualizationsComputer Graphics Forum10.1111/cgf.15271Online publication date: 4-Dec-2024
https://doi.org/10.1111/cgf.15271
Herrmanny KTorkamaan H(2021)Towards a User Integration Framework for Personal Health Decision Support and Recommender SystemsProceedings of the 29th ACM Conference on User Modeling, Adaptation and Personalization10.1145/3450613.3456816(65-76)Online publication date: 21-Jun-2021
https://dl.acm.org/doi/10.1145/3450613.3456816
Pereira MEtemad EPaulovich FHung CCerny TShin DBechini A(2020)Iterative learning to rank from explicit relevance feedbackProceedings of the 35th Annual ACM Symposium on Applied Computing10.1145/3341105.3374002(698-705)Online publication date: 30-Mar-2020
https://dl.acm.org/doi/10.1145/3341105.3374002
Show More Cited By

Index Terms

Interactive Modeling of Concept Drift and Errors in Relevance Feedback
1. Human-centered computing
  1. Interaction design
    1. Interaction design process and methods
      1. Interface design prototyping
2. Information systems
  1. Information retrieval
  2. World Wide Web
    1. Web searching and information discovery
      1. Personalization

Recommendations

Dealing with Concept Drift in Exploratory Search: An Interactive Bayesian Approach
IUI '16 Companion: Companion Publication of the 21st International Conference on Intelligent User Interfaces

In exploratory search, when the user formulates a query iteratively through relevance feedback, it is likely that the feedback given earlier requires adjustment later on. The main reason for this is that the user learns while searching, which causes ...
Interactive Intent Modeling from Multiple Feedback Domains
IUI '16: Proceedings of the 21st International Conference on Intelligent User Interfaces

In exploratory search, the user starts with an uncertain information need and provides relevance feedback to the system's suggestions to direct the search. The search system learns the user intent based on this feedback and employs it to recommend novel ...
Improving Controllability and Predictability of Interactive Recommendation Interfaces for Exploratory Search
IUI '15: Proceedings of the 20th International Conference on Intelligent User Interfaces

In exploratory search, when a user directs a search engine using uncertain relevance feedback, usability problems regarding controllability and predictability may arise. One problem is that the user is often modelled as a passive source of relevance ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

UMAP '16: Proceedings of the 2016 Conference on User Modeling Adaptation and Personalization

July 2016

366 pages

ISBN:9781450343688

DOI:10.1145/2930238

General Chairs:
Julita Vassileva
University of Saskatchewan, Canada
,
James Blustein
Dalhousie University, Canada
,
Program Chairs:
Lora Aroyo
VU Amsterdam, the Netherlands
,
Sidney D'Mello
University of Notre Dame, USA

Copyright © 2016 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 13 July 2016

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Conference

UMAP '16

Sponsor:

UMAP '16: User Modeling, Adaptation and Personalization Conference

July 13 - 17, 2016

Nova Scotia, Halifax, Canada

Acceptance Rates

UMAP '16 Paper Acceptance Rate 21 of 123 submissions, 17%;

Overall Acceptance Rate 162 of 633 submissions, 26%

Upcoming Conference

UMAP '25

Sponsor:
sigchi
sigchi

33rd ACM Conference on User Modeling, Adaptation and Personalization

June 16 - 19, 2025

New York City , NY , USA

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

7
Total Citations
View Citations
189
Total Downloads

Downloads (Last 12 months)7
Downloads (Last 6 weeks)2

Reflects downloads up to 18 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Yanez FConati COttley ANobre C(2024)The State of the Art in User‐Adaptive VisualizationsComputer Graphics Forum10.1111/cgf.15271Online publication date: 4-Dec-2024
https://doi.org/10.1111/cgf.15271
Herrmanny KTorkamaan H(2021)Towards a User Integration Framework for Personal Health Decision Support and Recommender SystemsProceedings of the 29th ACM Conference on User Modeling, Adaptation and Personalization10.1145/3450613.3456816(65-76)Online publication date: 21-Jun-2021
https://dl.acm.org/doi/10.1145/3450613.3456816
Pereira MEtemad EPaulovich FHung CCerny TShin DBechini A(2020)Iterative learning to rank from explicit relevance feedbackProceedings of the 35th Annual ACM Symposium on Applied Computing10.1145/3341105.3374002(698-705)Online publication date: 30-Mar-2020
https://dl.acm.org/doi/10.1145/3341105.3374002
Muthukrishnan HSzafir D(2019)Using Machine Learning and Visualization for Qualitative Inductive Analyses of Big Data2019 IEEE Workshop on Machine Learning from User Interaction for Visualization and Analytics (MLUI)10.1109/MLUI52769.2019.10075566(1-7)Online publication date: 20-Oct-2019
https://doi.org/10.1109/MLUI52769.2019.10075566
Jacucci GBarral ODaee PWenzel MSerim BRuotsalo TPluchino PFreeman JGamberini LKaski SBlankertz B(2019)Integrating neurophysiologic relevance feedback in intent modeling for information retrievalJournal of the Association for Information Science and Technology10.1002/asi.2416170:9(917-930)Online publication date: 2-Aug-2019
https://dl.acm.org/doi/10.1002/asi.24161
Margaris DVassilakis C(2018)Enhancing Rating Prediction Quality Through Improving the Accuracy of Detection of Shifts in Rating PracticesTransactions on Large-Scale Data- and Knowledge-Centered Systems XXXVII10.1007/978-3-662-57932-9_5(151-191)Online publication date: 2-Aug-2018
https://doi.org/10.1007/978-3-662-57932-9_5
Glowacka DKamps JKanoulas Ede Rijke MFang HYilmaz E(2017)Bandit Algorithms in Interactive Information RetrievalProceedings of the ACM SIGIR International Conference on Theory of Information Retrieval10.1145/3121050.3121108(327-328)Online publication date: 1-Oct-2017
https://dl.acm.org/doi/10.1145/3121050.3121108

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents