More Web Proxy on the site http://driver.im/

research-article

Impact of Interaction Strategies on User Relevance Feedback

Authors:

Omar Shahbaz Khan,

Björn Þór Jónsson,

Stevan Rudinac,

Marcel WorringAuthors Info & Claims

ICMR '21: Proceedings of the 2021 International Conference on Multimedia Retrieval

Pages 590 - 598

https://doi.org/10.1145/3460426.3463663

Published: 01 September 2021 Publication History

Abstract

User Relevance Feedback (URF) is a class of interactive learning methods that rely on the interaction between a human user and a system to analyze a media collection. To improve URF system evaluation and design better systems, it is important to understand the impact that different interaction strategies can have. Based on the literature and observations from real user sessions from the Lifelog Search Challenge and Video Browser Showdown, we analyze interaction strategies related to (a) labeling positive and negative examples, and (b) applying filters based on users' domain knowledge. Experiments show that there is no single optimal labeling strategy, as the best strategy depends on both the collection and the task. In particular, our results refute the common assumption that providing more training examples is always beneficial: strategies with a smaller number of prototypical examples lead to better results in some cases. We further observe that while expert filtering is unsurprisingly beneficial, aggressive filtering, especially by novice users, can hinder the completion of tasks. Finally, we observe that combining URF with filters leads to better results than using filters alone.

References

[1]

IJsbrand Jan Aalbersberg. 1992. Incremental Relevance Feedback. In Proceedings of the 15th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (Copenhagen, Denmark) (SIGIR '92). Association for Computing Machinery, New York, NY, USA, 11--22.

Digital Library

[2]

James Allan. 1996. Incremental Relevance Feedback for Information Filtering. In Proceedings of the 19th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (Zurich, Switzerland) (SIGIR '96). Association for Computing Machinery, New York, NY, USA, 270--278.

Digital Library

[3]

Fabian Berns, Luca Rossetto, Klaus Schoeffmann, Christian Beecks, and George Awad. 2019. V3C1 Dataset: An Evaluation of Content Characteristics. In Proceedings of the 2019 on International Conference on Multimedia Retrieval (Ottawa ON, Canada) (ICMR '19). Association for Computing Machinery, New York, NY, USA, 334--338.

Digital Library

[4]

Bogdan Boteanu, Ionuct Mironicua, and Bogdan Ionescu. 2017. Pseudo-relevance feedback diversification of social image retrieval results. Multimedia Tools and Applications, Vol. 76, 9 (2017), 11889--11916.

Digital Library

[5]

François Chollet. 2017. Xception: Deep Learning with Depthwise Separable Convolutions. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, Honolulu, HI, USA, 1800--1807.

[6]

Duc-Tien Dang-Nguyen, Luca Piras, Giorgio Giacinto, Giulia Boato, and Francesco GB DE Natale. 2017. Multimodal retrieval with diversification and relevance feedback for tourist attraction images. ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM), Vol. 13, 4 (2017), 1--24.

Digital Library

[7]

Lianli Gao, Jingkuan Song, Fuhao Zou, Dongxiang Zhang, and Jie Shao. 2015. Scalable Multimedia Retrieval by Deep Learning Hashing with Relative Similarity Learning. In Proceedings of the 23rd ACM International Conference on Multimedia (Brisbane, Australia) (MM'15). Association for Computing Machinery, New York, NY, USA, 903--906.

Digital Library

[8]

Ralph Gasser, Luca Rossetto, and Heiko Schuldt. 2019. Multimodal Multimedia Retrieval with Vitrivr. In Proceedings of the 2019 on International Conference on Multimedia Retrieval (Ottawa ON, Canada) (ICMR '19). Association for Computing Machinery, New York, NY, USA, 391--394.

Digital Library

[9]

Paula Gómez Duran, Eva Mohedano, Kevin McGuinness, Xavier Giró-i Nieto, and Noel E. O'Connor. 2018. Demonstration of an Open Source Framework for Qualitative Evaluation of CBIR Systems. In Proceedings of the 26th ACM International Conference on Multimedia (Seoul, Republic of Korea) (MM'18). Association for Computing Machinery, New York, NY, USA, 1256--1257.

[10]

Cathal Gurrin, Klaus Schoeffmann, Hideo Joho, Andreas Leibetseder, Liting Zhou, Aaron Duane, Dang Nguyen, Duc Tien, Michael Riegler, Luca Piras, et al. 2019. Comparing approaches to interactive lifelog search at the lifelog search challenge (LSC2018). ITE Transactions on Media Technology and Applications, Vol. 7, 2 (2019), 46--59.

[11]

Xiaofei He, Oliver King, Wei-Ying Ma, Mingjing Li, and Hong-Jiang Zhang. 2003. Learning a semantic space from user's relevance feedback for image retrieval. IEEE transactions on Circuits and Systems for Video technology, Vol. 13, 1 (2003), 39--48.

[12]

Thomas S Huang, Charlie K Dagli, Shyamsundar Rajaram, Edward Y Chang, Michael I Mandel, Graham E Poliner, and Daniel PW Ellis. 2008. Active learning for interactive multimedia retrieval. Proc. IEEE, Vol. 96, 4 (2008), 648--667.

[13]

Herve Jegou, Matthijs Douze, and Cordelia Schmid. 2010. Product quantization for nearest neighbor search. IEEE transactions on pattern analysis and machine intelligence, Vol. 33, 1 (2010), 117--128.

[14]

Lu Jiang, Teruko Mitamura, Shoou-I Yu, and Alexander G. Hauptmann. 2014. Zero-Example Event Search Using MultiModal Pseudo Relevance Feedback. In Proceedings of International Conference on Multimedia Retrieval (Glasgow, United Kingdom) (ICMR '14). Association for Computing Machinery, New York, NY, USA, 297--304.

[15]

Björn Þór Jónsson, Omar Shahbaz Khan, Dennis C. Koelma, Stevan Rudinac, Marcel Worring, and Jan Zahálka. 2020. Exquisitor at the Video Browser Showdown 2020. In MultiMedia Modeling, Yong Man Ro, Wen-Huang Cheng, Junmo Kim, Wei-Ta Chu, Peng Cui, Jung-Woo Choi, Min-Chun Hu, and Wesley De Neve (Eds.). Springer International Publishing, Cham, 796--802.

[16]

Omar Shahbaz Khan, Björn Þór Jónsson, Stevan Rudinac, Jan Zahálka, Hanna Ragnarsdóttir, Þórhildur Þorleiksdóttir, Gylfi Þór Guðmundsson, Laurent Amsaleg, and Marcel Worring. 2020 a. Interactive Learning for Multimedia at Large. In European Conference on Information Retrieval. Springer, Cham, 495--510.

Digital Library

[17]

Omar Shahbaz Khan, Mathias Dybkjær Larsen, Liam Alex Sonto Poulsen, Björn Þór Jónsson, Jan Zahálka, Stevan Rudinac, Dennis Koelma, and Marcel Worring. 2020 b. Exquisitor at the Lifelog Search Challenge 2020. In Proceedings of the Third Annual Workshop on Lifelog Search Challenge (Dublin, Ireland) (LSC '20). Association for Computing Machinery, New York, NY, USA, 19--22.

Digital Library

[18]

Miroslav Kratochvil, Frantivsek Mejzl'ik, Patrik Veselý, Tomávs Souçek, and Jakub Lokoć. 2020. SOMHunter: Lightweight Video Search System with SOM-Guided Relevance Feedback .Association for Computing Machinery, New York, NY, USA, 4481--4484.

[19]

Miroslav Kratochvíl, Patrik Veselý, Frantivs ek Mejzlík, and Jakub Lokovc. 2020. SOM-Hunter: Video Browsing with Relevance-to-SOM Feedback Loop. In MultiMedia Modeling. Springer International Publishing, Cham, 790--795.

[20]

Brian Kulis and Kristen Grauman. 2009. Kernelized Locality-Sensitive Hashing for Scalable Image Search. In 2009 IEEE 12th International Conference on Computer Vision. IEEE Computer Society, Los Alamitos, CA, USA, 2130 -- 2137.

[21]

Frantivsek Mejzl'ik, Patrik Veselý, Miroslav Kratochv'il, Tomávs Souvcek, and Jakub Lokovc. 2020. SOMHunter for Lifelog Search. In Proceedings of the Third Annual Workshop on Lifelog Search Challenge (Dublin, Ireland) (LSC '20). Association for Computing Machinery, New York, NY, USA, 73--75.

Digital Library

[22]

Pascal Mettes, Dennis C. Koelma, and Cees G.M. Snoek. 2016. The ImageNet Shuffle: Reorganized Pre-Training for Video Event Detection. In Proceedings of the 2016 ACM on International Conference on Multimedia Retrieval (ICMR '16). Association for Computing Machinery, New York, NY, USA, 175--182.

[23]

G. L. J. Pingen, M. H. T. de Boer, and R. B. N. Aly. 2017. Rocchio-Based Relevance Feedback in Video Event Retrieval. In MultiMedia Modeling. Springer International Publishing, Cham, 318--330.

[24]

J. J. Rocchio. 1965. Relevance feedback in information retrieval. Technical Report. University of Harvard, Computer Laboratory.

[25]

Ork De Rooij and Marcel Worring. 2012. Efficient Targeted Search Using a Focus and Context Video Browser. ACM Trans. Multimedia Comput. Commun. Appl., Vol. 8, 4, Article 51 (Nov. 2012), 19 pages.

Digital Library

[26]

Yong Rui, Thomas S Huang, and Sharad Mehrotra. 1997. Content-based image retrieval with relevance feedback in MARS. In Proceedings of International Conference on Image Processing, Vol. 2. IEEE, IEEE, Santa Barbara, CA, USA, 815--818.

[27]

Sheikh Muhammad Sarwar, John Foley, and James Allan. 2018. Term Relevance Feedback for Contextual Named Entity Retrieval. In Proceedings of the 2018 Conference on Human Information Interaction & Retrieval (New Brunswick, NJ, USA) (CHIIR '18). Association for Computing Machinery, New York, NY, USA, 301--304.

Digital Library

[28]

Klaus Schoeffmann. 2014. A User-Centric Media Retrieval Competition: The Video Browser Showdown 2012--2014. IEEE MM, Vol. 21, 4 (2014), 8--13.

[29]

Roberto Tronci, Gabriele Murgia, Maurizio Pili, Luca Piras, and Giorgio Giacinto. 2013. ImageHunter: A Novel Tool for Relevance Feedback in Content Based Image Retrieval .Springer Berlin Heidelberg, Berlin, Heidelberg, 53--70.

[30]

VOX-Pol. 2021. About Us. https://www.voxpol.eu/about-us

[31]

Xuanhui Wang, Hui Fang, and ChengXiang Zhai. 2008. A Study of Methods for Negative Relevance Feedback. In Proceedings of the 31st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (Singapore, Singapore) (SIGIR '08). Association for Computing Machinery, New York, NY, USA, 219--226.

Digital Library

[32]

Yunchao Wei, Yao Zhao, Canyi Lu, Shikui Wei, Luoqi Liu, Zhenfeng Zhu, and Shuicheng Yan. 2016. Cross-modal retrieval with CNN visual features: A new baseline. IEEE transactions on cybernetics, Vol. 47, 2 (2016), 449--460.

[33]

Rong Yan, Alexander Hauptmann, and Rong Jin. 2003 a. Multimedia Search with Pseudo-relevance Feedback. In Image and Video Retrieval. Springer Berlin Heidelberg, Berlin, Heidelberg, 238--247.

[34]

Rong Yan, Alexander G. Hauptmann, and Rong Jin. 2003 b. Negative Pseudo-Relevance Feedback in Content-Based Video Retrieval. In Proceedings of the Eleventh ACM International Conference on Multimedia (Berkeley, CA, USA) (MULTIMEDIA '03). Association for Computing Machinery, New York, NY, USA, 343--346.

Digital Library

[35]

Yi Yang, Feiping Nie, Dong Xu, Jiebo Luo, Yueting Zhuang, and Yunhe Pan. 2011. A multimedia retrieval framework based on semi-supervised ranking and relevance feedback. IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 34, 4 (2011), 723--742.

Digital Library

[36]

Jan Zahálka, Stevan Rudinac, Björn Þór Jónsson, Dennis C Koelma, and Marcel Worring. 2018. Blackthorn: Large-Scale Interactive Multimodal Learning. IEEE Transactions on Multimedia, Vol. 20, 3 (2018), 687--698.

Digital Library

[37]

Jan Zahálka, Stevan Rudinac, Björn Þór Jónsson, Dennis C. Koelma, and Marcel Worring. 2016. Interactive Multimodal Learning on 100 Million Images. In Proceedings of the 2016 ACM on International Conference on Multimedia Retrieval (New York, New York, USA) (ICMR '16). Association for Computing Machinery, New York, NY, USA, 333--337.

Digital Library

[38]

Jan Zahálka, Stevan Rudinac, and Marcel Worring. 2015. Analytic Quality: Evaluation of Performance and Insight in Multimedia Collection Analysis. In Proceedings of the 23rd ACM International Conference on Multimedia (Brisbane, Australia) (MM '15). Association for Computing Machinery, New York, NY, USA, 231--240.

Digital Library

[39]

Jan Zahálka, Marcel Worring, and Jarke J. Van Wijk. 2021. II-20: Intelligent and pragmatic analytic categorization of image collections. IEEE Transactions on Visualization and Computer Graphics, Vol. 27, 2 (2021), 422--431.

[40]

Shifeng Zhang, Jianmin Li, Mengqing Jiang, Peijiang Yuan, and Bo Zhang. 2017. Scalable discrete supervised multimedia hash learning with clustering. IEEE Transactions on Circuits and Systems for Video Technology, Vol. 28, 10 (2017), 2716--2729.

Digital Library

Cited By

Pegia MGurrin CKongkachandra RSchoeffmann KDang-Nguyen DRossetto LSatoh SZhou L(2024)Multimodality in Media RetrievalProceedings of the 2024 International Conference on Multimedia Retrieval10.1145/3652583.3657583(1219-1223)Online publication date: 30-May-2024
https://dl.acm.org/doi/10.1145/3652583.3657583
Vadicamo LArnold RBailer WCarrara FGurrin CHezel NLi XLokoc JLubos SMa ZMessina NNguyen TPeska LRossetto LSauter LSchöffmann KSpiess FTran MVrochidis S(2024)Evaluating Performance and Trends in Interactive Video Retrieval: Insights From the 12th VBS CompetitionIEEE Access10.1109/ACCESS.2024.340563812(79342-79366)Online publication date: 2024
https://doi.org/10.1109/ACCESS.2024.3405638
Khan OJónsson B(2023)User Relevance Feedback and Novices: Anecdotes from Exquisitor's Participation in Interactive Retrieval CompetitionsProceedings of the 20th International Conference on Content-based Multimedia Indexing10.1145/3617233.3617275(173-177)Online publication date: 20-Sep-2023
https://dl.acm.org/doi/10.1145/3617233.3617275

Index Terms

Impact of Interaction Strategies on User Relevance Feedback
1. Information systems
  1. Information retrieval
    1. Evaluation of retrieval results
    2. Specialized information retrieval
      1. Multimedia and multimodal retrieval

Recommendations

Influence of Late Fusion of High-Level Features on User Relevance Feedback for Videos
IMuR '22: Proceedings of the 2nd International Workshop on Interactive Multimedia Retrieval

Content-based media retrieval relies on multimodal data representations. For videos, these representations mainly focus on the textual, visual, and audio modalities. While the modality representations can be used individually, combining their ...
Relevance feedback: a power tool for interactive content-based image retrieval

Content-based image retrieval (CBIR) has become one of the most active research areas in the past few years. Many visual feature representations have been explored and many systems built. While these research efforts establish the basis of CBIR, the ...
An exploration of diversified user strategies for image retrieval with relevance feedback

Given the difficulty of setting up large-scale experiments with real users, the comparison of content-based image retrieval methods using relevance feedback usually relies on the emulation of the user, following a single, well-prescribed strategy. Since ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

ICMR '21: Proceedings of the 2021 International Conference on Multimedia Retrieval

August 2021

715 pages

ISBN:9781450384636

DOI:10.1145/3460426

General Chairs:
Wen-Huang Cheng
National Yang Ming Chiao Tung University, Taiwan
,
Mohan Kankanhalli
National University of Singapore, Singapore
,
Meng Wang
Hefei University of Technology, China
,
Program Chairs:
Wei-Ta Chu
National Cheng Kung University, Taiwan
,
Jiaying Liu
Peking University, China
,
Marcel Worring
University of Amsterdam, Netherlands

Copyright © 2021 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGMM: ACM Special Interest Group on Multimedia

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 September 2021

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

European Regional Development Fund

Conference

ICMR '21

Sponsor:

SIGMM

ICMR '21: International Conference on Multimedia Retrieval

August 21 - 24, 2021

Taipei, Taiwan

Acceptance Rates

Overall Acceptance Rate 254 of 830 submissions, 31%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
80
Total Downloads

Downloads (Last 12 months)17
Downloads (Last 6 weeks)1

Reflects downloads up to 11 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Pegia MGurrin CKongkachandra RSchoeffmann KDang-Nguyen DRossetto LSatoh SZhou L(2024)Multimodality in Media RetrievalProceedings of the 2024 International Conference on Multimedia Retrieval10.1145/3652583.3657583(1219-1223)Online publication date: 30-May-2024
https://dl.acm.org/doi/10.1145/3652583.3657583
Vadicamo LArnold RBailer WCarrara FGurrin CHezel NLi XLokoc JLubos SMa ZMessina NNguyen TPeska LRossetto LSauter LSchöffmann KSpiess FTran MVrochidis S(2024)Evaluating Performance and Trends in Interactive Video Retrieval: Insights From the 12th VBS CompetitionIEEE Access10.1109/ACCESS.2024.340563812(79342-79366)Online publication date: 2024
https://doi.org/10.1109/ACCESS.2024.3405638
Khan OJónsson B(2023)User Relevance Feedback and Novices: Anecdotes from Exquisitor's Participation in Interactive Retrieval CompetitionsProceedings of the 20th International Conference on Content-based Multimedia Indexing10.1145/3617233.3617275(173-177)Online publication date: 20-Sep-2023
https://dl.acm.org/doi/10.1145/3617233.3617275

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents