More Web Proxy on the site http://driver.im/

Article

Mutual relevance feedback for multimodal query formulation in video retrieval

Authors:

Haim PermuterAuthors Info & Claims

MIR '05: Proceedings of the 7th ACM SIGMM international workshop on Multimedia information retrieval

Pages 17 - 24

https://doi.org/10.1145/1101826.1101832

Published: 10 November 2005 Publication History

Abstract

Video indexing and retrieval systems allow users to find relevant video segments for a given information need. A multimodal video index may include speech indices, a text-from-screen (OCR) index, semantic visual concepts, content-based image features, audio features and more. Formulating an efficient multimodal query for a given information need is much less intuitive and more challenging for the user than of composing a text query in document search. This paper describes a video retrieval system that uses mutual relevance feedback for multimodal query formulation. Through an iterative search and browse session, the user provides relevance feedback on system's output and the system provides the user a mutual feedback which leads to better query and better retrieval results. Official evaluation at the NIST TRECVID 2004 Search Task is provided for both Manual and Interactive search. It is shown that in the Manual task the queries result from the mutual feedback on the training data significantly improve the retrieval performances. A further improvement over the manual search is achieved in the interactive task by using both browsing and mutual feedback on the test set.

References

[1]

A. Amir, S. Srinivasan and A. Efrat, "Search The Speech, Browse The Video - A Generic Paradigm for Video Collections", in EURASIP J on Applied Signal Processing, Special Issue on Unstructured Information Management from Multimedia Data Sources, EURASIP JASP 2003:2 (2003) pp. 209--222.

Digital Library

[2]

A. Amir, J. Argillander, M. Berg, S-F. Chang, M. Franz, W. Hsu, G. Iyengar, J. R Kender, L. Kennedy, C-Y. Lin, M. Naphade, A. Natsev, J. R. Smith, J. Tesic, G. Wu, R. Yan, D. Zhang, "IBM Research TRECVID-2004 Video Retrieval System", Proc. Of TRECVID 2004, NIST, Gaithersburg, MD, Nov. 2004.

[3]

M. J. Bates, "Where should the person stop and the information search interface start?" Information Processing and Management, 26(5):575--591, 1990.

Digital Library

[4]

T-S. Chua, S-Y. Neo, K-Y. Li, G. Wang, R. Shi, M. Zhao and H. Xu, "TRECVID 2004 Search and Feature Extraction Task by NUS PRIS", Proc. Of TRECVID 2004, NIST, Gaithersburg, MD, Nov. 2004.

[5]

W.B. Croft, "Experiments with Representation in a Document Retrieval System," Information Technology: Research and Development, Vol. 2, pp. 1--21, 1982.

[6]

M. Ferecatu, M. Crucianu and N. Boujemaa, "Relevance feedback for image retrieval: a short survey," in State of the Art in Audiovisual Content-Based Retrieval, Information Universal Access and Interaction, Including Datamodels and Languages, Report of the DELOS2 European Network of Excellence (FP6), 2004.

[7]

J.L. Gauvain, L. Lamel, and G. Adda, "The LIMSI Broadcast News Transcription System," Speech Communication, 37(1-2):89--108, 2002.

Digital Library

[8]

M. Haas, J. Rijsdam, B. Thomee and M. S. Lew, "Relevance feedback: perceptual learning and retrieval in bio-computing, photos, and video," Proceedings of the 6th ACM SIGMM Int. workshop on Multimedia Information Retrieval, October 15-16, 2004, New York, NY, USA.

Digital Library

[9]

D. Harman, "Relevance Feedback and other query modification techniques," In W. B. Frakes and R. Baeza-Yates, editors, Information Retrieval: Data structures & algorithms. Prentice Hall, 1992.

Digital Library

[10]

A. Hauptmann, M.-Y. Chen, M. Christel, C. Huang, W.-H. Lin, T. Ng, N. Papernick, A. Velivelli, J. Yang, R. Yan, H. Yang, and H.D. Wactlar, "Confounded Expectations: Informedia at TRECVID 2004," Proc. Of TRECVID 2004, NIST, Gaithersburg, MD, Nov. 2004.

[11]

M. A. Hearst, "User Interfaces and visualization," in R. Baeza-Yates and B. Ribeiro-Neto, Modern Information Retrieval, Addison Wesley, New York, May 1999.

[12]

Y. Ishikawa, R. Subramanya, and C. Faloutsos, "Mindreader: Querying databases through multiple examples," In Proc. of 24th Int. Conf. on Very Large Data Bases, pp. 218--227, 1998.

Digital Library

[13]

T. Joachims, Learning to Classify Text using Support Vector Machines, Kluwer, 2002.

Digital Library

[14]

Gerald Kowalski, Mark T. Maybury, Information Retrieval Systems- Theory and Implementation, Kluwer AP, 2000.

Digital Library

[15]

J. Koeneman and N.J. Belkin, "A case for interaction: A study of interactive information retrieval behavior and effectiveness," Proc. Of ACM SIGCHI Conference on Human Factors in Computing Systems 1996, pp.205--212.

Digital Library

[16]

W. Kraaij, A. F. Smeaton and P. Over, "TRECVID 2004 - An Introduction," Proc. Of TRECVID 2004, NIST, Gaithersburg, MD, Nov. 2004. http://www-nlpir.nist.gov/projects/trecvid/

[17]

C.-Y. Lin, B. L. Tseng and J. R. Smith, "Video Collaborative Annotation Forum: Establishing Ground-Truth Labels on Large Multimedia Datasets," Proc. of TRECVID-2003, NIST, Gaithersburg, MD, Nov. 2003.

[18]

H. Permuter, A. Amir and S. Srinivasan, A Multimodal Soft Boolean Search in Speech and Video Databases, submitted.

[19]

S. E. Robertson and K. Sparck-Jones, "Relevance weighting of search terms," Journal of the American Society for Information Science, 1976, 27 (3), 129--146.

[20]

S. E. Robertson, "On term selection for query expansion," Journal of Documentation, 46 (4), 1990, pp. 359--364.

Digital Library

[21]

J. J. Rocchio, "Relevance Feedback in Information Retrieval", in G. Salton, editor. The SMART retrieval system: experiments in automatic document processing., Prentice Hall, Inc., 1971, pp. 313--323.

[22]

I. Ruthven and M. Lalmas, "A survey on the use of relevance feedback in information access systems", Knowledge Engineering Review. Vol 18. Issue 2, 2003, pp. 95--145.

Digital Library

[23]

G. Salton and C. Buckley, "Improving retrieval performance by Relevance Feedback," J. of the American Society for Information Science, 1990.

[24]

L. Taycher, M. La Cascia and S. Sclaroff, "Image digestion and relevance feedback in the ImageRover WWW search engine", Int. Conf. on Visual Information, pp 85--92, 1997.

[25]

R. W. White, I. Ruthven, J.M. Jose and C. J. Van Rijsbergen, "Evaluating Implicit Feedback Models Using Searcher Simulations", ACM T. Information Systems, to appear.

Digital Library

[26]

Zhao Xu, Xiaowei Xu, KaiYu, and Volker Tresp, "A Hybrid Relevance-Feedback Approach to Text Retrieval," in F. Sebastiani (Ed.): ECIR-03, LNCS 2633, pp. 281--293, 2003.

Digital Library

[27]

R. Yan, A. Hauptmann, and R. Jin, "Negative Pseudo-Relevance Feedback in Content-based Video Retrieval", ACM-MM 03, Berkeley, CA, USA, 2003.

Digital Library

[28]

X. S. Zhou and T. S. Huang, "Relevance feedback for image retrieval: a comprehensive review," Multimedia Systems, 8(6):536--544, 2003.

Cited By

Rossetto LTăźNase CSchuldt H(2016)Dealing with Ambiguous Queries in Multimodal Video RetrievalProceedings, Part I, of the 22nd International Conference on MultiMedia Modeling - Volume 951610.1007/978-3-319-27671-7_75(898-909)Online publication date: 4-Jan-2016
https://dl.acm.org/doi/10.1007/978-3-319-27671-7_75
Mohedano EMcGuinness KHealy GO'Connor NSmeaton ASalvador APorta SGiró-i-Nieto XHauptmann ANgo CXue XJiang YSnoek CVasconcelos N(2015)Exploring EEG for Object Detection and RetrievalProceedings of the 5th ACM on International Conference on Multimedia Retrieval10.1145/2671188.2749368(591-594)Online publication date: 22-Jun-2015
https://dl.acm.org/doi/10.1145/2671188.2749368
Vallet DCantador IJose J(2012)Exploiting semantics on external resources to gather visual examples for video retrievalInternational Journal of Multimedia Information Retrieval10.1007/s13735-012-0017-12:2(117-130)Online publication date: 2-Sep-2012
https://doi.org/10.1007/s13735-012-0017-1
Show More Cited By

Index Terms

Mutual relevance feedback for multimodal query formulation in video retrieval
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
2. Information systems
  1. Information retrieval

Recommendations

Optimizing multimedia retrieval using multimodal fusion and relevance feedback techniques
MMM'12: Proceedings of the 18th international conference on Advances in Multimedia Modeling

This paper introduces a novel approach for search and retrieval of multimedia content. The proposed framework retrieves multiple media types simultaneously, namely 3D objects, 2D images and audio files, by utilizing an appropriately modified manifold ...
Contextual relevance feedback in web information retrieval
IIiX: Proceedings of the 1st international conference on Information interaction in context

In this paper, we present an alternative approach to the problem of contextual relevance feedback in web-based information retrieval. Our approach utilises a rich contextual model that exploits a user's implicit and explicit data. Each user's implicit ...
Query refinement suggestion in multimodal image retrieval with relevance feedback
ICMI '11: Proceedings of the 13th international conference on multimodal interfaces

In the literature, it has been shown that relevance feedback is a good strategy for the system to interact with the user and provide better results in a content-based image retrieval (CBIR) system. On the other hand, there are many retrieval systems ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

MIR '05: Proceedings of the 7th ACM SIGMM international workshop on Multimedia information retrieval

November 2005

274 pages

ISBN:1595932445

DOI:10.1145/1101826

General Chairs:
Hongjiang Zhang
Microsoft Research Advanced Technology Center, China
,
John Smith
IBM T. J. Watson Research Center, Hawthorne, NY, USA
,
Qi Tian
University of Texas at San Antonio, San Antonio, TX, USA

Copyright © 2005 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 10 November 2005

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Conference

MM&Sec '05

Sponsor:

MM&Sec '05: Multimedia and Security Workshop 2005

November 10 - 11, 2005

Hilton, Singapore

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

8
Total Citations
View Citations
618
Total Downloads

Downloads (Last 12 months)2
Downloads (Last 6 weeks)0

Reflects downloads up to 25 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Rossetto LTăźNase CSchuldt H(2016)Dealing with Ambiguous Queries in Multimodal Video RetrievalProceedings, Part I, of the 22nd International Conference on MultiMedia Modeling - Volume 951610.1007/978-3-319-27671-7_75(898-909)Online publication date: 4-Jan-2016
https://dl.acm.org/doi/10.1007/978-3-319-27671-7_75
Mohedano EMcGuinness KHealy GO'Connor NSmeaton ASalvador APorta SGiró-i-Nieto XHauptmann ANgo CXue XJiang YSnoek CVasconcelos N(2015)Exploring EEG for Object Detection and RetrievalProceedings of the 5th ACM on International Conference on Multimedia Retrieval10.1145/2671188.2749368(591-594)Online publication date: 22-Jun-2015
https://dl.acm.org/doi/10.1145/2671188.2749368
Vallet DCantador IJose J(2012)Exploiting semantics on external resources to gather visual examples for video retrievalInternational Journal of Multimedia Information Retrieval10.1007/s13735-012-0017-12:2(117-130)Online publication date: 2-Sep-2012
https://doi.org/10.1007/s13735-012-0017-1
de S. Fedel GMedeiros Cdos Santos J(2012)Sinimbu --- multimodal queries to support biodiversity studiesProceedings of the 12th international conference on Computational Science and Its Applications - Volume Part I10.1007/978-3-642-31125-3_47(620-634)Online publication date: 18-Jun-2012
https://dl.acm.org/doi/10.1007/978-3-642-31125-3_47
Vallet DCantador IJose JWang JBoujemaa NRamirez NNatsev A(2010)Exploiting external knowledge to improve video retrievalProceedings of the international conference on Multimedia information retrieval10.1145/1743384.1743406(101-110)Online publication date: 29-Mar-2010
https://dl.acm.org/doi/10.1145/1743384.1743406
Chen MChen SShyu M(2010)Content‐Based Retrieval of VideosSemantic Computing10.1002/9780470588222.ch3(33-48)Online publication date: 20-Jul-2010
https://doi.org/10.1002/9780470588222.ch3
Mamou JRamabhadran BSiohan OKraaij Wde Vries AClarke CFuhr NKando N(2007)Vocabulary independent spoken term detectionProceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval10.1145/1277741.1277847(615-622)Online publication date: 23-Jul-2007
https://dl.acm.org/doi/10.1145/1277741.1277847
Okabe MYamada S(2007)Semisupervised Query Expansion with Minimal FeedbackIEEE Transactions on Knowledge and Data Engineering10.1109/TKDE.2007.19064619:11(1585-1589)Online publication date: 1-Nov-2007
https://dl.acm.org/doi/10.1109/TKDE.2007.190646

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten