More Web Proxy on the site http://driver.im/

research-article

Open access

An Asynchronous Scheme for the Distributed Evaluation of Interactive Multimedia Retrieval

Authors:

Abraham Bernstein,

Luca RossettoAuthors Info & Claims

IMuR '22: Proceedings of the 2nd International Workshop on Interactive Multimedia Retrieval

Pages 33 - 39

https://doi.org/10.1145/3552467.3554797

Published: 10 October 2022 Publication History

Abstract

Evaluation campaigns for interactive multimedia retrieval, such as the Video Browser Shodown (VBS) or the Lifelog Search Challenge (LSC), so far imposed constraints on both simultaneity and locality of all participants, requiring them to solve the same tasks in the same place, at the same time and under the same conditions. These constraints are in contrast to other evaluation campaigns that do not focus on interactivity, where participants can process the tasks in any place at any time. The recent travel restrictions necessitated the relaxation of the locality constraint of interactive campaigns, enabling participants to take place from an arbitrary location. Born out of necessity, this relaxation turned out to be a boon since it greatly simplified the evaluation process and enabled organisation of ad-hoc evaluations outside of the large campaigns. However, it also introduced an additional complication in cases where participants were spread over several time zones. In this paper, we introduce an evaluation scheme for interactive retrieval evaluation that relaxes both the simultaneity and locality constraints, enabling participation from any place at any time within a predefined time frame. This scheme, as implemented in the Distributed Retrieval Evaluation Server (DRES), enables novel ways of conducting interactive retrieval evaluation and bridged the gap between interactive campaigns and non-interactive ones.

References

[1]

Cathal Gurrin, Björn Þór Jónsson, Klaus Schöffmann, Duc-Tien Dang-Nguyen, Jakub Lokoc, Minh-Triet Tran, Wolfgang Hürst, Luca Rossetto, and Graham Healy. 2021. Introduction to the Fourth Annual Lifelog Search Challenge, LSC'21. In ICMR '21: International Conference on Multimedia Retrieval, Taipei, Taiwan, August 21--24, 2021. ACM, New York, NY, USA, 690--691. https://doi.org/10.1145/ 3460426.3470945

Digital Library

[2]

Cathal Gurrin, Björn Þór Jónsson, Klaus Schöffmann, Duc-Tien Dang-Nguyen, Jakub Lokoč, Minh-Triet Tran, Wolfgang Hürst, Luca Rossetto, and Graham Healy2022. Introduction to the Fifth Annual Lifelog Search Challenge, LSC'22. In International Conference on Multimedia Retrieval (ICMR). Association for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/3512527.3531439

Digital Library

[3]

Cathal Gurrin, Tu-Khiem Le, Van-Tu Ninh, Duc-Tien Dang-Nguyen, Björn Þór Jónsson, Jakub Lokoc, Wolfgang Hürst, Minh-Triet Tran, and Klaus Schöffmann. 2020. Introduction to the Third Annual Lifelog Search Challenge (LSC'20). In Proceedings of the 2020 on International Conference on Multimedia Retrieval, ICMR 2020, Dublin, Ireland, June 8--11, 2020. ACM, New York, NY, USA, 584--585. https: //doi.org/10.1145/3372278.3388043

Digital Library

[4]

Silvan Heller, Ralph Gasser, Cristina Illi, Maurizio Pasquinelli, Loris Sauter, Florian Spiess, and Heiko Schuldt. 2021. Towards Explainable Interactive Multimodal Video Retrieval with Vitrivr. In MultiMedia Modeling - 27th International Conference, MMM 2021, Prague, Czech Republic, June 22--24, 2021, Proceedings, Part II (Lecture Notes in Computer Science, Vol. 12573). Springer, 435--440. https://doi.org/10.1007/978--3-030--67835--7_41

[5]

Silvan Heller, Viktor Gsteiger, Werner Bailer, Cathal Gurrin, Björn Þór Jónsson, Jakub Lokoc, Andreas Leibetseder, Frantisek Mejzlík, Ladislav Peska, Luca Rossetto, Konstantin Schall, Klaus Schoeffmann, Heiko Schuldt, Florian Spiess, Ly-Duyen Tran, Lucia Vadicamo, Patrik Veselý, Stefanos Vrochidis, and Jiaxin Wu. 2022. Interactive video retrieval evaluation at a distance: comparing sixteen interactive video search systems in a remote setting at the 10th Video Browser Showdown. International Journal on Multimedia Information Retrieval 11, 1 (2022), 1--18. https://doi.org/10.1007/s13735-021-00225--2

[6]

Nico Hezel, Konstantin Schall, Klaus Jung, and Kai Uwe Barthel. 2022. Efficient Search and Browsing of Large-Scale Video Collections with Vibro. In MultiMedia Modeling - 28th International Conference, MMM 2022, Phu Quoc, Vietnam, June 6--10, 2022, Proceedings, Part II (Lecture Notes in Computer Science, Vol. 13142). Springer, 487--492. https://doi.org/10.1007/978--3-030--98355-0_43

[7]

Bogdan Ionescu, Henning Müller, Renaud Péteri, Yashin Dicente Cid, Vitali Liauchuk, Vassili Kovalev, Dzmitri Klimuk, Aleh Tarasau, Asma Ben Abacha, Sadid A. Hasan, Vivek V. Datla, Joey Liu, Dina Demner-Fushman, Duc-Tien Dang-Nguyen, Luca Piras, Michael Riegler, Minh-Triet Tran, Mathias Lux, Cathal Gurrin, Obioma Pelka, Christoph M. Friedrich, Alba Garcia Seco de Herrera, Narciso García, Ergina Kavallieratou, Carlos Roberto del-Blanco, Carlos Cuevas, Nikos Vasilopoulos, Konstantinos Karampidis, Jon Chamberlain, Adrian F. Clark, and Antonio Campello. 2019. ImageCLEF 2019: Multimedia Retrieval in Medicine, Lifelogging, Security and Nature. In Experimental IR Meets Multilinguality, Multimodality, and Interaction - 10th International Conference of the CLEF Association, CLEF 2019, Lugano, Switzerland, September 9--12, 2019, Proceedings (Lecture Notes in Computer Science, Vol. 11696). Springer, 358--386. https://doi.org/10.1007/978--3-030--28577--7_28

[8]

Miroslav Kratochvíl, Patrik Veselý, Frantisek Mejzlík, and Jakub Lokoc. 2020. SOM-Hunter: Video Browsing with Relevance-to-SOM Feedback Loop. In MultiMedia Modeling - 26th International Conference, MMM 2020, Daejeon, South Korea, January 5--8, 2020, Proceedings, Part II (Lecture Notes in Computer Science, Vol. 11962). Springer, 790--795. https://doi.org/10.1007/978--3-030--37734--2_71

[9]

Martha Larson, Mohammad Soleymani, Guillaume Gravier, Bogdan Ionescu, and Gareth JF Jones. 2017. The benchmarking initiative for multimedia evaluation: MediaEval 2016. IEEE MultiMedia 24, 1 (2017), 93--96. https://doi.org/10.1109/ MMUL.2017.9

[10]

Jakub Lokoc, Werner Bailer, Kai Uwe Barthel, Cathal Gurrin, Silvan Heller, Björn Þór Jónsson, Ladislav Peska, Luca Rossetto, Klaus Schoeffmann, Lucia Vadicamo, Stefanos Vrochidis, and Jiaxin Wu. 2022. A Task Category Space for User-Centric Comparative Multimedia Search Evaluations. In MultiMedia Modeling - 28th International Conference, MMM 2022, Phu Quoc, Vietnam, June 6--10, 2022, Proceedings, Part I (Lecture Notes in Computer Science, Vol. 13141). Springer, 193--204. https://doi.org/10.1007/978--3-030--98358--1_16

[11]

Jakub Lokoč, Werner Bailer, Klaus Schoeffmann, Bernd Münzer, and George Awad. 2018. On influential trends in interactive video retrieval: video browser showdown 2015--2017. IEEE Transactions on Multimedia 20, 12 (2018), 3361--3376. https://doi.org/10.1109/TMM.2018.2830110

Digital Library

[12]

Jakub Lokoč, Gregor Kovalcík, Bernd Münzer, Klaus Schöffmann, Werner Bailer, Ralph Gasser, Stefanos Vrochidis, Phuong Anh Nguyen, Sitapa Rujikietgumjorn, and Kai Uwe Barthel. 2019. Interactive search or sequential browsing? A detailed analysis of the video browser showdown 2018. ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM) 15, 1 (2019), 1--18. https: //doi.org/10.1145/3295663

Digital Library

[13]

Giorgio Maria Di Nunzio, Nicola Ferro, Thomas Mandl, and Carol Peters. 2007. CLEF 2007: Ad Hoc Track Overview. In Advances in Multilingual and Multimodal Information Retrieval, 8th Workshop of the Cross-Language Evaluation Forum, CLEF 2007, Budapest, Hungary, September 19--21, 2007, Revised Selected Papers (Lecture Notes in Computer Science, Vol. 5152). Springer, 13--32. https://doi.org/10.1007/978- 3--540--85760-0_2

[14]

Paul Over, George Awad, Alan F. Smeaton, Colum Foley, and James Lanagan. 2009. Creating a Web-Scale Video Collection for Research (WSMC '09). Association for Computing Machinery, New York, NY, USA, 25--32. https://doi.org/10.1145/ 1631135.1631141

[15]

Paul Over, Clement H. C. Leung, Horace Ho-Shing Ip, and Michael Grubinger. 2004. Multimedia Retrieval Benchmarks. IEEE Multimedia 11, 2 (2004), 80--84. https://doi.org/10.1109/MMUL.2004.1289045

Digital Library

[16]

Luca Rossetto, Ralph Gasser, Silvan Heller, Mahnaz Parian-Scherb, Loris Sauter, Florian Spiess, Heiko Schuldt, Ladislav Peska, Tomás Soucek, Miroslav Kratochvíl, Frantisek Mejzlík, Patrik Veselý, and Jakub Lokoc. 2021. On the User-Centric Comparative Remote Evaluation of Interactive Video Search Systems. IEEE Multimedia 28, 4 (2021), 18--28. https://doi.org/10.1109/MMUL.2021.3066779

Digital Library

[17]

Luca Rossetto, Ralph Gasser, Loris Sauter, Abraham Bernstein, and Heiko Schuldt. 2021. A System for Interactive Multimedia Retrieval Evaluations. In MultiMedia Modeling - 27th International Conference, MMM 2021, Prague, Czech Republic, June 22--24, 2021, Proceedings, Part II (Lecture Notes in Computer Science, Vol. 12573). Springer, Cham, 385--390. https://doi.org/10.1007/978--3-030--67835--7_33

[18]

Luca Rossetto, Heiko Schuldt, George Awad, and Asad A. Butt. 2019. V3C - A Research Video Collection. In MultiMedia Modeling - 25th International Conference, MMM 2019, Thessaloniki, Greece, January 8--11, 2019, Proceedings, Part I (Lecture Notes in Computer Science, Vol. 11295). Springer, 349--360. https://doi.org/10.1007/ 978--3-030-05710--7_29

[19]

Klaus Schoeffmann. 2014. A user-centric media retrieval competition: The video browser showdown 2012--2014. IEEE MultiMedia 21, 4 (2014), 8--13. https: //doi.org/10.1109/MMUL.2014.56

[20]

Klaus Schoeffmann. 2019. Video Browser Showdown 2012--2019: A Review. In 2019 International Conference on Content-Based Multimedia Indexing, CBMI 2019, Dublin, Ireland, September 4--6, 2019, Cathal Gurrin, Björn Þór Jónsson, Renaud Péteri, Stevan Rudinac, Stéphane Marchand-Maillet, Georges Quénot, Kevin McGuinness, Gylfi Þór Guðmundsson, Suzanne Little, Marie Katsurai, and Graham Healy (Eds.). IEEE, 1--4. https://doi.org/10.1109/CBMI.2019.8877397

[21]

Alan F. Smeaton, Paul Over, and Wessel Kraaij. 2006. Evaluation Campaigns and TRECVid. In Proceedings of the 8th ACM International Workshop on Multimedia Information Retrieval (Santa Barbara, California, USA) (MIR '06). Association for Computing Machinery, New York, NY, USA, 321--330. https://doi.org/10.1145/ 1178677.1178722

Digital Library

[22]

Cees GM Snoek, Marcel Worring, Ork de Rooij, Koen EA van de Sande, Rong Yan, and Alexander G Hauptmann. 2008. VideOlympics: real-time evaluation of multimedia retrieval systems. IEEE Multimedia 15, 1 (2008), 86--91. https: //doi.org/10.1109/MMUL.2008.2

Digital Library

Cited By

Sauter LGasser RSchuldt HBernstein ARossetto L(2024)Performance Evaluation in Multimedia RetrievalACM Transactions on Multimedia Computing, Communications, and Applications10.1145/367888121:1(1-23)Online publication date: 14-Oct-2024
https://dl.acm.org/doi/10.1145/3678881
Heller SSpiess FSchuldt H(2023)A tale of two interfaces: vitrivr at the lifelog search challengeMultimedia Tools and Applications10.1007/s11042-023-15082-w82:24(37829-37853)Online publication date: 6-Apr-2023
https://dl.acm.org/doi/10.1007/s11042-023-15082-w

Index Terms

An Asynchronous Scheme for the Distributed Evaluation of Interactive Multimedia Retrieval
1. Information systems
  1. Information retrieval

Recommendations

Asynchronous evaluation based genetic programming: comparison of asynchronous and synchronous evaluation and its analysis
EuroGP'13: Proceedings of the 16th European conference on Genetic Programming

This paper compares an asynchronous evaluation based GP with a synchronous evaluation based GP to investigate the evolution ability of an asynchronous evaluation on the GP domain. As an asynchronous evaluation based GP, this paper focuses on Tierra-...
A System for Interactive Multimedia Retrieval Evaluations
MultiMedia Modeling
Abstract
The evaluation of the performance of interactive multimedia retrieval systems is a methodologically non-trivial endeavour and requires specialized infrastructure. Current evaluation campaigns have so far relied on a local setting, where all ...
Multimedia presentation techniques for interactive plots
ICMCS '97: Proceedings of the 1997 International Conference on Multimedia Computing and Systems

Interactive plots support the creation of engaging stories or games that are meaningfully interactive. The authors describe a novel multimedia presentation method for interactive plots that dynamically links story content with appropriate presentation ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

IMuR '22: Proceedings of the 2nd International Workshop on Interactive Multimedia Retrieval

October 2022

54 pages

ISBN:9781450394970

DOI:10.1145/3552467

General Chairs:
Luca Rossetto
University of Zurich, Switzerland
,
Werner Bailer
JOANNEUM RESEARCH, Austria
,
Klaus Schoeffmann
Klagenfurt University, Austria
,
Jakub Lokoč
Charles University, Czech Republic

Copyright © 2022 Owner/Author.

This work is licensed under a Creative Commons Attribution International 4.0 License.

Sponsors

SIGMM: ACM Special Interest Group on Multimedia

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 10 October 2022

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Swiss National Sci- ence Foundation

Conference

MM '22

Sponsor:

SIGMM

MM '22: The 30th ACM International Conference on Multimedia

October 14, 2022

Lisboa, Portugal

Acceptance Rates

IMuR '22 Paper Acceptance Rate 6 of 6 submissions, 100%;

Overall Acceptance Rate 6 of 6 submissions, 100%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
147
Total Downloads

Downloads (Last 12 months)91
Downloads (Last 6 weeks)13

Reflects downloads up to 30 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Sauter LGasser RSchuldt HBernstein ARossetto L(2024)Performance Evaluation in Multimedia RetrievalACM Transactions on Multimedia Computing, Communications, and Applications10.1145/367888121:1(1-23)Online publication date: 14-Oct-2024
https://dl.acm.org/doi/10.1145/3678881
Heller SSpiess FSchuldt H(2023)A tale of two interfaces: vitrivr at the lifelog search challengeMultimedia Tools and Applications10.1007/s11042-023-15082-w82:24(37829-37853)Online publication date: 6-Apr-2023
https://dl.acm.org/doi/10.1007/s11042-023-15082-w

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents