More Web Proxy on the site http://driver.im/

research-article

Interactive Multimodal Lifelog Retrieval with vitrivr at LSC 2021

Authors:

Mahnaz Parian-Scherb,

Florian Spiess,

Heiko SchuldtAuthors Info & Claims

LSC '21: Proceedings of the 4th Annual on Lifelog Search Challenge

Pages 35 - 39

https://doi.org/10.1145/3463948.3469062

Published: 21 August 2021 Publication History

Abstract

The Lifelog Search Challenge (LSC) is an annual benchmarking competition for interactive multimedia retrieval systems, where participating systems compete in finding events based on textual descriptions containing hints about structured, semi-structured, and/or unstructured data. In this paper, we present the multimedia retrieval system vitrivr, a long-time participant to LSC, with a focus on new functionality. Specifically, we introduce the image stabilisation module which is added prior to the feature extraction to reduce the image degradation caused by lifelogger movements, and discuss how geodata is used during query formulation, query execution, and result presentation.

References

[1]

Ahmed Alateeq, Mark Roantree, and Cathal Gurrin. 2020. Voxento: A Prototype Voice-controlled Interactive Search Engine for Lifelogs. In Proceedings of the Third Annual Workshop on the Lifelog Search Challenge (LSC '20). ACM.

Digital Library

[2]

Lukas Beck and Heiko Schuldt. 2016. City-Stories: A Spatio-Temporal Mobile Multimedia Search System. In IEEE International Symposium on Multimedia (ISM).

[3]

Tai-Te Chu, Chia-Chun Chang, An-Zi Yen, Hen-Hsen Huang, and Hsin-Hsi Chen. 2020. Multimodal Retrieval through Relations between Subjects and Objects in Lifelog Images. In Proceedings of the Third Annual Workshop on the Lifelog Search Challenge (LSC '20). ACM.

Digital Library

[4]

Ralph Gasser, Luca Rossetto, Silvan Heller, and Heiko Schuldt. 2020. Cottontail DB: an open source database system for multimedia retrieval and analysis. In Proceedings of the 28th ACM International Conference on Multimedia.

Digital Library

[5]

Ralph Gasser, Luca Rossetto, Silvan Heller, and Heiko Schuldt. 2021. Multimedia Retrieval and Analysis with Cottontail DB. ACM SIGMM Records, Open Source Column (2021).

[6]

Cathal Gurrin, Tu-Khiem Le, Van-Tu Ninh, Duc-Tien Dang-Nguyen, Björn Þór Jónsson, Jakub Lokoč, Wolfgang Hurst, Minh-Triet Tran, and Klaus Schoeffmann. 2020. An Introduction to the Third Annual Lifelog Search Challenge, LSC'20. In International Conference on Multimedia Retrieval. ACM.

Digital Library

[7]

Cathal Gurrin, Klaus Schoeffmann, Hideo Joho, Andreas Leibetseder, Liting Zhou, Aaron Duane, Dang Nguyen, Duc Tien, Michael Riegler, Luca Piras, Minh-Triet Tran, Jakub Lokoč, and Wolfgang Hürst. 2019. Comparing Approaches to Interactive Lifelog Search at the Lifelog Search Challenge (LSC2018). ITE Transactions on Media Technology and Applications, Vol. 7, 2 (2019).

[8]

Cathal Gurrin, Björn Þór Jónsson, Klaus Schöffmann, Duc-Tien Dang-Nguyen, Jakub Lokoč, Minh-Triet Tran, Wolfgang Hürst, Luca Rossetto, and Graham Healy. 2021. Introduction to the Fourth Annual Lifelog Search Challenge, LSC'21. In Proc. International Conference on Multimedia Retrieval (ICMR'21). ACM, Taipei, Taiwan.

Digital Library

[9]

Silvan Heller, Mahnaz Amiri Parian, Ralph Gasser, Loris Sauter, and Heiko Schuldt. 2020 a. Interactive lifelog retrieval with vitrivr. In Proceedings of the Third Annual Workshop on Lifelog Search Challenge.

Digital Library

[10]

Silvan Heller, Ralph Gasser, Cristina Illi, Maurizio Pasquinelli, Loris Sauter, Florian Spiess, and Heiko Schuldt. 2021. Towards explainable interactive multi-modal video retrieval with vitrivr. In International Conference on Multimedia Modeling.

Digital Library

[11]

Silvan Heller, Mahnaz Parian, Maurizio Pasquinelli, and Heiko Schuldt. 2020 b. Vitrivr-Explore: Guided Multimedia Collection Exploration for Ad-hoc Video Search. In International Conference on Similarity Search and Applications.

[12]

Silvan Heller, Loris Sauter, Heiko Schuldt, and Luca Rossetto. 2020 c. Multi-Stage Queries and Temporal Scoring in Vitrivr. In 2020 IEEE International Conference on Multimedia & Expo Workshops (ICMEW).

[13]

Gregor Kovalík, Vít krhák, Tomá Souek, and Jakub Loko. 2020. VIRET Tool with Advanced Visual Browsing and Feedback. In Proceedings of the Third Annual Workshop on the Lifelog Search Challenge (LSC '20). ACM.

Digital Library

[14]

Tu-Khiem Le, Van-Tu Ninh, Minh-Triet Tran, Thanh-An Nguyen, Hai-Dang Nguyen, Liting Zhou, Graham Healy, and Cathal Gurrin. 2020. LifeSeeker 2.0 : Interactive Lifelog Search Engine at LSC 2020. In Proceedings of the Third Annual Workshop on the Lifelog Search Challenge (LSC '20). ACM.

Digital Library

[15]

Jiayu Li, Min Zhang, Weizhi Ma, Yiqun Liu, and Shaoping Ma. 2020. A Multi-level Interactive Lifelog Search Engine with User Feedback. In Proceedings of the Third Annual Workshop on the Lifelog Search Challenge (LSC '20). ACM.

Digital Library

[16]

Jakub Lokovc, Gregor Kovalvcik, Bernd Münzer, Klaus Schöffmann, Werner Bailer, Ralph Gasser, Stefanos Vrochidis, Phuong Anh Nguyen, Sitapa Rujikietgumjorn, and Kai Uwe Barthel. 2019. Interactive search or sequential browsing? A detailed analysis of the video browser showdown 2018. ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM), Vol. 15, 1 (2019).

[17]

Frantiek Mejzlík, Patrik Veselý, Miroslav Kratochvíl, Tomá Souek, and Jakub Loko. 2020. SOMHunter for Lifelog Search. In Proceedings of the Third Annual Workshop on the Lifelog Search Challenge (LSC '20). ACM.

Digital Library

[18]

Luca Rossetto, Ralph Gasser, Silvan Heller, Mahnaz Amiri Parian, and Heiko Schuldt. 2019 a. Retrieval of structured and unstructured data with vitrivr. In Proceedings of the Second Annual Workshop on Lifelog Search Challenge.

Digital Library

[19]

Luca Rossetto, Ralph Gasser, Silvan Heller, Mahnaz Parian-Scherb, Loris Sauter, Florian Spiess, Heiko Schuldt, Ladislav Peska, Tomas Soucek, Miroslav Kratochvil, et al. 2021 a. On the User-centric Comparative Remote Evaluation of Interactive Video Search Systems. IEEE MultiMedia (2021).

[20]

Luca Rossetto, Ralph Gasser, Jakub Lokoc, Werner Bailer, Klaus Schoeffmann, Bernd Muenzer, Tomas Soucek, Phuong Anh Nguyen, Paolo Bolettieri, Andreas Leibetseder, et al. 2020. Interactive video retrieval in the age of deep learning-detailed evaluation of vbs 2019. IEEE Transactions on Multimedia (2020).

[21]

Luca Rossetto, Ralph Gasser, Loris Sauter, Abraham Bernstein, and Heiko Schuldt. 2021 b. A System for Interactive Multimedia Retrieval Evaluations. In International Conference on Multimedia Modeling.

Digital Library

[22]

Luca Rossetto, Ivan Giangreco, and Heiko Schuldt. 2014. Cineast: a multi-feature sketch-based video retrieval engine. In IEEE International Symposium on Multimedia, ISM. IEEE Computer Society.

Digital Library

[23]

Luca Rossetto, Ivan Giangreco, Claudiu Tanase, and Heiko Schuldt. 2016. vitrivr: A flexible retrieval stack supporting multiple query modes for searching in multimedia collections. In Proceedings of the 24th ACM international conference on Multimedia.

Digital Library

[24]

Luca Rossetto, Mahnaz Amiri Parian, Ralph Gasser, Ivan Giangreco, Silvan Heller, and Heiko Schuldt. 2019 b. Deep learning-based concept detection in vitrivr. In International Conference on Multimedia Modeling.

[25]

Loris Sauter, Mahnaz Amiri Parian, Ralph Gasser, Silvan Heller, Luca Rossetto, and Heiko Schuldt. 2020. Combining boolean and multimedia retrieval in vitrivr for large-scale video search. In International Conference on Multimedia Modeling .

Digital Library

[26]

Loris Sauter, Luca Rossetto, and Heiko Schuldt. 2018. Exploring Cultural Heritage in Augmented Reality with GoFind!. In IEEE International Conference on Artificial Intelligence and Virtual Reality, AIVR.

[27]

Klaus Schoeffmann. 2019. Video Browser Showdown 2012--2019: A Review. In International Conference on Content-Based Multimedia Indexing, CBMI.

[28]

Omar Shahbaz Khan, Mathias Dybkjær Larsen, Liam Alex Sonto Poulsen, Björn Þór Jónsson, Jan Zahálka, Stevan Rudinac, Dennis Koelma, and Marcel Worring. 2020. Exquisitor at the Lifelog Search Challenge 2020. In Proceedings of the Third Annual Workshop on the Lifelog Search Challenge (LSC '20). ACM.

Digital Library

[29]

Ly-Duyen Tran, Manh-Duy Nguyen, Nguyen Thanh Binh, Hyowon Lee, and Cathal Gurrin. 2020 a. Myscéal: An Experimental Interactive Lifelog Retrieval System for LSC'20. In Proceedings of the Third Annual Workshop on Lifelog Search Challenge .

Digital Library

[30]

Minh-Triet Tran, Thanh-An Nguyen, Quoc-Cuong Tran, Mai-Khiem Tran, Khanh Nguyen, Van-Tu Ninh, Tu-Khiem Le, Hoang-Phuc Trang-Trung, Hoang-Anh Le, Hai-Dang Nguyen, Trong-Le Do, Viet-Khoa Vo-Ho, and Cathal Gurrin. 2020 b. FIRST - Flexible Interactive Retrieval SysTem for Visual Lifelog Exploration at LSC 2020. In Proceedings of the Third Annual Workshop on the Lifelog Search Challenge (LSC '20). ACM.

Digital Library

[31]

Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, Ming-Hsuan Yang, and Ling Shao. 2021. Multi-Stage Progressive Image Restoration. CoRR, Vol. abs/2102.02808 (2021).

Cited By

Sauter LGasser RRettig LSchuldt HRossetto L(2024)General Purpose Multimedia Retrieval with vitrivr at LSC'24Proceedings of the 7th Annual ACM Workshop on the Lifelog Search Challenge10.1145/3643489.3661120(47-52)Online publication date: 10-Jun-2024
https://dl.acm.org/doi/10.1145/3643489.3661120
Vuong GHo VNguyen Dang TThai XNguyen-Ho TPham MLe TNinh VHealy GGurrin CTran M(2024)LifeInsight2.0: An Enhanced Approach for Automated Lifelog Retrieval in LSC'24Proceedings of the 7th Annual ACM Workshop on the Lifelog Search Challenge10.1145/3643489.3661112(1-6)Online publication date: 10-Jun-2024
https://dl.acm.org/doi/10.1145/3643489.3661112
Ho HPham MTran QPham QNguyen L(2024)Glimpse: A Multimodal-based Transforming Image Collection with Vector Database2024 International Conference on Information Networking (ICOIN)10.1109/ICOIN59985.2024.10572095(415-420)Online publication date: 17-Jan-2024
https://doi.org/10.1109/ICOIN59985.2024.10572095
Show More Cited By

Index Terms

Interactive Multimodal Lifelog Retrieval with vitrivr at LSC 2021
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. Interactive systems and tools
2. Information systems
  1. Information retrieval
    1. Specialized information retrieval
      1. Multimedia and multimodal retrieval
        Image search
    2. Users and interactive retrieval
      1. Search interfaces
  2. Information systems applications
    1. Multimedia information systems
      1. Multimedia databases

Recommendations

Exploring Intuitive Lifelog Retrieval and Interaction Modes in Virtual Reality with vitrivr-VR
LSC '21: Proceedings of the 4th Annual on Lifelog Search Challenge

The multimodal nature of lifelog data collections poses unique challenges for multimedia management and retrieval systems. The Lifelog Search Challenge (LSC) offers an annual evaluation platform for such interactive retrieval systems. They compete ...
General Purpose Multimedia Retrieval with vitrivr at LSC'24
LSC '24: Proceedings of the 7th Annual ACM Workshop on the Lifelog Search Challenge

The collection of lifelog data --- visual and multi-sensory data, including biometric and spatiotemporal metadata --- becomes easier and more supported by commercial products every year. Naturally, lifelog data is multi-modal, with arguably a major audio-...
Interactive Lifelog Retrieval with vitrivr
LSC '20: Proceedings of the Third Annual Workshop on Lifelog Search Challenge

The variety and amount of data being collected in our everyday life poses unique challenges for multimedia retrieval. In the Lifelog Search Challenge (LSC), multimedia retrieval systems compete in finding events based on descriptions containing hints ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

LSC '21: Proceedings of the 4th Annual on Lifelog Search Challenge

August 2021

102 pages

ISBN:9781450385336

DOI:10.1145/3463948

General Chairs:
Cathal Gurrin
Dublin City University, Ireland
,
Klaus Schoeffmann
Klagenfurt University, Austria
,
Björn Jónsson
IT University of Copenhagen, Denmark
,
Duc-Tien Dang-Nguyen
University of Bergen, Norway
,
Jakub Lokoč
Charles University, Czech Republic
,
Minh-Triet Tran
HCM University of Science, Vietnam
,
Wolfgang Hürst
Utrecht University, Netherlands
,
Luca Rossetto
University of Zurich, Switzerland
,
Graham Healy
Dublin City University, Ireland

Copyright © 2021 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGMM: ACM Special Interest Group on Multimedia

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 21 August 2021

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Swiss National Science Foundation
Hasler Foundation

Conference

ICMR '21

Sponsor:

SIGMM

ICMR '21: International Conference on Multimedia Retrieval

August 21, 2021

Taipei, Taiwan

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

17
Total Citations
View Citations
135
Total Downloads

Downloads (Last 12 months)23
Downloads (Last 6 weeks)8

Reflects downloads up to 26 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Sauter LGasser RRettig LSchuldt HRossetto L(2024)General Purpose Multimedia Retrieval with vitrivr at LSC'24Proceedings of the 7th Annual ACM Workshop on the Lifelog Search Challenge10.1145/3643489.3661120(47-52)Online publication date: 10-Jun-2024
https://dl.acm.org/doi/10.1145/3643489.3661120
Vuong GHo VNguyen Dang TThai XNguyen-Ho TPham MLe TNinh VHealy GGurrin CTran M(2024)LifeInsight2.0: An Enhanced Approach for Automated Lifelog Retrieval in LSC'24Proceedings of the 7th Annual ACM Workshop on the Lifelog Search Challenge10.1145/3643489.3661112(1-6)Online publication date: 10-Jun-2024
https://dl.acm.org/doi/10.1145/3643489.3661112
Ho HPham MTran QPham QNguyen L(2024)Glimpse: A Multimodal-based Transforming Image Collection with Vector Database2024 International Conference on Information Networking (ICOIN)10.1109/ICOIN59985.2024.10572095(415-420)Online publication date: 17-Jan-2024
https://doi.org/10.1109/ICOIN59985.2024.10572095
Do TNguyen HNguyen QTran MHuynh VGurrin CNinh TLe TNgo TNgo TDang-Nguyen DTran M(2023)News Event Retrieval from Large Video Collection in Ho Chi Minh City AI Challenge 2023Proceedings of the 12th International Symposium on Information and Communication Technology10.1145/3628797.3628940(1011-1017)Online publication date: 7-Dec-2023
https://dl.acm.org/doi/10.1145/3628797.3628940
Vuong GHo VNguyen-Dang TThai XNinh VPham MLe THealy GTran M(2023)NewsInsight: A Comprehensive Video Event Retrieval System with Spatial Insights and Query AssistanceProceedings of the 12th International Symposium on Information and Communication Technology10.1145/3628797.3628805(893-900)Online publication date: 7-Dec-2023
https://dl.acm.org/doi/10.1145/3628797.3628805
Spiess FGasser RSchuldt HRossetto L(2023)The Best of Both Worlds: Lifelog Retrieval with a Desktop-Virtual Reality Hybrid SystemProceedings of the 6th Annual ACM Lifelog Search Challenge10.1145/3592573.3593107(65-68)Online publication date: 12-Jun-2023
https://dl.acm.org/doi/10.1145/3592573.3593107
Nguyen-Dang TThai XVuong GHo VTran MNinh VPham MLe THealy G(2023)LifeInsight: An Interactive Lifelog Retrieval System with Comprehensive Spatial Insights and Query AssistanceProceedings of the 6th Annual ACM Lifelog Search Challenge10.1145/3592573.3593106(59-64)Online publication date: 12-Jun-2023
https://dl.acm.org/doi/10.1145/3592573.3593106
Schoeffmann K(2023)lifeXplore at the Lifelog Search Challenge 2023Proceedings of the 6th Annual ACM Lifelog Search Challenge10.1145/3592573.3593105(53-58)Online publication date: 12-Jun-2023
https://dl.acm.org/doi/10.1145/3592573.3593105
Tran QTran LNguyen BGurrin C(2023)MemoriEase: An Interactive Lifelog Retrieval System for LSC’23Proceedings of the 6th Annual ACM Lifelog Search Challenge10.1145/3592573.3593101(30-35)Online publication date: 12-Jun-2023
https://dl.acm.org/doi/10.1145/3592573.3593101
Nguyen TLe TNinh VGurrin CTran MNguyen THealy GCaputo ASmyth S(2023)E-LifeSeeker: An Interactive Lifelog Search Engine for LSC’23Proceedings of the 6th Annual ACM Lifelog Search Challenge10.1145/3592573.3593098(13-17)Online publication date: 12-Jun-2023
https://dl.acm.org/doi/10.1145/3592573.3593098
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents