More Web Proxy on the site http://driver.im/

research-article

Open access

lifeXplore at the Lifelog Search Challenge 2023

Author:

Klaus SchoeffmannAuthors Info & Claims

LSC '23: Proceedings of the 6th Annual ACM Lifelog Search Challenge

Pages 53 - 58

https://doi.org/10.1145/3592573.3593105

Published: 12 June 2023 Publication History

All formats PDF

Abstract

Searching substantial data archives of lifeloggers is a challenging task. The Lifelog Search Challenge (LSC) is an annually held competition with the aim of encouraging international teams to develop interactive content retrieval systems capable of searching large lifelog databases. LSC takes place as a live event co-located with the ACM International Conference on Multimedia Retrieval (ICMR), where teams compete against each other by solving retrieval tasks issued by the lifelogger. This paper presents our newest version of lifeXplore, a lifelog retrieval system that has been participating in LSC since 2018. For this year, we significantly redesign the entire system (backend, middleware, and frontend) and integrate free text-search using embeddings from vision transformers trained with large sets of text-image pairs. We present a novel architecture for multi-source search, where results from image embeddings are used together with results from traditional content analysis (for objects, concepts, and recognized text). We also perform intensive analysis of vision transformer models in order to know which one fits best to the requirements of the LSC.

References

[1]

Naushad Alam, Yvette Graham, and Cathal Gurrin. 2021. Memento: A Prototype Lifelog Search Engine for LSC’21. In Proceedings of the 4th Annual on Lifelog Search Challenge. 53–58.

Digital Library

[2]

Ahmed Alateeq, Mark Roantree, and Cathal Gurrin. 2020. Voxento: a prototype voice-controlled interactive search engine for lifelogs. In Proceedings of the Third Annual Workshop on Lifelog Search Challenge. 77–81.

Digital Library

[3]

Youngmin Baek, Bado Lee, Dongyoon Han, Sangdoo Yun, and Hwalsuk Lee. 2019. Character region awareness for text detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 9365–9374.

[4]

Duc Tien Dang Nguyen Graham Healy Jakub Lokoc Liting Zhou Luca Rossetto Minh-Triet Tran Wolfgang Hürst Werner Bailer Klaus Schoeffmann Cathal Gurrin, Björn Þór Jónsson. 2023. Introduction to the Sixth Annual Lifelog Search Challenge, LSC’23. In Proc. International Conference on Multimedia Retrieval (ICMR’23)(ICMR ’23). New York, NY, USA.

[5]

Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. 2022. Reproducible scaling laws for contrastive language-image learning. https://doi.org/10.48550/ARXIV.2212.07143

[6]

Aaron Duane and Bjorn Þór Jónsson. 2021. ViRMA: Virtual Reality Multimedia Analytics at LSC 2021. In Proceedings of the 4th Annual on Lifelog Search Challenge. 29–34.

Digital Library

[7]

Alexander Christian Faisst and Björn Þór Jónsson. 2021. LifeMon: A MongoDB-Based Lifelog Retrieval Prototype. In Proceedings of the 4th Annual on Lifelog Search Challenge. 75–80.

Digital Library

[8]

Silvan Heller, Ralph Gasser, Mahnaz Parian-Scherb, Sanja Popovic, Luca Rossetto, Loris Sauter, Florian Spiess, and Heiko Schuldt. 2021. Interactive multimodal lifelog retrieval with Vitrivr at LSC 2021. In Proceedings of the 4th Annual on Lifelog Search Challenge. 35–39.

Digital Library

[9]

Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2019. Billion-scale similarity search with GPUs. IEEE Transactions on Big Data 7, 3 (2019), 535–547.

[10]

Omar Shahbaz Khan, Aaron Duane, Björn Þór Jónsson, Jan Zahálka, Stevan Rudinac, and Marcel Worring. 2021. Exquisitor at the Lifelog Search Challenge 2021: Relationships Between Semantic Classifiers. In Proceedings of the 4th Annual on Lifelog Search Challenge. 3–6.

Digital Library

[11]

Tu-Khiem Le, Van-Tu Ninh, Mai-Khiem Tran, Graham Healy, Cathal Gurrin, and Minh-Triet Tran. 2022. AVSeeker: An Active Video Retrieval Engine at VBS2022. In MultiMedia Modeling: 28th International Conference, MMM 2022, Phu Quoc, Vietnam, June 6–10, 2022, Proceedings, Part II. Springer, 537–542.

Digital Library

[12]

Andreas Leibetseder and Klaus Schoeffmann. 2021. lifexplore at the lifelog search challenge 2021. In Proceedings of the 4th Annual on Lifelog Search Challenge. 23–28.

Digital Library

[13]

Jakub Lokoč, František Mejzlik, Patrik Veselỳ, and Tomáš Souček. 2021. Enhanced SOMHunter for Known-item Search in Lifelog Data. In Proceedings of the 4th Annual on Lifelog Search Challenge. 71–73.

Digital Library

[14]

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, 2021. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning. PMLR, 8748–8763.

[15]

Konstantin Schall, Kai Uwe Barthel, Nico Hezel, and Klaus Jung. 2022. GPR1200: A Benchmark for General-Purpose Content-Based Image Retrieval. In International Conference on Multimedia Modeling. Springer, 205–216.

Digital Library

[16]

Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, 2022. Laion-5b: An open large-scale dataset for training next generation image-text models. arXiv preprint arXiv:2210.08402 (2022).

[17]

Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. 2021. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.02114 (2021).

[18]

Jihye Shin, Alexandra Waldau, Aaron Duane, and Björn Þór Jónsson. 2021. PhotoCube at the Lifelog Search Challenge 2021. In Proceedings of the 4th Annual on Lifelog Search Challenge. 59–63.

Digital Library

[19]

Florian Spiess and Heiko Schuldt. 2022. Multimodal Interactive Lifelog Retrieval with vitrivr-VR. In Proceedings of the 5th Annual on Lifelog Search Challenge. 38–42.

Digital Library

[20]

Mingxing Tan and Quoc Le. 2019. Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference on machine learning. PMLR, 6105–6114.

[21]

Ly-Duyen Tran, Naushad Alam, Yvette Graham, Linh Khanh Vo, Nghiem Tuong Diep, Binh Nguyen, Liting Zhou, and Cathal Gurrin. 2022. An Exploration into the Benefits of the CLIP model for Lifelog Retrieval. In Proceedings of the 19th International Conference on Content-based Multimedia Indexing. 15–22.

Digital Library

[22]

Ly-Duyen Tran, Manh-Duy Nguyen, Duc-Tien Dang-Nguyen, Silvan Heller, Florian Spiess, Jakub Lokoč, Ladislav Peška, Thao-Nhu Nguyen, Omar Shahbaz Khan, Aaron Duane, 2023. Comparing Interactive Retrieval Approaches at the Lifelog Search Challenge 2021. IEEE Access (2023).

[23]

Ly-Duyen Tran, Manh-Duy Nguyen, Binh Nguyen, Hyowon Lee, Liting Zhou, and Cathal Gurrin. 2022. E-Myscéal: Embedding-based Interactive Lifelog Retrieval System for LSC’22. In Proceedings of the 5th Annual on Lifelog Search Challenge. 32–37.

Digital Library

[24]

Ly-Duyen Tran, Manh-Duy Nguyen, Nguyen Thanh Binh, Hyowon Lee, and Cathal Gurrin. 2021. Myscéal 2.0: A Revised Experimental Interactive Lifelog Retrieval System for LSC’21. In Proceedings of the 4th Annual on Lifelog Search Challenge. 11–16.

Digital Library

[25]

Chien-Yao Wang, Alexey Bochkovskiy, and Hong-Yuan Mark Liao. 2022. YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors. arXiv preprint arXiv:2207.02696 (2022).

[26]

Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. 2018. Places: A 10 million Image Database for Scene Recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence 40, 6 (2018), 1452–1464. https://doi.org/10.1109/TPAMI.2017.2723009

Cited By

Gurrin CZhou LHealy GBailer WDang Nguyen DHodges SJónsson BLokoč JRossetto LTran MSchöffmann KGurrin CKongkachandra RSchoeffmann KDang-Nguyen DRossetto LSatoh SZhou L(2024)Introduction to the Seventh Annual Lifelog Search Challenge, LSC'24Proceedings of the 2024 International Conference on Multimedia Retrieval10.1145/3652583.3658891(1334-1335)Online publication date: 30-May-2024
https://dl.acm.org/doi/10.1145/3652583.3658891
Vopálková ZYaghob JStroh MSchlegel ULokoc J(2024)Searching Temporally Distant Activities in Lifelog Data With PraK Tool V2Proceedings of the 7th Annual ACM Workshop on the Lifelog Search Challenge10.1145/3643489.3661131(111-116)Online publication date: 10-Jun-2024
https://dl.acm.org/doi/10.1145/3643489.3661131
Alateeq ARoantree MGurrin C(2024)Voxento-Pro: An Advanced Voice Lifelog Retrieval Interaction for Multimodal LifelogsProceedings of the 7th Annual ACM Workshop on the Lifelog Search Challenge10.1145/3643489.3661130(105-110)Online publication date: 10-Jun-2024
https://dl.acm.org/doi/10.1145/3643489.3661130
Show More Cited By

Index Terms

lifeXplore at the Lifelog Search Challenge 2023
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. Interactive systems and tools
2. Information systems
  1. Information retrieval
    1. Specialized information retrieval
      1. Multimedia and multimodal retrieval
    2. Users and interactive retrieval
      1. Search interfaces

Recommendations

lifeXplore at the Lifelog Search Challenge 2019
LSC '19: Proceedings of the ACM Workshop on Lifelog Search Challenge

First held in Yokohama, Japan in 2018, the Lifelogging Search Challenge (LSC) is an interactive lifelogging content retrieval competition, where international teams are faced with the task of finding specific moments within a large collection of ...
lifeXplore at the Lifelog Search Challenge 2022
LSC '22: Proceedings of the 5th Annual on Lifelog Search Challenge

Lifelogging creates substantial data archives that are challenging to manage and search. The annual Lifelog Search Challenge (LSC) aims at improving this situation by encouraging international teams to create interactive retrieval systems for searching ...
lifeXplore at the Lifelog Search Challenge 2024
LSC '24: Proceedings of the 7th Annual ACM Workshop on the Lifelog Search Challenge

Building on the success of lifeXplore 2023, which is the winner system of last year's Lifelog Search Challenge (LSC2023), we present a lifelog retrieval system that is both easy-to-use and effective in retrieving lifelog data. lifeXplore 2024 employs ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

LSC '23: Proceedings of the 6th Annual ACM Lifelog Search Challenge

June 2023

74 pages

ISBN:9798400701887

DOI:10.1145/3592573

Copyright © 2023 Owner/Author.

This work is licensed under a Creative Commons Attribution International 4.0 License.

Sponsors

SIGMM: ACM Special Interest Group on Multimedia

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 12 June 2023

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

ICMR '23

Sponsor:

SIGMM

ICMR '23: International Conference on Multimedia Retrieval

June 12 - 15, 2023

Thessaloniki, Greece

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

18
Total Citations
View Citations
615
Total Downloads

Downloads (Last 12 months)319
Downloads (Last 6 weeks)31

Reflects downloads up to 07 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Gurrin CZhou LHealy GBailer WDang Nguyen DHodges SJónsson BLokoč JRossetto LTran MSchöffmann KGurrin CKongkachandra RSchoeffmann KDang-Nguyen DRossetto LSatoh SZhou L(2024)Introduction to the Seventh Annual Lifelog Search Challenge, LSC'24Proceedings of the 2024 International Conference on Multimedia Retrieval10.1145/3652583.3658891(1334-1335)Online publication date: 30-May-2024
https://dl.acm.org/doi/10.1145/3652583.3658891
Vopálková ZYaghob JStroh MSchlegel ULokoc J(2024)Searching Temporally Distant Activities in Lifelog Data With PraK Tool V2Proceedings of the 7th Annual ACM Workshop on the Lifelog Search Challenge10.1145/3643489.3661131(111-116)Online publication date: 10-Jun-2024
https://dl.acm.org/doi/10.1145/3643489.3661131
Alateeq ARoantree MGurrin C(2024)Voxento-Pro: An Advanced Voice Lifelog Retrieval Interaction for Multimodal LifelogsProceedings of the 7th Annual ACM Workshop on the Lifelog Search Challenge10.1145/3643489.3661130(105-110)Online publication date: 10-Jun-2024
https://dl.acm.org/doi/10.1145/3643489.3661130
Gago AKaluza BBartolomeu EPandeirada JRibeiro RNeves A(2024)MEMORIA: A Memory Enhancement and MOment RetrIeval Application at the LSC2024Proceedings of the 7th Annual ACM Workshop on the Lifelog Search Challenge10.1145/3643489.3661129(99-104)Online publication date: 10-Jun-2024
https://dl.acm.org/doi/10.1145/3643489.3661129
Alam NGraham YGurrin C(2024)Memento 4.0: A Prototype Conversational Search System for LSC'24Proceedings of the 7th Annual ACM Workshop on the Lifelog Search Challenge10.1145/3643489.3661126(82-87)Online publication date: 10-Jun-2024
https://dl.acm.org/doi/10.1145/3643489.3661126
Hezel NSchall KSchilling BJung KBarthel K(2024)Libro - Lifelog Search BrowserProceedings of the 7th Annual ACM Workshop on the Lifelog Search Challenge10.1145/3643489.3661124(70-75)Online publication date: 10-Jun-2024
https://dl.acm.org/doi/10.1145/3643489.3661124
Rader MSchoeffmann K(2024)lifeXplore at the Lifelog Search Challenge 2024Proceedings of the 7th Annual ACM Workshop on the Lifelog Search Challenge10.1145/3643489.3661123(64-69)Online publication date: 10-Jun-2024
https://dl.acm.org/doi/10.1145/3643489.3661123
Le HNguyen TLe TTran MNguyen TNinh VZhou LGurrin C(2024)LifeSeeker 6.0: Leveraging the linguistic aspect of the lifelog system in LSC'24Proceedings of the 7th Annual ACM Workshop on the Lifelog Search Challenge10.1145/3643489.3661121(53-57)Online publication date: 10-Jun-2024
https://dl.acm.org/doi/10.1145/3643489.3661121
Sauter LGasser RRettig LSchuldt HRossetto L(2024)General Purpose Multimedia Retrieval with vitrivr at LSC'24Proceedings of the 7th Annual ACM Workshop on the Lifelog Search Challenge10.1145/3643489.3661120(47-52)Online publication date: 10-Jun-2024
https://dl.acm.org/doi/10.1145/3643489.3661120
Steffensen INystad SLiahjell MDang-Nguyen DKesavulu MNinh VGurrin CVuong GHo VTran M(2024)T@Retrospect: A Journey Through TimeProceedings of the 7th Annual ACM Workshop on the Lifelog Search Challenge10.1145/3643489.3661118(36-40)Online publication date: 10-Jun-2024
https://dl.acm.org/doi/10.1145/3643489.3661118
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Figures

Tables

Media

View Table of Conten