More Web Proxy on the site http://driver.im/

research-article

Open access

MEMORIA: A Memory Enhancement and MOment RetrIeval Application for LSC 2023

Authors:

Ricardo Ribeiro,

António J. R. Neves,

Pedro IglésiasAuthors Info & Claims

LSC '23: Proceedings of the 6th Annual ACM Lifelog Search Challenge

Pages 18 - 23

https://doi.org/10.1145/3592573.3593099

Published: 12 June 2023 Publication History

All formats PDF

Abstract

The continuous collection and storage of personal data, denoted Lifelogging, has gained popularity in recent years as a means of monitoring and improving personal health. One important aspect of lifelogging is the collection and analysis of image data, which can provide valuable insights into an individual’s lifestyle, dietary habits, and physical activity. The Lifelog Search Challenge provides a unique opportunity to explore the state-of-the-art in lifelogging research, particularly in the area of egocentric image retrieval and analysis. Researchers can propose their approaches and compete to solve lifelog retrieval challenges and evaluate the effectiveness of their systems on a rich multimodal dataset generated by an active lifelogger with 18 months of continuous capture of lifelogging data. This paper presents the second version of MEMORIA, a computational tool developed to participate in the Lifelog Search Challenge 2023. In this new version, the information retrieval is based on the use of natural language search with the possibility to filter the results based on keywords and time periods. The system applies image analysis algorithms to process visual lifelogs, from pre-processing algorithms to feature extraction methods, in order to enrich the annotation of the lifelogs. This new version explores the use of a graph database, more detailed image annotation, and event segmentation, in order to improve the performance and user interaction. Experimental results of the user interaction with our retrieval module are presented, confirming the effectiveness of the proposed approach and showing the most relevant functionalities of the system.

References

[1]

2022. ultralytics/yolov5: v7.0 - YOLOv5 SOTA Realtime Instance Segmentation. https://doi.org/10.5281/ZENODO.7347926

[2]

Naushad Alam, Yvette Graham, and Cathal Gurrin. 2022. Memento 2.0: An Improved Lifelog Search Engine for LSC’22. In Proceedings of the 5th Annual on Lifelog Search Challenge (Newark, NJ, USA) (LSC ’22). Association for Computing Machinery, New York, NY, USA, 2–7. https://doi.org/10.1145/3512729.3533006

Digital Library

[3]

Ahmed Alateeq, Mark Roantree, and Cathal Gurrin. 2022. Voxento 3.0: A Prototype Voice-Controlled Interactive Search Engine for Lifelog. In Proceedings of the 5th Annual on Lifelog Search Challenge (Newark, NJ, USA) (LSC ’22). Association for Computing Machinery, New York, NY, USA, 43–47. https://doi.org/10.1145/3512729.3533009

Digital Library

[4]

Youngmin Baek, Bado Lee, Dongyoon Han, Sangdoo Yun, and Hwalsuk Lee. 2019. Character Region Awareness for Text Detection. https://doi.org/10.48550/ARXIV.1904.01941

[5]

Cathal Gurrin, Liting Zhou, Graham Healy, Björn Þór Jónsson, Duc-Tien Dang-Nguyen, Jakub Lokoć, Minh-Triet Tran, Wolfgang Hürst, Luca Rossetto, and Klaus Schöffmann. 2022. Introduction to the Fifth Annual Lifelog Search Challenge, LSC’22. In Proceedings of the 2022 International Conference on Multimedia Retrieval (Newark, NJ, USA) (ICMR ’22). Association for Computing Machinery, New York, NY, USA, 685–687. https://doi.org/10.1145/3512527.3531439

Digital Library

[6]

Cathal Gurrin, Björn Þór Jónsson, Klaus Schöffmann, Duc-Tien Dang-Nguyen, Jakub Lokoč, Minh-Triet Tran, Wolfgang Hürst, Luca Rossetto, and Graham Healy. 2023. Introduction to the Sixth Annual Lifelog Search Challenge, LSC’23. In Proc. International Conference on Multimedia Retrieval (ICMR’23). ACM, Thessaloniki, Greece.

Digital Library

[7]

Silvan Heller, Luca Rossetto, Loris Sauter, and Heiko Schuldt. 2022. Vitrivr at the Lifelog Search Challenge 2022. In Proceedings of the 5th Annual on Lifelog Search Challenge (Newark, NJ, USA) (LSC ’22). Association for Computing Machinery, New York, NY, USA, 27–31. https://doi.org/10.1145/3512729.3533003

Digital Library

[8]

Jun Heo, Jaeyeon Won, Yejin Lee, Shivam Bharuka, Jaeyoung Jang, Tae Jun Ham, and Jae W. Lee. 2020. IIU. In Proceedings of the Twenty-Fifth International Conference on Architectural Support for Programming Languages and Operating Systems. ACM. https://doi.org/10.1145/3373376.3378521

Digital Library

[9]

Nhat Hoang-Xuan, Hoang-Phuc Trang-Trung, E-Ro Nguyen, Thanh-Cong Le, Mai-Khiem Tran, Tu-Khiem Le, Van-Tu Ninh, Cathal Gurrin, and Minh-Triet Tran. 2022. Flexible Interactive Retrieval SysTem 3.0 for Visual Lifelog Exploration at LSC 2022. In Proceedings of the 5th Annual on Lifelog Search Challenge (Newark, NJ, USA) (LSC ’22). Association for Computing Machinery, New York, NY, USA, 20–26. https://doi.org/10.1145/3512729.3533013

Digital Library

[10]

Andreas Leibetseder, Daniela Stefanics, and Klaus Schoeffmann. 2022. LifeXplore at the Lifelog Search Challenge 2022. In Proceedings of the 5th Annual on Lifelog Search Challenge (Newark, NJ, USA) (LSC ’22). Association for Computing Machinery, New York, NY, USA, 48–52. https://doi.org/10.1145/3512729.3533005

Digital Library

[11]

Leland McInnes, John Healy, and Steve Astels. 2017. hdbscan: Hierarchical density based clustering. The Journal of Open Source Software 2, 11 (2017), 205.

[12]

Nikolaj Mertz, Björn Þór Jónsson, and Aaron Duane. 2022. NeoCube. In Proceedings of the 2nd International Workshop on Interactive Multimedia Retrieval. ACM. https://doi.org/10.1145/3552467.3554799

Digital Library

[13]

Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient Estimation of Word Representations in Vector Space. https://doi.org/10.48550/ARXIV.1301.3781

[14]

Ron Mokady, Amir Hertz, and Amit H. Bermano. 2021. ClipCap: CLIP Prefix for Image Captioning. https://doi.org/10.48550/ARXIV.2111.09734

[15]

Thao-Nhu Nguyen, Tu-Khiem Le, Van-Tu Ninh, Minh-Triet Tran, Thanh Binh Nguyen, Graham Healy, Sinéad Smyth, Annalina Caputo, and Cathal Gurrin. 2022. LifeSeeker 4.0: An Interactive Lifelog Search Engine for LSC’22. In Proceedings of the 5th Annual on Lifelog Search Challenge (Newark, NJ, USA) (LSC ’22). Association for Computing Machinery, New York, NY, USA, 14–19. https://doi.org/10.1145/3512729.3533014

Digital Library

[16]

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning Transferable Visual Models From Natural Language Supervision. https://doi.org/10.48550/ARXIV.2103.00020

[17]

Ricardo Ribeiro, Alina Trifan, and António JR Neves. 2022. Impact of Blind Image Quality Assessment on the Retrieval of Lifelog Images. In Proceedings of the 2nd International Workshop on Interactive Multimedia Retrieval. 25–31.

Digital Library

[18]

Ricardo Ribeiro, Alina Trifan, and António JR Neves. 2023. Blind Image Quality Assessment with Deep Learning: A Replicability Study and Its Reproducibility in Lifelogging. Applied Sciences 13, 1 (2023), 59.

[19]

Ricardo Ribeiro, Alina Trifan, António JR Neves, 2022. Lifelog Retrieval From Daily Digital Data: Narrative Review. JMIR mHealth and uHealth 10, 5 (2022), e30517.

[20]

Florian Spiess and Heiko Schuldt. 2022. Multimodal Interactive Lifelog Retrieval with Vitrivr-VR. In Proceedings of the 5th Annual on Lifelog Search Challenge (Newark, NJ, USA) (LSC ’22). Association for Computing Machinery, New York, NY, USA, 38–42. https://doi.org/10.1145/3512729.3533008

Digital Library

[21]

Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. 2016. Rethinking the Inception Architecture for Computer Vision. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2818–2826. https://doi.org/10.1109/CVPR.2016.308

[22]

Ly-Duyen Tran, Manh-Duy Nguyen, Binh Nguyen, Hyowon Lee, Liting Zhou, and Cathal Gurrin. 2022. E-Myscéal: Embedding-Based Interactive Lifelog Retrieval System for LSC’22. In Proceedings of the 5th Annual on Lifelog Search Challenge (Newark, NJ, USA) (LSC ’22). Association for Computing Machinery, New York, NY, USA, 32–37. https://doi.org/10.1145/3512729.3533012

Digital Library

[23]

Ly-Duyen Tran, Manh-Duy Nguyen, Nguyen Thanh Binh, Hyowon Lee, and Cathal Gurrin. 2021. Myscéal 2.0: A Revised Experimental Interactive Lifelog Retrieval System for LSC’21. In Proceedings of the 4th Annual on Lifelog Search Challenge (Taipei, Taiwan) (LSC ’21). Association for Computing Machinery, New York, NY, USA, 11–16. https://doi.org/10.1145/3463948.3469064

Digital Library

[24]

Chien-Yao Wang, Alexey Bochkovskiy, and Hong-Yuan Mark Liao. 2022. YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors. https://doi.org/10.48550/ARXIV.2207.02696

[25]

Jialian Wu, Jianfeng Wang, Zhengyuan Yang, Zhe Gan, Zicheng Liu, Junsong Yuan, and Lijuan Wang. 2022. GRiT: A Generative Region-to-text Transformer for Object Understanding. arXiv preprint arXiv:2212.00280 (2022).

[26]

Yu Zheng, Hao Fu, Xing Xie, Wei-Ying Ma, and Quannan Li. 2011. Geolife GPS trajectory dataset - User Guide. Geolife GPS trajectories (July 2011).

Cited By

Vopálková ZYaghob JStroh MSchlegel ULokoc J(2024)Searching Temporally Distant Activities in Lifelog Data With PraK Tool V2Proceedings of the 7th Annual ACM Workshop on the Lifelog Search Challenge10.1145/3643489.3661131(111-116)Online publication date: 10-Jun-2024
https://dl.acm.org/doi/10.1145/3643489.3661131
Gago AKaluza BBartolomeu EPandeirada JRibeiro RNeves A(2024)MEMORIA: A Memory Enhancement and MOment RetrIeval Application at the LSC2024Proceedings of the 7th Annual ACM Workshop on the Lifelog Search Challenge10.1145/3643489.3661129(99-104)Online publication date: 10-Jun-2024
https://dl.acm.org/doi/10.1145/3643489.3661129
Alam NGraham YGurrin C(2024)Memento 4.0: A Prototype Conversational Search System for LSC'24Proceedings of the 7th Annual ACM Workshop on the Lifelog Search Challenge10.1145/3643489.3661126(82-87)Online publication date: 10-Jun-2024
https://dl.acm.org/doi/10.1145/3643489.3661126
Show More Cited By

Index Terms

MEMORIA: A Memory Enhancement and MOment RetrIeval Application for LSC 2023
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. Interactive systems and tools
2. Information systems
  1. Information retrieval
    1. Users and interactive retrieval
      1. Search interfaces

Recommendations

E-LifeSeeker: An Interactive Lifelog Search Engine for LSC’23
LSC '23: Proceedings of the 6th Annual ACM Lifelog Search Challenge

Lifelogging is referred to as the process of automatically capturing the everyday activities of an individual to ultimately create a digital diary for further sharing, which can be challenging to manage and retrieve due to its multimodal nature. ...
MyEachtra: Event-Based Interactive Lifelog Retrieval System for LSC’23
LSC '23: Proceedings of the 6th Annual ACM Lifelog Search Challenge

Retrieval is a fundamental challenge within the research community of lifelog and the Lifelog Search Challenge (LSC) has been an important annual benchmarking activity for interactive lifelog retrieval systems since 2018. This paper proposes MyEachtra (...
MemoriEase: An Interactive Lifelog Retrieval System for LSC’23
LSC '23: Proceedings of the 6th Annual ACM Lifelog Search Challenge

Lifelogging is an activity of recording all events that happen in the daily life of an individual. The events can contain images, audio, health index, etc which are collected through various devices such as wearable cameras, smartwatches, and other ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

LSC '23: Proceedings of the 6th Annual ACM Lifelog Search Challenge

June 2023

74 pages

ISBN:9798400701887

DOI:10.1145/3592573

Copyright © 2023 Owner/Author.

This work is licensed under a Creative Commons Attribution International 4.0 License.

Sponsors

SIGMM: ACM Special Interest Group on Multimedia

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 12 June 2023

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Funding Sources

FCT - Foundation for Science and Technology

Conference

ICMR '23

Sponsor:

SIGMM

ICMR '23: International Conference on Multimedia Retrieval

June 12 - 15, 2023

Thessaloniki, Greece

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
394
Total Downloads

Downloads (Last 12 months)229
Downloads (Last 6 weeks)18

Reflects downloads up to 17 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Vopálková ZYaghob JStroh MSchlegel ULokoc J(2024)Searching Temporally Distant Activities in Lifelog Data With PraK Tool V2Proceedings of the 7th Annual ACM Workshop on the Lifelog Search Challenge10.1145/3643489.3661131(111-116)Online publication date: 10-Jun-2024
https://dl.acm.org/doi/10.1145/3643489.3661131
Gago AKaluza BBartolomeu EPandeirada JRibeiro RNeves A(2024)MEMORIA: A Memory Enhancement and MOment RetrIeval Application at the LSC2024Proceedings of the 7th Annual ACM Workshop on the Lifelog Search Challenge10.1145/3643489.3661129(99-104)Online publication date: 10-Jun-2024
https://dl.acm.org/doi/10.1145/3643489.3661129
Alam NGraham YGurrin C(2024)Memento 4.0: A Prototype Conversational Search System for LSC'24Proceedings of the 7th Annual ACM Workshop on the Lifelog Search Challenge10.1145/3643489.3661126(82-87)Online publication date: 10-Jun-2024
https://dl.acm.org/doi/10.1145/3643489.3661126
Sauter LGasser RRettig LSchuldt HRossetto L(2024)General Purpose Multimedia Retrieval with vitrivr at LSC'24Proceedings of the 7th Annual ACM Workshop on the Lifelog Search Challenge10.1145/3643489.3661120(47-52)Online publication date: 10-Jun-2024
https://dl.acm.org/doi/10.1145/3643489.3661120

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents