More Web Proxy on the site http://driver.im/

research-article

Ten Questions in Lifelog Mining and Information Recall

Authors:

Hen-Hsen Huang,

Hsin-Hsi ChenAuthors Info & Claims

ICMR '21: Proceedings of the 2021 International Conference on Multimedia Retrieval

Pages 511 - 518

https://doi.org/10.1145/3460426.3463607

Published: 01 September 2021 Publication History

Abstract

With the advance of science and technology, people are used to recording their daily life events via writing blogs, uploading social media posts, taking photos, or filming videos. Such rich repository personal information is useful for supporting human living assistance, such as information recall service. The main challenges are how to store and manage personal knowledge from various sources, and how to provide support for people who may have difficulty recalling past experiences. In this position paper, we propose a research agenda on personal knowledge mining from various sources of lifelogs, personal knowledge base construction, and information recall for assisting people to recall their experiences. Ten research questions are formulated.

References

[1]

Krisztian Balog and Tom Kenter. 2019. Personal knowledge graphs: A research agenda. In Proceedings of the 2019 ACM SIGIR International Conference on Theory of Information Retrieval. 217--220.

Digital Library

[2]

Yoshua Bengio, R. Janda, Y. W. Yu, Daphne Ippolito, Max Jarvie, D. Pilat, Brooke Struck, Sekoul Krastev, and A. Sharma. 2020. The need for privacy with public digital contact tracing during the COVID-19 pandemic. The Lancet. Digital Health, Vol. 2 (2020), e342 -- e344.

[3]

Marc Bola nos and Petia Radeva. 2015. Ego-object discovery. arXiv preprint arXiv:1504.01639 (2015).

[4]

Vannevar Bush. 1945. As we may think. The atlantic monthly, Vol. 176, 1 (1945), 101--108.

[5]

Chia-Chun Chang, Min-Huan Fu, Hen-Hsen Huang, and Hsin-Hsi Chen. 2019. An interactive approach to integrating external textual knowledge for multimodal lifelog retrieval. In Proceedings of the ACM Workshop on Lifelog Search Challenge. 41--44.

Digital Library

[6]

Yi Chang, Jiliang Tang, Dawei Yin, Makoto Yamada, and Yan Liu. 2016. Timeline summarization from social media with life cycle models. In IJCAI. 3698--3704.

[7]

Yu Chen, Fang Luo, Tong Li, Tao Xiang, Zheli Liu, and Jin Li. 2020. A training-integrity privacy-preserving federated learning scheme with trusted execution environment. Information Sciences, Vol. 522 (2020), 69--79.

[8]

Smitashree Choudhury and Harith Alani. 2014. Detecting presence of personal events in twitter streams. In International conference on social informatics. Springer, 157--166.

[9]

Tzu-Hsuan Chu, Hen-Hsen Huang, and Hsin-Hsi Chen. 2019. Image recall on image-text intertwined lifelogs. In 2019 IEEE/WIC/ACM International Conference on Web Intelligence (WI). IEEE, 398--402.

Digital Library

[10]

Tai-Te Chu, Chia-Chun Chang, An-Zi Yen, Hen-Hsen Huang, and Hsin-Hsi Chen. 2020 a. Multimodal retrieval through relations between subjects and objects in lifelog images. In Proceedings of the Third Annual Workshop on Lifelog Search Challenge. 51--55.

Digital Library

[11]

Tai-Te Chu, Yi-Ting Liu, Chia-Chung Chang, An-Zi Yen, Hen-Hsen Huang, and Hsin-Hsi Chen. 2020 b. NLP301 at the NTCIR-15 Micro-activity retrieval task: incorporating region of interest features into supervised encoder. In Proceedings of the NTCIR-15 Conference.

[12]

Duc-Tien Dang-Nguyen, Luca Piras, Michael Riegler, Liting Zhou, Mathias Lux, and Cathal Gurrin. [n.d.]. Overview of ImageCLEFlifelog 2018: Daily living understanding and lifelog moment retrieval. In CLEF2018 Working Notes.

[13]

Duc-Tien Dang-Nguyen, Luca Piras, Michael Riegler, Liting Zhou, Mathias Lux, Minh-Triet Tran, Tu-Khiem Le, Van-Tu Ninh, and Cathal Gurrin. 2019. Overview of ImageCLEFlifelog 2019: Solve my life puzzle and lifelog moment retrieval. In CLEF2019 Working Notes.

[14]

Ana Garcia del Molino, J. Lim, and A. Tan. 2018. Predicting visual context for unsupervised event segmentation in continuous photo-streams. Proceedings of the 26th ACM international conference on Multimedia (2018).

Digital Library

[15]

Aiden R Doherty, Niamh Caprani, Vaiva Kalnikaite, Cathal Gurrin, Alan F Smeaton, Noel E O'Connor, et al. 2011. Passively recognising human activities through lifelogging. Computers in Human Behavior, Vol. 27, 5 (2011), 1948--1958.

Digital Library

[16]

Chenyou Fan, Zehua Zhang, and David J Crandall. 2018. Deepdiary: Lifelogging image captioning and summarization. Journal of Visual Communication and Image Representation, Vol. 55 (2018), 40--55.

Digital Library

[17]

Alircza Fathi, Jessica K Hodgins, and James M Rehg. 2012. Social interactions: A first-person perspective. In 2012 IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 1226--1233.

[18]

Min-Huan Fu, Chia-Chun Chang, Hen-Hsen Huang, and Hsin-Hsi Chen. 2019. Incorporating external textual knowledge for life event recognition and retrieval. In Proceedings of the 14th NTCIR Conference on Evaluation of Information Access Technologies. 61--71.

[19]

Min-Huan Fu, An-Zi Yen, Hen-Hsen Huang, and Hsin-Hsi Chen. 2020. Incorporating semantic knowledge for visual lifelog activity recognition. In Proceedings of the 2020 International Conference on Multimedia Retrieval. 450--456.

Digital Library

[20]

Cathal Gurrin, Rami Albatal, Hideo Joho, and Kaori Ishii. 2014a. A privacy by design approach to lifelogging. Digital Enlightenment Yearbook 2014 (2014).

[21]

Cathal Gurrin, Hideo Joho, Frank Hopfgartner, Liting Zhou, and Rami Albatal. 2016. Ntcir lifelog: The first test collection for lifelog research. In Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval. 705--708.

Digital Library

[22]

Cathal Gurrin, Hideo Joho, Frank Hopfgartner, Liting Zhou, Rashmi Gupta, Rami Albatal, and Duc Tien Dang Nguyen. 2017. Overview of NTCIR-13 Lifelog-2 task.

[23]

Cathal Gurrin, Hideo Joho, Frank Hopfgartner, Liting Zhou, V-T Ninh, T-K Le, Rami Albatal, D-T Dang-Nguyen, and Graham Healy. 2019 a. Overview of the NTCIR-14 lifelog-3 task. In Proceedings of the 14th NTCIR conference. NII, 14--26.

[24]

Cathal Gurrin, Tu-Khiem Le, Van-Tu Ninh, Duc-Tien Dang-Nguyen, Björn Þór Jónsson, Jakub Lokovs, Wolfgang Hürst, Minh-Triet Tran, and Klaus Schoeffmann. 2020. Introduction to the third annual lifelog search challenge (LSC'20). In Proceedings of the 2020 International Conference on Multimedia Retrieval. 584--585.

Digital Library

[25]

Cathal Gurrin, Klaus Schoeffmann, Hideo Joho, Bernd Munzer, Rami Albatal, Frank Hopfgartner, Liting Zhou, and Duc-Tien Dang-Nguyen. 2019 b. A test collection for interactive lifelog retrieval. In International Conference on Multimedia Modeling. Springer, 312--324.

[26]

Cathal Gurrin, Alan F Smeaton, and Aiden R Doherty. 2014b. LifeLogging: Personal big bata. Foundations and Trends in Information Retrieval, Vol. 8, 1 (2014), 1--125.

Digital Library

[27]

Lu Jiang, Junwei Liang, Liangliang Cao, Yannis Kalantidis, Sachin Farfade, and Alexander Hauptmann. 2017. Memexqa: Visual memex question answering. arXiv preprint arXiv:1708.01336 (2017).

[28]

Nebojsa Jojic, Alessandro Perina, and Vittorio Murino. 2010. Structural epitome: A way to summarize one's visual experience. In Advances in neural information processing systems. 1027--1035.

[29]

Pei-Wei Kao, An-Zi Yen, Hen-Hsen Huang, and Hsin-Hsi Chen. 2021. ConvLogMiner: A Real-Time Conversational Lifelog Miner. In Proceedings of the 30th International Joint Conference on Artificial Intelligence.

[30]

Amel Ksibi, Ala Saleh Alluhaidan, Amina Salhi, and Sahar A El-Rahman. 2021. Overview of lifelogging: Current challenges and advances. IEEE Access (2021).

[31]

Jiwei Li and Claire Cardie. 2014. Timeline generation: Tracking individuals on twitter. In WWW 2014. 643--652.

[32]

Jiwei Li, Alan Ritter, Claire Cardie, and Eduard Hovy. 2014. Major life event extraction from twitter based on congratulations/condolences speech acts. In EMNLP 2014. 1997--2007.

[33]

Kangqi Luo, Fengli Lin, Xusheng Luo, and Kenny Zhu. 2018. Knowledge base question answering via encoding of complex query graphs. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. 2185--2194.

[34]

Jana Machajdik, Allan Hanbury, Angelika Garz, and Robert Sablatnig. 2011. Affective computing for wearable diary and lifelogging systems: An overview. In Machine Vision-Research for High Quality Processes and Products-35th Workshop of the Austrian Association for Pattern Recognition. 2447--2456.

[35]

Takuya Maekawa. 2013. A sensor device for automatic food lifelogging that is embedded in home ceiling light: A preliminary investigation. In 2013 7th International Conference on Pervasive Computing Technologies for Healthcare and Workshops. IEEE, 405--407.

Digital Library

[36]

Mohsin Ali Memon, Sania Bhatti, and Naeem Mahoto. 2016. A digital diary: Remembering the past using the present context. Mehran University Research Journal of Engineering and Technology, Vol. 35, 2 (2016), 275--286.

[37]

Yongfeng Qian, Long Hu, Jing Chen, Xin Guan, Mohammad Mehedi Hassan, and Abdulhameed Alelaiwi. 2019. Privacy-aware service placement for mobile edge computing via federated learning. Information Sciences, Vol. 505 (2019), 562--570.

Digital Library

[38]

Estefania Talavera, Mariella Dimiccoli, Marc Bolanos, Maedeh Aghaei, and Petia Radeva. 2015. R-clustering for egocentric video segmentation. In Iberian Conference on Pattern Recognition and Image Analysis. Springer, 327--336.

[39]

Stacey Truex, Nathalie Baracaldo, Ali Anwar, Thomas Steinke, Heiko Ludwig, Rui Zhang, and Yi Zhou. 2019. A hybrid approach to privacy-preserving federated learning. In Proceedings of the 12th ACM Workshop on Artificial Intelligence and Security. 1--11.

Digital Library

[40]

Yu-Wun Wang, Hen-Hsen Huang, Kuan-Yu Chen, and Hsin-Hsi Chen. 2018. Discourse marker detection for hesitation events on mandarin Conversation. Proceedings of Interspeech 2018 (2018), 1721--1725.

[41]

Emma Woodberry, Georgina Browne, Steve Hodges, Peter Watson, Narinder Kapur, and Ken Woodberry. 2015. The use of a wearable camera improves autobiographical memory in patients with Alzheimer's disease. Memory, Vol. 23, 3 (2015), 340--349.

[42]

Changhua Yang, Kevin Hsin-Yih Lin, and Hsin-Hsi Chen. 2009. Writer meets reader: Emotion analysis of social media from both the writer's and reader's perspectives. In 2009 IEEE/WIC/ACM International Joint Conference on Web Intelligence and Intelligent Agent Technology, Vol. 1. IEEE, 287--290.

Digital Library

[43]

An-Zi Yen. 2020. Lifelog mining for multimodal information recall over personal knowledge base. Ph.D. dissertation. National Taiwan University. https://hdl.handle.net/11296/jntjdm.

[44]

An-Zi Yen, Chia-Chung Chang, Hen-Hsen Huang, and Hsin-Hsi Chen. 2021 a. Personal knowledge base construction from multimodal data. In Proceedings of the 2021 International Conference on Multimedia Retrieval.

Digital Library

[45]

An-Zi Yen, Hen-Hsen Huang, and Hsin-Hsi Chen. 2019 a. Multimodal joint learning for personal knowledge base construction from Twitter-based lifelogs. Information Processing & Management (2019). https://doi.org/10.1016/j.ipm.2019.102148.

[46]

An-Zi Yen, Hen-Hsen Huang, and Hsin-Hsi Chen. 2019 b. Personal knowledge base construction from text-based lifelogs. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval. 185--194.

Digital Library

[47]

An-Zi Yen, Hen-Hsen Huang, and Hsin-Hsi Chen. 2021 b. Unanswerable question correction in question answering over personal knowledge base. In Thirty-Fifth AAAI Conference on Artificial Intelligence (AAAI-21).

[48]

Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. 2017. Places: A 10 million image database for scene recognition. IEEE transactions on pattern analysis and machine intelligence, Vol. 40, 6 (2017), 1452--1464.

Cited By

강 찬원 유김 학이 유(2024)패션 라이프로깅 앱 채택에 관한 연구Journal of the Korean Society of Clothing and Textiles10.5850/JKSCT.2024.48.5.97448:5(974-990)Online publication date: 31-Oct-2024
https://doi.org/10.5850/JKSCT.2024.48.5.974
陈亮(2024)A Lifelog Management Model Based on EventsComputer Science and Application10.12677/CSA.2024.14100514:01(29-40)Online publication date: 2024
https://doi.org/10.12677/CSA.2024.141005
Lee YYen AHuang HChen HHui Yang GWang HHan SHauff CZuccon GZhang Y(2024)ConvLogRecaller: Real-Time Conversational Lifelog RecallerProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3626772.3657659(2724-2728)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.1145/3626772.3657659
Show More Cited By

Index Terms

Ten Questions in Lifelog Mining and Information Recall
1. Information systems
  1. Information retrieval
    1. Retrieval tasks and goals
      1. Information extraction
    2. Users and interactive retrieval
      1. Personalization

Recommendations

Personal Knowledge Base Construction from Multimodal Data
ICMR '21: Proceedings of the 2021 International Conference on Multimedia Retrieval

With the passage of time, people often have misty memories of their past experiences. Information recall support for people by collecting personal lifelogs is emerging. Recently, people tend to record their daily life via filming Video Weblog (VLog), ...
VidLife: A Dataset for Life Event Extraction from Videos
CIKM '21: Proceedings of the 30th ACM International Conference on Information & Knowledge Management

Filming video blogs, which is shortened to vlog, becomes a popular way for people to record their life experiences in recent years. In this work, we present a novel task that is aimed at extracting life events from videos and constructing personal ...
Information recall in a virtual reality disability simulation
VRST '17: Proceedings of the 23rd ACM Symposium on Virtual Reality Software and Technology

The purpose of this paper is to investigate the effect of the sense of presence on one aspect of learning, information recall, in an immersive virtual reality (VR) disability simulation. Previous research has shown that the use of VR technology in ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

ICMR '21: Proceedings of the 2021 International Conference on Multimedia Retrieval

August 2021

715 pages

ISBN:9781450384636

DOI:10.1145/3460426

General Chairs:
Wen-Huang Cheng
National Yang Ming Chiao Tung University, Taiwan
,
Mohan Kankanhalli
National University of Singapore, Singapore
,
Meng Wang
Hefei University of Technology, China
,
Program Chairs:
Wei-Ta Chu
National Cheng Kung University, Taiwan
,
Jiaying Liu
Peking University, China
,
Marcel Worring
University of Amsterdam, Netherlands

Copyright © 2021 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGMM: ACM Special Interest Group on Multimedia

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 September 2021

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Ministry of Science and Technology, Taiwan

Conference

ICMR '21

Sponsor:

SIGMM

ICMR '21: International Conference on Multimedia Retrieval

August 21 - 24, 2021

Taipei, Taiwan

Acceptance Rates

Overall Acceptance Rate 254 of 830 submissions, 31%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

5
Total Citations
View Citations
142
Total Downloads

Downloads (Last 12 months)27
Downloads (Last 6 weeks)4

Reflects downloads up to 30 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

강 찬원 유김 학이 유(2024)패션 라이프로깅 앱 채택에 관한 연구Journal of the Korean Society of Clothing and Textiles10.5850/JKSCT.2024.48.5.97448:5(974-990)Online publication date: 31-Oct-2024
https://doi.org/10.5850/JKSCT.2024.48.5.974
陈亮(2024)A Lifelog Management Model Based on EventsComputer Science and Application10.12677/CSA.2024.14100514:01(29-40)Online publication date: 2024
https://doi.org/10.12677/CSA.2024.141005
Lee YYen AHuang HChen HHui Yang GWang HHan SHauff CZuccon GZhang Y(2024)ConvLogRecaller: Real-Time Conversational Lifelog RecallerProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3626772.3657659(2724-2728)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.1145/3626772.3657659
Yen AFu MAng WChu TTsai SHuang HChen H(2023)Visual lifelog retrieval: humans and machines interpretation on first-person imagesMultimedia Tools and Applications10.1007/s11042-023-14344-x82:24(37757-37787)Online publication date: 16-Feb-2023
https://dl.acm.org/doi/10.1007/s11042-023-14344-x
Chakraborty PSanyal D(2023)A comprehensive survey of personal knowledge graphsWIREs Data Mining and Knowledge Discovery10.1002/widm.151313:6Online publication date: 10-Aug-2023
https://doi.org/10.1002/widm.1513

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents