MEMORIA: a Memory Enhancement and MOment RetrIeval Application for LSC 2023

Abstract

The continuous collection and storage of personal data, denoted Lifelogging, has gained popularity in recent years as a means of monitoring and improving personal health. One important aspect of lifelogging is the collection and analysis of image data, which can provide valuable insights into an individual’s lifestyle, dietary habits, and physical activity. The Lifelog Search Challenge provides a unique opportunity to explore the state-of-the-art in lifelogging research, particularly in the area of egocentric image retrieval and analysis. Researchers can propose their approaches and compete to solve lifelog retrieval challenges and evaluate the effectiveness of their systems on a rich multimodal dataset generated by an active lifelogger with 18 months of continuous capture of lifelogging data. This paper presents the second version of MEMORIA, a computational tool developed to participate in the Lifelog Search Challenge 2023. In this new version, the information retrieval is based on the use of natural language search with the possibility to filter the results based on keywords and time periods. The system applies image analysis algorithms to process visual lifelogs, from pre-processing algorithms to feature extraction methods, in order to enrich the annotation of the lifelogs. This new version explores the use of a graph database, more detailed image annotation, and event segmentation, in order to improve the performance and user interaction. Experimental results of the user interaction with our retrieval module are presented, confirming the effectiveness of the proposed approach and showing the most relevant functionalities of the system.