Exploring the Detection of Spontaneous Recollections during Video-viewing In-the-Wild using Facial Behavior Analysis
Authors: 
Bernd Dudzik, Hayley HungAuthors Info & Claims
Pages 236 - 246
Abstract
Intelligent systems might benefit from automatically detecting when a stimulus has triggered a user’s recollection of personal memories, e.g., to identify that a piece of media content holds personal significance for them. While computational research has demonstrated the potential to identify related states based on facial behavior (e.g., mind-wandering), the automatic detection of spontaneous recollections specifically has not been investigated this far. Motivated by this, we present machine learning experiments exploring the feasibility of detecting whether a video clip has triggered personal memories in a viewer based on the analysis of their Head Rotation, Head Position, Eye Gaze, and Facial Expressions. Concretely, we introduce an approach for automatic detection and evaluate its potential for predictions using in-the-wild webcam recordings. Overall, our findings demonstrate the capacity for above chance detections in both settings, with substantially better performance for the video-independent variant. Beyond this, we investigate the role of person-specific recollection biases for predictions of our video-independent models and the importance of specific modalities of facial behavior. Finally, we discuss the implications of our findings for detecting recollections and user-modeling in adaptive systems.
References
[1]
Sonja Annerer-Walcher, Simon M. Ceh, Felix Putze, Marvin Kampen, Christof Körner, and Mathias Benedek. 2021. How Reliably Do Eye Parameters Indicate Internal Versus External Attentional Focus?Cognitive Science 45, 4 (2021). https://doi.org/10.1111/cogs.12977
[2]
Anthony Bagnall, Jason Lines, Aaron Bostrom, James Large, and Eamonn Keogh. 2017. The great time series classification bake off: a review and experimental evaluation of recent algorithmic advances. Data Mining and Knowledge Discovery 31, 3 (may 2017), 606–660. https://doi.org/10.1007/s10618-016-0483-9
[3]
Tadas Baltrusaitis, Amir Zadeh, Yao Chong Lim, and Louis-Philippe Morency. 2018. OpenFace 2.0: Facial Behavior Analysis Toolkit. In 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018). IEEE, 59–66. https://doi.org/10.1109/FG.2018.00019
[4]
Krystian Barzykowski and Agnieszka Niedźwieńska. 2018. Involuntary autobiographical memories are relatively more often reported during high cognitive load tasks. Acta Psychologica 182 (jan 2018), 119–128. https://doi.org/10.1016/j.actpsy.2017.11.014
[5]
Dorthe Berntsen, Søren Risløv Staugaard, and Louise Maria Torp Sørensen. 2013. Why am I remembering this now? Predicting the occurrence of involuntary (spontaneous) episodic memories.Journal of Experimental Psychology: General 142, 2 (2013), 426–444. https://doi.org/10.1037/a0029128
[6]
Robert Bixler and Sidney D’Mello. 2014. Toward Fully Automated Person-Independent Detection of Mind Wandering. In User Modeling, Adaptation, and Personalization, Vania Dimitrova, Tsvi Kuflik, David Chin, Francesco Ricci, Peter Dolog, and Geert-Jan Houben (Eds.). Springer International Publishing, Cham, 37–48. https://doi.org/10.1007/978-3-319-08786-3_4
[7]
Nigel Bosch and Sidney K. D’Mello. 2021. Automatic Detection of Mind Wandering from Video in the Lab and in the Classroom. IEEE Transactions on Affective Computing 12, 4 (oct 2021), 974–988. https://doi.org/10.1109/TAFFC.2019.2908837
[8]
Kalina Christoff, Zachary C. Irving, Kieran C. R. Fox, R. Nathan Spreng, and Jessica R. Andrews-Hanna. 2016. Mind-wandering as spontaneous thought: a dynamic framework. Nature Reviews Neuroscience 17, 11 (nov 2016), 718–731. https://doi.org/10.1038/nrn.2016.113
[9]
Dan Cosley, Victoria Schwanda, S Tejaswi Peesapaty, Johnathon Schultz, and Jonathan Baxter. 2009. Experiences with a publicly deployed tool for reminiscing. In Proc. First Int’l Workshop on Reminiscence Systems. 31–36.
[10]
Angus Dempster, François Petitjean, and Geoffrey I. Webb. 2020. ROCKET: exceptionally fast and accurate time series classification using random convolutional kernels. Data Mining and Knowledge Discovery 34, 5 (sep 2020), 1454–1495. https://doi.org/10.1007/s10618-020-00701-z arxiv:1910.13051
[11]
Angus Dempster, Daniel F. Schmidt, and Geoffrey I. Webb. 2021. MiniRocket: A Very Fast (Almost) Deterministic Transform for Time Series Classification. Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (2021), 248–257. https://doi.org/10.1145/3447548.3467231 arxiv:2012.08791
[12]
Sidney K. D’mello and Jacqueline Kory. 2015. A Review and Meta-Analysis of Multimodal Affect Detection Systems. Comput. Surveys 47, 3 (apr 2015), 1–36. https://doi.org/10.1145/2682899
[13]
Bernd Dudzik, Simon Columbus, Tiffany Matej Hrkalovic, Daniel Balliet, and Hayley Hung. 2021. Recognizing Perceived Interdependence in Face-to-Face Negotiations through Multimodal Analysis of Nonverbal Behavior. In Proceedings of the 2021 International Conference on Multimodal Interaction. ACM, New York, NY, USA, 121–130. https://doi.org/10.1145/3462244.3479935
[14]
Bernd Dudzik, Hayley Hung, Mark Neerincx, and Joost Broekens. 2018. Artificial Empathic Memory. In Proceedings of the 2018 Workshop on Understanding Subjective Attributes of Data, with the Focus on Evoked Emotions - EE-USAD’18. ACM Press, New York, New York, USA, 1–8. https://doi.org/10.1145/3267799.3267801
[15]
Bernd Dudzik, Hayley Hung, Mark Neerincx, and Joost Broekens. 2020. Investigating the Influence of Personal Memories on Video-Induced Emotions. In Proceedings of the 28th ACM Conference on User Modeling, Adaptation and Personalization. ACM, New York, NY, USA, 53–61. https://doi.org/10.1145/3340631.3394842
[16]
Bernd Dudzik, Hayley Hung, Mark A. Neerincx, and Joost Broekens. 2021. Collecting Mementos: A Multimodal Dataset for Context-Sensitive Modeling of Affect and Memory Processing in Responses to Videos. IEEE Transactions on Affective Computing 3045, c (2021), 1–18. https://doi.org/10.1109/TAFFC.2021.3089584
[17]
Mohamad El Haj, Pascal Antoine, and Jean Louis Nandrino. 2016. More emotional facial expressions during episodic than during semantic autobiographical retrieval. Cognitive, Affective, & Behavioral Neuroscience 16, 2 (apr 2016), 374–381. https://doi.org/10.3758/s13415-015-0397-9
[18]
Myrthe Faber and Sidney K. D’Mello. 2018. How the stimulus influences mind wandering in semantically rich task contexts. Cognitive Research: Principles and Implications 3, 1 (dec 2018), 35. https://doi.org/10.1186/s41235-018-0129-0
[19]
James J. Gross and Robert W. Levenson. 1995. Emotion elicitation using films. Cognition & Emotion 9, 1 (1995), 87–108. https://doi.org/10.1080/02699939508408966
[20]
Kunal Gupta, Sam W. T. Chan, Yun Suen Pai, Nicholas Strachan, John Su, Alexander Sumich, Suranga Nanayakkara, and Mark Billinghurst. 2022. Total VREcall. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 6 (7 2022), 1–21. Issue 2. https://doi.org/10.1145/3534615
[21]
Jacek Gwizdka. 2019. Exploring Eye-Tracking Data for Detection of Mind-Wandering on Web Tasks. In Lecture Notes in Information Systems and Organisation. Vol. 29. Springer, Cham, 47–55. https://doi.org/10.1007/978-3-030-01087-4_6
[22]
Michael Xuelin Huang, Jiajia Li, Grace Ngai, Hong Va Leong, and Andreas Bulling. 2019. Moment-to-Moment Detection of Internal Thought during Video Viewing from Eye Vergence Behavior. In Proceedings of the 27th ACM International Conference on Multimedia. ACM, New York, NY, USA, 2254–2262. https://doi.org/10.1145/3343031.3350573 arxiv:1901.06572
[23]
S. Koelstra, C. Muhl, M. Soleymani, Jong-Seok Lee, A. Yazdani, T. Ebrahimi, T. Pun, A. Nijholt, and I. Patras. 2012. DEAP: A Database for Emotion Analysis ;Using Physiological Signals. IEEE Transactions on Affective Computing 3, 1 (jan 2012), 18–31. https://doi.org/10.1109/T-AFFC.2011.15
[24]
Blerina Lika, Kostas Kolomvatsos, and Stathes Hadjiefthymiades. 2014. Facing the cold start problem in recommender systems. Expert Systems with Applications 41, 4 (mar 2014), 2065–2073. https://doi.org/10.1016/j.eswa.2013.09.005
[25]
Mei Yii Lim. 2012. Memory Models for Intelligent Social Companions. In Studies in Computational Intelligence. Vol. 396. 241–262. https://doi.org/10.1007/978-3-642-25691-2_10
[26]
Daniel G. McDonald, Melanie A. Sarge, Shu-Fang Lin, James G. Collier, and Bridget Potocki. 2015. A Role for the Self: Media Content as Triggers for Involuntary Autobiographical Memories. Communication Research 42, 1 (feb 2015), 3–29. https://doi.org/10.1177/0093650212464771
[27]
J. O’Rourke, F. Tobin, S. O’Callaghan, R. Sowman, and D. Collins. 2011. ’YouTube’: a useful tool for reminiscence therapy in dementia?Age and Ageing 40, 6 (nov 2011), 742–744. https://doi.org/10.1093/ageing/afr100
[28]
Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, and Others. 2011. Scikit-learn: Machine learning in Python. Journal of machine learning research 12, Oct (2011), 2825–2830.
[29]
Marieke M.M. Peeters, Maaike Harbers, and Mark A Neerincx. 2016. Designing a personal music assistant that enhances the social, cognitive, and affective experiences of people with dementia. Computers in Human Behavior 63 (oct 2016), 727–737. https://doi.org/10.1016/j.chb.2016.06.003
[30]
Felix Putze, Dennis Küster, Sonja Annerer-Walcher, and Mathias Benedek. 2018. Dozing Off or Thinking Hard?. In Proceedings of the 2018 on International Conference on Multimodal Interaction - ICMI ’18. ACM Press, New York, New York, USA, 258–262. https://doi.org/10.1145/3242969.3243000
[31]
Felix Putze, Maximilian Scherer, and Tanja Schultz. 2016. Starring into the void? Classifying Internal vs. External Attention from EEG. In Proceedings of the 9th Nordic Conference on Human-Computer Interaction, Vol. 23-27-Octo. ACM, New York, NY, USA, 1–4. https://doi.org/10.1145/2971485.2971555
[32]
Tali Sharot, Matthew L. Davidson, Meredith M. Carson, and Elizabeth A. Phelps. 2008. Eye Movements Predict Recollective Experience. PLoS ONE 3, 8 (aug 2008), e2884. https://doi.org/10.1371/journal.pone.0002884
[33]
Mohammad Soleymani and Maja Pantic. 2012. Human-centered implicit tagging: Overview and perspectives. 2012 IEEE International Conference on Systems, Man, and Cybernetics (SMC), 3304–3309. https://doi.org/10.1109/ICSMC.2012.6378301
[34]
Victoria Schwanda Sosik and Dan Cosley. 2014. Leveraging social media content to support engagement in positive interventions. The Journal of Positive Psychology 9 (9 2014), 428–434. Issue 5. https://doi.org/10.1080/17439760.2014.910826
[35]
Angela Stewart, Nigel Bosch, Huili Chen, Patrick Donnelly, and Sidney D’Mello. 2017. Face Forward: Detecting Mind Wandering from Video During Narrative Film Comprehension. In Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). Vol. 10331 LNAI. Springer, Cham, 359–370. https://doi.org/10.1007/978-3-319-61425-0_30
[36]
Doménique van Gennip, Elise van den Hoven, and Panos Markopoulos. 2015. Things That Make Us Reminisce. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems - CHI ’15, Vol. 1. ACM Press, New York, New York, USA, 3443–3452. https://doi.org/10.1145/2702123.2702460
[37]
Manila Vannucci, Claudia Pelagatti, Maciej Hanczakowski, Giuliana Mazzoni, and Claudia Rossi Paccani. 2015. Why are we not flooded by involuntary autobiographical memories? Few cues are more effective than many. Psychological Research 79, 6 (nov 2015), 1077–1085. https://doi.org/10.1007/s00426-014-0632-y
[38]
Yuyu Zhang and Takatsune Kumada. 2018. Automatic detection of mind wandering in a simulated driving task with behavioral measures. PLOS ONE 13, 11 (nov 2018), e0207092. https://doi.org/10.1371/journal.pone.0207092
[39]
Yue Zhao, Christoph Lofi, and Claudia Hauff. 2017. Scalable mind-wandering detection for MOOCs: A webcam-based approach. In Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), Vol. 10474 LNCS. Springer, Cham, 330–344. https://doi.org/10.1007/978-3-319-66610-5_24
Index Terms
- Exploring the Detection of Spontaneous Recollections during Video-viewing In-the-Wild using Facial Behavior Analysis
Recommendations
Facial Landmark Detection and Tracking for Facial Behavior Analysis
ICMR '16: Proceedings of the 2016 ACM on International Conference on Multimedia RetrievalThe face is the most dominant and distinct communication tool of human beings. Automatic analysis of facial behavior allows machines to understand and interpret a human's states and needs for natural interactions. This research focuses on developing ...
Spontaneous vs. posed facial behavior: automatic analysis of brow actions
ICMI '06: Proceedings of the 8th international conference on Multimodal interfacesPast research on automatic facial expression analysis has focused mostly on the recognition of prototypic expressions of discrete emotions rather than on the analysis of dynamic changes over time, although the importance of temporal dynamics of facial ...
Mindfulness Is in the Eye of the Machine
Human and Artificial RationalitiesAbstractMindfulness can be defined relative to mind-wandering, while the former is associated with focused attention and enhanced monitoring of conscious experience, the latter corresponds to a process of distraction. The aim of this human-machine ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
November 2022
830 pages
ISBN:9781450393904
DOI:10.1145/3536221
Copyright © 2022 Owner/Author.
This work is licensed under a Creative Commons Attribution International 4.0 License.
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 07 November 2022
Check for updates
Author Tags
Qualifiers
- Research-article
- Research
- Refereed limited
Funding Sources
Conference
ICMI '22
Sponsor:
Acceptance Rates
Overall Acceptance Rate 453 of 1,080 submissions, 42%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 270Total Downloads
- Downloads (Last 12 months)151
- Downloads (Last 6 weeks)11
Reflects downloads up to 12 Dec 2024
Other Metrics
Citations
View Options
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderHTML Format
View this article in HTML Format.
HTML FormatLogin options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in