[go: up one dir, main page]
More Web Proxy on the site http://driver.im/ skip to main content
10.1145/3382507.3418856acmconferencesArticle/Chapter ViewAbstractPublication Pagesicmi-mlmiConference Proceedingsconference-collections
research-article

Towards Engagement Recognition of People with Dementia in Care Settings

Published: 22 October 2020 Publication History

Abstract

Roughly 50 million people worldwide are currently suffering from dementia. This number is expected to triple by 2050. Dementia is characterized by a loss of cognitive function and changes in behaviour. This includes memory, language skills, and the ability to focus and pay attention. However, it has been shown that secondary therapy such as the physical, social and cognitive activation of People with Dementia (PwD) has significant positive effects. Activation impacts cognitive functioning and can help prevent the magnification of apathy, boredom, depression, and loneliness associated with dementia. Furthermore, activation can lead to higher perceived quality of life. We follow Cohen's argument that activation stimuli have to produce engagement to take effect and adopt his definition of engagement as "the act of being occupied or involved with an external stimulus".

Supplementary Material

MP4 File (3382507.3418856.mp4)
The number of People with Dementia (PwD) is expected to triple by 2050 - and there is no medical cure in sight. The cornerstones of effective secondary therapy are physical, social, and cognitive activation, which require engagement. Whereas engagement has been exhaustively investigated in HCI contexts, little is known about the potential of visual cues for the automatic recognition of engagement of PwD. This task is especially challenging due to disease-related changes of facial expressions but also age-related changes of the face. Here, we show that emotional engagement can be automatically recognized based on visual and contextual information derived from interactions with the I-CARE activation system in an unconstrained care setting. Moreover, we show that model uncertainty can be utilized to identify likely errors of a classifier.

References

[1]
Mehmet Berkehan Akçay and Kaya O'uz. 2020. Speech emotion recognition: Emotional models, databases, features, preprocessing methods, supporting modalities, and classifiers. Speech Communication 116 (2020), 56 -- 76. https://doi.org/10.1016/j.specom.2019.12.001
[2]
S. M. Alarcão. 2017. Reminiscence therapy improvement using emotional information. In 2017 Seventh International Conference on Affective Computing and Intelligent Interaction (ACII). 561--565.
[3]
Kenneth Asplund, Astrid Norberg, Rolf Adolfsson, and Howard M. Waxman. 1991. Facial expressions in severely demented patients'a stimulus--response study of four patients with dementia of the Alzheimer type. International Journal of Geriatric Psychiatry 6, 8 (1991), 599--606. https://doi.org/10.1002/gps.930060809 arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1002/gps.930060809
[4]
Arlene J Astell, Maggie P Ellis, Lauren Bernardi, Norman Alm, Richard Dye, Gary Gowans, and Jim Campbell. 2010. Using a touch screen computer to support relationships between people with dementia and caregivers. Interacting with Computers 22, 4 (2010), 267--275.
[5]
T. Baltrusaitis, A. Zadeh, Y. C. Lim, and L. Morency. 2018. OpenFace 2.0: Facial Behavior Analysis Toolkit. In 2018 13th IEEE International Conference on Automatic Face Gesture Recognition (FG 2018). 59--66.
[6]
Sarah Adel Bargal, Emad Barsoum, Cristian Canton Ferrer, and Cha Zhang. 2016. Emotion recognition in the wild from videos using images. In Proceedings of the 18th ACM International Conference on Multimodal Interaction. 433--436.
[7]
Hanneke Beerens, Bram de Boer, Sandra Zwakhalen, Frans Tan, Dirk Ruwaard, Jan Hamers, and Hilde Verbeek. 2016. The association between aspects of daily life and quality of life of people with dementia living in long-term care facilities: a momentary assessment study. International Psychogeriatrics -1 (04 2016). https://doi.org/10.1017/S1041610216000466
[8]
Jiska Cohen-Mansfield. 2018. The impact of group activities and their content on persons with dementia attending them. Alzheimer's research & therapy 10, 1 (2018), 37.
[9]
Jiska Cohen-Mansfield, Maha Dakheel-Ali, and Marcia S Marx. 2009. Engagement in persons with dementia: the concept and its measurement. The American journal of geriatric psychiatry 17, 4 (2009), 299--307.
[10]
Jiska Cohen-Mansfield, Marcia S Marx, Khin Thein, and Maha Dakheel-Ali. 2011. The impact of stimuli on affect in persons with dementia. The Journal of clinical psychiatry 72, 4 (2011), 480.
[11]
Pasquale Dente, Dennis Küster, Lina Skora, and E. Krumhuber. 2017. Measures and metrics for automatic emotion classification via FACET. In Proceedings of the Conference on the Study of Artificial Intelligence and Simulation of Behaviour (AISB). 160--163.
[12]
Julia Eggert, Cheryl J Dye, Ellen Vincent, Veronica Parker, Shaundra B Daily, Hiep Pham, Alison Turner Watson, Hollie Summey, and Tania Roy. 2015. Effects of viewing a preferred nature image and hearing preferred music on engagement, agitation, and mental status in persons with dementia. SAGE Open Medicine 3 (2015), 2050312115602579. https://doi.org/10.1177/2050312115602579 arXiv: https://doi.org/10.1177/2050312115602579 26770801.
[13]
Yingruo Fan, Jacqueline C. K. Lam, and Victor O. K. Li. 2018. Video-Based Emotion Recognition Using Deeply-Supervised Neural Networks. In Proceedings of the 20th ACM International Conference on Multimodal Interaction (Boulder, CO, USA) (ICMI '18). Association for Computing Machinery, New York, NY, USA, 584--588. https://doi.org/10.1145/3242969.3264978
[14]
Maxi Freudenberg, Reginald B. Adams, Robert E. Kleck, and Ursula Hess. 2015. Through a glass darkly: facial wrinkles affect our processing of emotion in the elderly. Frontiers in Psychology 6 (2015). https://doi.org/10.3389/fpsyg.2015.01476
[15]
Yarin Gal and Zoubin Ghahramani. 2016. Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning. In Proceedings of the 33rd International Conference on International Conference on Machine Learning - Volume 48 (New York, NY, USA) (ICML'16). JMLR.org, 1050--1059.
[16]
Cindy Jones, Billy Sung, and Wendy Moyle. 2015. Assessing engagement in people with dementia: a new approach to assessment using video analysis. Archives of psychiatric nursing 29, 6 (2015), 377--382.
[17]
Cindy Jones, Billy Sung, and Wendy Moyle. 2018. Engagement of a Person with Dementia Scale: Establishing content validity and psychometric properties. Journal of advanced nursing 74, 9 (2018), 2227--2240.
[18]
Arvid Kappas, Eva Krumhuber, and Dennis Küster. 2013. Facial behavior. In In: Hall, Judith A.; Knapp, Mark L. (Ed.), Nonverbal communication (S. 131--166). Berlin: de Gruyter, 2013. de Gruyter, 131--166.
[19]
A. Kaur, A. Mustafa, L. Mehta, and A. Dhall. 2018. Prediction and Localization of Student Engagement in the Wild. In 2018 Digital Image Computing: Techniques and Applications (DICTA). 1--8. https://doi.org/10.1109/DICTA.2018.8615851
[20]
Heysem Kaya, Furkan Gürpnar, and Albert Ali Salah. 2017. Video-based emotion recognition in the wild using deep transfer learning and score fusion. Image and Vision Computing 65 (2017), 66--75.
[21]
Boris Knyazev, Roman Shvetsov, Natalia Efremova, and Artem Kuharenko. 2017. Convolutional neural networks pretrained on large face recognition datasets for emotion classification from video. (2017). arXiv:cs.CV/1711.04598
[22]
Eva Krumhuber, Antony S. R. Manstead, and Arvid Kappas. 2007. Temporal Aspects of Facial Displays in Person and Expression Perception: The Effects of Smile Dynamics, Head-tilt, and Gender. Journal of Nonverbal Behavior 31, 1 (March 2007), 39--56. https://doi.org/10.1007/s10919-006-0019-x
[23]
M Powell Lawton, Kimberly Van Haitsma, and Jennifer Klapper. 1996. Observed affect in nursing home residents with Alzheimer's disease. The Journals of Gerontology Series B: Psychological Sciences and Social Sciences 51, 1 (1996), P3-- P14.
[24]
Chuanhe Liu, Tianhao Tang, Kui Lv, and Minghao Wang. 2018. Multi-Feature Based Emotion Recognition for Video Clips. In Proceedings of the 20th ACM International Conference on Multimodal Interaction (Boulder, CO, USA) (ICMI '18). Association for Computing Machinery, New York, NY, USA, 630--634. https://doi.org/10.1145/3242969.3264989
[25]
Kaixin Ma, Xinyu Wang, Xinru Yang, Mingtong Zhang, Jeffrey M Girard, and Louis-Philippe Morency. 2019. ElderReact: A Multimodal Dataset for Recognizing Emotional Response in Aging Adults. In 2019 International Conference on Multimodal Interaction (Suzhou, China) (ICMI '19). Association for Computing Machinery, New York, NY, USA, 349--357. https://doi.org/10.1145/3340555.3353747
[26]
Carol Magai, Carl Cohen, David Gomberg, Chris Malatesta, and Clayton Culver. 1996. Emotional expression during mid-to late-stage dementia. International psychogeriatrics 8, 3 (1996), 383--395.
[27]
Margaret McAllister, Jeanne Dayton, Florin Oprescu, Mary Katsikitis, and Christian M Jones. 0. Memory Keeper: A prototype digital application to improve engagement with people with dementia in long-term care (innovative practice). Dementia 0, 0 (0), 1471301217737872. https://doi.org/10.1177/1471301217737872 arXiv: https://doi.org/10.1177/1471301217737872 29096546.
[28]
Wendy Moyle, Cindy J Jones, Jenny E Murfield, Lukman Thalib, Elizabeth RA Beattie, David KH Shum, Siobhan T O'Dwyer, M Cindy Mervin, and Brian M Draper. 2017. Use of a robotic seal as a therapeutic tool to improve dementia symptoms: a cluster-randomized controlled trial. Journal of the American Medical Directors Association 18, 9 (2017), 766--773.
[29]
Xuesong Niu, Hu Han, Jiabei Zeng, Xuran Sun, Shiguang Shan, Yan Huang, Songfan Yang, and Xilin Chen. 2018. Automatic Engagement Prediction with GAP Feature. In Proceedings of the 20th ACM International Conference on Multimodal Interaction (Boulder, CO, USA) (ICMI '18). ACM, New York, NY, USA, 599--603. https://doi.org/10.1145/3242969.3264982
[30]
Markus Ojala and Gemma C Garriga. 2010. Permutation tests for studying classifier performance. Journal of Machine Learning Research 11, Jun (2010), 1833--1863.
[31]
Richard V Olsen, B Lynn Hutchings, and Ezra Ehrenkrantz. 2000. 'Media Memory Lane' interventions in an Alzheimer's day care center. American Journal of Alzheimer's Disease 15, 3 (2000), 163--175.
[32]
Juri Opitz and Sebastian Burst. 2019. Macro F1 and Macro F1. arXiv:cs.LG/1911.03347
[33]
Viral Parekh, Pin Foong, Shengdong Zhao, and Ramanathan Subramanian. 2018. AVEID: Automatic Video System for Measuring Engagement In Dementia. 409-- 413. https://doi.org/10.1145/3172944.3173010
[34]
Omkar M. Parkhi, Andrea Vedaldi, and Andrew Zisserman. 2015. Deep Face Recognition., Article 41 (September 2015), 12 pages. https://doi.org/10.5244/C. 29.41
[35]
G. Perugia, M. Díaz-Boladeras, A. Català-Mallofré, E. I. Barakova, and M. Rauterberg. 2020. ENGAGE-DEM: A Model of Engagement of People with Dementia. IEEE Transactions on Affective Computing (2020), 1--1. https://doi.org/10.1109/ TAFFC.2020.2980275
[36]
Giulia Perugia, Daniel Rodríguez-Martín, Marta Díaz Boladeras, Andreu Català Mallofré, Emilia Barakova, and Matthias Rauterberg. 2017. Electrodermal activity: explorations in the psychophysiology of engagement with social robots in dementia. In 2017 26th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN). IEEE, 1248--1254.
[37]
Giulia Perugia, Roos van Berkel, Marta Díaz-Boladeras, Andreu Català-Mallofré, Matthias Rauterberg, and Emilia Barakova. 2018. Understanding Engagement in Dementia Through Behavior. The Ethographic and Laban-Inspired Coding System of Engagement (ELICSE) and the Evidence-Based Model of EngagementRelated Behavior (EMODEB). Frontiers in psychology 9 (2018), 690. https://doi.org/10.3389/fpsyg.2018.00690
[38]
M. Powell Lawton, K. Van Haitsma, M. Perkinson, and K. Ruckdeschel. 1999. Observed affect and quality of life in dementia: Further affirmations and problems. Journal of Mental Health and Aging 5, 1 (29 May 1999), 69--81.
[39]
Hans Ragneskog, Kenneth Asplund, Mona Kihlgren, and Astrid Norberg. 2001. Individualized music played for agitated patients with dementia: analysis of video-recorded sessions. International journal of nursing practice 7, 3 (2001), 146--155.
[40]
Susanna Re. 2003. Emotionales Ausdrucksverhalten bei schweren demenziellen Erkrankungen. Zeitschrift für Gerontologie und Geriatrie 36, 6 (2003), 447--453.
[41]
Karen Schmidt, Zara Ambadar, Jeffrey Cohn, and Lawrence Reed. 2006. Movement Differences between Deliberate and Spontaneous Facial Expressions: Zygomaticus Major Action in Smiling. Journal of nonverbal behavior 30 (02 2006), 37--52. https://doi.org/10.1007/s10919-005-0003-x
[42]
Karen L. Schmidt and Jeffrey F. Cohn. [n.d.]. Human facial expressions as adaptations: Evolutionary questions in facial expression research. 116 ([n. d.]), 3--24. Issue S33. https://doi.org/10.1002/ajpa.20001
[43]
Andrea S Schreiner, Eiko Yamamoto, and Hisako Shiotani. 2005. Positive affect among nursing home residents with Alzheimer's dementia: the effect of recreational activity. Aging & mental health 9, 2 (2005), 129--134.
[44]
Tanja Schultz, Felix Putze, Timo Schulze, Lars Steinert, Ralf Mikut, Wolfgang Doneit, Andreas Kruse, Anamaria Depner, Ingo Franz, Marc Engels, Philipp Gaerte, Sebastian Jünger, Rene Linden, Christof Ziegler, Michael Ricken, Todor Dimitrov, Joachim Herzig, Irene Maucher, Keni Bernardin, and Clarissa Simon. 2018. I-CARE - Ein Mensch-Technik Interaktionssystem zur Individuellen Aktivierung von Menschen mit Demenz.
[45]
Gary W Small, Peter V Rabins, Patricia P Barry, Neil S Buckholtz, Steven T DeKosky, Steven H Ferris, Sanford I Finkel, Lisa P Gwyther, Zaven S Khachaturian, Barry D Lebowitz, et al. 1997. Diagnosis and treatment of Alzheimer disease and related disorders: consensus statement of the American Association for Geriatric Psychiatry, the Alzheimer's Association, and the American Geriatrics Society. Jama 278, 16 (1997), 1363--1371.
[46]
Aimee Spector, Lene Thorgrimsen, BOB Woods, Lindsay Royan, Steve Davies, Margaret Butterworth, and Martin Orrell. 2003. Efficacy of an evidence-based cognitive stimulation therapy programme for people with dementia: randomised controlled trial. The British Journal of Psychiatry 183, 3 (2003), 248--254.
[47]
Chinchu Thomas, Nitin Nair, and Dinesh Babu Jayagopi. 2018. Predicting Engagement Intensity in the Wild Using Temporal Convolutional Network. In Proceedings of the 20th ACM International Conference on Multimodal Interaction (Boulder, CO, USA) (ICMI '18). ACM, New York, NY, USA, 604--610. https://doi.org/10.1145/3242969.3264984
[48]
Anthony J Viera, Joanne M Garrett, et al. 2005. Understanding interobserver agreement: the kappa statistic. Fam med 37, 5 (2005), 360--363.
[49]
K. Wada, Yousuke Ikeda, Kaoru Inoue, and Reona Uehara. 2010. Development and preliminary evaluation of a caregiver's manual for robot therapy using the therapeutic seal robot Paro. Proceedings - IEEE International Workshop on Robot and Human Interactive Communication, 533--538. https://doi.org/10.1109/ROMAN.2010.5598615
[50]
WHO. 2017. Dementia. https://www.who.int/news-room/fact-sheets/detail/ dementia. [Online; accessed 27-March-2019].
[51]
Bob Woods, Elisa Aguirre, Aimee E Spector, and Martin Orrell. 2012. Cognitive stimulation to improve cognitive functioning in people with dementia. Cochrane Database of Systematic Reviews 2 (2012).
[52]
Jianfei Yang, Kai Wang, Xiaojiang Peng, and Yu Qiao. 2018. Deep Recurrent Multi-instance Learning with Spatio-temporal Features for Engagement Intensity Prediction. In Proceedings of the 20th ACM International Conference on Multimodal Interaction (Boulder, CO, USA) (ICMI '18). ACM, New York, NY, USA, 594--598. https://doi.org/10.1145/3242969.3264981
[53]
Zhihong Zeng, Maja Pantic, Glenn I Roisman, and Thomas S Huang. 2009. A survey of affect recognition methods: Audio, visual, and spontaneous expressions. IEEE transactions on pattern analysis and machine intelligence 31, 1 (2009), 39--58.

Cited By

View all
  • (2024)Participation Role-Driven Engagement Estimation of ASD Individuals in Neurodiverse Group DiscussionsProceedings of the 26th International Conference on Multimodal Interaction10.1145/3678957.3685721(556-564)Online publication date: 4-Nov-2024
  • (2023)MultiPar-TProceedings of the Thirty-Second International Joint Conference on Artificial Intelligence10.24963/ijcai.2023/433(3893-3901)Online publication date: 19-Aug-2023
  • (2023)Teardrops on My Face: Automatic Weeping Detection From Nonverbal BehaviorIEEE Transactions on Affective Computing10.1109/TAFFC.2022.322874914:4(3001-3012)Online publication date: 1-Oct-2023
  • Show More Cited By

Index Terms

  1. Towards Engagement Recognition of People with Dementia in Care Settings

    Recommendations

    Comments

    Please enable JavaScript to view thecomments powered by Disqus.

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    ICMI '20: Proceedings of the 2020 International Conference on Multimodal Interaction
    October 2020
    920 pages
    ISBN:9781450375818
    DOI:10.1145/3382507
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 22 October 2020

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. dementia
    2. emotion
    3. engagement
    4. facial expressions
    5. lstm

    Qualifiers

    • Research-article

    Conference

    ICMI '20
    Sponsor:
    ICMI '20: INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION
    October 25 - 29, 2020
    Virtual Event, Netherlands

    Acceptance Rates

    Overall Acceptance Rate 453 of 1,080 submissions, 42%

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)49
    • Downloads (Last 6 weeks)6
    Reflects downloads up to 15 Jan 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2024)Participation Role-Driven Engagement Estimation of ASD Individuals in Neurodiverse Group DiscussionsProceedings of the 26th International Conference on Multimodal Interaction10.1145/3678957.3685721(556-564)Online publication date: 4-Nov-2024
    • (2023)MultiPar-TProceedings of the Thirty-Second International Joint Conference on Artificial Intelligence10.24963/ijcai.2023/433(3893-3901)Online publication date: 19-Aug-2023
    • (2023)Teardrops on My Face: Automatic Weeping Detection From Nonverbal BehaviorIEEE Transactions on Affective Computing10.1109/TAFFC.2022.322874914:4(3001-3012)Online publication date: 1-Oct-2023
    • (2023)Künstliche Intelligenz und Ethik im Gesundheitswesen – Spagat oder Symbiose?Artificial intelligence and ethics in healthcare—balancing act or symbiosis?Bundesgesundheitsblatt - Gesundheitsforschung - Gesundheitsschutz10.1007/s00103-022-03653-566:2(176-183)Online publication date: 17-Jan-2023
    • (2022)Sustaining Our Relationship: Dyadic Interactions Supported by Technology for People with Dementia and Their Informal CaregiversInternational Journal of Environmental Research and Public Health10.3390/ijerph19171095619:17(10956)Online publication date: 2-Sep-2022
    • (2022)Predicting Activation Liking of People With DementiaFrontiers in Computer Science10.3389/fcomp.2021.7704923Online publication date: 7-Jan-2022
    • (2022)Evaluation of an Engagement-Aware Recommender System for People with DementiaProceedings of the 30th ACM Conference on User Modeling, Adaptation and Personalization10.1145/3503252.3531318(89-98)Online publication date: 4-Jul-2022
    • (2022)Engagement estimation of the elderly from wild multiparty human–robot interactionComputer Animation and Virtual Worlds10.1002/cav.212033:6Online publication date: 18-Aug-2022
    • (2021)I-CARE-An Interaction System for the Individual Activation of People with DementiaGeriatrics10.3390/geriatrics60200516:2(51)Online publication date: 13-May-2021
    • (2021)A Multimodal Engagement-Aware Recommender System for People with Dementia2021 9th International Conference on Affective Computing and Intelligent Interaction Workshops and Demos (ACIIW)10.1109/ACIIW52867.2021.9666306(1-5)Online publication date: 28-Sep-2021

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Media

    Figures

    Other

    Tables

    Share

    Share

    Share this Publication link

    Share on social media