More Web Proxy on the site http://driver.im/

research-article

Multimodal approach for cognitive task performance prediction from body postures, facial expressions and EEG signal

Authors:

Ashwin Ramesh Babu,

Akilesh Rajavenkatanarayanan,

James Robert Brady,

Fillia MakedonAuthors Info & Claims

MCPMD '18: Proceedings of the Workshop on Modeling Cognitive Processes from Multimodal Data

Article No.: 14, Pages 1 - 7

https://doi.org/10.1145/3279810.3279849

Published: 16 October 2018 Publication History

Abstract

Recent developments in computer vision and the emergence of wearable sensors have opened opportunities for the development of advanced and sophisticated techniques to enable multi-modal user assessment and personalized training which is important in educational, industrial training and rehabilitation applications. They have also paved way for the use of assistive robots to accurately assess human cognitive and physical skills. Assessment and training cannot be generalized as the requirement varies for every person and for every application. The ability of the system to adapt to the individual's needs and performance is essential for its effectiveness. In this paper, the focus is on task performance prediction which is an important parameter to consider for personalization. Several research works focus on how to predict task performance based on physiological and behavioral data. In this work, we follow a multi-modal approach where the system collects information from different modalities to predict performance based on (a) User's emotional state recognized from facial expressions(Behavioral data), (b) User's emotional state from body postures(Behavioral data) (c) task performance from EEG signals (Physiological data) while the person performs a robot-based cognitive task. This multi-modal approach of combining physiological data and behavioral data produces the highest accuracy of 87.5 percent, which outperforms the accuracy of prediction extracted from any single modality. In particular, this approach is useful in finding associations between facial expressions, body postures and brain signals while a person performs a cognitive task.

References

[1]

Maher Abujelala, Cheryl Abellanoza, Aayush Sharma, and Fillia Makedon. 2016. Brain-ee: Brain enjoyment evaluation using commercial eeg headband. In Proceedings of the 9th ACM International Conference on PErvasive Technologies Related to Assistive Environments. ACM, 33.

Digital Library

[2]

Octavio Arriaga, Matias Valdenegro-Toro, and Paul Plöger. 2017. Real-time Convolutional Neural Networks for Emotion and Gender Classification. arXiv preprint arXiv:1710.07557 (2017).

[3]

Ashwin Ramesh Babu, Akilesh Rajavenkatanarayanan, Maher Abujelala, and Fillia Makedon. 2017. Votre: A vocational training and evaluation system to compare training approaches for the workplace. In International Conference on Virtual, Augmented and Mixed Reality. Springer, 203--214.

[4]

Maria Bannert. 2002. Managing cognitive load - recent trends in cognitive load theory. Learning and instruction 12, 1 (2002), 139--146.

[5]

Chris Berka, Daniel J Levendowski, Michelle N Lumicao, Alan Yau, Gene Davis, Vladimir T Zivkovic, Richard E Olmstead, Patrice D Tremoulet, and Patrick L Craven. 2007. EEG correlates of task engagement and mental workload in vigilance, learning, and memory tasks. Aviation, space, and environmental medicine 78, 5 (2007), B231--B244.

[6]

Nadia Bianchi-Berthouze, Paul Cairns, Anna Cox, Charlene Jennett, and Whan Woong Kim. 2006. On posture as a modality for expressing and recognizing emotions. In Emotion and HCI workshop at BCS HCI London.

[7]

KS Blair, BW Smith, DGV Mitchell, J Morton, M Vythilingam, L Pessoa, D Fridberg, A Zametkin, EE Nelson, WC Drevets, et al. 2007. Modulation of emotion by cognition and cognition by emotion. Neuroimage 35, 1 (2007), 430--440.

[8]

Carlos Busso, Zhigang Deng, Serdar Yildirim, Murtaza Bulut, Chul Min Lee, Abe Kazemzadeh, Sungbok Lee, Ulrich Neumann, and Shrikanth Narayanan. 2004. Analysis of emotion recognition using facial expressions, speech and multimodal information. In Proceedings of the 6th international conference on Multimodal interfaces. ACM, 205--211.

Digital Library

[9]

Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh. 2016. Realtime multi-person 2d pose estimation using part affinity fields. arXiv preprint arXiv:1611.08050 (2016).

[10]

Ginevra Castellano, Santiago D Villalba, and Antonio Camurri. 2007. Recognising human emotions from body movement and gesture dynamics. In International Conference on Affective Computing and Intelligent Interaction. Springer, 71--82.

Digital Library

[11]

Marco De Meijer.1989. The contribution of general features of body movement to the attribution of emotions. Journal of Nonverbal behavior 13, 4 (1989), 247--268.

[12]

Paul Ekman. 1999. Basic emotions. Handbook of cognition and emotion (1999), 45--60.

[13]

Mark J Fenske and John D Eastwood. 2003. Modulation of focused attention by faces expressing emotion: evidence from flanker tasks. Emotion 3, 4 (2003), 327.

[14]

Nesrine Fourati and Catherine Pelachaud. 2014. Emilya: Emotional body expression in daily actions database. In LREC. 3486--3493.

[15]

Kristin Fraser, Irene Ma, Elise Teteris, Heather Baxter, Bruce Wright, and Kevin McLaughlin. 2012. Emotion, cognitive load and learning outcomes during simulation training. Medical education 46, 11 (2012), 1055--1062.

[16]

Barbara L Fredrickson and Christine Branigan. 2005. Positive emotions broaden the scope of attention and thought-action repertoires. Cognition & emotion 19, 3 (2005), 313--332.

[17]

Susan E Gathercole and Alan D Baddeley. 2014. Working memory and language. Psychology Press.

[18]

Xavier Glorot and Yoshua Bengio. 2010. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics. 249--256.

[19]

M Melissa Gross, Elizabeth A Crane, and Barbara L Fredrickson. 2010. Methodology for assessing bodily expression of emotion. Journal of Nonverbal Behavior 34, 4 (2010), 223--248.

[20]

Stephen Grossberg and Lance R Pearson. 2008. Laminar cortical dynamics of cognitive and motor working memory, sequence learning and performance: toward a unified theory of how the cerebral cortex works. Psychological review 115, 3 (2008), 677.

[21]

Martijn Haak, Steven Bos, Sacha Panic, and LJM Rothkrantz. 2009. Detecting stress using eye blinks and brain activity from EEG signals. Proceeding of the 1st driver car interaction and interface (DCII 2008) (2009), 35--60.

[22]

Larry E Humes and Shari S Floyd. 2005. Measures of working memory, sequence learning, and speech recognition in the elderly. Journal of Speech, Language, and Hearing Research 48, 1 (2005), 224--235.

[23]

Wolfgang Klimesch. 1999. EEG alpha and theta oscillations reflect cognitive and memory performance: a review and analysis. Brain research reviews 29, 2--3 (1999), 169--195.

[24]

Wolfgang Klimesch, Michael Doppelmayr, H Russegger, Th Pachinger, and J Schwaiger. 1998. Induced alpha band power changes in the human EEG and attention. Neuroscience letters 244, 2 (1998), 73--76.

[25]

Mu Li and Bao-Liang Lu. 2009. Emotion classification based on gamma-band EEG. In Engineering in Medicine and Biology Society, 2009. EMBC 2009. Annual International Conference of the IEEE. IEEE, 1223--1226.

[26]

Elizabeth A Linnenbrink. 2007. The role of affect in student learning: A multi-dimensional approach to considering the interaction of affect, motivation, and engagement. In Emotion in education. Elsevier, 107--124.

[27]

Elizabeth A Linnenbrink and Paul R Pintrich. 2002. The role of motivational beliefs in conceptual change. In Reconsidering conceptual change: Issues in theory and practice. Springer, 115--135.

[28]

Elizabeth A Linnenbrink, Allison M Ryan, and Paul R Pintrich. 1999. The role of goals and affect in working memory functioning. Learning and Individual Differences 11, 2 (1999), 213--230.

[29]

Brent Morgan and Sidney DáĂ&Zacute;Mello. 2013. The Effect of Positive vs. Negative Emotion on Multitasking. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting, Vol. 57. SAGE Publications Sage CA: Los Angeles, CA, 848--852.

[30]

Muse. 2018. How Does MUSE- The Brain Sensing Headband Work. http://www.choosemuse.com/how-does-muse-work/

[31]

Ziad S Nasreddine, Natalie A Phillips, Valérie Bédirian, Simon Charbonneau, Victor Whitehead, Isabelle Collin, Jeffrey L Cummings, and Howard Chertkow. 2005. The Montreal Cognitive Assessment, MoCA: a brief screening tool for mild cognitive impairment. Journal of the American Geriatrics Society 53, 4 (2005), 695--699.

[32]

Michalis Papakostas, Konstantinos Tsiakas, Theodores Ginnakopoulos, and Fillia Makedon. {n. d.}. Towards Predicting Task Performance from EEG Signals. ({n. d.}).

[33]

Reinhard Pekrun. 1992. The impact of emotions on learning and achievement: Towards a theory of cognitive/motivational mediators. Applied Psychology 41, 4 (1992), 359--376.

[34]

Reinhard Pekrun. 2006. The control-value theory of achievement emotions: Assumptions, corollaries, and implications for educational research and practice. Educational psychology review 18, 4 (2006), 315--341.

[35]

Reinhard Pekrun and Lisa Linnenbrink-Garcia. 2012. Academic emotions and student engagement. In Handbook of research on student engagement. Springer, 259--282.

[36]

Luiz Pessoa. 2013. The cognitive-emotional brain: From interactions to integration. MIT press.

[37]

Akilesh Rajavenkatanarayanan, Ashwin Ramesh Babu, Konstantinos Tsiakas, and Fillia Makedon. 2018. Monitoring task engagement using facial expressions and body postures. In Proceedings of the 3rd International Workshop on Interactive and Spatial Computing. ACM, 103--108.

Digital Library

[38]

C Daniel Salzman and Stefano Fusi. 2010. Emotion, cognition, and mental state representation in amygdala and prefrontal cortex. Annual review of neuroscience 33 (2010), 173--202.

[39]

Jonathan F Schulz, Urs Fischbacher, Christian Thoni, and Verena Utikal. 2014. Affect and fairness: Dictator games under cognitive load. Journal of Economic Psychology 41 (2014), 77--87.

[40]

Jerritta Selvaraj, Murugappan Murugappan, Khairunizam Wan, and Sazali Yaacob. 2013. Classification of emotional states from electrocardiogram signals: a nonlinear approach based on hurst. Biomedical engineering online 12, 1 (2013), 44.

[41]

J Stastny, Pavel Sovka, and A Stancak. 2001. EEG signal classification. In Engineering in Medicine and Biology Society, 2001. Proceedings of the 23rd Annual International Conference of the IEEE, Vol. 2. IEEE, 2020--2023.

[42]

Paweł Tarnowski, Marcin Kołodziej, Andrzej Majkowski, and Remigiusz J Rak. 2017. Emotion recognition using facial expressions. Procedia Computer Science 108 (2017), 1175--1184.

[43]

Yelena Tonoyan, David Looney, Danilo P Mandic, and Marc M Van Hulle. 2016. Discriminating multiple emotional states from EEG using a data-adaptive, multi-scale information-theoretic approach. International journal of neural systems 26, 02 (2016), 1650005.

[44]

Konstantinos Tsiakas, Cheryl Abellanoza, Maher Abujelala, Michalis Papakostas, Tasnim Makada, and Fillia Makedon. {n. d.}. Towards Designing a Socially Assistive Robot for Adaptive and Personalized Cognitive Training. ({n. d.}).

[45]

Konstantinos Tsiakas, Maher Abujelala, Alexandros Lioulemes, and Fillia Makedon. 2016. An intelligent Interactive Learning and Adaptation framework for robot-based vocational training. In Computational Intelligence (SSCI), 2016 IEEE Symposium Series on. IEEE, 1--6.

[46]

Lotte F Van Dillen, Dirk J Heslenfeld, and Sander L Koole. 2009. Tuning down the emotional brain: an fMRI study of the effects of cognitive load on the processing of affective images. Neuroimage 45, 4 (2009), 1212--1219.

[47]

Jeroen JG Van Merrienboer and John Sweller. 2005. Cognitive load theory and complex learning: Recent developments and future directions. Educational psychology review 17, 2 (2005), 147--177.

[48]

Harald G Wallbott. 1998. Bodily expression of emotion. European journal of social psychology 28, 6 (1998), 879--896.

[49]

Shih-En Wei, Varun Ramakrishna, Takeo Kanade, and Yaser Sheikh. 2016. Convolutional pose machines. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 4724--4732.

[50]

Bernard Weiner. 1972. Theories of motivation: From mechanism to cognition. (1972).

Cited By

Nasihati Gilani SPollard KTraum D(2023)Multimodal Prediction of User's Performance in High-Stress Dialogue InteractionsCompanion Publication of the 25th International Conference on Multimodal Interaction10.1145/3610661.3617166(71-75)Online publication date: 9-Oct-2023
https://dl.acm.org/doi/10.1145/3610661.3617166
Kosch TKarolus JZagermann JReiterer HSchmidt AWoźniak P(2023)A Survey on Measuring Cognitive Workload in Human-Computer InteractionACM Computing Surveys10.1145/358227255:13s(1-39)Online publication date: 13-Jul-2023
https://dl.acm.org/doi/10.1145/3582272
Ji KSpina DHettiachchi DSalim FScholer FChen HDuh WHuang HKato MMothe JPoblete B(2023)Examining the Impact of Uncontrolled Variables on Physiological Signals in User Studies for Information Processing ActivitiesProceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3539618.3591981(1971-1975)Online publication date: 19-Jul-2023
https://dl.acm.org/doi/10.1145/3539618.3591981
Show More Cited By

Index Terms

Multimodal approach for cognitive task performance prediction from body postures, facial expressions and EEG signal

Recommendations

Monitoring task engagement using facial expressions and body postures
IWISC '18: Proceedings of the 3rd International Workshop on Interactive and Spatial Computing

As more industries adopt the use of robots to increase productivity, there is an increased need for effective human-robot interaction training, especially in the case of heavy and high precision robots. This implies the need for easy assessment methods ...
Facial expression recognition with Convolutional Neural Networks

Facial expression recognition has been an active research area in the past 10 years, with growing application areas including avatar animation, neuromarketing and sociable robots. The recognition of facial expressions is not an easy problem for machine ...
Face Recognition Through Different Facial Expressions

Face recognition has become an accessible issue for experts as well as ordinary people as it is a focal non-interfering biometric modality. In this paper, we introduced a new approach to perform face recognition under varying facial expressions. The ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

MCPMD '18: Proceedings of the Workshop on Modeling Cognitive Processes from Multimodal Data

October 2018

97 pages

ISBN:9781450360722

DOI:10.1145/3279810

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 16 October 2018

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ICMI '18

Sponsor:

SIGCHI

ICMI '18: INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION

October 16, 2018

Colorado, Boulder

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

19
Total Citations
View Citations
437
Total Downloads

Downloads (Last 12 months)51
Downloads (Last 6 weeks)11

Reflects downloads up to 24 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Nasihati Gilani SPollard KTraum D(2023)Multimodal Prediction of User's Performance in High-Stress Dialogue InteractionsCompanion Publication of the 25th International Conference on Multimodal Interaction10.1145/3610661.3617166(71-75)Online publication date: 9-Oct-2023
https://dl.acm.org/doi/10.1145/3610661.3617166
Kosch TKarolus JZagermann JReiterer HSchmidt AWoźniak P(2023)A Survey on Measuring Cognitive Workload in Human-Computer InteractionACM Computing Surveys10.1145/358227255:13s(1-39)Online publication date: 13-Jul-2023
https://dl.acm.org/doi/10.1145/3582272
Ji KSpina DHettiachchi DSalim FScholer FChen HDuh WHuang HKato MMothe JPoblete B(2023)Examining the Impact of Uncontrolled Variables on Physiological Signals in User Studies for Information Processing ActivitiesProceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3539618.3591981(1971-1975)Online publication date: 19-Jul-2023
https://dl.acm.org/doi/10.1145/3539618.3591981
Jaiswal AZaki Zadeh MHebri ARamesh Babu AMakedon F(2023)A Smart Sensor Suit (SSS) to Assess Cognitive and Physical Fatigue with Machine LearningDigital Human Modeling and Applications in Health, Safety, Ergonomics and Risk Management10.1007/978-3-031-35741-1_10(120-134)Online publication date: 9-Jul-2023
https://doi.org/10.1007/978-3-031-35741-1_10
Kalatzis AGirishan Prabhu VRahman SWittie MStanley L(2022)Emotions Matter: Towards Personalizing Human-System Interactions Using a Two-layer Multimodal ApproachProceedings of the 2022 International Conference on Multimodal Interaction10.1145/3536221.3556582(63-72)Online publication date: 7-Nov-2022
https://dl.acm.org/doi/10.1145/3536221.3556582
Pavel HKarim EZaki Zadeh MJaiswal AKapoor RMakedon FMakedon F(2022)Automated System to Measure Static Balancing in Children to Assess Executive FunctionProceedings of the 15th International Conference on PErvasive Technologies Related to Assistive Environments10.1145/3529190.3534750(569-575)Online publication date: 29-Jun-2022
https://dl.acm.org/doi/10.1145/3529190.3534750
Simpson TRafferty K(2022)EEG Correlates of Driving PerformanceIEEE Transactions on Human-Machine Systems10.1109/THMS.2021.313703252:2(232-247)Online publication date: Apr-2022
https://doi.org/10.1109/THMS.2021.3137032
Redmond PFleury AWard T(2022)An Open Source Multi-Modal Data-Acquisition Platform for Experimental Investigation of Blended Control of Scale Vehicles2022 IEEE International Conference on Metrology for Extended Reality, Artificial Intelligence and Neural Engineering (MetroXRAINE)10.1109/MetroXRAINE54828.2022.9967662(673-678)Online publication date: 26-Oct-2022
https://doi.org/10.1109/MetroXRAINE54828.2022.9967662
Khanam FHossain AAhmad M(2022)Electroencephalogram-based cognitive load level classification using wavelet decomposition and support vector machineBrain-Computer Interfaces10.1080/2326263X.2022.210985510:1(1-15)Online publication date: 8-Aug-2022
https://doi.org/10.1080/2326263X.2022.2109855
Montaño-Serrano VJacinto-Villegas JVilchis-González APortillo-Rodríguez O(2021)Artificial Vision Algorithms for Socially Assistive Robot Applications: A Review of the LiteratureSensors10.3390/s2117572821:17(5728)Online publication date: 25-Aug-2021
https://doi.org/10.3390/s21175728
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents