More Web Proxy on the site http://driver.im/

research-article

ECG-Coupled Multimodal Approach for Stress Detection

Authors:

Gelareh MohammadiAuthors Info & Claims

MuSe '23: Proceedings of the 4th on Multimodal Sentiment Analysis Challenge and Workshop: Mimicked Emotions, Humour and Personalisation

Pages 67 - 72

https://doi.org/10.1145/3606039.3613103

Published: 29 October 2023 Publication History

Abstract

Psychological stress undoubtedly impacts the lives of many people on a regular basis, and it is, therefore, necessary to have systems capable of adequately identifying the level of stress an individual is experiencing. With the emergence of compact devices capable of capturing biosignals such as ECG, it is now feasible to use such signals for different applications, including stress monitoring, in addition to the more popular audio and video modalities. As part of the MuSe-Personalisation sub-challenge for 2023, we develop a GRU-based regressor to continuously predict stress levels from features extracted from raw ECG signals, with our unimodal model achieving a reasonable combined CCC, which is comparable to almost all the baseline audio and video models. Moreover, we utilised this model in combination with the two best performing MuSe-Personalisation baseline models to construct a number of multimodal models via late fusion, with our best model attaining a combined CCC of .7808, thereby improving upon the best baseline system.

Supplementary Material

MP4 File (muse017-video.mp4)

Here, you can find a step by step walk-through of the paper, which highlights the key points.

Download
37.95 MB

References

[1]

F. Agrafioti, D. Hatzinakos, and A. K. Anderson. 2012. ECG Pattern Analysis for Emotion Detection. IEEE Transactions on Affective Computing, Vol. 3, 1 (1 2012), 102--115. https://doi.org/10.1109/T-AFFC.2011.28

Digital Library

[2]

Shahin Amiriparian, Lukas Christ, Andreas König, Eva-Maria Messner, Alan Cowen, Erik Cambria, and Björn W. Schuller. 2023. MuSe 2023 Challenge: Multimodal Prediction of Mimicked Emotions, Cross-Cultural Humour, and Personalised Recognition of Affects. In Proceedings of the 31st ACM International Conference on Multimedia (MM'23), October 29-November 2, 2023, Ottawa, Canada. Association for Computing Machinery, Ottawa, Canada. to appear.

Digital Library

[3]

Patricia J. Bota, Chen Wang, Ana L. N. Fred, and Hugo Placido Da Silva. 2019. A Review, Current Challenges, and Future Possibilities on Emotion Recognition Using Machine Learning and Physiological Signals. IEEE Access, Vol. 7 (2019), 140990--141020. https://doi.org/10.1109/ACCESS.2019.2944001

[4]

Lukas Christ, Shahin Amiriparian, Alice Baird, Alexander Kathan, Niklas Müller, Steffen Klug, Chris Gagne, Panagiotis Tzirakis, Lukas Stappen, Eva-Maria Meßner, Andreas König, Alan Cowen, Erik Cambria, and Björn W. Schuller. 2023. The MuSe 2023 Multimodal Sentiment Analysis Challenge: Mimicked Emotions, Cross-Cultural Humour, and Personalisation. In MuSe'23: Proceedings of the 4th Multimodal Sentiment Analysis Workshop and Challenge. Association for Computing Machinery. co-located with ACM Multimedia 2022, to appear.

[5]

Lukas Christ, Shahin Amiriparian, Alice Baird, Panagiotis Tzirakis, Alexander Kathan, Niklas Mü ller, Lukas Stappen, Eva-Maria Meßner, Andreas Kö nig, Alan Cowen, Erik Cambria, and Björn W. Schuller. 2022. The MuSe 2022 Multimodal Sentiment Analysis Challenge: Humor, Emotional Reactions, and Stress. In Proceedings of the 3rd International Multimodal Sentiment Analysis Workshop and Challenge (MuSe' 22). Association for Computing Machinery (ACM), 5--14. https://doi.org/10.1145/3551876.3554817

Digital Library

[6]

Sven-Åke Christianson. 1992. Emotional stress and eyewitness memory: A critical review. Psychological Bulletin, Vol. 112, 2 (1992), 284--309. https://doi.org/10.1037/0033--2909.112.2.284

[7]

Brianna Chu, Komal Marwaha, Terrence Sanvictores, and Derek Ayers. 2023. Physiology, Stress Reaction. StatPearls Publishing.

[8]

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. ImageNet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 248--255. https://doi.org/10.1109/CVPR.2009.5206848

[9]

Rayan Elalamy, Marios Fanourakis, and Guillaume Chanel. 2021. Multi-modal emotion recognition using recurrence plots and transfer learning on physiological signals. In 2021 9th International Conference on Affective Computing and Intelligent Interaction (ACII). IEEE, 1--7. https://doi.org/10.1109/ACII52823.2021.9597442

[10]

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2015. Deep Residual Learning for Image Recognition. (12 2015).

[11]

Jingshan Huang, Binqiang Chen, Bin Yao, and Wangpeng He. 2019. ECG Arrhythmia Classification Using STFT-Based Spectrogram and Convolutional Neural Network. IEEE Access, Vol. 7 (2019), 92871--92880. https://doi.org/10.1109/ACCESS.2019.2928017

[12]

Rateb Katmah, Fares Al-Shargie, Usman Tariq, Fabio Babiloni, Fadwa Al-Mughairbi, and Hasan Al-Nashash. 2021. A Review on Mental Stress Assessment Methods Using EEG Signals. Sensors, Vol. 21, 15 (7 2021), 5043. https://doi.org/10.3390/s21155043

[13]

Yea-Hoon Kwon, Sae-Byuk Shin, and Shin-Dug Kim. 2018. Electroencephalography Based Fusion Two-Dimensional (2D)-Convolution Neural Networks (CNN) Model for Emotion Recognition System. Sensors, Vol. 18, 5 (4 2018), 1383. https://doi.org/10.3390/s18051383

[14]

Wei Li, Zhen Zhang, and Aiguo Song. 2021. Physiological-signal-based emotion recognition: An odyssey from methodology to philosophy. Measurement, Vol. 172 (2 2021), 108747. https://doi.org/10.1016/j.measurement.2020.108747

[15]

Lawrence I-Kuei Lin. 1989. A Concordance Correlation Coefficient to Evaluate Reproducibility. Biometrics, Vol. 45, 1 (3 1989), 255. https://doi.org/10.2307/2532051

[16]

Mohsen Naji, Mohammd Firoozabadi, and Parviz Azadfallah. 2014. Classification of Music-Induced Emotions Based on Information Fusion of Forehead Biosignals and Electrocardiogram. Cognitive Computation, Vol. 6, 2 (6 2014), 241--252. https://doi.org/10.1007/s12559-013--9239--7

[17]

Pritam Sarkar and Ali Etemad. 2022. Self-Supervised ECG Representation Learning for Emotion Recognition. IEEE Transactions on Affective Computing, Vol. 13, 3 (7 2022), 1541--1554. https://doi.org/10.1109/TAFFC.2020.3014842

[18]

Lukas Stappen, Alice Baird, Lukas Christ, Lea Schumann, Benjamin Sertolli, Eva-Maria Messner, Erik Cambria, Guoying Zhao, and Björn W Schuller. 2021a. The MuSe 2021 multimodal sentiment analysis challenge: sentiment, emotion, physiological-emotion, and stress. In Proceedings of the 2nd on Multimodal Sentiment Analysis Challenge. 5--14.

Digital Library

[19]

Lukas Stappen, Lea Schumann, Benjamin Sertolli, Alice Baird, Benjamin Weigell, Erik Cambria, and Björn W. Schuller. 2021b. MuSe-Toolbox: The Multimodal Sentiment Analysis Continuous Annotation Fusion and Discrete Class Transformation Toolbox. In Proceedings of the 2nd on Multimodal Sentiment Analysis Challenge. ACM, New York, NY, USA, 75--82. https://doi.org/10.1145/3475957.3484451

Digital Library

[20]

World Health Organization. 2023. Stress. https://www.who.int/news-room/questions-and-answers/item/stress

[21]

Sarthak Yadav, Tilak Purohit, Zohreh Mostaani, Bogdan Vlasenko, and Mathew Magimai.-Doss. 2022. Comparing Biosignal and Acoustic feature Representation for Continuous Emotion Recognition. In MuSe 2022 - Proceedings of the 3rd International Multimodal Sentiment Analysis Workshop and Challenge. Association for Computing Machinery, Inc, 37--45. https://doi.org/10.1145/3551876.3554812

Digital Library

[22]

Özal Yldrm, Pawe Pławiak, Ru-San Tan, and U. Rajendra Acharya. 2018. Arrhythmia detection using deep convolutional neural network with long duration ECG signals. Computers in Biology and Medicine, Vol. 102 (11 2018), 411--420. https://doi.org/10.1016/j.compbiomed.2018.09.009 io

Digital Library

Cited By

Amiriparian SChrist LKönig ACowen AMeßner ECambria ESchuller BEl Saddik AMei TCucchiara RBertini MTobon Vallejo DAtrey PHossain M(2023)MuSe 2023 Challenge: Multimodal Prediction of Mimicked Emotions, Cross-Cultural Humour, and Personalised Recognition of AffectsProceedings of the 31st ACM International Conference on Multimedia10.1145/3581783.3610943(9723-9725)Online publication date: 26-Oct-2023
https://dl.acm.org/doi/10.1145/3581783.3610943

Index Terms

ECG-Coupled Multimodal Approach for Stress Detection
1. Computing methodologies
  1. Machine learning
2. Information systems
  1. Information retrieval
    1. Retrieval tasks and goals
      1. Sentiment analysis

Recommendations

A Hybrid Approach of a Deep Learning Technique for Real–Time ECG Beat Detection
Recent Advances in Modelling, Analysis and Implementation of Cyber-Physical Systems (Special section, pp. 345-413), Remigiusz Wiśniewski, Luis Gomes and Shaohua Wan (Eds.)
Abstract
This paper presents a new customized hybrid approach for early detection of cardiac abnormalities using an electrocardiogram (ECG). The ECG is a bioelectrical signal that helps monitor the heart’s electrical activity. It can provide health ...
Cardiac Ischemia Diagnosis Using Stress ECG Analysis
ICACC '13: Proceedings of the 2013 Third International Conference on Advances in Computing and Communications

Myocardial Ischemia is the most common heart disease. Stress ECG has been effectively used for analysis of the myocardial ischemia than normal ECG because of the reason that ischemic conditions will be dominated in stress conditions. According to the ...
Development of Lead System for ECG-Derived Respiration Aimed at Detection of Obstructive Sleep Apnea Syndrome
SITIS '13: Proceedings of the 2013 International Conference on Signal-Image Technology & Internet-Based Systems

Obstructive sleep apnea syndrome is the most common type of sleep apnea, characterized by repetitive pauses in breathing during sleep. Recent studies have investigated screening methods based only on an electrocardiogram (ECG). Generally, in ECG-based ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

MuSe '23: Proceedings of the 4th on Multimodal Sentiment Analysis Challenge and Workshop: Mimicked Emotions, Humour and Personalisation

November 2023

113 pages

ISBN:9798400702709

DOI:10.1145/3606039

General Chairs:
Shahin Amiriparian
University of Augsburg, Germany
,
Lukas Christ
University of Augsburg, Germany
,
Andreas Konig
University of Passau, Germany
,
Alan Cowen
Hume AI, USA
,
Eva-Maria Meßner
University of Ulm, Germany
,
Erik Cambria
Nanyang Technological University, Singapore
,
Bjorn W. Schuller
Imperial College London, UK

Copyright © 2023 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGMM: ACM Special Interest Group on Multimedia

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 29 October 2023

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

MM '23

Sponsor:

SIGMM

MM '23: The 31st ACM International Conference on Multimedia

October 29, 2023

Ottawa ON, Canada

Acceptance Rates

Overall Acceptance Rate 14 of 17 submissions, 82%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1
Total Citations
View Citations
146
Total Downloads

Downloads (Last 12 months)94
Downloads (Last 6 weeks)4

Reflects downloads up to 31 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Amiriparian SChrist LKönig ACowen AMeßner ECambria ESchuller BEl Saddik AMei TCucchiara RBertini MTobon Vallejo DAtrey PHossain M(2023)MuSe 2023 Challenge: Multimodal Prediction of Mimicked Emotions, Cross-Cultural Humour, and Personalised Recognition of AffectsProceedings of the 31st ACM International Conference on Multimedia10.1145/3581783.3610943(9723-9725)Online publication date: 26-Oct-2023
https://dl.acm.org/doi/10.1145/3581783.3610943

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents