Multiscale Transformer-Based for Multimodal Affective States Estimation from Physiological Signals
Pages 113 - 122
Abstract
In recent times, the estimation of affective states from physiological data has garnered considerable attention within the research community owing to its wide-ranging applicability in daily life scenarios. The advancement of wearable technology has facilitated the collection of physiological signals, thereby highlighting the necessity for a resilient system capable of effectively discerning and interpreting user states. This work introduces an innovative methodology aimed at addressing the Valence-Arousal estimation, through the utilization of physiological signals. Our proposed model presents an efficient multi-scale transformer-based architecture for fusing signals from multiple modern sensors to tackle Emotion Recognition task. Our approach involves applying a multi-modal technique combined with scaling data to establish the relationship between internal body signals and human emotions. Additionally, we utilize Transformer and Gaussian transformation techniques to improve signal encoding effectiveness and overall performance. Our proposed model demonstrates compelling performance on the CASE dataset, achieving an impressive Root Mean Squared Error (RMSE) of 1.45.
References
[1]
Ahmad Z and Khan N A survey on physiological signal-based emotion recognition Bioengineering 2022 9 11 688
[2]
Algarni M, Saeed F, Al-Hadhrami T, Ghabban F, and Al-Sarem M Deep learning-based approach for emotion recognition using electroencephalography (EEG) signals using bi-directional long short-term memory (Bi-LSTM) Sensors 2022 22 8 2976
[3]
Bradley MM and Lang PJ Measuring emotion: the self-assessment manikin and the semantic differential J. Behav. Ther. Exp. Psychiatry 1994 25 1 49-59
[4]
Domínguez-Jiménez JA, Campo-Landines KC, Martínez-Santos JC, Delahoz EJ, and Contreras-Ortiz SH A machine learning model for emotion recognition from physiological signals Biomed. Sig. Process. Control 2020 55 101646
[5]
Harper R and Southern J A Bayesian deep learning framework for end-to-end prediction of emotion from heartbeat IEEE Trans. Affect. Comput. 2020 13 2 985-991
[6]
Hinduja, S., Bilalpur, M., Jivnani, L., Canavan, S.: Multimodal temporal modeling of emotion using physiological signals (2023)
[7]
Hu L, Yang J, Chen M, Qian Y, and Rodrigues JJ SCAI-SVSC: smart clothing for effective interaction with a sustainable vital sign collection Fut. Gener. Comput. Syst. 2018 86 329-338
[8]
Li S and Deng W Deep facial expression recognition: a survey IEEE Trans. Affect. Comput. 2020 13 3 1195-1215
[9]
Li, S., et al.: Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting. In: Advances in Neural Information Processing Systems, vol. 32 (2019)
[10]
Li Y, Peng X, Zhang J, Li Z, and Wen M DCT-GAN: dilated convolutional transformer-based GAN for time series anomaly detection IEEE Trans. Knowl. Data Eng. 2021 35 3632-3644
[11]
Loshchilov, I., Hutter, F.: SGDR: stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608.03983 (2016)
[12]
Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 (2017)
[13]
Martens T, Niemann M, and Dick U Sensor measures of affective leaning Front. Psychol. 2020 11 379
[14]
Nakisa B, Rastgoo MN, Rakotonirainy A, Maire F, and Chandran V Long short term memory hyperparameter optimization for a neural network based emotion recognition framework IEEE Access 2018 6 49325-49338
[15]
Ragot M, Martin N, Em S, Pallamin N, and Diverrez J-M Ahram T and Falcão C Emotion recognition using physiological signals: laboratory vs. wearable sensors Advances in Human Factors in Wearable Technologies and Game Design 2018 Cham Springer 15-22
[16]
Rahimi, A., Recht, B.: Random features for large-scale kernel machines. In: Advances in Neural Information Processing Systems, vol. 20 (2007)
[17]
Rattanyu K and Mizukawa M Jacko JA Emotion recognition using biological signal in intelligent space Human-Computer Interaction. Towards Mobile and Intelligent Interaction Environments 2011 Heidelberg Springer 586-592
[18]
Rattanyu, K., Ohkura, M., Mizukawa, M.: Emotion monitoring from physiological signals for service robots in the living space. In: ICCAS 2010, pp. 580–583. IEEE (2010)
[19]
Romeo L, Cavallo A, Pepa L, Bianchi-Berthouze N, and Pontil M Multiple instance learning for emotion recognition using physiological signals IEEE Trans. Affect. Comput. 2019 13 1 389-407
[20]
Saganowski, S., Behnke, M., Komoszyńska, J., Kunc, D., Perz, B., Kazienko, P.: A system for collecting emotionally annotated physiological signals in daily life using wearables. In: 2021 9th International Conference on Affective Computing and Intelligent Interaction Workshops and Demos (ACIIW), pp. 1–3. IEEE (2021)
[21]
Saganowski S, Perz B, Polak A, and Kazienko P Emotion recognition for everyday life using physiological signals from wearables: a systematic literature review IEEE Trans. Affect. Comput. 2022 14 1876-1897
[22]
Santamaria-Granados L, Munoz-Organero M, Ramirez-Gonzalez G, Abdulhay E, and Arunkumar N Using deep convolutional neural network for emotion detection on a physiological signals dataset (amigos) IEEE Access 2018 7 57-67
[23]
Schmidt, P., Dürichen, R., Reiss, A., Van Laerhoven, K., Plötz, T.: Multi-target affect detection in the wild: an exploratory study. In: Proceedings of the 2019 ACM International Symposium on Wearable Computers, pp. 211–219 (2019)
[24]
Schuller BW Speech emotion recognition: two decades in a nutshell, benchmarks, and ongoing trends Commun. ACM 2018 61 5 90-99
[25]
Sharma K, Castellini C, van den Broek EL, Albu-Schaeffer A, and Schwenker F A dataset of continuous affect annotations and physiological signals for emotion analysis Sci. Data 2019 6 1 196
[26]
Shu L et al. A review of emotion recognition using physiological signals Sensors 2018 18 7 2074
[27]
Vaswani, A., et al.: Attention is all you need. In: Advances in Neural Information Processing Systems, vol. 30 (2017)
[28]
Vazquez-Rodriguez, J., Lefebvre, G., Cumin, J., Crowley, J.L.: Emotion recognition with pre-trained transformers using multimodal signals. In: 2022 10th International Conference on Affective Computing and Intelligent Interaction (ACII), pp. 1–8. IEEE (2022)
[29]
Vazquez-Rodriguez, J., Lefebvre, G., Cumin, J., Crowley, J.L.: Transformer-based self-supervised learning for emotion recognition. In: 2022 26th International Conference on Pattern Recognition (ICPR), pp. 2605–2612. IEEE (2022)
[30]
Wu, N., Green, B., Ben, X., O’Banion, S.: Deep transformer models for time series forecasting: The influenza prevalence case. arXiv preprint arXiv:2001.08317 (2020)
[31]
Wu, Y., Daoudi, M., Amad, A.: Transformer-based self-supervised multimodal representation learning for wearable emotion recognition. IEEE Trans. Affect. Comput. (2023)
[32]
Yang, K., et al.: Mobile emotion recognition via multiple physiological signals using convolution-augmented transformer. In: Proceedings of the 2022 International Conference on Multimedia Retrieval, pp. 562–570 (2022)
[33]
Zhao, B., Wang, Z., Yu, Z., Guo, B.: EmotionSense: emotion recognition based on wearable wristband. In: 2018 IEEE SmartWorld, Ubiquitous Intelligence & Computing, Advanced & Trusted Computing, Scalable Computing & Communications, Cloud & Big Data Computing, Internet of People and Smart City Innovation (SmartWorld/SCALCOM/UIC/ATC/CBDCom/IOP/SCI), pp. 346–355. IEEE (2018)
Recommendations
Emotion Recognition Using Physiological Signals
MIDI '15: Proceedings of the Mulitimedia, Interaction, Design and InnnovationIn this paper the problem of emotion recognition using physiological signals is presented. Firstly the problems with acquisition of physiological signals related to specific human emotions are described. It is not a trivial problem to elicit real ...
Affective ranking of movie scenes using physiological signals and content analysis
MS '08: Proceedings of the 2nd ACM workshop on Multimedia semanticsIn this paper, we propose an approach for affective ranking of movie scenes based on the emotions that are actually felt by spectators. Such a ranking can be used for characterizing the affective, or emotional, content of video clips. The ranking can ...
Evaluating multimodal affective fusion using physiological signals
IUI '11: Proceedings of the 16th international conference on Intelligent user interfacesIn this paper we present an evaluation of an affective multimodal fusion approach utilizing dimensional representations of emotion. The evaluation uses physiological signals as a reference measure of users' emotional states. Surface electromyography (...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
Nov 2023
414 pages
ISBN:978-3-031-47664-8
DOI:10.1007/978-3-031-47665-5
© The Author(s), under exclusive license to Springer Nature Switzerland AG 2023.
Publisher
Springer-Verlag
Berlin, Heidelberg
Publication History
Published: 05 November 2023
Author Tags
Qualifiers
- Article
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 0Total Downloads
- Downloads (Last 12 months)0
- Downloads (Last 6 weeks)0
Reflects downloads up to 13 Dec 2024