More Web Proxy on the site http://driver.im/

research-article

Adaptive Body Gesture Representation for Automatic Emotion Recognition

Authors:

Alessandra Staglianò,

Francesca Odone,

Antonio CamurriAuthors Info & Claims

ACM Transactions on Interactive Intelligent Systems (TiiS), Volume 6, Issue 1

Article No.: 6, Pages 1 - 31

https://doi.org/10.1145/2818740

Published: 09 March 2016 Publication History

Abstract

We present a computational model and a system for the automated recognition of emotions starting from full-body movement. Three-dimensional motion data of full-body movements are obtained either from professional optical motion-capture systems (Qualisys) or from low-cost RGB-D sensors (Kinect and Kinect2). A number of features are then automatically extracted at different levels, from kinematics of a single joint to more global expressive features inspired by psychology and humanistic theories (e.g., contraction index, fluidity, and impulsiveness). An abstraction layer based on dictionary learning further processes these movement features to increase the model generality and to deal with intraclass variability, noise, and incomplete information characterizing emotion expression in human movement. The resulting feature vector is the input for a classifier performing real-time automatic emotion recognition based on linear support vector machines. The recognition performance of the proposed model is presented and discussed, including the tradeoff between precision of the tracking measures (we compare the Kinect RGB-D sensor and the Qualisys motion-capture system) versus dimension of the training dataset. The resulting model and system have been successfully applied in the development of serious games for helping autistic children learn to recognize and express emotions by means of their full-body movement.

References

[1]

W. K. Allard, G. Chen, and M. Maggioni. 2012. Multi-scale geometric methods for data sets II: Geometric multi-resolution analysis. Appl. Comput. Harmonic Anal. 32, 3 (2012), 435--462.

[2]

M. Argyle. 2013. Bodily Communication. Routledge, London.

[3]

A. P. Atkinson, W. H. Dittrich, A. J. Gemmell, A. W. Young, and others. 2004. Emotion perception from dynamic and static body expressions in point-light and full-light displays. Perception (London) 33 (2004), 717--746.

[4]

T. Balomenos, A. Raouzaiou, S. Ioannou, A. Drosopoulos, K. Karpouzis, and S. Kollias. 2005. Emotion analysis in man-machine interaction systems. In Machine Learning for Multimodal Interaction. Springer, Berlin, 318--328.

Digital Library

[5]

C. Basso, M. Santoro, A. Verri, and S. Villa. 2011. PADDLE: Proximal algorithm for dual dictionaries learning. In Proceedings of the International Conference on Artificial Neural Networks and Machine Learning (ICANN’11). Springer, Berlin, 379--386.

Digital Library

[6]

R. T. Boone and J. G. Cunningham. 1998. Children’s decoding of emotion in expressive body movement: The development of cue attunement. Dev. Psychol. 34 (1998), 1007--1016.

[7]

P. E. Bull. 1987. Posture and Gesture. Pergamon Press, London.

[8]

A. Camurri. 1995. Interactive dance/music systems. In Proceedings of the 1995 International Computer Music Conference. 245--225.

[9]

A. Camurri, P. Coletta, G. Varni, and S. Ghisio. 2007. Developing multimodal interactive systems with eyesweb XMI. In Proceedings of the Conference on New Interfaces for Musical Expression (NIME), 2007. ACM, New York, NY, 302--305.

Digital Library

[10]

A. Camurri, I. Lagerlöf, and G. Volpe. 2003. Recognizing emotion from dance movement: Comparison of spectator recognition and automated techniques. Int. J. Human-Comput. Stud. 59, 1 (2003), 213--225.

Digital Library

[11]

A. Camurri, B. Mazzarino, and G. Volpe. 2004. Analysis of expressive gesture: The eyesweb expressive gesture processing library. In Gesture-Based Communication in Human-Computer Interaction. Springer, Berlin, 460--467.

[12]

A. Camurri, G. Volpe, G. De Poli, and M. Leman. 2005. Communicating expressiveness and affect in multimodal interactive systems. IEEE Multimedia 12, 1 (2005), 43--53.

Digital Library

[13]

G. Chen and M. Maggioni. 2010. Multiscale geometric wavelets for the analysis of point clouds. In Proceedings of the 44th Annual Conference on Information Sciences and Systems (CISS). IEEE, 1--6.

[14]

R. R. Cornelius. 1996. The Science of Emotion: Research and Tradition in the Psychology of Emotions. Prentice-Hall, Piscataway, NJ.

[15]

R. R. Cornelius. 2000. Theoretical approaches to emotion. In ISCA Tutorial and Research Workshop (ITRW) on Speech and Emotion.

[16]

M. Coulson. 2004. Attributing emotion to static body postures: Recognition accuracy, confusions, and viewpoint dependence. J. Nonverb. Behav. 28, 2 (2004), 117--139.

[17]

R. Cowie, E. Douglas-Cowie, N. Tsapatsoulis, G. Votsis, S. Kollias, W. Fellenz, and J. G. Taylor. 2001. Emotion recognition in human-computer interaction. IEEE Signal Process. Mag. 18, 1 (2001), 32--80.

[18]

B. de Gelder. 2009. Why bodies? Twelve reasons for including bodily expressions in affective neuroscience. Philos. Trans. Roy. Soc. B: Biol. Sci. 364, 1535 (2009), 3475--3484.

[19]

B. de Gelder and N. Hadjikhani. 2006. Non-conscious recognition of emotional body language. Neuroreport 17, 6 (2006), 583--586.

[20]

B. de Gelder, J. Van den Stock, H. K. M. Meeren, C. Sinke, M. E. Kret, and M. Tamietto. 2010. Standing up for the body. Recent progress in uncovering the networks involved in the perception of bodies and bodily expressions. Neurosci. Biobehav. Rev. 34, 4 (2010), 513--527.

[21]

M. de Meijer. 1989. The contribution of general features of body movement to the attribution of emotions. J. Nonverb. Behav. 13, 4 (1989), 247--268.

[22]

W. H. Dittrich, T. Troscianko, S. Lea, and D. Morgan. 1996. Perception of emotion from dynamic point-light displays represented in dance. Perception (London) 25, 6 (1996), 727--738.

[23]

P. Ekman. 1965. Differential communication of affect by head and body cues. J. Person. Soc. Psychol. 2, 5 (1965), 726.

[24]

P. L. Ekman and W. V. Friesen. 1974. Detecting deception from the body or face. J. Person. Soc. Psychol. 29, 3 (1974), 288.

[25]

J. L. Evenden. 1999. Varieties of impulsivity. Psychopharmacology 146, 4 (1999), 348--361.

[26]

T. Flash and N. Hogan. 1985. The coordination of arm movements: An experimentally confirmed mathematical model. J. Neurosci. 5, 7 (1985), 1688--1703.

[27]

D. Glowinski, N. Dael, A. Camurri, G. Volpe, M. Mortillaro, and K. Scherer. 2011. Toward a minimal representation of affective gestures. IEEE Trans. Affect. Comput. 2, 2 (2011), 106--118.

Digital Library

[28]

O. Golan, E. Ashwin, Y. Granader, S. McClintock, K. Day, V. Leggett, and S. Baron-Cohen. 2010. Enhancing emotion recognition in children with autism spectrum conditions: An intervention using animated vehicles with real emotional faces. J. Autism Dev. Disorders 40, 3 (2010), 269--279.

[29]

O. Golan, S. Baron-Cohen, and J. Hill. 2006. The Cambridge mindreading (CAM) face-voice battery: Testing complex emotion recognition in adults with and without Asperger syndrome. J. Autism Dev. Disorders 36, 2 (2006), 169--183.

[30]

P. Heiser, J. Frey, J. Smidt, C. Sommerlad, P. M. Wehmeier, J. Hebebrand, and H. Remschmidt. 2004. Objective measurement of hyperactivity, impulsivity, and inattention in children with hyperkinetic disorders before and after treatment with methylphenidate. Eur. Child Adolescent Psychiat. 13, 2 (2004), 100--104.

[31]

H. Hill and F. E. Pollick. 2000. Exaggerating temporal differences enhances recognition of individuals from point light displays. Psychol. Sci. 11, 3 (2000), 223--228.

[32]

B. R. Hoff. 1992. A Computational Description of the Organization of Human Reaching and Prehension. Ph.D. Dissertation. University of Southern California.

Digital Library

[33]

N. Hogan. 1984. An organizing principle for a class of voluntary movements. J. Neurosci. 4, 11 (1984), 2745--2754.

[34]

G. Johansson. 1973. Visual perception of biological motion and a model for its analysis. Perception & Psychophysics 14, 2 (1973), 201--211.

[35]

A. Kapoor, W. Burleson, and R. W. Picard. 2007. Automatic prediction of frustration. Int. J. Human-Comput. Stud. 65, 8 (2007), 724--736.

Digital Library

[36]

A. Kapur, A. Kapur, N. Virji-Babul, G. Tzanetakis, and P. F. Driessen. 2005. Gesture-based affective computing on motion capture data. In Affective Computing and Intelligent Interaction. Springer, Berlin, 1--7.

Digital Library

[37]

M. Karg, A. Samadani, R. Gorbet, K. Kuhnlenz, J. Hoey, and D. Kulic. 2013. Body movements for affective expression: A survey of automatic recognition and generation. IEEE Trans. Affect. Comput. 4, 4 (2013), 341--359.

[38]

A. Klautau, N. Jevtic, and A. Orlitsky. 2002. Combined binary classifiers with applications to speech recognition. In Proceedings of the Annual Conference of the International Speech Communication Association (INTERSPEECH).

[39]

Kinect. 2013. Kinect for Windows. Retrieved from http://www.microsoft.com/en-us/kinectforwindows/.

[40]

A. Kleinsmith and N. Bianchi-Berthouze. 2013. Affective body expression perception and recognition: A survey. IEEE Trans. Affect. Comput. 4, 1 (2013), 15--33.

Digital Library

[41]

R. Laban. 1963. Modern Educational Dance. Macdonald & Evans, London.

[42]

R. Laban and F. C. Lawrence. 1947. Effort. Macdonald & Evans, London.

[43]

H. Lee, A. Battle, R. Raina, and A. Ng. 2006. Efficient sparse coding algorithms. In Advances in Neural Information Processing Systems. 801--808.

[44]

Y. Ma, H. M. Paterson, and F. E. Pollick. 2006. A motion capture library for the study of identity, gender, and emotion perception from biological motion. Beh. Res. Methods 38, 1 (2006), 134--141.

[45]

J. Mairal, F. Bach, J. Ponce, and G. Sapiro. 2010. Online learning for matrix factorization and sparse coding. J. Machine Learn. Res. 11 (2010), 19--60.

Digital Library

[46]

B. Mazzarino and M. Mancini. 2009. The need for impulsivity & smoothness-improving HCI by qualitatively measuring new high-level human motion features. In Proceedings of the International Conference on Signal Processing and Multimedia Applications (SIGMAP). 62--67.

[47]

J. T. McConville, C. E. Clauser, T. D. Churchill, J. Cuzzi, and I. Kaleps. 1980. Anthropometric Relationships of Body and Body Segment Moments of Inertia. Technical Report. DTIC Document.

[48]

C. T. Nagoshi, J. R. Wilson, and L. A. Rodriguez. 2006. Impulsivity, sensation seeking, and behavioral and emotional responses to alcohol. Alcohol. Clin. Exp. Res. 15, 4 (2006), 661--667.

[49]

N. Noceti and F. Odone. 2012. Learning common behaviors from large sets of unlabeled temporal series. Image and Vision Computing 30, 11 (2012), 875--895.

Digital Library

[50]

N. Noceti, M. Santoro, and F. Odone. 2011. Learning behavioral patterns of time series for video-surveillance. In Machine Learning for Vision-Based Motion Analysis. Springer, Berlin, 275--304.

[51]

H. O’Reilly, D. Pigat, S. Berggren, S. Fridenson, S. Tal, O. Golan, S. Bolte, S. Baron-Cohen, and D. Lundqvist. 2014. The EU-Emotion Stimulus Set: A Validation Study. Retrieved from http://www. autismresearchcentre.com/projects/Emoticons.aspx.

[52]

S. Piana, A. Staglianò, F. Odone, and A. Camurri. 2014a. Emotional charades. In Proceedings of the 16th ACM International Conference on Multimodal Interaction (ICMI).

Digital Library

[53]

S. Piana, A. Staglianò, F. Odone, A. Verri, and A. Camurri. 2014b. Real-time automatic emotion recognition from body gestures. arXiv:1402.5047 (2014).

[54]

S. Piana, A. Staglianò, S. Semino, F. Odone, C. Usai, and A. Camurri. 2015. Evaluation of the emotional game for ASC children. (2015). In preparation (2015).

[55]

F. E. Pollick, H. M. Paterson, A. Bruderlin, and A. J. Sanford. 2001. Perceiving affect from arm movement. Cognition 82, 2 (2001), B51--B61.

[56]

Qualisys. 2013. Qualisys Motion Capture Systems. Retrieved from http://www.Qualisys.com.

[57]

C. L. Roether, L. Omlor, and M. A. Giese. 2008. Lateral asymmetry of bodily emotion expression. Curr. Biol. 18, 8 (2008), R329--R330.

[58]

R. Rubinstein, A. M. Bruckstein, and M. Elad. 2010. Dictionaries for sparse representation modeling. Proc. IEEE 98, 6 (2010), 1045--1057.

[59]

B. Schuller, E. Marchi, S. Baron-Cohen, H. O’Reilly, P. Robinson, I. Davies, O. Golan, S. Fridenson, S. Tal, S. Newman, N. Meir, R. Shillo, A. Camurri, S. Piana, S. Bölte, D. Lundqvist, S. Berggren, A. Baranger, and N. Sullings. 2013. ASC-inclusion: Interactive emotion games for social inclusion of children with autism spectrum conditions. In Proceedings of the 1st International Workshop on Intelligent Digital Games for Empowerment and Inclusion (IDGEI’13) held in conjunction with the 8th Foundations of Digital Games 2013 (FDG), B. Schuller, L. Paletta, and N. Sabouret (Eds.). ACM, SASDG (Chania, Greece, May 2013).

[60]

W. A. Sethares and T. W. Staley. 1999. Periodicity transforms. IEEE Trans. Signal Process. 47, 11 (1999), 2953--2964.

Digital Library

[61]

E. Todorov and M. I. Jordan. 1998. Smoothness maximization along a predefined path accurately predicts the speed profiles of complex arm movements. J. Neurophysiol. 80, 2 (1998), 696--714.

[62]

V. Vapnik. 1998. Statistical Learning Theory. Wiley, New York, NY.

[63]

H. G. Wallbott. 1998. Bodily expression of emotion. Eur. J. Soc. Psychol. 28, 6 (1998), 879--896.

Cited By

Niewiadomski RSuresh ASciutti ADi Cesare G(2025)Decoding communicative action vitality forms in social contextsFrontiers in Psychology10.3389/fpsyg.2025.147887516Online publication date: 5-Feb-2025
https://doi.org/10.3389/fpsyg.2025.1478875
Gilstrap TAlghamdi MPhan TLee S(2025)EMG-Based Continuous Estimation of Index Finger Movements With Varying Interjoint Coordination Patterns by Modeling Musculoskeletal DynamicsIEEE Access10.1109/ACCESS.2025.352890213(13454-13463)Online publication date: 2025
https://doi.org/10.1109/ACCESS.2025.3528902
Remland MJones T(2025)Emotional Expressions Beyond the Face and Conflict InterventionBody Language Communication10.1007/978-3-031-70064-4_11(259-284)Online publication date: 2-Jan-2025
https://doi.org/10.1007/978-3-031-70064-4_11
Show More Cited By

Index Terms

Adaptive Body Gesture Representation for Automatic Emotion Recognition
1. Computing methodologies
  1. Machine learning

Recommendations

Emotion Recognition via Body Gesture: Deep Learning Model Coupled with Keyframe Selection
MLMI '18: Proceedings of the 2018 International Conference on Machine Learning and Machine Intelligence

Many psychological research revealed that bodily gestures convey crucial information to emotion recognition. This valuable aspect, however, has not gained much the attention from the science community, compared to other modalities, such as facial ...
Automatic Recognition of Experienced Emotional State from Body Movement
Human-Computer Interaction. Theory, Methods and Tools
Abstract
Although body movement carries information about a person’s emotional state, this modality is not widely used in automatic emotion measurement systems. With this paper, we address the question of the automatic recognition of a person’s affective ...
Time-Dependent Body Gesture Representation for Video Emotion Recognition
MultiMedia Modeling
Abstract
Video emotion recognition has recently become a research hotspot in the field of affective computing. Although large parts of studies focus on facial cues, body gestures are the only available cues in some scenes such as video monitoring systems. ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Transactions on Interactive Intelligent Systems

ACM Transactions on Interactive Intelligent Systems Volume 6, Issue 1

Special Issue on New Directions in Eye Gaze for Interactive Intelligent Systems (Part 2 of 2), Regular Articles and Special Issue on Highlights of IUI 2015 (Part 1 of 2)

May 2016

219 pages

ISSN:2160-6455

EISSN:2160-6463

DOI:10.1145/2896319

Editors:
Anthony Jameson
German Research Center for Artifi cial Intelligence (DFKI), Germany
,
Krzysztof Gajos
Harvard University, U.S.A.

Issue’s Table of Contents

Copyright © 2016 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 09 March 2016

Accepted: 01 November 2015

Revised: 01 September 2015

Received: 01 December 2014

Published in TIIS Volume 6, Issue 1

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Funding Sources

European Commission

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

74
Total Citations
View Citations
1,184
Total Downloads

Downloads (Last 12 months)91
Downloads (Last 6 weeks)4

Reflects downloads up to 05 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Niewiadomski RSuresh ASciutti ADi Cesare G(2025)Decoding communicative action vitality forms in social contextsFrontiers in Psychology10.3389/fpsyg.2025.147887516Online publication date: 5-Feb-2025
https://doi.org/10.3389/fpsyg.2025.1478875
Gilstrap TAlghamdi MPhan TLee S(2025)EMG-Based Continuous Estimation of Index Finger Movements With Varying Interjoint Coordination Patterns by Modeling Musculoskeletal DynamicsIEEE Access10.1109/ACCESS.2025.352890213(13454-13463)Online publication date: 2025
https://doi.org/10.1109/ACCESS.2025.3528902
Remland MJones T(2025)Emotional Expressions Beyond the Face and Conflict InterventionBody Language Communication10.1007/978-3-031-70064-4_11(259-284)Online publication date: 2-Jan-2025
https://doi.org/10.1007/978-3-031-70064-4_11
Lai ZQing CTan JLuo WXu XCao ZJung TXu P(2024)Online Multi-level Contrastive Representation Distillation for Cross-Subject fNIRS Emotion RecognitionProceedings of the 1st International Workshop on Brain-Computer Interfaces (BCI) for Multimedia Understanding10.1145/3688862.3689110(29-37)Online publication date: 28-Oct-2024
https://dl.acm.org/doi/10.1145/3688862.3689110
Canovi NMontagna FNiewiadomski RSciutti ADi Cesare GBeyan C(2024)Diffusion-Based Unsupervised Pre-training for Automated Recognition of Vitality FormsProceedings of the 2024 International Conference on Advanced Visual Interfaces10.1145/3656650.3656689(1-9)Online publication date: 3-Jun-2024
https://dl.acm.org/doi/10.1145/3656650.3656689
Ghosh SMandi SSen SMitra BDe P(2024)Towards Estimating Missing Emotion Self-reports Leveraging User Similarity: A Multi-task Learning ApproachProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642833(1-19)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642833
Wang TLiu SHe FDai WDu MKe YMing D(2024)Emotion Recognition From Full-Body Motion Using Multiscale Spatio-Temporal NetworkIEEE Transactions on Affective Computing10.1109/TAFFC.2023.330519715:3(898-912)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1109/TAFFC.2023.3305197
Nalwaya APachori R(2024)Fourier–Bessel Domain Adaptive Wavelet Transform-Based Method for Emotion Identification From EEG SignalsIEEE Sensors Letters10.1109/LSENS.2023.33476488:2(1-4)Online publication date: Feb-2024
https://doi.org/10.1109/LSENS.2023.3347648
Wahyu Prabowo DAkhmad Setiawan NDebayle JNugroho H(2024)Enhancing EEG-Based Emotion Recognition Using Asymmetric Windowing Recurrence PlotsIEEE Access10.1109/ACCESS.2024.340938412(85969-85982)Online publication date: 2024
https://doi.org/10.1109/ACCESS.2024.3409384
Jekauc DBurkart DFritsch JHesenius MMeyer OSarfraz SStiefelhagen R(2024)Recognizing affective states from the expressive behavior of tennis players using Convolutional Neural NetworksKnowledge-Based Systems10.1016/j.knosys.2024.111856(111856)Online publication date: Apr-2024
https://doi.org/10.1016/j.knosys.2024.111856
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Issue’s Table of Contents