More Web Proxy on the site http://driver.im/

article

Hierarchical affective content analysis in arousal and valence dimensions

Authors:

Yong RuiAuthors Info & Claims

Signal Processing, Volume 93, Issue 8

Pages 2140 - 2150

https://doi.org/10.1016/j.sigpro.2012.06.026

Published: 01 August 2013 Publication History

Abstract

Different from the existing work focusing on emotion type detection, the proposed approach in this paper provides flexibility for users to pick up their favorite affective content by choosing either emotion intensity levels or emotion types. Specifically, we propose a hierarchical structure for movie emotions and analyze emotion intensity and emotion type by using arousal and valence related features hierarchically. Firstly, three emotion intensity levels are detected by using fuzzy c-mean clustering on arousal features. Fuzzy clustering provides a mathematical model to represent vagueness, which is close to human perception. Then, valence related features are used to detect five emotion types. Considering video is continuous time series data and the occurrence of a certain emotion is affected by recent emotional history, conditional random fields (CRFs) are used to capture the context information. Outperforming Hidden Markov Model, CRF relaxes the independence assumption for states required by HMM and avoids bias problem. Experimental results show that CRF-based hierarchical method outperforms the one-step method on emotion type detection. User study shows that majority of the viewers prefer to have option of accessing movie content by emotion intensity levels. Majority of the users are satisfied with the proposed emotion detection.

References

[1]

Flickner, M., Sawhney, H., Niblack, W., Ashley, J., Qian, H., Dom, B., Gorkani, M., Hafiner, J., Lee, D., Petkovic, D., Steele, D. and Yanker, P., Query by image and video content: the {QBIC} System. IEEE Computer Special Issue on Content-Based Retrieval. v28 i9. 23-32.

[2]

J. Wei, Z.N. Liu, I. Gertner, A novel motion-based active video indexing method, Proceedings of IEEE International Conference on Multimedia Computing and System, 2 (1999) 460-465.

[3]

E. Ardizzo, M.L. Cascia, V.D. Gesu, C. Valenti, Content-based indexing of image and video databases by global and shape features, in: Proceedings of 13th International Conference on Patter Recognition, vol. 3, 1996, pp. 140-144.

[4]

Hanjalic, A., Shot-boundary detection: unravelled and resolved. IEEE Transactions on Circuits and Systems for Video Technology. v12 i2. 90-105.

[5]

Wang, M., Hua, X.-S., Tang, J. and Hong, R., Beyond distance measurement: constructing neighborhood similarity for video annotation. IEEE Transactions on Multimedia. v11 i3. 465-476.

[6]

Wang, M., Hua, X.-S., Hong, R., Tang, J., Qi, G.-J. and Song, Y., Unified video annotation via multi-graph learning, in: IEE. Transactions on Circuits and Systems for Video Technology. v19 i5. 733-746.

[7]

Duan, L.Y., Xu, M., Xu, C. and Tian, Q., A unified framework for semantic shot classification in sports video. IEEE Transactions on Multimedia. v7 i6. 1066-1083.

[8]

M. Xu, N.C. Maddage, C.-S. Xu, M. Kankanhalli, Q. Tian, Creating audio keywords for event detection in soccer video, Proceedings of International Conference on Multimedia & Expo, 2 (2003) 143-154.

[9]

S. Zhang, Q. Huang, Q. Tian, S. Jiang, W. Gao, i.MTV: an integrated system for MTV affective analysis, in: Proceedings of ACM International Conference on Multimedia, 2008, 985-986.

[10]

H.-B. Kang, Affective content detection using HMMs, in: Proceedings of the ACM Multimedia Conference, 2003, pp. 259-262.

[11]

Hanjalic, A. and Xu, L.Q., Affective video content representation and modeling. IEEE Transaction on Multimedia. v7 i1. 143-154.

[12]

Rasheed, Z., Sheikh, Y. and Shah, M., On the use of computable features for film classification. IEEE Transactions on Circuits and Systems for Video Technology. v15 i1. 52-64.

[13]

C. Chan, G.J.F. Jones, Affect-based indexing and retrieval of films, in: Proceedings of the ACM Multimedia Conference, 2005, pp. 427-430.

[14]

Hanjalic, A., Extracting moods from pictures and sounds: towards truly personalized TV. IEEE Signal Processing Magazine. v23 i2. 90-100.

[15]

Y.-H. Chen, J.-H. Kuo, W.-T. Chu, J.-L. Wu, Movie emotional event detection based on music mood and video tempo, in: Proceedings of IEEE International Conference on Consumer Electronics, 2006, pp. 151-152.

[16]

Wang, H.L. and Cheong, L.F., Affective understanding in film. IEEE Transactions on Circuits and Systems for Video Technology. v16 i6. 689-704.

[17]

S. Arifin, P.Y.K. Cheung, User attention based arousal content modeling, in: Proceedings of the International Conference on Image Processing, 2006, pp. 433-436.

[18]

S. Arifin, P.Y.K. Cheung, A computation method for video segmentation utilizing the pleasure-arousal-dominance emotional information, in: Proceedings of the ACM Multimedia Conference, 2007, pp. 68-77.

Digital Library

[19]

G. Irie, K. Hidaka, T.Satou, A. Kojima, T. Yamasaki, K. Aizawa, Latent topic driving model for movie affective scene classification, in: Proceedings of the ACM Multimedia Conference, 2009, 565-568.

[20]

M. Soleymani, J.J.M. Kierkels, G. Chanel, T. Pun, A bayesian framework for video affective representation, in: Proceedings of the International Conference on Affective Computing and Intelligent interaction, 2009.

[21]

M. Bradley, Emotions: essays on emotion theory, in: Lawrence Erlbaum, Hillsdale, NJ, Stephanie H.M. van Goozen, N.E. van de Poll, J. A. Sergeant, 1994, 97-134.

[22]

P. J. Lang, The network model of emotion: motivational connections, Perspectives on Anger and Emotion, Advances in Social Cognition, Psychology Press, in: Robert S. Wyer, Thomas K. Srull, (Eds.), 1993, pp. 109-133.

[23]

Russell, J.A. and Mehrabian, A., Evidence for A Three-factor theory of emotions. Journal of Research in Personality. v11 i3. 273-294.

[24]

Kasap, Z., Benmoussa, M., Chaudhuri, P. and Magnenat-Thalmann, N., Making them remember-emotional virtual characters with memory. IEEE Computer Graphics and Applications. v29 i2. 20-29.

[25]

Huang, T.S., Hasegawa-Johnson, M.A., Chu, S.M., Zeng, Z. and Tang, H., Sensitive talking heads. IEEE Signal Processing Magazine. v26 i4. 67-72.

[26]

Sebe, N., Lew, M.S., Sun, Y., Cohen, I., Gevers, T. and Huang, T.S., Authentic facial expression analysis. Image and Vision Computing. v25 i12. 1856-1863.

[27]

Shan, C., Gong, S. and McOwan, P.W., Facial expression recognition based on local binary patterns: a comprehensive study. Image and Vision Computing. v27 i6. 803-816.

[28]

A. Khalili, C. Wu, H. Aghajan, Autonomous learning of user's preference of music and light services in smart home, in: Proceedings of the behavior monitoring and interpretation workshop at German AI Conference, 2009.

[29]

S. Zhang, Q. Tian, Q. Huang, W. Gao, S. Li, Utilizing affective analysis for efficient movie browsing, in: Proceedings of IEEE International Conference on Image Processing, 2009, 1853-1856.

[30]

Bezdek, J.C., Pattern Recognition with Fuzzy Objective Function Algoritms. 1981. Kluwer Academic Publishers, Norwell, MA, USA.

[31]

M. Xu, J.S. Jin, S. Luo, L. Duan, Hierarchical movie affective content analysis based on arousal and valence features, in: Proceedings of the ACM Multimedia Conference, 2008, pp. 677-680.

[32]

Nash Information Services, LLC, The Numbers - Movie Box Office Data, Film Stars, Idle Speculation <http://www.the-numbers.com/charts/weekly/2009/20091225.php>, 2009.

[33]

Universals and cultural differences in the judgments of facial expressions of emotion. Journal of Personality and Social Psychology. v54 i4. 712-717.

[34]

Valdez, P. and Mehrabian, A., Effects of color on emotions. Journal of Experimental Psychology: General. v123 i4. 394-409.

[35]

Plantinga, C. and Smith, G.M., Passionate Views: Film. 1999. The Johns Hopkins University Press.

[36]

M. Xu, S. Luo, J.S. Jin, Affective content detection by using timing features and fuzzy clustering, in: Proceedings of the Advances in Multimedia Information Processing - PCM 2008, Lecture Notes in Computer Science, vol. 5353, 2008, pp. 685-692.

[37]

Picard, R.W., Affective Computing. 2000. The MIT Press, Cambridge, MA.

[38]

Zeng, Z., Tu, J., Liu, M., Huang, T.S., Pianfetti, B., Roth, D. and Levinson, S., Audio-visual affective recognition, IEE. Transaction on Multimedia. v9. 424-428.

[39]

Chong, E. and Zak, S., . 1996. Wiley, New York.

[40]

Chang, S.-F., Zhong, D. and Kumar, R., Real-time content-based adaptive streaming of sports video. Proceedings of the IEEE Workshop Content-Based Access to Video/Image Library.

Cited By

Makantasis KLiapis AYannakakis G(2023)The Pixels and Sounds of Emotion: General-Purpose Representations of Arousal in GamesIEEE Transactions on Affective Computing10.1109/TAFFC.2021.306087714:1(680-693)Online publication date: 1-Jan-2023
https://dl.acm.org/doi/10.1109/TAFFC.2021.3060877
Huynh VYang HLee GKim S(2023)Prediction of evoked expression from videos with temporal position fusionPattern Recognition Letters10.1016/j.patrec.2023.07.002172:C(245-251)Online publication date: 1-Aug-2023
https://dl.acm.org/doi/10.1016/j.patrec.2023.07.002
Ghaemmaghami PAbadi MKia SAvesani PSebe N(2022)Movie Genre Classification by Exploiting MEG Brain SignalsImage Analysis and Processing — ICIAP 201510.1007/978-3-319-23231-7_61(683-693)Online publication date: 10-Mar-2022
https://dl.acm.org/doi/10.1007/978-3-319-23231-7_61
Show More Cited By

Recommendations

Hierarchical movie affective content analysis based on arousal and valence features
MM '08: Proceedings of the 16th ACM international conference on Multimedia

Emotional factors directly reflect audiences' attention, evaluation and memory. Affective contents analysis not only create an index for users to access their interested movie segments, but also provide feasible entry for video highlights. Most of the ...
Biometric valence and arousal recognition
OZCHI '07: Proceedings of the 19th Australasian conference on Computer-Human Interaction: Entertaining User Interfaces

A real-time user-independent emotion detection system using physiological signals has been developed. The system has the ability to classify affective states into 2-dimensions using valence and arousal. Each dimension ranges from 1 to 5 giving a total ...
The Effect of Different Affective Arousal Levels on Taste Perception
ICMI '20 Companion: Companion Publication of the 2020 International Conference on Multimodal Interaction

The emotions we experience shape our perception, and our emotion is shaped by our perceptions. Taste perception is also influenced by emotions. Positive and negative emotions alter sweetness, sourness, and bitterness perception. However, most previous ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Signal Processing

Signal Processing Volume 93, Issue 8

August, 2013

259 pages

ISSN:0165-1684

Issue’s Table of Contents

Copyright © Elsevier B.V. © 2012.

Publisher

Elsevier North-Holland, Inc.

United States

Publication History

Published: 01 August 2013

Author Tags

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

18
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 30 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Makantasis KLiapis AYannakakis G(2023)The Pixels and Sounds of Emotion: General-Purpose Representations of Arousal in GamesIEEE Transactions on Affective Computing10.1109/TAFFC.2021.306087714:1(680-693)Online publication date: 1-Jan-2023
https://dl.acm.org/doi/10.1109/TAFFC.2021.3060877
Huynh VYang HLee GKim S(2023)Prediction of evoked expression from videos with temporal position fusionPattern Recognition Letters10.1016/j.patrec.2023.07.002172:C(245-251)Online publication date: 1-Aug-2023
https://dl.acm.org/doi/10.1016/j.patrec.2023.07.002
Ghaemmaghami PAbadi MKia SAvesani PSebe N(2022)Movie Genre Classification by Exploiting MEG Brain SignalsImage Analysis and Processing — ICIAP 201510.1007/978-3-319-23231-7_61(683-693)Online publication date: 10-Mar-2022
https://dl.acm.org/doi/10.1007/978-3-319-23231-7_61
Zhu YZhang WZhang MZhang KZhu Y(2021)Image emotion distribution learning based on enhanced fuzzy KNN algorithm with sparse learningJournal of Intelligent & Fuzzy Systems: Applications in Engineering and Technology10.3233/JIFS-21025141:6(6443-6460)Online publication date: 1-Jan-2021
https://dl.acm.org/doi/10.3233/JIFS-210251
Wang SWang CChen TWang YShu YJi Q(2021)Video Affective Content Analysis by Exploring Domain KnowledgeIEEE Transactions on Affective Computing10.1109/TAFFC.2019.291237712:4(1002-1017)Online publication date: 1-Oct-2021
https://dl.acm.org/doi/10.1109/TAFFC.2019.2912377
Liu XWang M(2020)Context-Aware Attention Network for Human Emotion Recognition in VideoAdvances in Multimedia10.1155/2020/88434132020Online publication date: 1-Jan-2020
https://dl.acm.org/doi/10.1155/2020/8843413
Yadav AVishwakarma D(2020)A unified framework of deep networks for genre classification using movie trailerApplied Soft Computing10.1016/j.asoc.2020.10662496:COnline publication date: 1-Nov-2020
https://dl.acm.org/doi/10.1016/j.asoc.2020.106624
Lee OJung J(2019)Modeling affective character network for story analyticsFuture Generation Computer Systems10.1016/j.future.2018.01.03092:C(458-478)Online publication date: 1-Mar-2019
https://dl.acm.org/doi/10.1016/j.future.2018.01.030
Liu AShi YJing PLiu JSu Y(2018)Structured low-rank inverse-covariance estimation for visual sentiment distribution predictionSignal Processing10.1016/j.sigpro.2018.06.001152:C(206-216)Online publication date: 1-Nov-2018
https://dl.acm.org/doi/10.1016/j.sigpro.2018.06.001
Chen TWang YWang SChen SLiu QLienhart RWang HChen SBoll SChen PFriedland GLi JYan S(2017)Exploring Domain Knowledge for Affective Video Content AnalysesProceedings of the 25th ACM international conference on Multimedia10.1145/3123266.3123352(769-776)Online publication date: 23-Oct-2017
https://dl.acm.org/doi/10.1145/3123266.3123352
Show More Cited By

View Options

View options

Media

Figures

Other

Tables

View Issue’s Table of Contents