More Web Proxy on the site http://driver.im/

article

Multiple emotional tagging of multimedia data by exploiting dependencies among emotions

Authors:

Qiang JiAuthors Info & Claims

Multimedia Tools and Applications, Volume 74, Issue 6

Pages 1863 - 1883

https://doi.org/10.1007/s11042-013-1722-3

Published: 01 March 2015 Publication History

Abstract

Digital multimedia may elicit a mixture of human emotions. Most current emotional tagging research typically tags the multimedia data with a single emotion, ignoring the phenomenon of multi-emotion coexistence. To address this problem, we propose a novel multi-emotion tagging approach by explicitly modeling the dependencies among emotions. First, several audio or visual features are extracted from the multimedia data. Second, four traditional multi-label learning methods: Binary Relevance, Random k label sets, Binary Relevance k Nearest Neighbours and Multi-Label k Nearest Neighbours, are used as the classifiers to obtain the measurements of emotional tags. Then, a Bayesian network is automatically constructed to capture the relationships among emotional tags. Finally, the Bayesian network is used to infer the data's multi-emotion tags by combining the measurements obtained from those traditional methods with the dependencies among emotions. Experiments on two multi-label media data sets demonstrate the superiority of our approach to the existing methods.

References

[1]

Arifin S, Cheung PYK (2007) A novel probabilistic approach to modeling the pleasure-arousal-dominance content of the video based on "working memory". In: International Conference on Semantic Computing, ICSC 2007. IEEE, pp 147---154

Digital Library

[2]

Bhattacharya S, Sukthankar R, Shah M (2010) A framework for photo-quality assessment and enhancement based on visual aesthetics. In: Proceedings of the international conference on multimedia. ACM, pp 271---280

Digital Library

[3]

Bischoff K, Firan CS, Paiu R, Nejdl W, Laurier C, Sordo M (2009) Music mood and theme classification-a hybrid approach. In: Proceedings of the international conference on music information retrieval, pp 657---662

[4]

Datta R, Joshi D, Li J, Wang JZ (2006) Studying aesthetics in photographic images using a computational approach. In: Computer vision---ECCV 2006, pp 288---301

Digital Library

[5]

de Campos CP, Ji Q (2011) Efficient structure learning of Bayesian networks using constraints. J Mach Learn Res 12:663---689

Digital Library

[6]

Dorai C, Venkatesh S (2001) Computational media aesthetics: finding meaning beautiful. IEEE Multimedia 8(4):10---12

Digital Library

[7]

Eerola T, Lartillot O, Toiviainen P (2009) Prediction of multidimensional emotional ratings in music from audio using multivariate regression models. In: Proceedings of the international conference on music information retrieval, pp 621---626

[8]

Feng Y, Zhuang Y, Pan Y (2003) Popular music retrieval by detecting mood. In: Proceedings of the 26th annual international ACM SIGIR conference on research and development in informaion retrieval. ACM, pp 375---376

Digital Library

[9]

Fürnkranz J, Hüllermeier E, Loza Mencía E, Brinker K (2008) Multilabel classification via calibrated label ranking. Mach Learn 73(2):133---153

Digital Library

[10]

Ghamrawi N, McCallum A (2005) Collective multi-label classification. In: Proceedings of the 14th ACM international Conference on Information and Knowledge Management, CIKM '05. ACM, New York, pp 195---200

Digital Library

[11]

Godbole S, Sarawagi S (2004) Discriminative methods for multi-labeled classification. In: Proceedings of the 8th Pacific-Asia conference on knowledge discovery and data mining. Springer, pp 22---30

[12]

Gross JJ, Levenson RW (1995) Emotion elicitation using films. Cogn Emot 9(1):87---108

[13]

Hanjalic A, Xu LQ (2005) Affective video content representation and modeling. IEEE Trans Multimedia 7(1):143---154

Digital Library

[14]

Hevner K (1935) Expression in music: a discussion of experimental studies and theories. Psychol Rev 42(2):186

[15]

Holmes G, Read J, Pfahringer B, Frank E (2009) Classifier chains for multi-label classification. In: Proceedings of the European Conference on Machine Learning and Knowledge Discovery in Databases: part II, ECML PKDD '09. Springer, Berlin, Heidelberg, pp 254---269

Digital Library

[16]

Huang S-J, Yu Y, Zhou Z-H (2012) Multi-label hypothesis reuse. In: Proceedings of the 18th ACM SIGKDD international conference on Knowledge Discovery and Data mining, KDD '12. ACM, New York, pp 525---533

Digital Library

[17]

Joshi D, Datta R, Fedorovskaya E, Luong Q-T, Wang JZ, Li J, Luo J (2011) Aesthetics and emotions in images. IEEE Signal Process Mag 28(5):94---115

[18]

Kang HB (2003) Affective contents retrieval from video with relevance feedback. In: Sembok TMT, Zaman HB, Chen H, Urs SR, Myaeng S-H (eds) Digital libraries: technology and management of indigenous knowledge for global access. Lecture Notes in Computer Science, vol. 2911. Springer Berlin Heidelberg, pp 243---252

[19]

Kim EY, Kim SJ, Koo HJ, Jeong K, Kim JI (2005) Emotion-based textile indexing using colors and texture. In: Wang L, Jin Y (eds) Fuzzy systems and knowledge discovery. Lecture Notes in Computer Science, vol. 3613. Springer Berlin Heidelberg, pp 1077---1080

Digital Library

[20]

Kim YE, Schmidt EM, Migneco R, Morton BG, Richardson P, Scott J, Speck JA, Turnbull D (2010) Music emotion recognition: a state of the art review. In: Proc. ISMIR. Citeseer, pp 255---266

[21]

Li D, Sethi IK, Dimitrova N, McGee T (2001) Classification of general audio data for content-based retrieval. Pattern Recogn Lett 22(5):533---544

Digital Library

[22]

Li T, Ogihara M (2003) Detecting emotion in music. In: Proceedings of the international symposium on music information retrieval. Washington, pp 239---240

[23]

Liu D, Lu L, Zhang H-J (2003) Automatic mood detection from acoustic music data. In: Proceedings of the international symposium on music information retrieval, pp 81---87

[24]

Liu C-C, Yang Y-H, Wu P-H, Chen HH (2006) Detecting and classifying emotion in popular music. In: Proceedings of the joint international conference on information sciences, pp 996---999

[25]

Luca C, Sergio B, Riccardo L (2013) Affective recommendation of movies based on selected connotative features. IEEE Trans Circ Syst Video Technol 23(4):636---647

Digital Library

[26]

MacDorman KF, Ough S, Hoa C-C (2007) Automatic emotion prediction of song excerpts: index construction, algorithm design, and empirical comparison. J New Music Res 36(4):281---299

[27]

Marcelino R, Teixeira A, Yamasaki T, Aizawa K (2011) Determination of emotional content of video clips by low-level audiovisual features. Multimed Tools Appl 61(1):21---49

[28]

Moncrieff S, Dorai C, Venkatesh S (2001) Affect computing in film through sound energy dynamics. In: Proceedings of the ninth ACM international conference on multimedia. ACM, pp 525---527

Digital Library

[29]

Myint EEP, Pwint M (2010) An approach for mulit-label music mood classification. In: 2nd International Conference on Signal Processing Systems, (ICSPS) 2010, vol 1. IEEE, pp 290---294

[30]

Oliveira E, Martins P, Chambel T (2011) Ifelt: accessing movies through our emotions. In: Proceddings of the 9th international interactive conference on interactive television, EuroITV '11. ACM, pp 105---114

Digital Library

[31]

Pearl J (1988) Probabilistic reasoning in intelligent systems: networks of plausible inference. Morgan Kaufmann

Digital Library

[32]

Philippot P (1993) Inducing and assessing differentiated emotion-feeling states in the laboratory. Cogn Emot 7(2):171---193

[33]

Russell JA (1997) Reading emotion from and into faces: resurrecting a dimensional-contextual perspective. In: Russell JA, Fernández-Dols JM (eds) The psychology of facial expression. Studies in emotion and social interaction, 2nd series. Cambridge University Press, New York, NY, pp 295---320

[34]

Sanden C, Zhang JZ (2011) An empirical study of multi-label classifiers for music tag annotation. In: 12th international society for music information retrieval conference, (ISMIR 2011). Citeseer, pp 717---722

[35]

Santos AM, Canuto AMP, Neto AF (2011) A comparative analysis of classification methods to multi-label tasks in different application domains. Int J Comput Inform Syst Ind Manage Appl. 3:218---227

[36]

Schapire RE, Singer Y (2000) Boostexter: a boosting-based system for text categorization. Mach Learn 39(2):135---168

Digital Library

[37]

Schuller B, Johannes D, Gerhard R (2010) Determination of nonprototypical valence and arousal in popular music: features and performances. EURASIP Journal on Audio, Speech, and Music Processing 2010(735854):1---19.

Digital Library

[38]

Shibata T, Kato T (1999) "Kansei" image retrieval system for street landscape-discrimination and graphical parameters based on correlation of two images. In: IEEE international conference on systems, man, and cybernetics, 1999. IEEE SMC'99 conference proceedings, vol 6. IEEE, pp 247---252

[39]

Smith CA (1989) Dimensions of appraisal and physiological response in emotion. J Personal Soc Psychol 56(3):339

[40]

Smith CA, Lazarus RS (1991) Emotion and adaptation. Oxford University Press, New York

[41]

Sorower MS (2010) A literature survey on algorithms for multi-label learning. Ph.D Qualifying Review Paper, Oregon State University

[42]

Spyromitros E, Tsoumakas G, Vlahavas I (2008) An empirical study of lazy multilabel classification algorithms. In: Darzentas J, Vouros GA, Vosinakis S, Arnellos A (eds) Artificial intelligence: theories, models and applications. Lecture Notes in Computer Science, vol. 5138. Springer Berlin Heidelberg, pp 401---406

Digital Library

[43]

Sun K, Yu J (2007) Video affective content representation and recognition using video affective tree and hidden markov models. In: Paiva ACR, Prada R, Picard RW (eds) Affective computing and intelligent interaction. Lecture Notes in Computer Science, vol. 4738. Springer Berlin Heidelberg, pp 594---605

Digital Library

[44]

Sun L, Ji S, Ye J (2008) Hypergraph spectral learning for multi-label classification. In: Proceedings of the 14th ACM SIGKDD international conference on Knowledge Discovery and Data mining, KDD '08. ACM, New York, pp 668---676

Digital Library

[45]

Trohidis K, Tsoumakas G, Kalliris G, Vlahavas I (2008) Multi-label classification of music into emotions. In: ISMIR 2008: proceedings of the 9th international conference of music information retrieval, pp 325---330

[46]

Tsoumakas G, Vlahavas I (2007) Random k-labelsets: An ensemble method for multilabel classification. In: Proceedings of the 18th European conference on machine learning, pp 406---417

Digital Library

[47]

Tsoumakas G, Katakis I, Vlahavas I (2010) Mining multi-label data. In: Maimon O, Rokach L (eds) Data Mining and Knowledge Discovery Handbook, 2nd edn. Springer, New York, pp 667---685

[48]

Wang HL, Cheong L-F (2006) Affective understanding in film. IEEE Trans Circ Syst Video Technol 16(6):689---704

Digital Library

[49]

Wang CW, Cheng WH, Chen JC, Yang SS, Wu JL (2006) Film narrative exploration through the analysis of aesthetic elements. In: Cham TJ, Cai J, Dorai C, Rajan D, Chua TS, Chia LT (eds) Advances in multimedia modeling. Lecture Notes in Computer Science, vol. 4351. Springer Berlin Heidelberg, pp 606---615

Digital Library

[50]

Wang W, He Q (2008) A survey on emotional semantic image retrieval. In: 15th IEEE International Conference on Image Processing, ICIP 2008. IEEE, pp 117---120

[51]

Wang S, Wang X (2010) Emotional semantic detection from multimedia: a brief overview. In: Dai Y, Chakraborty B, Shi M (eds) Kansei engineering and soft computing: theory and practice 2010. IGI press, pp 126---146

[52]

Wang Z, Wang S, He M, Ji Q (2013) Emotional tagging of videos by exploring multi-emotion coexistence. In: IEEE international conference on automatic face & gesture recognition and workshops, (FG 2013). IEEE

[53]

Watanapa SC, Thipakorn B, Charoenkitarn N(2008) A sieving ANN for emotion-based movie clip classification. IEICE TRans Inf Syst 91(5):1562---1572

Digital Library

[54]

Wei CY, Dimitrova N, Chang SF (2004) Color-mood analysis of films based on syntactic and psychological models. In: IEEE International Conference on Multimedia and Expo, ICME'04, vol 2. IEEE, pp 831---834

[55]

Wei-ning W, Ying-lin Y, Sheng-ming J (2006) Image retrieval by emotional semantics: A study of emotional space and feature extraction. In: IEEE international conference on Systems, Man and Cybernetics, SMC'06, vol 4. IEEE, pp 3534---3539

[56]

Winoto P, Tang TY (2010) The role of user mood in movie recommendations. Expert Syst Appl 37(8):6086---6092

Digital Library

[57]

Wu T-L, Jeng S-K (2008) Probabilistic estimation of a novel music emotion model. In: Satoh S, Nack F, Etoh M (eds) Advances in Multimedia Modeling. Lecture Notes in Computer Science, vol. 4903. Springer Berlin Heidelberg, pp 487---497

Digital Library

[58]

Xu M, He X, Jin JS, Peng Y, Xu C, Guo W (2011) Using scripts for affective content retrieval.In: Qiu G, Lam KM, Kiya H, Xue XY, Kuo CCJ, Lew MS (eds) Advances in Multimedia Information Processing - PCM 2010. Lecture Notes in Computer Science, vol. 6298. Springer Berlin Heidelberg, pp 43---51

Digital Library

[59]

Xu M, Xu C, He X, Jin JS, Luo S, Rui Y (2013) Hierarchical affective content analysis in arousal and valence dimensions. Signal Process 93(8):2140---2150

Digital Library

[60]

Yang Y-H, Lin Y-C, Su Y-F, Chen HH (2007) Music emotion classification: a regression approach. In: IEEE international conference on multimedia and expo, 2007. IEEE, pp 208---211

[61]

Yang Y-H, Lin Y-C, Su Y-F, Chen HH (2008) A regression approach to music emotion recognition. IEEE Trans Audio Speech Lang Process 16(2):448---457

Digital Library

[62]

Yang Yh, Chen HH (2012) Machine recognition of music emotion: a review. ACM Trans Intell Syst Technol 3(3):40

Digital Library

[63]

Yoo HW, Cho SB(2007) Video scene retrieval with interactive genetic algorithm. Multimed Tools Appl 34(3):317---336

Digital Library

[64]

Zhang M-L, Zhou Z-H (2007) Ml-knn: a lazy learning approach to multi-label learning. Pattern Recogn 40(7):2038---2048

Digital Library

[65]

Zhang S, Tian Q, Jiang S, Huang Q, Gao W (2008) Affective mtv analysis based on arousal and valence features. In: IEEE international conference on multimedia and expo, 2008. IEEE, pp 1369---1372

[66]

Zhang S, Huang Q, Jiang S, Gao W, Tian Q (2010) Affective visualization and retrieval for music video. IEEE Trans Multimedia 12(6):510---522

Digital Library

[67]

Zhang M, Zhang K (2010) Multi-label learning by exploiting label dependency. In: Proceedings of the 16th ACM SIGKDD international conference on knowledge discovery and data mining. ACM, pp 999---1008

Digital Library

Cited By

Orjesek RJarina RChmulik M(2022)End-to-end music emotion variation detection using iteratively reconstructed deep featuresMultimedia Tools and Applications10.1007/s11042-021-11584-781:4(5017-5031)Online publication date: 1-Feb-2022
https://dl.acm.org/doi/10.1007/s11042-021-11584-7
Wang SPeng GZheng ZXu Z(2021)Capturing Emotion Distribution for Multimedia Emotion TaggingIEEE Transactions on Affective Computing10.1109/TAFFC.2019.290024012:4(821-831)Online publication date: 1-Oct-2021
https://dl.acm.org/doi/10.1109/TAFFC.2019.2900240
Xu ZWang SWang CWen Chen CCucchiara RHua XQi GRicci EZhang ZZimmermann R(2020)Exploiting Multi-Emotion Relations at Feature and Label Levels for Emotion TaggingProceedings of the 28th ACM International Conference on Multimedia10.1145/3394171.3413506(2955-2963)Online publication date: 12-Oct-2020
https://dl.acm.org/doi/10.1145/3394171.3413506
Show More Cited By

Multiple emotional tagging of multimedia data by exploiting dependencies among emotions
1. Computing methodologies

Recommendations

Multiple Emotion Tagging for Multimedia Data by Exploiting High-Order Dependencies Among Emotions
In this paper, a novel approach of multiple emotional multimedia tagging is proposed, which explicitly models the higher-order relations among emotions. First, multimedia features are extracted from the multimedia data. Second, a traditional multi-label ...
Emotional design and positive emotions in multimedia learning

The present eyetracking study examined the influence of emotions on learning with multimedia. Based on a 2 ý2 experimental design, participants received experimentally induced emotions (positive vs. neutral) and then learned with a multimedia ...
Capturing dependencies among labels and features for multiple emotion tagging of multimedia data
AAAI'17: Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence

In this paper, we tackle the problem of emotion tagging of multimedia data by modeling the dependencies among multiple emotions in both the feature and label spaces. These dependencies, which carry crucial top-down and bottom-up evidence for improving ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Multimedia Tools and Applications

Multimedia Tools and Applications Volume 74, Issue 6

March 2015

459 pages

ISSN:1380-7501

Issue’s Table of Contents

Copyright © Copyright © 2015 Springer Science+Business Media New York.

Publisher

Kluwer Academic Publishers

United States

Publication History

Published: 01 March 2015

Author Tags

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

12
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 26 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Orjesek RJarina RChmulik M(2022)End-to-end music emotion variation detection using iteratively reconstructed deep featuresMultimedia Tools and Applications10.1007/s11042-021-11584-781:4(5017-5031)Online publication date: 1-Feb-2022
https://dl.acm.org/doi/10.1007/s11042-021-11584-7
Wang SPeng GZheng ZXu Z(2021)Capturing Emotion Distribution for Multimedia Emotion TaggingIEEE Transactions on Affective Computing10.1109/TAFFC.2019.290024012:4(821-831)Online publication date: 1-Oct-2021
https://dl.acm.org/doi/10.1109/TAFFC.2019.2900240
Xu ZWang SWang CWen Chen CCucchiara RHua XQi GRicci EZhang ZZimmermann R(2020)Exploiting Multi-Emotion Relations at Feature and Label Levels for Emotion TaggingProceedings of the 28th ACM International Conference on Multimedia10.1145/3394171.3413506(2955-2963)Online publication date: 12-Oct-2020
https://dl.acm.org/doi/10.1145/3394171.3413506
Terroso-Saenz FMuñoz A(2020)Land use discovery based on Volunteer Geographic Information classificationExpert Systems with Applications: An International Journal10.1016/j.eswa.2019.112892140:COnline publication date: 1-Feb-2020
https://dl.acm.org/doi/10.1016/j.eswa.2019.112892
Rao TXu MLiu H(2018)Generating affective maps for imagesMultimedia Tools and Applications10.1007/s11042-017-5289-277:13(17247-17267)Online publication date: 1-Jul-2018
https://dl.acm.org/doi/10.1007/s11042-017-5289-2
Brito PStoyanova JCoelho A(2018)Augmented reality versus conventional interfaceMultimedia Tools and Applications10.1007/s11042-017-4658-177:6(7487-7516)Online publication date: 1-Mar-2018
https://dl.acm.org/doi/10.1007/s11042-017-4658-1
Lai L(2018)How Do They Tag? Senior Adults’ Tagging Behavior in Cultural Heritage InformationHCI in Business, Government, and Organizations10.1007/978-3-319-91716-0_38(475-484)Online publication date: 15-Jul-2018
https://dl.acm.org/doi/10.1007/978-3-319-91716-0_38
Wu SWang SJi QSingh SMarkovitch S(2017)Capturing dependencies among labels and features for multiple emotion tagging of multimedia dataProceedings of the Thirty-First AAAI Conference on Artificial Intelligence10.5555/3298239.3298390(1026-1032)Online publication date: 4-Feb-2017
https://dl.acm.org/doi/10.5555/3298239.3298390
Zhang SJia JNing Y(2017)Inferring emotions from heterogeneous social media data: A Cross-media Auto-Encoder solution2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)10.1109/ICASSP.2017.7952685(2891-2895)Online publication date: 5-Mar-2017
https://dl.acm.org/doi/10.1109/ICASSP.2017.7952685
Chiu MKo L(2017)Develop a personalized intelligent music selection system based on heart rate variability and machine learningMultimedia Tools and Applications10.1007/s11042-016-3860-x76:14(15607-15639)Online publication date: 1-Jul-2017
https://dl.acm.org/doi/10.1007/s11042-016-3860-x
Show More Cited By

View Options

View options

Media

Figures

Other

Tables

View Issue’s Table of Contents