Abstract
Music can trigger human emotion. This is a psychophysiological process. Therefore, using psychophysiological characteristics could be a way to understand individual music emotional experience. In this study, we explore a new method of personal music emotion recognition based on human physiological characteristics. First, we build up a database of features based on emotions related to music and a database based on physiological signals derived from music listening including EDA, PPG, SKT, RSP, and PD variation information. Then linear regression, ridge regression, support vector machines with three different kernels, decision trees, k-nearest neighbors, multi-layer perceptron, and Nu support vector regression (NuSVR) are used to recognize music emotions via a data synthesis of music features and human physiological features. NuSVR outperforms the other methods. The correlation coefficient values are 0.7347 for arousal and 0.7902 for valence, while the mean squared errors are 0.023 23 for arousal and 0.014 85 for valence. Finally, we compare the different data sets and find that the data set with all the features (music features and all physiological features) has the best performance in modeling. The correlation coefficient values are 0.6499 for arousal and 0.7735 for valence, while the mean squared errors are 0.029 32 for arousal and 0.015 76 for valence. We provide an effective way to recognize personal music emotional experience, and the study can be applied to personalized music recommendation.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Agrafioti F, Hatzinakos D, Anderson AK, 2012. ECG pattern analysis for emotion detection. IEEE Trans Affect Comput, 3(1):102–115. https://doi.org/10.1109/T-AFFC.2011.28
Ayadi ME, Kamel MS, Karray F, 2011. Survey on speech emotion recognition: features, classification schemes, and databases. Patt Recogn, 44(3):572–587. https://doi.org/10.1016/j.patcog.2010.09.020
Bradley MM, Lang PJ, 1994. Measuring emotion: the self-assessment manikin and the semantic differential. J Behav Ther Exp Psych, 25(1):49–59. https://doi.org/10.1016/0005-7916(94)90063-9
Chandler C, Cornes R, 2012. Biometric measurement of human emotions. TECHNIA - Int J Comput Sci Commun Technol, 4(2):164–168.
Cheng B, Liu GY, 2008. Emotion recognition from surface EMG signal using wavelet transform and neural network. J Comput Appl, 28(2):333–335 (in Chinese).
de Witte NAJ, Sütterlin S, Braet C, et al., 2017. Psychophysiological correlates of emotion regulation training in adolescent anxiety: evidence from the novel PIER task. J Affect Disord, 214:89–96. https://doi.org/10.1016/j.jad.2017.03.012
Gerdes ABM, Wieser MJ, Alpers GW, 2014. Emotional pictures and sounds: a review of multimodal interactions of emotion cues in multiple domains. Front Psychol, 5:1351. https://doi.org/10.3389/fpsyg.2014.01351
Katsis CD, Katertsidis NS, Fotiadis DI, 2011. An integrated system based on physiological signals for the assessment of affective states in patients with anxiety disorders. Biomed Signal Process Contr, 6(3):261–268. https://doi.org/10.1016/j.bspc.2010.12.001
Kim J, André E, 2008. Emotion recognition based on physiological changes in music listening. IEEE Trans Patt Anal Mach Intell, 30(12):2067–2083. https://doi.org/10.1109/TPAMI.2008.26
Koelstra S, Muhl C, Soleymani M, et al., 2012. DEAP: a database for emotion analysis; using physiological signals. IEEE Trans Affect Comput, 3(1):18–31. https://doi.org/10.1109/T-AFFC.2011.15
Krumhansl CL, 1997. An exploratory study of musical emotions and psychophysiology. Can J Exp Psychol, 51(4): 336–353. https://doi.org/10.1037/1196-1961.51.4.336
Li C, Xu C, Feng ZY, 2016. Analysis of physiological for emotion recognition with the IRS model. Neurocomputing, 178:103–111. https://doi.org/10.1016/j.neucom.2015.07.112
Li SS, Walters G, Packer J, et al., 2017. A comparative analysis of self-report and psychophysiological measures of emotion in the context of tourism advertising. J Travel Res, 57(8):1078–1092. https://doi.org/10.1177/0047287517733555
Lu Q, Chen XO, Yang DS, et al., 2010. Boosting for multi-modal music emotion classification. 11th Int Society for Music Information Retrieval Conf, p.105–110.
Maia CLB, Furtado ES, 2016. A study about psychophysiological measures in user experience monitoring and evaluation. Proc 15th Brazilian Symp on Human Factors in Computing Systems, p.7. https://doi.org/10.1145/3033701.3033708
Mitterschiffthaler MT, Fu CHY, Dalton JA, et al., 2007. A functional MRI study of happy and sad affective states induced by classical music. Hum Brain Mapp, 28(11):1150–1162. https://doi.org/10.1002/hbm.20337
Mori K, Iwanaga M, 2017. Two types of peak emotional responses to music: the psychophysiology of chills and tears. Sci Rep, 7:46063. https://doi.org/10.1038/srep46063
Nyklíček I, Thayer JF, van Doornen LJP, 1997. Cardiorespiratory differentiation of musically-induced emotions. J Psychophysiol, 11(4):304–321.
Pedregosa F, Varoquaux G, Gramfort A, et al., 2011. Scikitlearn: machine learning in Python. J Mach Learn Res, 12(10):2825–2830.
Picard RW, Vyzas E, Healey J, 2001. Toward machine emotional intelligence: analysis of affective physiological state. IEEE Trans Patt Anal Mach Intell, 23(10):1175–1191. https://doi.org/10.1109/34.954607
Rani P, Liu C, Sarkar N, et al., 2006. An empirical study of machine learning techniques for affect recognition in human-robot interaction. Patt Anal Appl, 9(1):58–69. https://doi.org/10.1007/s10044-006-0025-y
Ren P, Barreto A, Gao Y, et al., 2013. Affective assessment by digital processing of the pupil diameter. IEEE Trans Affect Comput, 4(1):2–14. https://doi.org/10.1109/T-AFFC.2012.25
Tzanetakis G, Cook P, 2000. MARSYAS: a framework for audio analysis. Organ Sound, 4(3):169–175. https://doi.org/10.1017/S1355771800003071
Wagner J, Kim J, Andre E, 2005. From physiological signals to emotions: implementing and comparing selected methods for feature extraction and classification. IEEE Int Conf on Multimedia and Expo, p.940–943. https://doi.org/10.1109/ICME.2005.1521579
Wang SH, Phillips P, Dong ZC, et al., 2018. Intelligent facial emotion recognition based on stationary wavelet entropy and Jaya algorithm. Neurocomputing, 272:668–676. https://doi.org/10.1016/j.neucom.2017.08.015
Wen WS, Liu GY, Cheng NP, et al., 2014. Emotion recognition based on multi-variant correlation of physiological signals. IEEE Trans Affect Comput, 5(2):126–140. https://doi.org/10.1109/TAFFC.2014.2327617
Xing BX, Zhang KJ, Sun SQ, et al., 2015. Emotion-driven Chinese folk music-image retrieval based on DE-SVM. Neurocomputing, 148:619-627. https://doi.org/10.1016/j.neucom.2014.08.007
Yang YH, Chen HH, 2011. Prediction of the distribution of perceived music emotions using discrete samples. IEEE Trans Audio Speech Lang Process, 19(7):2184–2196. https://doi.org/10.1109/TASL.2011.2118752
Yun T, Guan L, 2013. Human emotional state recognition using real 3D visual features from Gabor library. Patt Recogn, 46(2):529–538. https://doi.org/10.1016/j.patcog.2012.08.002
Zhang YD, Yang ZJ, Lu HM, et al., 2016. Facial emotion recognition based on biorthogonal wavelet entropy, fuzzy support vector machine, and stratified cross validation. IEEE Access, 4:8375–8385. https://doi.org/10.1109/ACCESS.2016.2628407
Zhu X, 2010. Emotion recognition of EMG based on BP neural network. Proc 2nd Int Symp on Networking and Network Security, p.227–229.
Author information
Authors and Affiliations
Corresponding author
Additional information
Project supported by the Philosophy and Social Science Planning Fund Project of Zhejiang Province, China (No. 20NDQN297YB) and the National Natural Science Foundation of China (No. 61702454)
Rights and permissions
About this article
Cite this article
Zhang, Lk., Sun, Sq., Xing, Bx. et al. Using psychophysiological measures to recognize personal music emotional experience. Frontiers Inf Technol Electronic Eng 20, 964–974 (2019). https://doi.org/10.1631/FITEE.1800101
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1631/FITEE.1800101