Abstract
Frustration is a common response during game interactions, typically decreasing a user’s engagement and leading to game failure. Artificially intelligent methods capable to automatically detect a user’s level of frustration at an early stage are hence of great interest for game designers, since this would enable optimisation of a player’s experience in real-time. Nevertheless, research in this context is still in its infancy, mainly relying on the use of pre-trained models and fine-tuning tailored to a specific dataset. Furthermore, this lack in research is due to the limited data available and to the ambiguous labelling of frustration, which leads to outcomes which are not generalisable in the real-world. Meanwhile, contrastive loss has been considered instead of the traditional cross-entropy loss in a variety of machine learning applications, showing to be more robust for system stability alternative in self-supervised learning. Following this trend, we hypothesise that using a supervised contrastive loss might overcome the limitations of the cross-entropy loss yielded by the labels’ ambiguity. In fact, our experiments demonstrate that using the supervised contrastive method as a loss function, results improve for the automatic recognition (binary frustration vs no-frustration) of game-induced frustration from speech with an Unweighted Average Recall increase from 86.4 % to 89.9 %.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Becker, S., Hinton, G.E.: Self-organizing neural network that discovers surfaces in random-dot stereograms. Nature 355(6356), 161–163 (1992)
Bevan, N.: What is the difference between the purpose of usability and user experience evaluation methods. In: Proceedings of the Workshop UXEM, pp. 1–4. Uppsala, Sweden (2009)
Byrd, D., McLaughlin, M., Khurana, S., Landes, M., Ucar, T.: Chimp: Children interacting with machines project
Ceylan, C., Gutmann, M.U.: Conditional noise-contrastive estimation of unnormalised models. In: Proceedings of the International Conference on Machine Learning, pp. 726–734. Vienna, Austria (2018)
Chen, T., Kornblith, S., Norouzi, M., Hinton, G.: A simple framework for contrastive learning of visual representations. In: Proceedings of the International Conference on Machine Learning, pp. 1597–1607. Virtual (2020)
Chopra, S., Hadsell, R., LeCun, Y.: Learning a similarity metric discriminatively, with application to face verification. In: Proceedings of the Computer Society Conference on Computer Vision and Pattern Recognition. vol. 1, pp. 539–546. San Diego, USA (2005)
DeFalco, J.A., Rowe, J.P., Paquette, L., Georgoulas-Sherry, V., Brawner, K., Mott, B.W., Baker, R.S., Lester, J.C.: Detecting and addressing frustration in a serious game for military training. Int. J. Artif. Intell. Educ. 28(2), 152–193 (2018)
Ekman, P., Keltner, D.: Universal facial expressions of emotion. In: Segerstrale U, P. Molnar, P., (eds.) Nonverbal communication: Where nature meets culture vol. 54 no. 2, pp. 27–46 (1997)
Franz, O., Drewitz, U., Ihme, K.: Facing driver frustration: towards real-time in-vehicle frustration estimation based on video streams of the face. In: Proceedings of the International Conference on Human-Computer Interaction, pp. 349–356. Virtual (2020)
Goetsu, S., Sakai, T.: Different types of voice user interface failures may cause different degrees of frustration. arXiv preprint arXiv:2002.03582 (2020)
Grafsgaard, J.F., Wiggins, J.B., Boyer, K.E., Wiebe, E.N., Lester, J.C.: Automatically recognizing facial indicators of frustration: a learning-centric analysis. In: Proceedings of the Humaine Association Conference on Affective Computing and Intelligent Interaction, pp. 159–165. Geneva, Switzerland (2013)
Grafsgaard, J.F., Wiggins, J.B., Vail, A.K., Boyer, K.E., Wiebe, E.N., Lester, J.C.: The additive value of multimodal features for predicting engagement, frustration, and learning during tutoring. In: Proc. International Conference on Multimodal Interaction, pp. 42–49. Istanbul, Turkey (2014)
Grewe, L., Hu, C.: Ulearn: understanding and reacting to student frustration using deep learning, mobile vision and nlp. In: Proceedings of the Signal Processing, Sensor/Information Fusion, and Target Recognition XXVIII. p. 110. Maryland, USA (2019)
Gutmann, M., Hyvärinen, A.: Noise-contrastive estimation: a new estimation principle for unnormalized statistical models. In: Proceedings of the International Conference on Artificial Intelligence and Statistics, pp. 297–304. Sardinia, Italy (2010)
Hadsell, R., Chopra, S., LeCun, Y.: Dimensionality reduction by learning an invariant mapping. In: Proceedings of the Computer Society Conference on Computer Vision and Pattern Recognition, pp. 1735–1742. New York, USA (2006)
Hansen, J.H., Busso, C., Zheng, Y., Sathyanarayana, A.: Driver modeling for detection and assessment of driver distraction: examples from the utdrive test bed. IEEE Signal Process. Mag. 34(4), 130–142 (2017)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the Computer Vision and Pattern Recognition, pp. 770–778. Las Vegas, USA (2016)
Henderson, N.L., Rowe, J.P., Mott, B.W., Brawner, K., Baker, R., Lester, J.C.: 4d affect detection: improving frustration detection in game-based learning with posture-based temporal data fusion. In: Proceedings of the Artificial Intelligence in Education, pp. 144–156. Beijing, China (2019)
Hertzum, M.: Frustration: a common user experience. DHRS2010 p. 11 (2010)
Hjelm, R.D., Fedorov, A., Lavoie-Marchildon, S., Grewal, K., Bachman, P., Trischler, A., Bengio, Y.: Learning deep representations by mutual information estimation and maximization. arXiv preprint arXiv:1808.06670 (2018)
Inoue, N., Goto, K.: Semi-supervised contrastive learning with generalized contrastive loss and its application to speaker recognition. In: Proceedings of the Asia-Pacific Signal and Information Processing Association Annual Summit and Conference, pp. 1641–1646. Virtual (2020)
Jaiswal, A., Babu, A.R., Zadeh, M.Z., Banerjee, D., Makedon, F.: A survey on contrastive self-supervised learning. Technologies 9(1), 2 (2021)
Khosla, P., et al.: Supervised contrastive learning. arXiv preprint arXiv:2004.11362 (2020)
Le-Khac, P.H., Healy, G., Smeaton, A.F.: Contrastive representation learning: a framework and review. IEEE Access (2020)
Macary, M., Tahon, M., Estève, Y., Rousseau, A.: Allosat: a new call center French corpus for satisfaction and frustration analysis. In: Proceedings of the Language Resources and Evaluation Conference, pp. 1590–1597. Virtual (2020)
Malta, L., Miyajima, C., Kitaoka, N., Takeda, K.: Analysis of real-world driver’s frustration. IEEE Trans. Intell. Transp. Syst. 12(1), 109–118 (2010)
Mnih, A., Teh, Y.W.: A fast and simple algorithm for training neural probabilistic language models. arXiv preprint arXiv:1206.6426 (2012)
Oehl, M., Ihme, K., Drewitz, U., Pape, A.A., Cornelsen, S., Schramm, M.: Towards a frustration-aware assistant for increased in-vehicle UX: F-RELACS. In: Proceedings of the Automotive User Interfaces and Interactive Vehicular Applications, pp. 260–264. Utrecht, Netherlands (2019)
Oh Song, H., Xiang, Y., Jegelka, S., Savarese, S.: Deep metric learning via lifted structured feature embedding. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4004–4012 (2016)
Parada-Cabaleiro, E., Batliner, A., Baird, A., Schuller, B.: The perception of emotional cues by children in artificial background noise. Int. J. Speech Technol. 23(1), 169–182 (2020). https://doi.org/10.1007/s10772-020-09675-1
Parada-Cabaleiro, E., Costantini, G., Batliner, A., Schmitt, M., Schuller, B.W.: DEMoS: an Italian emotional speech corpus: elicitation methods, machine learning, and perception. Lang. Resour. Eval. 54, 341–383 (2020)
Paszke, A., et al.: Automatic differentiation in PyTorch. In: NIPS-W (2017)
Schuller, B., Batliner, A.: Computational paralinguistics: emotion, affect and personality in speech and language processing. Wiley, Sussex, UK (2014)
Shoumy, N.J., Ang, L.M., Seng, K.P., Rahaman, D.M., Zia, T.: Multimodal big data affective analytics: a comprehensive survey using text, audio, visual and physiological signals. J. Netw. Comput. Appl. 149, 102447 (2020)
Song, M., et al.: Frustration recognition from speech during game interaction using wide residual networks. Virtual Reality & Intelligent Hardware 10, (2020)
Song, M., et al.: Predicting group work performance from physical handwriting features in a smart English classroom. In: Proceedings of the International Conference on Digital Signal Processing (ICDSP). Chengdu, China (2021)
Song, M., et al.: Audiovisual analysis for recognising frustration during game-play: introducing the multimodal game frustration database. In: Proceedings of the Affective Computing and Intelligent Interaction, pp. 517–523. Cambridge, the UK (2019)
Tian, Y., Sun, C., Poole, B., Krishnan, D., Schmid, C., Isola, P.: What makes for good views for contrastive learning. arXiv preprint arXiv:2005.10243 (2020)
Tyng, C.M., Amin, H.U., Saad, M.N., Malik, A.S.: The influences of emotion on learning and memory. Front. Psychol. 8, 1454 (2017)
Valdez, M.G., Hernández-Águila, A., Guervós, J.J.M., Soto, A.M.: Enhancing student engagement via reduction of frustration with programming assignments using machine learning. In: Proceedings of the International Joint Conference on Computational Intelligence, pp. 297–304. Funchal, Portugal (2017)
Wei, M., Su, J.C., Carrera, S., Lin, S.P., Yi, F.: Suppression and interpersonal harmony: a cross-cultural comparison between chinese and european americans. J. Couns. Psychol. 60(4), 625 (2013)
Xiao, T., Wang, X., Efros, A.A., Darrell, T.: What should not be contrastive in contrastive learning. arXiv preprint arXiv:2008.05659 (2020)
Zepf, S., Stracke, T., Schmitt, A., van de Camp, F., Beyerer, J.: Towards real-time detection and mitigation of driver frustration using SVM. In: Proceedings of the Machine Learning and Applications, pp. 202–209. Florida, USA (2019)
Zhang, M., Ihme, K., Drewitz, U.: Discriminating drivers’ fear and frustration through the dimension of power. In: Proceedings of the Humanist Conference, p. 98. Hague, Netherlands (2018)
Zhang, Z., Sabuncu, M.R.: Generalized cross entropy loss for training deep neural networks with noisy labels. arXiv preprint arXiv:1805.07836 (2018)
Zweig, G., Burges, C.J.: The microsoft research sentence completion challenge. Microsoft Research, Redmond, WA, USA, Technical report. MSR-TR-2011-129 (2011)
Acknowlwdgement
The authors acknowledge funding from the German Research Foundation (DFG) under the Reinhart Koselleck Project grant AUDI0NOMOUS (No. 442218748). The responsibility lies with the authors.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Song, M. et al. (2021). Supervised Contrastive Learning for Game-Play Frustration Detection from Speech. In: Antona, M., Stephanidis, C. (eds) Universal Access in Human-Computer Interaction. Design Methods and User Experience. HCII 2021. Lecture Notes in Computer Science(), vol 12768. Springer, Cham. https://doi.org/10.1007/978-3-030-78092-0_43
Download citation
DOI: https://doi.org/10.1007/978-3-030-78092-0_43
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-78091-3
Online ISBN: 978-3-030-78092-0
eBook Packages: Computer ScienceComputer Science (R0)