More Web Proxy on the site http://driver.im/

Article

Predicting Co-occurring Emotions from Eye-Tracking and Interaction Data in MetaTutor

Authors:

Sébastien Lallé,

Cristina Conati,

Roger AzevedoAuthors Info & Claims

Artificial Intelligence in Education: 22nd International Conference, AIED 2021, Utrecht, The Netherlands, June 14–18, 2021, Proceedings, Part I

Pages 241 - 254

https://doi.org/10.1007/978-3-030-78292-4_20

Published: 14 June 2021 Publication History

Abstract

Emotions in Intelligent Tutoring Systems (ITS) are often modeled as single affective states, however there is evidence that emotions co-occur during learning, with implications for affect-aware ITS that need to have a comprehensive understanding of a student’s affective state to react accordingly. In this paper we broaden the evidence that emotions co-occur in an educational context, and present a first attempt to predict these co-occurrences from data, using the MetaTutor ITS as a test-bed. We show that boredom+frustration, as well as curiosity+anxiety, frequently co-occur in MetaTutor, and that we can predict when these emotions co-occur significantly better than a baseline using eye-tracking and interaction data. These findings provide a first step toward building affect-aware ITS that can adapt to these complex co-occurring affective states.

References

[1]

Wortha, F., Azevedo, R., Taub, M., Narciss, S.: Multiple negative emotions during learning with digital learning environments–Evidence on their detrimental effect on learning from two methodological approaches. Front. Psychol. 10, 2678:1–2678:19 (2019).

[2]

Baker R, D’Mello S, Rodrigo MM, and Graesser AC Better to be frustrated than bored: The incidence, persistence, and impact of learners’ cognitive–affective states during interactions with three different computer-based learning environments Int. J. Hum.-Comput. Stud. 2010 68 223-241

[3]

Woolf B, Burleson W, Arroyo I, Dragon T, Cooper D, and Picard R Affect-aware tutors: recognising and responding to student affect Int. J. Learn. Technol. 2009 4 129-164

[4]

Grawemeyer, B., Mavrikis, M., Holmes, W., Gutierrez-Santos, S., Wiedmann, M., Rummel, N.: Affecting off-task behaviour: how affect-aware feedback can improve student learning. In: Proceedings of the Sixth International Conference on Learning Analytics & Knowledge. pp. 104–113. ACM, Edinburgh (2016).

[5]

Lallé, S., Conati, C., Azevedo, R.: Prediction of student achievement goals and emotion valence during interaction with pedagogical agents. In: Proceedings of the 17th International Conference on Autonomous Agents and Multiagent Systems, pp. 1222–1231. IFAAMAS, Stockholm (2018).

[6]

Salmeron-Majadas S, Santos OC, and Boticario JG An evaluation of mouse and keyboard interaction indicators towards non-intrusive and low cost affective modeling in an educational context Procedia Comput. Sci. 2014 35 691-700

[7]

Jaques N, Conati C, Harley JM, and Azevedo R Trausan-Matu S, Boyer KE, Crosby M, and Panourgia K Predicting affect from gaze data during interaction with an intelligent tutoring system Intelligent Tutoring Systems 2014 Cham Springer 29-38

[8]

Paquette L et al. Trausan-Matu S, Boyer KE, Crosby M, Panourgia K, et al. Sensor-free affect detection for a simulation-based science inquiry learning environment Intelligent Tutoring Systems 2014 Cham Springer 1-10

[9]

Sabourin J, Mott B, and Lester JC D’Mello S, Graesser A, Schuller B, and Martin J-C Modeling learner affect with theoretically grounded dynamic bayesian networks Affective Computing and Intelligent Interaction 2011 Heidelberg Springer 286-295

[10]

Baker, R.S., et al.: Towards sensor-free affect detection in cognitive tutor algebra. In: Proceedings of the 5th International Conference on Educational Data Mining, pp. 126–133. IEDMS, Montréal (2012)

[11]

Wixon, M., Arroyo, I., Muldner, K., Burleson, W., Rai, D., Woolf, B.: The opportunities and limitations of scaling up sensor-free affect detection. In: Proceedings of the International Conference on Educational Data Mining, pp. 145–152. IEDMS, London (2014)

[12]

Litman, D.J., Forbes-Riley, K.: Predicting student emotions in computer-human tutoring dialogues. In: Proceedings of the Annual Meeting on Association for Computational Linguistics, pp. 351–358, Barcelona, Spain (2004).

[13]

Bosch, N., D’Mello, S.: Co-occurring affective states in automated computer programming education. In: Proceedings of the Workshop on AI-supported Education for Computer Science (AIEDCS) at the 12th International Conference on Intelligent Tutoring Systems, pp. 21–30 (2014)

[14]

Dillon, J., et al.: Student emotion, co-occurrence, and dropout in a MOOC context. In: Proceedings of the 9th International Conference on Educational Data Mining, pp. 353–357. IEDMS, Raleigh (2016)

[15]

Harley JM, Bouchet F, and Azevedo R Cerri SA, Clancey WJ, Papadourakis G, and Panourgia K Measuring learners’ co-occurring emotional responses during their interaction with a pedagogical agent in MetaTutor Intelligent Tutoring Systems 2012 Heidelberg Springer 40-45

[16]

Gutica, M., Conati, C.: Student emotions with an edu-game: a detailed analysis. In: Proceedings of the Humaine Association Conference on Affective Computing and Intelligent Interaction. pp. 534–539. IEEE, Geneva (2013).

[17]

Sinclair J, Jang EE, Azevedo R, Lau C, Taub M, and Mudrick NV Nkambou R, Azevedo R, and Vassileva J Changes in emotion and their relationship with learning gains in the context of MetaTutor Intelligent Tutoring Systems 2018 Cham Springer 202-211

[18]

Azevedo R et al. Azevedo R, Aleven V, et al. Using trace data to examine the complex roles of cognitive, metacognitive, and emotional self-regulatory processes during learning with multi-agent systems International Handbook of Metacognition and Learning Technologies 2013 New York Springer 427-449

[19]

Petrovica S, Anohina-Naumeca A, and Ekenel HK Emotion recognition in affective tutoring systems: collection of ground-truth data Procedia Comput. Sci. 2017 104 437-444

[20]

Ekman, P.: Basic emotions. In: Handbook of Cognition and Emotion. pp. 45–60. Wiley (1999)

[21]

Pekrun, R., Frenzel, A.C., Goetz, T., Perry, R.P.: The control-value theory of achievement emotions: An integrative approach to emotions in education. In: Emotion in Education, pp. 13–36. Elsevier (2007)

[22]

Pekrun R, Vogl E, Muis KR, and Sinatra GM Measuring emotions during epistemic activities: the epistemically-related emotion scales Cogn. Emot. 2017 31 1268-1276

[23]

Ocumpaugh, J., Baker, R.S., Rodrigo, M.M.: Baker rodrigo ocumpaugh monitoring protocol (BROMP) 2.0 technical and training manual. Technical Report. Teachers College, Columbia University, New York. Ateneo Laboratory for the Learning Sciences, Manila (2015)

[24]

Malekzadeh, M., Mustafa, M., Lahsasna, A.: A review of emotion regulation in intelligent tutoring systems. Educ. Technol. Soc. 18, 435–445. https://www.jstor.org/stable/10.2307/jeductechsoci.18.4.435

[25]

Jarrell A, Harley JM, Lajoie S, and Naismith L Success, failure and emotions: examining the relationship between performance feedback and emotions in diagnostic reasoning Educ. Tech. Res Dev. 2017 65 5 1263-1284

[26]

Paquette, L., et al.: Sensor-free or sensor-full: a comparison of data modalities in multi-channel affect detection. In: Proceedings of the 8th International Conference on Educational Data Mining, pp. 93–100. IEDMS, Madrid (2016)

[27]

Kim J, Seo J, and Laine TH Detecting boredom from eye gaze and EEG Biomed. Sig. Process. Control 2018 46 302-313

[28]

Lallé, S., Conati, C., Carenini, G.: Predicting confusion in information visualization from eye tracking and interaction data. In: Proceedings on the 25th International Joint Conference on Artificial Intelligence, pp. 2529–2535. AAAI Press, New York (2016)

[29]

Henderson, N., Emerson, A., Rowe, J., Lester, J.: Improving sensor-based affect detection with multimodal data imputation. In: Proceedings of the 8th International Conference on Affective Computing and Intelligent Interaction, pp. 669–675. IEEE, Cambridge (2019)

[30]

Hutt, S., Mills, C., White, S., Donnelly, P.J., D’Mello, S.K.: The eyes have it: gaze-based detection of mind wandering during learning with an intelligent tutoring system. In: Proceedings of the 9th International Conference on Educational Data Mining, pp. 86–93. IEDMS, Raleigh (2016)

[31]

Kardan S and Conati C Carberry S, Weibelzahl S, Micarelli A, and Semeraro G Comparing and combining eye gaze and interface actions for determining user learning with an interactive simulation User Modeling, Adaptation, and Personalization 2013 Heidelberg Springer 215-227

[32]

Pekrun, R., Bühner, M.: Self-report measures of academic emotions. In: International Handbook of Emotions in Education. Routledge, London (2014)

[33]

Tan, P.-N., Kumar, V., Srivastava, J.: Selecting the right interestingness measure for association patterns. In: Proceedings of the 8th ACM International Conference on Knowledge Discovery and Data Mining, pp. 32–41. ACM, Edmonton (2002).

[34]

Villarica, R., Richards, D.: Intelligent and empathic agent to support student learning in virtual worlds. In: Proceedings of the Conference on Interactive Entertainment, pp. 1–9. ACM, Newcastle (2014).

[35]

Moridis CN and Economides AA Affective learning: empathetic agents with emotional facial and tone of voice expressions IEEE Trans. Affect. Comput. 2012 3 260-272

[36]

Craig S, Graesser A, Sullins J, and Gholson B Affect and learning: an exploratory look into the role of affect in learning with AutoTutor J. Educ. Media. 2004 29 241-250

[37]

Liu, Z., Pataranutaporn, V., Ocumpaugh, J., Baker, R.: Sequences of frustration and confusion, and learning. In: Proceedings of the International Conference on Educational Data Mining, pp. 114–120. IEDMS, Memphis (2013)

[38]

D’Mello S and Graesser A The half-life of cognitive-affective states during complex learning Cogn. Emot. 2011 25 1299-1308

[39]

Huang X and Mayer RE Benefits of adding anxiety-reducing features to a computer-based multimedia lesson on statistics Comput. Hum. Behav. 2016 63 293-303

[40]

Meyer, D.K.: Emotion regulation in K–12 classrooms. In: Handbook of Social Influences in School Contexts: Social-Emotional, Motivation, and Cognitive Outcomes. Routledge (2016)

[41]

Kardan, S., Lallé, S., Toker, D., Conati, C.: EMDAT: eye movement data analysis toolkit (1.x). The University of British Columbia (2021).

[42]

Bouchet F, Harley JM, Trevors GJ, and Azevedo R Clustering and profiling students according to their interactions with an intelligent tutoring system fostering self-regulated learning J. Educ. Data Min. 2013 5 104-146

[43]

Pedregosa F et al. Scikit-learn: machine learning in Python J. Mach. Learn. Res. 2011 12 2825-2830

[44]

Zeng Z, Pantic M, Roisman GI, and Huang TS A survey of affect recognition methods: audio, visual, and spontaneous expressions IEEE Trans. Pattern Anal. Mach. Intell. 2009 31 39-58

[45]

Chawla NV, Bowyer KW, Hall LO, and Kegelmeyer WP SMOTE: synthetic minority over-sampling technique J. Artif. Intell. Res. 2002 16 321-357

[46]

Holm, S.: A simple sequentially rejective multiple test procedure. Scand. J. Stat. 65–70 (1979)

Cited By

Yu ZCrane CChen LTesta MZheng Z(2024)A Computation Model to Estimate Interaction Intensity through Non-Verbal Behavioral Cues: A Case Study of Intimate Couples under the Impact of Acute Alcohol ConsumptionACM Transactions on Computing for Healthcare10.1145/36648265:3(1-23)Online publication date: 18-Sep-2024
https://dl.acm.org/doi/10.1145/3664826
Raković MLi YFoumani NSalehi MKuhlmann LMackellar GMartinez-Maldonado RHaffari GSwiecki ZLi XChen GGašević D(2024)Measuring Affective and Motivational States as Conditions for Cognitive and Metacognitive Processing in Self-Regulated LearningProceedings of the 14th Learning Analytics and Knowledge Conference10.1145/3636555.3636934(701-712)Online publication date: 18-Mar-2024
https://dl.acm.org/doi/10.1145/3636555.3636934
Anand SDevulapally NBhattacharjee SYuan JEl Saddik AMei TCucchiara RBertini MTobon Vallejo DAtrey PHossain M(2023)Multi-label Emotion Analysis in Conversation via Multimodal Knowledge DistillationProceedings of the 31st ACM International Conference on Multimedia10.1145/3581783.3612517(6090-6100)Online publication date: 26-Oct-2023
https://dl.acm.org/doi/10.1145/3581783.3612517
Show More Cited By

Index Terms

Predicting Co-occurring Emotions from Eye-Tracking and Interaction Data in MetaTutor

Index terms have been assigned to the content through auto-classification.

Recommendations

Predicting Co-occurring Emotions in MetaTutor when Combining Eye-Tracking and Interaction Data from Separate User Studies
LAK2023: LAK23: 13th International Learning Analytics and Knowledge Conference

Learning can be improved by providing personalized feedback adapting to the emotions that the learner may be experiencing. There is initial evidence that co-occurring emotions can be predicted during learning in Intelligent Tutoring Systems (ITS) through ...
Measuring learners' co-occurring emotional responses during their interaction with a pedagogical agent in metatutor
ITS'12: Proceedings of the 11th international conference on Intelligent Tutoring Systems

This paper extends upon traditional emotional measurement frameworks used by ITSs in which emotions are analyzed as single, discrete psychological experiences by examining co-occurring emotions (COEs) (e.g., Conati) through a novel methodological ...
Attentional bias to negative emotions in anxious individuals: an eye tracking study
ICEME '22: Proceedings of the 2022 13th International Conference on E-business, Management and Economics

(1) Background: While anxiety disorders are becoming more prevalent in our society, anxiety diagnosis still traditionally requires filling out assessments or requires professional consultation and evaluation, but now we could perform a simple and quick ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Guide Proceedings

Artificial Intelligence in Education: 22nd International Conference, AIED 2021, Utrecht, The Netherlands, June 14–18, 2021, Proceedings, Part I

Jun 2021

544 pages

ISBN:978-3-030-78291-7

DOI:10.1007/978-3-030-78292-4

Editors:
Ido Roll
Technion – Israel Institute of Technology, Haifa, Israel
,
Danielle McNamara
Arizona State University, Tempe, AZ, USA
,
Sergey Sosnovsky
Utrecht University, Utrecht, The Netherlands
,
Rose Luckin
London Knowledge Lab, London, UK
,
Vania Dimitrova
University of Leeds, Leeds, UK

© Springer Nature Switzerland AG 2021.

Publisher

Springer-Verlag

Berlin, Heidelberg

Publication History

Published: 14 June 2021

Author Tags

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

5
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 09 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Yu ZCrane CChen LTesta MZheng Z(2024)A Computation Model to Estimate Interaction Intensity through Non-Verbal Behavioral Cues: A Case Study of Intimate Couples under the Impact of Acute Alcohol ConsumptionACM Transactions on Computing for Healthcare10.1145/36648265:3(1-23)Online publication date: 18-Sep-2024
https://dl.acm.org/doi/10.1145/3664826
Raković MLi YFoumani NSalehi MKuhlmann LMackellar GMartinez-Maldonado RHaffari GSwiecki ZLi XChen GGašević D(2024)Measuring Affective and Motivational States as Conditions for Cognitive and Metacognitive Processing in Self-Regulated LearningProceedings of the 14th Learning Analytics and Knowledge Conference10.1145/3636555.3636934(701-712)Online publication date: 18-Mar-2024
https://dl.acm.org/doi/10.1145/3636555.3636934
Anand SDevulapally NBhattacharjee SYuan JEl Saddik AMei TCucchiara RBertini MTobon Vallejo DAtrey PHossain M(2023)Multi-label Emotion Analysis in Conversation via Multimodal Knowledge DistillationProceedings of the 31st ACM International Conference on Multimedia10.1145/3581783.3612517(6090-6100)Online publication date: 26-Oct-2023
https://dl.acm.org/doi/10.1145/3581783.3612517
Kobylyanskaya S(2022)Speech and Eye Tracking Features for L2 Acquisition: A Multimodal ExperimentArtificial Intelligence in Education. Posters and Late Breaking Results, Workshops and Tutorials, Industry and Innovation Tracks, Practitioners’ and Doctoral Consortium10.1007/978-3-031-11647-6_8(47-52)Online publication date: 27-Jul-2022
https://dl.acm.org/doi/10.1007/978-3-031-11647-6_8
Dever DSonnenfeld NWiedbusch MAzevedo R(2022)Pedagogical Agent Support and Its Relationship to Learners’ Self-regulated Learning Strategy Use with an Intelligent Tutoring SystemArtificial Intelligence in Education10.1007/978-3-031-11644-5_27(332-343)Online publication date: 27-Jul-2022
https://dl.acm.org/doi/10.1007/978-3-031-11644-5_27

View Options

View options

Media

Figures

Other

Tables

View Table of Contents