[go: up one dir, main page]
More Web Proxy on the site http://driver.im/ skip to main content
research-article

Using Video to Automatically Detect Learner Affect in Computer-Enabled Classrooms

Published: 20 July 2016 Publication History

Abstract

Affect detection is a key component in intelligent educational interfaces that respond to students’ affective states. We use computer vision and machine-learning techniques to detect students’ affect from facial expressions (primary channel) and gross body movements (secondary channel) during interactions with an educational physics game. We collected data in the real-world environment of a school computer lab with up to 30 students simultaneously playing the game while moving around, gesturing, and talking to each other. The results were cross-validated at the student level to ensure generalization to new students. Classification accuracies, quantified as area under the receiver operating characteristic curve (AUC), were above chance (AUC of 0.5) for all the affective states observed, namely, boredom (AUC = .610), confusion (AUC = .649), delight (AUC = .867), engagement (AUC = .679), frustration (AUC = .631), and for off-task behavior (AUC = .816). Furthermore, the detectors showed temporal generalizability in that there was less than a 2% decrease in accuracy when tested on data collected from different times of the day and from different days. There was also some evidence of generalizability across ethnicity (as perceived by human coders) and gender, although with a higher degree of variability attributable to differences in affect base rates across subpopulations. In summary, our results demonstrate the feasibility of generalizable video-based detectors of naturalistic affect in a real-world setting, suggesting that the time is ripe for affect-sensitive interventions in educational games and other intelligent interfaces.

References

[1]
Paul D. Allison. 1999. Multiple Regression: A Primer. Pine Forge Press.
[2]
Omar AlZoubi, Davide Fossati, Sidney D’Mello, and Rafael A. Calvo. 2015. Affect detection from non-stationary physiological data using ensemble classifiers. Evolving Systems 6, 2 (2015), 79--92.
[3]
Omar AlZoubi, M. S. Hussain, Sidney D’Mello, and Rafael A. Calvo. 2011. Affective modeling from multichannel physiology: Analysis of day differences. In Proceedings of the 4th International Conference on Affective Computing an Intelligent Interaction (ACII’11). Springer-Verlag, Berlin.
[4]
Ivon Arroyo, David G. Cooper, Winslow Burleson, Beverly Park Woolf, Kasia Muldner, and Robert Christopherson. 2009. Emotion sensors go to school. In Proceedings of the 14th International Conference on Artificial Intelligence in Education, V. Dimitrova, R. Mizoguchi, B. Du Boulay, and Art Graesser (Eds.). IOS Press, 17--24.
[5]
Ryan Baker, et al. 2012. Towards sensor-free affect detection in cognitive tutor algebra. In Proceedings of the 5th International Conference on Educational Data Mining. 126--133.
[6]
Nigel Bosch, Sidney D’Mello, Ryan Baker, Jaclyn Ocumpaugh, Valerie J. Shute, et al. 2015. Automatic detection of learning-centered affective states in the wild. In Proceedings of the 2015 International Conference on Intelligent User Interfaces (IUI’15). ACM, New York, NY, 379--388.
[7]
Nigel Bosch, Huili Chen, Ryan Baker, Valerie Shute, and Sidney D’Mello. 2015. Accuracy vs. availability heuristic in multimodal affect detection in the wild. In Proceedings of the 17th International Conference on Multimodal Interaction (ICMI’15). ACM, New York, NY, 267--274.
[8]
Nigel Bosch, Yuxuan Chen, and Sidney D’Mello. 2014. It's written on your face: Detecting affective states from facial expressions while learning computer programming. In Proceedings of the 12th International Conference on Intelligent Tutoring Systems (ITS’14), Stefan Trausan-Matu, Kristy Elizabeth Boyer, Martha Crosby, and Kitty Panourgia (Eds.). Lecture Notes in Computer Science. Springer, Berlin, 39--44.
[9]
Nigel Bosch, Sidney D’Mello, Ryan Baker, Jaclyn Ocumpaugh, and Valerie J. Shute. 2015. Temporal generalizability of face-based affect detection in noisy classroom environments. In Proceedings of the 17th International Conference on Artificial Intelligence in Education (AIED’15), Cristina Conati, Neil T. Heffernan, Antonija Mitrovic, and M. Felisa Verdejo (Eds.). Springer-Verlag, Berlin, 44--53.
[10]
Rafael A. Calvo and Sidney D’Mello. 2010. Affect detection: An interdisciplinary review of models, methods, and their applications. IEEE Transactions on Affective Computing 1, 1 (January 2010), 18--37.
[11]
Rafael A. Calvo, Sidney D’Mello, Jonathan Gratch, and Arvid Kappas. 2015. The Oxford Handbook of Affective Cmputing. Oxford University Press, New York, NY.
[12]
Rafael A. Calvo and Sidney K. D’Mello. 2011. New Perspectives on Affect and Learning Technologies. Springer, New York, NY.
[13]
Nitesh V. Chawla, Kevin W. Bowyer, Lawrence O. Hall, and W. Philip Kegelmeyer. 2011. SMOTE: Synthetic minority over-sampling technique. Journal of Artificial Intelligence Research 16 (June 2011), 321--357.
[14]
Gerald L. Clore and Jeffrey R. Huntsinger. 2007. How emotions inform judgment and regulate thought. Trends in Cognitive Sciences 11, 9 (September 2007), 393--399.
[15]
Jacob Cohen. 1988. Statistical Power Analysis for the Behavioral Sciences (2nd ed.). Erlbaum. Hillsdale, NJ.
[16]
Desmond L. Cook. 1962. The hawthorne effect in educational research. Phi Delta Kappan (1962), 116--122.
[17]
Abhinav Dhall, Roland Goecke, Jyoti Joshi, Michael Wagner, and Tom Gedeon. 2013. Emotion recognition in the wild challenge 2013. In Proceedings of the 15th ACM on International Conference on Multimodal Interaction (ICMI’13). ACM, New York, NY, 509--516.
[18]
Sidney D’Mello. 2013. A selective meta-analysis on the relative incidence of discrete affective states during learning with technology. Journal of Educational Psychology 105, 4 (2013), 1082--1099.
[19]
Sidney D’Mello, Nathan Blanchard, Ryan Baker, Jaclyn Ocumpaugh, and Keith Brawner. 2014. I feel your pain: A selective review of affect-sensitive instructional strategies. In Design Recommendations for Intelligent Tutoring Systems - Volume 2: Instructional Management, Robert Sottilare, Art Graesser, Xiangen Hu, and Benjamin Goldberg (Eds.). 35--48.
[20]
Sidney D’Mello and Rafael A. Calvo. 2013. Beyond the basic emotions: What should affective computing compute? In CHI’13 Extended Abstracts on Human Factors in Computing Systems. ACM, New York, NY, 2287--2294.
[21]
Sidney D’Mello and Art Graesser. 2011. The half-life of cognitive-affective states during complex learning. Cognition & Emotion 25, 7 (2011), 1299--1308.
[22]
Sidney K. D’Mello. 2016. On the influence of an iterative affect annotation approach on inter-observer and self-observer reliability. IEEE Transactions on Affective Computing 7, 2 (2016), 136--149.
[23]
Sidney K. D’Mello, Scotty D. Craig, Amy Witherspoon, Bethany McDaniel, and Arthur Graesser. 2008. Automatic detection of learner's affect from conversational cues. User Modeling and User-Adapted Interaction 18, 1--2 (2008), 45--80.
[24]
Sidney D’Mello and Jacqueline Kory. 2015. A review and meta-analysis of multimodal affect detection systems. ACM Computing Surveys 47, 3 (February 2015), 43, 1--43, 36.
[25]
Paul Ekman. 1992. An argument for basic emotions. Cognition and Emotion 6, 3--4 (May 1992), 169--200.
[26]
Paul Ekman, Wallace V. Freisen, and Sonia Ancoli. 1980. Facial signs of emotional experience. Journal of Personality and Social Psychology 39, 6 (1980), 1125--1134.
[27]
Paul Ekman and Wallace V. Friesen. 1978. Facial Action Coding System: A Technique for the Measurement of Facial Movement. Consulting Psychologists Press, Palo Alto, CA.
[28]
Jeffrey M. Girard, Jeffrey F. Cohn, Laszlo A. Jeni, Michael A. Sayette, and Fernando De La Torre. 2015. Spontaneous facial expression in unscripted social interactions can be measured automatically. Behavior Research Methods 47, 4 (December 2015), 1136--1147.
[29]
Art Graesser, Bethany McDaniel, Patrick Chipman, Amy Witherspoon, Sidney D’Mello, and Barry Gholson. 2006. Detection of emotions during learning with AutoTutor. In Proceedings of the 28th Annual Meetings of the Cognitive Science Society, Ron Sun and Naomi Miyake (Eds.). Cognitive Science Society, Austin, TX, 285--290.
[30]
Joseph F. Grafsgaard, Seung Y. Lee, Bradford W. Mott, Kristy Elizabeth Boyer, and James C. Lester. 2015. Modeling self-efficacy across age groups with automatically tracked facial expression. In Proceedings of the 17th International Conference on Artificial Intelligence in Education (AIED 2015). Springer-Verlag, Berlin, 582--585.
[31]
Joseph F. Grafsgaard, Joseph B. Wiggins, Kristy Elizabeth Boyer, Eric N. Wiebe, and James C. Lester. 2013. Automatically recognizing facial indicators of frustration: A learning-centric analysis. In Proceedings of the 5th International Conference on Affective Computing and Intelligent Interaction. 159--165.
[32]
Javier Hernandez, Mohammed Ehsan Hoque, Will Drevo, and Rosalind W. Picard. 2012. Mood meter: Counting smiles in the wild. In Proceedings of the 2012 ACM Conference on Ubiquitous Computing (UbiComp’12). ACM, New York, NY, 301--310.
[33]
Geoffrey Holmes, Andrew Donkin, and Ian H. Witten. 1994. WEKA: A machine learning workbench. In Proceedings of the 2nd Australian and New Zealand Conference on Intelligent Information Systems. 357--361.
[34]
Mohammed Ehsan Hoque, Daniel McDuff, and Rosalind W. Picard. 2012. Exploring temporal patterns in classifying frustrated and delighted smiles. IEEE Transactions on Affective Computing 3, 3 (2012), 323--334.
[35]
László A. Jeni, Jeffrey F. Cohn, and Fernando De La Torre. 2013. Facing imbalanced data--Recommendations for the use of performance metrics. In Proceedings of the 5th International Conference on Affective Computing and Intelligent Interaction. 245--251.
[36]
Samira Ebrahimi Kahou, et al. 2013. Combining modality specific deep neural networks for emotion recognition in video. In Proceedings of the 15th International Conference on Multimodal Interaction. ACM, New York, NY, 543--550.
[37]
Ashish Kapoor, Winslow Burleson, and Rosalind W. Picard. 2007. Automatic prediction of frustration. International Journal of Human-Computer Studies 65, 8 (2007), 724--736.
[38]
Ashish Kapoor and Rosalind W. Picard. 2005. Multimodal affect recognition in learning environments. In Proceedings of the 13th Annual ACM International Conference on Multimedia. ACM, New York, NY, 677--682.
[39]
Seong G. Kong, Jingu Heo, Besma R. Abidi, Joonki Paik, and Mongi A. Abidi. 2005. Recent advances in visual and infrared face recognition—a review. Computer Vision and Image Understanding 97, 1 (January 2005), 103--135.
[40]
Igor Kononenko. 1994. Estimating attributes: Analysis and extensions of RELIEF. In Proceedings of the European Conference on Machine Learning (ECML 94), Francesco Bergadano and Luc De Raedt (Eds.). Lecture Notes in Computer Science. Berlin, Springer, 171--182.
[41]
Mark R. Lepper, Maria Woolverton, Donna L. Mumme, and J. Gurtner. 1993. Motivational techniques of expert human tutors: Lessons for the design of computer-based tutors. Computers as Cognitive Tools 1993 (1993), 75--105.
[42]
Stan Z. Li, RuFeng Chu, ShengCai Liao, and Lun Zhang. 2007. Illumination invariant face recognition using near-infrared images. IEEE Transactions on Pattern Analysis and Machine Intelligence 29, 4 (April 2007), 627--639.
[43]
Gwen Littlewort et al. 2011. The computer expression recognition toolbox (CERT). In Proceedings of the 2011 IEEE International Conference on Automatic Face Gesture Recognition and Workshops (FG 2011). 298--305.
[44]
Bethany T. McDaniel, Sidney K. D’Mello, Brandon G. King, Patrick Chipman, Kristy Tapp, and Art Graesser. 2007. Facial features for affective state detection in learning environments. In Proceedings of the 29th Annual Cognitive Science Society. 467--472.
[45]
Daniel McDuff, Rana El Kaliouby, Thibaud Senechal, David Demirdjian, and Rosalind Picard. 2014. Automatic measurement of ad preferences from facial responses gathered over the Internet. Image and Vision Computing 32, 10 (October 2014), 630--640.
[46]
Daniel McDuff, R. El Kaliouby, T. Senechal, M. Amr, J. F. Cohn, and R. Picard. 2013. Affectiva-MIT facial expression dataset (AM-FED): Naturalistic and spontaneous facial expressions collected in-the-wild. In Proceedings of the 2013 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). 881--888.
[47]
Selene Mota and Rosalind W. Picard. 2003. Automated posture analysis for detecting learner's interest level. In Proceedings of the 2003 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). 49--56.
[48]
Jaclyn Ocumpaugh, Ryan Baker, and M. A. Mercedes Rodrigo, et al. 2015. HART: The human affect recording tool. In Proceedings of the ACM Special Interest Group on the Design of Communication (SIGDOC). ACM, New York, NY.
[49]
Jaclyn Ocumpaugh, Ryan Baker, Sujith Gowda, Neil Heffernan, and Cristina Heffernan. 2014. Population validity for educational data mining models: A case study in affect detection. British Journal of Educational Technology 45, 3 (May 2014), 487--501.
[50]
Jaclyn Ocumpaugh, Ryan Baker, M. A. Mercedes, and T. Rodrigo. 2015. Baker Rodrigo Ocumpaugh monitoring protocol (BROMP) 2.0 technical and training manual. In Technical Report. New York, NY: Teachers College, Columbia University. Manila, Philippines: Ateneo Laboratory for the Learning Sciences.
[51]
Rosalind W. Picard, Elias Vyzas, and Jennifer Healey. 2001. Toward machine emotional intelligence: Analysis of affective physiological state. IEEE Transactions on Pattern Analysis and Machine Intelligence 23, 10 (2001), 1175--1191.
[52]
Kaśka Porayska-Pomsta, Manolis Mavrikis, Sidney D’Mello, Cristina Conati, and Ryan Baker. 2013. Knowledge elicitation methods for affect modelling in education. International Journal of Artificial Intelligence in Education 22, 3 (January 2013), 107--140.
[53]
Rainer Reisenzein, Markus Studtmann, and Gernot Horstmann. 2013. Coherence between emotion and facial expression: Evidence from laboratory experiments. Emotion Review 5, 1 (January 2013), 16--23.
[54]
Thibaud Senechal, Kevin Bailly, and Lionel Prevost. 2014. Impact of action unit detection in automatic emotion recognition. Pattern Analysis and Applications 17, 1 (February 2014), 51--67.
[55]
Valerie J. Shute, Matthew Ventura, and Yoon Jeon Kim. 2013. Assessment and learning of qualitative physics in newton's playground. The Journal of Educational Research 106, 6 (2013), 423--430.
[56]
Michel F. Valstar, Marc Mehu, Bihan Jiang, Maja Pantic, and Klaus Scherer. 2012. Meta-analysis of the first facial expression recognition challenge. IEEE Transactions on Systems, Man, and Cybernetics, Part B: Cybernetics 42, 4 (2012), 966--979.
[57]
Jacqueline Kory Westlund, Sidney K. D’Mello, and Andrew M. Olney. 2015. Motion tracker: Camera-based monitoring of bodily movements using motion silhouettes. PLoS ONE 10, 6 (June 2015).
[58]
J. Whitehill, Z. Serpell, Yi-Ching Lin, A. Foster, and J. R. Movellan. 2014. The faces of engagement: Automatic recognition of student engagement from facial expressions. IEEE Transactions on Affective Computing 5, 1 (January 2014), 86--98.
[59]
Zhihong Zeng, Maja Pantic, Glenn I. Roisman, and Thomas S. Huang. 2009. A survey of affect recognition methods: Audio, visual, and spontaneous expressions. IEEE Transactions on Pattern Analysis and Machine Intelligence 31, 1 (2009), 39--58.

Cited By

View all
  • (2024)Measuring Student Engagement through Behavioral and Emotional Features Using Deep-Learning ModelsAlgorithms10.3390/a1710045817:10(458)Online publication date: 16-Oct-2024
  • (2024)ClassID: Enabling Student Behavior Attribution from Ambient Classroom Sensing SystemsProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36595868:2(1-28)Online publication date: 15-May-2024
  • (2024)Usability Evaluation of E-Learning Platforms Using UX/UI Design and ML Technique2024 International Conference on Advancements in Smart, Secure and Intelligent Computing (ASSIC)10.1109/ASSIC60049.2024.10507912(1-6)Online publication date: 27-Jan-2024
  • Show More Cited By

Index Terms

  1. Using Video to Automatically Detect Learner Affect in Computer-Enabled Classrooms

    Recommendations

    Comments

    Please enable JavaScript to view thecomments powered by Disqus.

    Information & Contributors

    Information

    Published In

    cover image ACM Transactions on Interactive Intelligent Systems
    ACM Transactions on Interactive Intelligent Systems  Volume 6, Issue 2
    Regular Articles, Special Issue on Highlights of IUI 2015 (Part 2 of 2) and Special Issue on Highlights of ICMI 2014 (Part 1 of 2)
    August 2016
    282 pages
    ISSN:2160-6455
    EISSN:2160-6463
    DOI:10.1145/2974721
    Issue’s Table of Contents
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 20 July 2016
    Accepted: 01 March 2016
    Revised: 01 January 2016
    Received: 01 July 2015
    Published in TIIS Volume 6, Issue 2

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. Affect detection
    2. classroom data
    3. generalization
    4. in the wild
    5. naturalistic facial expressions

    Qualifiers

    • Research-article
    • Research
    • Refereed

    Funding Sources

    • Bill & Melinda Gates Foundation
    • National Science Foundation (NSF)

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)77
    • Downloads (Last 6 weeks)13
    Reflects downloads up to 11 Dec 2024

    Other Metrics

    Citations

    Cited By

    View all
    • (2024)Measuring Student Engagement through Behavioral and Emotional Features Using Deep-Learning ModelsAlgorithms10.3390/a1710045817:10(458)Online publication date: 16-Oct-2024
    • (2024)ClassID: Enabling Student Behavior Attribution from Ambient Classroom Sensing SystemsProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36595868:2(1-28)Online publication date: 15-May-2024
    • (2024)Usability Evaluation of E-Learning Platforms Using UX/UI Design and ML Technique2024 International Conference on Advancements in Smart, Secure and Intelligent Computing (ASSIC)10.1109/ASSIC60049.2024.10507912(1-6)Online publication date: 27-Jan-2024
    • (2024)Measuring student behavioral engagement using histogram of actionsPattern Recognition Letters10.1016/j.patrec.2024.11.002186(337-344)Online publication date: Oct-2024
    • (2024)Joy is reciprocally transmitted between teachers and students: Evidence on facial mimicry in the classroomLearning and Instruction10.1016/j.learninstruc.2024.10189691(101896)Online publication date: Jun-2024
    • (2023)An Experimental Platform for Real-Time Students Engagement Measurements from Video in STEM ClassroomsSensors10.3390/s2303161423:3(1614)Online publication date: 2-Feb-2023
    • (2023)Informing Expert Feature Engineering through Automated Approaches: Implications for Coding Qualitative Classroom Video DataLAK23: 13th International Learning Analytics and Knowledge Conference10.1145/3576050.3576090(630-636)Online publication date: 13-Mar-2023
    • (2023)ExpresSense: Exploring a Standalone Smartphone to Sense Engagement of Users from Facial Expressions Using Acoustic SensingProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3581235(1-18)Online publication date: 19-Apr-2023
    • (2023)Multimodal Engagement Analysis From Facial Videos in the ClassroomIEEE Transactions on Affective Computing10.1109/TAFFC.2021.312769214:2(1012-1027)Online publication date: 1-Apr-2023
    • (2023)Active Learning for a Classroom Observer who Can’t Time Travel2023 11th International Conference on Affective Computing and Intelligent Interaction Workshops and Demos (ACIIW)10.1109/ACIIW59127.2023.10388151(1-8)Online publication date: 10-Sep-2023
    • Show More Cited By

    View Options

    Login options

    Full Access

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Media

    Figures

    Other

    Tables

    Share

    Share

    Share this Publication link

    Share on social media