[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to main content

Advertisement

Springer Nature Link
Log in

Online learners’ engagement detection via facial emotion recognition in online learning context using hybrid classification model

  • Original Article
  • Published:
Social Network Analysis and Mining Aims and scope Submit manuscript

Abstract

Writing, reading, viewing video lectures, completing online examinations, and attending online meetings are all activities that students participate in through the internet. While participating in these educational activities, they demonstrate various degrees of interest, including boredom, aggravation, delight, indifference, confusion, and learning gain. Online educators must accurately and efficiently monitor the degree of engagement of their online learners with the goal of giving focused pedagogical assistance to them through interventions. The objective of this paper is to propose a novel students engagement prediction model for online learners based on facial emotion, which will include four basic phases: (a) preprocessing, (b) feature extraction, (c) emotion recognition, and (d) student engagement prediction. The preprocessing step is the first phase in which the Face detection process is followed. Following the preprocessing step, the feature extraction phase proceeds with the extraction of the Improved Active Appearance Model (AAM), Shape Local Binary Texture (SLBT), Global Binary Pattern (GBP), and ResNet features. The retrieved characteristics are subsequently subjected to emotion recognition via the Hybrid Classification model, which incorporates models including Improved Deep Belief Network (IDBN) and Convolutional Neural Network (CNN). The student's involvement or engagement is identified based on the emotions recognized, as well as the way they performed via the enhanced entropy-based process. The execution of the suggested hybrid IDBN + CNN model is evaluated over the extant methods like DBN, SVM, CNN, LSTM-CNN, LSTM, and RF under various measures for two datasets. The hybrid model had the greatest accuracy of 0.95 at a learning percentage of 80% for the CK+ dataset. Also, the hybrid model has a higher sensitivity of 60% for FER-2013 datasets.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
£29.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price includes VAT (United Kingdom)

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

Data availability

The data underlying this article are available in https://www.kaggle.com/datasets/msambare/fer2013, and https://www.kaggle.com/datasets/shawon10/ckplus.

Abbreviations

CNN:

Convolutional neural network

DFSTN:

Deep facial spatiotemporal network

SVM:

Support vector machines

LSTM:

Long short-term memory

GALN:

LSTM network with global attention

ML:

Machine learning

FER:

Facial Expression Recognition

DL:

Deep learning

GPU:

Graphics processing unit

DBN:

Deep Belief Network

KNN:

K-nearest neighbor

FS:

Face-sensitive

EI:

Engagement index

PCA:

Principal component analysis

AAM:

Active Appearance Model

GBP:

Global Binary Pattern

IDBN:

Improved Deep Belief Network

DT:

Decision tree

SLBT:

Shape Local Binary Texture

EEG:

Electroencephalogram

GPU:

Graphics processing unit

DNN:

Deep neural network

AI:

Artificial intelligence

VGG:

Visual Geometry Group

References

  • Ashwin TS, Guddeti RMR (2020a) R.M.R. impact of inquiry interventions on students in e-learning and classroom environments using affective computing framework. User Model User-Adap Inter 30

  • Ashwin TS, Guddeti RMR (2020b) Automatic detection of students’ affective states in a classroom environment using hybrid convolutional neural networks. Educ Inf Technol 25

  • Barnouti NH, Al-Dabbagh SSM, Matti WE (2016) Naser MAS Face detection and recognition using viola-jones with PCA-LDA and square Euclidean distance. Int J Adv Comput Sci Appl 7(5)

  • Benabbes K, Housni K, Hmedna B, Zellou A, Mezouary AE (2023) A new hybrid approach to detect and track learner’s engagement in e-Learning. IEEE Access 11:70912–70929. https://doi.org/10.1109/ACCESS.2023.3293827

    Article  Google Scholar 

  • Buono P, De Carolis B, D’Errico F, Macchiarulo N, Palestra G (2022) Assessing student engagement from facial behaviour in online learning. Multimed Tools Appl 82:12859–12877

    Article  PubMed  PubMed Central  Google Scholar 

  • Cabada RZ, Rangel HR, Estrada MLB, Lopez HMC (2020) Hyperparameter optimization in CNN for learning-centred emotion recognition for intelligent tutoring systems. Soft Comput 24

  • Dewan MAA et al (2018) A deep learning approach to detecting engagement of online learners. In: 2018 IEEE SmartWorld, ubiquitous intelligence & computing, advanced & trusted computing, scalable computing & communications, cloud & big data computing, internet of people and smart city innovation (SmartWorld/SCALCOM/UIC/ATC/CBDCom/IOP/SCI). IEEE

  • Ding Y, Xing W (2022) Emotion recognition and achievement prediction for foreign language learners under the background of network teaching. Front Psychol. https://doi.org/10.3389/fpsyg.2022.1017570

    Article  PubMed  PubMed Central  Google Scholar 

  • Ghosh A, Sufian A, Sultana F, Chakrabarti A, De D (2020) Fundamental concepts of convolutional neural network. Chapter January 2020 https://doi.org/10.1007/978-3-030-32644-9_36.

  • Gupta S, Kumar P, Tekchandani RK (2022) Facial emotion recognition based real-time learner engagement detection system in online learning context using deep learning models. Multimed Tools Appl 8:11365–11394

    Google Scholar 

  • Hassouneh A, Mutawa AM, Murugappan M (2020) Development of a real-time emotion recognition system using facial expressions and EEG based on machine learning and deep neural network methods. Inform Med Unlock 20:100372

    Article  Google Scholar 

  • Hung JC, Lin K-C, Lai N-X (2019) Recognizing learning emotion based on convolutional neural networks and transfer learning. Appl Soft Comput 84:105724

    Article  Google Scholar 

  • Iqtait M, Mohamad FS, Mamat M (2018) Feature extraction for face recognition via active shape model (ASM) and active appearance model (AAM). In: IOP conferemce series: materials science and engineering vol 332, pp 012032. https://doi.org/10.1088/1757-899X/332/1/012032

  • Lakshmiprabha NS, Majumder S (2012) Face recognition system invariant to plastic surgery. In: 2012 12th international conference on intelligent systems design and applications (ISDA), Kochi. pp 258–263. https://doi.org/10.1109/ISDA.2012.6416547

  • Lasri I, Riadsolh A (2023) Elbelkacemi M (2023) Facial emotion recognition of deaf and hard-of-hearing students for engagement detection using deep learning. Educ Inf Technol 28:4069–4092. https://doi.org/10.1007/s10639-022-11370-4

    Article  Google Scholar 

  • Liao J, Liang Y, Pan J (2021) Deep facial spatiotemporal network for engagement prediction in online learning. Appl Intell 51:6609–6621

    Article  Google Scholar 

  • Mehta NK, Prasad SS, Saurav S, Saini R, Singh A (2022) Three-dimensional DenseNet self-attention neural network for automatic detection of student’s engagement. Appl Intell 52:13803–13823

    Article  Google Scholar 

  • Mhapsekar M, Mhapsekar P, Mhatre A, Sawant V (2015) Implementation of residual network (ResNet) for devanagari handwritten character recognition. In: Advanced computing technologies and applications, algorithms for intelligent systems, https://doi.org/10.1007/978-981-15-3242-9_14

  • Ngai WK, Xieb H, Zouc D, Chou KL (2022) Emotion recognition based on convolutional neural networks and heterogeneous bio-signal data sources. Inf Fusion 77:107–117

    Article  Google Scholar 

  • Ninaus M, Greipla S, Kiili K, Lindstedt A, Huber S, Klein E, Karnath H-O, Moeller K (2019) Increased emotional engagement in game-based learning—a machine learning approach on facial emotion detection data. Comput Educ 142:103641

    Article  Google Scholar 

  • Said Y, Barr M (2021) Human emotion recognition based on facial expressions via deep learning on high-resolution images. Multimed Tools Appl 80:25241–25253

    Article  Google Scholar 

  • Savchenko AV, Makarov IA (2022) Neural network model for video-based analysis of student’s emotions in E-learning. Opt Memory Neural Netw 3:237–244

    Article  Google Scholar 

  • Schoneveld L, Othmani A, Abdelkawy H (2021) Leveraging recent advances in deep learning for audio-visual emotion recognition. Pattern Recogn Lett 146:1–7

    Article  ADS  Google Scholar 

  • Sivri E, Kalkan S (2013) Global binary patterns: a novel shape descriptor. In: International IAPR conference on machine vision and applications (20–23 May 2013)

  • Vidyadhari S, Mishra DS (2020) Analysis of facial expressions for predicting student’s learning level. JCSE Int J Comput Sci Eng Open Access 8

  • Wang HZ, Wang GB, Li GQ, Peng JC, Liu YT (2016) Deep belief network based deterministic and probabilistic wind speed forecasting approach. Appl Energy 182:80–93

    Article  ADS  Google Scholar 

  • Xu R, Chen J, Han J, Tan L, Xu L (2020) Towards emotion-sensitive learning cognitive state analysis of big data in education: deep learning-based facial expression analysis using ordinal information. Computing 102:765–780

    Article  MathSciNet  Google Scholar 

  • Yuan Q (2022) Research on classroom emotion recognition algorithm based on visual emotion classification. Comput Intell Neurosci. https://doi.org/10.1155/2022/6453499

    Article  PubMed  PubMed Central  Google Scholar 

  • Zhang Z, Li Z, Liu H, Cao T, Liu S (2020) Data-driven online learning engagement detection via facial expression and mouse behaviour recognition technology. J Educ Comput Res 58

  • Zhu X, Chen Z (2020) Dual-modality spatiotemporal feature learning for spontaneous facial expression recognition in e-learning using hybrid deep neural network. Vis Comput 36:743–755

    Article  Google Scholar 

Download references

Funding

No specific fund was provided for this article.

Author information

Authors and Affiliations

  1. Department of Computer and Information Science, Faculty of Science, Annamalai University, Annamalainagar, Chennai, Tamilnadu, 608002, India

    Rama Bhadra Rao Maddu & S. Murugappan

Authors
  1. Rama Bhadra Rao Maddu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. S. Murugappan
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

RBRM conceived the suggested idea and designed the analysis. Along with that, MS helped him write the book and conduct the experiment. Each author contributed to the final manuscript and discussed the findings. The final manuscript was read and approved by all writers.

Corresponding author

Correspondence to Rama Bhadra Rao Maddu.

Ethics declarations

Conflict of interest

The authors say they have no conflicts of interest.

Informed consent

Not applicable.

Ethical approval

Not applicable.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Maddu, R.B.R., Murugappan, S. Online learners’ engagement detection via facial emotion recognition in online learning context using hybrid classification model. Soc. Netw. Anal. Min. 14, 43 (2024). https://doi.org/10.1007/s13278-023-01181-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s13278-023-01181-x

Keywords

Search

Navigation