Online Learning Behavior Analysis Based on Image Emotion Recognition

1.png

Figure 1. Flow of image emotion recognition based on online learning behavior analysis

The image emotion recognition of surveillance video on online learning students mainly covers the following tasks: recognizing the static emotional features based on the facial emotional changes in the key frames of the video, and recognizing the dynamic emotional features based on the dynamic changes of facial expressions in a series of video images. Figure 1 explains the flow of image emotion recognition based on online learning behavior analysis. It can be seen that any emotion recognition method must extract features from facial expressions. This paper relies on improved LBP and wavelet transform to extract the key frames from facial expression images.

The LBP can characterize the relationship between a pixel and its neighboring pixels in the surveillance video image. In this paper, the improved uniform mode LBP operator Δ_un-LBP is adopted to extract the face features from each image in the surveillance video on online learning students. Let h_D be the grayscale of the central pixel in the image; h_O(o=0, 1, L, O-1) be the grayscales of the other pixels in the circular neighborhood of the central pixel. Then, Δ_un-LBP can be expressed as:

${{\Delta }_{un-LBP}}=\left\{ \begin{align}  & \sum\limits_{i=0}^{O-1}{r\left( {{h}_{i}}-{{h}_{D}} \right),\text{   }if\text{  }V\left( {{\Delta }_{LBP}} \right)\le 2} \\ & O+1,otherwise \\\end{align} \right.$      (1)

where, V(Δ_un-LBP) can be calculated by:

$\begin{align}  & V\left( {{\Delta }_{LBP}} \right)=\left| r\left( {{h}_{o-1}}-{{h}_{D}} \right)-r\left( {{h}_{i}}-{{h}_{D}} \right) \right| \\ & +\sum\limits_{i=0}^{O-1}{\left| r\left( {{h}_{i}}-{{h}_{D}} \right)-r\left( {{h}_{i-1}}-{{h}_{D}} \right) \right|} \\\end{align}$      (2)

The binary function r(a) can be expressed as:

$r\left( a \right)=\left\{ \begin{align}  & 1,a\ge 0 \\ & 0,a<0 \\\end{align} \right.$     (3)

The two-dimensional (2D) Gabor wavelet was selected to process the texture of the images in the surveillance video on online learning students, aiming to extract the facial emotional features from the spatial domain and the time domain simultaneously. Let ||*|| be the remainder operation; λ and u be the direction and scale of Gabor filter, respectively; l_v,u=[l_ucosθ_v] and l_max be the central frequency and maximum central frequency, respectively, with l_u=l_max/g and θ_v=vπ/8; c=(a, b) be the coordinates of a pixel in an image; δ_λ-u be the kernel function of the 2D Gabor wavelet. Then, the wavelet transform of a grayscale image GR(a, b) in the surveillance video can be expressed as:

${{P}_{\lambda \text{-}u}}\left( a,b \right)=\left\| GR\left( a,b \right)*{{\delta }_{\lambda \text{-}u}} \right\|$      (4)

where, δ_λ-u can be calculated by:

${{\delta }_{\lambda \text{-}u}}=\frac{{{\left\| {{l}_{\lambda \text{-}u}} \right\|}^{2}}}{{{\varepsilon }^{2}}}{{e}^{-\frac{{{\left\| {{l}_{\lambda \text{-}u}} \right\|}^{2}}{{\left\| c \right\|}^{2}}}{2{{\varepsilon }^{2}}}}}\left( {{e}^{i{{l}_{\lambda \text{-}u}}c}}-{{e}^{\frac{{{\varepsilon }^{2}}}{2}}} \right)$      (5)

Let FI_SB(a, b) and FI_XB(a, b) be the real part filter and imaginary part filter of Gabor wavelet, respectively; (FI*GR) be the convolution operation of GR(a, b) and wavelet filter. After convolutional filtering of GR(a, b) by FI_SB(a, b) and FI_XB(a, b) in eight directions and on five scales, the resulting real and imaginary parts of GR(a, b) need to be computed by formula (6), in order to obtain the facial feature map after wavelet transform:

$FEF\left( x,y \right)=\sqrt{{{\left( F{{I}_{SB}}\left( x,y \right)*GR\left( x,y \right) \right)}^{2}}+{{\left( F{{I}_{XB}}\left( x,y \right)*GR\left( x,y \right) \right)}^{2}}}$      (6)

Finally, the grayscales of N facial expression feature maps were stretched to produce the one-dimensional (1D) Gabor eigenvector of the original facial expression image.

To effectively extract the emotional expressions of online learning students, it is necessary to associate the emotional information of the voice in the surveillance video with the emotional information of the facial expression image. Suppose there are m frames in each second of the surveillance video on online learning students. Then, each frame of the voice signal can be processed with Hamming window at the speed of m frames per second. Let FL and SH be the frame length and frame shift of the voice signal, respectively; SF=3FL is the sampling frequency. Then, the frame length can be calculated by:

$FL\times m-\left( m-1 \right)\times SH=SF$     (7)

Formula 7 is equivalent to:

$FL=3SF/\left( 2m+1 \right)$     (8)

Let {g(l), g(2), …, g(M)} be the voice series processed by Hamming window, with M being the total number of frames. Then, the time series g(i) of the voice signal in frame i after Hamming window processing can be expressed as:

$g\left( i \right)=\left\{ {{v}_{i}}\left( j \right),j=1,2,\cdots ,FL \right\}$     (9)

The mean r(i) of the absolute values of all amplitudes v_i(j) in g(i) can be computed by:

$r\left( i \right)=\frac{\sum\limits_{j=1}^{FL}{\left| {{v}_{i}}\left( j \right) \right|}}{FL}$     (10)

Then, a series can be constructed based on r(i):

$a=\left\{ r\left( i \right),i=1,2,...,M \right\}$       (11)

Next, the voice frames and image frames in the surveillance video were paired on the time axis, and series a was sorted again by size. Based on the new series a', the image frame series IF(i) was re-sorted into:

$y=\left\{ I\left( i \right),i=1,2,...,N \right\}$

$b=\left\{ IF\left( i \right),i=1,2,...,M \right\}$     (12)

Considering the sheer size of the surveillance video, the image frames corresponding to the top 25% of voice frames in terms of amplitude were selected for extracting facial expression eigenvectors, in order to reduce the time cost of image frame processing. Suppose the LBP eigenvector LBP_i of frame i has θ dimensions. Then, we have:

$LB{{P}_{i}}=\left\{ {{\Delta }_{lbp-i1}},{{\Delta }_{lbp-i2}},...,{{\Delta }_{lbp-i\theta }} \right\}$      (13)

For an m-frame surveillance video on online learning students, the series of LBP eigenvectors can be expressed as:

$c=\left\{ LB{{P}_{1}},LB{{P}_{2}},...,LB{{P}_{m}} \right\}$       (14)

The Euclidean distance between LBP_i and the other (m-1) frames can be calculated by:

${{q}_{i,j}}=\sqrt{{{\left( {{\Delta }_{lbp-i1}}-{{\Delta }_{lbp-j1}} \right)}^{2}}+{{\left( {{\Delta }_{lbp-i2}}-{{\Delta }_{lbp-j2}} \right)}^{2}}+...+{{\left( {{\Delta }_{lbp-i\theta }}-{{\Delta }_{lbp-j\theta }} \right)}^{2}}}$     (15)

The corresponding 1D vector can be described by:

${{Q}_{i}}=\left[ {{q}_{i,1}},{{q}_{i,2}},...,{{q}_{i,M-1}} \right]$      (16)

The mean Q^'_i of Q_i can be calculated by:

${{{Q}'}_{i}}=\sum\nolimits_{j=1}^{M-1}{{{q}_{i,j}}}$     (17)

The selected m frames of facial expressions demonstrate the emotions well. If a frame is highly similar as the previous and subsequent frames, the mean Q^'_i will be small. In this case, the frame must be very similar to other frames. That is, the frame is relatively good at emotion expression in the frame series. This paper chooses the l frames with the minimum Q^'_i and the best emotional expression effect as the targets of CNN-based emotion recognition.

For online learning behavior analysis, image emotion recognition is premised on a large labeled dataset of image emotions of online learners. Nevertheless, it is extremely difficult to label a large dataset of image emotions, owing to the subjectiveness of image emotions. This paper constructs an image emotion classification model based on the attention mechanism (Figure 4). Specifically, the model consists of a CNN to acquire facial emotional features, a non-extreme channel attention mechanism that suppresses the negative effects of labels, and a class-specific activation map mechanism targeting the salient features of each type of emotions.

4.png

Figure 4. Attention mechanism-based image emotion classification model

To begin with, the traditional CNN was implemented to extract facial expression features. On this basis, a feature matrix was derived for each facial expression image in the surveillance video. Let x=1, 2, 3, …, q and y = 1, 2, 3, …, q be the coordinates of the feature matrix; q be the length and width of the feature matrix; n be the number of convolution kernels; TZ be the number of classes in the emotional dataset. Then, the features extracted from the m-th image samples can be defined as A^m_x,y,n∈ℝ^TZ. As a nonlinear activation function, sigmoid function can be described as g_SIG(a)=1/(1+e^-a). Let ω and f be the weight and bias of the attention mechanism, respectively. The feature matrix can be converted into a 1D distribution of attention, using sigmoid function:

$D{{V}_{n}}={{g}_{SIG}}\left( {{\omega }^{T}}\cdot {{A}_{x,y,n}}+f \right)$      (22)

Then, the attention distribution value was normalized to reduce the feature values of noisy labels:

$F{{B}_{n}}=\frac{{{e}^{D{{V}_{n}}}}}{\sum\nolimits_{n}{{{e}^{D{{V}_{n}}}}}}$      (23)

The attention distribution value FB_n satisfies ∑FB_n=1. Let U be the multiplication of the elements of the matrix. By multiplying the normalized FB_n with the feature matrix of the previous facial expression images in the surveillance video, the attention feature map can be derived as:

$SF_{x,y,n}^{m}=A_{x,y,n}^{m}\otimes FB_{n}^{m}$      (24)

The softmax function can be described as g_SM=(e^a/∑e^a). To compare with the traditional attention mechanism, the channel attention mechanism FB_n in the field of image processing can be established as:

$F{{B}_{n}}={{g}_{SM}}\left( {{\omega }^{T}}\cdot {{A}_{x,y,n}}+f \right)$      (25)

The corresponding spatial attention mechanism can be expressed as:

$F{{B}_{x,y,n}}={{g}_{SM}}\left( {{\omega }^{T}}\cdot {{A}_{x,y,n}}+f \right)$      (26)

For comparison, the traditional attention mechanism can be expressed as:

$F{{B}_{x,y,n}}=\frac{{{e}^{{{A}_{x,y,n}}}}}{\sum\nolimits_{i}{\sum\nolimits_{j}{{{e}^{{{A}_{x,y,n}}}}}}}$     (27)

Let IN_n^ED_l be the input eigenvector of the emotion detector ED_l after global average pooling; n_SE be the dimensionality of a specific emotion vector; l be the number of classes of the emotion dataset. Based on IN_n^ED_l and SF^m_x,y,n, the emotion activation map EA^EDl_x,y,n can be established as:

$EA_{x,y,n}^{E{{D}_{l}}}=\sum\nolimits_{n}{IN_{n}^{E{{D}_{l}}}\cdot }SF_{x,y,n}^{m}$     (28)

Let L be the number of classes for the given emotions. Thus, a total of L emotion detectors were designed to derive L class activation maps EA^EDl_x,y,n for salient features. Taking EA^EDl_x,y,n as local features, an attention feature matrix, i.e., global feature, SF^m_x,y,n could be established. Let ○ be the connection between features. Then, we have:

${{L}_{x,y,n}}=\left[ SF_{x,y,n}^{m}\circ EA_{x,y,n}^{{{\gamma }_{1}}}\circ EA_{x,y,n}^{{{\gamma }_{2}}}\circ ...EA_{x,y,n}^{{{\gamma }_{l}}} \right]$      (29)

To classify the emotions for online learning behavior analysis, the pretrained densely connected CNN was selected as the modeling basis. For the final emotion classification, the number of image samples was denoted as M, the input image as ST_i, the corresponding label as B_i. Then, the supervised learning method can be denoted as {ST_i,B_i}^M_i=1. Further, softmax function g_SM=(e^a/∑e^a) was taken as the final loss function for emotion classification. Through global average pooling, the final image emotion vector o_i=GAP(γ_i, j, n) was obtained. On this basis, the minimize cross entropy loss function can be established as:

$LOS{{S}_{CLA}}=-\sum\nolimits_{i=1}^{M}{{{B}_{i}}}log\frac{{{e}^{{{\omega }^{T}}{{o}_{i}}}}}{\sum\nolimits_{\gamma }{{{e}^{{{\omega }^{T}}{{o}_{i}}}}}}$     (30)

Suppose the M-th image belongs to the emotion class b_i, i∈ΣL. If B_i=1, the target image has the emotion of the current emotion detector; if B_i=0, the target image does not have the emotion of the current emotion detector. Suppose B_i is a one-hot code. Then, the loss function of the binary classifier for a specific class activation map can be established by:

$LOS{{S}_{TC}}=-\sum\nolimits_{\beta }{\sum\nolimits_{i=1}^{M}{{{b}_{i}}log{{{{b}'}}_{i}}+\left( 1-{{b}_{i}} \right)}}\left( 1-log{{{{b}'}}_{i}} \right)$      (31)

where, b'_I can be calculated by:

${{{b}'}_{i}}=\frac{1}{\left( 1+{{e}^{-{{g}_{i}}}} \right)}$       (32)

To further improve the emotion classification effect, this paper treats emotion classification as a linear problem, and introduces two regularization terms for the output emotion classes: the central loss to reduce inter-class distance, and the triplet loss to increase the inter-class distance. Let γ_Bi be the class center of eigenvector o_i. Then, the central loss LOSS_CL can be described as:

$LOS{{S}_{CL}}=\frac{1}{2}\sum\nolimits_{i=1}^{M}{\left\| {{o}_{i}}-{{\gamma }_{{{B}_{i}}}} \right\|}_{2}^{2}$      (33)

where, γ_Bi can be updated by:

${{\gamma }_{{{B}_{i}}}}=\frac{{{B}_{i}}{{e}^{P{{A}^{T}}{{o}_{i}}}}}{o\cdot \sum\nolimits_{\gamma }{{{e}^{P{{A}^{T}}{{o}_{i}}}}}}$       (34)

Only if ∑^o_j=1B_ij=1, could γ_Bi be updated based on o_i. Suppose φ is a hyperparameter. Then, the triplet loss LOSS_TL can be described as:

$LOS{{S}_{TL}}=\sum\nolimits_{i=1}^{M}{\left[ \left\| h\left( o_{i}^{e} \right)-h\left( o_{i}^{r} \right) \right\|_{2}^{2}-\left\| h\left( o_{i}^{e} \right)-h\left( o_{i}^{m} \right) \right\|_{2}^{2}+\phi  \right]}$      (35)

The other hyperparameter h(*) can be calculated by:

$h\left( * \right)=\left\| \frac{{{e}^{P{{A}^{T}}o{}_{i}}}}{\sum\nolimits_{\gamma }{{{e}^{P{{A}^{T}}o{}_{i}}}}} \right\|_{2}^{2}$     (36)

where, o^e_i and o^r_i correspond to the same emotion class; o^e_i and o^m_i correspond to different emotion classes. Let μ and η be the weight coefficients that balance the contributions of LOSS_CL and LOSS_TL. The final emotion classification loss function can be established by:

$\begin{align}  &   LOSS=LOS{{S}_{CLA}} \left( ST , B \right)+LOS{{S}_{TC}} \left( ST , b \right) \\ & +\mu LOS{{S}_{CL}} \left( o \right)+ \eta LOS{{S}_{TL}}\left( o \right) \\\end{align}$      (37)

This paper analyzes online learning behaviors based on image emotion recognition. After detailing the flow of image emotion recognition for online learning behavior analysis, this paper extracts the key frames from facial expression images through improved LBP and wavelet transform. Then, the mean expression feature was solved form several extracted key frames. The data and histogram evidences of experiments show that the proposed method can improve the effect of image emotion recognition, compared with different key frame extraction methods. After that, the authors established the structure of online learning behavior analysis system, proposed a learning emotion recognition method based on facial expressions, and constructed an emotional classification model for online learning images based on the attention mechanism. Finally, the authors drew the 2D scatterplots on the test set with different training cycles, and presented the results of online learning behavior analysis. The results demonstrate the classification effectiveness of the regularization terms in the attention mechanism-based image emotion classification model, and the applicability of our model to online learning behavior analysis.