CN112699815A - Dynamic expression recognition method and system based on space-time motion enhancement network - Google Patents

Abstract

The invention discloses a dynamic expression recognition method and system based on a spatiotemporal motion enhancement network, comprising the following steps: acquiring a video sequence containing an expression image; inputting the video sequence into a Resnet-Emotion network to obtain the spatial feature of the expression; The spatial feature is input into the recursive refining unit network to obtain the motion feature of the expression; the motion feature is input into the gated recurrent unit network to obtain the temporal feature of the expression, and the recognition of the dynamic expression is completed. Based on the Resnet-Emotion network pre-trained by the expression data set, the invention realizes the dynamic expression recognition that enhances the expression movement feature of each video frame and mines the time information of the expression, and solves the problems existing in the current dynamic expression recognition methods. The problem of ignoring the motion characteristics of expression videos.

Description

Dynamic facial expression recognition method and system based on spatiotemporal motion enhancement network

technical field

The invention relates to the technical field of computer vision, in particular to a dynamic expression recognition method and system based on a spatiotemporal motion enhancement network.

Background technique

With the continuous improvement of facial expression recognition theory and the development of artificial intelligence technology, the field of facial expression recognition has received extensive attention. Expression recognition is mainly divided into two methods: expression recognition method based on static image and expression recognition method based on dynamic video. In the expression recognition method based on dynamic video, since the illumination, occlusion and head posture of the face are similar to the natural environment, if the expression recognition model can pay attention to the local motion details of the facial expression and highlight these motion details, it will undoubtedly improve the The accuracy of video expression recognition. In the video, these local motion details appear as the motion area between the upper and lower frames. How to use the motion characteristics in the video to represent the continuity of facial expressions has become the focus and difficulty of dynamic expression recognition.

SUMMARY OF THE INVENTION

In view of the deficiencies of the prior art, the purpose of the present invention is to provide a dynamic expression recognition method and system based on a spatiotemporal motion enhancement network, so as to solve the problem that the video expression recognition model in the prior art ignores the motion characteristics of the expression video.

In order to achieve the above object, the technical scheme adopted in the present invention is:

A dynamic expression recognition method, comprising the following steps:

Obtain a video sequence containing facial expressions;

Inputting the video sequence into the Resnet-Emotion network to obtain the spatial features of expressions;

Inputting the spatial feature into the recursive refining unit network to obtain the motion feature of the expression;

The motion features are input into the gated cyclic unit network to obtain the temporal features of expressions, and the recognition of dynamic expressions is completed.

Further, the acquisition process of the motion feature is as follows:

Obtain the original spatial feature of the current frame in the video sequence, the original spatial feature of the previous frame, and the motion feature output by the recursive refining unit network of the previous frame;

The original spatial features of the current frame, the original spatial features of the previous frame, and the motion features output by the recursive refining unit network of the previous frame are simultaneously input to the recursive refining unit network, and after being processed by the update gate model in the recursive refining unit network, we get overall attention map;

The overall attention map is input to the sigmoid function, and the motion feature of the current frame is output by the recursive refining unit network.

Further, the update gate model includes a transition layer, a spatial attention model and a channel attention model; the spatial attention model and the channel attention model are all connected with the transition layer.

Further, the process of updating the gate model is as follows:

The original spatial feature of the current frame, the original spatial feature of the previous frame, and the motion feature output by the recursive refining unit network of the previous frame are simultaneously input to the transition layer, and the first spatial feature is generated through the transition layer;

Inputting the first spatial feature into a spatial attention model, and obtaining a spatial attention map through the spatial attention model;

Inputting the first spatial feature into the channel attention model, and obtaining the channel attention map through the channel attention model;

The overall attention map is obtained by multiplying the spatial attention map with the channel attention map.

Further, the transition layer includes a convolutional layer, a batch normalization layer and a ReLU activation function that are set in sequence; the spatial attention model includes a global cross-channel average pooling layer, a first fully connected layer, a second A fully connected layer; the channel attention model includes a global spatial average pooling layer and a fully connected layer that are set in sequence.

Further, the spatial attention map is:

和

分别为第一全连接层、第二全连接层的参数，Relu为Relu激活函数，

为经全局跨通道平均池层后的空间特征；where Z _s is the spatial attention map,

and

are the parameters of the first fully connected layer and the second fully connected layer, Relu is the Relu activation function,

is the spatial feature after the global cross-channel average pooling layer;

The channel attention map is:

为经全局空间平均池层后的空间特征。Among them, Z _c is the channel attention map; W _c is the parameter of the fully connected layer;

is the spatial feature after the global spatial average pooling layer.

Further, the gated recurrent unit network includes an update gate, a reset gate, a candidate state and an output gate which are arranged in sequence.

Further, the update gate is:

z _t =σ(W _z ·[h _t-1 ,x _t ])

The reset gate is:

r _t =σ(W _r ·[h _t-1 ,x _t ])

Among them, z _t , r _t represent the update gate and the reset gate respectively; h _t-1 represents the output of the previous moment, x _t represents the input of the current moment, W _r represents the weight of the reset gate parameters, and σ represents the sigmoid function.

A dynamic expression recognition system, the system includes:

Acquisition module: used to acquire video sequences containing facial expressions;

The first input module: used to input the video sequence into the Resnet-Emotion network to obtain the spatial features of expressions;

The second input module: for inputting the spatial feature into the recursive refining unit network to obtain the motion feature of the expression;

The third input module is used for inputting the motion features into the gated recurrent unit network to obtain the temporal features of expressions and complete the recognition of dynamic expressions.

A dynamic expression recognition system, the system includes a processor and a storage medium;

the storage medium is used for storing instructions;

The processor is configured to operate in accordance with the instructions to perform the steps of the method described above.

Compared with the prior art, the beneficial effects achieved by the present invention are:

The present invention uses the pre-training residual network Resnet-Emotion based on migration learning, and inputs continuous frame video sequences on this basis, migrates the static facial expression recognition technology to the dynamic expression recognition technology, and uses the attention model in combination with the recursive refining unit, By enhancing the expression movement characteristics of each video frame, the problem that the dynamic expression recognition model ignores the movement characteristics of the expression video is solved; by using the recurrent neural network to obtain the time series information from the expression sequence, it can effectively help the network to construct the time series of the expression sequence. mold.

Description of drawings

Fig. 1 is the flow chart of the present invention;

Figure 2 is a schematic diagram of the spatiotemporal motion enhancement network structure;

Figure 3 is a schematic diagram of the structure of the recursive refining unit.

Detailed ways

The present invention will be further described below in conjunction with the accompanying drawings. The following examples are only used to illustrate the technical solutions of the present invention more clearly, and cannot be used to limit the protection scope of the present invention.

As shown in Figure 1 and Figure 2, the dynamic expression recognition method based on the spatiotemporal motion enhancement network first uses the Resnet-Emotion network pre-trained on the expression dataset to extract the spatial features of facial expressions, and then uses the recursive refining unit (Refining Recurrent Unit). , RRU) to enhance the expression motion features of each video frame, and finally use GRU to mine the temporal information of expressions. Specifically include the following steps:

S1, using a pre-trained Resnet-Emotion network based on transfer learning to extract the spatial information of expressions from video sequences:

Obtaining the spatial features of expressions in a video sequence includes the following processes:

S11, construct a set of video sequences with multiple expression images;

Since the number of expression frames detected in each video is different, when selecting the number of expression frames as n, we adopt the following strategy: first check the number N of expression frames in the sample, if N ≥ n, start from the [0th ,N-n] frame is randomly selected to start reading, and n frames of expression images are continuously read. If N<=n, start reading from the first frame, read the Nth frame, and then repeatedly read the Nth frame expression samples N-n times, so as to ensure that the read expressions are coherent.

S12 , feature extraction is performed on the expression images in the video sequence through the Res-Emotion feature extraction module, and a feature vector with a dimension of T×512×H×W is obtained, that is, the spatial feature of the facial expression.

S2, using a recursive refinement unit to enhance the expression motion features of each video frame.

The spatial features of facial expressions obtained by S1 are input into the RRU in chronological order for feature refinement, which specifically includes the following steps:

S21, as shown in Figure 3, the spatial feature input into the recursive refining unit each time includes the following three parts;

The original spatial feature X _i,k,t-1 of the previous frame, the motion feature S _i,k,t-1 output by the RRU of the previous frame, and the original spatial feature X _i,k,t of the current frame.

S22, input the original spatial feature of the previous frame, the motion feature outputted by the RRU of the previous frame, and the original spatial feature of the current frame into the recursive refining unit at the same time, the above-mentioned features are subjected to the update gate model of the recursive refining unit, and then output the spatial feature S _i,k,t ;

The update gate model of the recursive refining unit handles the above features as follows:

S221, the update gate model includes a transition layer, a spatial attention model, and a channel attention model set in sequence; the original spatial feature of the previous frame, the motion feature output by the RRU of the previous frame, and the original spatial feature of the current frame are sequentially input to transition layer, spatial attention model and channel attention model;

The transition layer consists of a convolutional layer of C 1×1 filters, a batch normalization (BN) layer, and a ReLU activation function. The transition layer integrates the information of each pixel on different channels, while retaining the motion information of each spatial position and reducing the feature dimension.

The original spatial features of the previous frame, the motion features output by the RRU of the previous frame, and the original spatial features of the current frame are sequentially input to the convolutional layer, batch normalization layer and ReLU activation function to obtain the first spatial feature Z _t ∈ R ^C×H×W .

The spatial attention model includes a global cross-channel average pooling layer and two FC layers;

In order to make the network pay more attention to the important spatial information parts in the feature map, a spatial attention model is used. For the first spatial feature Z _t ∈ R ^C×H×W , it is first input to the global cross-channel average pooling layer to obtain the overall response of each spatial location, and then input to the two FC layers in turn to generate the spatial attention map Z _s ∈ R ^1×H×W :

和

为FC层的参数，Relu为Relu激活函数，

为经全局跨通道平均池层后的空间特征。in,

and

is the parameter of the FC layer, Relu is the Relu activation function,

is the spatial feature after global cross-channel average pooling layer.

The channel attention model includes a global spatial average pooling layer and an FC (full connection) layer

然后输入至FC(全连接)层计算得到通道注意图Z_c∈R^C×1×1，公式如下：For neural networks, it is not enough to only focus on the spatial response of features. For the same feature map, the usefulness of the feature map of each channel is different, so the channel attention model is introduced. For the first spatial feature Z _t ∈ R ^C×H×W , it is first input to a global spatial average pooling layer to obtain the overall response of each channel

Then input to the FC (full connection) layer to calculate the channel attention map Z _c ∈ R ^C×1×1 , the formula is as follows:

为经全局空间平均池层后的空间特征。Among them, W _c is the parameter of the fully connected layer;

is the spatial feature after the global spatial average pooling layer.

After the spatial attention map and channel attention map of the feature are obtained, the spatial attention map and the channel attention map are multiplied to obtain the overall attention map of the motion feature. Finally, the overall attention map is input to the sigmoid function, which normalizes the overall attention map to a range between 0 and 1, and the output Z of the update gate is as follows:

Z=σ(Z _s ⊙Z _c )

Finally, the motion features S _{i, k, t} output by the RRU module are as follows:

S _i,k,t =(1-Z)⊙S _i,k,t-1 +Z⊙X _i,k,t

The value of each spatial position in Z represents the probability that the activation value of the corresponding position of the current frame feature X _i,k,t is preserved. A higher probability value indicates that the update gate considers the current feature at that location to be of higher quality, and the more features it should keep. Conversely, a lower probability indicates that the update gate considers the current feature at that location to be of lower quality, and the more refined features from the previous frame should be used. This approach can effectively utilize the useful motion information of historical frames and improve the quality of frame-level features.

S3, use GRU (Gated Recurrent Unit Network) to mine the temporal information of expressions.

The internal structure of GRU (Gated Recurrent Unit Network) includes update gate, reset gate, candidate state and output gate.

The forward propagation formula of GRU (Gated Recurrent Unit Network) is as follows:

z _t =σ(W _z ·[h _t-1 ,x _t ])

r _t =σ(W _r ·[h _t-1 ,x _t ])

表示当前时刻候选集，z_t,r_t分别表示为更新门和重置门。Among them, h _t-1 represents the output of the previous moment, x _t represents the input of the current moment, h _t represents the output of the current moment,

Represents the candidate set at the current moment, and z _t and r _t represent the update gate and the reset gate, respectively.

上，重置门越小，前一状态的信息被写入的越少。这两个门控机制的特殊之处在于，它们能够保存长期序列中的信息，且不会随时间而清除或因为与预测不相关而移除。The update gate is used to control the degree to which the state information of the previous moment is brought into the current state. The larger the value of the update gate, the more the state information of the previous moment is brought in. The reset gate controls how much information from the previous state is written to the current candidate set

On the other hand, the smaller the reset gate, the less information from the previous state is written. What is special about these two gating mechanisms is that they preserve information in long-term series and do not clear over time or remove as irrelevant to prediction.

A dynamic expression recognition system based on a spatiotemporal motion enhancement network, the system comprising:

Acquisition module: used to acquire video sequences containing facial expressions;

The first input module: used to input the video sequence into the Resnet-Emotion network to obtain the spatial features of expressions;

The second input module: for inputting the spatial feature into the recursive refining unit network to obtain the motion feature of the expression;

The third input module is used for inputting the motion features into the gated recurrent unit network to obtain the temporal features of expressions and complete the recognition of dynamic expressions.

A dynamic expression recognition system based on spatiotemporal motion enhancement network, the system includes a processor and a storage medium;

the storage medium is used for storing instructions;

The processor is configured to operate in accordance with the instructions to perform the steps of the method described above.

A computer-readable storage medium on which a computer program is stored, characterized in that, when the program is executed by a processor, the steps of the above-mentioned method are implemented.

The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present application. It will be understood that each flow and/or block in the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to the processor of a general purpose computer, special purpose computer, embedded processor or other programmable data processing device to produce a machine such that the instructions executed by the processor of the computer or other programmable data processing device produce Means for implementing the functions specified in a flow or flow of a flowchart and/or a block or blocks of a block diagram.

These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory result in an article of manufacture comprising instruction means, the instructions The apparatus implements the functions specified in the flow or flow of the flowcharts and/or the block or blocks of the block diagrams.

These computer program instructions can also be loaded on a computer or other programmable data processing device to cause a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process such that The instructions provide steps for implementing the functions specified in the flow or blocks of the flowcharts and/or the block or blocks of the block diagrams.

The above are only examples of the present invention, and are not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention are included in the application for pending approval of the present invention. within the scope of the claims.

Claims (10)

1. a dynamic expression recognition method, is characterized in that, comprises the following steps: Obtain a video sequence containing facial expressions; Inputting the video sequence into the Resnet-Emotion network to obtain the spatial features of expressions; Inputting the spatial feature into the recursive refining unit network to obtain the motion feature of the expression; The motion features are input into the gated cyclic unit network to obtain the temporal features of expressions, and the recognition of dynamic expressions is completed. 2. a kind of dynamic expression recognition method according to claim 1, is characterized in that, the acquisition process of described motion feature is as follows: Obtain the original spatial feature of the current frame in the video sequence, the original spatial feature of the previous frame, and the motion feature output by the recursive refining unit network of the previous frame; The original spatial features of the current frame, the original spatial features of the previous frame, and the motion features output by the recursive refining unit network of the previous frame are simultaneously input to the recursive refining unit network, and after being processed by the update gate model in the recursive refining unit network, we get overall attention map; The overall attention map is input to the sigmoid function, and the motion feature of the current frame is output by the recursive refining unit network. 3. a kind of dynamic expression recognition method according to claim 2, is characterized in that, described update gate model comprises transition layer, spatial attention model and channel attention model; Described space attention model, channel attention model are connected to the transition layer. 4. a kind of dynamic expression recognition method according to claim 3, is characterized in that, the processing procedure of described updating gate model is as follows: The original spatial feature of the current frame, the original spatial feature of the previous frame, and the motion feature output by the recursive refining unit network of the previous frame are simultaneously input to the transition layer, and the first spatial feature is generated through the transition layer; Inputting the first spatial feature into a spatial attention model, and obtaining a spatial attention map through the spatial attention model; Inputting the first spatial feature into the channel attention model, and obtaining the channel attention map through the channel attention model; The overall attention map is obtained by multiplying the spatial attention map with the channel attention map. 5. A kind of dynamic expression recognition method according to claim 3, is characterized in that, described transition layer comprises the convolution layer, batch normalization layer and ReLU activation function that are set in sequence; Described spatial attention model comprises sequentially A set global cross-channel average pooling layer, a first fully connected layer, and a second fully connected layer; the channel attention model includes a global spatial average pooling layer and a fully connected layer set in sequence. 6. a kind of dynamic expression recognition method according to claim 5, is characterized in that, described space attention map is:

where Z _s is the spatial attention map, W _s ¹ and

are the parameters of the first fully connected layer and the second fully connected layer, Relu is the Relu activation function,

is the spatial feature after the global cross-channel average pooling layer; The channel attention map is:

Among them, Z _c is the channel attention map; W _c is the parameter of the fully connected layer;

is the spatial feature after the global spatial average pooling layer. 7 . The dynamic expression recognition method according to claim 1 , wherein the gated cyclic unit network comprises an update gate, a reset gate, a candidate state and an output gate which are set in sequence. 8 . 8. a kind of dynamic expression recognition method according to claim 7, is characterized in that, described update gate is: z _t =σ(W _z ·[h _t-1 ,x _t ]) The reset gate is: r _t =σ(W _r ·[h _t-1 ,x _t ]) Among them, z _t , r _t represent the update gate and the reset gate respectively; h _t-1 represents the output of the previous moment, x _t represents the input of the current moment, W _r represents the weight of the reset gate parameters, and σ represents the sigmoid function. 9. A dynamic expression recognition system, wherein the system comprises: Acquisition module: used to acquire video sequences containing facial expressions; The first input module: used to input the video sequence into the Resnet-Emotion network to obtain the spatial features of expressions; The second input module: for inputting the spatial feature into the recursive refining unit network to obtain the motion feature of the expression; The third input module is used for inputting the motion features into the gated recurrent unit network to obtain the temporal features of expressions and complete the recognition of dynamic expressions. 10. A dynamic expression recognition system, wherein the system comprises a processor and a storage medium; the storage medium is used for storing instructions; The processor is adapted to operate in accordance with the instructions to perform the steps of the method according to any of claims 1-8.

Images