CN111444764A - Gesture recognition method based on depth residual error network - Google Patents

Abstract

In order to solve the problems that gesture recognition in the prior art requires many conditions and the calculation amount of the model is relatively large, the present invention proposes a gesture recognition method based on a deep residual network, which includes the following steps: S1. The human hand is used as the detection target for target detection, and the detected human hand is stored as an image; S2. The hand position coordinates are obtained according to the human hand image collected in step S1, and then based on the depth residual error The joint recognition model of the network performs gesture key point detection to obtain the hand key point coordinates; S3. Input the gesture key point coordinates obtained in step S2 into the SoftMax classifier for separation, obtain the classification of various gestures, and finally recognize the gesture. The speed of recognizing gestures of the present invention is faster than other solutions.

Description

A gesture recognition method based on deep residual network

technical field

The invention relates to the technical field of gesture recognition, in particular to a gesture recognition method based on a deep residual network.

Background technique

The term gesture recognition refers to the entire process of tracking human gestures, recognizing their representations, and converting them into semantically meaningful commands. Gesture recognition research aims to design and develop systems that can recognize gestures for device control as inputs and by mapping commands to outputs. Generally speaking, whether the way to collect gesture interaction information is contact or non-contact, the gesture interaction system can be divided into two types: touch-based sensors and non-contact-based sensors.

Touch-sensor-based gesture recognition is typically based on technologies such as data gloves, accelerometers, multi-touch screens, etc. that use multiple sensors. In 2004, Kevin et al. designed a wireless instrument glove for gesture recognition. In 2008, Ren Cheng et al. of Beihang University studied virtual hands in virtual reality systems with helmets and data gloves. In 2015, Lu Lei and others from Shandong Normal University studied a static gesture recognition method based on data gloves, which can recognize 25 gestures with a correct rate of 98.9%. In 2007, Bourke et al. proposed a recognition system that uses an accelerometer to detect normal gestures used in our daily activities. In 2017, Wang Linlin and others from the University of Electronic Science and Technology of China studied a gesture interaction method based on inertial sensors, with an accuracy rate of 96.7%. In 2014, Xue Jiao and others from the University of Chinese Academy of Sciences studied a touch screen-based gesture remote control system with an average recognition rate of 99%.

Gesture recognition based on non-contact sensors is usually based on the use of optical sensing, radar detection and other technologies. In 2002, gesture recognition using a camera to capture multi-scale color features was proposed. In 2010, Sha Liang of Tsinghua University and others studied human-computer interaction technology based on unmarked full gesture vision, and proposed a solution for a vehicle-mounted gesture-visual interaction system using a universal camera, with a recognition rate of 80% in complex environments. In 2011, Microsoft announced the Kinect, a camera that uses infrared light to recognize gestures. In 2015, Jiang Ke et al. of Jiangnan University used Kinect to study 3D gesture recognition based on depth images, with a recognition rate of 76.6%. In 2015, Google's ATAP division unveiled Project Soli, which uses tiny radars to recognize gesture movements that can capture tiny movements.

Although the prior art has achieved relatively accurate recognition on different gesture datasets, it still has the following deficiencies: (1) Some key parameters of preprocessing and feature extraction need to be set by human experience. (2) For gestures in various environments, gesture recognition only relying on an independent feature often cannot meet the requirements of gesture recognition.

Human-computer interaction technology is gradually changing from computer-centric to human-centric, and gesture recognition as an important human-computer interaction method has received extensive attention. Gesture recognition technology is the first to judge the user's gesture operation by obtaining the bending degree of the finger and the activity state of the hand through the wearable inductive glove sensor. This kind of technology requires wearing special equipment, and the cost is high and the interaction method is not natural enough. It has been gradually replaced by image recognition technology.

Traditional image recognition technology analyzes and distinguishes gesture types through the edge, brightness, color and other characteristics of the image, which is easily affected by factors such as light transformation, blind spots and complex backgrounds, and the algorithm has low robustness. With the introduction of deep learning in 2006, gesture recognition technology based on deep learning has become a research hotspot because of its lower hardware requirements, faster recognition speed and higher recognition accuracy. Since the human hand is a complex deformable body, and gestures have the characteristics of diversity, ambiguity, and time differences, there are also some difficulties in gesture recognition technology based on deep learning.

In the current performance of many different methods in different test datasets, most of the current methods have achieved gesture recognition in isolated hand scenes, but gesture recognition in complex background environments is still a challenge. For this reason, some Discriminative methods try to incorporate depth information to reduce the difficulty of hand rendering. Some literature first proposed a joint motion tracking method for two strongly interacting hands. The method uses a 54-dimensional parameter space to represent various possible shape structures of the two hands, in which each dimension parameter represents a motion structure with 26 degrees of freedom. At the same time, particle swarm optimization algorithm is added to perform asymptotic random optimization, so as to find the best explanation for RGB The shape structure of the hands in the observations provided by the -D sensor; however, this method mainly focuses on realizing the ability of the model to accurately recognize the overlapping hands, and the speed of real-time gesture detection is not mentioned. There are also methods that also hope to solve the problem of gesture recognition under objective occlusion or blind spots. It proposes to use differentiated learning on the fingers and associate the distinguishing point features of the fingers with gestures, and also consider the edge of the image, optical flow and collision, etc. However, due to the large number of conditions required for recognition and the large amount of calculation of the model, it is not applicable in actual human-computer interaction applications. .

SUMMARY OF THE INVENTION

In order to solve the problems that the gesture recognition in the prior art requires many conditions and the calculation amount of the model is large, the present invention proposes a gesture recognition method based on a deep residual network.

The technical solution adopted by the present invention to solve the above-mentioned technical problems is: a gesture recognition method based on a deep residual network, which is characterized in that it includes the following steps:

S1. Collect video data, use the human hand as a detection target for target detection, and store the detected human hand as an image;

S2. Obtain hand position coordinates according to the human hand image collected in step S1, and then perform gesture key point detection through a joint recognition model based on a deep residual network to obtain hand key point coordinates;

S3. Input the coordinates of the gesture key points obtained in step S2 into the SoftMax classifier for separation, to obtain the classification of various gestures, and finally recognize the gesture.

The specific method of using the human hand as the detection target to perform target detection in the step S1 is:

S101. Use a human hand video as an image through a convolutional neural network, and generate candidate frames of different sizes from the obtained feature map;

S102. Match and judge the candidate frame described in step S101 and the training sample calibration frame to obtain a target sample;

S103. Update the parameter setting of the convolutional neural network described in S101 by the total loss function obtained by the weighted sum of the position offset loss of the prediction frame and the calibration frame and the loss of the classification result, so as to obtain a more accurate model for human-hand target detection, and finally An image containing a human hand is obtained.

The criterion for matching judgment in the step S102 is: when the overlap area ratio of a candidate frame and the calibrated hand frame is greater than the overlap area ratio of all other candidate frames, it is considered that the candidate frame is successfully matched; When the matching degree between the frame and the calibrated hand frame is greater than a certain threshold, it is considered that the candidate frame is successfully matched; if one of the above two conditions is met, it is judged that the matching is successful, and when the candidate frame is judged to be successful, the candidate frame is determined. The excitation is a positive sample that obtains the predicted result, otherwise it is a negative sample.

The specific method for detecting gesture key points based on the joint recognition model of the deep residual network in the step S2 is:

S201. When training the recognition model, collect two-dimensional images of the same gesture at different angles at the same time, and use the pre-training detector to detect all the key points of the hand in the above-mentioned test video data, and obtain the recognition results of the same point at different angles ;

S202. Construct the three-dimensional position of each key point and the three-dimensional model of the entire hand through the RANSAC algorithm based on the identification results of the same point at different angles obtained in step S201, so that each moment becomes a picture frame with three-dimensional joint point characteristics, so as to Obtain the output image of hand joint point calibration;

S203. At this time, the calibrated hand image in the pre-training detector and the output image of the hand joint point calibration obtained in step S202 are used as new training samples to continue to train and update the above model, and obtain multi-view guidance for two-dimensional input. image;

S204. Finally, the Multiview Bootstrapping algorithm is used to identify the two-dimensional input image with the multi-view guidance described in step S203, and the coordinates of the key points of the hand are obtained.

The beneficial effects of the present invention are as follows: the present invention utilizes multiple optimization architectures of the deep residual network in deep learning to mix, and realizes a method that can perform independent gesture recognition functions, which ensures high accuracy and high robustness of the recognition results. At the same time, the speed of smooth recognition can be achieved; at the same time, the method of the present invention uses a single camera or a small number of cameras to recognize more gestures in complex background scenes than other methods when applying recognition. The edge, brightness, color and other characteristics of the image are used to analyze and identify the gesture type, which is easily affected by factors such as light and angle transformation, and the algorithm has low robustness. The present invention preprocesses the collected image and performs key point detection in subsequent steps. Changes in factors such as , light and other factors have little impact on the results, and the recognition response time is faster than other solutions.

Description of drawings

Fig. 1 is the working flow chart of the present invention.

Figure 2 shows the network structure of gesture target detection.

Detailed ways

The present invention will be further described below with reference to the accompanying drawings.

As shown in Figure 1, the described method for gesture recognition based on deep residual network includes the following steps:

Step (1): input video data to perform multi-scale human hand detection, use the human hand as a detection target to perform target detection, and store the detected human hand as an image;

Hand detection is a standard object detection problem. The traditional object detection method is to locate and classify the target objects by extracting the perceptual information of different color modules in the picture. But for the computer, it is faced with the RGB pixel matrix, it is difficult to directly get the abstract concept of the target object (such as cat, dog) from the image and locate its position, plus there are multiple objects mixed with the cluttered background The impact of the target detection is more difficult. Although there are many common feature sets for some specific research directions such as face detection and behavior detection in the traditional vision field, the calculation speed is difficult to improve due to the complex detection process. When the target detection method based on deep learning was first proposed, it required two steps of region nomination and region classification. Region nomination is to densely extract possible regions of interest in the image before the input image enters the convolutional neural network, and then identify and classify each nominated region. This method has a good guarantee for the accuracy of target detection. Compared with the traditional visual method, its detection speed is greatly improved, but it still cannot meet the real-time requirements.

Based on this, this step further optimizes the target detection model, and proposes to omit the step of region nomination, and directly put the entire input image into the deep residual network to generate the position coordinate value of the object.

The human hand object detection model is trained using an open source hand database. The input image is passed through the convolutional neural network, and the points of the obtained feature map generate candidate frames of different sizes in advance, and then match and judge with the calibration frame of the training sample to obtain the target sample. The target samples include the following positive samples and negative samples.

The criterion for matching judgment is: when the overlap area ratio between a candidate frame and the calibrated hand frame is greater than the overlap area ratio of all other candidate frames, the candidate frame is considered to be successfully matched; when a candidate frame and the calibrated hand frame overlap When the matching degree of the box is greater than a certain threshold, the candidate box is considered to be successfully matched. If one of the above two conditions is satisfied, it is judged that the matching is successful. When the candidate frame is judged to be successful, the candidate frame is excited as a positive sample for obtaining the predicted result, otherwise, it is a negative sample. After that, the total loss function obtained by the weighted sum of the position offset loss of the prediction frame and the calibration frame and the loss of the classification result is calculated to update the parameter settings of the convolutional neural network to obtain a more accurate model for human target detection. The specific detection process As shown in Figure 2 of the patent. The hand detection model can obtain all the detected hand position coordinates in the input image, and based on this, the next gesture key point recognition is performed.

Step (2): After confirming that there is a human hand in the input image and obtaining the position coordinates of the human hand, we need to identify and analyze the specific coordinates of the detected hand. Since there are many joints in the hand, and rich interaction scenarios are easily generated between the hand and the hand, and between the hand and general objects, it is an important basis for coordinate recognition to accurately identify the joints of the same hand from the same person.

The present invention adopts the Multiview Bootstrapping algorithm to realize gesture recognition under complex environment for the two-dimensional input image through multi-view guidance.

When training the recognition model, use multiple cameras to capture two-dimensional images of the same gesture at different angles at the same time, and use a pre-trained detector generated from a small number of calibrated hand images to detect all hands in the uncalibrated test video data. key points are detected.

For the recognition results of the same point at different angles, the 3D position of each key point and the 3D model of the entire hand are constructed through the RANSAC algorithm, so that each moment becomes a picture frame with the characteristics of 3D joint points, and the calibration of hand joint points is obtained. the output image. At this time, the initial small amount of calibration material and the calibration image detected by the pre-training detector are used as new training samples to continue to train and update the model.

Step (3): The hand key point detector can obtain the vector form output of the position coordinates of each hand key point, and directly input it to the SoftMax classifier for separation, and obtain the classification of various gestures, thereby realizing gesture recognition.

Specifically, the SoftMax classifier maps the signal to be separated to the corresponding label, and the signal after the convolutional neural network training will obtain a classification result. The result is compared with the corresponding label data to obtain the relative error value, and the classification error is continuously reduced and a model with better classification ability is obtained by multiple neural network training.

Embodiment 1: The SoftMax classifier uses ResNet-50 pre-trained on ImageNet as a feature extraction model, and sets the dimension of the output layer according to the number of IDs in the dataset. During the training process of the image classifier, the parameters of the BN layer, the conv1 layer and the res2 layer in the ResNet-50 network were frozen, and the parameters were updated using mini-batch SGD. The number of samples in a batch, the maximum number of iterations and momentum are set to 16, 50 and 0.9, respectively. The learning rate uses a class decay strategy, the initial learning rate is 0.001, and the learning rate decays to 0.0001 after 40 epochs until the end of training.

The loss function of the SoftMax classifier of the present invention is as follows:

where y _ik represents the real label information, p _ik represents the model prediction sample, N is the total number of training samples, K is the category of the sample, the regularization parameter λ=0.0005, for W={W ₁ …W ₁₀₂₄ }, where the W _i dimension is equal to the output dimension.

The above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art who is familiar with the technical scope disclosed by the present invention can easily change or replace them, all belonging to the scope of the present invention. within the protection scope of the present invention. Therefore, the protection scope of the present invention is described in accordance with the protection scope of the claims.

Claims (5)

1. A gesture recognition method based on a depth residual error network is characterized by comprising the following steps:

s1, collecting video data, carrying out target detection by taking a hand of a person as a detection target, and storing the detected hand of the person as an image;

s2, obtaining hand position coordinates according to the hand image of the person collected in the step S1, and detecting gesture key points through a joint recognition model based on a depth residual error network to obtain hand key point coordinates;

and S3, inputting the gesture key point coordinates obtained in the step S2 into a SoftMax classifier for separation to obtain various gesture classifications, and finally recognizing the gestures.

2. The gesture recognition method based on the deep residual error network of claim 1, wherein the specific method for performing target detection with human hands as detection targets in the step S1 is as follows:

s101, taking a hand video of a person as an image, and generating candidate frames with different sizes from the obtained feature map through a convolutional neural network;

s102, matching and judging the candidate frame and the training sample calibration frame in the step S101 to obtain a target sample;

and S103, updating S101 the parameter setting of the convolutional neural network by a total loss function obtained by weighted sum of the position offset loss and the classification result loss of the prediction frame and the calibration frame to obtain a more accurate model for detecting the human hand target, and finally obtaining an image containing the human hand.

3. The method according to claim 1, wherein the matching judgment criterion in step S102 is as follows: when the overlapping area ratio of one candidate frame to the calibrated hand frame is larger than the overlapping area ratio of all the other candidate frames, the candidate frame is considered to be successfully matched; when the matching degree of a certain candidate frame and the calibrated hand frame is greater than a certain threshold value, the candidate frame is considered to be successfully matched; if one of the two conditions is satisfied, the matching is judged to be successful, when the candidate frame is judged to be successful, the candidate frame is excited to be a positive sample for obtaining the prediction result, otherwise, the candidate frame is a negative sample.

4. The gesture recognition method based on the depth residual error network as claimed in claim 1, wherein the specific method for performing gesture key point detection based on the joint recognition model of the depth residual error network in the step S2 is as follows:

s201, when a recognition model is trained, collecting two-dimensional images of the same gesture at different angles at the same time, and detecting all hand key points in the test video data by using a pre-training detector to obtain recognition results of the same point at different angles;

s202, constructing a three-dimensional position of each key point and a three-dimensional model of the whole hand by using the RANSAC algorithm according to the recognition results of the same point at different angles obtained in the step S201, so that each moment becomes a picture frame with three-dimensional joint point characteristics to obtain an output image calibrated by the hand joint points;

s203, taking the calibrated hand image in the pre-training detector and the output image calibrated by the hand joint point obtained in the step S202 as new training samples to continue training and updating the model to obtain a multi-view guide pair two-dimensional input image;

and S204, finally, identifying the two-dimensional input image by the multi-view guide in the step S203 by adopting a Multiview boosting algorithm to obtain the coordinates of the key points of the hand.

5. The method according to claim 1, wherein the loss function of the SoftMax classifier is as follows:

wherein y is_ikRepresenting genuine label information, p_ikRepresents model prediction samples, N is the total number of training samples, K is the class of samples, the regularization parameter λ is 0.0005, and W is { W }₁…W₁₀₂₄In which W is_iIs equal to the output dimension.

Images