CN111209848B - A real-time fall detection method based on deep learning - Google Patents

Abstract

The invention relates to a real-time fall detection method based on deep learning, wherein the detection method comprises the following steps of: sequentially taking out single-frame pictures from a video stream after the analysis of a camera or a local video, inputting the single-frame pictures into an openpore algorithm model human body for detection to obtain key point coordinates of each part of the human body, using an SSD-Mobilene algorithm to exclude key points of non-human body areas, and a falling detection module: the method uses a common camera, has lower requirements on environment and use angle, and simultaneously has low cost, instantaneity, lower false detection rate, higher robustness and adaptability to different complex scenes.

Description

A real-time fall detection method based on deep learning

technical field

The invention relates to a real-time fall detection method based on deep learning. According to the openpose human body posture estimation algorithm combined with the SVDD classification algorithm, a more accurate and efficient real-time fall detection algorithm is designed, which belongs to the field of deep learning technology, human body posture estimation technology, and image processing technology.

Background technique

In recent years, with the acceleration of my country's population aging, the safety and health of the elderly has become a topic of widespread concern in society. According to the report of the World Health Organization, about 28-35% of the elderly aged 65 and above have fallen every year. Therefore, it is very necessary to accurately and quickly identify falls and notify the guardians or medical staff of the elderly. At present, there is a shortage of nursing staff in many nursing homes, and the problem of the physical safety of the elderly cannot be discovered in time, and there are great defects in the supervision of the elderly.

In recent years, researchers have proposed a variety of methods for automatic fall detection, including wearable device-based fall detection methods, environmental sensor-based fall detection methods, and machine vision-based fall detection methods.

Wearable fall detection system detection is not limited by space, but it often requires the person to be tested to wear wearable devices for a long time. These devices are usually cumbersome to install, and users will feel bored after wearing them for a long time. Moreover, the elderly usually have a certain degree of memory weakening and forget to wear wearable devices. The fall detection method based on the environmental sensor has the advantages of not needing to be worn, having a large detection area, and having the advantage of being used by multiple people for a long time after one installation, and can be applied to occasions such as families and nursing homes. The disadvantage is that compared with the other two methods, the requirements for environmental equipment are higher, the cost is relatively high, and the accuracy rate is also unstable.

With the development of technologies such as artificial intelligence and pattern recognition, more and more fall detection methods based on machine vision have been proposed, and monitoring systems have been widely used as an important technology in the field of public safety. However, most of the current monitoring systems only stay at the stage of manual monitoring of captured video signals and post-event video analysis, which requires a lot of material and manpower. The so-called intelligent monitoring system requires the computer to automatically analyze and process the captured image data in real time, detect, identify and track the target, and analyze the behavior of the target. When the target behaves abnormally, the monitoring system will perform a series of operations such as alarming and saving video data. Combining the intelligent monitoring system with fall detection, the guardian or the police can be notified as soon as the elderly fall, which greatly shortens the time for the elderly from falling to receiving treatment, reduces the possibility of secondary injury, and reduces the cost of later treatment. It is of great significance for improving the quality of life of the elderly in their later years and reducing the waste of social public resources.

Contents of the invention

The technical problem to be solved by the present invention is to improve the speed and accuracy of detection in view of the defects of false detection rate and real-time performance of the existing technology of fall detection using traditional image processing. .

In order to solve the above technical problems, the present invention provides a real-time fall detection method based on deep learning.

The main steps of the fall detection algorithm are as follows:

Step 1, using a camera and other equipment to collect human body images;

Step 2, carry out key point recognition of human skeleton;

Step 2.1, use the neural network convolution VGG-19 to extract features, predict the heat map of each key point, through multi-stage network iteration, the output of each stage is used as the input of the next stage, and optimize the results of the previous stage;

Step 2.2, add vector encoding to the predicted key points, combine the key points in the image, and connect different parts of the same person;

Step 2.3, transfer learning training image data set, so that the model can be better applied to specific scenarios;

Step 3, use the SSD-Mobilenet target detection algorithm to perform target detection on the detected human body key point area, and remove the non-human body part;

Step 3.1, first use the Mobilenet network to pre-train the COCO dataset to generate a pre-training model;

Step 3.2, using the data collected in step 2.3 to generate tfrecord data for transfer learning;

In step 3.3, step 3.3 transfers the data generated in step 3.2 and the fusion feature of the pre-training model generated in step 3.1 into the SSD network, so that the model can converge after a short period of training and has better detection ability;

Step 4, use the SVDD classification algorithm to classify the collected key points of human bones.

Beneficial effect

The advantages and innovations of this method are mainly reflected in the following aspects:

(1) Realize a fully automatic algorithm process without manual intervention or minimal intervention, which solves the shortcomings of traditional video surveillance methods that consume manpower and material resources.

(2) Human body pose estimation algorithm using the combination of openpose and SSD-Mobilenet algorithm

For the key points detected by the openpose algorithm, the SSD-Mobilenet target detection algorithm is used to detect human targets, filter out non-human areas, and reduce the misjudgment rate of the algorithm.

(3) For the problem of unbalanced data samples of falls, use the SVDD algorithm to classify, and pass

The parameters are optimized by grid search and other methods. Form the normal domain of positive sample training, and detect the fall behavior through the normal domain.

(4) Combining (2) and (3) to build a robust and accurate real-time fall detection framework, using a variety of target detection, image processing and other multi-field algorithms, with innovative algorithm applications.

Description of drawings

The above and/or additional aspects and advantages of the present invention will become apparent and easy to understand from the following description of the embodiments in conjunction with the accompanying drawings, wherein:

FIG. 1 is an overall flow chart of fall detection in a real-time example of the present invention.

FIG. 2 is a network structure diagram of the neural network feature extraction part of the embodiment of the present invention.

Fig. 3 is a diagram of detection results of key points of a human body according to an embodiment of the present invention.

Fig. 4 is a diagram of the SVDD classification result of the embodiment of the present invention.

Fig. 5 is a diagram of human fall detection results according to an embodiment of the present invention.

Detailed ways

Embodiments of the present invention are described in detail below, examples of which are shown in the drawings, wherein the same or similar reference numerals designate the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by referring to the figures are exemplary only for explaining the present invention and should not be construed as limiting the present invention.

As shown in Figure 1, the present invention proposes a real-time fall detection algorithm based on deep learning, and the specific steps are as follows:

Step 1, using a camera and other equipment to collect human body images;

Step 2, carry out key point recognition of human skeleton;

Step 2.1, use neural network convolution to extract features, predict the heat map of each key point, and through multi-stage learning, each stage optimizes the results of the previous stage;

As shown in Figure 2, this method uses the VGG-19 network for feature extraction. The feature map fear-map is initialized and fine-tuned by the first 10 layers of VGG-19 to generate a set of feature maps F that are input to the first stage of each branch. In the first stage, the network generates a set of confidence map predictions S ¹ = ρ ¹ (F) and a set of partial affinity fields where ρ ¹ and /> is the CNN used for stage 1 inference. In subsequent stages, the predictions from the two branches of the previous stage together with the original image F are used for the precise prediction. Features F are concatenated and used to produce accurate predictions

In the formula, ρ ^t and is the forecast at stage t.

There may be confusion between body parts in the early stage of the network, but through later network iterations, the prediction of human body key points becomes more and more accurate. The two branches of the network have two loss functions, one for each branch, using L2 loss between the maps and fields of the prediction and groudtruth to prevent overfitting, and weighting the calculation of the loss function.

and /> They are the L2 loss function from 1 to the different stage t of C, and f is the overall loss function. Personal confidence maps are generated for each person, and the network predicts a set of confidence levels.

In the process of network training, for a certain type of given key point j, for a certain person k, his confidence mark map has only one peak, and the confidence mark is the Gaussian distance from each point to the real point of the data. For multiple persons, non-maximum suppression (NMS) is used to cull low-scoring data from the network.

Step 2.2, add vector encodings for the predicted keypoints, and connect different parts of the same person; given a set of detected body parts, perform a confidence measure for each pair of body part detection associations, one way to detect associations is to detect additional midpoints between each pair of limbs on the limbs, and check the incidence between candidate part detections, these midpoints may provide false associations when the human body is superimposed. A 2D vector field for each limb, for each pixel in the region belonging to a particular limb, the 2D vector encodes the direction from one part of the limb to another. Each type of limb has a corresponding region of PAFs that connect two related body parts. Non-maximum suppression (NMS) is performed on the detection confidence map to obtain a discrete set of part candidate locations. For each part, there may be multiple candidate positions, which define a large set of possible limbs due to multiple people in the image or false positives. Each candidate limb is scored using a line integral computation over the PAF, and the problem of finding the best resolution corresponds to a K-dimensional matching problem known as NP-Hard, using the Hungarian algorithm to obtain the best match.

In step 2.3, in order to make the model more suitable for the application scenarios of this method, 1000 RGB pictures of various human actions were collected, including daily human behavior and fall images, and the collected pictures were trained using the tensorflow framework. shown.

Step 3, use the SSD-Mobilenet target detection model to perform target detection on the detected human body key point area, and remove the non-human body part.

Step 3.1 First, use the Mobilenet network to pre-train the COCO data set. This data set includes 80 types of targets, including 12 categories such as human body, animals, vehicles, and household appliances, to obtain pre-trained parameters.

Step 3.2 Use the self-built human body dataset in step 2.3, and use the LabelImg tool to label human samples. After the samples are labeled, generate a one-to-one corresponding xml file, and then call xml_to_csv.py to generate a csv table, and then use the generate_tfrecord.py file to convert the csv file into a tfrecord format that the tensorflow framework can recognize.

Step 3.3 Introduce the fusion features of the data generated in step 3.2 and the pre-trained model generated in step 3.1 to the SSD network, set the number of iterations to 10,000, stop training when the loss reaches 1.8, the total loss is 1.8, and the average accuracy is 78.4%. After this step is implemented, the redundant key points can be removed, and only the correct human skeleton key points can be kept.

Step 4, use the SVDD classification algorithm to classify the collected key points of human bones.

Although the trajectories of joint points are obtained, there is still the problem of data imbalance. In daily life, due to the diversity of fall types, obtaining enough actual sample falls not only requires a lot of manpower and resources, but also is difficult to obtain, which leads to sample imbalance, so how to use imbalanced data to detect falls has become a fundamental problem. This method adopts the SVDD anomaly detection method, and optimizes the parameters by grid search and other methods. Form the normal domain of positive sample training, and detect the fall behavior through the normal domain. SVDD was originally mainly used for anomaly detection and image classification. Unlike the classifier Support Vector Machine (SVM), SVDD is a one-class SVM algorithm. The main idea is to construct a normal domain hypersphere by training positive samples, while samples outside the hypersphere are abnormal samples. For a given positive sample set X, which contains N positive samples (Xi=0, 1, 2, ..., N), in order to reduce the impact of abnormal data points being included in the normal domain when constructing a hypersphere, the penalty factor C and the relaxation variable ξ _i , when the hypersphere center of the hemisphere R, let the positive samples be completely surrounded by a sphere, the corresponding optimization equation is:

For typical quadratic programming problems, by introducing Lagrangian multipliers and solving the corresponding functions. Typically, after removing outlier data points, the data will not be distributed in a spherical fashion. Therefore, the kernel function K is introduced to transform the nonlinear problem in the low-dimensional space into a high-dimensional linear problem. Figure 4 shows the two-dimensional SVDD anomaly detection map, the red dots are normal data, the blue intersections are anomalous data, and the decision boundary is surrounded by red dots. The boundary points in the decision are support vectors. Support vectors determine the shape and size of the decision boundary, which is the accuracy of the model.

The effect after this step is implemented is shown in Figure 5. The experiment proves the accuracy and reliability of this method.

Claims (1)

1. A real-time fall detection method based on deep learning, characterized in that, the concrete steps are as follows: Step 1, collecting human body images; Step 2, carry out key point recognition of human skeleton; Step 2.1, use neural network convolution to extract features, predict the heat map of each key point, and through multi-stage learning, each stage optimizes the results of the previous stage; Use the VGG-19 network for feature extraction to obtain the feature map feature-map, which is initialized and fine-tuned by the first 10 layers of VGG-19 to generate a set of feature maps F that are input to the first stage of each branch. In the first stage, the network generates a set of confidence map predictions S ¹ = ρ ¹ (F) and a set of partial affinity fields where ρ ¹ and /> is the CNN used for stage 1 inference; in subsequent stages, the predictions from the two branches of the previous stage along with the original image F are used for the precise prediction; the features F are concatenated and used to produce the precise prediction In the formula, ρ ^t and is the forecast at stage t; The two branches of the network have two loss functions, one for each branch, and weighted calculations for the loss function; and /> They are the L2 loss function of different stages t, and f is the overall loss function; a personal confidence map is generated for each person, and the network predicts the set of each confidence degree; In the process of network training, for a certain type of given key point j, for a certain person k, his confidence mark map has only one peak, and the confidence label is the Gaussian distance from each point to the real point of the data; for multiple people, use non-maximum value suppression to eliminate low-scoring data in the network; Step 2.2, add vector encoding to the predicted key points, and connect different parts of the same person; given a set of detected body parts, perform a confidence measure for each pair of body part detection associations, using the PAFs feature representation. Each type of limb has a corresponding region of PAFs used to connect two related body parts; non-maximum suppression is performed on the detection confidence map to obtain a discrete set of candidate positions of the part; for each part, there will be multiple candidate positions, and the candidate positions define a large set of candidate limbs; each candidate limb is scored using line integral calculations on PAFs, and the problem of finding the best resolution corresponds to the K-dimensional matching problem known as NP-Hard, using the Hungarian algorithm to obtain the best match; In step 2.3, the RGB pictures of various human actions are collected, including the daily behavior of the human body and images of falls, and the tensorflow framework is used to train the collected pictures. As the number of iterations increases, the value of the loss function does not decrease. Set the number of iterations to more than 5000 times, and use the model to test the test set; Step 3, use the SSD-Mobilenet target detection model to perform target detection on the detected human body key point area, and remove the non-human body part; Step 3.1 First, use the Mobilenet network to pre-train the COCO dataset, which includes 80 types of targets, and obtain the pre-trained parameters; Step 3.2 Use the human body data set in step 2.3, and use the LabelImg tool to label human samples. After the samples are marked, generate a one-to-one corresponding xml file, then call xml_to_csv.py to generate a csv table, and then use the generate_tfrecord.py file to convert the csv file into a tfrecord format that can be recognized by the tensorflow framework; step 3.3 transfer the data generated in step 3.2 and the fusion features of the pre-trained model generated in step 3.1 to the SSD network, and set iteration 1 More than 0000 times, stop training when the loss is less than 1.8; Step 4, using the SVDD classification algorithm to classify the collected key points of the human skeleton; Using the SVDD anomaly detection method, the parameters are optimized by grid search; the normal domain of positive sample training is formed, and the fall behavior is detected through the normal domain; For a given sample set X, which contains N samples, using the penalty factor C and the slack variable ξ _i , when the hypersphere center of the hemisphere R is used, the positive samples are completely surrounded by the sphere, and the corresponding optimization equation is: