CN112347967B - A Pedestrian Detection Method Fused with Motion Information in Complex Scenes - Google Patents

Abstract

The invention discloses a pedestrian detection method under a complex scene fusing motion information, which consists of two branches of a moving target identification network and a pedestrian detection network, wherein a video is input into the network to respectively obtain a moving target frame and a pedestrian detection proposal candidate frame, the two candidate frames are fused, then are divided into two groups according to the area of the frames, and are sent into two sub-networks to be respectively classified and regressed, and finally output results are combined; compared with other algorithms, the method can achieve higher detection rate for videos with lower resolution and smaller pedestrian size in the picture; the pedestrian detection network provided by the invention utilizes the motion information to reduce missing detection, and has good detection effect on dynamic and static pedestrian targets.

Description

A Pedestrian Detection Method Fused with Motion Information in Complex Scenes

technical field

The invention belongs to the technical field of target detection, and in particular relates to a pedestrian detection method for fusing motion information in complex scenes.

Background technique

Pedestrians are an important target in video surveillance tasks. Pedestrian detection is also one of the basic tasks and key technologies of computer vision research. This technology is used to judge whether there are pedestrians in images or video sequences and give precise positioning. Computer fields such as video surveillance and robot development are widely used.

Affected by many factors such as storage resources and shooting distance, in general practice, surveillance videos often have problems such as poor video quality, small proportion of pedestrian objects in the screen, and occlusion of pedestrian objects, which lead to pedestrian detection in complex scenes. There are still serious problems such as missed detection and false detection in the actual application of the technology. With the development of machine learning technology and computer vision, object detection technology based on deep learning has been widely applied to pedestrian detection tasks, and has achieved extraordinary results, which provides ideas for pedestrian detection algorithms in complex scenes.

In 2012, Lijun Guo et al. proposed a pedestrian detection method combining motion information and apparent features in the article "Pedestrian detection Method of Integrated Motion Information and APPearance Features", which is used for pedestrian detection tasks in complex scenes and integrates motion information into In the object segmentation algorithm based on image sequences, the detection accuracy of candidate detection windows is improved by obtaining more accurate segmentation results. However, the accuracy of this algorithm is much lower than that of the R-CNN series of pedestrian detection networks, and the amount of calculation is large. , the effect is not ideal in complex scenes.

In 2013, Zhang Zhiying designed a pedestrian detection classifier that combines target motion information in "Research and Implementation of Pedestrian Detection Based on Target Motion Information and HOG Features", forming a combination of HOG and SVM classifiers for pedestrian detection However, the accuracy of the detection module of this algorithm is not as good as the pedestrian detection network of the R-CNN series, and the operation speed is relatively slow. There is still room for improvement in the inter-frame difference method for motion information extraction.

In 2016, in the article "Scale-aware Fast R-CNN for Pedestrian Detection", Jianan Li et al. proposed a network that integrates large-scale subnetworks and small-scale subnetworks into the same framework for small-scale pedestrian targets in surveillance videos. , but the network is a picture-based pedestrian detection network, and the accuracy of the Fast R-CNN network used is low, and it cannot achieve better detection results for low-resolution pictures.

In 2016, Liliang Zhang et al. published an improved network that makes the general target detection Faster R-CNN network more suitable for pedestrian detection tasks in the article "Is Faster R-CNN Doing Well for Pedestrian Detection". The target is more discriminative, but the network is still an image-based detection network, which is not ideal when the image quality is low.

In 2018, Aixin Guo proposed a multi-scale pedestrian detection scheme based on deep convolutional feature fusion in "Multi-scale Pedestrain Detection based on Deep Convolutional Feature Fusion". For the lack of medium and small-scale pedestrian features, the underlying features and advanced semantics Features are combined, and the focus loss function is introduced to mine difficult samples to improve the accuracy of the algorithm. Although the detection rate of this scheme has been improved in the public data set, it is only applicable to the situation where the image quality is slightly better.

In 2019, Xia Jinming and others introduced a difficult sample mining strategy in "A Pedestrian Detection Algorithm Based on Faster R-CNN", which picks out samples in complex environments and adjusts the weights to make training more focused , to improve the generalization performance of the model, and this is also a picture-based pedestrian detection scheme. The recall rate of this scheme is slightly improved, but it is still only suitable for multi-scale pedestrian detection tasks with clear pictures.

In 2019, in the article "Video Pedestrian Detection Method Based on YOLO and GMM", Li Junyi and others proposed a scheme to fuse motion information in pedestrian detection tasks for pedestrian detection tasks under complex lighting conditions, but the YOLO algorithm used in this scheme It is not sensitive to small-sized targets, and adopts the method of lowering the pedestrian detection threshold and using motion information to remove false alarms, so this method is only applicable to the situation where all pedestrian targets are dynamic, and the performance in scenes with a large number of static pedestrian targets poor.

In 2019, Wang Lei disclosed a detection framework that uses motion information to assist border screening in "Research on High-precision Pedestrian Detection Algorithm and System Design for Railway Traffic Safety", using structural similarity to approximate the motion information of pedestrians between video frames , and combine the motion information with the confidence score of the network detection results to re-evaluate the frame. The premise of this method is that most pedestrians have motion displacement between adjacent frames, so there are disadvantages for the situation of stationary pedestrians.

Most of the current pedestrian detection algorithms in complex scenes are improved for a certain problem in complex scenes, such as low resolution, small-sized objects, etc. In actual scenes, the above problems often exist at the same time, and the detection network for a certain problem It cannot achieve good results in practical applications.

The existing pedestrian detection methods based on neural networks in complex scenes have the following disadvantages: (1) the general method is to detect pedestrians based on pictures, which requires a higher resolution of the test pictures, and the proportion of pedestrian targets in the picture is relatively large; 2) Although the pedestrian detection network designed by some methods is improved for small-sized objects, it still uses image detection, and the accuracy of the original object detection network used is low, and the detection effect is poor in low-resolution scenes; (3) 2019 Although the method of Li Junyi et al. is based on video design and incorporates motion information during pedestrian detection, the method of lowering the pedestrian detection threshold and using motion information to remove false alarms makes this method only applicable to all pedestrian targets. In dynamic situations, the performance is poor in scenes with a large number of static pedestrian targets; (4) The above methods are all aimed at the single problem of small pedestrian target size in complex scenes or low resolution of video scenes. However, the surveillance video actually obtained is limited by various factors and usually has all the above-mentioned problems. Therefore, there is currently no pedestrian detection scheme with better performance in complex scenes that fuses motion information.

Contents of the invention

In view of this, the purpose of the present invention is to provide a pedestrian detection method that integrates motion information in complex scenes, which can solve the problem of pedestrian detection in low-resolution videos in complex scenes, and is sensitive to small targets.

A pedestrian detection method for fusing motion information in complex scenes, comprising the following steps:

Step 1, obtain the original video, and process to obtain the image sequence;

Step 2, passing the picture sequence through the RPN network to obtain a target detection proposal candidate frame;

Step 3, passing the original video through a moving object recognition algorithm to obtain a moving object frame;

Step 4. Fusion the moving target frame obtained in step 3 with the target detection proposal candidate frame in step 2 to obtain all proposed candidate frames;

Step 5. Divide all proposed candidate frames obtained in step 4 into two groups according to size, and input them to two neural networks for classification and regression respectively;

Step 6. Output the pedestrian detection results output by the two neural networks in step 5 together to obtain a video with a pedestrian target frame.

Preferably, in the step 1, the pictures in the picture sequence are first scaled and then input into the convolutional network to obtain the feature map of each picture, and then the target detection proposal candidate frame is obtained, specifically:

a. Use anchor boxes of nine different sizes to classify each pixel in the picture to determine whether it is an object or a background;

b. Regression on the anchors to obtain the precise parameters of the classification;

c. Sort according to the softmax score of the anchor, and find the best 2000 classifications;

d. Map the anchor back to the original image;

e. Use the NMS algorithm to sort the anchors and output the first 256 proposed candidate boxes.

Preferably, the steps 2 and 3 are executed synchronously.

Preferably, in the step 3, the GMM algorithm is used to identify the moving object on the original video sequence to obtain the moving object frame.

Preferably, in the step 4, the non-maximum suppression value algorithm is used to perform the fusion.

Preferably, in step 5, sort all the proposed candidate frames obtained in step 4 according to the size of the area, divide the pictures with the top 50% of the area into the first group, and rank the pictures with the bottom 50% of the area into the second group.

Preferably, in the step 5, in each neural network, first adopt the ROI pooling layer of the Faster-RCNN network to normalize the feature map of the proposed candidate frame part; input this part of the feature map into Faster-RCNN The fully connected layer and the softmax layer of the RCNN network calculate what category each proposed target belongs to, and use the data set that only contains person labels for pre-training; at the same time, the target box regression is used again to obtain the position offset of each proposed target.

Preferably, in the step 5, the structures of the two neural networks are the same.

The present invention has following beneficial effect:

The purpose of the present invention is to overcome the shortcomings of existing pedestrian detection methods in complex scenes, and propose a pedestrian detection method in complex scenes that integrates motion information. The method consists of two branches, the moving object recognition network and the pedestrian detection network. The video is input into the network to obtain the moving object frame and the pedestrian detection proposal candidate frame respectively. Fusion, and then sorted according to the size of the box, the larger box is input into the large-size sub-network, and the smaller-sized box is input into the small-size sub-network, classification and regression are performed respectively, and finally the output results are combined. The method provides a pedestrian detection framework suitable for low-resolution video and sensitive to small targets, which solves the problem of pedestrian detection in complex scenes; the present invention is aimed at videos with lower resolution and smaller pedestrians in the picture. Compared with other algorithms, it can achieve a higher detection rate; the pedestrian detection network of the present invention uses motion information to reduce missed detection, and has a good detection effect on dynamic and static pedestrian targets; the existing general method is to reduce the threshold of pedestrian detection algorithms , using motion information to remove false alarms, the detection effect on static pedestrian targets is poor; compared with the existing general methods, the pedestrian detection network of the present invention is implemented in a general target detection framework with higher precision and more sensitivity to small targets It is improved, and the sub-network is distinguished according to the size of the pedestrian target, and a higher recognition rate can be achieved when the distance between the pedestrian target and the monitoring device is far away.

Description of drawings

Fig. 1 is a flow chart of the method of the present invention.

Figure 2 is a schematic representation of the recall rate and precision rate of the method of the present invention.

Detailed ways

The present invention will be described in detail below with reference to the accompanying drawings and examples.

The present invention proposes a pedestrian detection method based on the Faster-RCNN network and fusion of motion information, which is suitable for the detection of dynamic and static pedestrian targets. The target motion information is fused with the output of the RPN network, and input into different classification networks according to the size of the candidate frame, and finally the pedestrian target position is obtained, which is suitable for pedestrian detection tasks in complex scenes (low video resolution, small pedestrian pixel height, etc.).

As shown in Figure 1, the pedestrian detection method of fusion motion information under a kind of complex scene of the present invention, comprises the following steps:

Step 1. Get the original video.

The original video refers to the video taken by the surveillance camera in a natural scene and contains several pedestrian targets. Due to the limitation of storage resources, shooting distance and other factors, the resolution of the video is low, the pixel height of the pedestrian target is small, and the pedestrian target may be blocked. Subsequently, the video is input into two parallel branches to obtain the pedestrian target frame.

Step 2. Obtain the target detection proposal candidate frame through the RPN network.

Input the original video into the first branch, process it into a picture sequence, and input the feature map of each picture into the convolutional network after M×N scaling, and input these feature maps into the RPN network to obtain the detection target proposal candidate frame and obtain the candidate The coordinates (x*, y*, w*, h*) of the frame in the feature map, where (x*, y*) is the coordinate of the upper left corner of the candidate frame, and w*, h* are the width and height of the candidate frame, respectively.

Step 3. Obtain the moving target frame through the moving target recognition module.

Input the original video into the second branch, and use the GMM motion detection algorithm to obtain the coordinates (x, y, w, h) of the moving target in the current frame, where (x, y) are the coordinates of the upper left corner of the moving target frame, w, h are the width and height of the motion target box, respectively. After the coordinates are M×N scaled, they are projected to the feature layer to obtain the transformed coordinates (x*, y*, w*, h*).

Step 4. The target frame obtained in step (3) is fused with the candidate frame in (2) using the non-maximum suppression value algorithm to obtain all proposed candidate frames.

Step 5. Divide the proposed candidate boxes according to size, and input them into different neural networks for classification and regression.

All the proposed candidate frames obtained are sorted according to the size of the area, and a part with a large area is input into the large-size sub-network, and a part with a small area is input into the small-size sub-network. The structure of the two sub-networks is the same, but the size of the candidate frame is different. The candidate frame and the feature map are first pooled in the ROI layer in the sub-network, and then input into the subsequent network to classify and select the pedestrian target, and perform regression to correct the position of the candidate frame. , to get the final target box.

Step 6. Output the pedestrian detection results of the two sub-networks together to obtain a video with a pedestrian target frame.

Example:

(1) Get the original video.

The original video refers to the video taken by the surveillance camera in a natural scene and contains several pedestrian targets. Due to the limitation of storage resources, shooting distance and other factors, the resolution of the video is low, and the pixel height of the pedestrian target is small. Considering a dataset that meets the above requirements, choose the CUHK Square dataset from the Chinese University of Hong Kong.

(2) Through the RPN network, the target detection proposal candidate box is obtained.

This phase includes the following steps:

1) Process the video into a picture sequence;

2) Scale the image sequence and then input it into the convolutional network to obtain the feature map;

3) The feature map is input into the RPN network;

a. Use anchor boxes of nine different sizes to classify each pixel in the picture to determine whether it is an object or a background;

b. Regression on anchors to obtain precise parameters;

c. Sort according to the softmax score of the anchor, and find the best 2000;

d. Map the anchor back to the original image;

e. Use the NMS algorithm to sort the anchors and output the first 256.

Accordingly, we get the target proposal candidate box of the first branch.

(3) Obtain the moving target frame through the moving target recognition module.

This phase includes the following steps:

1) Use the GMM algorithm to identify moving objects in the video;

a. Initialize the background model, mean, standard deviation and difference threshold;

b. Set the threshold parameter, and judge whether the current pixel belongs to the foreground or the background according to whether the mean value is within the threshold range;

c. Update parameters to learn and update the background;

d. Repeat steps b.c. until the algorithm stops, and obtain the position information of the moving object in each frame.

2) Scale and project the coordinates of the moving target to the feature layer;

(4) The target frame obtained in step (3) is fused with the candidate frame in (2) using the non-maximum suppression value algorithm to obtain all proposed candidate frames.

(5) Divide the proposed candidate frame according to size, and input it into different neural networks for classification and regression.

Sort the proposed candidate boxes obtained in step (4) according to the size of the area, input the top 50% of the larger area into the large-size sub-network, and input the last 50% of the smaller-area sub-network into the small-size sub-network. The input of each sub-network is the original feature map and the proposed candidate boxes of different sizes output in the previous step, which are sent to the ROI pooling layer for size normalization, and the feature map of the proposed candidate box is calculated. Input this part of the feature map into the fully connected layer and the softmax layer to calculate what category each proposed target belongs to. The sub-network uses the data set that only contains the person label for pre-training, so only the person class and its probability vector are output; at the same time, the target box is used again Regression obtains the position offset of each proposed target, which is used to regress a more accurate pedestrian detection frame.

The present invention also carries out simulation verification on the effect of the method.

The simulation is carried out in the environment of ubuntu18.04, CUDA10.0, cuDNN7.6, OpenCV3.4, and PyTorch. The Faster R-CNN model based on the PyTorch framework is used for improvement and implemented based on Python.

Using the 80 types of weight parameters based on COCO training given by the author of Faster R-CNN to test multiple groups of videos, the AP (Average Precision) obtained is 72.61%, while the AP of the optimized Faster R-CNN network of the present invention reaches 77.63 %, the average accuracy is increased by 5.02%. Compared with the current advanced algorithm, that is, the video pedestrian detection method based on YOLO and GMM proposed by Li Junyi in 2019, the AP obtained by this algorithm is 74.82%, and the accuracy of this algorithm is improved by 2.81%.

Two indicators are used to evaluate the experimental results, namely Recall (recall rate) and Precision (accuracy rate). As shown in Figure 2, Recall is the ratio of the number of successfully detected pedestrians P _d,t to the total number of pedestrians P _d,t +P _n,t . Precision is the ratio of the number of successfully detected pedestrians P _d,t to all detected numbers P _d,t +P _d,f .

After multiple sets of complex scene video tests, the recall rate and precision rate of the current advanced algorithm are 0.77 and 0.51, respectively, and the recall rate and precision rate of the algorithm in this paper are 0.91 and 0.85, respectively.

To sum up, the above are only preferred embodiments of the present invention, and are not intended to limit the protection scope of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention shall be included within the protection scope of the present invention.

Claims (7)

1. A pedestrian detection method fusing motion information in a complex scene is characterized by comprising the following steps:

step 1, acquiring an original video, and processing to obtain a picture sequence;

step 2, the picture sequence is passed through an RPN network to obtain a target detection proposal candidate frame;

step 3, obtaining a moving target frame by the original video through a moving target recognition algorithm;

step 4, fusing the moving target frame obtained in the step 3 with the target detection proposal candidate frame in the step 2 to obtain all proposal candidate frames; the fusion is carried out by using a non-maximum suppression value algorithm;

step 5, dividing all the proposed candidate frames obtained in the step 4 into two groups according to the size, and respectively inputting the groups into two neural networks for classification and regression processing;

and 6, outputting the pedestrian detection results output by the two neural networks in the step 5 together to obtain a video with a pedestrian target frame.

2. The pedestrian detection method fusing motion information under complex scenes as claimed in claim 1, wherein in said step 1, the pictures in the picture sequence are scaled first and then input into a convolutional network to obtain a feature map of each picture, and then a target detection proposal candidate frame is obtained, specifically:

a. classifying each pixel point in the picture by utilizing nine anchor boxes with different sizes, and judging whether the pixel point is an object or a background;

b. regressing the anchors to obtain classified accurate parameters;

c. sorting according to the softmax score of the anchor, and finding 2000 optimal classifications;

d. mapping the anchor back to the original image;

e. the anchors are ranked using the NMS algorithm and the top 256 proposed candidate boxes are output.

3. The pedestrian detection method with fusion of motion information under complex scene as claimed in claim 1, wherein said steps 2 and 3 are executed synchronously.

4. The pedestrian detection method fusing motion information under complex scenes as claimed in claim 1, wherein in said step 3, a moving object frame is obtained by performing moving object recognition on an original video sequence using a GMM algorithm.

5. The method for detecting pedestrians fusing motion information under complex scenes as claimed in claim 1, wherein in step 5, all the proposed candidate boxes obtained in step 4 are sorted according to area size, the pictures with the area in the first 50% are divided into a first group, and the pictures with the area in the second 50% are divided into a second group.

6. The method according to claim 5, wherein in step 5, in each neural network, the characteristic map of the proposed candidate frame portion is normalized by using the ROI pooling layer of the Faster-RCNN network; inputting the partial feature map into a full connection layer and a softmax layer of a Faster-RCNN network to calculate the category of each proposed target, and pre-training by using a data set only containing character tags; while again using target box regression to obtain the position offset for each proposed target.

7. The pedestrian detection method based on fusion of motion information in complex scene as claimed in claim 5, wherein in the step 5, the two neural networks have the same structure.

Images