CN112257527B - Mobile phone detection method based on multi-target fusion and space-time video sequence - Google Patents

Abstract

The invention relates to a mobile phone detection method based on multiple target fusion and spatio-temporal video sequences, including training an improved yolo model to obtain a detection model, and inputting video image frames to run the detection model to obtain a first frame predicted value; Perform decoding, remove the frame whose score value is lower than the preset value and implement NMS with the Diou threshold, and suppress the mobile phone frame when only the mobile phone frame appears according to the decoding result of a certain frame of image; use the suppressed result as the target template, Input the video image frame as the candidate frame search area, input it into the fully connected twin network, and select the result with the highest score map similarity to frame the mobile phone in the video image frame; if the set number of frames has been tracked, repeat the above steps until the video image input ends. Based on the lightweight detection network in the One-stage detection algorithm, the invention refines the network structure, training and detection methods, and obtains higher detection accuracy without reducing the detection speed.

Description

Mobile phone detection method based on multi-object fusion and spatiotemporal video sequence

technical field

The invention relates to the technical field of image processing, in particular to a mobile phone detection method of multi-target fusion and spatiotemporal video sequences.

Background technique

Detection accuracy and speed have always been the core issues of target detection. In the process of target detection, in order to obtain a more accurate detection effect, a heavyweight detection algorithm that can obtain high precision is usually selected, which makes the system in the mobile terminal device. Inference speed is greatly limited.

The Chinese invention patent application with the application number of 202010048048.5 discloses an intelligent monitoring method, device and readable medium for identifying mobile phone anti-photography, which performs machine learning on the appearance of a large number of mobile phones through an intelligent monitoring system; camera probe, the camera probe communicates with the intelligent monitoring system in real time; the camera transmits the captured image to the intelligent monitoring system in real time; the intelligent monitoring system identifies whether there is a mobile phone; if there is a mobile phone, the intelligent monitoring system according to the shooting image, to determine whether there is an act of taking pictures with a mobile phone; the intelligent monitoring system determines that there is an act of taking pictures with a mobile phone, and outputs alarm information in real time to remind the staff to give timely reminders. Use the detection algorithm with Darknet53 as the Backbone for preliminary detection, and then cooperate with bone generation, motion recognition and other methods for monitoring; in addition, there are some methods that use similar algorithms for preliminary positioning, and then perform overall to local search methods for detection. However, such methods make the inference speed of the detection system on the mobile terminal basically not real-time.

SUMMARY OF THE INVENTION

The purpose of the present invention is to overcome the shortcomings of the prior art, to provide a mobile phone detection method for fusion of multiple targets and spatiotemporal video sequences, and to solve the shortcomings of the existing detection methods.

The object of the present invention is achieved through the following technical solutions: a mobile phone detection method based on multiple target fusion and spatiotemporal video sequences, the mobile phone detection method includes:

Train the improved yolo model to obtain the detection model, and input the video image frame to run the detection model to obtain the first frame prediction value;

The predicted value of the first frame is decoded, the frame whose score value is lower than the preset value is removed, and NMS (non-maximum suppression) is realized with the Diou threshold, and when only the mobile phone frame appears according to the decoding result of a certain frame of image, The mobile phone frame is suppressed;

Take the suppressed result as the target template, input the video image frame as the candidate frame search area, input the two into the fully connected twin network at the same time, and select the result with the largest score map similarity to frame the mobile phone in the video image frame. ;

If the set number of frames has been tracked, repeat the above steps until the video image input ends.

Further, the mobile phone detection method also includes if the number of frames is not set, then repeatedly using the suppressed result as the target template, inputting the video image frame as the candidate frame search area, inputting the two into the fully connected twin network at the same time, and The step of selecting the result with the highest score map similarity to frame the mobile phone in the video image frame.

Further, the mobile phone detection method further includes the steps of obtaining a training set and a test set before training the improved yolo model to obtain a detection model, and inputting video image frames to run the detection model to obtain the first frame predicted value.

Further, the step of obtaining the training set and the test set includes: performing frame processing on the recorded video and labeling the processed video pictures, extracting part of the pictures every frame to construct a data set, and dividing the data set into training sets according to a certain proportion. set and test set.

Further, the first frame predicted value is decoded, the frame whose score value is lower than the preset value is removed, and the Diou threshold is used to realize NMS, and according to the decoding result of a certain frame of image, when only the frame of the mobile phone appears, the frame of the mobile phone is Boxes for suppression include:

According to the decoding formulas bx=sigmoid(t _x )+cx, by=sigmoid(t _y )+cy, bw= _{p w} e ^tw , bh=ph e ^th , conf=sigmoid(raw_conf) and prob=sigmoid(raw_prob) decoding the predicted value of the first frame;

Remove the boxes whose confidence or class probability does not meet the requirements with a score threshold of 0.4 and implement NMS with a Diou threshold of 0.1;

For the decoding result of a certain frame of image, if there is a mobile phone frame but no human frame, hand frame or camera frame, the prediction frame about the mobile phone in the corresponding image is removed, so as to suppress the mobile phone frame.

Further, improvements to the yolo model include the following:

Add the s branch for detecting small objects to yolov3-tiny to improve the detection effect of small objects such as cameras;

On the basis of the model structure of the previous step, SPP (Spatial Pyramid Pooling), SAM (Spatial Attention Module), and CAM (Channel Attention Module) modules are added to connect with residuals to improve feature extraction capabilities.

The invention has the following advantages: a mobile phone detection method based on multi-target fusion and spatiotemporal video sequences, and a lightweight detection network based on the One-stage detection algorithm, refined modifications are made to the network structure, training and detection methods, and the detection speed is not reduced. In the case of high detection accuracy, at the same time, the detected target is tracked by using the tracking algorithm, which solves some difficult sample detection with large occlusion and angle inclination, and reduces the consumption of resources by the system, so that the overall On the mobile terminal, the reasoning speed of the system is greatly improved.

Description of drawings

FIG. 1 is a schematic flow chart of the method of the present invention.

Detailed ways

In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only It is a part of the embodiments of the present application, but not all of the embodiments. The components of the embodiments of the present application generally described and illustrated in the drawings herein may be arranged and designed in a variety of different configurations. Accordingly, the following detailed description of the embodiments of the application provided in the accompanying drawings is not intended to limit the scope of the application as claimed, but is merely representative of selected embodiments of the application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative work fall within the protection scope of the present application. The present invention will be further described below with reference to the accompanying drawings.

As shown in Figure 1, the present invention relates to a mobile phone detection method based on multi-target fusion and spatiotemporal video sequences, which specifically includes the following contents:

S1. Perform frame-by-frame processing on the video recorded by the camera in the actual application scene, and randomly extract some pictures every frame to construct a data set. Use LabelImg labeling software to label the mobile phone, human body, hand, and camera in each image, and divide the data set into training set and test set according to a certain proportion.

S2. Use the improved yolov3 network to train the detection model. The network training input is the training set image and the corresponding label, and the network output is the predicted t _x , t _y , t _w , _th offset value, original confidence and original class probability.

Further, in the training process, for the focalloss of the confidence loss, considering that the positive and negative sample imbalance of the yolov3 network model is much lower than that of ReTinanet, the value of α is selected as 0.4, and the calculation formula of the confidence loss is as follows:

L _focalloss = -α _t (1-p _t ) ^μ * ^γ log(p _t )

S3. Run the detection model to obtain the predicted value of the first frame.

S4. Decode the predicted value according to the following decoding formula, remove frames with lower confidence or class probability with a score threshold of 0.40, and implement NMS with a Diou threshold of 0.1;

bx=sigmoid(t _x )+cx

by=sigmoid(t _y )+cy

bw=p _w e ^tw

bh= ^ph e _th

conf=sigmoid(raw_conf)

prob=sigmoid(raw_prob)

Among them: bx, by, bh, bw represent the center horizontal and vertical coordinates and height and width of the prediction frame, respectively, and p _h and p _w represent the height and width of the prior frame, respectively. t _x and _ty represent the predicted offset of the object center from the upper left corner of the grid, tw and th represent the predicted offset of the object relative to the prior frame, and cx and cy represent the upper left corner of the grid. Coordinate, score=conf (confidence)×prob (class probability).

S5. For the decoding result of a certain frame of picture, if the mobile phone frame appears but the human body frame, the hand frame or the camera frame does not appear, the mobile phone frame will be suppressed.

S6. The suppressed result is used as the target template, and the video image frame is used as the candidate frame search area, and the two are simultaneously sent to the fully connected twin network to obtain template matching to obtain the similarity measurement result score map.

S7. Select the result with the greatest similarity to frame the mobile phone in the video image frame.

S8, determine whether the set number of frames has been tracked, if not, repeat steps S6-S8, if so, execute step S9;

S9, repeat steps S3-S9 until the video image input ends.

In terms of multi-target association, the contribution points of the present invention are as follows:

It is found that the position loss based on GIoU (Generalized Intersection over Union) (the position loss used in the present invention) will have an unbalanced situation opposite to the position loss based on the difference square. The average label box size and average position loss, combined with the ratio of the number of each branch box, use the negative exponential function (a·e ^-b/x ) as the basic function to perform unbalanced fitting correction, which solves the problem of GIoU-based size boxes. Balanced position loss problem.

Following the premise that when the amount of data is large enough, the average position loss of each branch box should be almost equal, and the first warm-up epoch (warm-up period, that is, the learning rate at the beginning of training) is counted during the training process. The average label box size and average position loss of the s, m, and l branches, combined with the number ratio of each branch box, use the negative exponential function (a·e ^-b/x ) as the basic function to unbalance The fitting correction of , adjusts the position loss weight of each branch in the subsequent iteration process, and solves the position loss problem of unbalanced size boxes based on GIoU.

The problem of rewriting labels in yolo is found, that is, the displacement anchor box assigned to an object has a chance to be covered by the following object, which makes it impossible to train the covered ones. The specific improvement steps are as follows:

If an anchor frame has been labelled by an original object, it is determined whether the original object has a unique frame;

If the original object has a unique frame, it is judged whether the existing object can be assigned an anchor. If it exists, the existing object cancels the assignment to an anchor box, otherwise the existing object goes down to find the next anchor box with the highest iou value for assignment;

If the original object does not have a unique frame, it is judged whether there is an anchor with the highest iou of the existing object and an anchor with a non-highest iou of the original object covering the original assignment; anchor; if there is, judge whether the existing object can have an assigned anchor, if so, the existing object cancels the assignment to a certain anchor box, otherwise the original assignment is overwritten; if there is an anchor of the existing object that is not the highest iou and the original object is not the highest iou anchor, the lower iou is covered.

Considering that the mobile phone and other auxiliary detection targets need to have a primary and secondary distinction, all the losses of the mobile phone category are multiplied by a priority coefficient, and this coefficient is 1.10 in the present invention.

The threshold value obtained by ATSS (Adaptive Training Sample Selection) is limited. When the threshold value is less than a certain value, it is considered that the quality of the training sample corresponding to the obtained threshold value is low. Therefore, the threshold selection method will be abandoned, and only the training sample to be selected will be selected. The one with the highest IoU. In the present invention, the certain value is taken as 0.10.

The multi-target objects are associated with basically no computing resources, and the computing resource consumption of the cognitive detection method is reduced.

In terms of spatio-temporal information fusion, the present invention utilizes the context information of two dimensions, temporal domain and spatial domain, and significantly improves the problems of occlusion and drift in the tracking process.

The foregoing are only preferred embodiments of the present invention, and it should be understood that the present invention is not limited to the forms disclosed herein, and should not be construed as an exclusion of other embodiments, but may be used in various other combinations, modifications, and environments, and Modifications can be made within the scope of the concepts described herein, from the above teachings or from skill or knowledge in the relevant field. However, modifications and changes made by those skilled in the art do not depart from the spirit and scope of the present invention, and should all fall within the protection scope of the appended claims of the present invention.

Claims (6)

1. The mobile phone detection method based on the fusion of various targets and the space-time video sequence is characterized in that: the mobile phone detection method comprises the following steps:

a1, training the improved yolo model to obtain a detection model, wherein the yolo model is specifically improved as follows:

if a certain anchor frame is endowed with a label by a certain original object, judging whether the original object has a unique frame;

if the original object has the unique frame, judging whether the current object can assign an anchor, if so, cancelling the assignment of the current object to a certain anchor frame, otherwise, searching the next anchor frame with the highest iou value for assignment;

if the original object has no unique frame, judging whether the anchor of the highest iou of the existing object and the anchor of the non-highest iou of the original object cover the original assignment; if so, judging whether an anchor of the non-highest iou of the existing object and an anchor of the highest iou of the original object exist or not; if yes, judging whether the current object can have an assignment anchor, if yes, canceling the assignment of the current object to a certain anchor frame, and if not, covering the original assignment; if the anchor of the existing object which is not the highest iou and the anchor of the original object which is not the highest iou exist, covering the person with the low iou;

a2, inputting a video image frame to operate a detection model to obtain a first frame prediction value;

a3, decoding the first frame prediction value, removing a frame with a score value lower than a preset value, realizing NMS (network management system) by using a Diou threshold value, and inhibiting a mobile phone frame according to the decoding result of a certain frame image when only the mobile phone frame appears;

a4, taking the suppressed result as a target template, inputting a video image frame as a candidate frame search area, simultaneously inputting the video image frame and the candidate frame search area to a full-connection twin network, and selecting the result with the largest score map similarity to mark the mobile phone in the video image frame;

a5, if the set frame number has been tracked, repeating steps A2-A4 until the video image input is finished.

2. The method for mobile phone detection based on multi-target fusion and spatio-temporal video sequence according to claim 1, wherein: if the frame number is not set, the steps of repeatedly taking the suppressed result as a target template, inputting a video image frame as a candidate frame search area, simultaneously inputting the video image frame and the candidate frame search area to a full-connection twin network, and selecting the result with the largest score map similarity to mark the frames of the mobile phones in the video image frame are repeated.

3. The mobile phone detection method based on multi-target fusion and spatio-temporal video sequence according to claim 1, characterized in that: the mobile phone detection method further comprises the step of acquiring a training set and a test set before the step of training the improved yolov3 model to obtain a detection model and inputting the video image frame to run the detection model to obtain a first frame predicted value.

4. The mobile phone detection method based on multi-target fusion and spatio-temporal video sequence according to claim 3, characterized in that: the step of obtaining the training set test set comprises: the method comprises the steps of performing frame division processing on a recorded video, labeling processed video pictures, extracting partial pictures at intervals of frames to construct a data set, and dividing the data set into a training set and a test set according to a certain proportion.

5. The mobile phone detection method based on multi-target fusion and spatio-temporal video sequence according to claim 1, characterized in that: the decoding the first frame prediction value, removing a frame with a score value lower than a preset value, realizing NMS (network management system) by using a Diou threshold value, and inhibiting a mobile phone frame when only the mobile phone frame appears according to a decoding result of a certain frame image comprises the following steps:

according to the decoding formula bx-sigmoid (t) _x )+cx、by＝sigmoid(t _y )+cy、bw＝p _w e ^tw 、bh＝p _h e ^th Decoding the first frame prediction value by conf ═ sigmoid (raw _ conf) and prob ═ sigmoid (raw _ prob), wherein bx, by, bh, and bw respectively represent a center horizontal and vertical coordinate and a height and width of a prediction frame, and p represents a center horizontal and vertical coordinate and a height and width of the prediction frame _h And p _w Respectively representing the height and width of the prior box, t _x And t _y The predicted offset of the center of the object from the position of the upper left corner of the grid is represented, the predicted offset of the object relative to the prior frame is represented by tw and th, the cx and cy represent coordinates of the upper left corner of the grid, conf is a confidence coefficient, and prob is a category probability;

removing the box with confidence or category probability not meeting the requirement with score threshold of 0.4 and implementing NMS with Diou threshold of 0.1;

and (3) rejecting a prediction frame related to the mobile phone in the corresponding image if the mobile phone frame does not have a human body frame, a hand frame or a camera frame according to the decoding result of the certain frame of image, so as to restrain the mobile phone frame.

6. The mobile phone detection method based on multi-target fusion and spatio-temporal video sequence according to claim 1, characterized in that: improvements to the yolo model include the following:

increasing an s branch for detecting a small object for yolov3-tiny to improve the detection effect of small objects such as a camera;

on the basis of the model structure of the previous step, the SPP module, the SAM module and the CAM module are added to be connected with the residual error, so that the feature extraction capability is improved.

Images