WO2021189507A1

WO2021189507A1 - Rotor unmanned aerial vehicle system for vehicle detection and tracking, and detection and tracking method

Info

Publication number: WO2021189507A1
Application number: PCT/CN2020/082257
Authority: WO
Inventors: 余犀; 董晓飞; 石霖; 曹峰; 孙明俊
Original assignee: 南京新一代人工智能研究院有限公司
Priority date: 2020-03-24
Filing date: 2020-03-31
Publication date: 2021-09-30
Also published as: CN111476116A

Abstract

Disclosed in the present application are a rotor unmanned aerial vehicle system for vehicle detection and tracking, and a detection and tracking method. The system comprises an unmanned aerial vehicle platform and a ground station platform, and the unmanned aerial vehicle platform is used for calculating and detecting a tracking target in real time; the ground station platform is used for carrying out tracking and video monitoring on the target and issuing a manual flight control instruction to the unmanned aerial vehicle platform. The present invention further provides a detection and tracking method based on a rotor unmanned aerial vehicle system. The present invention adopts an unmanned aerial vehicle end-side calculation scheme, so that the timeliness of the system is effectively improved; a YOLO Nano target detection algorithm is adopted, so that the operation pressure of an airborne computer is remarkably reduced; by using a Staple tracking algorithm having higher robustness and a target re-detection module, the tracking precision is improved, and the re-detection algorithm is automatically started to re-search a target when the target is shielded or lost in real-time tracking.

Description

Rotor unmanned aerial vehicle system for vehicle detection and tracking and detection and tracking method

Technical field

The invention relates to the field of rotary wing unmanned aerial vehicles, in particular to a rotary wing unmanned aerial vehicle system used for vehicle detection and tracking and a detection and tracking method.

Background technique

There are many types of drones. Among them, rotary-wing drones have many advantages such as no site restrictions, fixed-point hovering, slow flight, limited space flight, vertical take-off and landing, etc., making them suitable for aerial photography, agriculture, plant protection, micro selfies, and express transportation. , Disaster rescue, surveillance of infectious diseases, surveying and mapping, news reports, power inspections, disaster relief, film and television shooting and other fields have a wide range of applications. In recent years, artificial intelligence technology has developed rapidly, and the combination of drones and artificial intelligence technology has become a new research focus. Target detection and target tracking technology based on deep learning endows drones with "intelligence", enabling drones to have a wider information search area, further improving the ability to interpret and analyze local micro-information, and to perceive the surrounding environment. The accuracy of measurement and measurement has also been further improved. The empowerment of UAVs by artificial intelligence technology has added a pair of keen "eyes" to UAVs, enabling them to fly autonomously and perform higher-level tasks.

With the development of digital imaging technology, cameras as a sensor have been widely studied. Because people can estimate the position and distance of objects in the field of vision through their own vision, and the principle of the camera simulates human eyes, so imitating the characteristics of people, using the two-dimensional image of the camera can reverse the three-dimensional information of the object in the image. Visual perception systems have been used in UAVs for a long time. The detection and tracking of ground targets by rotary-wing UAVs is a popular scene application. Since deep learning-based target detection and target tracking algorithms both rely on powerful computing power, The end-side computing power of UAVs is limited. Therefore, most of the solutions at this stage use ground stations for calculations. UAVs are only responsible for collecting and transmitting images. For target tracking algorithms, the algorithms at this stage cannot do anything about occlusion and loss of targets. Effective processing.

At this stage, the mainstream rotary-wing drones for target detection algorithms use the YOLOv3 neural network architecture model. The ground station has strong computing power and can use large networks for object detection. The network model structure of YOLOv3 is mainly composed of 75 volume base layers. Without using a fully connected layer, the network can correspond to input images of any size. In addition, the pooling layer is not used. Instead, the stride of the volume base layer is set to 2 to achieve the down-sampling effect, while transferring the scale-invariant features to the next layer. In addition, YOLOv3 also uses structures similar to ResNet and FPN networks, these two structures are also of great benefit to improving the detection accuracy. This network is better for motor vehicle detection from the perspective of drones. For larger network models, the calculation of the above-mentioned network structure is time-consuming, and the real-time performance is poor, requiring strong computing power support, and cannot be deployed on the end side.

At this stage, the mainstream rotary-wing UAV target tracking algorithm is based on KCF technology. KCF (Kernel Correlation Filtering) is developed from the detection-based tracking loop structure (CSK) of the kernel, which uses online learning methods to solve tracking problems. It is a machine learning method without any prior knowledge. In the first frame, manually select an object of interest (OOI) area, and the KCF tracker converts this area into a multi-channel HOG feature descriptor. The HOG descriptor is used to perform ridge regression, and the regression function f(z) of the OOI area z is initialized. For a new frame, f(z) is evaluated on several areas near the last area of OOI. Finally, the region with the largest evaluation response is used as the output to update f(z). In order to speed up the calculation of the ridge regression matrix, KCF converts the descriptor of each channel of the HOG feature descriptor into a cyclic matrix through cyclic shift. The circulant matrix can be diagonalized by discrete Fourier transform (DFT). Therefore, matrix calculations can be effectively processed in the Fourier domain, especially matrix inversion. In addition, a kernel function is applied in the KCF tracker to improve the tracking performance, and the regression function f(z) is mapped to the nonlinear space. These solutions were introduced by CSK and optimized in KCF. In this way, the processing speed and average accuracy of KCF reached 172FPS and 73.2%, respectively.

Although the ground station is used for calculation and reasoning to ensure sufficient computing resources, the time consumption of large-scale data transmission is unacceptable, the uncertainty of the wireless transmission network also has time delay, and the ground station needs to pay for the speed of the reasoning calculation. High cost.

The real-time nature of flight control is a necessary factor for UAV safety. If the delay is high, it will inevitably affect the detection and tracking effect and flight safety. In addition, in the target tracking process, the KCF algorithm is affected by the uncertainties of the external environment, such as illumination changes, occlusion, etc., which will cause the target tracking to be lost, and the target cannot be repositioned after the target is lost, which will eventually lead to the tracking failure.

Summary of the invention

Purpose of the invention: In order to solve the problem of high flight control delay in the prior art, the present invention proposes a rotary wing UAV system for vehicle detection and tracking. The UAV end-side calculation scheme is adopted, and the ground station is only used for tracking video monitoring and tracking. Manual flight control command issuance operation effectively improves the timeliness performance of the system. Furthermore, the YOLO Nano target detection algorithm solves the problem of heavy computing load on the airborne computer; through the Staple-based tracking algorithm and the target re-detection module, the KCF algorithm is limited to the external environment and the tracking is unstable.

Technical solution: A rotary-wing UAV system for vehicle detection and tracking, including UAV platform and ground station platform, using UAV end-to-side computing solution, greatly reducing system delay and ensuring motor vehicle targets Real-time detection and tracking. The UAV platform is used for real-time calculation and detection of tracking targets; the ground station platform is used for tracking and video monitoring of the targets, and issuing manual flight control instructions to the UAV platform.

Further, the unmanned aerial vehicle platform includes: a visible light camera, an onboard computer, a first wireless image transmission terminal, and a flight control module, wherein the onboard computer is respectively connected to the visible light camera, the first wireless image transmission terminal, and the flight control module. Modules are connected; the ground station platform includes a PC and a second wireless image transmission terminal, which exchange information; the first wireless image transmission terminal and the second wireless image transmission terminal exchange information;

The visible light camera is used to collect image data;

The onboard computer is used to run a target detection algorithm and a target tracking algorithm;

The first wireless image transmission terminal is used to transmit a real-time video stream of target tracking and receive manual flight control instructions from a ground station;

The PC is used for real-time video stream monitoring of target tracking and manual flight control command issuance;

The second wireless image transmission terminal is used to receive a real-time video stream of target tracking and send manual flight control instructions.

Further, the target detection algorithm run by the onboard computer adopts the YOLO Nano algorithm.

Further, the target tracking algorithm run by the onboard computer adopts the Staple tracking algorithm.

Further, the on-board computer also includes a target re-detection module, which is used to determine whether the target is occluded according to the correlation value between the test sample of the Staple tracking algorithm and the training sample, and set the threshold of the correlation value. The threshold value judges that there is occlusion. If there is occlusion, the predicted value of the target is copied to the measured value, and the measured value is corrected to obtain the estimated value of the target position.

Further, the onboard computer is used to deploy the Ubuntu ROS operating system, which includes a camera node, a target detection node, a target tracking node, and a flight control node; among them, the camera node is used to collect image data, and the target detection node is used for all For vehicle positioning, the target tracking node is used to track the target vehicle, and the flight control node is used to control the flight of the rotary-wing UAV.

The detection and tracking method of the rotary wing unmanned aerial vehicle system for vehicle detection and tracking according to the present invention is characterized in that the method includes:

The UAV platform calculates and detects and tracks the target in real time; the ground station platform sends flight control instructions to the UAV through wireless communication to control the flight of the aircraft.

Further, the method specifically includes the following steps:

(1) The visible light camera collects image data and publishes image topics through the camera node of the onboard computer;

(2) The target detection node subscribes to the image topic and uses it as the input of the target detection node. The onboard computer calculates the vehicle coordinate information according to the target detection algorithm and publishes the topic of vehicle coordinate information;

(3) The target tracking node subscribes to the topic of vehicle coordinate information, and the onboard computer predicts the location of the target vehicle according to the target tracking algorithm, and publishes the topic of the target location;

(4) The flight control node subscribes to the target location topic, performs coordinate conversion, calculates the distance between the target and the UAV, and sends flight control instructions to the flight control module accordingly;

(5) The flight control module executes instructions to control the movement of the drone.

Further, the target detection algorithm adopts the YOLO Nano algorithm. The use of this compact network architecture greatly reduces the size of the model while ensuring the detection accuracy, making end-side computing time-consuming to meet the requirements, and adapting to the computing power of the airborne computer.

Further, the target tracking algorithm specifically includes the following steps:

(1) Perform the initialization of Kalman filter and Staple tracking algorithm;

(2) Perform target tracking in the image sequence;

(3) In the tracking process, first predict the position of the target vehicle in frame k based on the target state in frame k-1, and then sample the image block at the predicted position and input it to the Staple tracking algorithm to obtain the position of the target vehicle in the image. Measure the value, and then judge whether the target is occluded according to the correlation value between the test sample of the Staple tracking algorithm and the training sample. Set a threshold for the correlation value. If it is lower than the threshold, determine that there is occlusion. If there is occlusion, Copy the predicted value of the target vehicle to the measured value;

(4) Correct the target measurement value to obtain the estimated value of the target vehicle position;

(5) Update the target state of the previous frame.

When the complexity of the environment causes the UAV to lose the target during the tracking process, the re-detection algorithm based on Kalman filter can quickly search for the target again after the target is lost, ensuring that the UAV has a long-term and long-term response to the vehicle target in a complex environment. Stable tracking.

Beneficial effects: Deploying the algorithm on the UAV end side greatly reduces the system delay and ensures the real-time detection and tracking of motor vehicle targets; due to the computing power limitation of the UAV end side computing equipment, YOLO is adopted The Nano algorithm greatly reduces the size of the model while ensuring the accuracy of the detection, making the end-side computing time-consuming to meet the requirements. The Stape tracking algorithm is used and the re-detection module is embedded, so that the tracking algorithm can quickly search for the target again after the target is lost, ensuring the long-term tracking.

Description of the drawings

Figure 1 is a block diagram of the node design of the airborne computer ROS system of the present invention;

Figure 2 is a block diagram of the system hardware structure of the present invention;

Figure 3 is a block diagram of the system software structure of the present invention;

Figure 4 is a flow chart of the improved tracking algorithm of the present invention.

Detailed ways

The technical solution of the present invention will be further introduced below in conjunction with the drawings and specific embodiments.

The present invention proposes a rotary wing unmanned aerial vehicle system for vehicle detection and tracking, aiming at practical application scenarios, that is, automatic detection and tracking of motor vehicles. The rotary-wing drone obtains real-time video streams on the ground through the mounted visible light camera, and can transmit the video streams to the ground workstation in real time, and detect vehicle targets in the image through the on-board target detection algorithm. The ground station manually selects the vehicle targets to be tracked in the video. After that, the airborne target tracking algorithm is activated, and the rotary-wing UAV will automatically fly to track the selected target on the ground. The airborne computing device NVIADIA Jetson TX2 is used to run the Ubuntu ROS system, and the camera node, target detection node, target tracking node and flight control node are deployed and integrated on the ROS system.

Figure 1 is a block diagram of the node design of the airborne computer ROS system of the present invention. The workflow of the UAV platform includes the following steps:

(2) The target detection node subscribes to the image topic and uses it as the input of the target detection node. The onboard computer calculates the target coordinate information according to the target detection algorithm and publishes the target coordinate information topic;

(3) The target tracking node subscribes to the topic of coordinate information, and the onboard computer predicts the target location according to the target tracking algorithm, and publishes the target location topic;

(4) The flight control node subscribes to the target location topic, performs coordinate conversion, calculates the distance between the target and the aircraft, and sends flight control instructions to the flight control module accordingly;

Figure 2 is a block diagram of the hardware structure of the system of the present invention, including: the present invention is applied to the detection and tracking of ground motor vehicles by a rotary-wing drone. The drone load includes a visible light camera, TX2 airborne computer, wireless image transmission, etc., TX2 The Ubuntu camera deployed on the airborne computer adopts a gimbal camera with self-stabilization function, which can shoot 1080P resolution video, and the acquisition rate is 30FPS. The gimbal camera is fixed under the drone and shoots the ground at a fixed angle. The algorithm processing unit is a TX2 airborne computer, with the Ubuntu16.04 operating system installed, and the ROS version is kinetic.

Figure 3 is a block diagram of the system software structure of the present invention, which includes: Compared with ground station for inference calculation, the computing power of TX2 airborne computer is greatly reduced. Therefore, the target detection algorithm based on deep learning needs to be adjusted according to the computing power. The traditional The size of the YOLOv3 target detection model is 240M, the calculation complexity is too large, and it is no longer suitable for edge devices. The original network needs to be pruned. The size of YOLO Nano is only about 4.0MB, which is 15.1 times and 8.3 times smaller than Tiny YOLOv2 and Tiny YOLOv3, respectively. It requires 4.57B inference operations in calculation, which is 34% and 17% less than the latter two networks. In the above, 69.1% mAP was achieved in the VOC2007 data set, and the accuracy rate was improved by 12 points and 10.7 points respectively compared with the latter two. Therefore, deploying the YOLO Nano algorithm on TX2 can significantly reduce the computational pressure and ensure the accuracy of target detection.

(1) Design of target detection algorithm

Use the design principles of the YOLO series single-shot object detection network architecture to create YOLO Nano, which is a highly compact network with highly customized module-level macro-architecture and micro-architecture designs tailored to the application. The network structure mainly includes the residual PEP macro architecture and the fully connected attention macro architecture FCA.

PEP consists of a 1*1 convolutional mapping layer, which maps the input feature map to a lower-dimensional tensor. The num in PEP(num) is the lower dimension; a 1*1 convolutional expansion layer, It will expand the channel of the feature map to higher dimensions; a depth-wise convolution layer, which will perform spatial convolution on different expansion layer output channels through different filters; a 1*1 Convolutional mapping layer, which maps the output channels of the previous layer to lower dimensions. The first two steps combine cross-channel fusion features; the second step increases the feature dimension, so that more channel features in the third step are used for spatial feature fusion (to improve the abstraction and characterization capabilities of features); the third step is the deep convolution part ( Spatial convolution); the fourth step is the point-by-point convolution part (channel convolution), which reduces the huge amount of calculation caused by the reduction of the channel after the convolution; the latter two parts form a deep separable convolution, which can reduce the computational complexity Guarantee the characterization ability of the model. The use of the residual PEP macro-architecture can significantly reduce the complexity of architecture and calculations, while ensuring the characterization ability of the model. The FCA macro architecture consists of two fully connected layers, which can learn the dynamic and non-linear internal dependencies between channels, and re-weight the importance of the channels through channel-level multiplication. The use of FCA helps to focus on more informative features based on global information, because it calibrates the dynamic features again. This can make more effective use of the ability of neural networks, that is, express important information as much as possible with limited parameters. Therefore, this module can make a better trade-off between pruning the model architecture, reducing model complexity, and increasing model representation.

The size of YOLO Nano is only about 4.0MB, which is 15.1 times and 8.3 times smaller than Tiny YOLOv2 and Tiny YOLOv3, respectively. It requires 4.57B inference operations in calculation, which is 34% and 17% less than the latter two networks. In the above, 69.1% mAP was achieved in the VOC2007 data set, and the accuracy rate was improved by 12 points and 10.7 points respectively compared with the latter two. Therefore, deploying the YOLO Nano algorithm on TX2 can significantly reduce the computational pressure and ensure the accuracy of target detection.

(2) Target tracking algorithm design

The Staple algorithm is robust to motion blur and illuminance when considering HOG features for correlation filtering, but it is not robust to deformation. If the target is deformed, the color distribution of the entire target will basically not change. Therefore, the color histogram is very robust to deformation. On the other hand, the color histogram is not robust to light changes, which can be complemented by HOG features. Therefore, consider dividing into two channels and using these two features at the same time. Use HOG features to learn related filters to obtain a filter template, and update the template using the given formula. Use the color feature to learn the filter template, and then use the given update formula to update the learned template. Two templates are used to predict the target position and then weighted and averaged to obtain a composite response map, and the position of the maximum value in the response map is the target location. Although the Staple tracking algorithm overcomes some of the shortcomings of the KCF algorithm, there is still no reliable solution to target occlusion and loss. Therefore, the re-detection module is added to the Staple algorithm framework to effectively solve this problem. Specifically, the position of the target in the next frame is estimated, and then the estimated position is sampled to further lock the position of the target. The Kalman filter can establish a linear motion model of the target body, and the state of the target can be optimally estimated through the input value and output value of the model. Therefore, the Kalman filter is used to establish the target motion model to predict the next moment of the target. Position, camera shake can be regarded as Gaussian noise.

(3) Improvement based on Kalman filter

The motion model and observation equation of the target can be expressed in the following form. Since the sampling interval between the two frames of images is very short, the motion of the target between the two frames is simplified to a uniform motion. x and y respectively represent the component of the pixel distance between the position of the target and the center of the image on the u-axis and the v-axis,

with

Represent the components of the target's moving speed on the u-axis and v-axis, respectively. Because the acceleration of the target body movement is random, it can be

with

Think of it as Gaussian noise. Δt is the time interval between adjacent moments.

The observation value at time k is:

Where v _k is the measurement error at time k, so the state transition matrix A, control matrix G, and measurement matrix H of the system are respectively:

Fig. 4 is a flow chart of the improved tracking algorithm of the present invention, which includes: first perform the initialization work of the Kalman filter and the Staple tracking algorithm, and then perform the target tracking in the image sequence. In the tracking process, first predict the position of the target in the k frame according to the target state in the k-1 frame, and then sample the image block at the predicted position and input it to the Staple tracking algorithm to obtain the measured value of the target position in the image, and then according to Staple tracking algorithm tests the value of the correlation between the sample and the training sample to determine whether the target is occluded, and set a threshold for the correlation value. If it is lower than the threshold, it is determined that there is occlusion. If there is occlusion, the predicted value of the target is copied Give the measured value. The next step is to correct the target measurement value, and finally get the estimated value of the target position. In this process, after the target position of each frame is corrected, the target state of the previous frame is updated.

Claims

A rotary wing unmanned aerial vehicle system for vehicle detection and tracking. The system includes an unmanned aerial vehicle platform and a ground station platform, characterized in that the unmanned aerial vehicle platform is used for real-time calculation and detection and tracking of targets; the ground station platform It is used to track and monitor the target and issue manual flight control instructions to the UAV platform.
The rotary wing unmanned aerial vehicle system for vehicle detection and tracking according to claim 1, wherein the unmanned aerial vehicle platform comprises: a visible light camera, an onboard computer, a first wireless image transmission terminal and a flight control module, wherein , The onboard computer is respectively connected with a visible light camera, a first wireless image transmission terminal, and a flight control module; the ground station platform includes a PC and a second wireless image transmission terminal, which exchange information; the first wireless image transmission Information interaction between the terminal and the second wireless image transmission terminal;

The visible light camera is used to collect image data;

The onboard computer is used to run a target detection algorithm and a target tracking algorithm;

The first wireless image transmission terminal is used to transmit a real-time video stream of target tracking and receive manual flight control instructions from a ground station;

The PC is used for real-time video stream monitoring of target tracking and manual flight control command issuance;

The second wireless image transmission terminal is used to receive a real-time video stream of target tracking and send manual flight control instructions.
The rotary wing unmanned aerial vehicle system for vehicle detection and tracking according to claim 1, wherein the target detection algorithm run by the onboard computer adopts the YOLO Nano algorithm.
The rotary-wing unmanned aerial vehicle system for vehicle detection and tracking according to claim 1, wherein the target tracking algorithm run by the on-board computer adopts the Staple tracking algorithm.
The rotary wing unmanned aerial vehicle system for vehicle detection and tracking according to claim 1, wherein the onboard computer further comprises a target re-detection module for determining the correlation between the test sample and the training sample of the Staple tracking algorithm Value, determine whether the target is occluded, set the threshold of the correlation value, if it is lower than the threshold, it will determine that there is occlusion, if there is occlusion, copy the predicted value of the target to the measured value, correct the measured value, and get the target position Estimated value.
The rotary wing drone system for vehicle detection and tracking according to claim 1, wherein the onboard computer is used to deploy the Ubuntu ROS operating system, and the system includes a camera node, a target detection node, a target tracking node, Flight control node; among them, the camera node is used to collect image data, the target detection node is used to locate all vehicles, the target tracking node is used to track the target vehicle, and the flight control node is used to control the flight of the rotary-wing UAV.
A detection and tracking method for a rotary-wing UAV system for vehicle detection and tracking based on claim 1, wherein the method comprises:

The UAV platform calculates and detects and tracks the target in real time; the ground station platform sends flight control instructions to the UAV through wireless communication to control the flight of the aircraft.
The detection and tracking method of a rotary-wing UAV system for vehicle detection and tracking according to claim 7, wherein the UAV platform includes a visible light camera, an onboard computer, a first wireless image transmission terminal, and a flight controller. Module, wherein the onboard computer is connected to the visible light camera, the first wireless image transmission terminal, and the flight control module, and the Ubuntu ROS operating system is deployed on the onboard computer. The system includes camera nodes for collecting image data, The target detection node for all vehicle positioning, the target tracking node for tracking the target vehicle, and the flight control node for controlling the flight of the rotor drone; the ground station platform includes a PC and a second wireless image transmission terminal, The two information exchange; the first wireless image transmission terminal and the second wireless image transmission terminal information exchange; the method includes the following steps:

(1) The visible light camera collects image data and publishes image topics through the camera node of the onboard computer;

(2) The target detection node subscribes to the image topic and uses it as the input of the target detection node. The onboard computer calculates the vehicle coordinate information according to the target detection algorithm and publishes the topic of vehicle coordinate information;

(3) The target tracking node subscribes to the topic of vehicle coordinate information, and the onboard computer predicts the location of the target vehicle according to the target tracking algorithm, and publishes the topic of the target location;

(4) The flight control node subscribes to the target location topic, performs coordinate conversion, calculates the distance between the target and the UAV, and sends flight control instructions to the flight control module accordingly;

(5) The flight control module executes instructions to control the movement of the drone.
The detection and tracking method of a rotary-wing UAV system for vehicle detection and tracking according to claim 8, wherein the target detection algorithm adopts the YOLO Nano algorithm.
The detection and tracking method of a rotary-wing UAV system for vehicle detection and tracking according to claim 8, wherein the target tracking algorithm specifically includes the following steps:

(1) Perform the initialization of Kalman filter and Staple tracking algorithm;

(2) Perform target tracking in the image sequence;

(3) In the tracking process, first predict the position of the target vehicle in frame k based on the target state in frame k-1, and then sample the image block at the predicted position and input it to the Staple tracking algorithm to obtain the position of the target vehicle in the image. Measure the value, and then judge whether the target is occluded according to the correlation value between the test sample of the Staple tracking algorithm and the training sample. Set a threshold for the correlation value. If it is lower than the threshold, determine that there is occlusion. If there is occlusion, Copy the predicted value of the target vehicle to the measured value;

(4) Correct the target measurement value to obtain the estimated value of the target vehicle position;

(5) Update the target state of the previous frame.