CN108492324A - Aircraft method for tracing based on fully-connected network and Kalman filter - Google Patents

Abstract

The present invention relates to a kind of aircraft method for tracing based on fully-connected network and Kalman filter, includes the following steps：Video is detected frame by frame using fully-connected network R FCN, obtains the bounding box of previous frame image, so that trajectory corrector uses；State vector, description airplane motion track are built, state vector should represent the position of target aircraft central point, show the size and aspect ratio of bounding box again；Kalman filter and extended Kalman filter are combined, subvector is isolated from the state vector of structure to describe moving target；When testing result deviation is larger, effective range is limited according to the size of target object and improves detection speed, corrects the position of bounding box in consecutive frame, to realize the correction to movement locus.

Description

Aircraft Tracking Method Based on Fully Connected Network and Kalman Filter

technical field

The invention belongs to the field of computer vision and relates to a deep learning tracking method for aircraft in videos.

Background technique

Aircraft tracking is an important technology in aviation security and other fields, and the introduction of scientific and technological means in military reconnaissance to strengthen security measures has gradually attracted national attention. As an important field in computer vision, target tracking can realize the function of checking and even tracking the aircraft in the acquired video images.

In recent years, with the development of deep learning, machine learning algorithms have been gradually applied to various vision fields, and object detection and tracking technology based on deep learning has also developed rapidly. Compared with traditional methods, its tracking performance has been greatly improved. Popular object detection strategies include region proposal and region stepwise. However, the accuracy of aircraft tracking is mainly affected by complex environmental conditions, and there are still some challenging problems in visual tracking algorithms, such as sudden movement, attitude changes, deformation, occlusion, background clutter, lighting or viewpoint changes, etc. The accuracy of tracking decreases, and even leads to tracking failure. Currently there is no effective algorithm to solve the problem of aircraft tracking.

Contents of the invention

The purpose of the present invention is to establish a more accurate aircraft tracking method. The aircraft tracking method proposed by the invention includes three main parts: a detection model based on R-FCN, a state estimation model based on a Kalman filter, and an aircraft trajectory correction module. The technical solution is as follows:

An aircraft tracking method based on a fully connected network and a Kalman filter, comprising the following steps:

The first step: use the fully connected network R-FCN to detect the video frame by frame, and obtain the boundingbox of the previous frame image for trajectory correction;

Step 2: Construct a state vector to describe the trajectory of the aircraft. The state vector should not only indicate the position of the center point of the target aircraft, but also show the size and aspect ratio of the bounding box;

The third step: In order to avoid the target drift caused by the failure of target detection on a certain frame, the Kalman filter and the extended Kalman filter are combined to separate the sub-vectors from the constructed state vector to describe the moving target; specifically Methods as below:

(1) The Kalman filter processes the linear part: the sub-vector representing the position of the center point of the aircraft in the state vector is approximated by a linear model, and then the Kalman gain is calculated from the uncertainty of the prediction result and the current observation result, and the prediction result and the observation result Do a weighted average to get the state estimate at the current moment and the uncertainty of this state estimate;

(2) The extended Kalman filter is used to fit the nonlinear part that is not suitable for the linear model: create a state subvector that can represent the size and aspect ratio of the bounding box in the same way as in (1), but the state involved The matrix and mapping matrix are no longer constant matrices, and the state estimation of the nonlinear part at the current moment and the uncertainty of this state estimation are obtained;

(3) Add the nonlinear part to the linear system to describe the motion state of the aircraft;

Step 4: When the deviation of the detection result is large, limit the effective range according to the size of the target object to improve the detection speed, and correct the position of the bounding box in the adjacent frame, so as to realize the correction of the motion trajectory. If the bounding box and the aircraft are located If the window overlap IOU value is greater than the predefined threshold T, the position and size of the current detection frame will be modified according to the bounding box of the previous frame; The formula is as follows:

Among them, δ _a , δ _b represent the confidence of the detection results between the front and rear frames respectively, and the higher the value, the more accurate the model is; w _a , w _b , w _c represent the width of the front and rear frames and the corrected bounding box respectively, h _a , h _b , h _c represent the heights of the front and rear frames and the corrected bounding box respectively; (x _a , y _a ), (x _b , y _b ), (x _c , y _c ) represent the front and rear frames and the corrected bounding box respectively The horizontal and vertical coordinates of the center point of the box;

Step 5: Collect videos containing aircraft movement, process them into fixed-length videos, and form a training database together. The videos are randomly divided into two parts, and 80% of them are input into R-FCN as a training set. Forecast the next 20% of the video and get the aircraft tracking results.

The specific aircraft tracking algorithm proposed by the invention is realized on the basis of R-FCN and Kalman filter. R-FCN is a detection model used to obtain aircraft position information. In order to reduce the detection time, we also crop a specific region in each frame according to the position of the boundingbox in the previous frame. In addition, the size of the detection area can also be changed according to the size of the target. A Kalman filter is used as an evaluation model to adjust the predicted motion traces. When the detection difference of adjacent frames is large, in order to improve the certainty rate of detection, the bounding box of the next frame can be adjusted according to the detection result. Therefore, the method can accurately detect and track the trajectory of the aircraft.

Description of drawings

Figure 1: Flowchart

Figure 2: Schematic diagram of the calibration method

Detailed ways

The present invention provides an aircraft tracking method based on R-FCN ^[1] and Kalman filter ^[2, 3], including a detection model based on R-FCN, a state estimation model based on KF, and an aircraft trajectory There are three main parts of the correction module. Specifically, it can be expressed as the following steps:

Step 1: Obtain the bounding box of the previous frame through R-FCN.

The specific method is as follows:

(1) According to the R-FCN principle, the relative spatial position information of the Region of Interest (ROI) is encoded by creating a spatially sensitive score map. The present invention generates a 4-dimensional vector v=(v _x , v _y , v _w , v _h ) for each ROI by means of the bounding box regression method, which is used for subsequent calculation of the bounding box. Among them, v _x , v _y represent the x, y coordinates of the center point, v _w , v _h represent the width and height of the bounding box.

(2) Divide the ROI into four character areas: upper left, lower left, upper right, and lower right, and these sub-areas are used as score maps. Then the vertices of the bounding box can be represented as B _tl , B _tr , B _bl , B _bt , and the relationship with v _x , v _y , v _w , v _h is expressed as follows:

Step 2: Build the state vector Describe the trajectory of the aircraft.

in, Indicates the position of the center point of the target, Indicates the scale and aspect ratio of the bounding box, and black dots indicate derivative forms. This step mainly deals with the linear part, the specific method is as follows:

(1) Approximate by a linear model, separate subvectors from the constructed state vector to describe the goals of the exercise.

x _k ＝Ax _k-1 +Bu _k +w _k (2)

z _k ＝Hx _k +v _k (3)

Among them, x represents the state vector of the system, z is the observed value, A is the state transition matrix, and the control part composed of B and u can be ignored in the non-controllable system applied in the present invention. H is the observation model, that is, the observation matrix, which can map the real state to the observation space. w and v denote the noise in the state update and observation process, respectively. Previous studies have shown that the above noise obeys a Gaussian distribution. Formula (2) is called the state equation, formula (3) is called the observation equation, and the subscript k represents the value at time k.

(2) According to the KF principle:

Among them, P is the error covariance matrix between the predicted value and the real value, which is used to represent the uncertainty of the prediction result, and Q is the new uncertainty added in the prediction process. Formula (4) indicates that the current state is predicted from the state at the previous moment plus external input. Formula (5) represents the uncertainty that existed before, and a new uncertainty Q is added in the forecasting process.

(3) Calculate the Kalman gain K and the kth predicted value It can be calculated by the following formula:

Equation (6) expresses the uncertainty of the predicted results by and the uncertainty R of the observation results to calculate the Kalman gain (weight) K. Formula (7) expresses the weighted average of the prediction results and observation results to obtain the state estimation at the current moment.

(4) update , which represents the uncertainty of this state estimation.

The third step: use EKF to fit the nonlinear part of s and r which is not suitable for the linear model. Create the subvectors in the same way as in the third step But the state matrix A and the mapping matrix H are no longer constant matrices, they are expressed by the following formula:

Wherein, the matrices F _k and H _k are derived from the Jacobian matrix, and they correspond to A and H in KF respectively.

Step 4: Add the nonlinear part to the linear system to describe the motion state of the aircraft.

Step 5: Correct the motion track. Determine whether the bounding box can correctly detect the moving aircraft. If the value of the overlap (Intersection over Union, IOU) between the bounding box and the window where the aircraft is located is greater than the predefined threshold T, the position and size of the current detection frame are modified according to the bounding box of the previous frame. Otherwise, the target is used as the center, and the bounding box is drawn to input the detection network for training. The specific correction formula is as follows:

Among them, δ _a and δ _b represent the confidence of the detection results between the previous and subsequent frames respectively, and the higher the value, the more accurate the model is. w _a , w _b , w _c represent the widths of the front and rear frames and the corrected bounding box, respectively, and h _a , h _b , h _c represent the heights of the front and rear frames and the corrected bounding box, respectively. (x _a , y _a ), (x _b , y _b ), (x _c , y _c ) represent the horizontal and vertical coordinates of the front and back frames and the center point of the corrected bounding box, respectively.

Step 6: Collect videos containing aircraft movements, and process them into fixed-length videos to form a training database for the convenience of training. The videos are randomly divided into two parts, 80% of which are input into R-FCN as the training set, and the remaining 20% of the videos are predicted according to the trained model to obtain the aircraft tracking results.

references:

[1]Dai J, Li Y, He K, Sun J(2016) R-fcn: Object detection via region-based fully convolutional networks. In: Advances in Neural Information Processing Systems, pp 379C387.

[2] Kalman RE, et al(1960) A new approach to linear filtering and prediction problems. Journal of basic Engineering 82(1):35C45.

[3] Kalman RE, Bucy RS (1961) New results in linear filtering and prediction theory. Journal of basic engineering 83(1):95C108.

Claims (1)

1. An aircraft tracking method based on fully connected network and Kalman filter, comprising the following steps: The first step: use the fully connected network R-FCN to detect the video frame by frame, and obtain the boundingbox of the previous frame image for trajectory correction; Step 2: Construct a state vector to describe the trajectory of the aircraft. The state vector should not only indicate the position of the center point of the target aircraft, but also show the size and aspect ratio of the bounding box; The third step: In order to avoid the target drift caused by the failure of target detection on a certain frame, the Kalman filter and the extended Kalman filter are combined to separate the sub-vectors from the constructed state vector to describe the moving target; specifically Methods as below: (1) The Kalman filter processes the linear part: the sub-vector representing the position of the center point of the aircraft in the state vector is approximated by a linear model, and then the Kalman gain is calculated from the uncertainty of the prediction result and the current observation result, and the prediction result and the observation result Do a weighted average to get the state estimate at the current moment and the uncertainty of this state estimate; (2) The extended Kalman filter is used to fit the nonlinear part that is not suitable for the linear model: create a state subvector that can represent the size and aspect ratio of the bounding box in the same way as in (1), but the state involved The matrix and mapping matrix are no longer constant matrices, and the state estimation of the nonlinear part at the current moment and the uncertainty of this state estimation are obtained; (3) Add the nonlinear part to the linear system to describe the motion state of the aircraft; Step 4: When the deviation of the detection result is large, limit the effective range according to the size of the target object to improve the detection speed, and correct the position of the bounding box in the adjacent frame, so as to realize the correction of the motion trajectory. If the bounding box and the aircraft are located If the window overlap IOU value is greater than the predefined threshold T, the position and size of the current detection frame will be modified according to the bounding box of the previous frame; The formula is as follows: Among them, δ _a , δ _b represent the confidence of the detection results between the front and rear frames respectively, and the higher the value, the more accurate the model is; w _a , w _b , w _c represent the width of the front and rear frames and the corrected bounding box respectively, h _a , h _b , h _c represent the heights of the front and rear frames and the corrected bounding box respectively; (x _a , y _a ), (x _b , y _b ), (x _c , y _c ) represent the front and rear frames and the corrected bounding box respectively The horizontal and vertical coordinates of the center point of the box; Step 5: Collect videos containing aircraft movement, process them into fixed-length videos, and form a training database together. The videos are randomly divided into two parts, and 80% of them are input into R-FCN as a training set. Forecast the next 20% of the video and get the aircraft tracking results.