CN110706253A - Target tracking method, system and device based on apparent feature and depth feature - Google Patents

Abstract

The invention belongs to the technical field of computer vision tracking, and in particular relates to a target tracking method, system and device based on apparent features and depth features, and aims to solve the problem of low tracking accuracy caused by ignoring depth information of target scenes in existing target tracking methods. The method of the system includes obtaining the target area and the search area of the target to be tracked in the t-th frame image according to the target position of the t-1 frame and the preset target size; extracting the target area and searching The apparent features and depth features of the area; based on the preset weights, the apparent features and depth features of the target area and the search area are weighted and averaged respectively to obtain their respective fusion features; according to the fusion features of the target area and the search area, through correlation The filter obtains the response map of the target; the position corresponding to the peak of the response map is taken as the target position of the t-th frame. The invention extracts the depth information of the target scene and improves the accuracy of the target tracking.

Description

Target tracking method, system and device based on apparent feature and depth feature

technical field

The invention belongs to the technical field of computer vision tracking, and in particular relates to a target tracking method, system and device based on appearance features and depth features.

Background technique

Object tracking is one of the most fundamental problems in the field of computer vision, and its task is to estimate the motion trajectories of objects or image regions in video sequences. Object tracking has a very wide range of applications in real-world scenarios, often acting as a component in a larger computer vision system. For example, both autonomous driving and vision-based active safety systems rely on tracking the location of vehicles, cyclists, and pedestrians. In robotic systems, tracking objects of interest is a very important aspect of visual perception, which in turn extracts high-level information from camera sensors for decision-making and navigation. In addition to robotics-related applications, object tracking is often used for automatic video analysis, where information is first extracted by detecting and tracking players and objects involved in the game. Other applications include augmented reality and dynamic structures, whose task is usually to track different local image regions. As can be seen from the diversity of applications, the object tracking problem itself is very diverse.

In recent decades, although breakthroughs have been made in the field of target tracking, many classic research results have been produced. However, there are still many theoretical and technical problems to be solved in the field of target tracking, especially the complex problems encountered in the open environment such as background interference, illumination changes, scale changes, and occlusion during the tracking process. Therefore, how to adaptively, real-time and robustly track targets in complex scenes has always been a problem that researchers need to solve, and the research value and research space are still very large.

For single-target tracking, the quality of the feature directly determines the tracking performance. The early discriminative models based on hand-crafted features could only extract some shallow features of the target object, and could not describe the essence of the target object well. However, the recently emerged convolutional neural network can learn the target object through a hierarchical structure. The feature representation of objects at different levels, but ignores the global depth information of the target scene. Depth information can be used as an auxiliary feature to provide global information for the target and solve problems such as occlusion of the target, thereby improving the robustness of the model in complex scenes. Therefore, the present invention proposes a target tracking method based on appearance features and depth features.

SUMMARY OF THE INVENTION

In order to solve the above problem in the prior art, that is, in order to solve the problem that the existing target tracking method ignores the depth information of the target scene, resulting in low tracking accuracy, the first aspect of the present invention proposes a target based on appearance features and depth features. A method of tracking, which includes:

Step S10, according to the target position of the t-1 frame and the preset target size, obtain the area of the target to be tracked in the t-th frame image, and use this area as the target area; and based on the target position of the t-1 frame. The center point obtains an area N times the size of the target area, and uses it as the search area;

Step S20, through the apparent feature extraction network and the depth feature extraction network, respectively extract the target area, the apparent feature and the depth feature corresponding to the search area;

Step S30, based on a preset weight, perform a weighted average on the corresponding apparent features and depth features of the target area and the search area, respectively, to obtain a fusion feature of the target area and a fusion feature of the search area;

Step S40, according to the fusion feature of the target area and the fusion feature of the search area, obtain the response map of the target to be tracked through a correlation filter; take the position corresponding to the peak value of the response map as the target of the t-th frame Location;

in,

The apparent feature extraction network is constructed based on a convolutional neural network, and is used to obtain its corresponding apparent feature according to the input image;

The deep feature extraction network is constructed based on the ResNet network, and is used to obtain its corresponding deep features according to the input image.

In some preferred embodiments, in step S10 "according to the target position of frame t-1 and the preset target size, obtain the area of the target to be tracked in the t-th frame image, and use this area as the target area", the method of is: if t is equal to 1, obtain the target area of the target to be tracked according to the preset target position and the preset target size; if t is greater than 1, then according to the target position of the frame t-1 and the preset target size, Get the target area of the target to be tracked.

In some preferred embodiments, the apparent feature extraction network is structured as follows: two convolutional layers and one correlation filtering layer, each convolutional layer is connected to a maximum pooling layer and a ReLU activation function; the network During the training process, the back-propagation algorithm is used for training.

In some preferred embodiments, the structure of the deep feature extraction network is: 5 convolution layers and 5 deconvolution layers; the deep feature extraction network is trained by mutual reconstruction of binocular images during the training process.

In some preferred embodiments, in the process of extracting the depth feature of the deep feature extraction network, if t is equal to 1, the extraction method is:

Obtain the depth feature of the first frame image based on the depth feature extraction network;

Based on the depth feature of the first frame of image and the preset target position, the depth feature of the target area and the search area is acquired.

In some preferred embodiments, when the correlation filter filters the target area, different scales are obtained by a scale transformation method, and the target area is enlarged or reduced according to the different scales, and then filtered. . The scaling method is:

Among them, a is the scale coefficient, s is the scale pool, S is the preset number of scales, and a ^s is the scale.

In some preferred embodiments, after step S40, the update of the state value of the correlation filter is also included, and the method is as follows:

Obtain the state value of the correlation filter at frame t-1;

Based on the state value, the target position of the t-th frame, and the preset learning rate, the state value of the correlation filter frame t is updated.

In a second aspect of the present invention, a system for target tracking based on apparent features and depth features is proposed, the system includes an acquisition region module, a feature extraction module, a feature fusion module, and an output position module;

The acquisition area module is configured to acquire the area of the target to be tracked in the t-th frame image according to the target position of the t-1 frame and the preset target size, and use this area as the target area; and based on the t-1 The center point of the target position of the frame obtains an area N times the size of the target area, and takes it as the search area;

The feature extraction module is configured to extract the target area, the apparent feature and the depth feature corresponding to the search area through an apparent feature extraction network and a depth feature extraction network, respectively;

The feature fusion module is configured to perform a weighted average of the apparent features and depth features corresponding to the target area and the search area, respectively, based on a preset weight, to obtain a fusion feature of the target area and a fusion feature of the search area;

The output position module is configured to obtain a response map of the target to be tracked through a correlation filter according to the fusion feature of the target area and the fusion feature of the search area; take the position corresponding to the peak value of the response map as The target position of the t-th frame;

in,

The apparent feature extraction network is constructed based on a convolutional neural network, and is used to obtain its corresponding apparent feature according to the input image;

The deep feature extraction network is constructed based on the ResNet network, and is used to obtain its corresponding deep features according to the input image.

In a third aspect of the present invention, a storage device is provided, in which a plurality of programs are stored, and the program applications are loaded and executed by a processor to implement the above-mentioned method for target tracking based on apparent features and depth features.

In a fourth aspect of the present invention, a processing device is proposed, including a processor and a storage device; the processor is adapted to execute various programs; the storage device is adapted to store multiple programs; the programs are adapted to be loaded by the processor And execute to realize the above-mentioned target tracking method based on apparent feature and depth feature.

Beneficial effects of the present invention:

The invention extracts the depth information of the target scene and improves the accuracy of the target tracking. The present invention integrates the correlation filtering into the convolutional neural network, so that the learned convolution features can be closely coupled with the correlation filtering, so as to be more suitable for the target tracking task. Since the correlation filtering is derived in the frequency domain and maintains high efficiency, it can ensure that the algorithm can greatly improve the tracking effect under the premise of real-time tracking.

Moreover, the present invention provides a supplementary feature for a single feature that cannot well express the attributes of the target by fusing the depth feature with the apparent feature. Since the depth feature is extracted from the entire frame of the target scene, it has The global information includes depth information that is not available in the apparent features, which solves the problems of partial occlusion and deformation of the target, and makes the tracking algorithm more robust.

Description of drawings

Other features, objects and advantages of the present application will become more apparent upon reading the detailed description of non-limiting embodiments taken with reference to the following drawings.

1 is a schematic flowchart of a target tracking method based on apparent features and depth features according to an embodiment of the present invention;

2 is a schematic diagram of a framework of a target tracking method based on apparent features and depth features according to an embodiment of the present invention;

3 is a schematic frame diagram of a training process of a target tracking method based on apparent features and depth features according to an embodiment of the present invention;

FIG. 4 is an example diagram of a practical application of a target tracking method based on an apparent feature and a depth feature according to an embodiment of the present invention.

Detailed ways

In order to make the objectives, technical solutions and advantages of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not All examples. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

The present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the related invention, but not to limit the invention. In addition, it should be noted that, for the convenience of description, only the parts related to the related invention are shown in the drawings.

It should be noted that the embodiments in the present application and the features of the embodiments may be combined with each other in the case of no conflict.

The target tracking method based on apparent features and depth features of the present invention includes the following steps:

Step S10, according to the target position of the t-1 frame and the preset target size, obtain the area of the target to be tracked in the t-th frame image, and use this area as the target area; and based on the target position of the t-1 frame. The center point obtains an area N times the size of the target area, and uses it as the search area;

Step S20, through the apparent feature extraction network and the depth feature extraction network, respectively extract the target area, the apparent feature and the depth feature corresponding to the search area;

Step S30, based on a preset weight, perform a weighted average on the corresponding apparent features and depth features of the target area and the search area, respectively, to obtain a fusion feature of the target area and a fusion feature of the search area;

Step S40, according to the fusion feature of the target area and the fusion feature of the search area, obtain the response map of the target to be tracked through a correlation filter; take the position corresponding to the peak value of the response map as the target of the t-th frame Location;

in,

The apparent feature extraction network is constructed based on a convolutional neural network, and is used to obtain its corresponding apparent feature according to the input image;

The deep feature extraction network is constructed based on the ResNet network, and is used to obtain its corresponding deep features according to the input image.

In order to describe the target tracking method based on appearance features and depth features of the present invention more clearly, each step in an embodiment of the method of the present invention will be described in detail below with reference to FIG. 1 .

In the present invention, a computer with a 2.8G Hz central processing unit and a 1G-byte memory is used, the training process of the network is implemented under the Pytorch framework, and the training and testing processes of the entire network are processed in parallel by multiple NVIDIA TITAN XPGPUs, And the working program of the whole target tracking technology is compiled with python language, and the method of the present invention is realized.

In the preferred embodiment below, the apparent feature extraction network, the deep feature extraction network, and the correlation filter are first described in detail, and then the apparent feature extraction network, the deep feature extraction network, and the correlation filter are used to obtain the position of the target to be tracked. The object tracking method based on apparent features and deep features is described in detail.

1. Training of apparent feature extraction network, deep feature extraction network and correlation filter

Step A1, build a training dataset

In the present invention, the data of the training set comes from the OTB100 data set, which includes 100 videos annotated frame by frame, 11 kinds of target apparent change attributes, and 2 kinds of evaluation indicators. The 11 attributes are: illumination change, scale change, occlusion, non-rigid deformation, motion blur, fast motion, horizontal rotation, vertical rotation, out of view, background clutter, and low resolution.

The two evaluation indicators are the center location error (CLE) and the overlap rate of the rectangular box (Center location error, CLE). For the first evaluation index based on the position error of the center point, that is, the accuracy map, it is defined as the average Euclidean distance between the center position of the tracked target and the center point of the manually marked rectangular frame. The mathematical expression is as follows: (1) shows:

Among them, (x _g , y _g ) represents the position (ground truth) of the artificially labeled rectangular box, and (x _p , y _p ) is the predicted target position in the current frame.

If δ _gp is less than the given threshold, then the result of this frame is considered to be successful. In the accuracy map, the threshold for δ _gp is set to 20 pixels. The accuracy map does not give a comparison of the estimated object size and shape, because the center position error quantifies the difference in pixels. Therefore, we often use more robust success rate graphs to evaluate algorithms. For the second evaluation criterion based on the overlap rate, that is, the success rate map: Assuming that the predicted rectangular frame is r _t and the manually marked rectangular frame is _ra , then the calculation formula of the overlap ratio (Overlap Score, OS) is as follows: (2) shows:

S＝|r _t ∩r _a |/|r _t ∪r _a | (2)

Among them, ∪ and ∩ represent the union and intersection of two regions, respectively, and |·| represents the number of pixels in the region. The OS is used to determine if the tracking algorithm has successfully tracked the target in the current video frame. Frames with OS scores greater than the threshold are referred to as frames that successfully track the target. In the success rate graph, the threshold varies between 0-1, so the resulting graph is a graph of variation. We use the area under the curve of the accuracy plot and success rate plot to represent the performance of the algorithm.

Step A2, offline training apparent feature extraction network

获取期望的响应

在真实位置取得最高响应。其目标函数求解如公式(3)(4)(5)所示：First prepare the training data, the VID dataset in ImageNet, which contains more than 3000 videos. Secondly, the network structure is designed. Since the real-time performance of target tracking is also a very important indicator for evaluating the algorithm, the present invention designs a lightweight network. We use the Siamese Network as the basic structure of the network, which contains two convolution layers in total. The size of the data is 125*125, and the max pooling layer and the Relu activation function are connected after each layer of convolution. On this basis, a correlation filtering layer is added, and the back-propagation of the network is derived. The whole process can be described as: Given the characteristics of the search area

get the expected response

Get the highest response in the real location. The objective function solution is shown in formula (3)(4)(5):

为学习到的相关滤波器，⊙为矩阵点乘，

为提取到的搜索区域特征，z为搜索区域，

为相关滤波器，

为傅里叶逆变换，

为标准高斯响应的离散傅里叶变换的复共轭数，k为当前滤波器，

为目标区域的特征，

为目标区域特征的复共轭，λ为正则化系数，·^*表示取复共轭数。Among them, θ is the network parameter, y is the standard Gaussian response, γ is the regularization coefficient, L(θ) is the target loss function, D is the total number of video frames, l is the filter channel,

is the learned correlation filter, ⊙ is the matrix dot product,

is the extracted search area feature, z is the search area,

is the correlation filter,

is the inverse Fourier transform,

is the complex conjugate number of the discrete Fourier transform of the standard Gaussian response, k is the current filter,

is the feature of the target area,

is the complex conjugate of the feature of the target area, λ is the regularization coefficient, and ^* indicates the number of complex conjugates.

The objective function should contain the regularization shown, otherwise, the objective will get a non-convergence condition. This regularization is implied in conventional parameter optimization using a weight decay method. Furthermore, to limit the size of the feature map values and increase the stability of the training process, a Local Response Normalization (LRN) layer is added at the end of the convolutional layer. And based on the deep learning framework Pytorch, the detection branch and the learning branch are back-propagated. When the error is back-propagated to the real-valued feature map, the rest of the back-propagation can be conducted as a traditional CNN optimization. Since all operations of backpropagation in the correlation filter layer are still Hadamard operations in the Fourier frequency domain, it is possible to keep the (correlation filter) DCF efficient and apply offline training to large-scale datasets. After the offline training is completed, a specific feature extractor will be obtained for the online gender-related filter tracking algorithm.

Step A3, train deep feature network

Given a single image I at test time, the goal is to learn a function f that can predict the depth of the scene at each pixel, as in Equation (6):

为深度信息。in,

for depth information.

同样，也可以使用给定的右图像估计出左图像，

假设图像被校正，d就对应于图像视差，也就是模型将学会预测的每像素的标量值。这样将只需要单个左图像作为卷积层的输入，而右图像仅在训练期间使用。使用这种新颖的左右一致性损失实现两个视差图之间的一致性可以得到更准确的结果。而深度特征网络的结构由编码器和解码器的ResNet网络所组成，一共包含5个卷积层与5个反卷积层。解码器使用来自编码器激活块的跳跃式传递，使其能够解析更高分辨率的细节。关于训练的损失函数，在每个尺度f定义一个损失函数C_f，总的损失是每个尺度上的算术和，

损失模块又是三个主要的损失函数的组合，如公式(7)所示：Most existing learning-based methods treat it as a supervised learning problem. where they generally have color input images and their corresponding target depth values at training time. As an alternative, depth estimation can be treated as an image reconstruction problem during training. Specifically, we can input two images, which are left and right color images I ^l and I ^r acquired at the same time with a standard binocular camera, where l and r represent left and right. Instead of trying to predict depth information directly, try to find a correspondence ^dr that, when applied to the left image, can reconstruct the right image. Call the reconstructed image Il(dr) as

Similarly, the left image can also be estimated using the given right image,

Assuming the image is rectified, d corresponds to the image disparity, the scalar value per pixel that the model will learn to predict. This will only require a single left image as the input to the convolutional layer, and the right image will only be used during training. Using this novel left-right consistency loss to achieve consistency between two disparity maps leads to more accurate results. The structure of the deep feature network is composed of the ResNet network of the encoder and the decoder, which contains a total of 5 convolutional layers and 5 deconvolutional layers. The decoder uses skip passes from the encoder activation blocks, enabling it to resolve higher resolution details. Regarding the training loss function, a loss function C _f is defined at each scale f, and the total loss is the arithmetic sum at each scale,

The loss module is again a combination of three main loss functions, as shown in formula (7):

代表左、右图像的表观匹配损失函数，表示左、右图像的差异平滑度损失函数，

表示左、右图像的左右差异一致性损失函数，α_ap、α_ds、α_lr为三个损失函数的权重系数。每个主要的损失函数都包含左图像变体和右图像变体，但仅左图像输入到卷积层。where C _ap is the apparent matching loss function, indicating the similarity between the reconstructed image and the corresponding training input, C _ds is the difference smoothness loss function, C _lr is the left-right difference consistency loss function,

represents the apparent matching loss function of the left and right images, represents the difference smoothness loss function of the left and right images,

Represents the left and right difference consistency loss functions of the left and right images, and α _ap , α _ds , and α _lr are the weight coefficients of the three loss functions. Each of the main loss functions contains left and right image variants, but only the left image is input to the convolutional layer.

Step A4, training correlation filters

As shown in Figure 3, based on the training set constructed in step A1, the apparent features and depth features corresponding to the dataset are obtained through the apparent feature extraction network trained in step A2 and the deep feature extraction network trained in step A3.

In order to increase the robustness of the tracking performance and prevent the target from being interfered by the background, in Figure 3, the target template is the target area, which is obtained according to the target position of the t-1 frame and the preset target size, and selects two times the target area around the target area. The range is used as the search area, which is input into the apparent feature extraction network. The purpose of this is to add more background information, prevent target drift, and increase the discriminativeness of the model. Likewise, the deep features of the target template and the search region are extracted separately in the deep feature extraction network. In order to obtain more depth features, in the first frame of the video, the entire image is first input into the depth feature extraction network, the depth information of the entire image is extracted, and then the depth information is cropped at the target area, which is the depth we need. feature.

Since the dimensions of the extracted apparent features and depth features do not match, the apparent features are 32-dimensional, while the depth features are 1-dimensional. Therefore, in order to avoid the depth feature pairing algorithm due to the large difference in dimensions during the feature fusion process The influence of is weakened, and we introduce a weight coefficient α to fuse the features. The fusion process is shown in formula (8):

为第k帧提取的表观特征，为第k帧提取的深度特征，ψ(x)_k表示第k帧融合特征。in,

The apparent features extracted for the kth frame, is the depth feature extracted for the kth frame, ψ(x) _k represents the kth frame fusion feature.

训练出相关滤波器模板。其中，f(x_i)为估计出的目标响应，y_i为标准高斯响应，W为相关滤波器。The fusion feature is cyclically shifted to generate positive and negative samples, and the formula is

The relevant filter templates are trained. Among them, f(x _i ) is the estimated target response, y _i is the standard Gaussian response, and W is the correlation filter.

For training the correlation filter, the optimized formula for the difference between the predicted response map and the ideal response map is given by Equation (9):

为目标区域融合之后的特征，

为搜索区域融合之后的特征，

为目标区域融合之后的特征的复共轭。in,

is the feature after fusion of the target region,

is the feature after search area fusion,

is the complex conjugate of the feature after fusion of the target region.

When a new frame arrives, in order to detect the target, the apparent features and depth features are first extracted from the target region estimated in the previous frame, and then they are combined together to obtain a unified feature. With the help of the correlation filter layer, the response map of the template over the search area can be calculated by formula (10):

为相关滤波器的复共轭。in,

is the complex conjugate of the correlation filter.

The position of the object in the current video frame is then obtained by the maximum value on the response map. To ensure the robustness of the proposed model, the filter h is updated with a predefined learning rate η, as shown in Equation (11):

为第k+1帧的相关滤波器，

为第k帧的相关滤波器。in,

is the correlation filter of the k+1th frame,

is the correlation filter of the kth frame.

其中，a为尺度系数，s为尺度池，S为要取的尺度数。比如我们算法中采用3个尺度，就是S＝3，我们就可以得出尺度池s＝(-1,0,1)，然后a的s次方就是我们要取的尺度，即(0.97,1,1.03)，然后把这个尺度乘到目标区域上，0.97意思就是范围缩小0.97倍，目标就会变小，1.03就是放大，然后进行滤波，响应值最大的就是我们要的尺度。As for the scale change of the target, we use pyramid image patches with scale factors for scale filtering

Among them, a is the scale coefficient, s is the scale pool, and S is the number of scales to be taken. For example, we use 3 scales in our algorithm, that is, S=3, we can get the scale pool s=(-1,0,1), and then the s power of a is the scale we want to take, that is (0.97,1 , 1.03), and then multiply this scale to the target area, 0.97 means that the range is reduced by 0.97 times, the target will become smaller, 1.03 is to enlarge, and then filter, the largest response value is the scale we want.

The obtained target response map corresponds to the probability value that the point is the target. Therefore, the largest point in the response map is selected as the estimated target position, and the motion model is updated when the target position in a new frame is obtained. During online tracking, the filter is only updated over time. The optimization problem of the objective can be expressed in incremental mode, as shown in Equation (12):

为第p个样本的相关滤波器，为第t个样本的特征。Among them, ε is the output loss, p is the p-th sample, t is the current sample, β _t is the impact factor of the current sample,

is the correlation filter of the p-th sample, is the feature of the t-th sample.

The parameter β _t > 0 is the influence factor of the sample x _t . At the same time, the closed-form solution in the equation can also be extended to the time series, as shown in formula (13):

为第p个样本的相关滤波器，

为样本x_t的特征，

为第k个滤波器得到的样本x_t的特征，

为第k个滤波器得到的样本x_t的特征的复共轭。in,

is the correlation filter of the p-th sample,

is the feature of the sample _xt ,

is the feature of the sample x _t obtained by the kth filter,

is the complex conjugate of the features of the sample _xt obtained for the kth filter.

The advantage of this incremental update is that we don't need to save a large sample set, so it only needs to take up very little space.

2. Target tracking method based on apparent features and depth features

A target tracking method based on appearance features and depth features according to an embodiment of the present invention includes the following steps:

Step S10, according to the target position of the t-1 frame and the preset target size, obtain the area of the target to be tracked in the t-th frame image, and use this area as the target area; and based on the target position of the t-1 frame. The center point acquires an area N times the size of the target area, and takes it as the search area.

In this embodiment, the target position and target size are given by a rectangular frame in the first frame of the video. In the case of other frames, the position area of the target in the current frame image will be obtained according to the target position of the previous frame and the preset target size, which will be used as the target area, and the target area of the previous frame will be obtained as the center. An area twice the size of the target area is used as the search area. This increases the robustness of the tracking performance and prevents the target from being disturbed by the background.

The target size in this embodiment is set according to the actual application.

Step S20 , extract the target area, the apparent feature and the depth feature corresponding to the search area through an apparent feature extraction network and a depth feature extraction network, respectively.

In this embodiment, the target area and the search area obtained in step S10 are respectively input into the apparent feature extraction network and the depth feature extraction network to obtain their corresponding apparent features and depth features.

When extracting the depth feature of the first frame image of the video, first input the whole image into the depth feature extraction network, extract the depth information of the whole image, and then crop the depth information at the target position, that is, the depth feature we need.

Among them, the apparent feature contains a total of 32 dimensions, the size is 125*125*32, and the size of the depth feature is limited to 125*125*1.

Step S30, based on a preset weight, weighted average of the apparent features and depth features corresponding to the target area and the search area, respectively, to obtain a fusion feature of the target area and a fusion feature of the search area.

In this embodiment, since the dimensions of the extracted apparent features and depth features do not match, the apparent features are 32-dimensional, while the depth features are 1-dimensional. Therefore, in order to avoid a large difference in dimensions during the feature fusion process As a result, the influence of depth features on the algorithm is weakened. We introduce a weight coefficient to fuse the features, and obtain the fusion features of the target area and the search area respectively.

Step S40, according to the fusion feature of the target area and the fusion feature of the search area, obtain the response map of the target to be tracked through a correlation filter; take the position corresponding to the peak value of the response map as the target of the t-th frame Location.

In this embodiment, based on the fusion feature of the target area and the search area, a trained correlation filter is used to obtain a response graph, and the position of the target to be tracked in the current video frame is obtained by the maximum value on the response graph.

After obtaining the target position of the current frame, the relevant filter is updated based on the state value of the filter in the previous frame and the preset learning rate.

Step S10-Step S40, can refer to Fig. 4, wherein based on the target area image (Current image) and the search area image (Search area image) through the apparent feature extraction network (ANET) and the depth feature extraction network (DNET), respectively, to obtain the table. The visual features and depth features are weighted and averaged based on the preset weight α to obtain the respective fusion features of the current image and the search area image, and the response map is obtained based on the correlation filter (DCF). value to get the position of the target.

A target tracking system based on apparent features and depth features according to the second embodiment of the present invention, as shown in FIG. 2 , includes: a region acquisition module 100, a feature extraction module 200, a feature fusion module 300, and an output position module 400;

The acquisition area module 100 is configured to acquire the area of the target to be tracked in the t-th frame image according to the target position of the t-1 frame and the preset target size, and use this area as the target area; and based on the t-1 frame The center point of the target position obtains an area N times the size of the target area, and takes it as the search area;

The feature extraction module 200 is configured to extract the target area, the apparent feature and the depth feature corresponding to the search area respectively through the apparent feature extraction network and the depth feature extraction network;

The feature fusion module 300 is configured to perform a weighted average of the apparent features and depth features corresponding to the target area and the search area, respectively, based on a preset weight, to obtain a fusion feature of the target area and a fusion feature of the search area;

The output position module 400 is configured to obtain a response map of the target to be tracked through a correlation filter according to the fusion feature of the target area and the fusion feature of the search area; take the position corresponding to the peak value of the response map as the first the target position of the t frame;

in,

The apparent feature extraction network is constructed based on a convolutional neural network, and is used to obtain its corresponding apparent feature according to the input image;

The deep feature extraction network is constructed based on the ResNet network, and is used to obtain its corresponding deep features according to the input image.

Those skilled in the technical field can clearly understand that, for the convenience and brevity of description, for the specific working process and related description of the system described above, reference may be made to the corresponding process in the foregoing method embodiments, which will not be repeated here.

It should be noted that, the target tracking system based on apparent features and depth features provided in the above embodiments is only illustrated by the division of the above functional modules. In practical applications, the above functions can be allocated by different The modules or steps in the embodiments of the present invention are further decomposed or combined. For example, the modules in the above-mentioned embodiments can be combined into one module, or can be further split into multiple sub-modules to complete all the above-described or some functions. The names of the modules and steps involved in the embodiments of the present invention are only for distinguishing each module or step, and should not be regarded as an improper limitation of the present invention.

A storage device according to the third embodiment of the present invention stores a plurality of programs, and the programs are suitable for being loaded by a processor and implementing the above-mentioned target tracking method based on appearance features and depth features.

A processing device according to a fourth embodiment of the present invention includes a processor and a storage device; the processor is adapted to execute various programs; the storage device is adapted to store multiple programs; the programs are adapted to be loaded and executed by the processor In order to realize the above-mentioned target tracking method based on appearance features and depth features.

Those skilled in the technical field can clearly understand that the undescribed convenience and brevity are not described. Repeat.

Those skilled in the art should be aware that the modules and method steps of each example described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, computer software or a combination of the two, and the programs corresponding to the software modules and method steps Can be placed in random access memory (RAM), internal memory, read only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or as known in the art in any other form of storage medium. In order to clearly illustrate the interchangeability of electronic hardware and software, the components and steps of each example have been described generally in terms of functionality in the foregoing description. Whether these functions are performed in electronic hardware or software depends on the specific application and design constraints of the technical solution. Skilled artisans may use different methods of implementing the described functionality for each particular application, but such implementations should not be considered beyond the scope of the present invention.

The terms "first," "second," etc. are used to distinguish between similar objects, and are not used to describe or indicate a particular order or sequence.

The term "comprising" or any other similar term is intended to encompass a non-exclusive inclusion such that a process, method, article or device/means comprising a list of elements includes not only those elements but also other elements not expressly listed, or Also included are elements inherent to these processes, methods, articles or devices/devices.

So far, the technical solutions of the present invention have been described with reference to the preferred embodiments shown in the accompanying drawings, however, those skilled in the art can easily understand that the protection scope of the present invention is obviously not limited to these specific embodiments. Without departing from the principle of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will fall within the protection scope of the present invention.

Claims (10)

1. A method for object tracking based on appearance features and depth features, the method comprising:

step S10, acquiring the area of the target to be tracked in the t frame image according to the target position of the t-1 frame and the preset target size, and taking the area as the target area; acquiring a region with the size N times that of the target region based on the central point of the target position of the t-1 frame, and taking the region as a search region;

step S20, respectively extracting the apparent features and the depth features corresponding to the target area and the search area through an apparent feature extraction network and a depth feature extraction network;

step S30, respectively carrying out weighted average on the apparent features and the depth features corresponding to the target region and the search region based on preset weights to obtain fusion features of the target region and the search region;

step S40, obtaining a response graph of the target to be tracked through a relevant filter according to the fusion characteristics of the target area and the fusion characteristics of the search area; taking the position corresponding to the peak value of the response image as the target position of the t frame;

wherein,

the apparent feature extraction network is constructed based on a convolutional neural network and is used for acquiring corresponding apparent features according to an input image;

the depth feature extraction network is constructed based on a ResNet network and is used for acquiring the corresponding depth features according to the input image.

2. The method for tracking the target according to claim 1, wherein in step S10, "acquiring the region of the target to be tracked in the image of the t-th frame according to the target position of the t-1 frame and the preset target size, and using the region as the target region", the method comprises: if t is equal to 1, acquiring a target area of a target to be tracked according to a preset target position and a preset target size; and if t is larger than 1, acquiring a target area of the target to be tracked according to the target position of the t-1 frame and the preset target size.

3. The method for tracking an object based on the appearance features and the depth features according to claim 1, wherein the appearance feature extraction network has a structure that: the filter comprises two convolutional layers and a relevant filter layer, wherein a maximum pooling layer and a ReLU activation function are connected behind each convolutional layer; the network is trained in a training process by adopting a back propagation algorithm.

4. The method for tracking the target based on the appearance feature and the depth feature of claim 1, wherein the depth feature extraction network has a structure that: 5 convolutional layers, 5 deconvolution layers; the depth feature extraction network is trained through mutual reconstruction of binocular images in the training process.

5. The target tracking method based on the appearance features and the depth features according to claim 2, wherein in the process of extracting the depth features, if t is equal to 1, the extraction method of the depth feature extraction network is as follows:

acquiring the depth feature of the first frame image based on a depth feature extraction network;

and acquiring the depth features of the target area and the search area based on the depth feature of the first frame image and the preset target position.

6. The object tracking method based on the appearance feature and the depth feature of claim 1, wherein the correlation filter obtains different scales by a scale transformation method when filtering the object region, and enlarges or reduces the object region according to the different scales and then filters the object region. The scale transformation method comprises the following steps:

wherein a is a scale coefficient, S is a scale pool, S is a preset scale degree, a^sIs a scale.

7. The method for tracking an object based on an apparent feature and a depth feature according to any one of claims 1 to 6, further comprising updating the state values of the correlation filters after step S40 by:

acquiring the state value of a correlation filter in a t-1 frame;

and updating the state value of the t frame of the relevant filter based on the state value, the target position of the t frame and a preset learning rate.

8. A target tracking system based on appearance features and depth features is characterized by comprising an acquisition region module, a feature extraction module, a feature fusion module and an output position module;

the acquisition region module is configured to acquire a region of a target to be tracked in a t-th frame image according to the target position of the t-1 frame and a preset target size, and the region is used as a target region; acquiring a region with the size N times that of the target region based on the central point of the target position of the t-1 frame, and taking the region as a search region;

the feature extraction module is configured to extract the apparent features and the depth features corresponding to the target area and the search area respectively through an apparent feature extraction network and a depth feature extraction network;

the feature fusion module is configured to perform weighted average on the target region and the apparent feature and the depth feature corresponding to the search region respectively based on preset weights to obtain a fusion feature of the target region and a fusion feature of the search region;

the output position module is configured to obtain a response map of the target to be tracked through a relevant filter according to the fusion feature of the target area and the fusion feature of the search area; taking the position corresponding to the peak value of the response image as the target position of the t frame;

wherein,

the apparent feature extraction network is constructed based on a convolutional neural network and is used for acquiring corresponding apparent features according to an input image;

the depth feature extraction network is constructed based on a ResNet network and is used for acquiring the corresponding depth features according to the input image.

9. A storage device having stored therein a plurality of programs, wherein said program applications are loaded and executed by a processor to implement the apparent feature and depth feature based object tracking method of any of claims 1-7.

10. A processing device comprising a processor, a storage device; a processor adapted to execute various programs; a storage device adapted to store a plurality of programs; characterized in that the program is adapted to be loaded and executed by a processor to implement the apparent and depth feature based object tracking method of any of claims 1-7.

Images