CN116681728A - Multi-target tracking method and system based on Transformer and graph embedding - Google Patents

Abstract

The invention discloses a multi-target tracking method based on a transform and graph embedding, which comprises the following steps: acquiring a video sequence, and reading the video sequence frame by frame to acquire all frames in the video sequence; setting a counter cnt 1=1; judging whether the cnt1 is equal to the total number of frames in the video sequence, if not, inputting the cnt1 frame and the cnt1+1 frame in the video sequence into a multi-target tracking model trained in advance to obtain an allocation matrix between a target in the cnt1 frame and a corresponding target in the cnt1+1 frame, wherein the cnt1 frame is a previous frame, and the cnt+1 frame is a subsequent frame; and acquiring all targets associated with each target in the cnt1 frame in the cnt+1 frame according to the obtained allocation matrix between the target in the cnt1 frame and the corresponding target in the cnt1+1 frame, and forming the tracking track of the target by the target in the cnt1 frame and all the obtained targets. The method can solve the technical problems that the global correlation learning adopted by the existing transform-based multi-target tracking method not only increases the computational complexity, but also has excessive redundancy calculation.

Description

A multi-target tracking method and system based on Transformer and graph embedding

technical field

The invention belongs to the technical field of pattern recognition, and more particularly relates to a multi-target tracking method and system based on Transformer and graph embedding.

Background technique

Multi-Object Tracking (MOT) is a hot research topic in the field of computer vision, and has a wide range of applications in the fields of automatic driving, video surveillance, intelligent robots, etc. The position of the target of interest, and retrieve the trajectory of the same identity, involves two tasks of target detection and identification.

At present, with the rise of the Transformer network, the Transformer-based multi-target tracking method has been widely used, which uses the sparse object query in the Transformer-based detector (Detection Transformer, referred to as DETR) architecture method to detect new objects and initialize tracking. Trajectory, using tracking query to maintain the information of different trajectories across frames, and realize the trajectory association between multiple frames. The common idea behind these works is to expand existing trajectories frame-by-frame using the object query mechanism in DETR, which enables migration propagation of objects in the temporal domain.

However, the above-mentioned existing Transformer-based multi-target tracking methods have some defects that cannot be ignored:

First, the success of the Transformer model in the visual field is due to its global correlation modeling feature based on the attention mechanism, but this also leads to the inherent defects of the Transformer architecture itself. The global correlation learning not only increases the computational complexity, but also has excessive redundant calculation;

Second, the Transformer integrated tracking model is derived from the detection task, emphasizing the correlation of scene structure in modeling long-term spatial domain span, the model learning is free, the learning process lacks interpretability, and there is no clear priority for neighborhood local information learning ;

Third, the Transformer integrated MOT framework adopts a method based on query ranking to indirectly realize the identity association between the tracking target and the detection response. The correlation output of the query target does not consider the similarity results of other query targets and detection targets, that is, there is no Clarify the optimal reasoning of multi-target data association, so the association results are often suboptimal, which will lead to the distortion of target information in difficult and complex scenes, and the output results based on encoding query sorting are not credible, which in turn leads to identity switching in the tracking process (Identity Switches, referred to as IDS) indicators are relatively poor.

Contents of the invention

In view of the above defects or improvement needs of the prior art, the present invention provides a multi-target tracking method and system based on Transformer and graph embedding, the purpose of which is to solve the global correlation learning adopted by the existing Transformer-based multi-target tracking method The technical problems of high computational complexity, excessive redundant calculations, lack of interpretability in the learning process, no clear priority for neighborhood local information learning, and poor IDS indicators in the tracking process. .

To achieve the above object, according to one aspect of the present invention, a multi-target tracking method based on Transformer and graph embedding is provided, including:

(1) Obtain a video sequence, and read the video sequence frame by frame to obtain all frames therein;

(2) setting counter cnt1=1;

(3) judge whether cnt1 has been equal to the total number of frames in the video sequence, if so the process ends, otherwise enter step (4);

(4) Input the cnt1th frame and the cnt1+1th frame in the video sequence into the pre-trained multi-target tracking model to obtain the distribution between the target in the cnt1th frame and the corresponding target in the cnt1+1th frame Matrix, where the cntth frame is the previous frame, and the cnt+1th frame is the next frame;

(5) According to the allocation matrix between the target in the cnt1 frame obtained in step (4) and the corresponding target in the cnt1+1 frame, obtain all targets associated with each target in the cnt+1 frame, Constitute the target in the cnt frame together with all the obtained targets to track the target;

(6) Set the counter cnt1=cnt1+1, and return to step (3).

Preferably, the multi-target tracking model includes a benchmark deep visual feature extraction network, a Transformer encoder, a Transformer decoder, a graph convolutional network GCN, and an association network connected in sequence.

Preferably, the first layer is a benchmark deep visual feature extraction network, which includes 1 convolutional layer, 16 building block structures and 1 fully connected layer, and the input of the benchmark deep visual feature extraction network is the previous frame and the next frame, The output is the reference depth visual feature corresponding to the previous frame with a dimension of h×w×C and the reference depth corresponding to the next frame with a dimension of h×w×C, where h represents the height of the visual feature of the reference depth, and w represents The height of the reference depth visual feature, C represents the number of channels of the reference depth visual feature, and C=2048.

The second layer is the Transformer encoder, whose input is the reference depth visual feature obtained through the reference depth visual feature extraction network of the previous frame and the next frame, and is encoded by the Transformer of the L layer, and the output is corresponding to the previous frame, with a dimension of 256 The global feature of dimension and the global feature of dimension 256 corresponding to the next frame, where the value range of L is 3 to 10.

The third layer is the Transformer decoder. For the previous frame, its input is the object query and the global features obtained by the Transformer encoder. First, N output embeddings are obtained through the mutual attention mechanism, and then N output embeddings are obtained through the feedforward neural network. Frame coordinates and class labels; for the next frame, the input is the result of the object query and the N output embeddings obtained in the previous frame and the global features obtained in the second layer. First, through the mutual attention mechanism, get M output embeddings and then get M box coordinates and class labels through the feedforward neural network;

The fourth layer is a graph convolutional network, whose input is the N output embeddings of the previous frame and the M output embeddings of the next frame obtained by the third layer. First, by determining the neighbor relationship, the previous frames obtained respectively are concatenated. The output embedding of the concatenated output of the previous frame and the output embedding of the next frame concatenated, and then iteratively process the output embedding of the concatenated previous frame and the output embedding of the concatenated next frame through the graph convolutional network, and output the former The local correlation enhanced features of one frame and the local correlation enhanced features of the next frame;

The fifth layer is the association network, whose input is the local correlation enhanced feature of the previous frame and the local correlation enhanced feature of the next frame obtained by the fourth layer, and the two are processed based on the correlation between the features to obtain the previous frame. One frame and the next frame correspond to the assignment matrix between objects.

Preferably, the multi-target tracking model is trained through the following steps:

(4-1) Obtain the MOT17 data set and the CrowdHuman image database, divide the obtained MOT17 data set into the first half and the second half according to the proportion, and use the first half of the obtained MOT17 data set and the entire CrowdHuman image database as a training set , using the second half of the resulting MOT17 dataset as the test set.

(4-2) setting counter cnt3=1;

(4-3) counter cnt2=1 is set, judge whether cnt3 is equal to the total number of video sequences in the training set, if enter step (4-16), otherwise enter step (4-4);

(4-4) judge whether cnt2 is equal to the total number of frames in the cnt3 video sequence in the training set, if it is then proceed to step (4-14), otherwise enter step (4-5);

(4-5) Initialize N _obj learnable object queries (objectqueries) as the object queries of the previous frame and the next frame respectively, wherein N _obj =500;

(4-6) Use the cnt2 frame in the cnt3 video sequence in the training set as the previous frame, use the cnt2+1 frame as the next frame, and input the previous frame and the next frame into the benchmark depth visual feature extraction network respectively , to obtain the reference depth visual feature I ₁ corresponding to the previous frame with a dimension of 2048, and the reference depth visual feature I ₂ with a dimension of 2048 corresponding to the next frame.

(4-7) For the previous frame and the next frame in the cnt3th video sequence in the training set obtained in step (4-6), the reference depth corresponding to the previous frame obtained in step (4-6) The visual feature I ₁ and the reference depth visual feature I ₂ corresponding to the next frame are used as the query, key and value of the Transformer encoder, and are respectively input to the multi-head self-attention layer of the Transformer encoder to obtain feature vectors with a dimension of 256 , respectively normalize the 256-dimensional feature vectors to obtain the normalized feature vectors respectively, and input the normalized features into the feedforward neural network of the Transformer encoder and normalize them respectively to obtain the normalized feature vectors respectively A 512-dimensional global feature F ₁ corresponding to the previous frame and a 512-dimensional global feature F ₂ corresponding to the next frame are obtained.

(4-8) For the previous frame in the cnt3th video sequence in the training set obtained in step (4-6), input the object query of the previous frame to the multi-head self-attention layer and the layer normalization layer to obtain For an object query with a dimension of 256, the global feature F ₁ corresponding to the previous frame obtained in step (4-7) is used as the key (Key) and value (Value) of the Transformer decoder, and the 256-dimensional object query is used as a query, input Go to the multi-head mutual attention layer of the Transformer decoder and normalize to obtain the output embedding of the N previous frames, and then embed the output into the feedforward neural network and normalize to obtain a 512-dimensional feature vector, and convert the 512 The feature vector of dimension is input into the feed-forward neural network to obtain N tracking trajectories of the previous frame, each tracking track includes frame coordinates and class labels;

(4-9) For the next frame in the cnt3th video sequence in the training set obtained in step (4-6), the tracking query and object query of the latter frame are first cascaded and then input to the multi-head self-attention layer and layer normalization layer to obtain a cascaded query with a dimension of 256 dimensions, and use the cascaded query as a query, and the global feature F ₂ obtained in steps (4-7) as a key (Key) and value (Value) to be input to the Transformer The multi-head mutual attention layer of the decoder is normalized to obtain M output embeddings of the next frame, and the output embeddings are input to the feedforward neural network and normalized to obtain a 512-dimensional feature vector, and the 512-dimensional The feature vector is input into the feedforward neural network to obtain M detection targets in the next frame, each detection target includes frame coordinates and class labels;

(4-10) Establish graph models G 1 , G ₂ for the N tracking trajectories obtained in step (4-8) and the M detection targets obtained in step ( _4-9 );

(4-11) Input the graph models G ₁ and G ₂ of the previous frame and the next frame obtained in step (4-10) into the graph convolutional network respectively, and by increasing the number of iterations of the graph convolutional network, the former The local correlation enhanced feature E ₁ of one frame and the local correlation enhanced feature E ₂ of the next frame.

(4-12) Input the local correlation enhancement feature E ₁ of the previous frame and the local correlation enhancement feature E ₂ of the next frame obtained in step (4-11) into the association network (as shown in Figure 8), and perform feature-based Correlation processing between to obtain the allocation matrix between the tracking track of the previous frame and the detection target of the next frame;

(4-13) Use the N frame coordinates and class labels obtained in step (4-8), and the M frame coordinates and class labels obtained in step (4-9) to Transformer encoder, Transformer decoder, graph convolution Network and associated network for training, according to the previous frame and the next frame in the training set obtained by the Transformer encoder, Transformer decoder and GCN for step (4-6), the frame coordinates, class labels and local Correlation enhancement features and the frame coordinates, class labels, and local correlation enhancement features of the detection target in the next frame, the distribution matrix between the tracking trajectory of the previous frame and the detection target of the next frame is obtained according to the association network, and the frame coordinates are obtained by , class label and assignment matrix are input into the defined loss function to get detection loss, tracking loss and focal loss;

(4-14) cnt2=cnt2+1, and return to step (4-4);

(4-15) cnt3=cnt3+1, and return to step (4-3);

(4-16) According to the detection loss, tracking loss and focal loss obtained in step (4-13), and use the backpropagation method to iteratively train the multi-target tracking model until the multi-target tracking model converges, thereby obtaining a preliminary A trained multi-object tracking model.

(4-17) Use the test set obtained in step (4-1) to verify the multi-target tracking model preliminarily trained in step (4-16) until the obtained tracking accuracy reaches the optimal level, so as to obtain the multi-target tracking model trained Object tracking model.

Preferably, the step (4-10) is specifically, firstly determine the vertex set, the vertex set V1=(v1, v2...vN) is composed of all tracking tracks in the previous frame, the vertex set V2=(v1, v2...vM) It is composed of all detection targets in the next frame, and the vertex values are the output embedding of the corresponding tracking trajectory of the previous frame and the output embedding of the detection target of the next frame, and then determine the neighbors of all vertices in the vertex sets V1 and V2 respectively Nodes form edges, and finally graph models G ₁ and G ₂ are obtained.

Preferably, the process of determining the neighbor nodes of all vertices in V1 and V2 in step (4-10) is as follows, first obtain the frame coordinates and output embeddings corresponding to all vertices in the V1 set, and then obtain the two vertices corresponding to the frame coordinates determined by V1 The intersection ratio IoU between the frame coordinates and the output embedding obtain the cosine similarity between the two vertex features, and select the weight aggregation coefficient between each vertex and all vertices except itself The largest top K as its neighbor nodes to form an edge, and /> As the weight of the edges between vertices and neighbor nodes, the process of determining the neighbor nodes of all vertices in V2 is the same as that of V1.

The weight of the edge between vertices is the clustering coefficient Represents the correlation between vertices, defined as the feature and spatial distance similarity of vertex i and vertex j:

cos(·,·) represents the cosine similarity between feature vectors, IoU(·,·) represents the intersection and union ratio IoU of two target position coordinates, l represents the number of GCN iteration layers, Indicates the vertex feature of vertex i in the l-level GCN, b _i indicates the frame coordinates of vertex i, i∈[1,M], j∈[1,N].

Preferably, the step (4-11) is specifically, first constructing the feature matrix of each vertex in G ₁ and G ₂ obtained in the step (4-10), the total number of each vertex and its neighbor nodes is Num=K+1 , the feature matrix Ft is obtained by cascading the output embedding of the vertex and the neighbor node, Ft∈R ^Num×d is the feature matrix of the vertex, where d is the feature dimension of each node, and the dimension is 256. Then, get the two-dimensional adjacency matrix Mt∈R ^Nu×Num of each vertex in _G1 and _G2 and all its neighbor nodes respectively, set the target node index as 1, then the two-dimensional adjacency matrix initialization is defined as:

Among them, i, j∈{1,2,...,Num}, Mt=1 means that there is an edge connecting vertex i and vertex j, and Num means the total number of the target node and its neighbor nodes. Then each vertex in _G1 and _G2 is input into the graph convolutional network to obtain aggregated features, and finally the aggregated features are input into the feedforward neural network to obtain the local correlation enhanced features _E1 and The local correlation of the next frame enhances the feature E ₂ .

Preferably, the step (4-12) is specifically, first, before using the association network for data association, it is necessary to construct a feature matrix, since the appearance features of each target contain high-dimensional information, in order to make full use of these rich features information, the present invention adopts the method of splicing the local correlation enhanced feature E ₁ of the previous frame and the local correlation enhanced feature E ₂ of the next frame obtained in step (4-11) to obtain the feature matrix E.

Then the feature matrix E is sent to three different 1*1 convolutional layers, and compressed to the MN×2C dimension through the Reshape operation to obtain Q∈R ^MN×2C , K∈R ^MN×2C , V∈R ^MN×2C , where C=256, perform matrix multiplication on Q and K to obtain the correlation between two tracking targets, and then obtain E _{Q, K} after Softmax operation, and then perform a matrix on E _{Q, K} and V multiplication to get a further incidence matrix.

Finally, the matrix is sent to the feed-forward network layer and through two residual connections, the tracking trajectory in the previous frame and the deeper correlation features of the detection target in the next frame are obtained, and the assignment matrix A∈RM ^×N is finally obtained.

Preferably, the detection loss, tracking loss and focal loss acquisition steps in step (4-13) are as follows, first obtain the newly detected target in the subsequent frame and the target associated with the subsequent frame to the previous frame, and T _t-1 and T _t represents the tracking trajectory of the previous frame and the detection target of the next frame respectively. When any trajectory trk∈T _t /T _t-1 , it corresponds to a new target in T _t but does not belong to T _t-1 , and then the following The frame of the newly detected target in one frame is input to the coordinates and class label to the detection loss function to obtain the detection loss; the frame coordinates and class labels of the target associated with the previous frame are input into the tracking loss function, to get the tracking loss. Subsequently, the distribution matrix between the tracking trajectory of the previous frame and the detection target of the next frame obtained in step (4-12) is input into the focal loss function to obtain the focal loss.

The detection loss function is defined as follows:

L _det ＝λ _cls L _cls +λ _L1 L _box +λ _giou L _giou

Among them, L _cls is the focal loss between the class label based on the new detection target and the true value class label of the training set, and L _box and L _giou are the L1 distance between the normalized frame coordinates and the true value box coordinates of the training set and generalized IoU and λ _cls , λ _L1 , λ _giou are the weight parameters corresponding to each part;

The tracking loss function L _trk is as follows:

L _trk ＝λ _cls L _cls +λ _L1 L _box +λ _giou L _giou

The focal loss function is equal to:

Among them, the focal loss adds a modulation coefficient γ and a balance factor α on the basis of the cross-entropy loss function. The value range of the modulation coefficient γ is 1 to 5, preferably 2, and the value range of α is 0.1-0.7, preferably 0.5. The loss of negative sample classification can be reduced, so that the loss weight pays more attention to the classification of positive samples, y=1 means a successful match, and y=0 means no match.

According to another aspect of the present invention, a multi-target tracking system based on Transformer and graph embedding is provided, including:

The first module is used to obtain a video sequence, and reads the video sequence frame by frame to obtain all frames therein;

The second module is used to set the counter cnt1=1;

The third module is used to judge whether cnt1 has been equal to the total number of frames in the video sequence, if so, the process ends, otherwise enter the fourth module;

The fourth module is used to input the cnt1th frame and the cnt1+1th frame in the video sequence into the pre-trained multi-target tracking model to obtain the target in the cnt1th frame and the corresponding target in the cnt1+1th frame The allocation matrix between, where the cntth frame is the previous frame, and the cnt+1th frame is the next frame;

The fifth module is used to obtain all the objects associated with each target in the cnt1th frame in the cnt+1th frame according to the allocation matrix between the target in the cnt1th frame obtained by the fourth module and the corresponding target in the cnt1+1th frame Target, the target in the cnt frame and all the obtained targets constitute the tracking track of the target;

The sixth module is used to set the counter cnt1=cnt1+1, and return to the third module.

Generally speaking, compared with the prior art, the above technical solutions conceived by the present invention can achieve the following beneficial effects:

(1) Since the present invention adopts steps (4-7) to steps (4-11), the GCN is embedded into the Transformer integrated MOT architecture, which enhances the effectiveness of space-time domain topology information learning of adjacent multi-targets in the coding space, and simultaneously constructs Apply the association network for linear programming to solve the defect of Transformer integrated architecture that lacks global reasoning for multi-objective identity matching during the test process. The model proposed in the present invention combines the advantages of Transformer and GCN models, and embeds the target detection, feature expression and data association reasoning in MOT into the deep network model, which greatly exerts the advantages of multi-task joint optimization learning and improves model convergence. , therefore, the present invention can solve the inherent defect of the existing Transformer model, the global correlation learning not only increases the computational complexity, but also has the problem of slow convergence caused by excessive redundant calculation.

(2) Since the present invention adopts steps (4-10) to steps (4-11), there is a specific group motion rule among multiple targets between adjacent frames, and on the one hand, there is a consistent motion between the target groups that constitute the neighbors. mode, and there is a fixed neighbor relationship. This social group feature reflects the local correlation between moving targets. As a priori information, it can provide an effective basis for target detection and tracking in complex scenes. Therefore, the present invention can solve the problem of Transformer model learning freedom and lack of learning process. Interpretability, the problem of no clear priority for neighborhood-local information learning.

(3) Since the present invention adopts steps (4-12), inspired by the "attention is all your need" model of the Transformer architecture, for the multi-target tracking data association problem, an association network is proposed to realize a linear programming network model based on a full attention mechanism In order to embed the data association formalized neural network into the integrated MOT learning model, this autonomous learning scenario is suitable for the feature expression and measurement of the data association optimization problem, and further improves the effective characteristics of the model. In this way, it can solve the problem that the target information is distorted in difficult and complex scenes, resulting in suboptimal association results.

(4) Since the association network in step (4-12) of the present invention can also be embedded in the neural network, it is possible to use the powerful learning ability of deep learning to further improve the data association according to the internal connection between the data association model and the state reasoning model. performance of models and state inference models;

(5) The present invention proposes a framework based on Transformer model and graph embedding, which combines the advantages of Transformer and GCN for multi-target tracking problems, so the backpropagation algorithm can be used to automatically solve the parameters in the Transformer model, while also It can be combined with the neural network in steps (4-11) and steps (4-12) for end-to-end learning to obtain more suitable parameters and further improve the performance of the entire model;

(6) The present invention has a wide range of applications, not only for pedestrian tracking, but also for any known type of moving target track tracking.

Description of drawings

Fig. 1 is the flow chart of the multi-target tracking method based on Transformer and graph embedding of the present invention;

Fig. 2 is the frame extracted in the step (4) of the inventive method, wherein Fig. 2 (a) is a previous frame, and Fig. 2 (b) is a subsequent frame;

Fig. 3 is the structural representation of Transformer encoder in the step (4-7) of the inventive method;

Fig. 4 is the structural representation of Transformer decoder in the step (4-8) and (4-9) of the inventive method;

Fig. 5 is the target in the frame that extracts in the step (4-8), (4-9) of the inventive method, wherein Fig. 5 (a) is the target in the previous frame, and Fig. 5 (b) is the following frame target in

Fig. 6 is the schematic diagram of constructing graph in the step (4-10) of the method of the present invention;

Fig. 7 is the structural representation of GCN in the step (4-11) of the method of the present invention;

Fig. 8 is a schematic structural diagram of an association network in step (4-12) of the method of the present invention.

Detailed ways

In order to make the object, technical solution and advantages of the present invention clearer, the present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain the present invention, not to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not constitute a conflict with each other.

The basic idea of the present invention is to propose a "Transformer+GCN" end-to-end integrated multi-target tracking framework suitable for multi-target tracking tasks. On the one hand, the GCN model is formalized into the Transformer decoder, and the GCN vertex transfer characteristics are used to explicitly guide the modeling of local correlations between adjacent target groups, so as to improve the effectiveness of the Transformer in learning the characteristics of multiple target groups in the scene. sex. On the other hand, inspired by the "attention is all your need" model of the Transformer architecture, for the multi-target tracking data association problem, an association network is proposed to implement a linear programming network model based on the full attention mechanism, so that the data association can be formalized by the neural network. The implementation form is embedded in the integrated MOT learning model, which is applicable to the feature expression and measurement problems of data association optimization problems in this autonomous learning scenario, and further improves the effective characteristics of the model.

As shown in Figure 1, the present invention provides a multi-target tracking method based on Transformer and graph embedding, including the following steps:

(1) Obtain a video sequence, and read the video sequence frame by frame to obtain all frames therein;

Specifically, this step is a certain video sequence obtained from the MOT17 dataset.

The duration of the video frame in this step is 450 to 1500 frames.

(2) setting counter cnt1=1;

(3) judge whether cnt1 has been equal to the total number of frames in the video sequence, if so the process ends, otherwise enter step (4);

(4) Input the cnt1th frame (the previous frame) and the cnt1+1th frame (the next frame) in the video sequence into the pre-trained multi-target tracking model to get the target in the cnt1th frame and the cnt1th frame Allocation matrix between corresponding targets in +1 frame;

For example, the allocation matrix obtained in this step is as follows, where the rows of the matrix represent the cnt1th frame, and the columns represent the cnt1+1th frame:

As can be seen from the above allocation matrix, the first target in the cnt1 frame is associated with the second target in the cnt1+1 frame; the second target in the cnt1 frame is associated with the first target in the cnt1+1 frame; The third object in the cnt1 frame is associated with the fourth object in the cnt1+1 frame; the fourth object in the cnt1 frame is associated with the third object in the cnt1+1 frame.

As shown in Figure 1, the multi-target tracking model of the present invention includes a benchmark depth visual feature extraction network (which is a ResNet-50 network), a Transformer encoder, a Transformer decoder, and a graph convolutional network (Graphconvolutionalnetwork, referred to as GCN) connected in sequence , and associated networks.

The first layer is the benchmark deep visual feature extraction network, which contains 1 convolutional layer, 16 building block (Building block) structures and 1 fully connected layer. The input of the benchmark deep visual feature extraction network is the previous frame and the next frame ( It is an RGB image, the width W is 750, the height H is 1333, and the number of channels is 3), the output is the reference depth visual feature corresponding to the previous frame with a dimension of h×w×C and the corresponding frame with a dimension of A reference depth of h×w×C, where h represents the height of the reference depth visual feature, w represents the height of the reference depth visual feature, C represents the number of channels of the reference depth visual feature, and C=2048.

The second layer is the Transformer encoder, whose input is the reference depth visual feature obtained through the reference depth visual feature extraction network of the previous frame and the next frame, and is encoded by the Transformer of the L layer, and the output is corresponding to the previous frame, with a dimension of 256 The global feature of the dimension and the global feature corresponding to the next frame with a dimension of 256 dimensions, wherein the value range of L is 3 to 10, preferably 6.

The third layer is the Transformer decoder. For the previous frame, its input is the global feature obtained by the object query (objectqueries) and the Transformer encoder. First, N output embeddings (dimension 256) are obtained through the mutual attention mechanism, and then through The feed-forward neural network obtains N frame coordinates and class labels, as shown in Figure 5(a); for the next frame, its input is object queries and N output embeddings obtained in the previous frame (the As the result of cascading the tracking query) and the global features obtained by the second layer, first obtain M output embeddings (dimension 256) through the mutual attention mechanism, and then obtain M frame coordinates and class labels through the feedforward neural network. As shown in Figure 5(b);

The fourth layer is a graph convolutional network (as shown in Figure 7). Its input is the N output embeddings of the previous frame and the M output embeddings of the next frame obtained by the third layer. First, by determining the neighbor relationship, the previous The output embedding after cascading of one frame and the output embedding after cascading of the next frame, and then iterate the output embedding of the concatenated previous frame and the output embedding of the next frame through the graph convolutional network respectively Processing, respectively output the local correlation enhancement feature of the previous frame and the local correlation enhancement feature of the next frame;

The fifth layer is the association network (as shown in Figure 8), and its input is the local correlation enhancement feature of the previous frame and the local correlation enhancement feature of the next frame obtained by the fourth layer, and the correlation between the two is based on the feature Processing to obtain the allocation matrix between the corresponding targets of the previous frame and the next frame.

The multi-target tracking model of the present invention is trained through the following steps:

(4-1) Obtain the MOT17 data set and the CrowdHuman image database, divide the obtained MOT17 data set into the first half and the second half according to the proportion, and use the first half of the obtained MOT17 data set and the entire CrowdHuman image database as a training set , using the second half of the resulting MOT17 dataset as the test set.

It should be noted that in this step, the main labeling targets of the CrowdHuman image database are moving pedestrians and vehicles. The MOT17 dataset contains a total of 14 video sequences, of which 7 video sequences are training sets with label information, and the other 7 The video sequence is the test set. The 7 test video sequences come from 7 different scenes, and their shooting angles and camera movements are different. The total length of the test set is 5919 frames, including 188076 detection responses and 785 trajectories. The CrowdHuman image database is a database for pedestrian detection in crowded environments, which contains 15,000 images and contains a total of 450K objects.

In addition, in view of the relatively small amount of data in the MOT17 dataset, in deep learning, the small amount of data used for training will lead to more one-sided features learned by the convolutional neural network, and the resulting model has poor generalization ability and is prone to overfitting. combine. In order to avoid changes in the characteristics and shapes of pedestrian images, the present invention expands the data set through data enhancement. Specifically, the present invention uses another part from the CrowHuman image database to artificially obtain a single image by random scaling and translation. adjacent frames. This can effectively reduce the generalization error of the model and increase the robustness of the model.

Furthermore, the training set in the present invention is used to adjust parameters such as trainable weights and biases in the multi-target tracking model, and the test set is used to adjust hyperparameters such as the learning rate of the multi-target tracking model. The test set does not participate in the model The training is used to statistically test the final prediction effect of the multi-target tracking model.

The advantage of this step is that more video scene information can be simulated, thereby mining more potential information of objects in the video.

(4-2) Set counter cnt3=1 (it is used as the pointer of the video sequence in the whole training set);

(4-3) setting counter cnt2=1 (it is used as the pointer of different frames in the video sequence), judge whether cnt3 is equal to the total number of video sequences in the training set, if enter step (4-16), otherwise enter step (4 -4);

(4-4) judge whether cnt2 is equal to the total number of frames in the cnt3 video sequence in the training set, if it is then proceed to step (4-14), otherwise enter step (4-5);

(4-5) Initialize N _obj learnable object queries (objectqueries) as the object queries of the previous frame and the next frame respectively, wherein N _obj =500;

Specifically, the number of object queries is much larger than the total number of objects in one frame.

(4-6) Use the cnt2 frame in the cnt3 video sequence in the training set as the previous frame, use the cnt2+1 frame as the next frame, and input the previous frame and the next frame into the benchmark depth visual feature extraction network respectively , to obtain the reference depth visual feature I ₁ corresponding to the previous frame with a dimension of 2048, and the reference depth visual feature I ₂ with a dimension of 2048 corresponding to the next frame.

(4-7) For the previous frame and the next frame in the cnt3th video sequence in the training set obtained in step (4-6), the reference depth corresponding to the previous frame obtained in step (4-6) The visual feature I ₁ and the reference depth visual feature I ₂ corresponding to the next frame are used as the query (Query), key (Key) and value (Value) of the Transformer encoder, and are respectively input to the multi-head self-attention layer of the Transformer encoder , to obtain feature vectors with dimensions of 256, respectively, layer-normalize the 256-dimensional feature vectors to obtain normalized feature vectors, and input the normalized features to the front of the Transformer encoder. Feed the neural network and normalize to obtain the 512-dimensional global feature F ₁ corresponding to the previous frame and the 512-dimensional global feature F ₂ corresponding to the next frame.

Furthermore, Figure 3 shows the structure of the Transformer encoder used in this step, and the network structure is: the whole is composed of L Transformer layers, and each Transformer layer is composed of three parts: multi-head self-attention layer, feedforward neural network Network and layer normalization layer composition.

The advantage of this step is that the Transformer encoder can obtain the global information of the entire input frame target. The utilization of such global information helps the model to better extract the feature and position information of the target in the whole input frame.

(4-8) For the previous frame in the cnt3 video sequence in the training set obtained in step (4-6), the object query of the previous frame is input to the multi-head self-attention layer and the layer normalization layer, Obtaining dimension is the object query of 256 dimensions, the global feature F ₁ corresponding to the previous frame obtained by step (4-7) is used as the key (Key) and value (Value) of Transformer decoder, and the object query of 256 dimensions is used as query ( Query), input to the multi-head mutual attention layer of the Transformer decoder and normalized to obtain the output embedding of the N previous frames (dimension 256), the output is embedded into the feedforward neural network and normalized to obtain 512 dimensional feature vector, and input the 512-dimensional eigenvector into the feedforward neural network to obtain N tracking trajectories of the previous frame, each tracking track includes frame coordinates and class labels;

As shown in Figure 5(a), the N output embeddings are used as track queries for the next frame;

Furthermore, in the previous frame, the Transformer decoder only accepts the object query as the query input, and the Transformer decoder implements the same function as the detector at this time.

Furthermore, Figure 4 shows that this step uses the Transformer decoder structure, and the network structure is: the whole is composed of L Transformer layers, and each Transformer layer is composed of four parts: multi-head self-attention layer, multi-head mutual attention layer, a feed-forward neural network, and a layer normalization layer. Among them, the multi-head self-attention layer, feed-forward neural network and layer normalization layer are exactly the same structure as the Transformer encoder. The difference between the multi-head mutual attention layer and the multi-head self-attention layer is that the source of the input feature F is different. Transformer The input features in the decoder come from object queries or track queries and the global features from steps (4-7).

(4-9) For the next frame in the cnt3th video sequence in the training set obtained in step (4-6), first concatenate the tracking query and object query of the latter frame and then input it to the multi-head self-attention Layer and layer normalization layer to obtain a cascaded query with a dimension of 256, using the cascaded query as the query (Query), and the global feature _F2 obtained in steps (4-7) as the key (Key) and value (Value ) is input to the multi-head mutual attention layer of the Transformer decoder and normalized to obtain M output embeddings of the next frame (dimension 256), and the output embeddings are input to the feedforward neural network and normalized to obtain a 512-dimensional feature vector, and input the 512-dimensional feature vector into the feed-forward neural network to obtain M detection targets in the next frame, each detection target includes frame coordinates and class labels, as shown in Figure 5(b);

Furthermore, in the next frame, the tracking trajectory, that is, Figure 5(a), is used as track queries, and object queries are used as the labeling basis for new targets to achieve the detection and calibration of new targets and feature expression. Tracking queries carry the identities of targets throughout the video sequence while matching their changing positions in an autoregressive fashion. To this end, a tracking query is initialized each time a new object is generated, which is derived from the output embedding from the previous frame.

The advantage of the above steps (4-7) to (4-9) is that the encoder and decoder of the Transformer are stacked by multiple identical modules, and each module has the same structure and parameters. This modular design makes Transformer very flexible and easy to expand and modify. Modules can be added or deleted freely according to task requirements, making Transformer suitable for various processing tasks.

(4-10) Establish graph models G 1 , G ₂ for the N tracking trajectories obtained in step (4-8) and the M detection targets obtained in step ( _4-9 );

The establishment process of this step is shown in Figure 6. Specifically, first determine the vertex set. The vertex set V1=(v1, v2...vN) is composed of all tracking trajectories in the previous frame, and the vertex set V2=(v1, v2...vM ) is composed of all detection targets in the next frame, and the vertex values (features) are the output embedding of the corresponding tracking track of the previous frame and the output embedding of the detection target of the next frame, and then determine the vertices in the set V1 and V2 respectively. Neighbor nodes of all vertices constitute edges, and finally graph models G ₁ and G ₂ are obtained.

Furthermore, the process of determining the neighbor nodes of all vertices in V1 and V2 is as follows. First, obtain the frame coordinates and output embeddings corresponding to all vertices in the V1 set, and then obtain the frame coordinates corresponding to the two vertices according to the frame coordinates determined by V1. Intersection over Union (IoU for short) and output embedding obtain the cosine similarity between two vertex features, and select the weight aggregation coefficient between each vertex and all vertices except itself The largest top K as its neighbor nodes, the value of K ranges from 1 to 3, preferably 2, to form an edge, and /> As the weight of the edge between the vertex and the neighbor node, the process of determining the neighbor nodes of all vertices in V2 is the same as that of V1, and will not be repeated here.

Furthermore, the weight of the edge between vertices is the aggregation coefficient (Aggregationweightcoefficient, referred to as AWC) Represents the correlation between vertices, defined as the feature and spatial distance similarity of vertex i and vertex j:

cos(·,·) represents the cosine similarity between feature vectors, IoU(·,·) represents the intersection over Union (IoU for short) of the two target position coordinates, l represents the number of GCN iteration layers, Indicates the vertex feature of vertex i in the l-level GCN, b _i indicates the frame coordinates of vertex i, i∈[1,M], j∈[1,N].

(4-11) Input the graph models G ₁ and G ₂ of the previous frame and the next frame obtained in step (4-10) respectively into the graph convolutional network (as shown in Figure 7), through the graph convolutional network As the number of iterations increases, the local correlation enhanced feature E ₁ of the previous frame and the local correlation enhanced feature E ₂ of the next frame are obtained.

Specifically, this step is first to construct the feature matrix of each vertex in G ₁ and G ₂ obtained in step (4-10), because each vertex (target node) is connected to its K neighbor nodes, then each vertex and its neighbors The total number of nodes is Num=K+1, and the feature matrix Ft is obtained by cascading the vertex and the output embedding of the neighbor nodes. Ft∈R ^Num×d is the feature matrix of the vertex, where d is the feature dimension of each node, and the dimension is 256. Then, the two-dimensional adjacency matrix Mt∈R ^Num×Num of each vertex in G ₁ and G ₂ and all its neighbor nodes is obtained respectively, and the index of the target node is set to 1, then the initialization of the two-dimensional adjacency matrix is defined as:

Among them, i, j∈{1,2,...,Num}, Mt=1 means that there is an edge connecting vertex i and vertex j, and Num means the total number of the target node and its neighbor nodes. Then each vertex in G ₁ and G ₂ is input into the graph convolutional network to obtain aggregated features, where each vertex feature update will update the neighbor relationship, the formula is That is, the vertices in the neighbors measure the neighbor correlation according to the similarity, set /> Recorded as the normalization of the adjacency matrix (normalized), the feature information update of a GCN model is defined as:

expands to:

in is the feature matrix corresponding to each vertex i in layer l, MLP is a multi-layer perceptron (Multi-Layer Perceptron, referred to as MLP), and l is the number of GCN iteration layers, as shown in Figure 7, as the number of iterations increases, the adjacent Nodes in the domain that contribute consistently to the target node continuously receive positive feedback. This consistent contribution reflects the unique correlation of neighbor nodes to the target node. Finally, the aggregated features are input into the feedforward neural network to obtain the local correlation of the previous frame. Sex enhancement feature E ₁ and local correlation enhancement feature E ₂ of the next frame.

The advantage of the above steps (4-10) to (4-11) is that the GCN model is formalized into the Transformer decoder, and the characteristics of graph vertex transfer are used to explicitly guide the modeling of local correlations between adjacent target groups , so as to improve the effectiveness of learning the characteristics of multi-target groups in the scene.

(4-12) Input the local correlation enhancement feature E ₁ of the previous frame and the local correlation enhancement feature E ₂ of the next frame obtained in step (4-11) into the association network (as shown in Figure 8), and perform feature-based Correlation processing between to obtain the allocation matrix between the tracking track of the previous frame and the detection target of the next frame;

Specifically, first of all, before using the association network for data association, a feature matrix needs to be constructed. Since the appearance features of each target contain high-dimensional information, in order to make full use of these rich feature information, the present invention adopts the steps of (4-11) The obtained local correlation enhanced feature E ₁ of the previous frame and the local correlation enhanced feature E ₂ of the next frame are spliced in pairs to obtain the feature matrix E.

Then the feature matrix E is sent to three different 1*1 convolutional layers, and compressed to the MN×2C dimension through the Reshape operation to obtain Q∈R ^MN×2C , K∈R ^MN×2C , V∈R ^MN×2C , where C=256, perform matrix multiplication on Q and K to obtain the correlation between two tracking targets, and then obtain E _{Q, K} after Softmax operation, and then perform a matrix on E _{Q, K} and V multiplication to get a further incidence matrix.

Finally, the matrix is sent to the feed-forward network layer and through two residual connections, the tracking trajectory in the previous frame and the deeper correlation features of the detection target in the next frame are obtained, and the assignment matrix A∈RM ^×N is finally obtained.

Furthermore, if the tracking track in the previous frame is [R ₁ , R ₂ ,..., R _n ], the detection target in the next frame is [r ₁ , r ₂ ,..., r _m ], where n represents the previous frame The total number of tracking trajectories, m represents the total number of detected targets in the next frame; then the allocation matrix constructed in this step is:

Among them, the element P _yz in the matrix represents the correlation confidence between the yth tracking track of the previous frame and the zth detection target of the next frame. In this example, y∈[1,n], and z ∈ [1, m].

The advantage of this step is that by implementing the linear programming network model based on the self-attention mechanism, the data association can be formalized into the realizable form of the neural network and embedded into the integrated MOT learning model. The data association optimization problem and measurement problem of feature expression further improve the effective characteristics of the model.

(4-13) Use the N frame coordinates and class labels obtained in step (4-8), and the M frame coordinates and class labels obtained in step (4-9) to Transformer encoder, Transformer decoder, graph convolution Network and associated network for training, according to the previous frame and the next frame in the training set obtained by the Transformer encoder, Transformer decoder and GCN for step (4-6), the frame coordinates, class labels and local Correlation enhancement features and the frame coordinates, class labels, and local correlation enhancement features of the detection target in the next frame, the distribution matrix between the tracking trajectory of the previous frame and the detection target of the next frame is obtained according to the association network, and the frame coordinates are obtained by , class label and assignment matrix are input into the defined loss function to get detection loss, tracking loss and focal loss;

Specifically, the detection loss, tracking loss, and focal loss are obtained as follows. First, the newly detected target in the next frame and the target associated with the previous frame from the next frame are obtained, and T _t-1 and T _t respectively represent the previous Frame tracking trajectory and the detection target of the next frame, when any trajectory trk∈T _t /T _t-1 , it corresponds to a new target in T _t but does not belong to T _t-1 , and the present invention considers that the trajectory trk is the next frame In the newly detected target, the detection loss is used here; when any trajectory trk∈T _k ∩T _k-1 , that is, the target shared by the previous frame and the next frame, the present invention considers that the trajectory trk is associated with the previous frame The target of one frame, and then input the box of the newly detected target in the next frame to the coordinates and the class label to the detection loss function to obtain the detection loss; associate the next frame with the frame coordinates and class of the target of the previous frame The labels are fed into the tracking loss function to get the tracking loss. Subsequently, the distribution matrix between the tracking trajectory of the previous frame and the detection target of the next frame obtained in step (4-12) is input into the focal loss function to obtain the focal loss.

Furthermore, the detection loss function is defined as follows:

L _det ＝λ _cls L _cls +λ _L1 L _box +λ _giou L _giou

Among them, L _cls is the focal loss between the class label based on the new detection target and the true value class label of the training set, and L _box and L _giou are the L1 distance between the normalized frame coordinates and the true value box coordinates of the training set and generalized IoU and λ _cls , λ _L1 , and λ _giou are the weight parameters corresponding to each part, which are 2, 2, and 3 respectively. Similarly, the tracking loss function L _trk is as follows:

L _trk ＝λ _cls L _cls +λ _L1 L _box +λ _giou L _giou

When the association network gets the distribution matrix, the successfully matched targets in the matrix will be far less than the unmatched targets, so a large number of negative samples will be generated. If the cross-entropy loss function is used, the loss function will be caused by the imbalance of positive and negative samples. Therefore, the present invention adopts focalloss as the loss function for calculating the matching relationship in the association network. Focalloss is an improved version of the cross-entropy loss function, which is specially designed to solve the problem of imbalance between positive and negative samples. It is defined as follows:

Among them, the focal loss adds a modulation coefficient γ and a balance factor α on the basis of the cross-entropy loss function. The value range of the modulation coefficient γ is 1 to 5, preferably 2, and the value range of α is 0.1-0.7, preferably 0.5. The loss of negative sample classification can be reduced, so that the loss weight pays more attention to the classification of positive samples, y=1 means a successful match, and y=0 means no match.

Furthermore, to set the Transformer network training parameters, the model uses the ResNet-50 pre-trained on ImageNet as the CNN backbone, uses the Encoder and Decoder architectures in Deformable-DETR, and combines the pre-training model based on CrowHuman on MOT17 and Joint fine-tuning on CrowHuman, this model adopts Trackformer training strategy, fine-tuning directly on MOT17 and CrowHuan, using AdamW optimizer, this is because the optimizer has the advantages of momentum and adaptive learning rate, the initial learning rate is 2× 10 ^-4 , the number of batches is 2; set the graph convolutional network training parameters, the algorithm adopts the training method from zero, the number of graph convolutional network layers is 3, use the ImageNet pre-trained ResNet-50 model to initialize the model weight, and the optimizer is SGD, the initial learning rate is 1×10 ^-3 , and it will be reduced to 1×10 ^-4 after 40 rounds; the training parameters of the associated network are set, and the training of the associated network model is carried out according to the method created by DeepMOT. The ground-truth labels consist of an assignment matrix. The assignment matrix is a two-dimensional matrix composed of 0s and 1s, representing the true matching results between labeled targets. Therefore, this data set is suitable for the association network designed by the present invention to be used for training the data association process. The initial learning rate is set to 0.005, the learning rate is reduced by 10 times every 10 epochs, and the optimizer is set to SGD;

The advantage of this step is that the target detection, feature expression and data association reasoning in multi-target tracking are unified embedded into the deep network model, which greatly exerts the advantages of multi-task joint optimization learning.

(4-14) cnt2=cnt2+1, and return to step (4-4);

(4-15) cnt3=cnt3+1, and return to step (4-3);

(4-16) According to the detection loss, tracking loss and focal loss obtained in step (4-13), and use the backpropagation method to iteratively train the multi-target tracking model until the multi-target tracking model converges, thereby obtaining a preliminary A trained multi-object tracking model.

(4-17) Use the test set obtained in step (4-1) to verify the multi-target tracking model preliminarily trained in step (4-16) until the obtained tracking accuracy reaches the optimal level, so as to obtain the multi-target tracking model trained Object tracking model.

(5) According to the allocation matrix between the target in the cnt1 frame obtained in step (4) and the corresponding target in the cnt1+1 frame, obtain all targets associated with each target in the cnt+1 frame, Constitute the target in the cnt frame together with all the obtained targets to track the target;

(6) Set the counter cnt1=cnt1+1, and return to step (3).

In summary, the present invention proposes a multi-target tracking algorithm based on Transformer and graph embedding. On the one hand, the GCN model is formalized into the Transformer decoder, and the GCN vertex transfer characteristics are used to explicitly guide the modeling of local correlations between adjacent target groups, so as to improve the effectiveness of the Transformer in learning the characteristics of multi-target groups in the scene. sex. On the other hand, inspired by the "attention is all your need" model of the Transformer architecture, for the multi-target tracking data association problem, an association network is proposed to implement a linear programming network model based on a full attention mechanism, so as to formalize data association into a neural network The form can be embedded into the integrated MOT learning model, which is suitable for feature expression and measurement problems of data association optimization problems in this autonomous learning scenario, and further improves the effective characteristics of the model. Therefore, the present invention can effectively reflect the correlation of real data in the process of multi-target tracking, and the accuracy of tracking results is high.

Experimental results

The actual effect of the present invention is illustrated here by the test results on the MOT17 test set. By using the following standard evaluation indicators, the tracking results of the multi-target tracking algorithm proposed by the present invention on the MOT17 data set are evaluated. These standard evaluation indicators include: multiple objects and accuracy (Multiple Object Tracking Accuracy, referred to as MOTA), multi-object Tracking accuracy (Multiple Object Tracking Precision, referred to as MOTP), tracking accuracy (ID F1 Score, referred to as IDF1), false positives (False Positives, referred to as FP), false negatives (False Negatives, referred to as FN), high-order tracking accuracy (Higher Order Tracking Accuracy (HOTA for short), Association Accuracy (AssA for short), Detection Accuracy (DetA for short), and ID Switches (IDS for short). "↑" means the higher the better, '↓' means the lower the better. As shown in Table 1 below, it lists the detailed comparison of the test results of the present invention and the existing superior performance TransCenter algorithm, TransTrack algorithm and TrackFormer algorithm on the MOT17 test set.

Table 1

It can be seen from Table 1 that: (1) the method of the present invention ranks first on HOTA, and the three indexes of IDF1 and AssA surpass the other three algorithms. Among them, HOTA is the main indicator of evaluating the overall performance of the algorithm performance. Compared with the other three tracking algorithms, the tracking algorithm proposed by the present invention has achieved the best results, which shows that the tracking algorithm proposed by the present invention is better than other tracking algorithms in terms of overall performance. three algorithms;

(2) Higher IDF1 and higher AssA indicate that the method proposed in the present invention can significantly improve the performance of Re-ID in MOT by guiding the Transformer to explicitly mine the local context information of the scene through the advantage of transferring the neighborhood characteristics through the graph model, thus Improve tracking performance.

It is easy for those skilled in the art to understand that the above descriptions are only preferred embodiments of the present invention, and are not intended to limit the present invention. Any modifications, equivalent replacements and improvements made within the spirit and principles of the present invention, All should be included within the protection scope of the present invention.

Claims (10)

1. A multi-target tracking method based on Transformer and graph embedding, is characterized in that, comprises the following steps: (1) Obtain a video sequence, and read the video sequence frame by frame to obtain all frames therein; (2) setting counter cnt1=1; (3) judge whether cnt1 has been equal to the total number of frames in the video sequence, if so the process ends, otherwise enter step (4); (4) Input the cnt1th frame and the cnt1+1th frame in the video sequence into the pre-trained multi-target tracking model to obtain the distribution between the target in the cnt1th frame and the corresponding target in the cnt1+1th frame Matrix, where the cntth frame is the previous frame, and the cnt+1th frame is the next frame; (5) According to the allocation matrix between the target in the cnt1 frame obtained in step (4) and the corresponding target in the cnt1+1 frame, obtain all targets associated with each target in the cnt+1 frame, Constitute the target in the cnt frame together with all the obtained targets to track the target; (6) Set the counter cnt1=cnt1+1, and return to step (3). 2. The multi-target tracking method based on Transformer and graph embedding according to claim 1, wherein the multi-target tracking model includes a sequentially connected benchmark depth visual feature extraction network, Transformer encoder, Transformer decoder, graph convolution Network GCN, and associated networks. 3. the multi-target tracking method based on Transformer and graph embedding according to claim 1 or 2, characterized in that, The first layer is the benchmark deep visual feature extraction network, which includes 1 convolutional layer, 16 building block structures and 1 fully connected layer. The input of the benchmark deep visual feature extraction network is the previous frame and the next frame, and the output is the previous frame. The reference depth visual feature with dimensions h×w×C corresponding to one frame and the reference depth with dimension h×w×C corresponding to the next frame, where h represents the height of the reference depth visual feature, and w represents the reference depth visual feature The height of the feature, C represents the channel number of the reference depth visual feature, and C=2048. The second layer is the Transformer encoder, whose input is the reference depth visual feature obtained through the reference depth visual feature extraction network of the previous frame and the next frame, and is encoded by the Transformer of the L layer, and the output is corresponding to the previous frame, with a dimension of 256 The global feature of dimension and the global feature of dimension 256 corresponding to the next frame, where the value range of L is 3 to 10. The third layer is the Transformer decoder. For the previous frame, its input is the object query and the global features obtained by the Transformer encoder. First, N output embeddings are obtained through the mutual attention mechanism, and then N output embeddings are obtained through the feedforward neural network. Frame coordinates and class labels; for the next frame, the input is the result of the object query and the N output embeddings obtained in the previous frame and the global features obtained in the second layer. First, through the mutual attention mechanism, get M output embeddings and then get M frame coordinates and class labels through the feedforward neural network; The fourth layer is a graph convolutional network, whose input is the N output embeddings of the previous frame and the M output embeddings of the next frame obtained by the third layer. First, by determining the neighbor relationship, the previous frames obtained respectively are concatenated. The output embedding of the concatenated output of the previous frame and the output embedding of the next frame concatenated, and then iteratively process the output embedding of the concatenated previous frame and the output embedding of the concatenated next frame through the graph convolutional network, and output the former The local correlation enhanced features of one frame and the local correlation enhanced features of the next frame; The fifth layer is the association network, whose input is the local correlation enhanced feature of the previous frame and the local correlation enhanced feature of the next frame obtained by the fourth layer, and the two are processed based on the correlation between the features to obtain the previous frame. One frame and the next frame correspond to the assignment matrix between objects. 4. The multi-target tracking method based on Transformer and graph embedding according to any one of claims 1 to 3, wherein the multi-target tracking model is obtained through the following steps of training: (4-1) Obtain the MOT17 data set and the CrowdHuman image database, divide the obtained MOT17 data set into the first half and the second half according to the proportion, and use the first half of the obtained MOT17 data set and the entire CrowdHuman image database as a training set , using the second half of the resulting MOT17 dataset as the test set. (4-2) setting counter cnt3=1; (4-3) counter cnt2=1 is set, judge whether cnt3 is equal to the total number of video sequences in the training set, if enter step (4-16), otherwise enter step (4-4); (4-4) judge whether cnt2 is equal to the total number of frames in the cnt3 video sequence in the training set, if it is then proceed to step (4-14), otherwise enter step (4-5); (4-5) Initialize N _obj learnable object queries (objectqueries) as the object queries of the previous frame and the next frame respectively, wherein N _obj =500; (4-6) Use the cnt2 frame in the cnt3 video sequence in the training set as the previous frame, use the cnt2+1 frame as the next frame, and input the previous frame and the next frame into the benchmark depth visual feature extraction network respectively , to obtain the reference depth visual feature I ₁ corresponding to the previous frame with a dimension of 2048, and the reference depth visual feature I ₂ with a dimension of 2048 corresponding to the next frame. (4-7) For the previous frame and the next frame in the cnt3th video sequence in the training set obtained in step (4-6), the reference depth corresponding to the previous frame obtained in step (4-6) The visual feature I ₁ and the reference depth visual feature I ₂ corresponding to the next frame are used as the query, key and value of the Transformer encoder, and are respectively input to the multi-head self-attention layer of the Transformer encoder to obtain feature vectors with a dimension of 256 , respectively normalize the 256-dimensional feature vectors to obtain the normalized feature vectors respectively, and input the normalized features into the feedforward neural network of the Transformer encoder and normalize them respectively to obtain the normalized feature vectors respectively A 512-dimensional global feature F ₁ corresponding to the previous frame and a 512-dimensional global feature F ₂ corresponding to the next frame are obtained. (4-8) For the previous frame in the cnt3th video sequence in the training set obtained in step (4-6), input the object query of the previous frame to the multi-head self-attention layer and the layer normalization layer to obtain For an object query with a dimension of 256, the global feature F ₁ corresponding to the previous frame obtained in step (4-7) is used as the key and value of the Transformer decoder, and the 256-dimensional object query is used as a query, and input to the multi-head of the Transformer decoder The mutual attention layer is normalized to obtain the output embedding of N previous frames, and the output is embedded into the feedforward neural network and normalized to obtain a 512-dimensional feature vector, and the 512-dimensional feature vector is input to In the feed-forward neural network, to obtain N tracking trajectories of the previous frame, each tracking track includes frame coordinates and class labels; (4-9) For the next frame in the cnt3th video sequence in the training set obtained in step (4-6), the tracking query and object query of the latter frame are first cascaded and then input to the multi-head self-attention layer And layer normalization layer to obtain a cascaded query with a dimension of 256 dimensions, and use the cascaded query as a query, and the global feature _F2 obtained in steps (4-7) as a key and value input to the multi-head mutual attention of the Transformer decoder The force layer is normalized to obtain the output embedding of the M subsequent frames, and the output embedding is input to the feedforward neural network and normalized to obtain a 512-dimensional feature vector, and the 512-dimensional feature vector is input to the feedforward In the neural network, to obtain M detection targets in the next frame, each detection target includes frame coordinates and class labels; (4-10) Establish graph models G 1 , G ₂ for the N tracking trajectories obtained in step (4-8) and the M detection targets obtained in step ( _4-9 ); (4-11) Input the graph models G ₁ and G ₂ of the previous frame and the next frame obtained in step (4-10) into the graph convolutional network respectively, and by increasing the number of iterations of the graph convolutional network, the former The local correlation enhanced feature E ₁ of one frame and the local correlation enhanced feature E ₂ of the next frame. (4-12) Input the local correlation enhancement feature E ₁ of the previous frame and the local correlation enhancement feature E ₂ of the next frame obtained in step (4-11) into the association network, and perform correlation processing based on features, To obtain the allocation matrix between the tracking track of the previous frame and the detection target of the next frame; (4-13) Use the N frame coordinates and class labels obtained in step (4-8), and the M frame coordinates and class labels obtained in step (4-9) to Transformer encoder, Transformer decoder, graph convolution Network and associated network for training, according to the previous frame and the next frame in the training set obtained by the Transformer encoder, Transformer decoder and GCN for step (4-6), the frame coordinates, class labels and local Correlation enhancement features and the frame coordinates, class labels, and local correlation enhancement features of the detection target in the next frame, the distribution matrix between the tracking trajectory of the previous frame and the detection target of the next frame is obtained according to the association network, and the frame coordinates are obtained by , class label and assignment matrix are input into the defined loss function to get detection loss, tracking loss and focal loss; (4-14) cnt2=cnt2+1, and return to step (4-4); (4-15) cnt3=cnt3+1, and return to step (4-3); (4-16) According to the detection loss, tracking loss and focal loss obtained in step (4-13), and use the backpropagation method to iteratively train the multi-target tracking model until the multi-target tracking model converges, thereby obtaining a preliminary A trained multi-object tracking model. (4-17) Use the test set obtained in step (4-1) to verify the multi-target tracking model preliminarily trained in step (4-16) until the obtained tracking accuracy reaches the optimal level, so as to obtain the multi-target tracking model trained Object tracking model. 5. The multi-target tracking method based on Transformer and graph embedding according to claim 4, wherein the step (4-10) is specifically, at first determining the vertex set, and the vertex set V1=(v1, v2...vN) is Consists of all tracking trajectories in the previous frame, the vertex set V2=(v1, v2...vM) is composed of all detection targets in the next frame, and the vertex values are the output embedding and the following tracking trajectories of the corresponding previous frame The output of the detected target of the frame is embedded, and then the neighbor nodes of all vertices in the vertex sets V1 and V2 are respectively determined to form edges, and finally the graph models G ₁ and G ₂ are obtained. 6. the multi-target tracking method based on Transformer and graph embedding according to claim 5 is characterized in that, step (4-10) determines the neighbor node process of all vertices in V1, V2 as follows, at first obtains all vertices in the V1 collection Corresponding frame coordinates and output embedding, and then according to the frame coordinates determined by V1 to obtain the intersection ratio IoU between the frame coordinates corresponding to the two vertices and the output embedding to obtain the cosine similarity between the two vertex features, and select each Weight aggregation coefficient between a vertex and all vertices except itself The largest top K as its neighbor nodes to form an edge, and /> As the weight of the edges between vertices and neighbor nodes, the process of determining the neighbor nodes of all vertices in V2 is the same as that of V1. weight clustering factor Indicates the correlation between vertices, which is equal to: cos(·,·) represents the cosine similarity between feature vectors, IoU(·,·) represents the intersection and union ratio IoU of two target position coordinates, l represents the number of GCN iteration layers, Indicates the vertex feature of vertex i in the l-level GCN, b _i indicates the frame coordinates of vertex i, i∈[1,M], j∈[1,N]. 7. The multi-target tracking method based on Transformer and graph embedding according to claim 6, characterized in that, step (4-11) is specifically, at first constructing _G1 and _G2 that step (4-10) obtains respectively The feature matrix of each vertex, the total number of each vertex and its neighbor nodes is Num=K+1, the feature matrix Ft is obtained by cascading the output of the vertex and the neighbor node, and Ft∈R ^Num×d is the feature matrix of the vertex , where d is the feature dimension of each node, and the dimension is 256. Then, the two-dimensional adjacency matrix Mt∈R ^Num×Num of each vertex in G ₁ and G ₂ and all its neighbor nodes is obtained respectively, and the index of the target node is set to 1, then the initialization of the two-dimensional adjacency matrix is defined as: Among them, i,j∈{1,2,…,Num}, Mt=1 means there is an edge connection between vertex i and vertex j, Num means the total number of the target node and its neighbor nodes, and then each of G ₁ and G ₂ Vertices are input into the graph convolutional network to obtain aggregated features, where each vertex feature update will update the neighbor relationship, the formula is That is, the vertices in the neighbors measure the neighbor correlation according to the similarity, set /> Recorded as the normalization of the adjacency matrix, the feature information update of a GCN model is defined as: expands to: in is the feature matrix corresponding to each vertex i in layer l, MLP is a multi-layer perceptron MLP, and l is the number of GCN iteration layers. As the number of iterations increases, the nodes in the neighborhood that contribute consistently to the target node continue to receive positive feedback. This consistency contribution embodies the unique correlation of neighbor nodes to the target node, and finally the aggregated features are input into the feedforward neural network to obtain the local correlation enhancement feature E ₁ of the previous frame and the local correlation of the next frame Enhancement feature E ₂ . 8. the multi-target tracking method based on Transformer and graph embedding according to claim 7, is characterized in that, Step (4-12) is specifically, first, before using the association network for data association, it is necessary to construct a feature matrix. Since the appearance features of each target contain high-dimensional information, in order to make full use of these rich feature information, this paper The invention adopts the method of splicing the local correlation enhanced feature E ₁ of the previous frame and the local correlation enhanced feature E ₂ of the next frame obtained in step (4-11) to obtain the feature matrix E. Then the feature matrix E is sent to three different 1*1 convolutional layers, and compressed to the MN×2C dimension through the Reshape operation to obtain Q∈R ^MN×2C , K∈R ^MN×2C , V∈R ^MN×2C , where C=256, perform matrix multiplication on Q and K to obtain the correlation between two tracking targets, and then obtain E _{Q, K} after Softmax operation, and then perform a matrix on E _{Q, K} and V multiplication to get a further incidence matrix. Finally, the matrix is sent to the feed-forward network layer and through two residual connections, the tracking trajectory in the previous frame and the deeper correlation features of the detection target in the next frame are obtained, and the assignment matrix A∈RM ^×N is finally obtained. 9. the multi-target tracking method based on Transformer and graph embedding according to claim 8, is characterized in that, The process of obtaining detection loss, tracking loss and focal loss in step (4-13) is as follows. First, obtain the newly detected target in the subsequent frame and the target associated with the subsequent frame to the previous frame. T _t-1 and T _t Represents the tracking trajectory of the previous frame and the detection target of the next frame respectively. When any trajectory trk∈T _t /T _t-1 , it corresponds to a new target in T _t but does not belong to T _t-1 , and then the next frame The frame coordinates and class labels of the newly detected target are input into the detection loss function to obtain the detection loss; the frame coordinates and class labels of the target associated with the previous frame are input into the tracking loss function to obtain Track loss. Subsequently, the distribution matrix between the tracking trajectory of the previous frame and the detection target of the next frame obtained in step (4-12) is input into the focal loss function to obtain the focal loss. The detection loss function is defined as follows: L _det ＝λ _cls L _cls +λ _L1 L _box +λ _giou L _giou Among them, L _cls is the focal loss between the class label based on the new detection target and the true value class label of the training set, and L _box and L _giou are the L1 distance between the normalized frame coordinates and the true value box coordinates of the training set and generalized IoU and λ _cls , λ _L1 , λ _giou are the weight parameters corresponding to each part; The tracking loss function L _trk is as follows: L _trk ＝λ _cls L _cls +λ _L1 L _box +λ _giou L _giou The focal loss function is equal to: The value range of the modulation coefficient γ is 1 to 5, preferably 2, the value range of α is 0.1-0.7, preferably 0.5, y=1 indicates successful matching, and y=0 indicates no matching. 10. A multi-target tracking system based on Transformer and graph embedding, characterized in that, comprising: The first module is used to obtain a video sequence, and reads the video sequence frame by frame to obtain all frames therein; The second module is used to set the counter cnt1=1; The third module is used to judge whether cnt1 has been equal to the total number of frames in the video sequence, if so, the process ends, otherwise enter the fourth module; The fourth module is used to input the cnt1 frame and the cnt1+1 frame in the video sequence into the pre-trained multi-target tracking model to obtain the target in the cntl frame and the corresponding target in the cnt1+1 frame The allocation matrix between, where the cntth frame is the previous frame, and the cnt+1th frame is the next frame; The fifth module is used to obtain all the objects associated with each target in the cnt1th frame in the cnt+1th frame according to the allocation matrix between the target in the cnt1th frame obtained by the fourth module and the corresponding target in the cnt1+1th frame Target, the target in the cnt frame and all the obtained targets constitute the tracking track of the target; The sixth module is used to set the counter cnt1=cnt1+1, and return to the third module.