CN117877068B - Mask self-supervision shielding pixel reconstruction-based shielding pedestrian re-identification method - Google Patents

Abstract

The invention provides a mask self-supervision occlusion pixel reconstruction-based pedestrian re-identification method, and belongs to the field of pedestrian re-identification in multimedia information processing. The method comprises a mask self-encoder fine-tuning image complement model based on mask guidance and a blocked pedestrian re-recognition network based on dynamic graph-graph convolution. Firstly, the image complement model performs self-supervision training by randomly deleting image blocks and generating complete pictures through residual image blocks, and the difference between the generated pictures and original pictures is reduced by using mean square error loss. And then training the blocked pedestrian re-recognition network, and training together by using the triplet loss, the ID loss and the center loss to obtain robust and discriminant features. And in the test process, the image with shielding is complemented by using an image complement model and a mask guiding method, and the pixels of the pedestrian body of the part of the image shielded by the obstacle are reconstructed. And then, inputting the completed pedestrian image into a pedestrian re-recognition blocking network to obtain the pedestrian characteristics, and implementing pedestrian re-recognition. Compared with other methods, the method and the device have the advantages that the accuracy of shielding the re-identification of the pedestrians is obviously improved.

Description

A method for re-identification of occluded pedestrians based on masked self-supervised reconstruction of occluded pixels

Technical Field

The present invention belongs to the field of pedestrian re-identification in computer vision, and in particular relates to an occluded pedestrian re-identification method based on mask self-supervised occluded pixel reconstruction.

Background Art

Person re-identification is a task to retrieve specific pedestrians captured by different cameras. It plays a vital role in surveillance systems and has attracted widespread attention. However, due to challenges such as camera position differences, low resolution, illumination changes, and obstacle occlusion during data collection, existing person re-identification methods lack robustness and have relatively low recognition accuracy. Among them, the occlusion problem is one of the most difficult problems in the current field of person re-identification.

At present, most deep learning-based methods rely on human key point detection or human structure detection. However, due to occlusion problems, key points or structures are often blocked by other obstacles, resulting in the inability of existing methods to accurately identify them. Moreover, similar obstacles will cause similar content to appear in different pedestrian images, which greatly reduces the discriminability of neural network feature extraction, further affecting the accuracy and robustness of pedestrian re-identification methods. Therefore, a new pedestrian re-identification technology that can effectively overcome the occlusion problem is needed to improve recognition performance in complex scenes.

Summary of the invention

In order to solve the above problems, the present invention provides a method for re-identifying occluded pedestrians based on mask self-supervised reconstruction of occluded pixels, the method comprising the steps of:

The training data A, training data B and test data are collected from the surveillance camera device. The training data A includes the pedestrian re-identification pictures with occlusion and the annotations and pedestrian numbers at the instance segmentation level. The training data B includes the pedestrian re-identification pictures with relatively complete human body parts. The test data is divided into query pictures and library pictures, and the test data only includes the original pedestrian re-identification pictures with occlusion and pedestrian numbers.

First, the image completion model is trained to convergence using training data A and training data B. The test image is input into the converged image completion model, and the obscured pedestrian pixels in the image are reconstructed to obtain a de-occluded pedestrian image. Then, the obscured pedestrian re-identification network is trained to convergence using training data B. Finally, the test data is input into the image completion module and a de-occluded pedestrian image is obtained. The de-occluded pedestrian image is input into the pedestrian re-identification network to obtain a better pedestrian recognition accuracy.

The mask-guided masked autoencoder fine-tuning image completion model is characterized by performing de-occlusion processing on the occluded pedestrian image. The completion model training process is as follows:

(1) Using training data A, the existing instance segmentation network Mask2former is trained until convergence, and an instance segmentation network that can output human body masks in pedestrian images is obtained. The converged instance segmentation network can predict the pedestrian mask _Mi corresponding to the image _Xi with identity i. Among them, the pixels of pedestrians that are not blocked in the original image _Xi in the corresponding pedestrian mask _Mi image position are white, and the other pixels are black.

(2) Use the image modeling network guided by the self-supervised mask to reconstruct the obscured pedestrian pixels in the obscured pedestrian image. The obscured pedestrian image and the _mask corresponding to the image are reconstructed. (H and W are the height and width of the image, and 3 is the three dimensions of the RGB image) Use the image block function to convert it to an image block embedding, where the height and width of each image block are _Ph and _Pw . For the image _Xi , the image block function is to convolve each image _Xi , the convolution kernel size is _Ph and _Pw , and the step size is also _Ph and _Pw , and the output dimension of each image block is C. The image is converted to an image block embedding using the image block function and flattened to obtain the image block embedding as follows:

_XiP = Patch( _Xi ,θ)

M _iP = Random(M _i )

Among them, Patch is the image block function. Where C is the number of dimensions. (P _h ,P _w ) represents the resolution of each image block. is the total number of image blocks, and θ is a learnable parameter. The Random function calculates the number of image blocks according to the input mask image size and randomly generates the corresponding number of pixel retention scores. _{M iP} represents the pixel retention score of each image block in the current image Xi _, and its value ranges from 0 to 105. Image blocks with pixel retention scores less than 60 will be marked as reconstructed blocks. The image block embedding corresponding to the marked image blocks will be discarded and will not enter the encoder part. During the training phase, M _iP is generated using a random function.

(3) The non-discarded image block embedding and the image block position coding are added and then input into the pixel reconstruction encoder. The position coding uses the 2D position coding formula as follows:

The pos _X and pos _Y are the horizontal and vertical coordinates of the image block in the original image. The d _model is the dimension of the position code, and in the present invention, the d _model is the same as the image block embedding C. The value of i is a positive integer from 0 to 0.5C-1. After each item of the position code is calculated, the obtained codes are arranged in the following manner to obtain the 2D position code: PE(pos _X , 0), PE(pos _Y , 1), PE(pos _X , 2), PE(pos _Y , 3)...PE(pos _X , C-2), PE(pos _Y , C-1).

The original image block is embedded into the input reconstruction encoder to obtain an intermediate tensor. The learnable tensor is inserted into the position of the discarded pixel block in the intermediate tensor to obtain the intermediate tensor to be learned. The shape of the learnable tensor is the same as the embedding shape of the replaced image block, which is C represents the number of dimensions of the image block embedding. The intermediate tensor to be learned is input into the decoder to obtain the reconstructed image block embedding, and the reconstructed image block embedding is unpacked to obtain the reconstructed image.

During the training process, the image modeling network guided by the self-supervised mask is trained using the training data B and its mask image. Specifically, the black pixels in the mask of each pedestrian image are covered on the corresponding pedestrian image, and the white pixels are not processed. The training data B_withMask is obtained. The image modeling network is self-supervised trained using the training data B_withMask.

The self-supervised mask-guided image modeling network only calculates the loss for image patches with a pixel retention score less than 60, and uses the mean square error as the loss function.

Furthermore, during the prediction process, the test data is fed into the converged instance segmentation network to obtain the test image mask _MiTest . The test data and the test image mask _MiTest are fed into the image modeling network to obtain a de-occluded pedestrian image. The image modeling network process during the prediction process is as follows:

The pedestrian image and the test image mask _MiTest are input into the image segmentation function together to obtain the image block embedding and the flattened image block embedding and the pixel retention score _MiP . The image blocks with a pixel retention score _MiP less than 60 will be discarded. The non-discarded image block embedding is added to the image block position code and input into the pixel reconstruction encoder to obtain an intermediate tensor. The learnable tensor is inserted into the position of the discarded pixel block in the intermediate tensor to obtain the intermediate tensor to be learned. The intermediate tensor to be learned is input into the decoder to obtain the reconstructed image block embedding, and the reconstructed image block embedding is unpacked to obtain the reconstructed image.

The only difference between the prediction process and the training process is the way to obtain the pixel retention score M _iP . The method for obtaining the pixel retention score M _iP in the test process is as follows:

_MiP =Patch_formask( _MiTest ,1)

M _iP represents the pixel retention score of all image blocks in the current image Xi _. Each score ranges from 0 to 105. Image blocks with pixel retention scores less than 60 will be marked as reconstructed blocks. _{The XiP} corresponding to the marked image blocks will be discarded and will not enter the encoder part.

Patch_formask is an image segmentation function with the same structure as the Patch function. The difference between them is that the learnable parameter of the convolution kernel in the Patch function is θ, while the parameters in the convolution kernel in the Patch_formask function are not learnable and are fixed to 1. The output dimension of the pixel retention score is 1, that is, each image patch corresponds to a pixel retention score. In the Patch_formask function, 1 means that the parameters in the convolution kernel are not learnable and are fixed to 1.

Furthermore, the pedestrian re-identification network uses a dynamic graph structure module and a graph convolution feature propagation module to assist in propagating features. The dynamic graph structure module establishes different graph structures at multiple positions in the convolutional neural network, that is, converts the feature graph into a K nearest neighbor graph structure at multiple positions in the convolutional neural network. The dynamic graph structure module inputs a feature graph with a height of H and a width of W, and outputs an adjacency matrix corresponding to the feature graph. The pseudo code of the dynamic graph structure module is as follows:

Furthermore, the correlation of each node in the feature graph is calculated to obtain the correlation matrix. The formula is as follows:

R＝θ(F)φ(F) ^T

Where R is the correlation matrix, is the feature map output by the convolutional layer, C is the number of dimensions of the feature map, and W and H are the width and height of the feature map, respectively. ^XT represents the transpose of the matrix X. θ(F) and φ(F) represent the input of the feature map F into two transfer functions with exactly the same structure but different parameters. The transfer function consists of a 1×1 convolutional layer, a batch normalization layer, and a ReLU activation function.

Next, multiply the correlation matrix R by the adjacency matrix A and normalize it using the softmax function to obtain the similarity adjacency matrix. The formula is as follows:

in, is the similarity adjacency matrix, N is the number of nodes in the feature graph, A is the adjacency matrix output by the dynamic graph structure module, and ⊙ represents the Hadamard product of the matrix.

Then, node feature propagation is performed on the graph structure A, and the formula is as follows:

in, is the feature after feature propagation, It is the feature obtained by matrix transformation of the feature map output by the convolution layer. Through matrix transformation, the propagated features can be converted back into feature maps.

All the processes of the above dynamic graph structure module and graph convolution feature propagation module are denoted as OGA(F). It is the feature map output by the convolutional layer.

Furthermore, the residual structure is introduced into the feature propagation process. At the same time, multiple stacking of OGA modules can make the features propagate more fully. The stacking method of the OGA modules with residual structure is as follows:

Where β represents the learnable parameter.

Furthermore, during the training process, the loss function of the occluded person re-identification network is expressed as:

L＝L _ID +L _Triplet +εL _C

Among them, L _Triplet is the triplet loss, L _ID is the ID loss, and L _C is the Center loss. ε is the balance weight of the Center loss, which is set to 0.1 in this network.

The L _Triplet expression is as follows:

In the above formula, B is the number of training mini-batch samples. represents the benchmark sample, represents a positive sample of the same category but different from the benchmark sample, represents a negative sample of a different category from the benchmark sample. α represents a set training interval, which is set to 0.2 in the present invention. f(x) is the feature of image x.

The L _ID expression is as follows:

In the above formula, y represents the true label value of the training sample. N represents the number of pedestrian identities in the data set. f( _xi ) represents the embedding predicted by the network for the image _xi . In the present invention, the ε value is a constant, set to 0.1.

The L _C expression is as follows:

In the above formula, f( _xi ) is the feature of image _xi . B is the number of training mini-batch samples. Represents the center of all features of category _yi , that is, the average value of the features of all images of category _yi in the mini-batch.

Furthermore, after all networks are trained to convergence, the test data is sent to a masked autoencoder fine-tuned image completion model based on mask guidance to obtain a de-occluded pedestrian image. Then, the de-occluded pedestrian image is sent to an occluded pedestrian re-identification network based on dynamic graphs and graph convolutions to obtain the features of each image. For each query image, sort by feature distance, and the library images with closer distances will be ranked in front, and the 10 library images with the closest distances are taken as the query results of the query image. The library images with the same identity label are taken as the correct results matched to the query graph, and the average accuracy of the query image is calculated. The average accuracy of all query images is calculated to obtain the average accuracy mean of the present invention on the data set. The average of the accuracy of the library images with the closest feature distances of all query images is calculated to obtain the first hit rate.

The present invention provides a masked pedestrian re-identification method for self-supervised reconstruction of occluded pixels, which has the following advantages:

(1) The self-supervised mask-guided image modeling network of the method is trained in a self-supervised manner, which avoids the time waste of large-scale labeling of image data, and its effect is comparable to that of supervised methods.

(2) The method adopts pixel completion technology, so that the network can completely complete the occluded pedestrians in the occluded data, so that the existing method based on human key points can be applied to occluded images.

(3) The method adopts an occluded pedestrian re-identification network based on a combination of dynamic graph neural networks and graph convolution, which effectively utilizes the image spatial information. The dynamic graph neural network is conducive to increasing the effective receptive field of the convolutional neural network and improving the pedestrian recognition performance in different scenes and occlusion conditions.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

FIG1 is a flow chart of a method for re-identifying an occluded person by masked self-supervised reconstruction of occluded pixels provided by the present invention;

FIG2 is a schematic diagram of the completion model (including an instance segmentation network and an image modeling network);

FIG3 is a schematic diagram of the completion model's completion effect on an occluded image;

FIG4 is a schematic diagram of a network structure for occluded person re-identification based on dynamic graphs and graph convolution;

DETAILED DESCRIPTION

In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention is further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings. It should be understood that these descriptions are only exemplary and are not intended to limit the scope of the present invention. In addition, in the following description, the description of well-known structures and technologies is omitted to avoid unnecessary confusion of the concept of the present invention.

Exemplary Methods

As shown in FIG1 , the present invention provides a flow chart of a method for re-identifying an occluded person based on mask self-supervised reconstruction of occluded pixels, and the method steps are as follows:

Step S110: First, all test set images in the data set are retained, and the training data is divided into images with relatively complete pedestrian human body structures (referred to as training data B) and pedestrian re-identification images with large occlusions (referred to as training data A). In addition, pedestrians in the training data A images are labeled at the instance segmentation level. All data are converted to 210 pixels in height and 98 pixels in width.

Step S120: Train the instance segmentation network using the training data A. The instance segmentation network uses the Mask2former network and the loss function uses Diceloss. After completion, the images in the training data B are sent to the instance segmentation network to obtain the pedestrian mask of each image in the training data B.

The self-supervised mask-guided image modeling network is trained until convergence using the training data B and its mask image. Specifically, the black pixels in the mask of each pedestrian image are covered on the corresponding pedestrian image, and the white pixels are not processed. The training data B_withMask is obtained. The image modeling network is self-supervisedly trained using the training data B_withMask (the image modeling network training guided by the self-supervised mask does not use the image mask, and the M _iP is generated using a random function).

The image modeling network will predict the image and the corresponding mask _Xi , (H and W are the image height and width, and 3 is the dimension of the RGB image) is converted into multiple image blocks, where each image block is 15 pixels high and 7 pixels wide. For the image _Xi , the Patch operation is to convolve each image _Xi , with a convolution kernel size of 15 high and 7 wide, and a stride of 15 high and 7 wide. The output dimension of each image block is C. The image is converted into image blocks and flattened to get the image block embedding result as follows:

_XiP = Patch( _Xi ,θ)

M _iP = Random(M _i )

Where C is the number of dimensions, and in the present invention, C is 768. The number of dimensions of the mask is 1. (P _h ,P _w ) represents the resolution of each image block, i.e., (15, 7). is the total number of image blocks, which is 196 in the present invention. Patch is an image block function, and the learnable parameter of the convolution kernel in the Patch function is θ. The Random function calculates the number of image blocks according to the input mask image size, and randomly generates a corresponding number of pixel retention scores. The M _iP represents the pixel retention score of each image block in the current image Xi _, ranging from 0 to 105. Image blocks with pixel retention scores less than 60 will be marked as reconstructed blocks, and the image block embedding corresponding to the marked image blocks will be discarded and will not enter the encoder part. M _iP is randomly generated during training.

The non-discarded image block embedding and the image block position coding are added and then input into the pixel reconstruction encoder. The position coding uses the 2D position coding formula as follows:

The pos _X and pos _Y are the horizontal and vertical coordinates of the image block in the original image. The d _model is the dimension of the position code, and in the present invention, the d _model is the same as the image block embedding C. The value of i is a positive integer from 0 to 0.5C-1. After each item of the position code is calculated, the obtained codes are arranged in the following manner to obtain the 2D position code: PE (pos _X , 0), PE (pos _Y , 1), PE (pos _X , 2), PE (pos _Y , 3) ... PE (pos _X , 766), PE (pos _Y , 767).

After the reconstruction encoder, the original image block embedding is converted into an intermediate tensor. The learnable tensor is inserted into the position of the discarded pixel block in the intermediate tensor to obtain the decoder input tensor. The shape of the learnable tensor is the same as the shape of the replaced image block data, which is C represents the number of dimensions of the image block embedding, that is, 768. The decoder input tensor is input into the decoder to obtain embeddings of all reconstructed image blocks after pixel reconstruction, and the reconstructed image blocks are embedded in the deblocking to obtain a reconstructed image.

The self-supervised mask-guided image modeling network only calculates the loss for image patches with pixel retention scores less than 60, and the loss uses the mean square error loss.

Furthermore, during the test process, the test data is fed into the converged instance segmentation network to obtain the test image mask _MiTest . The test data and the test image mask _MiTest are jointly fed into the image modeling network to obtain a de-occluded pedestrian image.

During the test, only the method for obtaining the pixel retention score M _iP is different from that during the training process. The method for obtaining the pixel retention score M _iP during the test is as follows:

_MiP =Patch_formask( _MiTest ,1)

M _iP represents the pixel retention score of all image blocks in the current image M _i . Each score ranges from 0 to 105. Image blocks with pixel retention scores less than 60 will be marked as reconstructed blocks. The _XiP corresponding to the marked image blocks will be discarded and will not enter the encoder part.

Patch_formask is an image segmentation function with the same structure as the Patch function. The difference between them is that the learnable parameter of the convolution kernel in the Patch function is θ, while the parameters in the convolution kernel in the Patch_formask function are not learnable and are fixed to 1. The output dimension of the pixel retention score is 1, that is, each image block corresponds to one pixel retention score.

In the Patch_formask function, 1 means that the parameters in the convolution kernel are not learnable and are fixed to 1.

The rest of the process is exactly the same as the training process. That is, the test data is fed into the strength segmentation network to obtain the test image mask. The test data and the test image mask are fed into the self-supervised mask-guided image modeling network to obtain the de-occluded pedestrian image. The de-occluded pedestrian image is used as the test data for the occluded pedestrian re-identification network based on dynamic graph and graph convolution.

Step S130: Use training set B to train the occluded pedestrian re-identification network. The pedestrian re-identification network includes a ResNet-50 network, a dynamic graph structure module and a graph convolution feature propagation module. The ResNet-50 network is used to extract feature maps in pedestrian re-identification images, and the dynamic graph neural network feature propagation module is used to propagate features on the standardized feature maps to obtain more robust and discriminative features. Specifically, the dynamic graph structure establishment module inserts a multi-layer dynamic graph structure establishment module and a dynamic graph convolution feature propagation module after stage 1, 2, 3, and 4 of ResNet-50 respectively (the module only changes the content of the feature map, and does not change its shape).

The feature map with a height of H and a width of W is input into the dynamic graph structure module to obtain the adjacency matrix R corresponding to the feature map.

The pseudo code of the dynamic graph structure module is as follows:

Furthermore, the correlation of each node in the feature graph is calculated to obtain the correlation matrix. The formula is as follows:

R＝θ(F)φ(F) ^T

Where R is the correlation matrix, is the feature map output by the convolutional layer, C is the number of dimensions of the feature map, and W and H are the width and height of the feature map, respectively. ^XT represents the transpose of the matrix X. θ(F) and φ(F) represent the input of the feature map F into two transfer functions with exactly the same structure but different parameters. The transfer function consists of a 1×1 convolutional layer, a batch normalization layer, and a ReLU activation function.

Next, multiply the correlation matrix R by the adjacency matrix A and normalize it using the softmax function to obtain the similarity adjacency matrix. The formula is as follows:

in, is the similarity adjacency matrix, N is the number of nodes in the feature graph, A is the adjacency matrix output by the dynamic graph structure module, and ⊙ represents the Hadamard product of the matrix.

Then, node feature propagation is performed on the graph structure A, and the formula is as follows:

in, is the feature after feature propagation, It is the feature obtained by matrix transformation of the feature map output by the convolution layer. Through matrix transformation, the propagated features can be converted back into feature maps.

All the processes of the above dynamic graph structure module and graph convolution feature propagation module are denoted as OGA(F). It is the feature map output by the convolutional layer.

Furthermore, the residual structure is introduced into the feature propagation process. At the same time, multiple stacking of OGA modules can make the features propagate more fully. The stacking method of the OGA modules with residual structure is as follows:

Where β represents the learnable parameter.

Furthermore, during the training process, the loss function of the occluded person re-identification network is expressed as:

L＝L _ID +L _Triplet +εL _C

Among them, L _Triplet is the triplet loss, L _ID is the ID loss, and L _C is the Center loss. ε is the balance weight of the Center loss, which is set to 0.1 in this network.

The L _Triplet expression is as follows:

In the above formula, B is the number of training mini-batch samples. represents the benchmark sample, represents a positive sample of the same category but different from the benchmark sample, represents a negative sample of a different category from the benchmark sample. α represents a set training interval, which is set to 0.2 in the present invention. f(x) is the feature of image x.

The L _ID expression is as follows:

In the above formula, y represents the true label value of the training sample. N represents the number of pedestrian identities in the data set. f( _xi ) represents the embedding predicted by the network for the image _xi . In the present invention, the ε value is a constant, set to 0.1.

The L _C expression is as follows:

In the above formula, f( _xi ) is the feature of image _xi . B is the number of training mini-batch samples. Represents the center of all features of category _yi , that is, the average value of the features of all images of category _yi in the mini-batch.

The network is trained using the gradient descent algorithm with the default configuration of Adam. The training ends when the loss value reaches the minimum.

Step S140: After the network is trained to convergence, the test data can be used to test the occluded pedestrian re-identification network based on dynamic graph and graph convolution. The de-occluded pedestrian image obtained in step S120 is sent to the pedestrian re-identification network for feature extraction to obtain the features of each image. For each query image, sort by feature distance, and the library images with closer distances will be ranked in front. The 10 library images with the closest distances are taken as the query results of the query image. The library images with the same identity label are taken as the correct results matched to the query graph, and the average accuracy of the query image is calculated. The average accuracy of all query images is calculated to obtain the average accuracy mean of the present invention on the data set. Calculate the average accuracy of the library images with the closest feature distance of all query images to obtain the first hit rate.

Step S150: Calculate the pedestrian re-identification accuracy rate on the occluded pedestrian re-identification dataset Occluded-DukeMTMC according to the obtained classification results.

Through this implementation, the data set is first divided into two parts: one with a relatively complete human body and one with large occluded pixels (training data A and B, respectively). Then, the image completion model is trained until convergence using training data A and B. The occluded pedestrian re-identification network is trained using training data B. During the test process, the test data is first sent to the image completion model: first sent to the instance segmentation network to obtain the mask corresponding to each test data. The test data and the test image mask are simultaneously sent to the image modeling network for occluded pixel completion. Finally, the de-occluded pedestrian image is sent to the pedestrian re-identification network for feature extraction and prediction to obtain the classification result predicted by the network.

To further illustrate, assuming that an occluded pedestrian re-identification dataset is classified according to this implementation, a classification result with higher accuracy than most methods will be obtained.

Detailed description of the invention Results

This implementation uses the publicly available occluded person re-identification dataset Occluded-DukeMTMC. The details of the dataset are as follows:

In this dataset, all query images are occluded by various objects (e.g., trees, cars, other people). The training, query, and gallery sets contain 14%/15%/10% occluded images, respectively. The training set of Occluded-DukeMTMC contains 15,618 images, covering a total of 702 identities. The test set contains 1,110 identities, including 17,661 gallery images and 2,210 query images. The Occluded-DukeMTMC dataset is the largest and most difficult dataset for the current occluded person re-identification task.

In order to verify the superiority of this implementation (Ours), this implementation is compared with several existing occluded pedestrian re-identification methods, including BoT, MHSA, OAMN, HG and other methods. The mean average precision (mAP) and the first hit rate (Rank-1) of these methods on the above public datasets will be compared. The specific data comparison is shown in Table 1.

Table 1 Accuracy of Occluded-DukeMTMC dataset (%)

By comparing the data in the above table, we can clearly see that Ours achieves the best performance and significantly improves the accuracy of occluded pedestrian re-identification. The quantitative results fully demonstrate the superiority of Ours, because Ours can better complete the human key point information and perspective difference information in the occluded image. Ours uses a combination of convolutional neural networks and dynamic graph neural networks to effectively expand the network's receptive field, thereby improving the robustness of occluded pedestrian re-identification data. A large number of experiments show that this method is superior to existing methods.

This embodiment proposes an occluded person re-identification method based on mask self-supervised occluded pixel reconstruction, which is used to perform pedestrian re-identification tasks on occluded images.

By training an image completion model consisting of an instance segmentation network and an image modeling network guided by a self-supervised mask, the occluded pedestrian pixels in the image are reconstructed to obtain a de-occluded pedestrian image. Then, an occluded pedestrian re-identification network based on dynamic graphs and graph convolution is used to re-identify the de-occluded pedestrian image. The occluded pedestrian re-identification network uses a dynamic graph structure to effectively expand the receptive field of the convolutional neural network and increase the discriminability and robustness of the feature graph learned. The experimental results on the public dataset Occluded-DukeMTMC show that this implementation has higher accuracy and better superiority than other methods.

It should be understood that the above specific embodiments of the present invention are only used to illustrate or explain the principles of the present invention, and do not constitute a limitation of the present invention. Therefore, any modifications, equivalent substitutions, improvements, etc. made without departing from the spirit and scope of the present invention should be included in the protection scope of the present invention. In addition, the appended claims of the present invention are intended to cover all changes and modifications that fall within the scope and boundaries of the appended claims, or the equivalent forms of such scope and boundaries.

Claims (4)

1. An occlusion pedestrian re-identification method based on mask self-supervision occlusion pixel reconstruction, which is characterized by comprising the following steps:

Collecting training data D, training data E and test data from a monitoring camera device, wherein the training data D comprises blocked pedestrian re-identification pictures and instance segmentation level labels and pedestrian numbers, the training data E comprises pedestrian re-identification pictures of relatively complete human body parts, and the test data only comprises original blocked pedestrian re-identification pictures and pedestrian numbers; constructing a mask self-encoder fine-tuning image complement model based on mask guidance, wherein the image complement model consists of an instance segmentation network and an image modeling network guided by self-supervision mask, the instance segmentation network is used for segmenting pedestrian instances of an input image to obtain masks of each pedestrian, a random function is used for randomly generating pixel retention scores of different image blocks in a pedestrian mask image, the image modeling network guided by self-supervision mask is used for reconstructing the image blocks with the pixel retention scores smaller than 60 in the pedestrian image to obtain a de-occluded pedestrian image, and training data D and training data E are used for training the image complement model to be converged;

And predicting test data by using the complement model: inputting the test data into the example segmentation network to obtain pedestrian masks corresponding to the test data, calculating pixel retention scores of different image blocks in pedestrian mask images by using an image block function, and reconstructing the image blocks with the pixel retention scores smaller than 60 in the pedestrian images by using an image modeling network guided by a self-supervision mask to obtain a de-blocked pedestrian image;

Constructing a shielding pedestrian re-recognition network based on dynamic graph and graph convolution, wherein the pedestrian re-recognition network consists of a ResNet-50 network, a dynamic graph structure module and a graph convolution feature propagation module, extracting a feature graph in a pedestrian image by using the ResNet-50 network, constructing a topological graph structure corresponding to the feature graph by using the dynamic graph structure module, performing feature propagation on the topological graph structure by using the graph convolution feature propagation module, and training the shielding pedestrian re-recognition network to convergence by using training data E;

In the prediction process of the shielding pedestrian re-recognition network, inputting a pedestrian image subjected to de-shielding into the converged pedestrian re-recognition model, so that characteristics of each pedestrian image can be obtained, sequencing each query image according to characteristic distances from small to large, taking 10 library images with the nearest distance as query results of the query image, taking library images with the same identity label as correct results matched with the query image, and calculating average accuracy and first hit rate of the query image;

the training process of the blocked pedestrian re-identification network based on the convolution of the dynamic graph and the graph is as follows:

The pedestrian re-recognition network uses a dynamic graph structure module and a graph convolution feature propagation module to assist in propagation of features, the dynamic graph structure module establishes different graph structures at a plurality of positions in a convolutional neural network, namely, the feature graph is converted into a K nearest neighbor graph structure at the plurality of positions in the convolutional neural network, the dynamic graph structure module inputs a feature graph with height of H and width of W, an adjacent matrix A corresponding to the feature graph is output, and pseudo codes of the dynamic graph structure module are as follows:

then, calculating the correlation of each node in the feature map to obtain a correlation matrix, wherein the formula is as follows:

R＝θ(F)φ(F)^T

Wherein R is a correlation matrix, The method comprises the steps that a characteristic diagram output by a convolution layer is represented by C, the dimension number of the characteristic diagram is represented by W and H, the width and the height of the characteristic diagram are represented by X ^T, the transposition of a matrix X is represented by X (F), and two transfer functions with identical structures and different parameters are respectively input into the characteristic diagram F, wherein the transfer functions consist of a 1X 1 convolution layer, a batch normalization layer and a ReLU activation function;

Multiplying the correlation matrix R by the adjacency matrix A, and normalizing by a softmax function to obtain a similarity adjacency matrix, wherein the formula is as follows:

Wherein, The similarity adjacency matrix is G is the number of nodes in the feature graph, A is the adjacency matrix output by the dynamic graph structure module, and the Hadamard product of the matrix is represented by the following formula:

Wherein, Is a feature that has been propagated through the feature,The characteristic diagram output by the convolution layer is obtained by matrix transformation, and the propagated characteristic can be converted into the characteristic diagram again through matrix transformation;

All the processes of the dynamic graph structure module and the graph rolling characteristic propagation module are denoted as OGA (F), Is a feature map of the convolutional layer output; introducing the residual structure into the characteristic propagation process, and stacking the OGA modules for a plurality of times can enable the characteristics to be propagated more fully, wherein the stacking mode of the OGA modules with the residual structure is as follows:

Wherein β represents a learnable parameter;

in the training process, the loss function expression of the pedestrian re-recognition network is as follows:

L＝L_ID+L_Triplet+εL_C

Where L _Triplet is the triplet loss, L _ID is the ID loss, L _C is the Center loss, ε is the balance weight of the Center loss, which is set to 0.1 in the present network;

The expression of L _Triplet is as follows:

where B is a training small sample number, A reference sample is represented and a reference sample is represented,Representing positive samples of the same class as the reference sample but different,Negative samples of different categories from the reference sample are represented, alpha represents a set training interval, and is set to 0.2, and f (x) is the characteristic of picture x;

The expression of L _ID is as follows:

In the above formula, y represents the true tag value of the training sample, Z represents the number of pedestrians in the data set, f (x _i) represents the embedding of the network predicted by the picture x _i, and epsilon is constant and is set to 0.1;

the expression of L _C is as follows:

Where f (x _i) is the feature of picture x _i, B is a training small sample number, The center of all features of category y _i, i.e., the average of the features of all pictures of category y _i in a small lot, is represented.

2. The method for re-identifying the blocked pedestrians based on the mask self-supervision blocked pixel reconstruction according to claim 1, wherein the full model is used for carrying out de-blocking processing on the blocked pedestrian picture, and the full model training process comprises the following steps:

(1) Training the existing example segmentation network Mask2former to be converged by using training data D to obtain an example segmentation network capable of outputting a human Mask in a pedestrian picture, wherein the converged example segmentation network can predict a pedestrian Mask M _i corresponding to a picture X _i with an identity of i, wherein pixels of non-occluded pedestrian pixels in an original picture X _i in the corresponding pedestrian Mask M _i picture position are white, and other pixels are black;

(2) Reconstructing an occluded part of pedestrian pixels in an occluded pedestrian image by using an image modeling network guided by a self-supervision mask, and carrying out image reconstruction on the occluded pedestrian image H and W are the height and width of the picture, 3 is the three dimensions of the RGB picture, the three dimensions are converted into image block embedding by using an image block function, wherein the height and width of each image block are P _h and P _w, for the picture X _i and the mask M _i, the image block function is to convolve each picture X _i, the convolution kernel size is P _h and P _w, the step size is also P _h and P _w, the output dimension of each image block is C, and the image block embedding obtained by converting the picture into the image block embedding by using the image block function and leveling is as follows:

X_iP＝Patch(X_i,θ)

M_iP＝Random(M_i)

Where Patch is an image blocking function, Where C is the number of dimensions, (P _h,P_w) represents the resolution of each image block,Is the total number of image blocks, θ is a learnable parameter, the Random function calculates the number of image blocks according to the size of the input mask image, and randomly generates a corresponding number of pixel retention scores, M _iP represents the pixel retention scores of all the image blocks in the current picture X _i, each score takes a value from 0 to 105, the image blocks with pixel retention scores less than 60 are marked as reconstruction blocks, X _iP corresponding to the marked image blocks are discarded and do not enter the encoder part, and M _iP is generated using a Random function in the training phase;

(3) The non-discarded image blocks are embedded and added with the image block position codes, and then are input into a pixel reconstruction encoder, wherein the position codes use a 2D position coding formula as follows:

The pos _X and pos _Y are the abscissa and ordinate of the image block in the original image, the D _model is the dimension number of the position code, D _model is the same as the embedded C of the image block, the value of i is a positive integer from 0 to 0.5C-1, after each position code is calculated, the obtained codes are arranged as follows to obtain the 2D position code ：PE(pos_X,0),PE(pos_Y,1),PE(pos_X,2),PE(pos_Y,3)...PE(pos_X,C-2),PE(pos_Y,C-1);

Embedding an original image block into an input reconstruction encoder to obtain an intermediate tensor, inserting a learnable tensor into the discarded pixel block position in the intermediate tensor to obtain an intermediate tensor to be learned, wherein the shape of the learnable tensor is the same as the embedded shape of a replaced image block, and the shape of the learnable tensor is as followsC represents the dimension number of the image block embedding, the intermediate tensor to be learned is input into a decoder to obtain the reconstructed image block embedding, and the reconstructed image block embedding is unpacked to obtain a reconstructed image;

The self-supervised mask guided image modeling network computes the loss only for image blocks with pixel retention scores less than 60, using the mean square error as the loss function.

3. The method for re-identifying the blocked pedestrian based on the mask self-supervision blocked pixel reconstruction according to claim 1, wherein the pixel retention score M _iP is obtained in a different manner from the training process in the completion model prediction process;

Sending the test data into a converged instance segmentation network to obtain a test picture mask M _iTest, inputting the test data and the test picture mask M _iTest into the image modeling network together to obtain a pedestrian image with de-occlusion, wherein the image modeling network flow in the prediction process is as follows:

The pedestrian picture and the test picture mask M _iTest are input into an image blocking function together to obtain an image block embedding and pixel retention score M _iP obtained by embedding and leveling the image blocks, the image blocks with the pixel retention score M _iP smaller than 60 are discarded, the non-discarded image block embedding and the image block position coding are added, the image block is input into a pixel reconstruction encoder to obtain an intermediate tensor, a learnable tensor is inserted into the discarded pixel block position in the intermediate tensor to obtain an intermediate tensor to be learned, the intermediate tensor to be learned is input into a decoder to obtain a reconstructed image block embedding, and the reconstructed image block is obtained after the reconstructed image block embedding unpacking;

The pixel retention score M _iP in the prediction process is obtained by the following formula:

M_iP＝Patch_formask(M_iTest,1)

M _iP represents the pixel retention score of each image block in the current picture X _i, which takes on a value from 0 to 105, the image block with pixel retention score less than 60 is marked as a reconstructed block, the X _iP corresponding to the marked image block is discarded and does not enter the encoder section, patch_ formask is an image block function with the same structure as the Patch function, the difference is that the learnable parameter of the convolution kernel in the Patch function is θ, none of the parameters in the convolution kernel in the patch_ formask function is learnable and fixed to 1, The number of output dimensions of the pixel retention score is 1, i.e., one pixel retention score for each image block, no parameter in the convolution kernel is learnable and fixed at 1 for 1 in the Patch_ formask function.

4. The method for re-identifying the blocked pedestrian based on the mask self-supervision blocked pixel reconstruction according to claim 1, wherein the prediction process of the blocked pedestrian re-identification network based on the convolution of the dynamic graph and the graph is as follows:

After all networks are trained to be converged, sending test data into a mask self-encoder fine-tuning image complement model based on mask guidance to obtain a de-blocked pedestrian image, sending the de-blocked pedestrian image into a blocked pedestrian re-recognition network based on dynamic graph and graph convolution to obtain the characteristics of each picture, sequencing each query picture according to the characteristic distance from small to large, and taking the 10 library pictures with the nearest distance as the query result of the query picture;

And taking the library pictures with the same identity labels as the correct results matched with the query picture, calculating the average accuracy of all the query pictures to obtain the average accuracy on the data set, and calculating the average of the accuracy of all the query picture features from the nearest library picture to obtain the first hit rate.