CN116935100A - A multi-label image classification method based on feature fusion and self-attention mechanism - Google Patents

Abstract

The invention discloses a multi-label image classification method based on feature fusion and self-attention mechanism. The method mainly includes the following steps: image global feature extraction, using a deep convolutional neural network to extract image global features; image local feature extraction, in On the feature map generated by the middle layer of the above-mentioned deep convolutional neural network, multiple convolution operations with a convolution kernel size of 1*1 are implemented to extract local features of the image; feature fusion is based on the self-attention mechanism to extract the global features of the image. Features and local features are fused to generate feature expressions for each category; image multi-label classification, based on the fused feature expressions, achieves the generation of image labels through fully connected layers and sigmoid activation functions. The image multi-label classification method of the present invention provides a way to fuse local features and global features of the image, which can effectively model the visual features of small targets in the image, take into account the semantic correlation between labels, and improve the image quality. Multi-label classification accuracy.

Description

A multi-label image classification method based on feature fusion and self-attention mechanism

Technical field

The invention belongs to the field of image recognition, and specifically relates to a multi-label image classification method based on the fusion of global features and local features of the image and the introduction of a self-attention mechanism.

Background technique

In the information age, images have become a medium and carrier for conveying information and are widely used in various fields. Achieving rapid and accurate classification of massive digital images in the information age is the main research content in the current image application field. Although convolutional neural networks (CNN) have shown good performance in single-label image classification tasks, most images in the real world contain more than one scene or object, and one image can be labeled with multiple labels. These labels Can correspond to different objects, scenes, actions and attributes in an image.

To extract this rich semantic information from images, you need to use image multi-label generation technology to identify all categories in the image as accurately as possible. However, traditional classification is often a hard classification, that is, a piece of data is only assigned to In a class, it is exclusive, which is reflected in image annotation by only annotating an image with one label, which has certain limitations. In addition, for a typical multi-label image, objects of different categories are located in different locations, with different proportions and postures. Occlusion, overlap, lighting and other reasons between objects will make the recognition and classification of multi-label images more difficult. Multi-label image classification is a more common and practical problem. Modeling the rich semantic information and their dependencies in images and efficiently and accurately completing the classification and recognition of multi-label images has become a key research direction (see "Jizhong, Li Huihui, He Yuqing. Zero-shot multi-label image classification based on deep example differentiation [J]. Computer Science and Exploration, 2019, 13(1): 9."), in such fields as image retrieval, portrait grouping, medical image recognition, scene It is widely used in many fields such as understanding.

The success of CNN in single-label image classification provides some inspiration for solving multi-label image classification problems. Thanks to the translation invariance of the convolution operation in CNN, no matter where the target appears in the image, it will detect the same features and output the same response. This is also true when multiple targets appear in the image. Therefore, you can simply use the Sigmoid function to convert the vector output by the fully connected layer in the CNN model into a probability value between 0 and 1, thereby calculating the probability that the sample belongs to each category. The probability distribution of each category output by the model here is independent, that is, the multi-label problem is decomposed into multiple independent two-classification problems. However, this method ignores the semantic correlation between tags, that is, when an image is annotated with a certain tag, there is a high probability that the image also has the content of another tag. For example, the sky and clouds often appear together, while water and cars almost never appear together. In addition, in deep convolutional neural networks, although multiple convolution and pooling operations reduce the number of model parameters by sharing weights and downsampling, at the same time, the receptive fields of neurons are also constantly expanding, and the feature maps of deep layers of the model are also expanding. will reflect more of the global characteristics of the image, which is advantageous in single-label image classification tasks where there is usually only a single target in the image. However, in multi-label image classification tasks, there are objects with different sizes, locations, and shapes in the image. Small targets, the local features contained in these small targets are often ignored or diluted in the larger receptive field deep in the model. Therefore, if global features are directly extracted from the entire image, it is difficult to avoid losing the visual features of small targets during the feature extraction process, thus affecting the accuracy of multi-label classification.

Xiao Lin et al. (see "Xiao Lin, Chen Boli, Huang Xin, et al. Multi-label text classification based on label semantic attention [J]. Journal of Software, 2020, 31(4): 11.") proposed a method based on label semantic attention. The method of multi-label text classification relies on the text of the document and the corresponding label, uses two-way long short-term memory to obtain the hidden representation of each word, and obtains the weight of each word in the document by using the label semantic attention mechanism. In addition, the label is semantically Spaces are often interconnected. Zhang Yong et al. (see "Zhang Yong, Liu Haoke, Zhang Jie. Multi-label classification algorithm based on generic characteristics and instance correlation [J]. Pattern Recognition and Artificial Intelligence, 2020, 33(5): 10.") proposed based on The multi-label classification algorithm of generic features and instance correlation not only considers the correlation of labels but also the correlation of instance features, and learns the similarity of the instance feature space by constructing a similarity graph. Mou Jiapeng et al. (see "Mou Jiapeng, Cai Jian, Yu Mengchi, Xu Jian. Multi-label classification algorithm for generic attributes based on label correlation [J]. Computer Application Research, 2020, 37(9): 4.") proposed a method based on Label correlation multi-label classification algorithm for generic attributes. This algorithm uses label distance to measure the correlation between labels, and completes the introduction of label correlation by appending relevant labels to the generic attribute space to achieve the purpose of improving classification performance. . Chen et al. (see "ChenZ M, Wei ") proposes to use graph convolutional networks (GCN) to explicitly model the correlation between category labels, and learn interdependent target classifiers based on the mapping function of GCN. The generated classifier can be applied to any CNN model learned Image features with high scalability and flexibility. Lanchantin et al. (see "Lanchantin J, Wang T, Ordonez V, et al. General multi-label image classification with transformers[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021: 16478-16488." ) proposed to use the Transformer model and the Label Mask Training training strategy to randomly mask some real labels during training, allowing the model to predict the masked labels, thereby exploring the complex dependencies between image features and labels and within the label set.

In view of the problem that general methods will lose the visual features of some small targets in the image in the process of extracting global features of the image, and considering the dependence between labels in multi-label classification problems, it is necessary to design an efficient multi-label method. Image classification model to effectively model the local features possessed by small objects in images and the dependencies between multiple labels.

Contents of the invention

In order to overcome the shortcomings of the above-mentioned existing technologies, the present invention proposes a multi-label image classification method based on feature fusion and self-attention mechanism. This method combines the global features of the image extracted by the deep convolutional neural network and the local features of the image. The implicit representation of the middle layer of the network is combined, and a self-attention mechanism is introduced to model the dependencies between multiple labels, resulting in higher multi-label classification accuracy.

In order to achieve the above objects, the present invention adopts the following technical solutions:

Step 1: Initialize the ResNet50 model structure and parameters, and implement a 1*1 convolution operation on the feature map output by the third convolution block of ResNet50 to extract local features of the image. At this time, the number of channels of the 1*1 convolution kernel should be consistent with the total number of categories in the current multi-label classification task.

Step 2: The feature map output by the original ResNet50 model continues to pass through subsequent convolution blocks and average pooling (Average Pooling) to obtain the global feature matrix of the image.

Step 3: In order to explore the dependencies between labels, the above feature vectors are fused through the self-attention mechanism, which specifically includes the following steps:

(1) Flatten the local feature matrix of the image obtained in step 1 above into a one-dimensional vector in each dimension of the channel dimension; flatten the global feature matrix of the image obtained in step 2 above into a one-dimensional vector; splice these vectors row by row into A matrix E, in which the local feature vectors should be linearly transformed so that their dimensions are consistent with the global feature vector dimensions;

(2) Initialize the weight matrices W ^Q , W ^K , W ^V ;

(3) Multiply the feature matrix E described in (1) with the weight vectors W ^Q , W ^K , and W ^V described in (2) respectively, and finally obtain the Query matrix, Key matrix and Value matrix. Each row vector of the matrix is Associated with the aforementioned one-dimensional global feature vector or local feature vector.

(4) Calculate the attention score. Multiply the Query matrix and the transpose of the Key matrix to obtain the attention score matrix Score and divide it by the value. ( ^dk is the number of columns of the Key matrix), and then use the Softmax function to normalize each row in the matrix. At this time, the value of each element in the Score matrix represents the attention score between the two feature vectors in the feature matrix E;

(5) Multiply the attention score matrix Score and the Value matrix. In this way, each row vector in the Value matrix is obtained by the weighted sum of the attention score and other row vectors.

Step 4: Input the Value matrix into a fully connected neural network for calculation. Finally, through the Sigmoid activation function, a vector with a dimension equal to the number of categories is generated for each image. The value of each dimension of the vector represents that the image belongs to the corresponding category. Probability.

The beneficial effects of the present invention are specifically stated as follows:

(1) This invention solves the problem of loss of small target feature information in traditional image feature extraction networks to a certain extent by simultaneously taking into account the global features and local features of the image;

(2) The present invention performs convolution operations through a convolution kernel of 1*1 size with the number of channels equal to the total number of categories, and independently calculates feature maps for each category. Compared with the general method of sharing feature maps, the classification accuracy is improved;

(3) The present invention uses a self-attention mechanism to model the dependencies between multiple labels, and uses the semantic correlation of labels in multi-label classification problems to significantly improve the classification performance of the model;

(4) The present invention has good anti-interference ability and strong robustness, and can meet the needs of actual multi-label image classification applications.

Description of the drawings

Figure 1 is a flow chart of a multi-label image classification method based on feature fusion and self-attention mechanism.

Figure 2 is a flow chart of the feature fusion process based on the self-attention mechanism.

Figure 3 is a schematic diagram of the neural network model of the multi-label image classification method based on feature fusion and self-attention mechanism.

Detailed ways

The present invention will be further clarified below by taking the accompanying drawings and specific implementations as examples. It should be understood that these embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention. After reading the present invention, those skilled in the art will be familiar with various aspects of the present invention. Modifications in various equivalent forms all fall within the scope defined by the appended claims of this application.

The following takes the total number of classification categories as 5 as an example for illustration.

S1: Initialize the ResNet50 model structure and parameters. The parameters here refer to the weight data obtained by ResNet50 pre-trained on the ImageNet large-scale visual recognition data set. Afterwards, a 1*1 convolution operation is implemented on the feature map output by the third convolution block of ResNet50 to extract local features of the image. Specifically, a convolution kernel with a shape of 5*1*1 is used to perform a convolution operation on the feature map with a shape of 512*28*28, and a feature map in the form of 5*28*28 can be obtained, with each channel of corresponding categories.

S2: The feature map output by the original ResNet50 model continues to pass through subsequent convolution blocks and average pooling (Average Pooling) to obtain the global feature map of the image, with a shape of 1*1*2048.

S3: In order to explore the dependencies between labels, the above feature vectors are fused through the self-attention mechanism, which specifically includes the following steps:

S31 flatten the image local feature matrix obtained in the above step S1 into a one-dimensional vector in each dimension of the channel dimension, that is, flatten the feature map of the shape 5*28*28 into 5*784, recorded as F ^regional ; The image global feature matrix obtained in step S2 is flattened into a 2048-dimensional one-dimensional vector, recorded as F ^global ; these vectors are spliced row by row into a matrix E, in which the local feature vectors should be linearly transformed so that their dimensions are the same as the global feature vector dimensions. Consistent, the dimension of the E matrix should be 6*2048;

Define a 784*2048-dimensional parameter matrix θ, and linearly transform the feature matrix F ^regional :

F′＝ ^Fregional ·θ (9)

E＝(F′；F ^global ) _concat (10)

S32 initializes the 2048*512-dimensional weight matrix W ^Q , W ^K , and W ^V ;

S33 Multiply the feature matrix E mentioned in S31 with the weight vectors W ^Q , W ^K , and W ^V mentioned in S32 respectively, and finally obtain the Query matrix, Key matrix and Value matrix. Each row vector of the matrix is related to the aforementioned one-dimensional global feature. Vectors or local eigenvectors are associated. The matrix dimensions are all 6*512.

The specific calculation method is:

Query＝E·W ^Q (11)

Key＝E·W ^K (12)

Value＝E·W ^V (13)

S34 Calculate attention score. Multiply the Query matrix and the transpose of the Key matrix to obtain the attention score matrix Score (the dimension should be 6*6), and divide it by the value ( ^dk is the number of columns of the Key matrix), and then use the Softmax function to normalize each row in the matrix. At this time, the value of each element in the Score matrix represents the attention score between the two feature vectors in the feature matrix E;

The specific calculation method is:

S35 Multiply the attention score matrix Score and the Value matrix. In this way, each row vector in the resulting matrix F ^mixed is obtained by the weighted sum of the attention score and other corresponding row vectors in the Value matrix.

F ^mixed =Score·Value (15)

S4: Input the F ^mixed matrix into a fully connected neural network for calculation. Finally, through the Sigmoid activation function, a vector with a dimension equal to the number of categories is generated for each image. The value of each dimension of the vector represents that the image belongs to the corresponding category. Probability. The fully connected neural network is specifically defined as follows, where the dimension of the parameter matrix ω ^c is 512*256, the dimension of ω ^b is 256*64, and the dimension of ω ^a is 64*5:

out=Sigmoid(((F ^mixed ·ω ^c )x _elu ·ω ^b ) _relu ·ω ^a ) (16).

Claims (5)

1. a multi-label image classification method based on feature fusion and self-attention mechanism is characterized by comprising the following steps:

step 1: initializing a model and extracting local features of an image;

step 2: extracting global features of the image;

step 3: fusing global features and local features of the image through a self-attention mechanism;

step 4: and based on the fused characteristics, performing image multi-label classification by using a fully connected neural network.

2. The multi-label image classification method based on feature fusion and self-attention mechanism according to claim 1, wherein the step 1 initializes a model, extracts image local features, and the method comprises the following steps:

the ResNet50 model structure and parameters are initialized, where the parameters refer to weight data from the ResNet50 pre-training on the ImageNet large-scale visual recognition dataset. Thereafter, a 1*1 convolution operation is performed on the feature map output by the third convolution block of ResNet50 to extract image local features.

3. The multi-label image classification method based on feature fusion and self-attention mechanism according to claim 1, wherein the step 2 extracts global features of the image, and the method is as follows:

and (3) continuously passing the feature map output by the original ResNet50 model in the step (1) through a subsequent convolution block, and carrying out Average Pooling (Average Pooling) to obtain a global feature matrix of the image.

4. The multi-label image classification method based on feature fusion and self-attention mechanism according to claim 1, wherein the step 3 fuses the global features and the local features of the image by the self-attention mechanism, and the method is as follows:

(1) Flattening the image local feature matrix obtained in the step 2 into one-dimensional vectors in each dimension of the channel dimension respectively, and marking as F ^regional The method comprises the steps of carrying out a first treatment on the surface of the Flattening the global feature matrix of the image obtained in the step 3 into a one-dimensional vector, and marking the one-dimensional vector as F ^global The method comprises the steps of carrying out a first treatment on the surface of the Splicing the vectors into a matrix E according to rows, wherein the local eigenvectors are subjected to linear transformation to ensure that the dimensions of the local eigenvectors are consistent with those of the global eigenvectors; defining a parameter matrix theta and a characteristic matrix F ^regional Performing linear transformation:

F′＝F ^regional ·Θ (1)

E＝(F′；F ^global ) _concat (2)

(2) Initializing a weight matrix W ^Q 、W ^K 、W ^V ；

(3) Respectively combining (1) the feature matrix E with (2) the weight vector W ^Q 、W ^K 、W ^V Multiplying to obtain a Query matrix, a Key matrix and a Value matrix, wherein each row of vectors of the matrix are associated with the one-dimensional global feature vector or the local feature vector.

The specific calculation method comprises the following steps:

Query＝E·W ^Q (3)

Key＝E·W ^K (4)

Value＝E·W ^V (5)

(4) An attention score is calculated. The Query matrix and KeyTransposed multiplication of the matrices to obtain the attention Score matrix Score, and division by the value(d ^k The number of columns of the Key matrix) and then normalize each row in the matrix using the Softmax function. At this time, the numerical value of each element in the Score matrix represents the attention Score between every two feature vectors in the feature matrix E;

the specific calculation method comprises the following steps:

(5) Multiplying the attention Score matrix Score by the Value matrix, so that the resulting matrix F ^mixed Each row vector in the Value matrix is obtained by weighted summation of the attention score and other corresponding row vectors in the Value matrix.

F ^mixed ＝Score·Value (7)

5. The method for classifying images according to claim 1, wherein the step 4 is based on the fused features and uses a fully connected neural network to classify the images, and the method is as follows:

will F ^mixed The matrix is input into a fully-connected neural network for calculation, finally, a vector with the dimension equal to the category number is generated for each image through a Sigmoid activation function, and the numerical value of each dimension of the vector represents the probability that the image belongs to the corresponding category. The fully-connected neural network is specifically defined as follows:

out＝Sigmoid(((F ^mixed ·ω ^c ) _relu ·ω ^b ) _relu ·ω ^a ) (8)。