CN114694003A - Multi-scale feature fusion method based on target detection - Google Patents

Abstract

The invention belongs to the field of target detection, and relates to a multi-scale feature fusion method based on target detection, which comprises the following steps: 1) carrying out convolution processing on the feature maps with different sizes to obtain feature maps with the same channel number; 2) respectively carrying out channel dimension fusion processing and space dimension fusion processing on the feature map obtained in the step 1) to respectively obtain a channel dimension fusion processing feature map and a space dimension fusion processing feature map; 3) fusing the channel dimension fusion processing characteristic diagram and the space dimension fusion processing characteristic diagram obtained in the step 2), so as to realize the characteristic fusion of the characteristic diagrams with different sizes in two dimensions of space and channel. The invention provides a multi-scale feature fusion method based on target detection, which can obviously improve the detection precision.

Description

A multi-scale feature fusion method based on target detection

technical field

The invention belongs to the field of target detection, and relates to a feature fusion method, in particular to a multi-scale feature fusion method based on target detection.

Background technique

The main task of object detection is to locate the objects of interest from the input image, and then accurately determine the category of each object of interest. The current target detection technology has been widely used in daily life safety, robot navigation, intelligent video surveillance, traffic scene detection, aerospace and other fields.

In the context of the development of deep learning, convolutional neural networks have been recognized by more and more people, and their applications have become more and more common. Object detection algorithms based on deep learning use convolutional neural networks (CNN) to automatically select features, and then input the features into the detector to classify and locate objects.

In a picture, the size of the objects is different. If the feature maps of the same size are used for training, the effect is very unsatisfactory. If the feature maps are cropped into different sizes and input for training multiple times, the complexity and training time of the network will be greatly increased. In the target detection algorithm, in order to solve the problem of predicting objects of different sizes, Lin et al. proposed the famous feature pyramid network FPN, as shown in Figure 1. The basic idea is to combine the fine-grained spatial information of the shallow feature map and the deep layer. The semantic information of the feature map is used to detect objects at multiple scales. On this basis, many people have proposed an improved FPN structure. Liu et al. proposed PANet, which first uses up-sampling to fuse feature maps of different sizes, and then performs down-sampling feature fusion on this basis to reconstruct a pyramid that enhances spatial information; GolnazGhaisi et al. proposed NAS-FPN , the network uses a neural network structure search in an extensible search space covering all cross-scale connections, and can fuse features across ranges; Guo et al. proposed AugFPN, which used Consistent Supervision, Residual Supervision, Residual for the shortcomings of FPN. Feature Augmentation and Soft RoISelection; Tan et al. proposed the BiFPN architecture. BiFPN also weighted and fused features, allowing the network to learn the importance of different input features; Siyuan Qiao et al. proposed Recursive-FPN, which combines traditional FPN The fused feature map is then input to Backbone for a second cycle. The pyramid network FPN can balance the classification accuracy and localization accuracy of objects, and effectively solve the multi-scale prediction problem. However, this structure only effectively fuses the features in the spatial structure, and the information between different channels may be correlated or redundant.

SUMMARY OF THE INVENTION

In order to solve the above technical problems existing in the background art, the present invention provides a multi-scale feature fusion method based on target detection that can significantly improve detection accuracy.

In order to achieve the above object, the present invention adopts the following technical solutions:

A multi-scale feature fusion method based on target detection, characterized in that: the target detection-based multi-scale feature fusion method comprises the following steps:

1) Convolve feature maps of different sizes to obtain feature maps with the same number of channels;

2) Perform channel dimension fusion processing and spatial dimension fusion processing on the feature maps obtained in step 1), respectively, to obtain channel dimension fusion processing feature maps and spatial dimension fusion processing feature maps respectively;

3) Perform fusion processing on the channel dimension fusion processing feature map and the spatial dimension fusion processing feature map obtained in step 2), so as to realize the feature fusion of the feature maps of different sizes in the two dimensions of space and channel.

As preferably, the concrete implementation mode of step 1) adopted by the present invention is:

1.1) Obtain feature maps of different sizes, which at least include feature map C3, feature map C4 and feature map C5; the size of feature map C3 is twice that of feature map C4; the size of feature map C4 is 2 times the feature map C5;

1.2) Perform convolution processing on feature maps of different sizes, so that the number of channels after convolution processing is the same.

Preferably, the specific implementation method of the channel dimension fusion processing in step 2) adopted in the present invention is: compress the feature map into a three-dimensional vector in the passband dimension, and then fuse the feature maps of different scales in the channel dimension.

Preferably, the present invention compresses the feature map into a three-dimensional vector in the passband dimension as follows:

The feature map is subjected to the unfold operation to convert the feature map from a four-dimensional tensor to a three-dimensional tensor; the dimension of the four-dimensional tensor is [B, C, H, W], and the dimension of the three-dimensional tensor is [B, C', L] ;

in:

C' satisfies the following relationship:

C'=C*K*K

in:

C is the channel size of the original feature map;

K is the size of the convolution kernel;

C' is the sliding window size;

L satisfies the following relationship:

L=H'*W'

in:

H is the length of the original feature map;

W is the width of the original feature map;

pading is the padding size;

K is the size of the convolution kernel;

stride is the step size;

L is the number of sliding windows.

Preferably, the specific implementation of the present invention to fuse feature maps of different scales in the channel dimension is as follows:

a) Feature map C3 obtains C'3 after one unfold operation; feature map C4 obtains C'4_1 and C'4_2 after two unfold operations; feature map C5 obtains C'5 after one unfold operation; among which: C' 3 and C'4_1 are equal in size, C'4_2 and C'5 are equal in size;

b) fuse C'3 and C'4_1 to obtain a three-dimensional tensor, and fuse C'4_2 and C'5 to obtain a three-dimensional tensor;

c) Restore the result of step b) to four four-dimensional tensors through reshape, wherein the size of the first four-dimensional tensor is the same as the size of the feature map C3, the size of the second four-dimensional tensor and the size of the third four-dimensional tensor They are the same size as feature map C4, and the size of the fourth four-dimensional tensor is the same as that of feature map C5;

d) The first four-dimensional tensor obtains the feature map P3_1 fused on the channel dimension; the fourth four-dimensional tensor obtains the feature map P5_1 fused on the channel dimension; the second four-dimensional tensor and the third four-dimensional tensor are added to obtain the feature map P5_1. The feature map P4_1 fused in the channel dimension; the size of the feature map P3_1, the size of the feature map P4_1, and the size of the feature map P5_1 are all non-equivalent.

Preferably, the spatial dimension fusion processing in step 2) adopted in the present invention is to upsample the feature maps, and then fuse the feature maps of different scales in the spatial dimension.

As preferably, the concrete implementation mode of spatial dimension fusion processing in step 2) adopted by the present invention is:

The feature map C5 is upsampled and the feature map C4 is fused to obtain P4_2, and the P4_2 is then upsampled and the feature map C3 is fused to obtain P3_2, and finally the feature map P3_2, feature map P4_2 and feature map P5_2 fused in the spatial dimension are obtained; The size of the feature map P3_2, the size of the feature map P4_2, and the size of the feature map P5_2 are not equivalent.

As preferably, the specific implementation mode of the fusion processing in step 3) adopted by the present invention is:

3.1) Integrate feature maps of the same size in channel dimension and spatial dimension;

3.2) Perform ablation and down-sampling operations on the integrated results of step 3.1), and finally realize feature fusion of feature maps of different sizes in the two dimensions of space and channel.

The advantages of the present invention are:

The invention provides a multi-scale feature fusion method based on target detection. The method firstly convolves feature maps of different sizes to obtain feature maps with the same number of channels, and secondly, the obtained feature maps are respectively channel dimension. Fusion processing and spatial dimension fusion processing, respectively, obtain the channel dimension fusion processing feature map and the spatial dimension fusion processing feature map, that is, two branches are used, and the first branch adopts the compression operation to compress the feature map in the passband dimension as three-dimensional vector, and then fuse the feature maps of different scales in the channel dimension; the second branch adopts the upsampling operation, and then fuses the feature maps of different scales in the spatial dimension; finally, fuse the obtained channel dimension to process the feature map And spatial dimension fusion processing feature map for fusion processing, to achieve feature fusion of feature maps of different sizes in the two dimensions of space and channel. The multi-scale feature fusion method based on target detection provided by the present invention adds a branch on the basis of FPN, the branch compresses all channel information together and fuses semantic information, and finally obtains rich semantic space information. Through experimental comparison, it is found that the network structure constructed by the present invention achieves a certain improvement on different networks, especially the detection accuracy of medium targets and large targets is significantly improved.

Description of drawings

1 is a schematic structural diagram of a feature pyramid network FPN in the prior art;

Fig. 2 is the structural representation of MFPN provided by the present invention;

3 is a schematic diagram of the overall structure of a target detection network formed based on the MFPN provided by the present invention;

4 to 7 are detection effect diagrams for different detection objects based on different target detection networks.

Detailed ways

Referring to Fig. 2, the present invention provides a multi-scale feature fusion method based on target detection, the method includes:

1) Convolve feature maps of different sizes to obtain feature maps with the same number of channels;

1.1) Obtain feature maps of different sizes. Feature maps of different sizes include feature map C3, feature map C4 and feature map C5 at least; the size of feature map C3 is twice that of feature map C4; the size of feature map C4 is the feature map 2 times of C5;

1.2) Perform convolution processing on feature maps of different sizes, so that the number of channels after convolution processing is the same.

2) Perform channel dimension fusion processing and spatial dimension fusion processing on the feature maps obtained in step 1), respectively, to obtain channel dimension fusion processing feature maps and spatial dimension fusion processing feature maps respectively;

in:

The specific implementation method of channel dimension fusion processing is to compress the feature map into a three-dimensional vector in the passband dimension, and then fuse the feature maps of different scales in the channel dimension. The specific implementation of compressing the feature map into a three-dimensional vector in the passband dimension is:

The feature map is subjected to the unfold operation to convert the feature map from a four-dimensional tensor to a three-dimensional tensor; the dimension of the four-dimensional tensor is [B, C, H, W], and the dimension of the three-dimensional tensor is [B, C', L];

in:

C' satisfies the following relationship:

C'=C*K*K

in:

C is the channel size of the original feature map;

K is the size of the convolution kernel;

C' is the sliding window size;

L satisfies the following relationship:

L=H'*W'

in:

H is the length of the original feature map;

W is the width of the original feature map;

pading is the padding size;

K is the size of the convolution kernel;

stride is the step size;

L is the number of sliding windows.

The specific implementation of fusing feature maps of different scales in the channel dimension is as follows:

a) Feature map C3 obtains C'3 after one unfold operation; feature map C4 obtains C'4_1 and C'4_2 after two unfold operations; feature map C5 obtains C'5 after one unfold operation; among which: C' 3 and C'4_1 are equal in size, C'4_2 and C'5 are equal in size;

b) fuse C'3 and C'4_1 to obtain a three-dimensional tensor, and fuse C'4_2 and C'5 to obtain a three-dimensional tensor;

Assuming that the size of C5 is a four-dimensional tensor [b, c, h, w], from the introduction of steps 1.1) and 1.2), the size of C4 is [b.c, 2*h, 2*w], and the size of C3 is [b, c, 4*h, 4*w]. According to the formula in step 1.2), after the unfold operation, the size of C'3 is [b, c*k*k, h*w], and the size of C'4_1 is [b, c*k*k, h* w], the size of C'4_2 is [b, c*k*k, 4*h*w], and the size of C'5 is [b, c*k*k, 4*h*w], so C' 3 and C'4_1 are merged to obtain a three-dimensional tensor [b, c*k*k, h*w]; C'4_2 and C'3 are merged to obtain a three-dimensional tensor [b, c*k*k, 4*h*w ];

c) Restore the result of step b) to four four-dimensional tensors through reshape, wherein the size of the first four-dimensional tensor is the same as the size of the feature map C3, the size of the second four-dimensional tensor and the size of the third four-dimensional tensor They are the same size as feature map C4, and the size of the fourth four-dimensional tensor is the same as that of feature map C5;

d) The first four-dimensional tensor obtains the feature map P3_1 fused on the channel dimension; the fourth four-dimensional tensor obtains the feature map P5_1 fused on the channel dimension; the second four-dimensional tensor and the third four-dimensional tensor are added to obtain the feature map P5_1. The feature map P4_1 fused in the channel dimension; the size of the feature map P3_1, the size of the feature map P4_1, and the size of the feature map P5_1 are all non-equivalent.

The spatial dimension fusion processing is to upsample the feature maps, and then fuse the feature maps of different scales in the spatial dimension.

The specific implementation of spatial dimension fusion processing is as follows:

The feature map C5 is upsampled and the feature map C4 is fused to obtain P4_2, and the P4_2 is then upsampled and the feature map C3 is fused to obtain P3_2, and finally the feature map P3_2, feature map P4_2 and feature map P5_2 (C5) fused in the spatial dimension are obtained. The direct output of ); the size of the feature map P3_2, the size of the feature map P4_2 and the size of the feature map P5_2 are not equivalent.

3) Perform fusion processing on the channel dimension fusion processing feature map and the spatial dimension fusion processing feature map obtained in step 2), so as to realize the feature fusion of the feature maps of different sizes in the two dimensions of space and channel, wherein the specific fusion processing is performed. The way to do it is:

3.1) Integrate feature maps of the same size in channel dimension and spatial dimension;

3.2) Perform ablation and down-sampling operations on the integrated results of step 3.1), and finally realize feature fusion of feature maps of different sizes in the two dimensions of space and channel.

Specifically, first obtain feature maps of different sizes [C3, C4, C5] (the size of C3 is 2 times that of C4, and the size of C4 is 2 times that of C5) after convolution processing, the number of channels becomes 256, and then Then two parallel branches branch1 and branch2 are used for feature fusion in different dimensions, and three feature maps of different sizes [P3_1, P4_1, P5_1] and [P3_2, P4_2, P5_2] are obtained respectively, and the features of the same size in different dimensions are obtained. Figure to obtain new feature maps [P3, P4, P5] (P3 is twice the size of P4, P4 is twice the size of P5), and finally through ablation and downsampling operations to obtain 5 different sizes of feature maps [P3 , P4, P5, P6, P7].

The technical solutions of the present invention will be further described below with reference to the accompanying drawings and embodiments.

The present invention is an optimized neural network based on MFPN network, and its steps include:

Step 1: Data input and preprocessing.

The Ms CoCo2017 dataset was used. The COCO dataset contains a total of 80 categories for detection. They are "people", "bicycles", "cars", "motorcycles", "airplanes", "buses", "trains", "trucks", "boats", "traffic lights" and other common individuals in daily life , which is a large, rich dataset for object detection, segmentation and captioning. Contains four files: annotations, test2017, train2017, and val2017. where train contains 118287 images, val contains 5000 images, and test contains 28660 images. annotations is a collection of annotation types: objectinstances, object keypoints and image captions, stored using JSON files.

Adjust the size of all images to 512x512 for multi-scale training, and use data augmentation to perform various operations on the image dataset: random flip operations, pad filling operations for images that do not meet the requirements, and random cropping for images that do not meet the specified size. operation, normalization processing operation, image distortion processing operation.

Step 2: Construction of the model.

As shown in Figure 3, a target detection network is proposed based on Vfnet, which consists of three parts: backbone, neck and heads. For different object detection networks, they usually use the same backbone and neck, but the heads of different object detection networks are different. The backbone uses resnet50, which is used to extract the features of the picture. The network outputs 4 feature maps of different sizes [C2, C3, C4, C5], the stride is [4, 8, 16, 32], and the channel size is [ 256, 512, 1024, 2048]. The Neck structure is used to connect backbones and heads for fusing features. The structure uses three feature maps [C3, C4, C5] of the backbone. After 1*1 convolution, the channels are reduced to 256, MFPN feature fusion structure, and then C5 is downsampled twice to obtain C6 and C7, and finally Use 3*3 convolution to ablate the feature map, and output 5 feature maps of different sizes [P3, P4, P5, P6, P7], the stride is [8, 16, 32, 64, 128], the channel size Both are 256. Heads are used for object detection, classification and regression of objects.

Step 3: Train and test.

The evaluation standard of the experiment adopts the average precision (Average-Precision, AP), AP50, AP75, AP_S, AP_M, AP_L as the main evaluation standard.

Experimental environment: Build a Python compilation environment with PyTorch1.6.0, torchvision=0.7.0, CUDA10.0, and CUDNN7.4 as the deep learning framework, and implement it on the platform mmdetection2.6.

Experimental equipment:

CPU: Intel Xeon E5-2683 V3@2.00GHz ;

RAM: 32GB;

Graphics card: Nvidia GTX 1080Ti;

Hard disk: 500GB;

The influence of multi-scale fusion structure on object accuracy is tested, and comparative experiments are carried out on multiple networks. The experimental results are shown in Table 1.

Table 1 The effect of MFPN on different networks

The MFPN structure is improved to varying degrees on the four networks. On the ATSS network, AP increased from 32.7% to 34%, and AP50 and AP_L even increased by 2%. The AP in FCOS has improved by 0.9% from 29.1%, and other indicators can also improve by more than 1%. The AP of Vfnet increased by 0.4% from 34.1%, but AP_L increased from 50.5% to 52.8%. The MFPN structure has the most obvious improvement in Foveabox, its AP is improved by 2.4%, and AP_L is improved from 43.8% to 47.8%. This shows that the semantic information obtained by the MFPN module is richer than that obtained by the FPN module.

MFPN has different effects on different networks. It increases AP by 0.4% on VFNet and 2.4% in AP_L on Foveabox. Think it has something to do with the benchmark network. The original network AP of VFNet is as high as 34.1%, while that of Foveabox is only 28.5%. MFPN has limited improvement for small objects, but has significant improvements in detection accuracy for medium and large objects. In the field of object detection, in order to improve the detection accuracy of small objects, larger size feature maps are required. Although branch1 of the MFPN structure integrates feature maps in the channel dimension, the sliding window of the feature map is C*K*K, and its feature map size is K*K in space, so MFPN is not ideal for improving small objects.

The present invention tests the detection effects of MFPN and other networks, and the results are shown in Figures 4 to 7 . Wherein (a) is the detection effect diagram of fcos, (b) is the detection effect diagram of atss, (c) is the detection effect diagram of foveabox, (d) is the detection effect diagram of the MFPN network provided by the present invention . In the vehicle detection in Figure 4, it can be seen that for dense objects, the MFPN network provided by the present invention can detect more objects. In the detection of a single dog in Figure 5, all networks can accurately detect the target, but the MFPN network provided by the present invention has a higher classification accuracy, reaching 96%. In the multi-object detection in FIG. 6 , the MFPN network provided by the present invention can not only detect people and bicycles well, but also detect half-occluded objects. In the graph of FIG. 7 , the MFPN network provided by the present invention can better separate the detection of humans and horses, and the detection accuracy also reaches the highest level.

Claims (8)

1. a multi-scale feature fusion method based on target detection, is characterized in that: the described multi-scale feature fusion method based on target detection comprises the following steps: 1) Convolve feature maps of different sizes to obtain feature maps with the same number of channels; 2) Perform channel dimension fusion processing and spatial dimension fusion processing on the feature maps obtained in step 1), respectively, to obtain channel dimension fusion processing feature maps and spatial dimension fusion processing feature maps respectively; 3) Perform fusion processing on the channel dimension fusion processing feature map and the spatial dimension fusion processing feature map obtained in step 2), so as to realize the feature fusion of the feature maps of different sizes in the two dimensions of space and channel. 2. the multi-scale feature fusion method based on target detection according to claim 1, is characterized in that: the concrete implementation mode of described step 1) is: 1.1) Obtain feature maps of different sizes, which at least include feature map C3, feature map C4 and feature map C5; the size of feature map C3 is twice that of feature map C4; the size of feature map C4 is 2 times the feature map C5; 1.2) Perform convolution processing on feature maps of different sizes, so that the number of channels after convolution processing is the same. 3. The multi-scale feature fusion method based on target detection according to claim 2, is characterized in that: the concrete implementation mode of channel dimension fusion processing in the described step 2) is: the feature map is compressed into a three-dimensional vector on the passband dimension , and then fuse feature maps of different scales in the channel dimension. 4. The multi-scale feature fusion method based on target detection according to claim 3, characterized in that: the specific implementation of compressing the feature map into a three-dimensional vector in the passband dimension is: The feature map is subjected to the unfold operation to convert the feature map from a four-dimensional tensor to a three-dimensional tensor; the dimension of the four-dimensional tensor is [B, C, H, W], and the dimension of the three-dimensional tensor is [B, C', L] ; in: C' satisfies the following relationship: C'=C*K*K in: C is the channel size of the original feature map; K is the size of the convolution kernel; C' is the sliding window size; L satisfies the following relationship: L=H'*W'

in: H is the length of the original feature map; W is the width of the original feature map; pading is the padding size; K is the size of the convolution kernel; stride is the step size; L is the number of sliding windows. 5. The multi-scale feature fusion method based on target detection according to claim 4, characterized in that: the specific implementation of the feature maps of different scales in channel dimension fusion is: a) The feature map C3 obtains C'3 after one unfold operation; the feature map C4 obtains C'4_1 and C'4_2 after two unfold operations; the feature map C5 obtains C'5 after one unfold operation; among which: C' 3 and C'4_1 are equal in size, C'4_2 and C'5 are equal in size; b) fuse C'3 and C'4_1 to obtain a three-dimensional tensor, and fuse C'4_2 and C'5 to obtain a three-dimensional tensor; c) Restore the result of step b) to four four-dimensional tensors by reshape, wherein the size of the first four-dimensional tensor is the same as that of the feature map C3, the size of the second four-dimensional tensor and the size of the third four-dimensional tensor They are the same size as feature map C4 respectively, and the size of the fourth four-dimensional tensor is the same as that of feature map C5; d) The first four-dimensional tensor obtains the feature map P3_1 fused on the channel dimension; the fourth four-dimensional tensor obtains the feature map P5_1 fused on the channel dimension; the second four-dimensional tensor and the third four-dimensional tensor are added to obtain the feature map P5_1. The feature map P4_1 fused in the channel dimension; the size of the feature map P3_1, the size of the feature map P4_1, and the size of the feature map P5_1 are all non-equivalent. 6. The multi-scale feature fusion method based on target detection according to claim 1, characterized in that: in the step 2), the spatial dimension fusion processing is to upsample the feature maps, and then the feature maps of different scales are stored in the space. Dimensional fusion. 7. The multi-scale feature fusion method based on target detection according to claim 6, characterized in that: the specific implementation of spatial dimension fusion processing in the step 2) is: The feature map C5 is upsampled and the feature map C4 is fused to obtain P4_2, and the P4_2 is then upsampled and the feature map C3 is fused to obtain P3_2, and finally the feature map P3_2, feature map P4_2 and feature map P5_2 fused in the spatial dimension are obtained; The size of the feature map P3_2, the size of the feature map P4_2, and the size of the feature map P5_2 are not equivalent. 8. The multi-scale feature fusion method based on target detection according to any one of claims 1-7, wherein the specific implementation of the fusion processing in the step 3) is: 3.1) Integrate feature maps of the same size in channel dimension and spatial dimension; 3.2) Perform ablation and down-sampling operations on the integrated results of step 3.1), and finally realize feature fusion of feature maps of different sizes in the two dimensions of space and channel.

Images