CN111369563B - A Semantic Segmentation Method Based on Pyramid Atrous Convolutional Network - Google Patents

Abstract

The invention discloses a semantic segmentation method based on a pyramid hole convolution network, comprising the following steps: obtaining a medical image data set containing real segmentation results, performing preprocessing operations such as data enhancement on the data set; passing the preprocessed image through residual The shallow image features are obtained by the difference recursive convolution module and the pooling layer; the deep image features are obtained through the parallel network of the pyramid pooling module and the hole convolution module; the deconvolution layer, the skip connection and the residual recursive convolution module Deep image feature decoding; input the decoding result to the softmax layer to obtain the category of each pixel; train the pyramid hole convolution network, establish a loss function, and determine the network parameters through training samples; input the test image into the trained pyramid hole convolution network , to get the semantic segmentation result of the image. The atrous convolution and pyramid pooling methods adopted in the present invention can effectively extract multi-scale semantic information and detailed information, and improve the segmentation effect of the network.

Description

A semantic segmentation method based on pyramid atrous convolutional network

Technical Field

The present invention relates to the field of computer vision technology, and in particular to a semantic segmentation method based on a pyramid hole convolutional network.

Background Art

In recent years, with the rapid development of deep learning technology, its application in the field of medical image analysis has become more and more extensive. Among them, semantic segmentation technology plays a huge role in various application scenarios such as treatment planning, disease diagnosis, and pathological research. For medical images, accurate identification of the categories of objects in the image requires professional knowledge background and takes a certain amount of time for the professional authority. Through the study of semantic segmentation technology, it is possible to automatically and accurately segment the input medical images, so that doctors can make more accurate judgments and design better treatment plans.

Traditional semantic segmentation algorithms include watershed-based segmentation methods, clustering-based segmentation methods, and statistical feature-based segmentation methods. However, with the development of deep learning technology, semantic segmentation methods based on CNNs models have become mainstream. In particular, with the introduction of FCN networks, a new door has been opened for the development of semantic segmentation technology. More and more researchers have proposed many improved semantic segmentation models based on FCN models. In particular, the U-Net model is widely used in the field of medical image semantic segmentation because it has the advantage of being able to perform well even when the training set is relatively small.

In the encoder structure of the U-Net model, downsampling is performed by maximum pooling. Pooling can increase the receptive field and obtain deeper semantic information. However, after pooling, the resolution of the feature map of the image will also be reduced accordingly, resulting in the loss of detail information. Although multi-scale detail information is obtained through skip connections in the U-Net network, it still leads to the loss of boundary position information and the decline of the model's spatial discrimination ability.

In the process of proposing the present invention, it is at least found that the dilated convolution is widely used because it can increase the receptive field without reducing the resolution of the feature image. At the same time, in order to further improve the effect of the U-Net model, attention mechanism, pyramid pooling module, recursive convolution, residual connection, dense connection and other technologies are also used to combine with the U-Net model.

Summary of the invention

The purpose of the present invention is to solve the above-mentioned defects in the prior art and provide a semantic segmentation method based on a pyramid dilated convolutional network, which extracts features of different scales by using multiple residual recursive convolution modules, dilated convolution modules, and pyramid pooling modules, and then uses multi-layer upsampling and skip connections to restore the size of the feature image.

The technical purpose of the present invention is achieved through the following technical solutions:

A semantic segmentation method based on a pyramid dilated convolutional network, wherein the pyramid dilated convolutional network comprises a first residual recursive convolutional module, a second residual recursive convolutional module, a pooling layer, a pyramid pooling module, a dilated convolutional module, a deconvolutional layer, a third residual recursive convolutional module, a fourth residual recursive convolutional module, and a softmax prediction layer, wherein the structural connection mode is as follows: the first residual recursive convolutional module is sequentially connected in series with the pooling layer, the second residual recursive convolutional module, and the pooling layer, the pyramid pooling module and the dilated convolutional module are connected in parallel and then connected in series with the aforementioned pooling layer, and then the deconvolutional layer, the third residual recursive convolutional module, the deconvolutional layer, the fourth residual recursive convolutional module, and the softmax prediction layer are sequentially connected in series; the semantic segmentation method comprises the following steps:

S1. Obtain a medical image dataset containing real segmentation results, and perform preprocessing operations on the dataset to achieve data enhancement;

S2, passing the preprocessed image through the first residual recursive convolution module, the pooling layer, the second residual recursive convolution module, and the pooling layer in sequence, extracting the semantic information of the image at multiple scales, and obtaining shallow image features F ₁₁ , F ₁₂ , F ₂₁ , and F ₂₂ respectively;

S3, passing the image feature _F22 through a network consisting of a pyramid pooling module and a dilated convolution module in parallel, wherein the image feature _F22 is passed through the pyramid pooling module to obtain the image feature _F3 , and the image feature _F22 is passed through the dilated convolution module to obtain the image feature _F4 ; performing a channel-by-channel aggregation operation on the image features _F3 and _F4 , and then passing through a convolution layer with a convolution kernel of 1×1 to obtain a deep image feature _F5 , so as to further extract deep semantic information;

S4, passing the image feature _F5 through a deconvolution layer, and then performing a channel-by-channel aggregation operation with the shallow image feature _F21 transmitted through the jump connection to obtain the image feature _F61 ; then passing the image feature _F61 through the third residual recursive convolution module to obtain the image feature _F62 , wherein the jump connection directly transmits the shallow feature and performs channel-by-channel aggregation with the result after passing through the deconvolution layer; by using the jump connection, more detail information of the original image can be retained in the output image feature, thereby making the boundary of the predicted segmented image smoother.

S5, passing the image feature F ₆₂ through a deconvolution layer, and then performing a channel-by-channel aggregation operation with the shallow image feature F ₁₁ transmitted through the skip connection to obtain the image feature F ₇₁ ; then passing the image feature F ₇₁ through the fourth residual recursive convolution module to obtain the image feature F ₇₂ ;

S6, input the image feature F ₇₂ into the softmax prediction layer to obtain the category to which each pixel in the original input image belongs;

S7, train the pyramid hole convolution network, establish the loss function, and determine the network parameters through training samples;

S8. Input the test image to be segmented into the trained pyramid hollow convolutional network to obtain the semantic segmentation result of the image.

Furthermore, the preprocessing operations in step S1 include rotation, slicing, normalization, and adaptive histogram equalization.

Furthermore, the first residual recursive convolution module, the second residual recursive convolution module, the third residual recursive convolution module, and the fourth residual recursive convolution module have the same structure, and each residual recursive convolution module first passes the input through two series-connected recursive convolution layers, and then adds the input in a residual manner to obtain the output; the structural connection of the recursive convolution layer is conv, ReLU, Add, conv, ReLU, Add, conv, ReLU, where conv is a convolution layer with a convolution kernel of 3×3, and Add is pixel-by-pixel addition to the input. Compared with using ordinary convolution layers, using residual connections can help train deeper networks, and using recursive convolution networks can better extract semantic information contained in images.

Further, the pyramid pooling module in step S3 includes four adaptive average pooling layers with different pooling sizes, which are used to obtain the semantic information contained in the image feature _F22 obtained in step S2 from multiple scales. The pooling sizes adopted by the four pooling layers are N, N/2, N/3, and N/6, respectively, where N represents the resolution size of the image feature _F22 ; the image features of different sizes obtained by different pooling layers are respectively passed through a convolution layer with a convolution kernel of 1×1, and then transposed convolution is performed to obtain image features _F31 , _F32 , _F33 , and _F34 of the same size as the image feature _F22 , and then the upsampling results of each scale are aggregated with the input image feature _F22 . Finally, in order to reduce the number of channels, the aggregated image features are passed through a convolution layer with a convolution kernel of 3×3 to obtain the image feature _F3 , that is, _F3 ＝Conv(Concatenate( _F22 , _F31 , _F32 , _F33 , F34 ₎ )), where Concatenate is an aggregation operation and Conv is a 3×3 convolution operation. By performing pooling operations at multiple scales, the detailed information and deeper semantic information contained in the image can be better obtained.

Furthermore, in the step S3, the dilated convolution module is composed of three dilated convolution units with different dilated factors connected in series, the dilated factors of the three dilated convolution units are 1, 2, and 4, respectively, and the size of the dilated convolution kernel is 3×3; after the image feature F ₂₂ is input, the image features obtained by the three dilated convolution units are F ₄₁ , F ₄₂ , and F ₄₃ , respectively; the dilated convolution units are connected in a densely connected manner, and the densely connected manner is that the input of each dilated convolution unit is added to the output of the dilated convolution unit as the output; after passing through the dilated convolution module, an image feature F ₄ with a resolution equal to that of the image feature F ₂₂ can be obtained, F ₄ =Add(F ₂₂ ,F ₄₁ ,F ₄₂ ,F ₄₃ ), where Add is a pixel-by-pixel addition operation. By using dilated convolution instead of ordinary convolution plus pooling, we can not only obtain deeper semantic information by increasing the receptive field, but also avoid the problem of losing detail information due to reduced resolution caused by pooling operations.

Furthermore, the deconvolution layer in step S4 and step S5 adopts a transposed convolution.

Furthermore, in step S6, the established pyramid hollow convolutional network is trained end-to-end, the training strategy adopts the stochastic gradient descent algorithm, and the loss function uses categorical_crossentropy, and the formula is:

表示体素f_s是否属于类别k，

表示体素f_s属于类别k的可能性大小。Among them, l _c represents the category cross entropy loss of the segmented feature map F _s , f _s represents a voxel in the feature map F _s , M is the number of voxels in the feature map F _s , K is the number of categories,

Indicates whether the voxel _fs belongs to category k,

It represents the possibility that voxel _fs belongs to category k.

Compared with the prior art, the present invention has the following advantages and effects:

(1) The present invention uses a dilated convolution module to extract deep semantic information. Compared with the traditional convolution + pooling method, the dilated convolution module can increase the receptive field without reducing the resolution. At the same time, the dilated convolution module contains three dilated convolution layers with different dilation factors, and the three dilated convolution layers are connected in a densely connected manner, so that semantic information can be obtained at multiple scales.

(2) The present invention combines the use of a pyramid spatial pooling module to extract information of multiple scales contained in an image, thereby effectively obtaining deep semantic information and shallow detail information contained in the image.

(3) The present invention uses residual recursive convolution to replace ordinary convolution, which can help train deeper network structures and obtain better feature representation capabilities for segmentation tasks.

(4) The residual recursive convolution, dilated convolution, pyramid pooling and other modules included in the present invention are an algorithm that can be trained end-to-end. Compared with the two-stage algorithm, it has a smaller number of parameters and is more convenient to train.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a flow chart of a semantic segmentation method based on a pyramid dilated convolutional network disclosed in the present invention;

FIG. 2( a ) is a schematic diagram of a residual recursive convolution module according to an embodiment of the present invention, and FIG. 2( b ) is a schematic diagram of a recursive convolution unit used in FIG. 2( a );

FIG3 is a schematic diagram of a spatial pyramid pooling module in an embodiment of the present invention;

FIG4 is a simplified schematic diagram of a dilated convolution module in an embodiment of the present invention.

DETAILED DESCRIPTION

In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

As shown in FIG1 , this embodiment provides a semantic segmentation method based on a pyramid dilated convolutional network, which specifically includes the following steps:

S1. Obtain a medical image dataset containing real segmentation results, and perform preprocessing operations such as data enhancement on the dataset. Since most medical image datasets have the characteristics of small capacity and low contrast, the images in the dataset are first rotated, sliced, standardized, and adaptively histogram equalized.

S2. The preprocessed image is sequentially passed through the first residual recursive convolution module, the pooling layer, the second residual recursive convolution module, and the pooling layer to extract the semantic information of the image at multiple scales, and obtain shallow image features _F11 , _F12 , _F21 , and _F22 respectively. Specifically, as shown in FIG2(a), the residual recursive convolution module first passes the input through two serially connected recursive convolution layers, and then adds the input in a residual manner to obtain the output; wherein, as shown in FIG2(b), the structural connection of the recursive convolution unit is conv, ReLU, Add, conv, ReLU, Add, conv, and ReLU, wherein conv is a convolution layer with a convolution kernel of 3×3, and Add is pixel-by-pixel addition to the input; the pooling layer adopts a maximum pooling layer with a step size of 2.

S3, passing the image feature _F22 through a network consisting of a pyramid pooling module and a dilated convolution module in parallel, wherein the image feature _F22 is passed through the pyramid pooling module to obtain the image feature _F3 , and the image feature _F22 is passed through the dilated convolution module to obtain the image feature _F4 ; then performing a channel-by-channel aggregation operation on the image features _F3 and _F4 and passing them through a convolution layer with a convolution kernel of 1×1 to obtain a deep image feature _F5 , so as to further extract deep semantic information; specifically:

As shown in FIG3 , the pyramid pooling module includes four adaptive average pooling layers with different pooling sizes (the adaptive average pooling layer is avgpool in FIG3 ), which are used to obtain the semantic information contained in the image feature F ₂₂ obtained in step S2 from multiple scales. The pooling sizes adopted by the four pooling layers are N, N/2, N/3, and N/6, respectively, where N represents the resolution size of the image feature F _22. The image features of different sizes obtained by different pooling layers are respectively passed through a convolution layer with a convolution kernel of 1×1 (i.e., conv 1×1 in FIG3 ), and then transposed convolution is performed (transposed convolution is up-conv in FIG3 ) to obtain image features F ₃₁ , F ₃₂ , F ₃₃ , and F ₃₄ of the same size as the image feature F _22. The upsampling results of each scale are then aggregated with the input image feature F _22. Finally, in order to reduce the number of channels, the aggregated image features are passed through a convolution layer with a convolution kernel of 3×3 to obtain the image feature F ₃ , i.e., F ₃ =Conv(Concatenate(F ₂₂ ,F ₃₁ ,F ₃₂ ,F ₃₃ ,F ₃₄ )), where Concatenate is an aggregation operation and Conv is a 3×3 convolution operation.

As shown in Figure 4, the dilated convolution module is composed of three dilated convolution units with different dilated factors connected in series, the dilated factors of the three dilated convolution units are 1, 2, and 4 respectively, and the size of the dilated convolution kernel is 3×3; after the image feature F ₂₂ is input, the image features obtained after the three dilated convolution units are F ₄₁ , F ₄₂ , and F ₄₃ respectively; the dilated convolution units are connected in a densely connected manner, that is, the input of each dilated convolution unit is added to the output of the dilated convolution unit as the output; after passing through the dilated convolution module, an image feature F ₄ with a resolution equal to that of the image feature F ₂₂ can be obtained, F ₄ =Add(F ₂₂ ,F ₄₁ ,F ₄₂ ,F ₄₃ ), where Add is a pixel-by-pixel addition operation.

In this embodiment, the channel-by-channel aggregation operation refers to the aggregation operation being performed on the channel dimension, that is, assuming that the number of channels of the image feature _F3 is _C1 and the number of channels of the image feature _F4 is _C2 , the number of channels of the image feature obtained after aggregation is _C1 + _C2 .

S4, pass the image feature _F5 through a deconvolution layer, and then perform a channel-by-channel aggregation operation with the shallow image feature _F21 transmitted through the jump connection to obtain the image feature _F61 ; then pass the image feature _F61 through the third residual recursive convolution module to obtain the image feature _F62 ; specifically: the deconvolution layer adopts a transposed convolution; the jump connection directly transmits the shallow feature, and performs a channel-by-channel aggregation with the result after passing through the deconvolution layer, and the channel-by-channel aggregation operation is as described in the previous step S3.

S5. Pass the image feature _F62 through a deconvolution layer, and then perform a channel-by-channel aggregation operation with the shallow image feature _F11 transmitted through the jump connection to obtain the image feature _F71 ; then pass the image feature _F71 through the fourth residual recursive convolution module to obtain the image feature _F72 ; specifically: the deconvolution layer adopts a transposed convolution; the jump connection directly transmits the shallow feature, and performs a channel-by-channel aggregation with the result after passing through the deconvolution layer, and the channel-by-channel aggregation operation is as described in the previous step S3.

S6. Input the image feature F ₇₂ into the softmax prediction layer to obtain the category to which each pixel in the original input image belongs.

S7. Train the pyramid dilated convolutional network, establish a loss function, and determine the network parameters through training samples. The network parameters specifically include learning rate, weight drop, momentum term, and training strategy. Perform end-to-end training on the established pyramid dilated convolutional network. The training strategy uses the stochastic gradient descent algorithm. The initial learning rate is set to 0.001, the weight drop is set to 10 ^-4 , and a momentum term of 0.9 is added. The loss function uses categorical_crossentropy. The difference from the original cross entropy loss function is that categorical_crossentropy adds a corresponding loss weight v ^k to the loss function of the k ^th category voxel. The weight is inversely proportional to the voxel belonging to the k ^th category. The formula is:

表示体素f_s是否属于类别k，

表示体素f_s属于类别k的可能性大小。Among them, l _c represents the category cross entropy loss of the segmented feature map F _s , f _s represents a voxel in the feature map F _s , M is the number of voxels in the feature map F _s , K is the number of categories,

Indicates whether the voxel _fs belongs to category k,

It represents the possibility that voxel _fs belongs to category k.

S8. Input the test image to be segmented into the trained pyramid hollow convolutional network to obtain the semantic segmentation result of the image.

In summary, the present embodiment discloses a semantic segmentation method based on a pyramid dilated convolutional network, proposes and trains a pyramid dilated convolutional network, establishes a loss function, and determines network parameters through training samples; the test image is input into the trained pyramid dilated convolutional network to obtain the semantic segmentation result of the image. The dilated convolution and pyramid pooling methods used in the present embodiment can effectively extract multi-scale semantic information and detail information, and improve the segmentation effect of the network.

The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.

Claims (7)

1. The semantic segmentation method based on the pyramid hole convolution network is characterized in that the pyramid hole convolution network comprises a first residual error recursive convolution module, a second residual error recursive convolution module, a pooling layer, a pyramid pooling module, a hole convolution module, an deconvolution layer, a third residual error recursive convolution module, a fourth residual error recursive convolution module and a softmax prediction layer, and the structural connection mode is as follows: the pyramid pooling module and the cavity convolution module are connected in parallel and then connected in series with the pooling layer, and then sequentially connected in series with the deconvolution layer, the third residual recursive convolution module, the deconvolution layer, the fourth residual recursive convolution module and the softmax prediction layer; the semantic segmentation method comprises the following steps:

s1, acquiring a medical image data set containing a real segmentation result, and preprocessing the data set to realize data enhancement;

s2, the preprocessed image sequentially passes through the first residual recursive convolution module, the pooling layer, the second residual recursive convolution module and the pooling layer, the semantic information of the image is extracted in a multi-scale mode, and the shallow image characteristics F are obtained respectively ₁₁ 、F ₁₂ 、F ₂₁ 、F ₂₂ ；

S3, image feature F ₂₂ By a network of pyramid pooling modules and hole convolution modules in parallel, wherein the image features F are ₂₂ Obtaining image characteristics F through a pyramid pooling module ₃ Image feature F ₂₂ Obtaining image characteristics F through a cavity convolution module ₄ (ii) a Image feature F ₃ 、F ₄ Performing aggregation operation channel by channel, and performing convolution layer with convolution kernel of 1 × 1 to obtain deep image feature F ₅ Thereby further extracting deep semantic information;

s4, image feature F ₅ Through an inverse convolution layer and then coupled with the shallow image features F delivered through the skip-join ₂₁ Performing channel-by-channel aggregation operation to obtain image feature F ₆₁ (ii) a Then image feature F ₆₁ Obtaining image characteristics F through a third residual error recursive convolution module ₆₂ Wherein, the jump connection directly transmits the shallow feature and carries out channel-by-channel aggregation with the result after passing through the reverse convolution layer;

s5, image feature F ₆₂ Through an inverse convolution layer and then coupled with the shallow image features F delivered through the skip-join ₁₁ Performing channel-by-channel aggregation operation to obtain image feature F ₇₁ (ii) a Then image feature F ₇₁ Obtaining image characteristics F through a fourth residual error recursive convolution module ₇₂ ；

S6, image feature F ₇₂ Inputting the image into a softmax prediction layer to obtain the category of each pixel in the original input image;

s7, training a pyramid cavity convolution network, establishing a loss function, and determining network parameters through training samples;

and S8, inputting the test image to be segmented into the trained pyramid cavity convolution network to obtain the semantic segmentation result of the image.

2. The method for semantic segmentation based on the pyramid hole convolutional network of claim 1, wherein the preprocessing operation in step S1 includes rotation, slicing, normalization, and adaptive histogram equalization.

3. The semantic segmentation method based on the pyramid hole convolution network according to claim 1, characterized in that the first residual recursive convolution module, the second residual recursive convolution module, the third residual recursive convolution module and the fourth residual recursive convolution module have the same structure, and each residual recursive convolution module is formed by first passing an input through two recursive convolution layers connected in series and then adding the input and the input in a residual manner to obtain an output; the structure connection of the recursive convolutional layer is conv, reLU, add, conv and ReLU in sequence, wherein conv is a convolutional layer with a convolution kernel of 3 multiplied by 3, and Add is pixel-by-pixel addition with input;

4. the semantic segmentation method based on the pyramid hole convolutional network of claim 1, wherein the pyramid pooling module in step S3 comprises four adaptive average pooling layers with different pooling sizes, and is used for obtaining the image feature F obtained in step S2 from multiple scales ₂₂ The four pooling layers adopt the pooling sizes of N, N respectivelyN2, N/3, N/6, where N represents an image feature F ₂₂ The resolution of (2); then, the image features with different sizes obtained by different pooling layers are respectively passed through a convolution layer with convolution kernel of 1 × 1, and then the transposition convolution is carried out to obtain the image features F ₂₂ Image features F of uniform size ₃₁ 、F ₃₂ 、F ₃₃ 、F ₃₄ Then the up-sampling result of each scale and the input image characteristic F are combined ₂₂ Polymerizing, and passing the polymerized image features through a convolution layer with convolution kernel of 3 × 3 to obtain image features F ₃ I.e. F ₃ ＝Conv(Concatenate(F ₂₂ ,F ₃₁ ,F ₃₂ ,F ₃₃ ,F ₃₄ ) Concatenate is an aggregation operation and Conv is a convolution operation of 3 x 3).

5. The semantic segmentation method based on the pyramid hole convolution network according to claim 1, wherein in step S3, the hole convolution module is formed by connecting three hole convolution units with different hole factors in series, the hole factors of the three hole convolution units are 1, 2 and 4, respectively, and the sizes of the hole convolution kernels are all 3 × 3; input image feature F ₂₂ Then, the image characteristics obtained by the three cavity convolution units are respectively F ₄₁ 、F ₄₂ 、F ₄₃ (ii) a The cavity convolution units are connected in a dense connection mode, wherein the dense connection mode is that the input of each cavity convolution unit is added with the output of the cavity convolution unit to be used as output; after passing through the cavity convolution module, a resolution and an image characteristic F can be obtained ₂₂ Equal image features F ₄ ，F ₄ ＝Add(F ₂₂ ,F ₄₁ ,F ₄₂ ,F ₄₃ ) Where Add is a pixel-by-pixel addition operation.

6. The method as claimed in claim 1, wherein the deconvolution layer in step S4 and step S5 is a transposed convolution.

7. The method for semantic segmentation based on the pyramid hole convolutional network of claim 1, wherein in step S6, the established pyramid hole convolutional network is trained end to end, a random gradient descent algorithm is adopted as a training strategy, and a loss function uses catagorical _ cross, and the formula is as follows:

wherein l _c Representing a segmented feature map F _s Class cross entropy loss of f _s Representation feature mapping F _s M is a feature map F _s K is the number of classes,

representing a voxel f _s Whether it belongs to the category k, or not>

Representing a voxel f _s The probability of belonging to class k. />

Images