CN110599499B - MRI image heart structure segmentation method based on multipath convolutional neural network - Google Patents

Abstract

The invention relates to an MRI image heart structure segmentation method based on a multi-path convolution neural network, which comprises the steps of collecting heart movie MRI training data of normal people and heart patients, manually marking the heart structures in the training data by experienced doctors to be used as heart segmentation marking results, training a heart region extraction model based on the training data to enable the heart region extraction model to accurately extract heart regions, training the heart segmentation network according to the heart regions extracted from the training data to segment various structures of a heart, and measuring the segmentation performance of the constructed heart segmentation network by using standard segmentation marking results as standards. According to the method, the heart is extracted by adopting the heart region extraction model based on the generated countermeasure network, so that the accuracy of heart region extraction is improved; meanwhile, context information between adjacent layers is utilized through the multi-path convolutional neural network, and the segmentation precision and accuracy are improved.

Description

A segmentation method of cardiac structure in MRI images based on multi-channel convolutional neural network

technical field

The invention relates to the field of medical image processing, in particular to a method for segmenting a cardiac structure of an MRI image based on a multi-channel convolutional neural network.

Background technique

According to the World Health Organization, cardiovascular disease is the disease with the highest fatality rate in the world, and about 19.7 million people died of cardiovascular disease in 2016. In clinical practice, cardiac function analysis plays an important role in cardiac disease diagnosis, risk assessment, patient management, and treatment decision-making. This is typically the quantitative analysis of global or local cardiac function with the aid of digital images of the heart by assessing a range of clinical indicators such as ventricular volume, ejection fraction, stroke volume, myocardial mass, etc. Due to the good discrimination of soft tissue, the evaluation of left and right ventricular ejection fraction, stroke volume, left ventricular mass, and myocardial thickness through cine MRI images has become the gold standard for cardiac function analysis. The evaluation of these quantitative indicators requires The endocardium and epicardium of the left ventricle, as well as the endocardium of the right ventricle, were accurately segmented during both diastolic and end-systolic phases. In clinical practice, manual segmentation by doctors is not only time-consuming and labor-intensive, but also depends on the doctor's experience, and the two segmentation results of different doctors or even the same doctor also have great variability. Therefore, accurate automatic segmentation methods are urgently needed.

The segmentation of cardiac structures in cardiac cine MRI images currently includes manual segmentation methods, traditional automatic segmentation methods, and segmentation methods based on deep learning. The manual manual segmentation method is that doctors manually describe layer by layer on the two-dimensional image, and then conduct further analysis and diagnosis according to the results of the manual description. However, the manual labeling workload is large, time-consuming and labor-intensive, and the repeatability is poor, so that the doctor's labeling ability is not enough to meet the needs of a large number of potential patients; at the same time, the professional level and experience of the labeling personnel are quite different, and the results of manual segmentation are quite different, and the quality cannot be ensure.

Traditional automatic segmentation methods, methods based on images or deformable models can complete the segmentation process by interacting with users, requiring manual confirmation of segmentation results and adjustment of annotation results. Model-based segmentation methods, such as active shape model and graph model, can be used to reduce user interaction by constructing a general model with a large amount of data to complete the automatic segmentation process. However, image- and deformable-model-based cardiac segmentation methods usually require user interaction, suffer from poor robustness, and have low segmentation accuracy. Although model-based methods such as active shape models and atlas model methods can reduce user interaction, the heart shapes and dynamics of different people (including normal people and patients with heart disease) are diverse, and a universal model that includes all possible shapes of the ventricle is established. The model is very difficult, and there are problems of model generality and poor generalization performance.

With the development of deep learning in recent years, segmentation methods based on deep learning have been introduced into cardiac MRI image segmentation. Deep learning-based segmentation methods automatically extract features from raw cardiac images to complete the automatic segmentation process, usually without user interaction. The method based on deep learning can obtain more accurate automatic segmentation results, but the existing heart segmentation methods based on deep learning mostly use 2D segmentation methods without considering the context information between layers, which is important for accurate segmentation and improved segmentation. Performance is valuable. Ignoring inter-layer contextual information is not in line with the clinician's actual workflow. At the same time, due to the characteristics of thick slices and large spacing in the scanning of cardiac cine MRI images, the direct use of inter-layer context information through 3D segmentation method not only has high computational cost but also may not bring about performance improvement.

Therefore, how to improve the accuracy of automatic segmentation of cardiac cine MRI images and improve the segmentation performance has become an urgent problem to be solved.

SUMMARY OF THE INVENTION

In view of the deficiencies of the prior art, the present invention proposes a method for segmenting the cardiac structure of an MRI image based on a multi-channel convolutional neural network, the method comprising:

Step 1: collecting cardiac cine MRI training data, including normal cardiac cine MRI images of normal persons and abnormal cardiac cine MRI images of cardiac patients, the cardiac cine MRI training data including cine MRI images of the diastolic and systolic phases of the heart;

Step 2: The cardiac structure in the cardiac cine MRI training data is manually labeled layer by layer by an experienced doctor, and the labeling result is used as the standard result of cardiac segmentation;

Step 3: heart region extraction generative adversarial network training, design and build a heart region extraction model based on generative adversarial network, extract cardiac region images from the collected cardiac cine MRI training data, and the extracted cardiac region images are composed of multiple Each cardiac MRI slice image is superimposed;

Step 4: Design and build a deep convolutional segmentation network combining context information between layers based on the training of the heart segmentation network of the multi-channel convolutional neural network. The steps include:

Step 41: segment the cardiac structure on the heart region extracted in step 3, and use the MRI slice image information of the adjacent layer and the existing segmentation result information of the adjacent upper layer as the inter-layer context information in an iterative manner;

Step 42: Input the inter-layer context information into the respective corresponding feature extraction branches, each of the feature extraction branches adopts an independent parallel structure, that is, each inter-layer context feature extraction branch adopts the same network structure but Independently process a corresponding inter-layer context information;

Step 43: Each branch extracts the high-level abstract features of the image by stacking multiple convolution and pooling operations, and fuses them through the feature fusion module;

Step 44: The feature fusion module first concatenates the high-level abstract features extracted by each of the feature extraction branches, and then further fuses these high-level features through the ASPP module, and the fused features are subjected to upsampling by the decoding module and compensation of local detail information. And the convolution operation restores the image to the size of the input into the segmentation network, obtains an end-to-end dense multi-structure simultaneous segmentation probability map, and determines the category of each pixel from the probability map to obtain the final segmentation result;

Step 5: compare the segmentation result obtained by the cardiac structure segmentation network in step 4 with the cardiac standard segmentation result obtained in step 2, and perform quantitative evaluation of the segmentation result through the performance evaluation index;

Step 6: Collect the cardiac cine MRI data to be segmented, extract the cardiac region using the cardiac region extraction model trained in Step 3, record the extraction location information, and then input the extracted MRI slice images of the cardiac region into the cardiac segmentation trained in Step 4. In the network, iteratively completes the segmentation of the volume data of the heart region from the bottom of the heart to the top of the heart, removes the discontinuous and scattered areas that may exist in each cardiac structure in the segmentation result, and obtains the initial segmentation result of the cardiac structure;

Step 7: According to the extracted position information, restore the initial segmentation result of the cardiac structure to the original image size to obtain a final segmentation result.

According to a preferred embodiment, the heart region extraction model in step 3 includes a generator and a discriminator,

The generator takes cardiac MRI slice images as input, and adopts an encoding-decoding structure, that is, first extracts features through convolution, pooling, and downsampling, and then generates and input cardiac MRI through upsampling, local detail compensation, and convolution operations. Pseudo-heart contour images with the same size of slice images;

The discriminator takes the cardiac MRI slice image and the corresponding real cardiac region contour image pair or the cardiac MRI slice image and the pseudo-heart contour image pair generated by the generator as input, extracts features through convolution and pooling operations, and discriminates the input heart. whether the contour image is real or generated by a generator;

After the generative adversarial network is trained, the generator generates accurate cardiac contour images corresponding to each cardiac MRI slice image, locates the position of the heart on the image, and extracts a three-dimensional cardiac region image.

The beneficial effects of the present invention are:

1. The present invention adopts the heart region extraction model based on the generative confrontation network, which can automatically and more accurately extract the heart region, and record the extraction position, without manual interaction, and has good versatility and generalization ability.

2. The technical solution of the present invention is to use the context information between layers, that is, use the spatial correlation information between adjacent layers, to improve the accuracy and accuracy of the segmentation of the cardiac structure in the training of the neural network for segmenting the cardiac structure. The trained neural network Automatic segmentation of cardiac structures can be accomplished.

3. Since the direct use of inter-layer context information through the 3D segmentation method has the disadvantages of high computational overhead and limited performance under limited data sets, the present invention adopts an independent parallel structure to process the corresponding inter-layer context under the framework of the 2D segmentation method. information, which effectively improves the segmentation performance.

Description of drawings

Fig. 1 is the flow chart of the cardiac structure automatic segmentation method of the present invention;

Fig. 2 is the working principle diagram of the present invention based on generative adversarial network extraction heart; and

FIG. 3 is a working principle diagram of a heart segmentation network based on a multi-channel convolutional neural network according to the present invention.

Detailed ways

In order to make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the specific embodiments and the accompanying drawings. It should be understood that these descriptions are exemplary only and are not intended to limit the scope of the invention. Also, in the following description, descriptions of well-known structures and techniques are omitted to avoid unnecessarily obscuring the concepts of the present invention.

The cardiac cine MRI image in the present invention refers to an image obtained by cardiac magnetic resonance cine imaging technology, which is a type of cardiac MRI image.

The ASPP module in the present invention refers to a pyramid pooling module with dilated convolution.

Cardiac magnetic resonance cine imaging technology is a commonly used cardiac magnetic resonance imaging technology, which usually refers to the rapid acquisition of multiple images at various specific stages in the cardiac cycle, which are displayed in the form of movies. Cardiac magnetic resonance cine imaging technology can be used not only to evaluate ventricular function and wall abnormalities, but also to evaluate the shape and function of heart valves.

The original image in the present invention refers to the image obtained by the original MRI scan, that is, the collected cardiac cine MRI image.

In view of the deficiencies in the current prior art, the present invention proposes a method for segmenting the cardiac structure of an MRI image based on a multi-channel convolutional neural network. As shown in FIG. 1 , the method of the present invention includes:

Step 1: Collect cardiac cine MRI training data, including normal cardiac cine MRI images of normal people and abnormal cardiac cine MRI images of cardiac patients. Cardiac cine MRI images capture images of the diastolic and systolic phases of the cardiac cycle. The scanned cardiac cine MRI data is volume data, that is, composed of a plurality of slice MRI data.

Simultaneous use of normal and cardiac patient heart data is to ensure the generalization capability of the technical solution of the present invention, that is, the automatic segmentation method of the present invention is suitable for segmentation of normal cardiac structures and abnormal cardiac structures at the same time.

In the training stage, the data of the diastolic and systolic stages of the heart are used. On the one hand, this part of the data is marked, and more importantly, the segmentation of the cardiac structure in these two stages is usually a clinical concern.

Step 2: An experienced doctor manually labels the cardiac structures in the cardiac cine MRI training data layer by layer, and uses the labeling results as the standard results of cardiac segmentation. The cardiac structures to be segmented include normal cardiac structures and abnormal cardiac structures. The annotated cardiac structures mainly include left ventricle, right ventricle, and myocardium. In practical applications, the endocardium and epicardium of the left ventricle can also be annotated. Specifically, layer-by-layer refers to the MRI slice layer from the bottom of the heart to the top of the heart, and all training data are labeled.

Considering that the cardiac cine MRI image data collected in the actual operation all have a large scanning range and cover a large area around the heart, the display ratio of the cardiac region on the image is relatively small, considering the computational efficiency. And in order to avoid the problem of class imbalance to a certain extent, it is necessary to extract the heart region before performing the segmentation of the heart structure, so that the extracted heart region occupies a larger display ratio on the image, which also reduces the subsequent segmentation processing. amount of calculation.

Therefore, the technical solution of the present invention designs and establishes a generative adversarial network-based cardiac region extraction model before segmentation, and trains the model for cardiac cine MRI training data to extract the cardiac region, as shown in step 3.

Step 3: Heart region extraction Generative adversarial network training, design and build a heart region extraction model based on generative adversarial network, extract cardiac images from the collected cardiac cine MRI training data, and the extracted cardiac region is a three-dimensional image, which is composed of multiple MRI slice images are superimposed.

Generative adversarial network is a deep learning model, including generator and discriminator. The working principle of Generative Adversarial Network is shown in Figure 2. Heart region extraction can be done manually or automatically based on some methods. Since the generative adversarial network has achieved good performance in the field of image processing in recent years, the present invention chooses to apply the generative adversarial network to the heart region extraction model to achieve automatic extraction and obtain excellent extraction performance.

The generator takes the cardiac MRI slice images in the training data as input, and adopts an encoding-decoding structure, that is, the features are extracted by downsampling through convolution and pooling operations, and then generated and inputted through upsampling, local detail compensation, and convolution operations. Pseudo-heart contour images consistent with the size of MRI slice images. During the training phase of the heart region extraction model, the generator generates a pseudo-heart contour image, and the image marked by the doctor is the real heart region contour image. The goal of generative adversarial network training is that the generator can generate real heart contour images, that is, after training, the ideal effect is that the generator can generate heart contour images that are consistent with the doctor's labeled images.

There is no method for extracting heart region based on generative adversarial network in the prior art. The technical solution of the present invention mainly utilizes generative adversarial network to realize automatic extraction of heart region without manual interaction and without manual design features.

The discriminator takes as input a cardiac MRI slice image and a corresponding true cardiac region contour image pair or a cardiac MRI slice image and a pseudo cardiac contour image generated by the generator. Features are extracted through convolution and pooling operations to determine whether the contour image of the heart region input to the discriminator is real or generated by the generator. Specifically, the discriminator discriminates whether the input real heart region contour image or pseudo heart contour image is real or generated by the generator.

In order to enable the generator to generate segmentation results that are close to the real heart contour image, and at the same time, the discriminator can accurately distinguish the real heart contour image and the pseudo heart contour image generated by the generator as the optimization goal, and train the heart region extraction generative adversarial network. , so that the generator can finally generate a high-accuracy heart contour image. After the network is trained, the generator is used to generate accurate cardiac contour images corresponding to each cardiac MRI slice image, locate the position of the heart on the image, and extract the three-dimensional cardiac region. The present invention regards the heart region extraction problem as an image pixel-level two-classification problem, that is, an image segmentation problem. Referring to the method of image segmentation using a generative adversarial network, the present invention uses image pairs as the input of the discriminator.

In a preferred embodiment, it can be determined whether the generative adversarial network has been trained by specifying the number of training iterations and combining with the training loss curve.

The generative adversarial network is used to extract the heart area; for this purpose, it is divided into two steps, first, you need to know where the heart is in the image, and second, extract the corresponding area. For the first step, the determination of the position of the heart is achieved through the generative adversarial network. After training the generative adversarial network, the generator can generate an accurate heart contour image, that is, knowing which position on the image is the heart, that is, knowing the heart s position. The second step is to extract the determined heart region.

After the generative adversarial network is trained, the heart region is extracted based on the located heart position. The extraction method includes: for a cardiac cine MRI image data of the heart region to be extracted, first, use the trained generator to generate each MRI slice The heart contour image corresponding to the image, and then generate a rectangular area surrounding the heart according to the contour, take the largest rectangular area in each slice and expand the length and width by 0.3 times to ensure that the complete heart area can be extracted. The expanded rectangular area is used as the area to be extracted. Finally, the region is extracted for all slices and stacked together to extract a 3D cardiac region image for subsequent training of the cardiac segmentation network.

Step 4: Design and build a deep convolutional segmentation network combining contextual information between layers based on the training of the heart segmentation network of the multi-channel convolutional neural network. The working principle of heart structure segmentation based on multi-channel convolutional neural network is shown in Figure 3.

Step 41 : segment the cardiac structure on the heart region extracted in step 3, and use the MRI slice image information of the adjacent layer and the existing segmentation result information of the adjacent upper layer as the inter-layer context information in an iterative manner.

In a preferred embodiment, the MRI slice image information of two adjacent layers, that is, the adjacent upper layer and the adjacent lower layer, and the existing segmentation result information of an adjacent upper layer are used as the inter-layer context information.

Step 42: Input these inter-layer contextual information into their respective feature extraction branches, each inter-layer contextual feature extraction branch adopts an independent parallel structure, that is, each inter-layer contextual feature extraction branch adopts the same network structure but processes its features independently. A corresponding inter-layer context information. The contextual feature extraction branch performs feature extraction through convolution and pooling operations.

Specifically, the number of inter-layer context feature extraction branches is determined according to the number of inter-layer context information to be processed. In a preferred embodiment, the MRI slice images of two adjacent upper and lower layers and the segmentation result information of one adjacent upper layer are taken as the inter-layer context information, then there are three inter-layer context feature extraction branches. In addition, plus the feature extraction branch of the MRI slice currently to be segmented, there are a total of 4 feature extraction branches. Fig. 3 is a working principle diagram of the cardiac structure segmentation of the four feature extraction branches of the present embodiment. As shown in Fig. 3, input M[i+1] represents the segmentation result of the adjacent upper-layer MRI slice images, and input S[i- 1] and input S[i+1] represent the adjacent upper and lower layers of MRI slice images, and input S[i] represents the current MRI slice image to be segmented.

Step 43: Each branch extracts the high-level abstract features of the image by stacking multiple convolution and pooling operations, and fuses them through a feature fusion module.

Step 44: The feature fusion module first concatenates the high-level abstract features extracted by each branch, and then further fuses these high-level features through the designed ASPP module. The fused features are subjected to upsampling, local detail information compensation and convolution by the decoding module. The operation restores the image to the size of the input into the cardiac structure segmentation network, and obtains an end-to-end dense multi-structure joint segmentation probability map. The segmentation probability map means that each pixel on the MRI slice image represents that the corresponding pixel of the image to be segmented belongs to the corresponding classification The probability of each pixel is determined by the probability map to obtain the final segmentation result.

Step 5: Compare the segmentation result obtained by the cardiac structure segmentation network with the cardiac standard segmentation result obtained in step 2, and perform quantitative evaluation of the segmentation result through the performance evaluation index. The present invention uses Dice coefficient and/or ASSD index to quantitatively evaluate the segmentation result. At the same time, the performance evaluation index is used to evaluate the quality of the segmentation results.

Step 6: Collect the cine MRI data of the heart to be segmented, extract the heart region through the automatic heart region extraction network trained in step 3, and record the extraction location information. The extracted heart region is three-dimensional volume data, which is composed of multiple MRI slice images superimposed. Then, the extracted MRI slice images of the heart region are input into the deep convolutional segmentation network trained in step 4, and the segmentation of the volume data of the heart region is completed iteratively from the bottom of the heart to the top of the heart, and the possible discontinuities of each structure in the segmentation results are eliminated. The scattered area of is obtained, and the initial segmentation result of the cardiac structure is obtained.

Step 6 is to segment the cardiac structure by applying the method proposed by the present invention.

Step 7: According to the extracted position information, restore the initial segmentation result of the heart structure to the original image size to obtain the final segmentation result. In practical applications, the segmentation result can also be seen without restoring to the original image, but the size is inconsistent with the original image, but it does not affect the viewing of the segmentation result.

For example, the original image is 400*400 in size, and the extracted heart is 20*20 in size and located in the center of the original image. By default, the area outside 20*20 is the background, that is, the default segmentation is non-heart structure; The initial segmentation of the heart structure is the segmentation result of the 20*20 image, and the final result should be the segmentation result corresponding to the 400*400 image. Therefore, during segmentation, it is necessary to record the extraction position of the heart, so as to know where the heart segmentation result corresponds to the original image size. For example, middle, lower left, and lower right.

It should be noted that the above-mentioned specific implementation is exemplary, and those skilled in the art can come up with various solutions inspired by the disclosure of the present invention, and these solutions also belong to the disclosure scope of the present invention and fall into the present invention within the scope of protection. It should be understood by those skilled in the art that the description of the present invention and the accompanying drawings are illustrative rather than limiting to the claims. The protection scope of the present invention is defined by the claims and their equivalents.

Claims (2)

1. A method for segmenting a cardiac structure of an MRI image based on a multi-path convolutional neural network, the method comprising:

step 1: collecting cardiac cine MRI training data, including normal cardiac cine MRI images of normal persons and abnormal cardiac cine MRI images of cardiac patients, the cardiac cine MRI training data including cine MRI images of diastolic and systolic phases;

step 2: manually labeling the heart structure in the heart cine MRI training data layer by an experienced doctor, and taking a labeling result as a heart segmentation standard result;

and step 3: the method comprises the steps of extracting a heart region, generating confrontation network training, designing and establishing a heart region extraction model based on the generated confrontation network, extracting heart region images from collected cardiac cine MRI training data, wherein the extracted heart region images are formed by overlapping a plurality of cardiac MRI slice images;

the cardiac region extraction model includes a generator and an arbiter:

the generator takes a cardiac MRI slice image as input and adopts a coding and decoding structure, namely, features are extracted through convolution, pooling operation and down-sampling, and then a pseudo cardiac contour image with the same size as the input cardiac MRI slice image is generated through up-sampling, local detail compensation and convolution operation;

the discriminator takes a heart MRI slice image and a corresponding real heart area contour image pair or a heart MRI slice image and a pseudo heart contour image pair generated by the generator as input, extracts features through convolution and pooling operation, and discriminates whether the input heart contour image is real or generated by the generator;

and 4, step 4: training a heart segmentation network based on a multipath convolutional neural network, designing and establishing a deep convolutional segmentation network combined with interlayer context information, and comprising the following steps of:

step 41: segmenting the heart structure of the heart region extracted in the step 3, and taking MRI slice image information of an adjacent layer and the existing segmentation result information of an adjacent upper layer as interlayer context information in an iterative mode;

step 42: respectively inputting the interlayer context information into respective corresponding feature extraction branches, wherein each feature extraction branch adopts an independent parallel structure, namely each interlayer context feature extraction branch adopts the same network structure but independently processes the corresponding interlayer context information;

step 43: each branch extracts high-level abstract features of the image by superposing a plurality of convolution and pooling operations and is fused by a feature fusion module;

step 44: the feature fusion module firstly connects the high-level abstract features extracted by each feature extraction branch in series, then further fuses the high-level features through an ASPP module, the fused features restore the image to the size when the image is input into a segmentation network through the up-sampling, local detail information compensation and convolution operation of a decoding module, an end-to-end dense multi-structure simultaneous segmentation probability map is obtained, and the class of each pixel is determined through the probability map so as to obtain a final segmentation result;

and 5: comparing the segmentation result obtained by the heart structure segmentation network in the step 4 with the heart standard segmentation result obtained in the step 2, and quantitatively evaluating the segmentation result through a performance evaluation index;

step 6: collecting cardiac cine MRI data to be segmented, extracting a cardiac region by using the cardiac region extraction model trained in the step 3, recording extraction position information, inputting an MRI slice image of the extracted cardiac region into the cardiac segmentation network trained in the step 4, sequentially iterating from the heart bottom to the heart top to complete the segmentation of the volumetric data of the cardiac region, and eliminating discontinuous scattered regions possibly existing in each cardiac structure in a segmentation result to obtain an initial segmentation result of the cardiac structure;

and 7: and restoring the initial segmentation result of the heart structure to the original image size according to the extracted position information to obtain a final segmentation result.

2. The method for segmenting the cardiac structure of the MRI image according to claim 1, wherein after the training of the countermeasure network is generated, the generator generates accurate cardiac contour images corresponding to the MRI slice images of the heart, positions the heart on the images and extracts the three-dimensional cardiac region images.

Images