CN107610194A - MRI super resolution ratio reconstruction method based on Multiscale Fusion CNN - Google Patents

Abstract

The present invention relates to a kind of MRI super resolution ratio reconstruction method based on Multiscale Fusion CNN, low-resolution image and corresponding high-definition picture are pre-processed first, and build training dataset and label data collection, then the structure fusion full convolutional neural networks of multi-scale information, training dataset is input in the fusion multi-scale information convolutional neural networks of structure and be trained, convolutional neural networks model after being learnt, it will test in the convolutional neural networks after low-resolution image is input to study, obtain rebuilding high-definition picture.The present invention passes through Multiscale Fusion unit, the Feature Mapping of different convolutional layers is merged, overcome the flat layered structures that the multilayer convolutional layer of traditional convolutional neural networks stacks, the convergence rate of network can be accelerated, quickly reconstruct the image detail of low-resolution image loss, reconstruction time is reduced, improves and rebuilds efficiency, avoid the wasting of resources.

Description

Super-resolution reconstruction method of magnetic resonance images based on multi-scale fusion CNN

technical field

The invention belongs to the field of image processing, in particular to a method for super-resolution reconstruction of magnetic resonance images based on multi-scale fusion CNN.

Background technique

Structural magnetic resonance images with higher spatial resolution have fewer artifacts, which directly affect the accuracy of subsequent image processing and medical diagnosis, such as registration and segmentation. However, due to limitations in physical equipment, acquisition technology, and economy, the spatial resolution of existing magnetic resonance images is affected to a certain extent.

In the field of image processing, traditional super-resolution reconstruction methods mainly use interpolation methods, such as bilinear interpolation, B-spline interpolation and other methods. These methods assume the smooth nature of local regions and estimate newly interpolated voxel values from neighboring voxels. However, the interpolation method is not suitable for non-uniform areas, and it is easy to cause blurred images.

For magnetic resonance images, according to different reconstruction stages, super-resolution reconstruction methods are mainly divided into two types: the first reconstruction directly reconstructs the K-space data during the acquisition process; the second reconstruction is in the post-processing stage, usually Traditional reconstruction methods were applied to structural magnetic resonance image data. The most commonly used methods are non-local mean methods and sparse coding methods. Since the prior knowledge reconstructed by the non-local mean method still comes from local image blocks, the ideal reconstruction effect cannot be obtained. Sparse coding methods use machine learning methods to learn low-resolution and high-resolution dictionaries from low-resolution image patches and corresponding high-resolution image patches; the low-resolution images are then considered to sparsely represent linear Combination, solve its sparse coefficient. And project the sparse coefficients to the high-resolution dictionary space to obtain the reconstructed high-resolution image. But patch-based sparse representation cannot guarantee the optimal reconstruction of the overall image.

The training of traditional convolutional neural networks requires a large number of samples to ensure a better final effect. In the medical field, it is difficult to obtain a large amount of magnetic resonance image data, so it is difficult to guarantee the convergence and reconstruction accuracy of the network directly using the traditional convolutional neural network.

Contents of the invention

Aiming at the deficiencies in the prior art, the present invention proposes a method for super-resolution reconstruction of magnetic resonance images based on multi-scale fusion CNN, the method comprising:

Step 1: Perform preprocessing operations on the low-resolution structural magnetic resonance image and its corresponding high-resolution structural magnetic resonance image, and construct a training data set and a label data set;

Step 11: Input the low-resolution structural magnetic resonance image and the high-resolution structural magnetic resonance image in the standard format, and perform format conversion;

Step 12: removing the skull part from the low-resolution structural magnetic resonance image and the high-resolution structural magnetic resonance image converted in step 11, and only retaining the brain region;

Step 13: Perform normalization processing on the low-resolution structural magnetic resonance image and the high-resolution structural magnetic resonance image after the skull is removed in step 12, and normalize them to the [0-1] interval;

Step 14: Sliding window method is used to extract a plurality of two-dimensional image blocks sequentially on each layer from the normalized resolution structural magnetic resonance image and the high-resolution structural magnetic resonance image in step 13, wherein the low The high-resolution image patches constitute the training dataset, and the high-resolution image patches constitute the label dataset;

Step 2: Constructing a fusion multi-scale information convolutional neural network, the convolutional neural network includes an input layer, at least three stacked multi-scale fusion units and a reconstruction layer;

Step 21: the input layer is used to receive the training data set;

Step 22: Construct at least three multi-scale fusion units;

Step 23: Constructing a reconstruction layer, the reconstruction layer is a convolution layer composed of a convolution kernel;

Step 3: input the training data set into the convolutional neural network constructed in step 2 for training, and obtain the learned convolutional neural network model;

Step 31: Divide the training data set into multiple batches of training data, and initialize the convolution kernel weights and biases in all convolutional layers in the multi-scale information convolutional neural network constructed in step 2 to be 0 inversely to the loss function ,which is:

△W ^(l) ＝0

Δb ^(l) = 0

Among them, W represents the weight of the convolution kernel, b represents the bias pair loss function, and l represents the lth layer;

Step 32: Input a batch of training data each time and calculate the parameters of each node in the multi-scale fusion unit to realize the forward propagation of neural network training, and finally obtain the output high-resolution data through the reconstruction layer;

Step 33: Using the Euclidean distance, the error between the output high-resolution data obtained in step 32 and the label dataset:

Wherein, I, J represent the size of the image block;

Step 34: Based on the error, use the gradient descent method to reversely calculate the derivative of the convolution kernel weight and bias to the loss function with and add it to △W ^(l) and △b ^(l) , that is:

Step 35: Repeat steps 32 to 34 until all the training data are processed and an iteration is completed. According to the above △W ^(l) and △b ^(l) , use the batch gradient descent algorithm to obtain the updated network parameters:

Where m represents the number of batches of training data, α is the learning rate, and λ is the kinetic energy;

Step 36: Repeat steps 32 to 35 until the preset number of iterations is reached;

Step 4: Input the test low-resolution structural magnetic resonance image into the convolutional neural network trained in step 3, and output the reconstructed high-resolution structural magnetic resonance image;

Step 41: directly input each layer of the test low-resolution structural magnetic resonance image into the input layer in the convolutional neural network model trained in step 3;

Step 42: Input the test low-resolution structural magnetic resonance image received in step 41 into the learned convolutional neural network model, perform operations from front to back, and finally output the reconstructed high-resolution structural magnetic resonance image in the reconstruction layer.

According to a preferred embodiment, the multi-scale fusion unit includes a main path, at least one sub-path and a fusion layer, the main path consists of a convolutional layer plus a ReLU activation function, and the sub-path consists of a convolution Layers plus a ReLU activation function are alternately formed, and the last layer is a convolutional layer. The fusion layer adds and fuses the output of the main path and the sub-path to output to the next multi-scale fusion unit.

Compared with prior art, the beneficial effect of the present invention is:

1. The present invention fuses the feature maps of different convolutional layers through a multi-scale fusion unit, which overcomes the stacked flat layer structure of the traditional convolutional neural network, and can accelerate the convergence speed of the network and achieve faster Reconstruct the image details lost in low-resolution images, reduce the time required for reconstruction, and reduce the waste of resources.

2. Compared with the existing reconstruction method, the reconstruction method of the present invention obtains better reconstruction effect, the restored detail information and structural information are closer to the real high-resolution image, and the obtained peak signal-to-noise ratio is also higher.

Description of drawings

Fig. 1 is a schematic flow chart of the super-resolution reconstruction method of the present invention;

Fig. 2 is a schematic structural diagram of a multi-scale fusion unit of the present invention;

Fig. 3 is a schematic structural diagram of the fusion multi-scale information convolutional neural network of the present invention;

Fig. 4 is a schematic structural diagram of a traditional convolutional neural network;

Figure 5 is the feature map output by each part of the multi-scale fusion unit;

Figure 6 is a reconstruction rendering of various methods on the simulation data set Brainweb; and

Figure 7 Reconstruction renderings of various methods on real datasets.

detailed description

In order to make the object, technical solution and advantages of the present invention clearer, the present invention will be further described in detail below in combination with specific embodiments and with reference to the accompanying drawings. It should be understood that these descriptions are exemplary only, and are not intended to limit the scope of the present invention. Also, in the following description, descriptions of well-known structures and techniques are omitted to avoid unnecessarily obscuring the concept of the present invention.

The multi-scale fusion unit MFU of the present invention: The Multi-scale Fusion Unit.

Fig. 1 is a schematic flowchart of the super-resolution reconstruction method of the present invention. As shown in Fig. 1, a kind of multi-scale fusion CNN proposed by the present invention super-resolution reconstruction method of magnetic resonance image, the method comprises:

Step 1: Perform preprocessing operations on the low-resolution structural magnetic resonance image and its corresponding high-resolution structural magnetic resonance image, and construct a training data set and a label data set. The high-resolution images come from real images collected by 3T magnetic resonance equipment, and the low-resolution images come from downsampling the high-resolution images. The low-resolution structural MRI images in step 1 are used as training samples to train the convolutional neural network.

Step 11: Input low-resolution structural magnetic resonance images and high-resolution structural magnetic resonance images in standard formats, and perform format conversion. The original magnetic resonance image data format is DCM format, which is converted to NII format by SPM. The reason is that the original DCM format is that a person's magnetic resonance data is composed of N DCM files, and after being converted to NII format, a person's magnetic resonance data is composed of one NII file, which is convenient for subsequent data processing.

Step 12: Remove the skull part from the low-resolution structural MRI image and the high-resolution structural MRI image converted in Step 11, and only keep the brain region.

Step 13: Normalize the low-resolution structural MRI image and the high-resolution structural MRI image after removing the skull in step 12, and normalize them to the [0-1] interval. Since the original collected magnetic resonance image data ranges from 0 to tens of thousands, image processing usually transforms its range to [0-1] in order to put all the data in the same range.

Step 14: For the normalized low-resolution structural magnetic resonance image and high-resolution structural magnetic resonance image in step 13, a plurality of two-dimensional image blocks are sequentially extracted on each layer in a sliding window manner, among which the low-resolution The high-resolution image patches constitute the training dataset, and the high-resolution image patches constitute the label dataset. The number of two-dimensional image blocks is controlled by the size of the image and the size of the sliding window, and generally tens of thousands are taken. Specifically, the magnetic resonance data of the human brain is three-dimensional data M*N*S. The magnetic resonance machine scans the human brain layer by layer from top to bottom, S represents the number of layers, and M*N represents the size of the layer of the brain.

Step 2: Construct a convolutional neural network that fuses multi-scale information.

Fig. 3 is a schematic structural diagram of a multi-scale information convolutional neural network of the present invention. Figure 4 is a schematic diagram of the structure of a traditional convolutional neural network. As shown in Figure 4, the traditional convolutional neural network is a flat layer structure with multiple convolutional layers stacked. As shown in Figure 3, the multi-scale information convolutional neural network includes an input layer, multiple stacked multi-scale fusion units and a reconstruction layer. Specifically, at least three stacked multi-scale fusion units. The fusion multi-scale convolutional neural network of the present invention overcomes the shortcomings of the traditional convolutional neural network, can accelerate the convergence speed of the neural network, reconstruct the image details lost in the low-resolution image faster, reduce the time required for reconstruction, and have higher efficiency , reducing waste of resources.

Step 21: The input layer is used to receive the training data set.

Step 22: Construct at least three multi-scale fusion units. Fig. 2 is a schematic diagram of the structure of a multi-scale fusion unit. As shown in Figure 2, a multi-scale fusion unit includes a main path, at least one sub-path and a fusion layer. The main path consists of a convolutional layer plus a ReLU activation function, the sub-paths alternately consist of a convolutional layer plus a ReLU activation function, and the last layer is a convolutional layer. The fusion layer adds and fuses the output of the main path and the sub-path to output to the next multi-scale fusion unit.

Step 23: Construct a reconstruction layer, which is a convolution layer composed of a convolution kernel.

Step 3: Input the training data set into the convolutional neural network constructed in step 2 for training, and obtain the learned convolutional neural network model.

Step 31: Divide the training data set into multiple batches of training data. Because the data volume of the training data set is large, the constructed neural network cannot process all the training data at one time, so the training data set needs to be divided into multiple batches for processing. The specific number of batches depends on the number of training samples and the number of samples in each batch. For example, if there are 10,000 training samples and each batch is 100, then it is divided into 100 batches of training data. And initialize the convolution kernel weights and biases in all convolutional layers in the multi-scale information convolutional neural network built in step 2 to be 0 inverse to the loss function, namely:

△W ^(l) ＝0

Δb ^(l) = 0

Among them, W represents the weight of the convolution kernel, b represents the bias pair loss function, and l represents the lth layer. All convolutional layers include the convolutional layers in the multi-scale fusion unit and the convolutional layers that constitute the reconstruction layer.

Step 32: Input a batch of training data each time and calculate the parameters of each node in the multi-scale fusion unit to realize the forward propagation of neural network training, and finally obtain the output high-resolution data through the reconstruction layer;

Step 33: Using the Euclidean distance, calculate the error between the output high-resolution data and the label dataset:

Among them, I, J represent the size of the image block.

Step 34: Based on the error calculated in step 33, use the gradient descent method to reversely calculate the derivative of the convolution kernel weight and bias to the loss function with And add it to △W ^(l) and △b ^(l) , namely:

Step 35: Steps S32-S34 are repeated until all training samples are processed, and one iteration is completed. According to the above △W ^(l) and △b ^(l) , the batch gradient descent algorithm is used to obtain the updated network parameters.

Among them, m represents the batch number of training samples, α is the learning rate, and λ is the kinetic energy, which determines the influence of the parameters of the last iteration during the parameter update process.

Step 36: Repeat steps 32 to 35 until the preset number of iterations is reached. Generally, the number of iterations is taken to the 5th power of 10, or the loss is less than about 0.02, which can be determined by the loss function. After the iteration stops, the trained convolutional neural network model is obtained.

Step 4: Input the test low-resolution structural magnetic resonance image into the convolutional neural network model trained in step 3, and output the reconstructed high-resolution structural magnetic resonance image. Test low-resolution structural magnetic resonance images as test samples.

Step 41: Input each layer of the test low-resolution structural magnetic resonance image directly into the input layer in the convolutional neural network model trained in step 3.

Step 42: The test low-resolution structural magnetic resonance image received in step 41 is input into the trained convolutional neural network, the operation is performed from front to back, and finally the reconstructed high-resolution structural magnetic resonance image is output in the reconstruction layer. The reconstructed high-resolution structural magnetic resonance image is a high-resolution structural magnetic resonance image learned by multi-scale fusion CNN.

Fig. 5 is a feature map output by each part of the multi-scale fusion unit.

The first row of images in Figure 5 is a low-resolution structural magnetic resonance image, the second row of images is the feature map output by the main path of the multi-scale fusion unit, and the third row of images is the feature map output by the sub-path of the multi-scale fusion unit. The fourth row of images is the feature map output by the multi-scale fusion layer.

Figure 6 is a reconstruction rendering of various methods on the simulation data set Brainweb. Figure 7 Reconstruction renderings of various methods on real datasets. In Figure 7, HR Real Data represents real high-resolution images, LR Real Data represents real low-resolution images, and high-resolution images are reconstructed from real low-resolution images. The second row of images in Fig. 6 and Fig. 7 is a partially enlarged image of the reconstructed image. It can be seen intuitively from Fig. 6 and Fig. 7 that the reconstruction effect of the method MFCN (Multi-scale FusionConvolution Network) of the present invention is the best. The reconstructed high-resolution edge detail information and structure are closer to the real high-resolution image, especially the parts marked by ellipses in Figures 6 and 7. Further, PSNR is used to objectively evaluate the reconstruction effect, and the higher the peak signal-to-noise ratio, the better the reconstruction effect. It can be seen from Fig. 6 and Fig. 7 that the peak signal-to-noise ratio obtained by the method of the present invention is higher than that of several existing methods.

The super-resolution reconstruction method of the present invention is not only applicable to the reconstruction of magnetic resonance images, but also applicable to image reconstruction in other fields, such as reconstruction of weather radar echoes, CT images, PET-CT images, etc.

The present invention is based on the super-resolution reconstruction method of multi-scale fusion CNN, which overcomes the shortcomings of the traditional convolutional neural network that it is difficult to ensure the convergence and reconstruction accuracy of the neural network, and does not need to obtain a large amount of magnetic resonance image data, and maps the features of different convolutional layers Fusion overcomes the stacked flat layer structure of the traditional convolutional neural network, which can speed up the convergence speed of the network, reconstruct the image details lost in low-resolution images faster, reduce the time required for reconstruction, and reduce the Waste of resources. Furthermore, it can be seen from the intuitive and objective indicators that the method of the present invention obtains higher reconstruction accuracy than the existing reconstruction technology.

It should be noted that the above specific embodiments are exemplary, and those skilled in the art can come up with various solutions inspired by the disclosure of the present invention, and these solutions also belong to the scope of the disclosure of the present invention and fall within the scope of this disclosure. within the scope of protection of the invention. Those skilled in the art should understand that the description and drawings of the present invention are illustrative rather than limiting to the claims. The protection scope of the present invention is defined by the claims and their equivalents.

Claims (2)

1. A magnetic resonance image super-resolution reconstruction method based on multi-scale fusion CNN, is characterized in that, described method comprises: Step 1: Perform preprocessing operations on the low-resolution structural magnetic resonance image and its corresponding high-resolution structural magnetic resonance image, and construct a training data set and a label data set; Step 11: Input the low-resolution structural magnetic resonance image and the high-resolution structural magnetic resonance image in the standard format, and perform format conversion; Step 12: removing the skull part from the low-resolution structural magnetic resonance image and the high-resolution structural magnetic resonance image converted in step 11, and only retaining the brain region; Step 13: Perform normalization processing on the low-resolution structural magnetic resonance image and the high-resolution structural magnetic resonance image after the skull is removed in step 12, and normalize them to the [0-1] interval; Step 14: Sliding windows are used to sequentially extract a plurality of two-dimensional image blocks on each layer from the normalized low-resolution structural magnetic resonance image and the high-resolution structural magnetic resonance image in step 13, wherein The low-resolution image patches constitute the training dataset, and the high-resolution image patches constitute the label dataset; Step 2: Constructing a fusion multi-scale information convolutional neural network, the convolutional neural network includes an input layer, at least three stacked multi-scale fusion units and a reconstruction layer; Step 21: the input layer is used to receive the training data set; Step 22: Construct at least three multi-scale fusion units; Step 23: Constructing a reconstruction layer, the reconstruction layer is a convolution layer composed of a convolution kernel; Step 3: input the training data set into the convolutional neural network constructed in step 2 for training, and obtain the learned convolutional neural network model; Step 31: Divide the training data set into multiple batches of training data, and initialize the convolution kernel weights and biases in all convolutional layers in the multi-scale information convolutional neural network constructed in step 2 to be 0 inversely to the loss function ,which is: △W ^(l) ＝0 Δb ^(l) = 0 Among them, W represents the weight of the convolution kernel, b represents the bias pair loss function, and l represents the lth layer; Step 32: Input a batch of training data each time and calculate the parameters of each node in the multi-scale fusion unit to realize the forward propagation of neural network training, and finally obtain the output high-resolution data through the reconstruction layer; Step 33: Using the Euclidean distance, the error between the output high-resolution data obtained in step 32 and the label dataset: <mrow><mi>E</mi><mo>=</mo><munderover><mo>&amp;Sigma;</mo><mrow><mi>i</mi><mo>,</mo><mi>j</mi><mo>=</mo><mn>0</mn></mrow><mrow><mi>I</mi><mo>,</mo><mi>J</mi></mrow></munderover><mo>|</mo><mo>|</mo><msub><mi>X</mi><mrow><mi>i</mi><mi>j</mi></mrow></msub><mo>-</mo><msub><mover><mi>X</mi><mo>&amp;OverBar;</mi>mo></mover><mrow><mi>i</mi><mi>j</mi></mrow></msub><mo>|</mo><msup><mo>|</mo><mn>2</mn></msup></mrow> Wherein, I, J represent the size of the image block; Step 34: Based on the error, use the gradient descent method to reversely calculate the derivative of the convolution kernel weight and bias to the loss function with and add it to △W ^(l) and △b ^(l) , that is: <mrow><msup><mi>&amp;Delta;W</mi><mrow><mo>(</mo><mi>l</mi><mo>)</mo></mrow></msup><mo>=</mo><msup><mi>&amp;Delta;W</mi><mrow><mo>(</mo><mi>l</mi><mo>)</mo></mrow></msup><mo>+</mo><msub><mo>&amp;dtri;</mo><msup><mi>W</mi><mrow><mo>(</mo><mi>l</mi><mo>)</mo></mrow></msup></msub><mi>J</mi><mrow><mo>(</mo><mi>W</mi><mo>,</mo><mi>b</mi><mo>;</mo><mi>x</mi><mo>,</mo><mi>y</mi><mo>)</mo></mrow></mrow> <mrow><msup><mi>&amp;Delta;b</mi><mrow><mo>(</mo><mi>l</mi><mo>)</mo></mrow></msup><mo>=</mo><msup><mi>&amp;Delta;b</mi><mrow><mo>(</mo><mi>l</mi><mo>)</mo></mrow></msup><mo>+</mo><msub><mo>&amp;dtri;</mo><msup><mi>b</mi><mrow><mo>(</mo><mi>l</mi><mo>)</mo></mrow></msup></msub><mi>J</mi><mrow><mo>(</mo><mi>W</mi><mo>,</mo><mi>b</mi><mo>;</mo><mi>x</mi><mo>,</mo><mi>y</mi><mo>)</mo></mrow></mrow> Step 35: Repeat steps 32 to 34 until all the training data are processed and an iteration is completed. According to the above △W ^(l) and △b ^(l) , use the batch gradient descent algorithm to obtain the updated network parameters. The mathematical expression is as follows : <mrow><msup><mi>W</mi><mrow><mo>(</mo><mi>l</mi><mo>)</mo></mrow></msup><mo>=</mo><msup><mi>W</mi><mrow><mo>(</mo><mi>l</mi><mo>)</mo></mrow></msup><mo>-</mo><mi>&amp;alpha;</mi><mo>&amp;lsqb;</mo><mrow><mo>(</mo><mfrac><mn>1</mn><mi>m</mi></mfrac><msup><mi>&amp;Delta;W</mi><mrow><mo>(</mo><mi>l</mi><mo>)</mo></mrow></msup><mo>)</mo></mrow><mo>+</mo><msup><mi>&amp;lambda;W</mi><mrow><mo>(</mo><mi>l</mi><mo>)</mo></mrow></msup><mo>&amp;rsqb;</mo></mrow> <mrow><msup><mi>b</mi><mrow><mo>(</mo><mi>l</mi><mo>)</mo></mrow></msup><mo>=</mo><msup><mi>b</mi><mrow><mo>(</mo><mi>l</mi><mo>)</mo></mrow></msup><mo>-</mo><mi>&amp;alpha;</mi><mo>&amp;lsqb;</mo><mfrac><mn>1</mn><mi>m</mi></mfrac><msup><mi>&amp;Delta;b</mi><mrow><mo>(</mo><mi>l</mi><mo>)</mo></mrow></msup><mo>&amp;rsqb;</mo></mrow> Among them, m represents the number of batches of training data, α is the learning rate, and λ is the kinetic energy; Step 36: Repeat steps 32 to 35 until the preset number of iterations is reached; Step 4: Input the test low-resolution structural magnetic resonance image into the convolutional neural network trained in step 3, and output the reconstructed high-resolution structural magnetic resonance image; Step 41: directly input each layer of the test low-resolution structural magnetic resonance image into the input layer in the convolutional neural network model trained in step 3; Step 42: Input the test low-resolution structural magnetic resonance image received in step 41 into the learned convolutional neural network model, perform operations from front to back, and finally output the reconstructed high-resolution structural magnetic resonance image in the reconstruction layer. 2. The super-resolution reconstruction method according to claim 1, wherein the multi-scale fusion unit comprises a main path, at least one sub-path and a fusion layer, and the main path is activated by a convolutional layer plus a ReLU function, the sub-path is composed of a convolutional layer plus a ReLU activation function alternately in turn, and the last layer is a convolutional layer, and the fusion layer fuses the output of the main path and the sub-path by adding to output to the next multi-scale fusion unit.