Abstract
To reconstruct magnetic resonance (MR) images from undersampled Cartesian k-space data, we propose an algorithm based on two deep-learning architectures: (1) a multi-layer perceptron (MLP) that estimates a target image from 1D inverse Fourier transform (IFT) of k-space; and (2) a convolutional neural network (CNN) that estimates the target image from the estimated image of the MLP. The MLP learns the relationship between 1D IFT of undersampled k-space which is transformed along the frequency-encoding direction and the target fully-sampled image. The MLP is trained line by line rather than by a whole image, because each frequency-encoding line of the 1D IFT of k-space is not correlated with each other. It can dramatically decrease the number of parameters to be learned because the number of input/output pixels decrease from N2 to N. The next CNN learns the relationship between an estimated image of the MLP and the target fully-sampled image to reduce remaining artifacts in the image domain. The proposed deep-learning algorithm (i.e., the combination of the MLP and the CNN) exhibited superior performance over a single MLP and a single CNN. And it outperformed the comparison algorithms including CS-MRI, DL-MRI, a CNN-based algorithm (denoted as Wang’s algorithm), PANO, and FDLCP in both qualitative and quantitative evaluation. Consequently, the proposed algorithm is applicable up to a sampling ratio of 25% in Cartesian k-space.
You have full access to this open access chapter, Download conference paper PDF
Similar content being viewed by others
Keywords
- Magnetic resonance imaging
- Undersampling
- Multi-layer perceptron
- Convolutional neural networks
- 1D Fourier transform
1 Introduction
Magnetic resonance imaging (MRI) is an imaging technique that can provide various contrast mechanisms for visualizing anatomical structures and physiological functions in human body. However, MRI is relatively slow because it is not possible to simultaneously sample multiple data points in the raw data domain (i.e. 2D Fourier domain of an image, which is referred to as k-space).
To accelerate MRI acquisition, k-space can be subsampled at a frequency that is lower than the Nyquist rate (i.e., it can be undersampled) instead of acquiring the fully-sampled k-space (i.e., sampling the data at the Nyquist rate). However, a simple reconstruction (i.e., 2D inverse Fourier transform) from the undersampled k-space brings out aliasing artifacts in the image domain and obscures many anatomical and physiological information. To reduce the aliasing artifacts and recover the missing information in images, various reconstruction algorithms such as compressed sensing (CS) [1] and parallel imaging (PI) [2] have been developed. The CS algorithms have been developed to the combination with low-rank constraint terms [3] and to image-adaptive algorithms that enforce sparsity on image patches [4].
In recent, deep-learning based reconstruction algorithms have been introduced and regarded as alternatives of CS. The first deep-learning algorithm applied to MRI reconstruction comprises a 3-layer convolutional neural network (CNN) that learns the relationship between undersampled images and fully-sampled images, and the conventional CS is followed at the end (hereafter denoted as Wang’s algorithm) [5]. After the study, various deep-learning algorithms such as CNN-based sparse residual (artifacts) learning algorithm [6], a cascaded CNN with interleaved data fidelity [7], multi-layer perceptron (MLP)-based parallel imaging (PI) algorithms have been introduced [8]. Especially, the automated transform by manifold approximation (AUTOMAP) algorithm provided a new perspective for reconstruction algorithm by directly translating k-space to the target image with neural networks [9].
In this study for accelerating MRI, we propose an algorithm that can efficiently translate 1D inverse Fourier transform (IFT) of undersampled k-space in data-acquisition direction (i.e., frequency-encoding direction) to the target fully-sampled images using two different deep-learning architectures: (1) a multi-layer perceptron (MLP) that estimates a target image from 1D inverse Fourier transform of k-space; and (2) a convolutional neural network (CNN) that reduces remaining artifacts in the output of the MLP. The proposed algorithm can utilize the maximum possible extent of the raw k-space while reducing the number of optimizing parameters dramatically compared with existing deep-learning algorithms, which enables efficient learning and results in better reconstructions than comparison algorithms.
2 Methods
2.1 Problem Formulation
Let \( {\mathbf{y}} \in {\complement }^{{{\text{n}}_{\text{kx}} \times {\text{n}}_{\text{ky}} }} \), where \( {\text{n}}_{\text{kx}} \) and \( {\text{n}}_{\text{ky}} \) represent the number of horizontal and vertical pixels of k-space respectively, denote a 2D complex-valued MR k-space. Our purpose is to reconstruct a fully-sampled image \( {\mathbf{x}} \in {\complement }^{{{\text{n}}_{\text{x}} \times {\text{n}}_{\text{y}} }} \), where \( {\text{n}}_{\text{x}} \) and \( {\text{n}}_{\text{y}} \) represent the number of horizontal and vertical pixels of x, from the undersampled k-space, \( {\mathbf{y}}_{{\mathbf{u}}} \), obtained by multiplication of a binary undersampling mask U and y as follows:
where \( {\mathbf{y}}_{{\mathbf{u}}} \in {\complement }^{{{\text{n}}_{\text{kx}} \times {\text{n}}_{\text{ky}} }} \) denotes the undersampled k-space; \( {\mathbf{U}} \in {\mathbb{R}}^{{{\text{n}}_{\text{kx}} \times {\text{n}}_{\text{ky}} }} \) denotes the binary undersampling mask; \( \circ \) denotes element-wise multiplication; \( {\mathcal{F}}_{2D} \) and \( {\mathcal{F}}_{2D}^{ - 1} \) denote the 2D Fourier transform (FT) and IFT, respectively; \( {\mathbf{y}}_{{{\mathbf{u}},{\mathbf{r}}}} \in {\mathbb{R}}^{{{\text{n}}_{\text{kx}} \times {\text{n}}_{\text{ky}} }} \) and \( {\mathbf{y}}_{{{\mathbf{u}},{\mathbf{i}}}} \in {\mathbb{R}}^{{{\text{n}}_{\text{kx}} \times {\text{n}}_{\text{ky}} }} \) denote the real and imaginary channels of \( {\mathbf{y}}_{{\mathbf{u}}} \), respectively; \( {\mathbf{x}}_{{\mathbf{u}}} \) denotes the undersampled image; and \( {\mathbf{x}}_{{{\mathbf{u}},{\mathbf{r}}}} \) and \( {\mathbf{x}}_{{{\mathbf{u}},{\mathbf{i}}}} \) denote the real and imaginary channels of \( {\mathbf{x}}_{{\mathbf{u}}} \), respectively.
To estimate x from a small number of samples in the k-space, \( {\mathbf{y}}_{{\mathbf{u}}} \), we introduce an algorithm comprising two deep-learning architectures, one is MLP that translates 1D IFT of undersampled k-space to an image, and the other is CNN that removes remaining artifacts in image domain. The objective function to optimize the parameters of the MLP, \( {\varvec{\Theta}}_{\text{MLP}} \), is
where \( {\mathbf{z}}_{\text{u}} = {\mathcal{F}}_{1D}^{ - 1} ({\mathbf{y}}_{\text{u}} ) \in {\complement }^{{{\text{n}}_{\text{x}} \times {\text{n}}_{\text{ky}} }} \) denotes the 1D IFT of \( {\mathbf{y}}_{\text{u}} \) along the frequency-encoding (horizontal) direction; and \( H_{\text{MLP}} \) denotes the hypothesis function that estimates fully-sampled image x from 1D IFT of undersampled k-space, \( {\mathbf{z}}_{\text{u}} \), by the MLP. The other objective function to optimize the parameters of the CNN for artifacts removal, \( {\varvec{\Theta}}_{\text{CNN}} \), is
where \( H_{\text{CNN}} \) denotes the hypothesis function for reducing the remaining artifacts in the image reconstructed by the MLP, \( {\hat{\mathbf{x}}}_{\text{MLP}} = H_{\text{MLP}} \left( {{\mathbf{z}}_{\text{u}} ;{\varvec{\Theta}}_{\text{MLP}} } \right) \); and \( \uplambda \) is the regularization parameter for data fidelity. The combined equation of Eqs. (3) and (4) to optimize \( {\varvec{\Theta}}_{\text{MLP}} \) and \( {\varvec{\Theta}}_{\text{CNN}} \), which is the final objective function of this study, can be represented as
In Fig. 1, the proposed deep-learning frameworks to solve Eq. (5) are presented.
2.2 The Proposed MLP: Translation from 1D IFT of K-space to an Image
The proposed MLP learns the relationship between 1D IFT of the undersampled Cartesian k-space which is transformed along the frequency-encoding (horizontal) direction, \( {\mathbf{z}}_{\text{u}} \), and the target fully-sampled image x. The proposed MLP is trained line by line rather than by a whole image, because each phase-encoding (vertical) line of the 1D IFT of k-space is not correlated with each other. Then, the estimation of a vertical line of the fully-sampled image x from the corresponding line of the 1D IFT of undersampled k-space, \( {\mathbf{z}}_{\text{u}} \), using the MLP can be represented as
where
where x and y denote horizontal and vertical indices in the image domain, respectively; \( {\mathbf{w}}_{\text{n}} \) and \( {\mathbf{b}}_{\text{n}} \) are weight and bias matrices of the MLP, respectively, where n = 1, 2, 3, …, NFL; \( \sigma \) is the activation function; and NFL denotes the number of fully-connected layers of the MLP. As depicted in Eq. (9), The loss function for training the MLP is defined as the mean-squared error between \( {\mathbf{Z}}({\text{x}}) \) and the estimation of \( {\mathbf{Z}}({\text{x}}) \) (i.e., \( H_{\text{MLP}} \left( {{\mathbf{Z}}_{\text{u}} ({\text{x}});{\varvec{\uptheta}}_{\text{MLP}} } \right) = \widehat{{\mathbf{X}}}_{\text{MLP}} ({\text{x}}) \)).
where
where \( {\mathbf{x}}\left( {{\text{x}}_{\text{m}} } \right) \) and \( {\mathbf{Z}}_{\text{u}} \left( {{\text{x}}_{\text{m}} } \right) \) denote the m-th vertical line of the fully-sampled image and the m-th vertical line of 1D IFT of the undersampled k-space, respectively; and M denotes the batch size. This line-by-line training can dramatically decrease the number of parameters to be learned because the number of input/output pixels (i.e., \( {\mathbf{Z}}_{\text{u}} ({\text{x}}) \) and \( \widehat{{\mathbf{X}}}_{\text{MLP}} ({\text{x}}) \)) decrease from \( {\text{n}}_{\rm{x}} {\text{n}}_{\rm{ky}} \) to \( {\text{n}}_{\rm{ky}} \). Finally, the hypothesis function and the parameters (i.e., weight and bias matrices) of the MLP that estimates the 2D fully-sampled image x from \( {\mathbf{z}}_{\text{u}} \) can be represented as
where
where \( {\mathbf{W}}_{\text{n}} = 1{\mathbf{w}}_{\text{n}} \), \( {\mathbf{B}}_{\text{n}} = 1{\mathbf{w}}_{\text{n}} \), \( 1 = \left[ {1, 1, \cdots ,1} \right]^{{\mathbf{T}}} \in {\mathbb{R}}^{{1 \times {\text{n}}_{\text{ky}} }} \), and n = 1, 2, 3, …, NFL.
More details for the MLP, including optimizer parameters, network specifications, including network depths, the number of outputs, activation function, deep-learning libraries, and training/testing times, are provided in the supporting information.
2.3 The Proposed CNN with Data Fidelity: Artifacts Removal in the MLP Output
The proposed CNN aims to remove the artifacts remaining in the output images of the previous MLP. The hypothesis function of \( H_{\text{CNN}} \) that estimates the fully-sampled image x from the output of the MLP, \( {\hat{\mathbf{x}}}_{\text{MLP}} \), using CNN can be represented as
where
where \( {\varvec{\upomega}}_{\text{n}} \) and \( {\varvec{\upbeta}}_{\text{n}} \) are convolution and bias matrices in each convolution layer, respectively, where \( {\text{n}} = 1,2,3, \ldots ,{\text{NCL}} \); \( * \) denotes the convolution operator; and NCL denotes the number of convolution layers. The loss function used for training the CNN is
where \( {\mathbf{x}}_{\text{m}} \) and \( {\hat{\mathbf{x}}}_{{{\text{MLP}},{\text{m}}}} \) denote the m-th fully-sampled image and the m-th estimated image of the MLP in the training dataset, respectively; and M denotes the batch size.
The final data fidelity is performed in k-space as a closed form solution [4] as
where
where \( {\text{k}}_{\text{x}} \) and \( {\text{k}}_{\text{y}} \) are horizontal and vertical indices of k-space, respectively. Then, the final solution of the proposed algorithm is obtained by IFT of \( {\hat{\mathbf{y}}}_{\text{f}} \) as \( {\hat{\mathbf{x}}}_{\text{f}} = {\mathcal{F}}_{2D}^{ - 1} \left( {{\hat{\mathbf{y}}}_{\text{f}} } \right) \).
More details for the CNN, including optimizer parameters, network specifications, including network depths, filter sizes, activation function, deep-learning libraries, and training/testing times, are provided in the supporting information.
2.4 Experimental Frameworks
All experiments conducted in the present study were approved by the institutional review board. Written informed consent was obtained from all human subjects. Two MR datasets were used: T2-fluid attenuated inversion recovery (T2-FLAIR) brain real-valued dataset provided by the Alzheimer’s Disease Neuroimaging Initiative (ADNI) [10], and T2-weighted complex-valued dataset acquired at our local institute. Details of data acquisition, including scanner information, sequence parameters, and the number of slices used for training and testing, are provided in the supporting information.
Undersampled k-space data were retrospectively obtained by subsampling the fully-sampled k-space data. Before undersampling, all MR images were normalized to a maximum magnitude of 1. A Cartesian undersampling scheme in a phase encoding direction (1D random) was used for generating undersampled k-space dataset. The sampling ratio for the undersampling was 25%. The binary undersampling masks is presented in the supporting information (Supporting Figure S1).
The proposed algorithm was compared with the following six algorithms: baseline zero-filling, CS-MRI [1], DL-MRI [4], a CNN-based algorithm by Wang et al. [5] (denoted as Wang’s algorithm), PANO [11], and FDLCP [12]. The parameters of the comparison algorithms are provided in the supporting information. Peak signal-to-noise ratio (PSNR) and structure similarity (SSIM) [13] were used for quantitative metrics. The patch size used to calculate SSIM was 11.
3 Results
3.1 The Conventional Image-to-Image MLP vs. the Proposed MLP
In Fig. 2, we evaluated the two different MLPs: The conventional MLP introduced in [8] which translates each vertical line of the 2D IFT of the undersampled k-space (i.e., the undersampled image) to the corresponding line of the target image and the proposed MLP which translates each vertical line of 1D IFT of the undersampled k-space to the corresponding line of the target image. Variables such as the number of outputs and the network depths in MLPs were exactly same. In Fig. 2, the proposed MLP (h) showed better reconstructions than the conventional image-to-image MLP (g) in terms of restoring details and reducing aliasing artifacts.
3.2 Comparison Study with Conventional Algorithms
Qualitative and quantitative comparisons are provided for the six conventional algorithms and the proposed algorithm. Figure 3 depicts the resultant images from the complex-valued T2-weighted dataset (from our local institute). The results of the real-valued T2-FLAIR dataset are provided in the supporting information (Supporting Figure S3). Table 1 depicts quantitative results for the two data sets. The proposed algorithm outperformed the six comparison algorithms in both qualitative and quantitative tests.
4 Conclusion
The current study presents a domain-transform algorithm comprising two deep-learning architectures (MLP and CNN) for reconstructing MR images from undersampled Cartesian k-space. Experimental results on the two different datasets demonstrated that the proposed algorithm outperforms the comparison state-of-the-art reconstruction algorithms. More discussion is provided in the supporting information.
References
Lusting, M., Donoho, D., Pauly, J.M.: The application of compressed sensing for rapid MR imaging. Magn. Reson. Med. 58(6), 1182–1195 (2007). https://doi.org/10.1002/mrm.21391
Griswold, M.A., et al.: Generalized autocalibrating partially parallel acquisitions (GRAPPA). Magn. Reson. Med. 247(6), 1202–1210 (2002). https://doi.org/10.1002/mrm.10171
Otazo, R., Candès, E., Sodickson, D.K.: Low‐rank plus sparse matrix decomposition for accelerated dynamic MRI with separation of background and dynamic components. Magn. Reson. Med. 73(3), 1125–1136 (2015). https://doi.org/10.1002/mrm.25240
Ravishankar, S., Bresler, Y.: MR image reconstruction from highly undersampled k-space data by dictionary learning. IEEE Trans. Med. Imaging 30(5), 1028–1041 (2015). https://doi.org/10.1109/TMI.2010.2090538
Wang, S., et al.: Accelerating magnetic resonance imaging via deep learning. In: Proceedings of the IEEE 13th International Symposium on Biomedical Imaging (ISBI), pp. 514–517 (2016)
Lee, D., Yoo, J., Ye, J.: Deep residual learning for compressed sensing MRI. In: Proceedings of the IEEE 14th International Symposium on Biomedical Imaging (ISBI), pp. 15–18 (2017)
Schlemper, J., Caballero, J., Hajnal, J.V., Price, A., Rueckert, D.: A deep cascade of convolutional neural networks for mr image reconstruction. In: Niethammer, M., et al. (eds.) IPMI 2017. LNCS, vol. 10265, pp. 647–658. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-59050-9_51
Kwon, K., Kim, D., Park, H.: A parallel MR imaging method using multilayer perceptron. Med. Phys. 44(12), 6209–6224 (2017). https://doi.org/10.1002/mp.12600
Zhu, B., Liu, J.Z., Rosen, R.B., Rosen, M.S.: Image reconstruction by domain transform manifold learning. Nature 555(7697), 487–492 (2018). https://doi.org/10.1038/nature25988
Jack, C.R., et al.: The Alzheimer’s disease neuroimaging initiative (ADNI): MRI methods. J. Magn. Reson. Imaging 27(4), 685–691 (2008). https://doi.org/10.1002/jmri.21049
Qu, X., Hou, Y., Lam, F., Guo, D., Zhong, J., Chen, Z.: Magnetic resonance image reconstruction from undersampled measurements using a patch–based nonlocal operator. Med. Image Anal. 18(6), 843–856 (2014). https://doi.org/10.1016/j.media.2013.09.007
Zhan, Z., Cai, J.-F., Guo, D., Liu, Y., Chen, Z., Qu, X.: Fast multiclass dictionaries learning with geometrical directions in MRI reconstruction. IEEE Trans. Biomed. Eng. 63(9), 1850–1861 (2016). https://doi.org/10.1109/TBME.2015.2503756
Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 600–612 (2004). https://doi.org/10.1109/TIP.2003.819861
Acknowledgements
This research was supported by the National Research Foundation of Korea (NRF) grant funded by the Korean government (MSIP) (2016R1A2B4015016) and was partially supported by the Graduate School of YONSEI University Research Scholarship Grants in 2017 and the Brain Korea 21 Plus Project of Dept. of Electrical and Electronic Engineering, Yonsei University in 2018.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Copyright information
© 2018 Springer Nature Switzerland AG
About this paper
Cite this paper
Eo, T., Shin, H., Kim, T., Jun, Y., Hwang, D. (2018). Translation of 1D Inverse Fourier Transform of K-space to an Image Based on Deep Learning for Accelerating Magnetic Resonance Imaging. In: Frangi, A., Schnabel, J., Davatzikos, C., Alberola-López, C., Fichtinger, G. (eds) Medical Image Computing and Computer Assisted Intervention – MICCAI 2018. MICCAI 2018. Lecture Notes in Computer Science(), vol 11070. Springer, Cham. https://doi.org/10.1007/978-3-030-00928-1_28
Download citation
DOI: https://doi.org/10.1007/978-3-030-00928-1_28
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-00927-4
Online ISBN: 978-3-030-00928-1
eBook Packages: Computer ScienceComputer Science (R0)