ISSN 0253-2778

CN 34-1054/N

Open AccessOpen Access JUSTC Life Sciences 01 April 2024

IDDNet: a deep interactive dual-domain convolutional neural network with auxiliary modality for fast MRI reconstruction

Cite this:
https://doi.org/10.52396/JUSTC-2023-0169
More Information
  • Author Bio:

    Yi Cao is currently pursuing a master’s degree at the University of Science and Technology of China. Her research focuses mainly on fast MRI reconstruction based on deep learning

    Hongwei Du is currently an Associate Professor at the School of Information Science and Technology, University of Science and Technology of China (USTC). He received his Ph.D. degree from the Department of Electronic Science and Technology, USTC, in 2007. His research interests include medical image processing based on deep learning, fast MRI imaging principle and algorithm, and advanced biomedical imaging system

  • Corresponding author: E-mail: duhw@ustc.edu.cn
  • Received Date: 31 December 2023
  • Accepted Date: 22 February 2024
  • Available Online: 01 April 2024
  • Reconstructing a complete image accurately from an undersampled k-space matrix is a viable approach for magnetic resonance imaging (MRI) acceleration. In recent years, numerous deep learning (DL)-based methods have been employed to improve MRI reconstruction. Among these methods, the cross-domain method has been proven to be effective. However, existing cross-domain reconstruction algorithms sequentially link the image domain and k-space networks, disregarding the interplay between different domains, consequently leading to a deficiency in reconstruction accuracy. In this work, we propose a deep interactive dual-domain network (IDDNet) with an auxiliary modality for accelerating MRI reconstruction to effectively extract pertinent information from multiple MR domains and modalities. The IDDNet first extracts shallow features from low-resolution target modalities in the image domain to obtain visual representation information. In the following feature processing, a parallel interactive architecture with dual branches is designed to extract deep features from relevant information of dual domains simultaneously to avoid redundant priority priors in sequential links. Furthermore, the model uses additional information from the auxiliary modality to refine the structure and improve the reconstruction accuracy. Numerous experiments at different sampling masks and acceleration rates on the MICCAI BraTS 2019 brain and fastMRI knee datasets show that IDDNet achieves excellent accelerated MRI reconstruction performance.
    We propose IDDNet, a parallel dual-domain interaction framework with an auxiliary modality, to accelerate MRI reconstruction.
    Reconstructing a complete image accurately from an undersampled k-space matrix is a viable approach for magnetic resonance imaging (MRI) acceleration. In recent years, numerous deep learning (DL)-based methods have been employed to improve MRI reconstruction. Among these methods, the cross-domain method has been proven to be effective. However, existing cross-domain reconstruction algorithms sequentially link the image domain and k-space networks, disregarding the interplay between different domains, consequently leading to a deficiency in reconstruction accuracy. In this work, we propose a deep interactive dual-domain network (IDDNet) with an auxiliary modality for accelerating MRI reconstruction to effectively extract pertinent information from multiple MR domains and modalities. The IDDNet first extracts shallow features from low-resolution target modalities in the image domain to obtain visual representation information. In the following feature processing, a parallel interactive architecture with dual branches is designed to extract deep features from relevant information of dual domains simultaneously to avoid redundant priority priors in sequential links. Furthermore, the model uses additional information from the auxiliary modality to refine the structure and improve the reconstruction accuracy. Numerous experiments at different sampling masks and acceleration rates on the MICCAI BraTS 2019 brain and fastMRI knee datasets show that IDDNet achieves excellent accelerated MRI reconstruction performance.
    • We extract and interactively utilize relevant information from different MR domains by designing a hierarchical feature extraction and dual-domain parallel interaction network structure.
    • We combine multimodal auxiliary imaging technology and leverage similar information between different MR modalities to enhance the reconstruction effect.
    • The experimental results show that our model provides better visual and quantitative reconstruction results at both low and high acceleration rates with various sampling masks.

  • loading
  • [1]
    Griswold M A, Jakob P M, Heidemann R M, et al. Generalized autocalibrating partially parallel acquisitions (GRAPPA). Magnetic Resonance in Medicine, 2002, 47 (6): 1202–1210. doi: 10.1002/mrm.10171
    [2]
    Lustig M, Donoho D, Pauly J M. Sparse MRI: The application of compressed sensing for rapid MR imaging. Magnetic Resonance in Medicine, 2007, 58 (6): 1182–1195. doi: 10.1002/mrm.21391
    [3]
    Pruessmann K P, Weiger M, Scheidegger M B, et al. SENSE: sensitivity encoding for fast MRI. Magnetic Resonance in Medicine, 1999, 42 (5): 952–962. doi: 10.1002/(SICI)1522-2594(199911)42:5<952::AID-MRM16>3.0.CO;2-S
    [4]
    Lustig M, Pauly J M. SPIRiT: Iterative self-consistent parallel imaging reconstruction from arbitrary k -space. Magnetic Resonance in Medicine, 2010, 64 (2): 457–471. doi: 10.1002/mrm.22428
    [5]
    Uecker M, Lai P, Murphy M J, et al. ESPIRiT—an eigenvalue approach to autocalibrating parallel MRI: where SENSE meets GRAPPA. Magnetic Resonance in Medicine, 2014, 71 (3): 990–1001. doi: 10.1002/mrm.24751
    [6]
    Lustig M, Donoho D L, Santos J M, et al. Compressed sensing MRI. IEEE Signal Processing Magazine, 2008, 25 (2): 72–82. doi: 10.1109/MSP.2007.914728
    [7]
    Qu X, Cao X, Guo D, et al. Combined sparsifying transforms for compressed sensing MRI. Electronics Letters, 2010, 46 (2): 121–123. doi: 10.1049/el.2010.1845
    [8]
    Haldar J P, Zhuo J W. P-LORAKS: Low-rank modeling of local k -space neighborhoods with parallel imaging data. Magnetic Resonance in Medicine, 2016, 75 (4): 1499–1514. doi: 10.1002/mrm.25717
    [9]
    Block K T, Uecker M, Frahm J. Undersampled radial MRI with multiple coils. Iterative image reconstruction using a total variation constraint. Magnetic Resonance in Medicine, 2007, 57 (6): 1086–1098. doi: 10.1002/mrm.21236
    [10]
    Wang S S, Su Z H, Ying L, et al. Accelerating magnetic resonance imaging via deep learning. In: 2016 IEEE13th International Symposium on Biomedical Imaging (ISBI). IEEE, 2016: 514-517.
    [11]
    Schlemper J, Caballero J, Hajnal J V, et al. A deep cascade of convolutional neural networks for dynamic MR image reconstruction. IEEE Transactions on Medical Imaging, 2018, 37 (2): 491–503. doi: 10.1109/TMI.2017.2760978
    [12]
    Sun L Y, Fan Z W, Huang Y, et al. Compressed sensing MRI using a recursive dilated network. In: Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence(AAAI-18). New Orleans, LA: AAAI, 2018: 2444-2451.
    [13]
    Ding P L K, Li Z Q, Zho Y X, et al. Deep residual dense U-Net for resolution enhancement in accelerated MRI acquisition. In: Proc. SPIE 10949, Medical Imaging 2019: Image Processing. SPIE, 2019 , 10949: 109490F.
    [14]
    Dai Y X, Zhuang P X. Compressed sensing MRI via a multi-scale dilated residual convolution network. Magnetic Resonance Imaging, 2019, 63: 93–104. doi: 10.1016/j.mri.2019.07.014
    [15]
    Yang G, Yu S M, Dong H, et al. DAGAN: deep de-aliasing generative adversarial networks for fast compressed sensing MRI reconstruction. IEEE Transactions on Medical Imaging, 2018, 37 (6): 1310–1321. doi: 10.1109/TMI.2017.2785879
    [16]
    Quan T M, Nguyen-Duc T, Jeong W K. Compressed sensing MRI reconstruction using a generative adversarial network with a cyclic loss. IEEE Transactions on Medical Imaging, 2018, 37 (6): 1488–1497. doi: 10.1109/TMI.2018.2820120
    [17]
    Eo T, Jun Y, Kim T, et al. KIKI-net: cross-domain convolutional neural networks for reconstructing undersampled magnetic resonance images. Magnetic Resonance in Medicine, 2018, 80 (5): 2188–2201. doi: 10.1002/mrm.27201
    [18]
    Wang Z L, Jiang H T, Du H W, et al. IKWI-net: A cross-domain convolutional neural network for undersampled magnetic resonance image reconstruction. Magnetic Resonance Imaging, 2020, 73: 1–10. doi: 10.1016/j.mri.2020.06.015
    [19]
    Zhou B, Zhou S K. DuDoRNet: learning a dual-domain recurrent network for fast MRI reconstruction with deep T1 prior. In: Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE Press, 2020: 4272-4281.
    [20]
    Ran M S, Xia W J, Huang Y Q, et al. MD-Recon-Net: a parallel dual-domain convolutional neural network for compressed sensing MRI. IEEE Transactions on Radiation and Plasma Medical Sciences, 2021, 5 (1): 120–135. doi: 10.1109/TRPMS.2020.2991877
    [21]
    Liu Y, Pang Y W, Liu X H, et al. DIIK-Net: A full-resolution cross-domain deep interaction convolutional neural network for MR image reconstruction. Neurocomputing, 2023, 517: 213–222. doi: 10.1016/j.neucom.2022.09.048
    [22]
    Han Y, Sunwoo L, Ye J C. k -space deep learning for accelerated MRI. IEEE Transactions on Medical Imaging, 2020, 39 (2): 377–386. doi: 10.1109/TMI.2019.2927101
    [23]
    Ronneberger O, Fischer P, Brox T. U-Net: Convolutional Networks for Biomedical Image Segmentation. In: Navab N, Hornegger J, Wells W, et al. , editors. Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015. Cham: Springer, 2015: 234-241.
    [24]
    Hinton G E, Salakhutdinov R R. Reducing the dimensionality of data with neural networks. Science, 2006, 313 (5786): 504–507. doi: 10.1126/science.1127647
    [25]
    Martínez H P, Yannakakis G N. Deep multimodal fusion: Combining discrete events and continuous signals. In: ICMI ’14: Proceedings of the 16th International Conference on Multimodal Interaction. New York: ACM, 2014: 34-41.
    [26]
    Xiang L, Chen Y, Chang W T, et al. Deep-learning-based multi-modal fusion for fast MR reconstruction. IEEE Transactions on Biomedical Engineering, 2018, 66 (7): 2105–2114. doi: 10.1109/TBME.2018.2883958
    [27]
    Xuan K, Xiang L, Huang X Q, et al. Multimodal MRI reconstruction assisted with spatial alignment network. IEEE Transactions on Medical Imaging, 2022, 41 (9): 2499–2509. doi: 10.1109/TMI.2022.3164050
    [28]
    Zhang Y L, Tian Y P, Kong Y, et al. Residual Dense Network for Image Super-Resolution. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE Press, 2018: 2472-2481.
    [29]
    Kim D W, Chung J R, Jung S W. GRDN: grouped residual dense network for real image denoising and GAN-based real-world noise modeling. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). Piscataway: IEEE Press, 2019: 2086-2094.
    [30]
    Hu J, Shen L, Sun G. Squeeze-and-Excitation Networks. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE Press, 2018: 7132-7141.
    [31]
    Woo S H, Park J, Lee J Y, et al. CBAM: Convolutional Block Attention Module. In: Ferrari V, Hebert M, Sminchisescu C, et al. , editors. Computer Vision – ECCV 2018. Cham: Springer, 2018: 3-19.
    [32]
    Qin C, Schlemper J, Caballero J, et al. Convolutional recurrent neural networks for dynamic MR image reconstruction. IEEE Transactions on Medical Imaging, 2019, 38 (1): 280–290. doi: 10.1109/TMI.2018.2863670
    [33]
    Bakas S, Reyes M, Jakab A, et al. Identifying the best machine learning algorithms for brain tumor segmentation, progression assessment, and overall survival prediction in the BRATS challenge. arXiv: 1811.02629, 2018 .
    [34]
    Menze B H, Jakab A, Bauer S, et al. The Multimodal Brain Tumor Image Segmentation Benchmark (BRATS). IEEE Transactions on Medical Imaging, 2015, 34 (10): 1993–2024. doi: 10.1109/TMI.2014.2377694
    [35]
    Zbontar J K F, Sriram A. fastMRI: An Open Dataset and Benchmarks for Accelerated MRI. arXiv: 1811.08839, 2018 .
    [36]
    Xuan K, Sun S, Xue Z, et al. Learning MRI k-space subsampling pattern using progressive weight pruning. In: Medical Image Computing and Computer Assisted Intervention – MICCAI 2020. Cham: Springer, 2020: 178-187.
    [37]
    Wang P Q, Chen P F, Yuan Y, et al. Understanding convolution for semantic segmentation. In: 2018 IEEE Winter Conference on Applications of Computer Vision (WACV). Piscataway: IEEE Press, 2018: 1451-1460.
  • 加载中

Catalog

    Figure  1.  Relationships among the different MRI modalities used for reconstruction. The FFT operator refers to the 2D fast Fourier transform, and the IFFT operator refers to the 2D inverse fast Fourier transform.

    Figure  2.  Visualization of data in different domains. (a) Spatial (image) domain visualization of cosine waves; (b) frequency domain visualization of cosine waves; (c) image domain visualization of MR data; (d) frequency domain (k-space) visualization of MR data.

    Figure  3.  Overview of the architecture of the proposed IDDNet.

    Figure  4.  Block diagrams of component modules. (a) The residual dense (RD) block, which contains several 3×3 convolution and LeakyReLU combined convolution layers and one 1×1 convolution layer for dimensionality reduction. The growth rate of the dense blocks is set to 32, and the number of combined convolution layers is set to 6. (b) Fusion attention (FA) block. (c) The residual dilated dense attention (RDDA) block, which uses the residual dilated dense (RDD) block and FA block for feature extraction, and 1×1 convolution for channel dimensionality reduction. The RDD module replaced the ordinary convolution in Fig. 4(a) with dilated convolution of different dilation rates. (d) Dual-domain interaction (DDI) block.

    Figure  5.  The masks used in the experiments. The masks displayed in the sequence from left to right are labeled 4× Random mask, 4× Equispaced mask, 8× Random mask and 8× Equispaced mask.

    Figure  6.  Visual comparisons and error maps of IDDNet with the state-of-the-art methods on the BraTS2019 dataset.

    Figure  7.  Visual comparisons and error maps of IDDNet with the state-of-the-art methods on the fastMRI dataset.

    Figure  8.  Visual comparisons of IDDNet and its variations on the BraTs2019 dataset.

    [1]
    Griswold M A, Jakob P M, Heidemann R M, et al. Generalized autocalibrating partially parallel acquisitions (GRAPPA). Magnetic Resonance in Medicine, 2002, 47 (6): 1202–1210. doi: 10.1002/mrm.10171
    [2]
    Lustig M, Donoho D, Pauly J M. Sparse MRI: The application of compressed sensing for rapid MR imaging. Magnetic Resonance in Medicine, 2007, 58 (6): 1182–1195. doi: 10.1002/mrm.21391
    [3]
    Pruessmann K P, Weiger M, Scheidegger M B, et al. SENSE: sensitivity encoding for fast MRI. Magnetic Resonance in Medicine, 1999, 42 (5): 952–962. doi: 10.1002/(SICI)1522-2594(199911)42:5<952::AID-MRM16>3.0.CO;2-S
    [4]
    Lustig M, Pauly J M. SPIRiT: Iterative self-consistent parallel imaging reconstruction from arbitrary k -space. Magnetic Resonance in Medicine, 2010, 64 (2): 457–471. doi: 10.1002/mrm.22428
    [5]
    Uecker M, Lai P, Murphy M J, et al. ESPIRiT—an eigenvalue approach to autocalibrating parallel MRI: where SENSE meets GRAPPA. Magnetic Resonance in Medicine, 2014, 71 (3): 990–1001. doi: 10.1002/mrm.24751
    [6]
    Lustig M, Donoho D L, Santos J M, et al. Compressed sensing MRI. IEEE Signal Processing Magazine, 2008, 25 (2): 72–82. doi: 10.1109/MSP.2007.914728
    [7]
    Qu X, Cao X, Guo D, et al. Combined sparsifying transforms for compressed sensing MRI. Electronics Letters, 2010, 46 (2): 121–123. doi: 10.1049/el.2010.1845
    [8]
    Haldar J P, Zhuo J W. P-LORAKS: Low-rank modeling of local k -space neighborhoods with parallel imaging data. Magnetic Resonance in Medicine, 2016, 75 (4): 1499–1514. doi: 10.1002/mrm.25717
    [9]
    Block K T, Uecker M, Frahm J. Undersampled radial MRI with multiple coils. Iterative image reconstruction using a total variation constraint. Magnetic Resonance in Medicine, 2007, 57 (6): 1086–1098. doi: 10.1002/mrm.21236
    [10]
    Wang S S, Su Z H, Ying L, et al. Accelerating magnetic resonance imaging via deep learning. In: 2016 IEEE13th International Symposium on Biomedical Imaging (ISBI). IEEE, 2016: 514-517.
    [11]
    Schlemper J, Caballero J, Hajnal J V, et al. A deep cascade of convolutional neural networks for dynamic MR image reconstruction. IEEE Transactions on Medical Imaging, 2018, 37 (2): 491–503. doi: 10.1109/TMI.2017.2760978
    [12]
    Sun L Y, Fan Z W, Huang Y, et al. Compressed sensing MRI using a recursive dilated network. In: Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence(AAAI-18). New Orleans, LA: AAAI, 2018: 2444-2451.
    [13]
    Ding P L K, Li Z Q, Zho Y X, et al. Deep residual dense U-Net for resolution enhancement in accelerated MRI acquisition. In: Proc. SPIE 10949, Medical Imaging 2019: Image Processing. SPIE, 2019 , 10949: 109490F.
    [14]
    Dai Y X, Zhuang P X. Compressed sensing MRI via a multi-scale dilated residual convolution network. Magnetic Resonance Imaging, 2019, 63: 93–104. doi: 10.1016/j.mri.2019.07.014
    [15]
    Yang G, Yu S M, Dong H, et al. DAGAN: deep de-aliasing generative adversarial networks for fast compressed sensing MRI reconstruction. IEEE Transactions on Medical Imaging, 2018, 37 (6): 1310–1321. doi: 10.1109/TMI.2017.2785879
    [16]
    Quan T M, Nguyen-Duc T, Jeong W K. Compressed sensing MRI reconstruction using a generative adversarial network with a cyclic loss. IEEE Transactions on Medical Imaging, 2018, 37 (6): 1488–1497. doi: 10.1109/TMI.2018.2820120
    [17]
    Eo T, Jun Y, Kim T, et al. KIKI-net: cross-domain convolutional neural networks for reconstructing undersampled magnetic resonance images. Magnetic Resonance in Medicine, 2018, 80 (5): 2188–2201. doi: 10.1002/mrm.27201
    [18]
    Wang Z L, Jiang H T, Du H W, et al. IKWI-net: A cross-domain convolutional neural network for undersampled magnetic resonance image reconstruction. Magnetic Resonance Imaging, 2020, 73: 1–10. doi: 10.1016/j.mri.2020.06.015
    [19]
    Zhou B, Zhou S K. DuDoRNet: learning a dual-domain recurrent network for fast MRI reconstruction with deep T1 prior. In: Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE Press, 2020: 4272-4281.
    [20]
    Ran M S, Xia W J, Huang Y Q, et al. MD-Recon-Net: a parallel dual-domain convolutional neural network for compressed sensing MRI. IEEE Transactions on Radiation and Plasma Medical Sciences, 2021, 5 (1): 120–135. doi: 10.1109/TRPMS.2020.2991877
    [21]
    Liu Y, Pang Y W, Liu X H, et al. DIIK-Net: A full-resolution cross-domain deep interaction convolutional neural network for MR image reconstruction. Neurocomputing, 2023, 517: 213–222. doi: 10.1016/j.neucom.2022.09.048
    [22]
    Han Y, Sunwoo L, Ye J C. k -space deep learning for accelerated MRI. IEEE Transactions on Medical Imaging, 2020, 39 (2): 377–386. doi: 10.1109/TMI.2019.2927101
    [23]
    Ronneberger O, Fischer P, Brox T. U-Net: Convolutional Networks for Biomedical Image Segmentation. In: Navab N, Hornegger J, Wells W, et al. , editors. Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015. Cham: Springer, 2015: 234-241.
    [24]
    Hinton G E, Salakhutdinov R R. Reducing the dimensionality of data with neural networks. Science, 2006, 313 (5786): 504–507. doi: 10.1126/science.1127647
    [25]
    Martínez H P, Yannakakis G N. Deep multimodal fusion: Combining discrete events and continuous signals. In: ICMI ’14: Proceedings of the 16th International Conference on Multimodal Interaction. New York: ACM, 2014: 34-41.
    [26]
    Xiang L, Chen Y, Chang W T, et al. Deep-learning-based multi-modal fusion for fast MR reconstruction. IEEE Transactions on Biomedical Engineering, 2018, 66 (7): 2105–2114. doi: 10.1109/TBME.2018.2883958
    [27]
    Xuan K, Xiang L, Huang X Q, et al. Multimodal MRI reconstruction assisted with spatial alignment network. IEEE Transactions on Medical Imaging, 2022, 41 (9): 2499–2509. doi: 10.1109/TMI.2022.3164050
    [28]
    Zhang Y L, Tian Y P, Kong Y, et al. Residual Dense Network for Image Super-Resolution. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE Press, 2018: 2472-2481.
    [29]
    Kim D W, Chung J R, Jung S W. GRDN: grouped residual dense network for real image denoising and GAN-based real-world noise modeling. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). Piscataway: IEEE Press, 2019: 2086-2094.
    [30]
    Hu J, Shen L, Sun G. Squeeze-and-Excitation Networks. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE Press, 2018: 7132-7141.
    [31]
    Woo S H, Park J, Lee J Y, et al. CBAM: Convolutional Block Attention Module. In: Ferrari V, Hebert M, Sminchisescu C, et al. , editors. Computer Vision – ECCV 2018. Cham: Springer, 2018: 3-19.
    [32]
    Qin C, Schlemper J, Caballero J, et al. Convolutional recurrent neural networks for dynamic MR image reconstruction. IEEE Transactions on Medical Imaging, 2019, 38 (1): 280–290. doi: 10.1109/TMI.2018.2863670
    [33]
    Bakas S, Reyes M, Jakab A, et al. Identifying the best machine learning algorithms for brain tumor segmentation, progression assessment, and overall survival prediction in the BRATS challenge. arXiv: 1811.02629, 2018 .
    [34]
    Menze B H, Jakab A, Bauer S, et al. The Multimodal Brain Tumor Image Segmentation Benchmark (BRATS). IEEE Transactions on Medical Imaging, 2015, 34 (10): 1993–2024. doi: 10.1109/TMI.2014.2377694
    [35]
    Zbontar J K F, Sriram A. fastMRI: An Open Dataset and Benchmarks for Accelerated MRI. arXiv: 1811.08839, 2018 .
    [36]
    Xuan K, Sun S, Xue Z, et al. Learning MRI k-space subsampling pattern using progressive weight pruning. In: Medical Image Computing and Computer Assisted Intervention – MICCAI 2020. Cham: Springer, 2020: 178-187.
    [37]
    Wang P Q, Chen P F, Yuan Y, et al. Understanding convolution for semantic segmentation. In: 2018 IEEE Winter Conference on Applications of Computer Vision (WACV). Piscataway: IEEE Press, 2018: 1451-1460.

    Article Metrics

    Article views (73) PDF downloads(443)
    Proportional views

    /

    DownLoad:  Full-Size Img  PowerPoint
    Return
    Return