Reproducibility of Deep Learning Algorithms Developed for Medical Imaging Analysis: A Systematic Review

Since 2000, there have been more than 8000 publications on radiology artificial intelligence (AI). AI breakthroughs allow complex tasks to be automated and even performed beyond human capabilities. However, the lack of details on the methods and algorithm code undercuts its scientific value. Many science subfields have recently faced a reproducibility crisis, eroding trust in processes and results, and influencing the rise in retractions of scientific papers. For the same reasons, conducting research in deep learning (DL) also requires reproducibility. Although several valuable manuscript checklists for AI in medical imaging exist, they are not focused specifically on reproducibility. In this study, we conducted a systematic review of recently published papers in the field of DL to evaluate if the description of their methodology could allow the reproducibility of their findings. We focused on the Journal of Digital Imaging (JDI), a specialized journal that publishes papers on AI and medical imaging. We used the keyword “Deep Learning” and collected the articles published between January 2020 and January 2022. We screened all the articles and included the ones which reported the development of a DL tool in medical imaging. We extracted the reported details about the dataset, data handling steps, data splitting, model details, and performance metrics of each included article. We found 148 articles. Eighty were included after screening for articles that reported developing a DL model for medical image analysis. Five studies have made their code publicly available, and 35 studies have utilized publicly available datasets. We provided figures to show the ratio and absolute count of reported items from included studies. According to our cross-sectional study, in JDI publications on DL in medical imaging, authors infrequently report the key elements of their study to make it reproducible.

AI:

Artificial intelligence

DL:

Deep learning

JDI:

Journal of Digital Imaging

CLAIM:

Checklist for Artificial Intelligence in Medical Imaging

PRISMA:

Preferred Reporting Items for Systematic Reviews and Meta-analyses

Authors and Affiliations

1. Artificial Intelligence Lab, Department of Radiology, Mayo Clinic, Rochester, MN, USA

   Mana Moassefi, Pouria Rouzrokh, Gian Marco Conte, Sanaz Vahdati, Shiba Kuanar, Bradley J. Erickson & Shahriar Faghani

2. Orthopedic Surgery Artificial Intelligence Laboratory (OSAIL), Department of Orthopedic Surgery, Mayo Clinic, Rochester, MN, USA

   Pouria Rouzrokh

3. Department of Radiology, University Hospitals Cleveland, Cleveland, OH, USA

   Tianyuan Fu

4. Department of Radiology, Thomas Jefferson University, Philadelphia, PA, USA

   Aylin Tahmasebi

5. Cleveland Clinic Children’s, Cleveland, OH, USA

   Mira Younis

6. National Cancer Institute, National Institutes of Health, Bethesda, MA, USA

   Keyvan Farahani

7. Department of Radiology, University of California, San Diego, CA, USA

   Amilcare Gentili

8. Department of Radiology, Mayo Clinic, Rochester, MN, USA

   Timothy Kline

9. DasaInova, Diagnósticos da América S.A, São Paulo, Brazil

   Felipe C. Kitamura

10. Department of Electrical Engineering & Computer Science, Vanderbilt University, Nashville, TN, USA

    Yuankai Huo

11. Phillips Research North America, Cambridge, MD, USA

    Khaled Younis

Contributions

All authors contributed to the study conception and design. Material preparation, and analysis were performed mostly by Shahriar Faghani, Mana Moassefi, and data collection was performed by the team. The first draft of the manuscript was written by Mana Moassefi and all authors commented on previous versions of the manuscript. All authors read and approved the final manuscript.

Corresponding authors

Correspondence to Mana Moassefi or Shahriar Faghani.

Ethics Approval

This paper is a review. We have not received any approval from universities for conducting this.

Consent to Participate

Not applicable.

Consent to Publish

Not applicable.

Conflict of Interest

The authors declare no competing interests.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions