[go: up one dir, main page]
More Web Proxy on the site http://driver.im/ Skip to main content
Springer Nature Link
Log in

Integrating Active Learning and Transfer Learning for Carotid Intima-Media Thickness Video Interpretation

  • Published:
Journal of Digital Imaging Aims and scope Submit manuscript

Abstract

Cardiovascular disease (CVD) is the number one killer in the USA, yet it is largely preventable (World Health Organization 2011). To prevent CVD, carotid intima-media thickness (CIMT) imaging, a noninvasive ultrasonography method, has proven to be clinically valuable in identifying at-risk persons before adverse events. Researchers are developing systems to automate CIMT video interpretation based on deep learning, but such efforts are impeded by the lack of large annotated CIMT video datasets. CIMT video annotation is not only tedious, laborious, and time consuming, but also demanding of costly, specialty-oriented knowledge and skills, which are not easily accessible. To dramatically reduce the cost of CIMT video annotation, this paper makes three main contributions. Our first contribution is a new concept, called Annotation Unit (AU), which simplifies the entire CIMT video annotation process down to six simple mouse clicks. Our second contribution is a new algorithm, called AFT (active fine-tuning), which naturally integrates active learning and transfer learning (fine-tuning) into a single framework. AFT starts directly with a pre-trained convolutional neural network (CNN), focuses on selecting the most informative and representative AU s from the unannotated pool for annotation, and then fine-tunes the CNN by incorporating newly annotated AU s in each iteration to enhance the CNN’s performance gradually. Our third contribution is a systematic evaluation, which shows that, in comparison with the state-of-the-art method (Tajbakhsh et al., IEEE Trans Med Imaging 35(5):1299–1312, 2016), our method can cut the annotation cost by >81% relative to their training from scratch and >50% relative to their random selection. This performance is attributed to the several advantages derived from the advanced active, continuous learning capability of our AFT method.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
£29.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price includes VAT (United Kingdom)

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

References

  1. World Health Organization. Global atlas on cardiovascular disease prevention and control. Available at http://www.who.int/cardiovascular_diseases/publications (September 19, 2011)

  2. Al Rahhal Ms, Bazi Y, AlHichri H, Alajlan N, Melgani F, Yager R: Deep learning approach for active classification of electrocardiogram signals. Inf Sci 345: 340–354, 2016

    Article  Google Scholar 

  3. Carneiro G, Nascimento J, Bradley A: Unregistered multiview mammogram analysis with pre-trained deep learning models. In: (Navab N, Hornegger J, Wells WM, Frangi AF, Eds.) Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015, Lecture Notes in Computer Science, vol. 9351, pp. 652–660. Springer International Publishing, 2015. https://doi.org/10.1007/978-3-319-24574-4_78

  4. Chen H, Ni D, Qin J, Li S, Yang X, Wang T, Heng PA: Standard plane localization in fetal ultrasound via domain transferred deep neural networks. IEEE J Biomed Health Inform 19 (5): 1627–1636, 2015

    Article  PubMed  Google Scholar 

  5. Delsanto S, Molinari F, Giustetto P, Liboni W, Badalamenti S, Suri JS: Characterization of a completely user-independent algorithm for carotid artery segmentation in 2-d ultrasound images. IEEE Trans Instrum Meas 56 (4): 1265–1274, 2007

    Article  Google Scholar 

  6. Deng J, Dong W, Socher R, Li LJ, Li K: Fei-fei, L.: ImageNet: A Large-Scale Hierarchical Image Database CVPR09, 2009

  7. Gao M, Bagci U, Lu L, Wu A, Buty M, Shin HC, Roth H, Papadakis GZ, Depeursinge A, Summers RM, et al: Holistic classification of ct attenuation patterns for interstitial lung diseases via deep convolutional neural networks.. In: The 1st workshop on deep learning in medical image analysis, international conference on medical image computing and computer assisted intervention, at MICCAI-DLMIA’15, 2015

  8. Gepner AD, Young R, Delaney JA, Tattersall MC, Blaha MJ, Post WS, Gottesman RF, Kronmal R, Budoff MJ, Burke GL, et al: A comparison of coronary artery calcium presence, carotid plaque presence, and carotid intima-media thickness for cardiovascular disease prediction in the multi-ethnic study of atherosclerosis (mesa). Circulation Cardiovascular Imaging 8 (1): e002262, 2015

    Article  PubMed  PubMed Central  Google Scholar 

  9. Greenspan H, van Ginneken B, Summers RM: Guest editorial deep learning in medical imaging: Overview and future promise of an exciting new technique. IEEE Trans Med Imaging 35 (5): 1153–1159, 2016

    Article  Google Scholar 

  10. Guyon I, Cawley G, Dror G, Lemaire V, Statnikov A (2011) JMLR Workshop and conference proceedings (volume 16): Active learning challenge microtome publishing

  11. He K, Zhang X, Ren S, Sun J: Deep residual learning for image recognition.. In: The IEEE conference on computer vision and pattern recognition (CVPR), 2016

  12. Hoo-Chang S, Roth HR, Gao M, Lu L, Xu Z, Nogues I, Yao J, Mollura D, Summers RM: Deep convolutional neural networks for computer-aided detection: Cnn architectures, dataset characteristics and transfer learning. IEEE Trans Med Imaging 35 (5): 1285, 2016

    Article  PubMed Central  Google Scholar 

  13. Hurst RT, Burke RF, Wissner E, Roberts A, Kendall CB, Lester SJ, Somers V, Goldman ME, Wu Q, Khandheria B: Incidence of subclinical atherosclerosis as a marker of cardiovascular risk in retired professional football players. Am J Cardiol 105 (8): 1107–1111, 2010

    Article  PubMed  Google Scholar 

  14. Jia Y, Shelhamer E, Donahue J, Karayev S, Long J, Girshick R, Guadarrama S, Darrell T (2014) Caffe: Convolutional architecture for fast feature embedding. arXiv:http://arXiv.org/abs/1408.5093

  15. Kass M, Witkin A, Terzopoulos D: Snakes: Active contour models. Int J Comput Vis 1 (4): 321–331, 1988

    Article  Google Scholar 

  16. Krizhevsky A, Sutskever I, Hinton GE: Imagenet classification with deep convolutional neural networks.. In: Advances in neural information processing systems, pp 1097–1105, 2012

  17. Li J: Active learning for hyperspectral image classification with a stacked autoencoders based neural network.. In: 2016 IEEE International conference on image processing (ICIP), pp 1062–1065, 2016. https://doi.org/10.1109/ICIP.2016.7532520

  18. Liang J, McInerney T, Terzopoulos D: United snakes. Med Image Anal 10 (2): 215–233, 2006

    Article  PubMed  Google Scholar 

  19. Liang Q, Wendelhag I, Wikstrand J, Gustavsson T: A multiscale dynamic programming procedure for boundary detection in ultrasonic artery images. IEEE Trans Med Imaging 19 (2): 127–142, 2000

    Article  CAS  PubMed  Google Scholar 

  20. Loizou CP, Pattichis CS, Pantziaris M, Nicolaides A: An integrated system for the segmentation of atherosclerotic carotid plaque. IEEE Trans Inf Technol Biomed 11 (6): 661–667, 2007

    Article  PubMed  Google Scholar 

  21. Long J, Shelhamer E, Darrell T: Fully convolutional networks for semantic segmentation.. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3431–3440, 2015

  22. Lorenz MW, Markus HS, Bots ML, Rosvall M, Sitzer M: Prediction of clinical cardiovascular events with carotid intima-media thickness. Circulation 115 (4): 459–467, 2007

    Article  Google Scholar 

  23. Margeta J, Criminisi A, Cabrera Lozoya R, Lee DC, Ayache N: Fine-tuned convolutional neural nets for cardiac mri acquisition plane recognition. Comput Methods Biomech Biomed Eng: Imaging Visual 5 (5): 339–349, 2017

    Google Scholar 

  24. Menchón-Lara RM, Bastida-Jumilla MC, González-López A, Sancho-Gómez JL: Automatic evaluation of carotid intima-media thickness in ultrasounds using machine learning.. In: Natural and artificial computation in engineering and medical applications, pp 241–249. Springer, 2013

  25. Menchón-Lara RM, Sancho-Gómez JL: Fully automatic segmentation of ultrasound common carotid artery images based on machine learning. Neurocomputing 151: 161–167, 2015

    Article  Google Scholar 

  26. Schlegl T, Ofner J, Langs G: Unsupervised pre-training across image domains improves lung tissue classification.. In: Medical computer vision: Algorithms for big data, pp 82–93. Springer, 2014

  27. Shin HC, Lu L, Kim L, Seff A, Yao J, Summers RM: Interleaved text/image deep mining on a very large-scale radiology database.. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 1090–1099, 2015

  28. Shin J, Tajbakhsh N, Todd Hurst R, Kendall CB, Liang J: Automating carotid intima-media thickness video interpretation with convolutional neural networks.. In: The IEEE conference on computer vision and pattern recognition (CVPR), 2016

  29. Simonyan K, Zisserman A: Very deep convolutional networks for large-scale image recognition.. In: ICLR, 2015

  30. Stark F, Hazırbas C, Triebel R., Cremers D.: Captcha recognition with active deep learning.. In: Workshop new challenges in neural computation 2015, p 94. Citeseer, 2015

  31. Stein JH, Korcarz CE, Hurst RT, Lonn E, Kendall CB, Mohler ER, Najjar SS, Rembold CM, Post WS: Use of carotid ultrasound to identify subclinical vascular disease and evaluate cardiovascular disease risk: a consensus statement from the american society of echocardiography carotid intima-media thickness task force endorsed by the society for vascular medicine. J Am Soc Echocardiogr 21 (2): 93–111, 2008

    Article  PubMed  Google Scholar 

  32. Szegedy C, Liu W, Jia Y, Sermanet P, Reed S, Anguelov D, Erhan D, Vanhoucke V, Rabinovich A: Going deeper with convolutions.. In: The IEEE conference on computer vision and pattern recognition (CVPR), 2015

  33. Tajbakhsh N, Shin JY, Gurudu SR, Hurst RT, Kendall CB, Gotway MB, Liang J: Convolutional neural networks for medical image analysis: Full training or fine tuning? IEEE Trans Med Imaging 35 (5): 1299–1312, 2016

    Article  PubMed  Google Scholar 

  34. Tajbakhsh N, Shin JY, Hurst RT, Kendall CB, Liang J: Automatic interpretation of carotid intima–media thickness videos using convolutional neural networks.. In: Deep learning for medical image analysis, pp 105–131. Elsevier, 2017

  35. Wang D, Shang Y: A new active labeling method for deep learning.. In: 2014 International joint conference on neural networks (IJCNN), pp 112–119, 2014. https://doi.org/10.1109/IJCNN.2014.6889457

  36. Yang L, Zhang Y, Chen J, Zhang S, Chen DZ: Suggestive annotation: a deep active learning framework for biomedical image segmentation.. In: International conference on medical image computing and computer-assisted intervention, pp 399–407. Springer, 2017

  37. Zhou Z, Shin J, Zhang L, Gurudu S, Gotway M, Liang J: Fine-tuning convolutional neural networks for biomedical image analysis: actively and incrementally.. In: IEEE Conference on computer vision and pattern recognition, hawaii, pp 7340–7349, 2017

Download references

Acknowledgements

This research has been supported partially by NIH under Award Number R01HL128785 and partially by ASU and Mayo Clinic through the Discovery Translation Program. The content is solely the responsibility of the authors and does not necessarily represent the official views of NIH.

Author information

Authors and Affiliations

  1. Arizona State University, 13212 E Shea Blvd, Scottsdale, AZ, 85259, USA

    Zongwei Zhou, Jae Shin, Ruibin Feng & Jianming Liang

  2. Mayo Clinic, 13400 E Shea Blvd, Scottsdale, AZ, 85259, USA

    R. Todd Hurst & Christopher B. Kendall

Authors
  1. Zongwei Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jae Shin
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ruibin Feng
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. R. Todd Hurst
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Christopher B. Kendall
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Jianming Liang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jianming Liang.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhou, Z., Shin, J., Feng, R. et al. Integrating Active Learning and Transfer Learning for Carotid Intima-Media Thickness Video Interpretation. J Digit Imaging 32, 290–299 (2019). https://doi.org/10.1007/s10278-018-0143-2

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10278-018-0143-2

Keywords

Search

Navigation