More Web Proxy on the site http://driver.im/

demonstration

Comprehensible reasoning and automated reporting of medical examinations based on deep learning analysis

Authors:

Steven Alexander Hicks,

Konstantin Pogorelov,

Thomas de Lange,

Mattis Jeppsson,

Kristin Ranheim Randel,

Sigrun Eskeland,

Pål Halvorsen,

Michael RieglerAuthors Info & Claims

MMSys '18: Proceedings of the 9th ACM Multimedia Systems Conference

Pages 490 - 493

https://doi.org/10.1145/3204949.3208113

Published: 12 June 2018 Publication History

Abstract

In the future, medical doctors will to an increasing degree be assisted by deep learning neural networks for disease detection during examinations of patients. In order to make qualified decisions, the black box of deep learning must be opened to increase the understanding of the reasoning behind the decision of the machine learning system. Furthermore, preparing reports after the examinations is a significant part of a doctors work-day, but if we already have a system dissecting the neural network for understanding, the same tool can be used for automatic report generation. In this demo, we describe a system that analyses medical videos from the gastrointestinal tract. Our system dissects the Tensorflow-based neural network to provide insights into the analysis and uses the resulting classification and rationale behind the classification to automatically generate an examination report for the patient's medical journal.

References

[1]

2015. The Crohnoid Blog. (2015). http://www.wrestlingtheoctopus.com/the-a-to-z-of-my-crohns/

[2]

Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, and others. 2016. Tensorflow: Large-scale machine learning on heterogeneous distributed systems. arXiv preprint arXiv:1603.04467 (2016).

[3]

Apache. 2018. Apache HTTP Server Project. (2018). https://httpd.apache.org/

[4]

Michael Bretthauer, Lars Aabakken, Evelien Dekker, Michal F Kaminski, Thomas Rösch, Rolf Hultcrantz, Stepan Suchanek, Rodrigo Jover, Ernst J Kuipers, Raf Bisschops, and others. 2016. Reporting systems in gastrointestinal endoscopy: Requirements and standards facilitating quality improvement: European Society of Gastrointestinal Endoscopy position statement. United European gastroenterology journal 4, 2 (2016), 172--176.

[5]

Thomas de Lange, Stig Larsen, and Lars Aabakken. 2005. Image documentation of endoscopic findings in ulcerative colitis: photographs or video clips? Gastrointestinal Endoscopy 61, 6 (2005), 715--720.

[6]

Facebook. 2018. React. (2018). https://reactjs.org/

[7]

FFmpeg. 2018. FFmpeg. (2018). https://www.ffmpeg.org/

[8]

Flask. 2018. Flask. (2018). http://flask.pocoo.org/

[9]

Steven Hicks. 2018. Demo Docker Repositoryma. (2018). https://hub.docker.com/r/stevenah/mmsys-demo/

[10]

Steven Hicks. 2018. Demo Github Repository. (2018). https://github.com/Stevenah/mmsys-demo

[11]

Baoyu Jing, Pengtao Xie, and Eric Xing. 2017. On the Automatic Generation of Medical Imaging Reports. arXiv preprint arXiv:1711.08195 (2017).

[12]

Keras. 2018. Keras: The Python Deep Learning library. (2018). https://keras.io/

[13]

Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. 2015. Deep learning. nature 521, 7553 (2015), 436.

[14]

Grégoire Montavon, Wojciech Samek, and Klaus-Robert Müller. 2017. Methods for interpreting and understanding deep neural networks. Digital Signal Processing (2017).

[15]

NGINX. 2018. NGINX. (2018). https://nginx.org/en/

[16]

Nvidia. 2018. Nvidia CUDA Toolkit. (2018). https://developer.nvidia.com/cuda-toolkit

[17]

Nvidia. 2018. Nvidia CuDNN. (2018). https://developer.nvidia.com/cudnn

[18]

Konstantin Pogorelov, Kristin Ranheim Randel, Carsten Griwodz, Sigrun Losada Eskeland, Thomas de Lange, Dag Johansen, Concetto Spampinato, Duc-Tien Dang-Nguyen, Mathias Lux, Peter Thelin Schmidt, Michael Riegler, and Pål Halvorsen. 2017. KVASIR: A Multi-Class Image Dataset for Computer Aided Gastrointestinal Disease Detection. In Proc. of MMSYS. 164--169.

Digital Library

[19]

Konstantin Pogorelov, Michael Riegler, Sigrun Losada Eskeland, Thomas de Lange, Dag Johansen, Carsten Griwodz, Peter Thelin Schmidt, and Pål Halvorsen. 2017. Efficient disease detection in gastrointestinal videos - global features versus neural networks. Multimedia Tools and Applications 76, 21 (01 Nov 2017), 22493--22525.

Digital Library

[20]

Michael Riegler, Mathias Lux, Carsten Griwodz, Concetto Spampinato, Thomas de Lange, Sigrun L Eskeland, Konstantin Pogorelov, Wallapak Tavanapong, Peter T Schmidt, Cathal Gurrin, and others. 2016. Multimedia and Medicine: Teammates for Better Disease Detection and Survival. In Proc. of ACM MM.

Digital Library

[21]

Michael Riegler, Konstantin Pogorelov, Pål Halvorsen, Thomas de Lange, Carsten Griwodz, Peter Thelin Schmidt, Sigrun L. Eskeland, and Dag Johansen. 2016. EIR - Efficient Computer Aided Diagnosis Framework for Gastrointestinal Endoscopies. In Proc. of CBMI.

[22]

Michael Riegler, Konstantin Pogorelov, Jonas Markussen, Mathias Lux, Håkon Kvale Stensland, Thomas de Lange, Carsten Griwodz, Pål Halvorsen, Dag Johansen, Peter T Schmidt, and Sigrun L. Eskeland. 2016. Computer Aided Disease Detection System for Gastrointestinal Examinations. In Proc. of MMSys.

Digital Library

[23]

Michael Riegler, Konstantin Pogorelov, Jonas Markussen, Mathias Lux, Håkon Kvale Stensland, Thomas de Lange, Carsten Griwodz, Pål Halvorsen, Dag Johansen, Peter T. Schmidt, and Sigrun L. Eskeland. 2016. Computer Aided Disease Detection System for Gastrointestinal Examinations. In Proc. of MMSYS. 29:1--29:4.

Digital Library

[24]

Christin Seifert, Aisha Aamir, Aparna Balagopalan, Dhruv Jain, Abhinav Sharma, Sebastian Grottel, and Stefan Gumhold. 2017. Visualizations of Deep Neural Networks in Computer Vision: A Survey. In Transparent Data Mining for Big and Small Data. Springer, 123--144.

[25]

Ramprasaath R. Selvaraju, Abhishek Das, Ramakrishna Vedantam, Michael Cogswell, Devi Parikh, and Dhruv Batra. 2016. Grad-CAM: Why did you say that? Visual Explanations from Deep Networks via Gradient-based Localization. CoRR abs/1610.02391 (2016). arXiv:1610.02391 http://arxiv.org/abs/1610.02391

[26]

Karen Simonyan and Andrew Zisserman. 2014. Very Deep Convolutional Networks for Large-Scale Image Recognition. CoRR abs/1409.1556 (2014). arXiv:1409.1556 http://arxiv.org/abs/1409.1556

[27]

Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, and Martin A. Riedmiller. 2014. Striving for Simplicity: The All Convolutional Net. CoRR abs/1412.6806 (2014). arXiv:1412.6806 http://arxiv.org/abs/1412.6806

[28]

OpenCV team. 2018. Open Source Computer Vision Library (OpenCV). (2018). https://opencv.org/

[29]

Rene Vidal, Joan Bruna, Raja Giryes, and Stefano Soatto. 2017. Mathematics of Deep Learning. arXiv preprint arXiv:1712.04741 (2017).

[30]

Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. 2015. Show, attend and tell: Neural image caption generation with visual attention. In Proc. of ML. 2048--2057.

Digital Library

[31]

Jason Yosinski, Jeff Clune, Anh Nguyen, Thomas Fuchs, and Hod Lipson. 2015. Understanding neural networks through deep visualization. arXiv preprint arXiv:1506.06579 (2015).

[32]

Zizhao Zhang, Yuanpu Xie, Fuyong Xing, Mason McGough, and Lin Yang. 2017. Mdnet: A semantically and visually interpretable medical image diagnosis network. In Proc. of IEEE CVPR. 6428--6436.

[33]

Bolei Zhou, Aditya Khosla, Àgata Lapedriza, Aude Oliva, and Antonio Torralba. 2015. Learning Deep Features for Discriminative Localization. CoRR abs/1512.04150 (2015). arXiv:1512.04150 http://arxiv.org/abs/1512.04150

Cited By

Pang TLi PZhao L(2023)A survey on automatic generation of medical imaging reports based on deep learningBioMedical Engineering OnLine10.1186/s12938-023-01113-y22:1Online publication date: 18-May-2023
https://doi.org/10.1186/s12938-023-01113-y
Garcia E(2023)Integrating artificial intelligence and natural language processing for computer-assisted reporting and report understanding in nuclear cardiologyJournal of Nuclear Cardiology10.1007/s12350-022-02996-530:3(1180-1190)Online publication date: Jun-2023
https://doi.org/10.1007/s12350-022-02996-5
Messina PPino PParra DSoto ABesa CUribe SAndía MTejos CPrieto CCapurro D(2022)A Survey on Deep Learning and Explainability for Automatic Report Generation from Medical ImagesACM Computing Surveys10.1145/352274754:10s(1-40)Online publication date: 23-Mar-2022
https://dl.acm.org/doi/10.1145/3522747
Show More Cited By

Index Terms

Comprehensible reasoning and automated reporting of medical examinations based on deep learning analysis
1. Applied computing
  1. Life and medical sciences
    1. Health informatics
2. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Computer vision tasks
        Video summarization
  2. Machine learning

Recommendations

Mimir: an automatic reporting and reasoning system for deep learning based analysis in the medical domain
MMSys '18: Proceedings of the 9th ACM Multimedia Systems Conference

Automatic detection of diseases is a growing field of interest, and machine learning in form of deep learning neural networks are frequently explored as a potential tool for the medical video analysis. To both improve the "black box"-understanding and ...
Medical thermograms’ classification using deep transfer learning models and methods
Abstract
Infrared thermal imaging and deep learning provide intelligent monitoring systems that detect diseases in early phases. However, deep learning models require thousands of labeled images to be effectively trained from scratch. Since such a dataset ...
Medical image analysis based on deep learning approach
Abstract
Medical imaging plays a significant role in different clinical applications such as medical procedures used for early detection, monitoring, diagnosis, and treatment evaluation of various medical conditions. Basicsof the principles and ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

MMSys '18: Proceedings of the 9th ACM Multimedia Systems Conference

June 2018

604 pages

ISBN:9781450351928

DOI:10.1145/3204949

General Chair:
Pablo Cesar
Centrum Wiskunde & Informatica, The Netherlands
,
Program Chairs:
Michael Zink
University of Massachusetts Amherst
,
Niall Murray
Athlone Institute of Technology, Ireland

Copyright © 2018 Owner/Author.

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

Sponsors

SIGMM: ACM Special Interest Group on Multimedia

In-Cooperation

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 12 June 2018

Check for updates

Author Tags

Qualifiers

Demonstration

Conference

MMSys '18

Sponsor:

SIGMM

MMSys '18: 9th ACM Multimedia Systems Conference

June 12 - 15, 2018

Amsterdam, Netherlands

Acceptance Rates

Overall Acceptance Rate 176 of 530 submissions, 33%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

7
Total Citations
View Citations
204
Total Downloads

Downloads (Last 12 months)4
Downloads (Last 6 weeks)0

Reflects downloads up to 31 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Pang TLi PZhao L(2023)A survey on automatic generation of medical imaging reports based on deep learningBioMedical Engineering OnLine10.1186/s12938-023-01113-y22:1Online publication date: 18-May-2023
https://doi.org/10.1186/s12938-023-01113-y
Garcia E(2023)Integrating artificial intelligence and natural language processing for computer-assisted reporting and report understanding in nuclear cardiologyJournal of Nuclear Cardiology10.1007/s12350-022-02996-530:3(1180-1190)Online publication date: Jun-2023
https://doi.org/10.1007/s12350-022-02996-5
Messina PPino PParra DSoto ABesa CUribe SAndía MTejos CPrieto CCapurro D(2022)A Survey on Deep Learning and Explainability for Automatic Report Generation from Medical ImagesACM Computing Surveys10.1145/352274754:10s(1-40)Online publication date: 23-Mar-2022
https://dl.acm.org/doi/10.1145/3522747
Hakkoum HAbnane IIdri A(2022)Interpretability in the medical fieldApplied Soft Computing10.1016/j.asoc.2021.108391117:COnline publication date: 12-May-2022
https://dl.acm.org/doi/10.1016/j.asoc.2021.108391
Hakkoum HIdri AAbnane I(2021)Assessing and comparing interpretability techniques for artificial neural networks breast cancer classicationComputer Methods in Biomechanics and Biomedical Engineering: Imaging & Visualization10.1080/21681163.2021.1901784(1-13)Online publication date: 27-Mar-2021
https://doi.org/10.1080/21681163.2021.1901784
Borgli HThambawita VSmedsrud PHicks SJha DEskeland SRandel KPogorelov KLux MNguyen DJohansen DGriwodz CStensland HGarcia-Ceja ESchmidt PHammer HRiegler MHalvorsen Pde Lange T(2020)HyperKvasir, a comprehensive multi-class image and video dataset for gastrointestinal endoscopyScientific Data10.1038/s41597-020-00622-y7:1Online publication date: 28-Aug-2020
https://doi.org/10.1038/s41597-020-00622-y
Schroeder DStyp-Rekowski KSchmidt FAcker AKao O(2019)Graph-based Feature Selection Filter Utilizing Maximal Cliques2019 Sixth International Conference on Social Networks Analysis, Management and Security (SNAMS)10.1109/SNAMS.2019.8931841(297-302)Online publication date: Oct-2019
https://doi.org/10.1109/SNAMS.2019.8931841

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents