poster

Deep Learning–Based Perceptual Stimulus Encoder for Bionic Vision

Authors:

Lucas Relic,

Bowen Zhang,

Yi-Lin Tuan,

Michael BeyelerAuthors Info & Claims

AHs '22: Proceedings of the Augmented Humans International Conference 2022

Pages 323 - 325

https://doi.org/10.1145/3519391.3524034

Published: 18 April 2022 Publication History

Get Access

Abstract

Retinal implants have the potential to treat incurable blindness, yet the quality of the artificial vision they produce is still rudimentary. An outstanding challenge is identifying electrode activation patterns that lead to intelligible visual percepts (phosphenes). Here we propose a perceptual stimulus encoder (PSE) based on convolutional neural networks (CNNs) that is trained in an end-to-end fashion to predict the electrode activation patterns required to produce a desired visual percept. We demonstrate the effectiveness of the encoder on MNIST using a psychophysically validated phosphene model tailored to individual retinal implant users. The present work constitutes an essential first step towards improving the quality of the artificial vision provided by retinal implants.

References

[1]

Lauren N. Ayton, Nick Barnes, Gislin Dagnelie, Takashi Fujikado, Georges Goetz, Ralf Hornig, Bryan W. Jones, Mahiul M. K. Muqit, Daniel L. Rathbun, Katarina Stingl, James D. Weiland, and Matthew A. Petoe. 2020. An update on retinal prostheses. Clinical Neurophysiology 131, 6 (June 2020), 1383–1398. https://doi.org/10.1016/j.clinph.2019.11.029

Crossref

Google Scholar

[2]

Michael Beyeler, Devyani Nanduri, James D. Weiland, Ariel Rokem, Geoffrey M. Boynton, and Ione Fine. 2019. A model of ganglion axon pathways accounts for percepts elicited by retinal implants. Scientific Reports 9, 1 (2019). https://doi.org/10.1038/s41598-019-45416-4

Crossref

Google Scholar

[3]

Wm H. Dobelle. 2000. Artificial Vision for the Blind by Connecting a Television Camera to the Visual Cortex. ASAIO Journal 46, 1 (Feb. 2000), 3–9.

Crossref

Google Scholar

[4]

Cordelia Erickson-Davis and Helma Korzybska. 2021. What do blind people “see” with retinal prostheses? Observations and qualitative reports of epiretinal implant users. PLOS ONE 16, 2 (Feb. 2021), e0229189. https://doi.org/10.1371/journal.pone.0229189 Publisher: Public Library of Science.

Crossref

Google Scholar

[5]

Jacob Granley and Michael Beyeler. 2021. A Computational Model of Phosphene Appearance for Epiretinal Prostheses. 4477–4481. https://doi.org/10.1109/EMBC46164.2021.9629663 ISSN: 2694-0604.

Crossref

Google Scholar

[6]

Nicole Han, Sudhanshu Srivastava, Aiwen Xu, Devi Klein, and Michael Beyeler. 2021. Deep Learning–Based Scene Simplification for Bionic Vision. In Augmented Humans Conference 2021(AHs’21). Association for Computing Machinery, New York, NY, USA, 45–54. https://doi.org/10.1145/3458709.3458982

Digital Library

Google Scholar

[7]

Lachlan Horne, Jose Alvarez, Chris McCarthy, Mathieu Salzmann, and Nick Barnes. 2016. Semantic labeling for prosthetic vision. Computer Vision and Image Understanding 149 (2016), 113–125.

Digital Library

Google Scholar

[8]

Y. H. Luo and L. da Cruz. 2016. The Argus((R)) II Retinal Prosthesis System. Prog Retin Eye Res 50 (Jan. 2016), 89–107. https://doi.org/10.1016/j.preteyeres.2015.09.003

Crossref

Google Scholar

[9]

Michael Beyeler, Geoffrey M. Boynton, Ione Fine, and Ariel Rokem. 2017. pulse2percept: A Python-based simulation framework for bionic vision. In Proceedings of the 16th Python in Science Conference, Katy Huff, David Lippa, Dillon Niederhut, and M Pacer (Eds.). 81 – 88. https://doi.org/10.25080/shinma-7f4c6e7-00c

Crossref

Google Scholar

[10]

Jaap de Ruyter van Steveninck, Umut Guclu, Richard JA van Wezel, and Marcel AJ van Gerven. 2020. End-to-end optimization of prosthetic vision. bioRxiv (2020).

Google Scholar

[11]

Ying Zhao, Qi Li, Donghui Wang, and Aiping Yu. 2018. Image Processing Strategies Based on Deep Neural Network for Simulated Prosthetic Vision. In 2018 11th International Symposium on Computational Intelligence and Design (ISCID), Vol. 01. 200–203. https://doi.org/10.1109/ISCID.2018.00052

Crossref

Google Scholar

Cited By

View all

Mahadiuzzaman AHoque MAlam SChawdhury ZHasan MRashid A(2024)Visual Neuroprostheses for Impaired Human Nervous System: State‐of‐the‐Art and Future OutlookInternational Journal of Cell Biology10.1155/ijcb/26517632024:1Online publication date: 2-Dec-2024
https://doi.org/10.1155/ijcb/2651763
Castro DGrayden DMeffin HSpencer M(2024)Neural activity shaping in visual prostheses with deep learningJournal of Neural Engineering10.1088/1741-2552/ad618621:4(046025)Online publication date: 25-Jul-2024
https://doi.org/10.1088/1741-2552/ad6186
Hou YNanduri DGranley JWeiland JBeyeler M(2024)Axonal stimulation affects the linear summation of single-point perception in three Argus II usersJournal of Neural Engineering10.1088/1741-2552/ad31c421:2(026031)Online publication date: 8-Apr-2024
https://doi.org/10.1088/1741-2552/ad31c4
Show More Cited By

Index Terms

Deep Learning–Based Perceptual Stimulus Encoder for Bionic Vision
1. Applied computing
  1. Life and medical sciences
2. Computing methodologies
  1. Artificial intelligence
  2. Machine learning
    1. Machine learning approaches
      1. Neural networks

Index terms have been assigned to the content through auto-classification.

Recommendations

A Deep Learning-based Classification Algorithm for the Origin of Premature Ventricular Contractions
ICBBB '24: Proceedings of the 2024 14th International Conference on Bioscience, Biochemistry and Bioinformatics

Catheter ablation is an effective and safe method for treating outflow tract ventricular arrhythmias. Preoperative preliminary localization of abnormal excitation origin is of great value in designing ablation strategies and improving the efficiency of ...
Comprehensive electrocardiographic diagnosis based on deep learning
Highlights
- Deep learning techniques for classification of MI, CAD and CHF are discussed.
- ...
Abstract
Cardiovascular disease (CVD) is the leading cause of death worldwide, and coronary artery disease (CAD) is a major contributor. Early-stage CAD can progress if undiagnosed and left untreated, leading to myocardial infarction (MI) that ...
Perceptual learning of simple stimuli modifies stimulus representations in posterior inferior temporal cortex

Practicing simple visual detection and discrimination tasks improves performance, a signature of adult brain plasticity. The neural mechanisms that underlie these changes in performance are still unclear. Previously, we reported that practice in ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

AHs '22: Proceedings of the Augmented Humans International Conference 2022

March 2022

350 pages

ISBN:9781450396325

DOI:10.1145/3519391

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 18 April 2022

Check for updates

Author Tags

Qualifiers

Poster
Research
Refereed limited

Conference

AHs 2022

AHs 2022: Augmented Humans 2022

March 13 - 15, 2022

Kashiwa, Chiba, Japan

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

9
Total Citations
View Citations
167
Total Downloads

Downloads (Last 12 months)45
Downloads (Last 6 weeks)7

Reflects downloads up to 15 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

View all

Mahadiuzzaman AHoque MAlam SChawdhury ZHasan MRashid A(2024)Visual Neuroprostheses for Impaired Human Nervous System: State‐of‐the‐Art and Future OutlookInternational Journal of Cell Biology10.1155/ijcb/26517632024:1Online publication date: 2-Dec-2024
https://doi.org/10.1155/ijcb/2651763
Castro DGrayden DMeffin HSpencer M(2024)Neural activity shaping in visual prostheses with deep learningJournal of Neural Engineering10.1088/1741-2552/ad618621:4(046025)Online publication date: 25-Jul-2024
https://doi.org/10.1088/1741-2552/ad6186
Hou YNanduri DGranley JWeiland JBeyeler M(2024)Axonal stimulation affects the linear summation of single-point perception in three Argus II usersJournal of Neural Engineering10.1088/1741-2552/ad31c421:2(026031)Online publication date: 8-Apr-2024
https://doi.org/10.1088/1741-2552/ad31c4
Donati EValle G(2024)Neuromorphic hardware for somatosensory neuroprosthesesNature Communications10.1038/s41467-024-44723-315:1Online publication date: 16-Jan-2024
https://doi.org/10.1038/s41467-024-44723-3
Wu KMina MSahyoun JKalevar ATran S(2023)Retinal Prostheses: Engineering and Clinical Perspectives for Vision RestorationSensors10.3390/s2313578223:13(5782)Online publication date: 21-Jun-2023
https://doi.org/10.3390/s23135782
Wu YKaretić IStegmaier JWalter PMerhof D(2023)A Deep Learning-based in silico Framework for Optimization on Retinal Prosthetic Stimulation2023 45th Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC)10.1109/EMBC40787.2023.10340288(1-4)Online publication date: 24-Jul-2023
https://doi.org/10.1109/EMBC40787.2023.10340288
Wang CFang CZou YYang JSawan M(2023)Artificial intelligence techniques for retinal prostheses: a comprehensive review and future directionJournal of Neural Engineering10.1088/1741-2552/acb29520:1(011003)Online publication date: 15-Feb-2023
https://doi.org/10.1088/1741-2552/acb295
Granley JRelic LBeyeler MKoyejo SMohamed SAgarwal ABelgrave DCho KOh A(2022)Hybrid neural autoencoders for stimulus encoding in visual and other sensory neuroprosthesesProceedings of the 36th International Conference on Neural Information Processing Systems10.5555/3600270.3601917(22671-22685)Online publication date: 28-Nov-2022
https://dl.acm.org/doi/10.5555/3600270.3601917
Beyeler MSanchez-Garcia M(2022)Towards a Smart Bionic Eye: AI-powered artificial vision for the treatment of incurable blindnessJournal of Neural Engineering10.1088/1741-2552/aca69d19:6(063001)Online publication date: 7-Dec-2022
https://doi.org/10.1088/1741-2552/aca69d

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

HTML Format

View this article in HTML Format.

HTML Format

Abstract

References

Cited By

Index Terms

Recommendations

A Deep Learning-based Classification Algorithm for the Origin of Premature Ventricular Contractions

Comprehensive electrocardiographic diagnosis based on deep learning

Perceptual learning of simple stimuli modifies stimulus representations in posterior inferior temporal cortex

Comments

Information

Published In

Publisher

Publication History

Check for updates

Author Tags

Qualifiers

Conference

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Cited By

Login options

Full Access

View options

PDF

eReader

HTML Format

Figures

Other

Share

Share this Publication link

Share on social media

Affiliations