Say and Find it: A Multimodal Wearable Interface for People with Visual Impairment
Pages 27 - 29
Abstract
Recent advances in computer vision and natural language processing using deep neural networks (DNNs) have enabled rich and intuitive multimodal interfaces. However, research on intelligent assistance systems for persons with visual impairment has not been well explored. In this work, we present an interactive object recognition and guidance interface based on multimodal interaction for blind and partially sighted people using an embedded mobile device. We demonstrate that the proposed solution using DNNs can effectively assist visually impaired people. We believe that this work will provide new and helpful insights for designing intelligent assistance systems in the future.
References
[1]
Rupert RA Bourne, Seth R Flaxman, Tasanee Braithwaite, Maria V Cicinelli, Aditi Das, Jost B Jonas, Jill Keeffe, John H Kempen, Janet Leasher, Hans Limburg, and others. 2017. Magnitude, temporal trends, and projections of the global prevalence of blindness and distance and near vision impairment: a systematic review and meta-analysis. The Lancet Global Health 5, 9 (2017), e888--e897.
[2]
Joseph Paul Cohen. 2015. A mobile app that gives a "sense of vision" to the blind with deep learning. https://github.com/ieee8023/blindtool
[3]
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition. Ieee, 248--255.
[4]
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018).
[5]
Manfred Diaz, Roger Girgis, Thomas Fevens, and Jeremy Cooperstock. 2017. To veer or not to veer: Learning from experts how to stay within the crosswalk. In Proceedings of the IEEE International Conference on Computer Vision. 1470--1479.
[6]
ETRI. 2017. AI API Data. (2017). http://aiopen.etri.re.kr/guide_recognition.php.
[7]
Jonggi Hong, Alisha Pradhan, Jon E Froehlich, and Leah Findlater. 2017. Evaluating Wrist-Based Haptic Feedback for Non-Visual Target Finding and Path Tracing on a 2D Surface. In Proceedings of the 19th International ACM SIGACCESS Conference on Computers and Accessibility. ACM, 210--219.
[8]
Anita Meier, Denys JC Matthies, Bodo Urban, and Reto Wettach. 2015. Exploring vibrotactile feedback on the body and foot for the purpose of pedestrian navigation. In Proceedings of the 2nd international Workshop on Sensor-based Activity Recognition and Interaction. ACM, 11.
[9]
NAVER. 2017. Clova Speech Synthesis. (2017). https://www.ncloud.com/product/aiService/css.
[10]
Sabrina A Paneels, Adriana Olmos, Jeffrey R Blum, and Jeremy R Cooperstock. 2013. Listen to it yourself!: evaluating usability of what's around me? for the blind. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 2107--2116.
[11]
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. 2018. MobileNetV2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 4510--4520.
[12]
Stefano Scheggi, A Talarico, and Domenico Prattichizzo. 2014. A remote guidance system for blind and visually impaired people via vibrotactile haptic feedback. In 22nd Mediterranean Conference on Control and Automation. IEEE, 20--23.
[13]
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems. 5998--6008.
[14]
Martin Weiss, Margaux Luck, Roger Girgis, Chris Pal, and Joseph Cohen. 2018. A Survey of Mobile Computing for the Visually Impaired. arXiv preprint arXiv:1811.10120 (2018).
Index Terms
- Say and Find it: A Multimodal Wearable Interface for People with Visual Impairment
Recommendations
The challenges in adopting assistive technologies in the workplace for people with visual impairments
OzCHI '18: Proceedings of the 30th Australian Conference on Computer-Human InteractionThere are many barriers to employment for people with visual impairments. Assistive technologies (ATs), such as computer screen readers and enlarging software, are commonly used to help overcome employment barriers and enable people with visual ...
Chinese FingerReader: a wearable device to explore Chinese printed text
SIGGRAPH '17: ACM SIGGRAPH 2017 PostersReading is an essential part of daily life. When reading books, drug information, textual icons on electronic devices (e.g., microwave), and information on signs and maps (e.g., location and floor level), people must be able to recognize the words to ...
Game accessibility for visually impaired people: a review
AbstractPlaying games is an important way to promote the integration, inclusion, and socialization of participants. This is especially the case of persons with disabilities, such as visually impaired people. Unfortunately, very few games are accessible to ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
October 2019
192 pages
ISBN:9781450368179
DOI:10.1145/3332167
- General Chair:
- François Guimbretière,
- Program Chairs:
- Michael Bernstein,
- Katharina Reinecke
Copyright © 2019 Owner/Author.
Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 14 October 2019
Check for updates
Author Tags
Qualifiers
- Poster
Funding Sources
- the Ministry of Education of the Republic of Korea and the National Research Foundation of Korea
- the Industrial Technology Innovation Program funded by the Ministry of Trade, Industry & Energy
Conference
UIST '19: The 32nd Annual ACM Symposium on User Interface Software and Technology
October 20 - 23, 2019
LA, New Orleans, USA
Acceptance Rates
Overall Acceptance Rate 355 of 1,733 submissions, 20%
Upcoming Conference
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 240Total Downloads
- Downloads (Last 12 months)5
- Downloads (Last 6 weeks)1
Reflects downloads up to 11 Dec 2024
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in