More Web Proxy on the site http://driver.im/

research-article

Open access

Increasing Web3D Accessibility with Audio Captioning

Authors:

Nicholas Polys,

Sheeban Mohd WasiAuthors Info & Claims

Web3D '23: Proceedings of the 28th International ACM Conference on 3D Web Technology

Article No.: 6, Pages 1 - 10

https://doi.org/10.1145/3611314.3615902

Published: 09 October 2023 Publication History

All formats PDF

Abstract

Situational awareness plays a critical role in daily life, enabling individuals to comprehend their surroundings, make informed decisions, and navigate safely. However, individuals with low vision or visual impairments face difficulties in perceiving their real or virtual environment. In order to address this challenge, we propose a 3D computer vision-based accessibility solution, empowered by object-detection and text-to-speech technology. Our application describes the visual content of a Web3D scene from the user’s perspective through auditory channels, thereby enhancing situational awareness for individuals with visual impairments in virtual and physical environments. We conducted a user study of 44 participants to compare a set of algorithms for specific tasks, such as Search or Summarize, and assessed the effectiveness of our captioning algorithms based on user ratings of naturalness, correctness, and satisfaction. Our study results indicate positive subjective results in accessibility for both normal and visually-impaired subjects and also distinguish significant effects between the task and the captioning algorithm.

References

[1]

Dragan Ahmetovic, Federico Avanzini, Adriano Baratè, Cristian Bernareggi, Gabriele Galimberti, Luca A Ludovico, Sergio Mascetti, and Giorgio Presti. 2019. Sonification of rotation instructions to support navigation of people with visual impairment. In 2019 IEEE International Conference on Pervasive Computing and Communications (PerCom. IEEE, 1–10.

[2]

Jesse Anderson. 2022. How Can a Blind Person Use Virtual Reality. Retrieved February 21, 2023 from https://equalentry.com/how-can-a-blind-person-use-virtual-reality/

[3]

Nida Aziz, Tony Stockman, and Rebecca Stewart. 2019. An investigation into customisable automatically generated auditory route overviews for pre-navigation. Georgia Institute of Technology.

[4]

Johannes Behr, Peter Eschler, Yvonne Jung, and Michael Zöllner. 2009. X3DOM: A DOM-Based HTML5/X3D Integration Model. Association for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/1559764.1559784

Digital Library

[5]

Johannes Behr, Yvonne Jung, Timm Drevensek, and Andreas Aderhold. 2011. Dynamic and Interactive Aspects of X3DOM. Association for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/2010425.2010440

Digital Library

[6]

J. Behr, Y. Jung, J. Keil, T. Drevensek, M. Zoellner, P. Eschler, and D. Fellner. 2010. A Scalable Architecture for the HTML5/X3D Integration Model X3DOM. Association for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/1836049.1836077

Digital Library

[7]

Pramod Chundury, Biswaksen Patnaik, Yasmin Reyazuddin, Christine Tang, Jonathan Lazar, and Niklas Elmqvist. 2021. Towards understanding sensory substitution for accessible visualization: An interview study. IEEE transactions on visualization and computer graphics 28, 1 (2021), 1084–1094.

[8]

Angela Constantinescu, Karin Müller, Monica Haurilet, Vanessa Petrausch, and Rainer Stiefelhagen. 2020. Bring the environment to life: A sonification module for people with visual impairments to improve situation awareness. In Proceedings of the 2020 International Conference on Multimodal Interaction. 50–59.

Digital Library

[9]

Tilman Dingler, Jeffrey Lindsay, and Bruce N Walker. 2008. Learnabiltiy of sound cues for environmental features: Auditory icons, earcons, spearcons, and speech. International Community for Auditory Display.

[10]

Niklas Elmqvist. 2023. Visualization for the Blind. Retrieved February 18, 2023 from https://interactions.acm.org/archive/view/january-february-2023/visualization-for-the-blind

Digital Library

[11]

Mark Everingham, S. M. Eslami, Luc Gool, Christopher K. Williams, John Winn, and Andrew Zisserman. 2015. The Pascal Visual Object Classes Challenge: A Retrospective. 111, 1 (2015). https://doi.org/10.1007/s11263-014-0733-5

Digital Library

[12]

Mexhid Ferati, Steve Mannheimer, and Davide Bolchini. 2011. Usability evaluation of acoustic interfaces for the blind. In Proceedings of the 29th ACM international conference on Design of communication. 9–16.

Digital Library

[13]

Michele Geronazzo, Alberto Bedin, Luca Brayda, Claudio Campus, and Federico Avanzini. 2016. Interactive spatial sonification for non-visual exploration of virtual maps. International Journal of Human-Computer Studies 85 (2016), 4–15.

Digital Library

[14]

Sneha Gupta, Suchismita Chakraborti, R Yogitha, and G Mathivanan. 2022. Object Detection with Audio Comments using YOLO v3. In 2022 International Conference on Applied Artificial Intelligence and Computing (ICAAIC). IEEE, 903–909.

[15]

Andreas Hub, Joachim Diepstraten, and Thomas Ertl. 2003. Design and development of an indoor navigation and object identification system for the blind. ACM Sigaccess Accessibility and Computing77-78 (2003), 147–152.

[16]

Brian FG Katz, Slim Kammoun, Gaëtan Parseihian, Olivier Gutierrez, Adrien Brilhault, Malika Auvray, Philippe Truillet, Michel Denis, Simon Thorpe, and Christophe Jouffrais. 2012. NAVIG: Augmented reality guidance system for the visually impaired: Combining object localization, GNSS, and spatial audio. Virtual Reality 16 (2012), 253–269.

Digital Library

[17]

Nitin Kumar and Anuj Jain. 2022. A Deep Learning Based Model to Assist Blind People in Their Navigation. Journal of Information Technology Education: Innovations in Practice 21 (2022), 095–114.

[18]

Min Lin, Qiang Chen, and Shuicheng Yan. 2013. Network In Network. https://doi.org/10.48550/ARXIV.1312.4400

[19]

Mansi Mahendru and Sanjay Kumar Dubey. 2021. Real time object detection with audio feedback using Yolo vs. Yolo_v3. In 2021 11th International Conference on Cloud Computing, Data Science & Engineering (Confluence). IEEE, 734–740.

[20]

Sergio Mascetti, Lorenzo Picinali, Andrea Gerino, Dragan Ahmetovic, and Cristian Bernareggi. 2016. Sonification of guidance data during road crossing for people with visual impairments or blindness. International Journal of Human-Computer Studies 85 (2016), 16–26.

Digital Library

[21]

Nicholas Polys, Jessica Hotter, Madison Lanier, Laura Purcell, Jordan Wolf, W. Cully Hession, Peter Sforza, and James D. Ivory. 2017. Finding Frogs: Using Game-Based Learning to Increase Environmental Awareness. In Proceedings of the 22nd International Conference on 3D Web Technology (Brisbane, Queensland, Australia) (Web3D ’17). Association for Computing Machinery, New York, NY, USA, Article 10, 8 pages. https://doi.org/10.1145/3055624.3075955

Digital Library

[22]

Nicholas Polys, Cecile Newcomb, Todd Schenk, Thomas Skuzinski, and Donna Dunay. 2018. The Value of 3D Models and Immersive Technology in Planning Urban Density. In Proceedings of the 23rd International ACM Conference on 3D Web Technology (Poznań, Poland) (Web3D ’18). Association for Computing Machinery, New York, NY, USA, Article 13, 4 pages. https://doi.org/10.1145/3208806.3208824

Digital Library

[23]

Nicholas F Polys, Kathleen Meaney, John Munsell, and Benjamin J Addlestone. 2021. X3D Field Trips for Remote Learning. In The 26th International Conference on 3D Web Technology. 1–7.

Digital Library

[24]

Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. 2015. You Only Look Once: Unified, Real-Time Object Detection. https://doi.org/10.48550/ARXIV.1506.02640

[25]

Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. 2014. ImageNet Large Scale Visual Recognition Challenge. https://doi.org/10.48550/ARXIV.1409.0575

[26]

Manaswi Saha, Alexander J Fiannaca, Melanie Kneisel, Edward Cutrell, and Meredith Ringel Morris. 2019. Closing the gap: Designing for the last-few-meters wayfinding problem for people with visual impairments. In Proceedings of the 21st International ACM SIGACCESS Conference on Computers and Accessibility. 222–235.

Digital Library

[27]

Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. 2014. Going Deeper with Convolutions. https://doi.org/10.48550/ARXIV.1409.4842

[28]

Christoph Urbanietz, Gerald Enzner, Alexander Orth, Patrick Kwiatkowski, and Nils Pohl. 2019. A radar-based navigation assistance device with binaural sound interface for vision-impaired people. Georgia Institute of Technology.

[29]

Karen Vines, Chris Hughes, Laura Alexander, Carol Calvert, Chetz Colwell, Hilary Holmes, Claire Kotecki, Kaela Parks, and Victoria Pearson. 2019. Sonification of numerical data for education. Open Learning: The Journal of Open, Distance and e-Learning 34, 1 (2019), 19–39.

[30]

Bruce N Walker and Lisa M Mauney. 2010. Universal design of auditory graphs: A comparison of sonification mappings for visually impaired and sighted listeners. ACM Transactions on Accessible Computing (TACCESS) 2, 3 (2010), 1–16.

Digital Library

[31]

WHO. 2022. Vision Impairment and blindness. Retrieved February 18, 2023 from https://www.who.int/news-room/fact-sheets/detail/blindness-and-visual-impairment

[32]

Limin Zeng, Markus Simros, and Gerhard Weber. 2017. Camera-based mobile electronic travel aids support for cognitive mapping of unknown spaces. In Proceedings of the 19th international conference on human-computer interaction with mobile devices and services. 1–10.

Digital Library

[33]

Minye Zhan, Rainer Goebel, and Beatrice de Gelder. 2018. Ventral and Dorsal Pathways Relate Differently to Visual Awareness of Body Postures under Continuous Flash Suppression. eNeuro 5, 1 (2018). https://doi.org/10.1523/ENEURO.0285-17.2017 arXiv:https://www.eneuro.org/content/5/1/ENEURO.0285-17.2017.full.pdf

Cited By

Liu TFazli PJeong H(2024)Artificial Intelligence in Virtual Reality for Blind and Low Vision Individuals: Literature ReviewProceedings of the Human Factors and Ergonomics Society Annual Meeting10.1177/1071181324126683268:1(1333-1338)Online publication date: 9-Sep-2024
https://doi.org/10.1177/10711813241266832

Index Terms

Index terms have been assigned to the content through auto-classification.

Recommendations

User experience and usage scenarios of audio-tactile interaction with virtual objects in a physical environment
DPPI '13: Proceedings of the 6th International Conference on Designing Pleasurable Products and Interfaces

Augmented reality (AR) systems enable new user experiences while the user is interacting with virtual objects in the physical space. The virtual objects have mostly been presented visually, overlaid on the physical world. In this paper, we present an ...
Visually impaired people and the emerging connected TV: a comparative study of TV and Web applications’ accessibility
Abstract
With the TV signal digitization and the current market growth of connected TVs, the authors envision the appearance of accessibility barriers to visually impaired persons. The paper addresses the following hypothesis: (1) visually impaired users ...
Improved and accessible e-book reader application for visually impaired people
SA '17: SIGGRAPH Asia 2017 Posters

This¹paper presents a study of an accessible e-book reader application for visually impaired people. We interviewed 27 visually impaired people to understand their usage patterns of e-books and user requirements in terms of functions and interface of an ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

Web3D '23: Proceedings of the 28th International ACM Conference on 3D Web Technology

October 2023

244 pages

ISBN:9798400703249

DOI:10.1145/3611314

Editors:
Aitor Moreno
Vicomtech
,
Alberto Jaspe-Villanueva
KAUST
,
Imanol Muñoz-Pandiella
UPC
,
Jorge Posada
Vicomtech

Copyright © 2023 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGGRAPH: ACM Special Interest Group on Computer Graphics and Interactive Techniques

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 09 October 2023

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

Web3D '23

Sponsor:

SIGGRAPH

Web3D '23: The 28th International ACM Conference on 3D Web Technology

October 9 - 11, 2023

San Sebastian, Spain

Acceptance Rates

Overall Acceptance Rate 27 of 71 submissions, 38%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1
Total Citations
View Citations
288
Total Downloads

Downloads (Last 12 months)230
Downloads (Last 6 weeks)31

Reflects downloads up to 04 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Liu TFazli PJeong H(2024)Artificial Intelligence in Virtual Reality for Blind and Low Vision Individuals: Literature ReviewProceedings of the Human Factors and Ergonomics Society Annual Meeting10.1177/1071181324126683268:1(1333-1338)Online publication date: 9-Sep-2024
https://doi.org/10.1177/10711813241266832

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents