Speech-based Mark for Data Sonification
Article No.: 98, Pages 1 - 5
Abstract
Sonification serves as a powerful tool for data accessibility, especially for people with vision loss. Among various modalities, speech is a familiar means of communication similar to the role of text in visualization. However, speech-based sonification is underexplored. We introduce SpeechTone, a novel speech-based mark for data sonification and extension to the existing Erie declarative grammar for sonification. It encodes data into speech attributes such as pitch, speed, voice and speech content. We demonstrate the efficacy of SpeechTone through three examples.
Supplemental Material
ZIP File
"The supplementary material include: 1. A directory named `/Demo` that contains three speech sonification samples. These samples correspond to the three demonstrations described in the paper: 1.1. ""Demo_1_Num-Car-Models-Per-Origin_Pitch.mp3"": Refer to Section 4.1 in the paper. 1.2. ""Demo_2_Num-Car-Models-Per-Year_Rate.mp3"": Refer to Section 4.2 in the paper. 1.3. ""Demo_3_Fuel-Efficiency_Origin_Voice.mp3"": Refer to Section 4.3 in the paper. 2. A ZIP file named ""SpeechTone-Erie-Demo.zip"". This is a sample project that creates the three speech demonstrations. You can find details on how to run this project in the ""README.md"" file included in the file. 3. A ZIP file named ""SpeechTone-Erie-Source.zip"". This file contains the source code for the extended Erie library, which includes SpeechTone. Instructions on how to build the extended Erie library with SpeechTone are provided in the README.md file included in the file."
- Download
- 4.78 MB
References
[1]
Md Zubair Ibne Alam, Shehnaz Islam, and Enamul Hoque. 2023. SeeChart: Enabling Accessible Visualizations Through Interactive Natural Language Interface For People with Visual Impairments. (Feb 2023). https://doi.org/10.1145/3581641.3584099 arXiv:2302.07742 [cs].
[2]
Jérémy Albouys-Perrois, Jérémy Laviole, Carine Briant, and Anke M. Brock. 2018. Towards a Multisensory Augmented Reality Map for Blind and Low Vision People: a Participatory Design Approach. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. ACM, Montreal QC Canada, 1–14. https://doi.org/10.1145/3173574.3174203
[3]
FLOE Project at OCAD University. [n. d.]. Inclusive Learning Design Handbook: What is Sonification?https://handbook.floeproject.org/approaches/sonification/
[4]
Sina Bahram. 2013. Multimodal eyes-free exploration of maps: TIKISI for maps. ACM SIGACCESS Accessibility and Computing106 (Jun 2013), 3–11. https://doi.org/10.1145/2505401.2505402
[5]
Richard Brath. 2020. Visualizing with Text. CRC Press. https://books.google.ca/books?id=dAoHEAAAQBAJ
[6]
Andy Brown, Steve Pettifer, and Robert Stevens. 2003. Evaluation of a non-visual molecule browser. In Proceedings of the 6th international ACM SIGACCESS conference on Computers and accessibility(Assets ’04). Association for Computing Machinery, New York, NY, USA, 40–47. https://doi.org/10.1145/1028630.1028639
[7]
Sanjana Shivani Chintalapati, Jonathan Bragg, and Lucy Lu Wang. 2022. A Dataset of Alt Texts from HCI Publications: Analyses and Uses Towards Producing More Descriptive Alt Texts of Data Visualizations in Scientific Papers. In Proceedings of the 24th International ACM SIGACCESS Conference on Computers and Accessibility(ASSETS ’22). Association for Computing Machinery, New York, NY, USA, 1–12. https://doi.org/10.1145/3517428.3544796
[8]
Benjamin K. Davison and Bruce N. Walker. 2007. Sonification Sandbox Reconstruction: Software Standard for Auditory Graphs. In Proceedings of the 13th International Conference on Auditory Display. Montréal, Canada, 4. http://sonify.psych.gatech.edu/publications/pdfs/2007ICAD-DavisonWalker.pdf
[9]
Kajetan Enge, Alexander Rind, Michael Iber, Robert Höldrich, and Wolfgang Aigner. 2022. Towards Multimodal Exploratory Data Analysis: SoniScope as a Prototypical Implementation. (2022), 5 pages. https://doi.org/10.2312/EVS.20221095
[10]
Kajetan Enge, Alexander Rind, Michael Iber, Robert Höldrich, and Wolfgang Aigner. 2023. Towards a unified terminology for sonification and visualization. Personal and Ubiquitous Computing (Aug. 2023). https://doi.org/10.1007/s00779-023-01720-5
[11]
Danyang Fan, Alexa Fay Siu, Wing-Sum Adrienne Law, Raymond Ruihong Zhen, Sile O’Modhrain, and Sean Follmer. 2022. Slide-Tone and Tilt-Tone: 1-DOF Haptic Techniques for Conveying Shape Characteristics of Graphs to Blind Users. In CHI Conference on Human Factors in Computing Systems. ACM, New Orleans LA USA, 1–19. https://doi.org/10.1145/3491102.3517790 Citation Key: fan-SlideToneTiltTone1DOF-2022b.
[12]
Ciaran Frame. [n. d.]. BOTSOT: A Generative Sonification Toolbox. Available at https://github.com/ciaranframe/botsot.
[13]
Leona M Holloway, Cagatay Goncu, Alon Ilsar, Matthew Butler, and Kim Marriott. 2022. Infosonics: Accessible Infographics for People who are Blind using Sonification and Voice. In CHI Conference on Human Factors in Computing Systems. ACM, New Orleans LA USA, 1–13. https://doi.org/10.1145/3491102.3517465 Citation Key: holloway-InfosonicsAccessibleInfographics-2022.
[14]
Mirko Horstmann*, Martin Lorenz, A Watkowski, G Ioannidis, Otthein Herzog, Alasdair King, D Gareth Evans, Cornelius Hagen, Christoph Schlieder, A-M Burn, 2004. Automated interpretation and accessible presentation of technical diagrams for blind people. New Review of Hypermedia and Multimedia 10, 2 (2004), 141–163. https://doi.org/10.1080/13614560512331326017
[15]
Nikolaos Kaklanis, Konstantinos Votis, and Dimitrios Tzovaras. 2013. A mobile interactive maps application for a visually impaired audience. In Proceedings of the 10th International Cross-Disciplinary Conference on Web Accessibility(W4A ’13). Association for Computing Machinery, New York, NY, USA, 1–2. https://doi.org/10.1145/2461121.2461152
[16]
Shaun K. Kane, Meredith Ringel Morris, Annuska Z. Perkins, Daniel Wigdor, Richard E. Ladner, and Jacob O. Wobbrock. 2011. Access overlays: improving non-visual access to large touch screens for blind users. In Proceedings of the 24th annual ACM symposium on User interface software and technology(UIST ’11). Association for Computing Machinery, New York, NY, USA, 273–282. https://doi.org/10.1145/2047196.2047232
[17]
Hyeok Kim, Yea-Seul Kim, and Jessica Hullman. 2024. Erie: A Declarative Grammar for Data Sonification. In Proceedings of the CHI Conference on Human Factors in Computing Systems. ACM, Honolulu HI USA, 1–19. https://doi.org/10.1145/3613904.3642442
[18]
Alasdair King, Paul Blenkhorn, David Crombie, Sijo Dijkstra, Gareth Evans, and John Wood. 2004. Presenting UML Software Engineering Diagrams to Blind People. In Computers Helping People with Special Needs(Lecture Notes in Computer Science), Klaus Miesenberger, Joachim Klaus, Wolfgang L. Zagler, and Dominique Burger (Eds.). Springer, Berlin, Heidelberg, 522–529. https://doi.org/10.1007/978-3-540-27817-7_76
[19]
Johannes Lang and Miguel A. Nacenta. 2022. Perception of letter glyph parameters for InfoTypography. ACM Transactions on Graphics 41, 4 (July 2022), 1–21. https://doi.org/10.1145/3528223.3530111
[20]
Hans Lindetorp and Kjetil Falkenberg. 2021. Sonification for Everyone Everywhere - Evaluating the WebAudioXML Sonification Toolkit for Browsers. In Proceedings of the 26th International Conference on Auditory Display (ICAD 2021). International Community for Auditory Display, Virtual Conference, 15–21. https://doi.org/10.21785/icad2021.009
[21]
Miguel Nacenta, Uta Hinrichs, and Sheelagh Carpendale. 2012. FatFonts: combining the symbolic and visual aspects of numbers. In Proceedings of the International Working Conference on Advanced Visual Interfaces(AVI ’12). Association for Computing Machinery, New York, NY, USA, 407–414. https://doi.org/10.1145/2254556.2254636
[22]
John G. Neuhoff. 2019. Is Sonification Doomed to Fail?. In Proceedings of the 25th International Conference on Auditory Display (ICAD 2019). Department of Computer and Information Sciences, Northumbria University, Newcastle upon Tyne, 327–330. https://doi.org/10.21785/icad2019.069
[23]
Sandra Pauletto and Andy Hunt. 2004. A Toolkit for Interactive Sonification. In Proceedings of the 15th International Conference on Auditory Display. 7. https://www.academia.edu/35199105/A_Toolkit_for_Interactive_Sonification
[24]
Helen Petrie, Neil King, Anne-Marie Burn, and Peter Pavan. 2006. Providing interactive access to architectural floorplans for blind people. British Journal of Visual Impairment 24, 1 (Jan 2006), 4–11. https://doi.org/10.1177/0264619606060010
[25]
Helen Petrie, Christoph Schlieder, Paul Blenkhorn, Gareth Evans, Alasdair King, Anne-Marie O’Neill, George T. Ioannidis, Blaithin Gallagher, David Crombie, Rolf Mager, and Maurizio Alafaci. 2002. TeDUB: A System for Presenting and Exploring Technical Drawings for Blind People. In Computers Helping People with Special Needs(Lecture Notes in Computer Science), Klaus Miesenberger, Joachim Klaus, and Wolfgang Zagler (Eds.). Springer, Berlin, Heidelberg, 537–539. https://doi.org/10.1007/3-540-45491-8_102
[26]
Sean Phillips and Andrés Cabrera. 2019. Sonification Workstation. In Proceedings of the 25th International Conference on Auditory Display (ICAD 2019). Department of Computer and Information Sciences, Northumbria University, Newcastle upon Tyne, 184–190. https://doi.org/10.21785/icad2019.056
[27]
Benjamin Poppinga, Charlotte Magnusson, Martin Pielot, and Kirsten Rassmus-Gröhn. 2011. TouchOver map: audio-tactile exploration of interactive maps. In Proceedings of the 13th International Conference on Human Computer Interaction with Mobile Devices and Services. ACM, Stockholm Sweden, 545–550. https://doi.org/10.1145/2037373.2037458
[28]
Dennis Reinsch and Thomas Hermann. 2023. Interacting with Sonifications: The Mesonic Framework for Interactive Auditory Data Science. Zenodo (Jan 2023). https://doi.org/10.5281/zenodo.755224
[29]
Freedom Scientific. [n. d.]. JAWS®. https://www. freedomscientifc.com/products/software/jaws
[30]
Ather Sharif, Olivia H. Wang, Alida T. Muongchan, Katharina Reinecke, and Jacob O. Wobbrock. 2022. VoxLens: Making Online Data Visualizations Accessible with an Interactive JavaScript Plug-In. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems(CHI ’22). Association for Computing Machinery, New York, NY, USA, 1–19. https://doi.org/10.1145/3491102.3517431
[31]
John R Thompson, Jesse J Martinez, Alper Sarikaya, Edward Cutrell, and Bongshin Lee. 2023. Chart Reader: Accessible Visualization Experiences Designed with Screen Reader Users. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems(CHI ’23). Association for Computing Machinery, New York, NY, USA, 1–18. https://doi.org/10.1145/3544548.3581186
[32]
Vega. [n. d.]. Vega Datasets. https://github.com/vega/vega-datasets
[33]
Bruce N. Walker and Joshua T. Cothran. 2003. Sonification Sandbox: A Graphical Toolkit for Auditory Graphs. In Proceedings of the 2003 International Conference on Auditory Display. Boston, MA, USA. http://sonify.psych.gatech.edu/publications/pdfs/2003ICAD-WalkerCothran-Sandbox.pdf
[34]
Yanan Wang, Ruobin Wang, Crescentia Jung, and Yea-Seul Kim. 2022. What makes web data tables accessible? Insights and a tool for rendering accessible tables for people with visual impairments. In CHI Conference on Human Factors in Computing Systems. ACM, New Orleans LA USA, 1–20. https://doi.org/10.1145/3491102.3517469
[35]
David Worrall, Michael Bylstra, Stephen Barrass, and Roger Dean. 2007. SoniPy: The Design of an Extendable Software Framework for Sonification Research and Auditory Display. In Proceedings of the 13th International Conference on Auditory Display. Montréal, Canada. https://www.researchgate.net/publication/254057374_SONIPY_THE_DESIGN_OF_AN_EXTENDABLE_SOFTWARE_FRAMEWORK_FOR_SONIFICATION_RESEARCH_AND_AUDITORY_DISPLAY
[36]
Zhuohao Zhang and Jacob O. Wobbrock. 2022. A11yBoard: Using Multimodal Input and Output to Make Digital Artboards Accessible to Blind Users. In Adjunct Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology(UIST ’22 Adjunct). Association for Computing Machinery, New York, NY, USA, 1–4. https://doi.org/10.1145/3526114.3558695
[37]
Haixia Zhao, Catherine Plaisant, Ben Shneiderman, and Jonathan Lazar. 2008. Data Sonification for Users with Visual Impairment: A Case Study with Georeferenced Data. ACM Transactions on Computer-Human Interaction 15, 1 (May 2008), 4:1–4:28. https://doi.org/10.1145/1352782.1352786
[38]
Yichun Zhao, Miguel A Nacenta, Mahadeo A Sukhai, and Sowmya Somanath. 2024. TADA: Making Node-link Diagrams Accessible to Blind and Low-Vision People. In Proceedings of the CHI Conference on Human Factors in Computing Systems (CHI ’24), May 11–16, 2024, Honolulu, HI, USA. ACM, Honolulu, HI, USA. https://doi.org/10.1145/3613904.3642222
[39]
Yichun Zhao and George Tzanetakis. 2022. Interactive Sonification for Health and Energy using ChucK and Unity. In Proceedings of the Conference on Sonification of Health and Environmental Data (SoniHED 2022). Zenodo. https://doi.org/10.5281/zenodo.7243950
[40]
Jonathan Zong, Crystal Lee, Alan Lundgard, JiWoong Jang, Daniel Hajas, and Arvind Satyanarayan. 2022. Rich Screen Reader Experiences for Accessible Data Visualization. Computer Graphics Forum 41, 3 (2022), 15–27. https://doi.org/10.1111/cgf.14519
[41]
Jonathan Zong, Isabella Pedraza Pineros, Mengzhu (Katie) Chen, Daniel Hajas, and Arvind Satyanarayan. 2024. Umwelt: Accessible Structured Editing of Multi-Modal Data Representations. In Proceedings of the CHI Conference on Human Factors in Computing Systems. ACM, Honolulu HI USA, 1–20. https://doi.org/10.1145/3613904.3641996
Index Terms
- Speech-based Mark for Data Sonification
Recommendations
Articulatory Speech Re-synthesis: Profiting from Natural Acoustic Speech Data
Cross-Modal Analysis of Speech, Gestures, Gaze and Facial ExpressionsThe quality of static phones (e.g. vowels, fricatives, nasals, laterals) generated by articulatory speech synthesizers has reached a high level in the last years. Our goal is to expand this high quality to dynamic speech, i.e. whole syllables, words, ...
Psycho-acoustics inspired automatic speech recognition
AbstractUnderstanding the human spoken language recognition process is still a far scientific goal. Nowadays, commercial automatic speech recognisers (ASRs) achieve high performance at recognising clean speech, but their approaches are poorly ...
Highlights- We propose a novel Automatic Speech Recognizer inspired by psycho-acoustic studies.
Effects of Speaking Rate on Speech and Silent Speech Recognition
CHI EA '22: Extended Abstracts of the 2022 CHI Conference on Human Factors in Computing SystemsSpeaking rate or the speed at which a person speaks is a fundamental user characteristic. This work investigates the rate in which users speak when interacting with speech and silent speech-based methods. Results revealed that native users speak about ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
October 2024
1475 pages
ISBN:9798400706776
DOI:10.1145/3663548
Copyright © 2024 Owner/Author.
Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 27 October 2024
Check for updates
Author Tags
Qualifiers
- Poster
- Research
- Refereed limited
Conference
ASSETS '24
Sponsor:
ASSETS '24: The 26th International ACM SIGACCESS Conference on Computers and Accessibility
October 27 - 30, 2024
NL, St. John's, Canada
Acceptance Rates
Overall Acceptance Rate 436 of 1,556 submissions, 28%
Upcoming Conference
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 47Total Downloads
- Downloads (Last 12 months)47
- Downloads (Last 6 weeks)16
Reflects downloads up to 02 Mar 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign inFull Access
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderHTML Format
View this article in HTML Format.
HTML Format