More Web Proxy on the site http://driver.im/

research-article

Open access

Notably Inaccessible — Data Driven Understanding of Data Science Notebook (In)Accessibility

Authors:

Venkatesh Potluri,

Sudheesh Singanamalla,

Nussara Tieanklin,

Jennifer MankoffAuthors Info & Claims

ASSETS '23: Proceedings of the 25th International ACM SIGACCESS Conference on Computers and Accessibility

Article No.: 13, Pages 1 - 19

https://doi.org/10.1145/3597638.3608417

Published: 22 October 2023 Publication History

All formats PDF

Abstract

Computational notebooks, tools that facilitate storytelling through exploration, data analysis, and information visualization, have become the widely accepted standard in the data science community. These notebooks have been widely adopted through notebook software such as Jupyter, Datalore and Google Colab, both in academia and industry. While there is extensive research to learn how data scientists use computational notebooks, identify their pain points, and enable collaborative data science practices, very little is known about the various accessibility barriers experienced by blind and visually impaired (BVI) users using these notebooks. BVI users are unable to use computational notebook interfaces due to (1) inaccessibility of the interface, (2) common ways in which data is represented in these interfaces, and (3) inability for popular libraries to provide accessible outputs. We perform a large scale systematic analysis of 100000 Jupyter notebooks to identify various accessibility challenges in published notebooks affecting the creation and consumption of these notebooks. Through our findings, we make recommendations to improve accessibility of the artifacts of a notebook, suggest authoring practices, and propose changes to infrastructure to make notebooks accessible.

References

[1]

ACM. 2017. Software system award goes to three for pioneering open source initiatives. https://awards.acm.org/software-system

[2]

Khaled Albusays, Stephanie Ludi, and Matt Huenerfauth. 2017. Interviews and observation of blind software developers at work to understand code navigation challenges. In Proceedings of the 19th International ACM SIGACCESS Conference on Computers and Accessibility (Baltimore, Maryland, USA) (ASSETS ’17). Association for Computing Machinery, New York, NY, USA, 91–100. https://doi.org/10.1145/3132525.3132550

Digital Library

[3]

Mohammad Almogbel. 2022. JupyterLab horizon theme. https://github.com/mohirio/jupyterlab-horizon-theme. (Accessed on 05/03/2023).

[4]

Apple. 2022. Apple Developer Documentation: Audio Graphs. https://developer.apple.com/documentation/accessibility/audio_graphs.

[5]

Catherine M. Baker, Lauren R. Milne, and Richard E. Ladner. 2015. StructJumper: A tool to help blind programmers navigate and understand the structure of code. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (Seoul, Republic of Korea) (CHI ’15). Association for Computing Machinery, New York, NY, USA, 3043–3052. https://doi.org/10.1145/2702123.2702589

Digital Library

[6]

Stanley J Cantrell, Bruce N Walker, and Øystein Moseng. 2021. Highcharts Sonification Studio: An online, open-source, extensible, and accessible data sonification tool. In 26th International Conference on Auditory Display (ICAD’21). Georgia Institute of Technology, International Community for Auditory Display (ICAD), Virtual Online Event, 210–216.

[7]

SAS Help Center. 2023. ALTDESC, NOALTDESC Graphics Options. https://documentation.sas.com/doc/en/pgmsascdc/v_034/graphref/n1gj4oarw072dxn1bl3pcqbdv644.htm. (Accessed on 05/03/2023).

[8]

Morakot Choetkiertikul, Apirak Hoonlor, Chaiyong Ragkhitwetsagul, Siripen Pongpaichet, Thanwadee Sunetnanta, Tasha Settewong, Vacharavich Jiravatvanich, and Urisayar Kaewpichai. 2023. Mining the characteristics of Jupyter notebooks in data science projects. arxiv:2304.05325 [cs.SE]

[9]

Asa Dotzler. 2022. Cache the World Opt In Preview - Mozilla Accessibility. https://blog.mozilla.org/accessibility/cache-the-world-opt-in-preview/. (Accessed on 05/03/2023).

[10]

Md Ehtesham-Ul-Haque, Syed Mostofa Monsur, and Syed Masum Billah. 2022. Grid-Coding: An accessible, efficient, and structured coding paradigm for Blind and low-vision programmers. In Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology (Bend, OR, USA) (UIST ’22). Association for Computing Machinery, New York, NY, USA, Article 44, 21 pages. https://doi.org/10.1145/3526113.3545620

Digital Library

[11]

Frank Elavsky, Cynthia Bennett, and Dominik Moritz. 2022. How accessible is my visualization? Evaluating visualization accessibility with chartability. Computer Graphics Forum 41, 3 (2022), 57–70. https://doi.org/10.1111/cgf.14522 arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1111/cgf.14522

[12]

Tony Fast. 2023. Notebook authoring accessibility checklist. https://github.com/Iota-School/notebooks-for-all/blob/main/resources/event-hackathon/notebook-authoring-checklist.md. (Accessed on 05/03/2023).

[13]

Tony Fast. 2023. Rendering dataframes for screen readers with pandas. Independent. https://tonyfast.github.io/tonyfast/xxiii/2023-01-02-accessible-dataframes-basic-indexes.html

[14]

Alena Guzharina. 2020. We downloaded 10,000,000 Jupyter notebooks from GitHub – This is what we learned | The JetBrains Datalore blog. https://blog.jetbrains.com/datalore/2020/12/17/we-downloaded-10-000-000-jupyter-notebooks-from-github-this-is-what-we-learned/. (Accessed on 01/18/2023).

[15]

Leona M Holloway, Cagatay Goncu, Alon Ilsar, Matthew Butler, and Kim Marriott. 2022. Infosonics: Accessible infographics for people who are blind using sonification and voice. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (New Orleans, LA, USA) (CHI ’22). Association for Computing Machinery, New York, NY, USA, Article 480, 13 pages. https://doi.org/10.1145/3491102.3517465

Digital Library

[16]

Web Accessibility Initiative. 2020. Web Content Accessibility Guidelines (WCAG) 2 Level AA Conformance | Web Accessibility Initiative (WAI) | W3C. https://www.w3.org/WAI/WCAG2AA-Conformance. (Accessed on 05/03/2023).

[17]

JetBrains. 2013. Platform theme colors. https://jetbrains.design/intellij/principles/platform_theme_colors/. (Accessed on 05/03/2023).

[18]

K. V. Jobin, Ajoy Mondal, and C. V. Jawahar. 2019. DocFigure: A dataset for scientific document figure classification. In 2019 International Conference on Document Analysis and Recognition Workshops (ICDARW), Vol. 1. IEEE, Sydney, Australia, 74–79. https://doi.org/10.1109/ICDARW.2019.00018

[19]

Man From Jupyter. 2020. Accessibility issues needing addressing for WCAG 2.1 compliance (As of Version 2.2.6) · Issue #9399 · jupyterlab/jupyterlab. https://github.com/jupyterlab/jupyterlab/issues/9399. (Accessed on 01/18/2023).

[20]

Hernisa Kacorri, Utkarsh Dwivedi, Sravya Amancherla, Mayanka Jha, and Riya Chanduka. 2020. IncluSet: A data surfacing repository for accessibility datasets. In Proceedings of the 22nd International ACM SIGACCESS Conference on Computers and Accessibility (Virtual Event, Greece) (ASSETS ’20). Association for Computing Machinery, New York, NY, USA, Article 72, 4 pages. https://doi.org/10.1145/3373625.3418026

Digital Library

[21]

N. W. Kim, S. C. Joyner, A. Riegelhuth, and Y. Kim. 2021. Accessible Visualization: Design space, opportunities, and challenges. Computer Graphics Forum 40, 3 (2021), 173–188. https://doi.org/10.1111/cgf.14298 arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1111/cgf.14298

[22]

Donald Ervin Knuth. 1984. Literate programming. The computer journal 27, 2 (1984), 97–111.

[23]

Jingyi Li, Son Kim, Joshua A. Miele, Maneesh Agrawala, and Sean Follmer. 2019. Editing spatial layouts through tactile templates for people with visual impairments. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI ’19). ACM, New York, NY, USA, Article 206, 11 pages. https://doi.org/10.1145/3290605.3300436

Digital Library

[24]

Junchen Li, Garreth W. Tigwell, and Kristen Shinohara. 2021. Accessibility of high-fidelity prototyping tools. In CHI ’21: CHI Conference on Human Factors in Computing Systems, Virtual Event / Yokohama, Japan, May 8-13, 2021, Yoshifumi Kitamura, Aaron Quigley, Katherine Isbister, Takeo Igarashi, Pernille Bjørn, and Steven Mark Drucker (Eds.). ACM, New York, NY, USA, 493:1–493:17. https://doi.org/10.1145/3411764.3445520

Digital Library

[25]

Jiasheng Li, Zeyu Yan, Ebrima Haddy Jarjue, Ashrith Shetty, and Huaishu Peng. 2022. TangibleGrid: Tangible web layout design for Blind users. In Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology (Bend, OR, USA) (UIST ’22). Association for Computing Machinery, New York, NY, USA, Article 47, 12 pages. https://doi.org/10.1145/3526113.3545627

Digital Library

[26]

Alan Lundgard and Arvind Satyanarayan. 2021. Accessible visualization via natural language descriptions: A four-level model of semantic content. IEEE transactions on visualization and computer graphics 28, 1 (2021), 1073–1083.

Digital Library

[27]

Kelly Mack, Emma McDonnell, Dhruv Jain, Lucy Lu Wang, Jon E. Froehlich, and Leah Findlater. 2021. What do we mean by “Accessibility Research?” A literature survey of accessibility papers in CHI and ASSETS from 1994 to 2019. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Yokohama, Japan) (CHI ’21). Association for Computing Machinery, New York, NY, USA, 12 pages. https://doi.org/10.1145/3411764.3445412

Digital Library

[28]

Jennifer Mankoff, Holly Fait, and Tu Tran. 2005. Is your web page accessible? A comparative study of methods for assessing web page accessibility for the blind. In Proceedings of the 2005 Conference on Human Factors in Computing Systems, CHI 2005, Portland, Oregon, USA, April 2-7, 2005, Gerrit C. van der Veer and Carolyn Gale (Eds.). ACM, New York, NY, USA, 41–50. https://doi.org/10.1145/1054972.1054979

Digital Library

[29]

Rowan Manning, Hollie Kay, Jude Robinson, Glynn Phillips, Andrew Mee, Perry Harlock, and Alex Kilgour. 2022. Pa11y–Automated accessibility testing pal. pa11y. https://pa11y.org/

[30]

Sean Mealin and Emerson Murphy-Hill. 2012. An exploratory study of blind software developers. In 2012 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC). IEEE, Innsbruck, Austria, 71–74. https://doi.org/10.1109/VLHCC.2012.6344485

[31]

Oriol Mirosa. 2022. ‘Material Darker’ JupyterLab extension. https://github.com/oriolmirosa/jupyterlab_materialdarker. (Accessed on 05/03/2023).

[32]

NVAccess. 2023. NVDA 2023.1 User Guide. https://www.nvaccess.org/files/nvda/documentation/userGuide.html. (Accessed on 05/04/2023).

[33]

Maulishree Pandey, Sharvari Bondre, Sile O’Modhrain, and Steve Oney. 2022. Accessibility of UI frameworks and libraries for programmers with visual impairments. In 2022 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC). IEEE, Rome, Italy, 1–10. https://doi.org/10.1109/VL/HCC53370.2022.9833098

[34]

Maulishree Pandey, Vaishnav Kameswaran, Hrishikesh V. Rao, Sile O’Modhrain, and Steve Oney. 2021. Understanding accessibility and collaboration in programming for people with visual impairments. In Proceedings of the CSCW Conference on Computer Supported Cooperative Work (Virtual) (CSCW ’21). Association for Computing Machinery, New York, NY, USA, 30 pages.

Digital Library

[35]

João Felipe Pimentel, Leonardo Murta, Vanessa Braganholo, and Juliana Freire. 2021. Understanding and improving the quality and reproducibility of Jupyter notebooks. Empirical Software Engineering 26, 4 (2021), 1–55.

Digital Library

[36]

Venkatesh Potluri, Liang He, Christine Chen, Jon E. Froehlich, and Jennifer Mankoff. 2019. A multi-modal approach for blind and visually impaired developers to edit webpage designs. In The 21st International ACM SIGACCESS Conference on Computers and Accessibility (Pittsburgh, PA, USA) (ASSETS ’19). Association for Computing Machinery, New York, NY, USA, 612–614. https://doi.org/10.1145/3308561.3354626

Digital Library

[37]

Venkatesh Potluri, Maulishree Pandey, Andrew Begel, Michael Barnett, and Scott Reitherman. 2022. CodeWalk: Facilitating shared awareness in mixed-ability collaborative software development. In Proceedings of the 24th International ACM SIGACCESS Conference on Computers and Accessibility (Athens, Greece) (ASSETS ’22). Association for Computing Machinery, New York, NY, USA, Article 20, 16 pages. https://doi.org/10.1145/3517428.3544812

Digital Library

[38]

Venkatesh Potluri, Sudheesh Singanamalla, Nussara Tieanklin, and Jennifer Mankoff. 2023. Notably inaccessible – data driven understanding of data science notebook (in)accessibility (Dataset). Zenodo. https://doi.org/10.5281/zenodo.8185050

[39]

Venkatesh Potluri, Sudheesh Singanamalla, Nussara Tieanklin, and Jennifer Mankoff. 2023. Notably inaccessible – data driven understanding of data science notebook (in)accessibility (Source code). https://github.com/make4all/notebooka11y

[40]

Venkatesh Potluri, John Thompson, James Devine, Bongshin Lee, Nora Morsi, Peli De Halleux, Steve Hodges, and Jennifer Mankoff. 2022. PSST: Enabling Blind or visually impaired developers to author sonifications of streaming sensor data. In Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology (Bend, OR, USA) (UIST ’22). Association for Computing Machinery, New York, NY, USA, Article 46, 13 pages. https://doi.org/10.1145/3526113.3545700

Digital Library

[41]

Venkatesh Potluri, Priyan Vaithilingam, Suresh Iyengar, Y. Vidya, Manohar Swaminathan, and Gopal Srinivasa. 2018. CodeTalk: Improving programming environment accessibility for visually impaired developers. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (Montreal QC, Canada) (CHI ’18). Association for Computing Machinery, New York, NY, USA, 1–11. https://doi.org/10.1145/3173574.3174192

Digital Library

[42]

Fernando Pérez, Brian Granger, and Min Ragan-Kelley. 2014. Jupyter. Jupyter. https://jupyter.org/

[43]

Luigi Quaranta, Fabio Calefato, and Filippo Lanubile. 2021. KGTorrent: A dataset of python Jupyter notebooks from Kaggle. In 2021 IEEE/ACM 18th International Conference on Mining Software Repositories (MSR). IEEE, IEEE Computer Society, Virtual (originally Madrid, Spain), 550–554.

[44]

Adam Rule, Aurélien Tabard, and James D. Hollan. 2018. Exploration and explanation in computational notebooks. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (Montreal QC, Canada) (CHI ’18). Association for Computing Machinery, New York, NY, USA, 1–12. https://doi.org/10.1145/3173574.3173606

Digital Library

[45]

Emmanuel Schanzer, Sina Bahram, and Shriram Krishnamurthi. 2019. Accessible AST-based programming for visually-impaired programmers. In Proceedings of the 50th ACM Technical Symposium on Computer Science Education (Minneapolis, MN, USA) (SIGCSE ’19). Association for Computing Machinery, New York, NY, USA, 773–779. https://doi.org/10.1145/3287324.3287499

Digital Library

[46]

Ethan Schoonover. 2011. Solarized. https://ethanschoonover.com/solarized/. (Accessed on 05/03/2023).

[47]

Ather Sharif, Sanjana Shivani Chintalapati, Jacob O. Wobbrock, and Katharina Reinecke. 2021. Understanding screen-reader users’ experiences with online data visualizations. In ASSETS ’21: The 23rd International ACM SIGACCESS Conference on Computers and Accessibility, Virtual Event, USA, October 18-22, 2021, Jonathan Lazar, Jinjuan Heidi Feng, and Faustina Hwang (Eds.). ACM, New York, NY, USA, 14:1–14:16. https://doi.org/10.1145/3441852.3471202

Digital Library

[48]

Ather Sharif, Olivia H. Wang, Alida T. Muongchan, Katharina Reinecke, and Jacob O. Wobbrock. 2022. VoxLens: Making online data visualizations accessible with an interactive JavaScript plug-in. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (New Orleans, LA, USA) (CHI ’22). Association for Computing Machinery, New York, NY, USA, Article 478, 19 pages. https://doi.org/10.1145/3491102.3517431

Digital Library

[49]

Ashrith Shetty, Ebrima Jarjue, and Huaishu Peng. 2020. Tangible web layout design for blind and visually impaired people: An initial investigation. In Adjunct Publication of the 33rd Annual ACM Symposium on User Interface Software and Technology (Virtual Event, USA) (UIST ’20 Adjunct). Association for Computing Machinery, New York, NY, USA, 37–39. https://doi.org/10.1145/3379350.3416178

Digital Library

[50]

Yan Song, Peiseng Wang, Xinhai Hong, and Ian McLoughlin. 2017. Fisher vector based CNN architecture for image classification. In 2017 IEEE International Conference on Image Processing (ICIP). IEEE, Beijing, China, 565–569. https://doi.org/10.1109/ICIP.2017.8296344

Digital Library

[51]

Space Telescope Science Institute, Iota School and Quansight Labs. 2023. Astronomy notebooks for all. GitHub. https://github.com/Iota-School/notebooks-for-all

[52]

Andreas Stefik, Roger Alexander, Robert Patterson, and Jonathan Brown. 2007. WAD: A feasibility study using the wicked audio debugger. In Proceedings of the 15th IEEE International Conference on Program Comprehension(ICPC ’07). IEEE Computer Society, USA, 69–80. https://doi.org/10.1109/ICPC.2007.42

Digital Library

[53]

Andreas Stefik, Andrew Haywood, Shahzada Mansoor, Brock Dunda, and Daniel Garcia. 2009. Sodbeans. In 2009 IEEE 17th International Conference on Program Comprehension. IEEE, Vancouver, BC, Canada, 293–294.

[54]

Kevin M. Storer, Harini Sampath, and M. Alice Merrick. 2021. “It’s just everything outside of the IDE that’s the problem”: Information seeking by software developers with visual impairments. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Yokohama, Japan) (CHI ’21). Association for Computing Machinery, New York, NY, USA, 12 pages. https://doi.org/10.1145/3411764.3445090

Digital Library

[55]

WebAIM Surveys. 2021. Screen reader user survey #9 results. https://webaim.org/projects/screenreadersurvey9/. (Accessed on 05/03/2023).

[56]

Marcy Sutton. 2023. Automating peace of mind with accessibility resting & continuous integration. Deque. https://marcysutton.github.io/a11y-and-ci/#/

[57]

Jupyter Development Team. 2020. nbformat. Jupyter. https://github.com/jupyter/nbformat

[58]

Jiawei Wang, Li Li, and Andreas Zeller. 2020. Better code, better sharing: On the need of analyzing Jupyter Notebooks. In Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering: New Ideas and Emerging Results (Seoul, South Korea) (ICSE-NIER ’20). Association for Computing Machinery, New York, NY, USA, 53–56. https://doi.org/10.1145/3377816.3381724

Digital Library

[59]

Yanan Wang, Ruobin Wang, Crescentia Jung, and Yea-Seul Kim. 2022. What makes web data tables accessible? Insights and a tool for rendering accessible tables for people with visual impairments. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (New Orleans, LA, USA) (CHI ’22). Association for Computing Machinery, New York, NY, USA, Article 595, 20 pages. https://doi.org/10.1145/3491102.3517469

Digital Library

[60]

Marco Zehe. 2017. Rethinking web accessibility on windows. https://www.marcozehe.de/rethinking-web-accessibility-on-windows/. (Accessed on 05/03/2023).

[61]

Jonathan Zong, Crystal Lee, Alan Lundgard, JiWoong Jang, Daniel Hajas, and Arvind Satyanarayan. 2022. Rich screen reader experiences for accessible data visualization. Computer Graphics Forum 41, 3 (2022), 13 pages. https://doi.org/10.1111/cgf.14519

Cited By

Chundury PThakkar UReyazuddin YJordan JElmqvist NLazar J(2024)Understanding the Visualization and Analytics Needs of Blind and Low-Vision ProfessionalsProceedings of the 26th International ACM SIGACCESS Conference on Computers and Accessibility10.1145/3663548.3688496(1-5)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3663548.3688496

Index Terms

Recommendations

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

ASSETS '23: Proceedings of the 25th International ACM SIGACCESS Conference on Computers and Accessibility

October 2023

1163 pages

ISBN:9798400702204

DOI:10.1145/3597638

Copyright © 2023 Owner/Author.

This work is licensed under a Creative Commons Attribution International 4.0 License.

Sponsors

SIGACCESS: ACM Special Interest Group on Accessible Computing

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 22 October 2023

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Funding Sources

NSF
UW CREATE

Conference

ASSETS '23

Sponsor:

SIGACCESS

ASSETS '23: The 25th International ACM SIGACCESS Conference on Computers and Accessibility

October 22 - 25, 2023

NY, New York, USA

Acceptance Rates

ASSETS '23 Paper Acceptance Rate 55 of 182 submissions, 30%;

Overall Acceptance Rate 436 of 1,556 submissions, 28%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1
Total Citations
View Citations
610
Total Downloads

Downloads (Last 12 months)533
Downloads (Last 6 weeks)78

Reflects downloads up to 04 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Chundury PThakkar UReyazuddin YJordan JElmqvist NLazar J(2024)Understanding the Visualization and Analytics Needs of Blind and Low-Vision ProfessionalsProceedings of the 26th International ACM SIGACCESS Conference on Computers and Accessibility10.1145/3663548.3688496(1-5)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3663548.3688496

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents