More Web Proxy on the site http://driver.im/

research-article

Understanding Personalized Accessibility through Teachable AI: Designing and Evaluating Find My Things for People who are Blind or Low Vision

Authors:

Cecily Morrison,

Martin Grayson,

Rita Faia Marques,

Daniela Massiceti,

Camilla Longden,

Edward CutrellAuthors Info & Claims

ASSETS '23: Proceedings of the 25th International ACM SIGACCESS Conference on Computers and Accessibility

Article No.: 31, Pages 1 - 12

https://doi.org/10.1145/3597638.3608395

Published: 22 October 2023 Publication History

Abstract

The opportunity for artificial intelligence, or AI, to enable accessibility is rapidly growing, but widely impactful applications can be challenging to build given the diversity of user need within and across disability communities. Teachable AI systems give users with disabilities a way to leverage the power of AI to personalize applications for their own specific needs, as long as the effort of providing examples is balanced with the benefit of the personalization received. As an example, this paper presents the design and evaluation of Find My Things, an end-to-end application that can be taught by people who are blind or low vision to find their personal things. Through synthesis of the design process, this paper offers design considerations for the teaching loop that is so critical to realizing the power of teachable AI for accessibility.

References

[1]

Dragan Ahmetovic, Daisuke Sato, Uran Oh, Tatsuya Ishihara, Kris Kitani, and Chieko Asakawa. 2020. Recog: Supporting blind people in recognizing personal objects. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, 1–12.

Digital Library

[2]

Saleema Amershi, Maya Cakmak, William Bradley Knox, and Todd Kulesza. 2014. Power to the People: The Role of Humans in Interactive Machine Learning. AI Magazine 35, 4: 105–120.

Digital Library

[3]

Saleema Amershi, James Fogarty, Ashish Kapoor, and Desney Tan. 2009. Designing for End-User Interactive Concept Learning in CueFlik. Neural Information Processing Systems (NIPS) Workshop on Analysis and Design of Algorithms for Interactive Machine Learning.

[4]

C. Andrews. 2014. Accessible Participatory Design: Engaging and Including Visually Impaired Participants. In Inclusive Designing. Springer International Publishing, 201–210.

[5]

Harshadha Balasubramanian, Cecily Morrison, Martin Grayson, Zhanat Makhataeva, Rita Marques, Thomas Gable, Dalya Perez, and Edward Cutrell. 2023. Enable Blind Users’ Experience in 3D Virtual Environments: The Scene Weaver Prototype. In In 2023 CHI EA Conference on Human Factors in Computing Systems, 1–4.

[6]

Michelle Carney, Barron Webster, Irene Alvarado, Kyle Phillips, Noura Howell, Jordan Griffith, Jonas Jongejan, Amit Pitaru, and Alexander Chen. 2020. Teachable machine: Approachable Web-based tool for exploring machine learning classification. In The 2020 CHI EA Conference on Human Factors in Computing Systems, 1–8.

Digital Library

[7]

Da Chen, Yuefeng Chen, Yuhong Li, Feng Mao, Yuan He, and Hui Xue. 2021. Self-Supervised Learning for Few-Shot Image Classification. In ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 1745–1749.

[8]

Rebecca Fiebrink, Perry R. Cook, and Dan Trueman. 2011. Human model evaluation in interactive supervised learning. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 147–156.

Digital Library

[9]

C. Finn, P. Abbeel, and S. Levine. 2017. Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the 2017 ICML International Conference on Machine Learning, 1126–1135.

[10]

Stephan Forchhammer, Amar Abu-Ghazaleh, Gisela Metzler, Claus Garbe, and Thomas Eigentler. 2022. Development of an Image Analysis-Based Prognosis Score Using Google's Teachable Machine in Melanoma. Cancers 14, 9: 2243.

[11]

Steven M. Goodman, Ping Liu, Dhruv Jain, Emma J. McDonnell, Jon E. Froehlich, and Leah Findlater. 2021. Toward User-Driven Sound Recognizer Personalization with People Who Are d/Deaf or Hard of Hearing. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 5, 2: 1–23.

Digital Library

[12]

Jonggi Hong, Jaina Gandhi, Ernest Essuah Mensah, Farnaz Zamiri Zeraati, Ebrima Haddy Jarjue, Kyungjun Lee, and Hernisa Kacorri. 2022. Blind Users Accessing Their Training Images in Teachable Object Recognizers.

[13]

Jonggi Hong, Kyungjun Lee, June Xu, and Hernisa Kacorri. 2020. Crowdsourcing the Perception of Machine Teaching. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, 1–14.

Digital Library

[14]

Hernisa Kacorri. 2017. Teachable machines for accessibility. ACM SIGACCESS Accessibility and Computing 119: 10–18.

Digital Library

[15]

Hernisa Kacorri, Kris M. Kitani, Jeffrey P. Bigham, and Chieko Asakawa. 2017. People with visual impairment training personal object recognizers: Feasibility and challenges. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, 5839–5849.

Digital Library

[16]

Rie, Kamikubo, Kyungjun Lee, and Hernisa Kacorri. 2023. Contributing to Accessibility Datasets: Reflections on Sharing Study Data by Blind People . In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, 1–18.

[17]

Simon Katan, Mick Grierson, and Rebecca Fiebrink. 2015. Using Interactive Machine Learning to Support Interface Development Through Workshops with Disabled People. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, 251–254.

Digital Library

[18]

Todd Kulesza, Weng-Keen Wong, Simone Stumpf, Stephen Perona, Rachel White, Margaret M. Burnett, Ian Oberst, and Amy J. Ko. 2009. Fixing the program my computer learned. In Proceedings of the 14th international conference on Intelligent user interfaces, 187–196.

[19]

Brenden M Lake, Ruslan Salakhutdinov, Jason Gross, and Joshua B Tenenbaum. 2011. One shot learning of simple visual concepts. Cognitive Science 33, 2568 – 2573.

[20]

Kyungjun Lee, Abhinav Shrivastava, and Hernisa Kacorri. 2020. Hand-Priming in Object Localization for Assistive Egocentric Vision. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 3422-3432.

[21]

Kelly Mack, Emma McDonnell, Dhruv Jain, Lucy Lu Wang, Jon E. Froehlich, and Leah Findlater. 2021. What Do We Mean by “Accessibility Research”? In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, 1–18.

Digital Library

[22]

Daniela Massiceti, Luisa Zintgraf, John Bronskill, Lida Theodorou, Matthew Tobias Harris, Edward Cutrell, Cecily Morrison, Katja Hofmann, and Simone Stumpf. 2021. ORBIT: A Real-World Few-Shot Dataset for Teachable Object Recognition.

[23]

Yuri Nakao and Yusuke Sugano. 2020. Use of Machine Learning by Non-Expert DHH People: Technological Understanding and Sound Perception. In Proceedings of the 11th Nordic Conference on Human-Computer Interaction: Shaping Experiences, Shaping Society, 1–12.

Digital Library

[24]

Eunbyung Park and Junier B Oliva. 2019. Meta-Curvature. In Advances in Neural Information Processing Systems 32, 1-11.

[25]

Gonzalo Ramos, Christopher Meek, Patrice Simard, Jina Suh, and Soroush Ghorashi. 2020. Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35, 5–6: 413–451.

[26]

Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. 2015. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision 115, 211-252.

Digital Library

[27]

Téo Sanchez, Baptiste Caramiaux, Jules Françoise, Frédéric Bevilacqua, and Wendy E. Mackay. 2021. How do People Train a Machine? Proceedings of the ACM on Human-Computer Interaction 5, CSCW1: 1–26.

[28]

Dan Shapiro. 2005. Participatory design: the will to succeed. In Proceedings of the CC Conference on Critical Computing, 29-38.

Digital Library

[29]

Jonathan Silvertown. 2009. A new dawn for citizen science. Trends in Ecology & Evolution 24, 9: 467–471.

[30]

Jake Snell, Kevin Swersky, and Richard Zemel. 2017. Prototypical Networks for Few-shot Learning. In Advances in Neural Information Processing Systems 30, 1 – 11.

[31]

Mingxing Tan and Quoc Le. 2019. EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks. In Proceedings of the ICML International Conference on Machine Learning, 6105–6114.

[32]

Anja Thieme, Cynthia L. Bennett, Cecily Morrison, Edward Cutrell, and Alex S. Taylor. 2018. "I can do everything but see!–How People with Vision Impairments Negotiate their Abilities in Social Contexts. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, 203.

[33]

Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Koray Kavukcuoglu, and Daan Wierstra. 2016. Matching Networks for One Shot Learning. In Advances in Neural Information Processing Systems, 29, 1-9.

[34]

Michele A Williams, Amy Hurst, and Shaun K Kane. 2013. “Pray Before You Step out”: Describing Personal and Situational Blind Navigation Behaviors. In Proceedings of the ACM SIGACCESS Conference on Computers and Accessibility: 1-8.

Digital Library

[35]

Yuhang Zhao, Edward Cutrell, Christian Holz, Meredith Ringel Morris, Eyal Ofek, and Andrew D. Wilson. 2019. SeeingVR. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, 1–14.

[36]

Luisa Zintgraf, Kyriacos Shiarli, Vitaly Kurin, Katja Hofmann, and Shimon Whiteson. 2019. Fast Context Adaptation via Meta-Learning. In Proceedings of the ICML Conference on Machine Learning, 7693–7702.

Cited By

Yu RLee SXie JBillah SCarroll J(2024)Human–AI Collaboration for Remote Sighted Assistance: Perspectives from the LLM EraFuture Internet10.3390/fi1607025416:7(254)Online publication date: 18-Jul-2024
https://doi.org/10.3390/fi16070254
Luo SLiu JHu B(2024)Hearing the Bullseye: An Auditory-Cued Archery Exergame for the Visually Impaired and Their Sighted Family and FriendsCompanion Proceedings of the 2024 Annual Symposium on Computer-Human Interaction in Play10.1145/3665463.3678829(384-391)Online publication date: 14-Oct-2024
https://dl.acm.org/doi/10.1145/3665463.3678829
Luo SLiu JHu B(2024)Designing a Safe Auditory-Cued Archery Exertion Game for the Visually Impaired and Sighted to Enjoy TogetherProceedings of the 26th International ACM SIGACCESS Conference on Computers and Accessibility10.1145/3663548.3688510(1-6)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3663548.3688510
Show More Cited By

Index Terms

Understanding Personalized Accessibility through Teachable AI: Designing and Evaluating Find My Things for People who are Blind or Low Vision
1. Human-centered computing
2. Social and professional topics
  1. Professional topics
    1. Computing profession
      1. Assistive technologies
  2. User characteristics
    1. People with disabilities

Index terms have been assigned to the content through auto-classification.

Recommendations

Find My Things: Personalized Accessibility through Teachable AI for People who are Blind or Low Vision
CHI EA '24: Extended Abstracts of the CHI Conference on Human Factors in Computing Systems

The opportunity for artificial intelligence, or AI, to enable accessibility is rapidly growing, but widely impactful applications can be challenging to build given the diversity of user need within and across disability communities. Teachable AI systems ...
Misfitting With AI: How Blind People Verify and Contest AI Errors
ASSETS '24: Proceedings of the 26th International ACM SIGACCESS Conference on Computers and Accessibility

Blind people use artificial intelligence-enabled visual assistance technologies (AI VAT) to gain visual access in their everyday lives, but these technologies are embedded with errors that may be difficult to verify non-visually. Previous studies have ...
Drawing and Understanding Diagrams: An Accessible Approach Dedicated to Blind People
Universal Access in Human-Computer Interaction. User and Context Diversity
Abstract
Our research activity focuses on accessibility for document design and consultation, especially in the field of computer science. In this article, we focus on the design and reading of class diagrams according to the UML standard. Our goal is to ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

ASSETS '23: Proceedings of the 25th International ACM SIGACCESS Conference on Computers and Accessibility

October 2023

1163 pages

ISBN:9798400702204

DOI:10.1145/3597638

Copyright © 2023 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGACCESS: ACM Special Interest Group on Accessible Computing

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 22 October 2023

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

ASSETS '23

Sponsor:

SIGACCESS

ASSETS '23: The 25th International ACM SIGACCESS Conference on Computers and Accessibility

October 22 - 25, 2023

NY, New York, USA

Acceptance Rates

ASSETS '23 Paper Acceptance Rate 55 of 182 submissions, 30%;

Overall Acceptance Rate 436 of 1,556 submissions, 28%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

10
Total Citations
View Citations
498
Total Downloads

Downloads (Last 12 months)367
Downloads (Last 6 weeks)52

Reflects downloads up to 13 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Yu RLee SXie JBillah SCarroll J(2024)Human–AI Collaboration for Remote Sighted Assistance: Perspectives from the LLM EraFuture Internet10.3390/fi1607025416:7(254)Online publication date: 18-Jul-2024
https://doi.org/10.3390/fi16070254
Luo SLiu JHu B(2024)Hearing the Bullseye: An Auditory-Cued Archery Exergame for the Visually Impaired and Their Sighted Family and FriendsCompanion Proceedings of the 2024 Annual Symposium on Computer-Human Interaction in Play10.1145/3665463.3678829(384-391)Online publication date: 14-Oct-2024
https://dl.acm.org/doi/10.1145/3665463.3678829
Luo SLiu JHu B(2024)Designing a Safe Auditory-Cued Archery Exertion Game for the Visually Impaired and Sighted to Enjoy TogetherProceedings of the 26th International ACM SIGACCESS Conference on Computers and Accessibility10.1145/3663548.3688510(1-6)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3663548.3688510
Adnin RDas M(2024)"I look at it as the king of knowledge": How Blind People Use and Understand Generative AI ToolsProceedings of the 26th International ACM SIGACCESS Conference on Computers and Accessibility10.1145/3663548.3675631(1-14)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3663548.3675631
Kamikubo RZamiri Zeraati FLee KKacorri H(2024)AccessShare: Co-designing Data Access and Sharing with Blind PeopleProceedings of the 26th International ACM SIGACCESS Conference on Computers and Accessibility10.1145/3663548.3675612(1-16)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3663548.3675612
Herskovitz JXu AAlharbi RGuo A(2024)ProgramAlly: Creating Custom Visual Access Programs via Multi-Modal End-User ProgrammingProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3654777.3676391(1-15)Online publication date: 13-Oct-2024
https://dl.acm.org/doi/10.1145/3654777.3676391
Chang RLiu YGuo A(2024)WorldScribe: Towards Context-Aware Live Visual DescriptionsProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3654777.3676375(1-18)Online publication date: 13-Oct-2024
https://dl.acm.org/doi/10.1145/3654777.3676375
Wang YLi MKim YLee BDanilovich MLazar AConroy DKacorri HChoe E(2024)Redefining Activity Tracking Through Older Adults' Reflections on Meaningful ActivitiesProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642170(1-15)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642170
Massiceti DLongden CSłowik AWills SGrayson MMorrison C(2024)Explaining CLIP's Performance Disparities on Data from Blind/Low Vision Users2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR52733.2024.01157(12172-12182)Online publication date: 16-Jun-2024
https://doi.org/10.1109/CVPR52733.2024.01157
Fisseler B(2024)Trends in Technology for People with Special NeedsComputers Helping People with Special Needs10.1007/978-3-031-62849-8_54(440-448)Online publication date: 8-Jul-2024
https://dl.acm.org/doi/10.1007/978-3-031-62849-8_54

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Table of Contents