More Web Proxy on the site http://driver.im/

research-article

Inclusive Multimodal Voice Interaction for Code Navigation

Authors:

Bharat Paudyal,

Maite Frutos-PascualAuthors Info & Claims

ICMI '22: Proceedings of the 2022 International Conference on Multimodal Interaction

Pages 509 - 519

https://doi.org/10.1145/3536221.3556600

Published: 07 November 2022 Publication History

Abstract

Navigation of source code typically requires extensive use of a traditional mouse and keyboard which can present significant barriers for developers with physical impairments. We present research exploring how commonly used code navigation approaches (e.g. locating references to user-defined identifiers, jumping to function definitions, conducting a search for specific syntax, etc.) can be optimized for multimodal voice interaction. An exploratory study was initially conducted with five developers who have physical impairments to elicit insights around their experiences in navigating code within existing voice-controlled development environments. Findings from this study informed the design of a code editor integrating different navigation features tailored for multimodal speech input. A user evaluation with 14 developers with physical impairments was conducted with results demonstrating that all participants were able to successfully complete a series of standard navigation tasks. Participants also highlighted that the code navigation techniques were intuitive to use and provided a high-level of usability.

References

[1]

Khaled Albusays, Stephanie Ludi, and Matt Huenerfauth. 2017. Interviews and Observation of Blind Software Developers at Work to Understand Code Navigation Challenges. In Proceedings of the 19th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS ’17), Association for Computing Machinery, New York, NY, USA, 91–100.

Digital Library

[2]

Vinay Augustine, Patrick Francis, Xiao Qu, David Shepherd, Will Snipes, Christoph Braunlich, and Thomas Fritz. 2015. A Field Study on Fostering Structural Navigation with Prodet. In 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering, 229–238.

[3]

Catherine M. Baker, Lauren R. Milne, and Richard E. Ladner. 2015. Struct jumper: A tool to help blind programmers navigate and understand the structure of code. In Conference on Human Factors in Computing Systems - Proceedings, Association for Computing Machinery, New York, New York, USA, 3043–3052.

Digital Library

[4]

Aaron Bangor, Philip Kortum, and James Miller. 2009. Determining what individual SUS scores mean: adding an adjective rating scale. Journal of Usability Studies 4, 3 (2009), 114–123.

Digital Library

[5]

Andrew Begel and Susan L. Graham. 2005. Spoken programs. In Proceedings - 2005 IEEE Symposium on Visual Languages and Human-Centric Computing, IEEE, 99–106.

Digital Library

[6]

Andrew Begel and Susan L Graham. 2006. An assessment of a speech-based programming environment. In Proceedings - IEEE Symposium on Visual Languages and Human-Centric Computing, VL/HCC 2006, 116–120.

Digital Library

[7]

Brackets. 2018. Brackets - A modern, open source code editor that understands web design. Retrieved from http://brackets.io/

[8]

Andrew Bragdon, Robert Zeleznik, Steven P Reiss, Suman Karumuri, William Cheung, Joshua Kaplan, Christopher Coleman, Ferdi Adeputra, and Joseph J. LaViola. 2010. Code bubbles. In Proceedings of the 28th international conference on Human factors in computing systems - CHI ’10 (ICSE ’10), ACM, New York, NY, USA, 2503–2503.

Digital Library

[9]

Andrew Bragdon, Robert Zeleznik, Steven P. Reiss, Suman Karumuri, William Cheung, Joshua Kaplan, Christopher Coleman, Ferdi Adeputra, and Joseph J. LaViola. 2010. Code bubbles: a working set-based interface for code understanding and maintenance. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’10), Association for Computing Machinery, New York, NY, USA, 2503–2512.

Digital Library

[10]

Hashmeet Chadha, Satyam Mhatre, Unnati Ganatra, and Sujata Pathak. 2018. HTML Voice. In Proceedings - 2018 4th International Conference on Computing, Communication Control and Automation, ICCUBEA 2018, IEEE, 1–4.

[11]

Harendra Chhekur. 2022. Colon IDE. Retrieved May 11, 2022 from https://github.com/Chhekur/colon-ide

[12]

Chris Creed, Ian Williams, and Maite Frutos-Pascual. 2019. Multimodal Gaze Interaction for Creative Design. (2019).

Digital Library

[13]

Robert DeLine and Kael Rowan. 2010. Code canvas. In Proceedings of the 32nd ACM/IEEE International Conference on Software Engineering - ICSE ’10 (ICSE ’10), ACM, New York, NY, USA, 207–207.

Digital Library

[14]

Hartmut Glücker, Felix Raab, Florian Echtler, and Christian Wolff. 2014. EyeDE: Gaze-enhanced software development environments. In Conference on Human Factors in Computing Systems - Proceedings (CHI EA ’14), ACM, New York, NY, USA, 1555–1560.

Digital Library

[15]

Quinn Hanam, Fernando S.De M. Brito, and Ali Mesbah. 2016. Discovering bug patterns in Javascript. In Proceedings of the ACM SIGSOFT Symposium on the Foundations of Software Engineering, Association for Computing Machinery, 144–156.

Digital Library

[16]

Austin Z Henley and Scott D Fleming. 2014. The patchworks code editor: toward faster navigation with less code arranging and fewer navigation mistakes. In Proc. CHI (CHI ’14), ACM, New York, NY, USA, 2511–2520.

Digital Library

[17]

Pranav Joshi and Doina Bein. 2020. Audible Code, a Voice-Enabled Programming Extension of Visual Studio Code. In Advances in Intelligent Systems and Computing, Springer, 335–341.

[18]

Hyunhoon Jung, Seongeun So, Changhoon Oh, Hee Jae Kim, and Jinjoong Kim. 2019. TurtleTalk: An educational programming game for children with voice user interface. In Conference on Human Factors in Computing Systems - Proceedings, Association for Computing Machinery, New York, NY, USA, 1–6.

Digital Library

[19]

Thorsten Karrer, Jan Peter Krämer, Jonathan Diehl, Björn Hartmann, and Borchers Jan. 2011. Stacksplorer: Call graph navigation helps increasing code maintenance efficiency. In UIST’11 - Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology (UIST ’11), ACM, New York, NY, USA, 217–224.

Digital Library

[20]

Yea Seul Kim, Mira Dontcheva, Eytan Adar, and Jessica Hullman. 2019. Vocal shortcuts for creative experts. In Conference on Human Factors in Computing Systems - Proceedings, ACM.

Digital Library

[21]

Benjamin P. Klein and Austin Z. Henley. 2021. CodeRibbon: More Efficient Workspace Management and Navigation for Mainstream Development Environments. In 2021 IEEE International Conference on Software Maintenance and Evolution (ICSME), 604–608.

[22]

Andrew J. Ko, Brad A. Myers, Michael J. Coblenz, and Htet Htet Aung. 2006. An Exploratory Study of How Developers Seek, Relate, and Collect Relevant Information during Software Maintenance Tasks. IEEE Trans. Softw. Eng. 32, 12 (December 2006), 971–987.

Digital Library

[23]

Jan-Peter Krämer, Thorsten Karrer, Joachim Kurz, Moritz Wittenhagen, and Jan Borchers. 2013. How tools in IDEs shape developers’ navigation behavior. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems - CHI ’13 (2013), 3073–3073.

Digital Library

[24]

Jan-Peter Krämer, Joachim Kurz, Thorsten Karrer, and Jan Borchers. 2012. Blaze: Supporting Two-phased Call Graph Navigation in Source Code. In CHI ’12 Extended Abstracts on Human Factors in Computing Systems (CHI EA ’12), ACM, New York, NY, USA, 2195–2200.

Digital Library

[25]

Thomas D Latoza, Gina Venolia, and Robert Deline. 2006. Maintaining Mental Models: A Study of Developer Work Habits. Retrieved from http://delivery.acm.org/10.1145/1140000/1134355/p492-latoza.pdf?ip=134.225.30.51&id=1134355&acc=ACTIVE SERVICE&key=3B621857D79A46E2.3B621857D79A46E2.7F82F319CD1C5BCF.4D4702B0C3E38B35&__acm__=1567591177_fde2357a954c9ec7f294e6bc257d8b64

[26]

Yun Young Lee, Sam Harwell, Sarfraz Khurshid, and Darko Marinov. 2013. Temporal code completion and navigation. In 2013 35th International Conference on Software Engineering (ICSE), 1181–1184.

[27]

Rinor S. Maloku and Besart Xh. Pllana. 2016. HyperCode: Voice aided programming. IFAC-PapersOnLine 49, 29 (January 2016), 263–268.

[28]

Frolin Ocariza, Kartik Bajaj, Karthik Pattabiraman, and Ali Mesbah. 2013. An empirical study of client-side JavaScript bugs. In International Symposium on Empirical Software Engineering and Measurement, 55–64.

[29]

Bharat Paudyal, Chris Creed, Maite Frutos-Pascual, and Ian Williams. 2020. Voiceye: A multimodal inclusive development environment. In DIS 2020 - Proceedings of the 2020 ACM Designing Interactive Systems Conference, 21–33.

Digital Library

[30]

David J. Piorkowski, Scott D. Fleming, Irwin Kwan, Margaret M. Burnett, Christopher Scaffidi, Rachel K.E. Bellamy, and Joshua Jordahl. 2013. The whats and hows of programmers’ foraging diets. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’13), Association for Computing Machinery, New York, NY, USA, 3063–3072.

Digital Library

[31]

Stevche Radevski, Hideaki Hata, and Kenichi Matsumoto. 2016. EyeNav. In Proceedings of the 9th Nordic Conference on Human-Computer Interaction - NordiCHI ’16 (NordiCHI ’16), ACM, New York, NY, USA, 1–4.

Digital Library

[32]

Lucas Rosenblatt, Patrick Carrington, Kotaro Hara, and Jeffrey P Bigham. 2018. Vocal Programming for People with Upper-Body Motor Impairments. In W4A, 10–10.

Digital Library

[33]

Korok Sengupta, Sabin Bhattarai, Sayan Sarcar, I. Scott MacKenzie, and Steffen Staab. 2020. Leveraging Error Correction in Voice-based Text Entry by Talk-and-Gaze. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, (25/04/20 - 30/04/20), 1–11.

Digital Library

[34]

Asma Shakil, Christof Lutteroth, and Gerald Weber. 2019. CodeGazer: Making Code Navigation Easy and Natural With Gaze Input. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI ’19), Association for Computing Machinery, New York, NY, USA, 1–12.

Digital Library

[35]

David Shepherd, Kostadin Damevski, Bartosz Ropski, and Thomas Fritz. 2012. Sando: an extensible local code search framework. In Proceedings of the ACM SIGSOFT 20th International Symposium on the Foundations of Software Engineering (FSE ’12), Association for Computing Machinery, New York, NY, USA, 1–2.

Digital Library

[36]

J. Smith, C. Brown, and E. Murphy-Hill. 2017. Flower: Navigating program flow in the IDE. In 2017 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC), 19–23.

[37]

S. Stumpf, A. Peters, S. Bardzell, M. Burnett, D. Busse, J. Cauchard, and E. Churchill. 2020. Gender-Inclusive HCI Research and Design: A Conceptual Review. Foundations and Trends in Human–Computer Interaction 13, 1 (March 2020), 1–69.

Digital Library

[38]

A. Teckchandani. 2018. Slack: A Unified Communications Platform to Improve Team Collaboration. AMLE 17, 2 (June 2018), 226–228.

[39]

Tobias Kahlert and Kay Giza. 2016. Visual Studio Code - Code Editing. Redefined. Mikrosoft 1, March (2016), 1–26.

[40]

Jessica Van Brummelen, Kevin Weng, Phoebe Lin, and Catherine Yeo. 2020. CONVO: What does conversational programming need? In Proceedings of IEEE Symposium on Visual Languages and Human-Centric Computing, VL/HCC, Institute of Electrical and Electronics Engineers (IEEE), 1–5.

[41]

Amber Wagner and Jeff Gray. 2015. An Empirical Evaluation of a Vocal User Interface for Programming by Voice. International Journal of Information Technologies and Systems Approach 8, 2 (2015), 47–63.

[42]

2018. Talon 0.0.7.7 documentation. Retrieved from https://talonvoice.com/docs/index.html#document-index

[43]

Cursorless - Visual Studio Marketplace. Retrieved March 26, 2022 from https://marketplace.visualstudio.com/items?itemName=pokey.cursorless

[44]

Serenade | Documentation. Retrieved from https://serenade.ai/docs/

[45]

Dragon Speech Recognition - Get More Done by Voice | Nuance. Retrieved from https://www.nuance.com/dragon.html

[46]

Monaco Editor. Retrieved March 26, 2022 from https://microsoft.github.io/monaco-editor/

[47]

Using the Web Speech API - Web APIs | MDN. Retrieved from https://developer.mozilla.org/en-US/docs/Web/API/Web_Speech_API/Using_the_Web_Speech_API

[48]

Wit.ai. Retrieved from https://wit.ai/

[49]

Atom. Retrieved from https://atom.io/

[50]

Video Conferencing, Web Conferencing, Webinars, Screen Sharing - Zoom. Retrieved from https://zoom.us/

[51]

Microsoft Teams | Group Chat, Team Chat & Collaboration. Retrieved from https://www.microsoft.com/en-GB/microsoft-teams/group-chat-software

Cited By

da Rocha Tde Souza CTeran LMota M(2024)Effective Inclusion of People with Disabilities in Software Development TeamsProceedings of the 18th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement10.1145/3674805.3690749(447-453)Online publication date: 24-Oct-2024
https://dl.acm.org/doi/10.1145/3674805.3690749
Creed CSarcar S(2024)Voice coding experiences for developers with physical impairmentsJournal of Enabling Technologies10.1108/JET-02-2024-0021Online publication date: 22-Oct-2024
https://doi.org/10.1108/JET-02-2024-0021

Recommendations

Voice Snapping: Inclusive Speech Interaction Techniques for Creative Object Manipulation
DIS '22: Proceedings of the 2022 ACM Designing Interactive Systems Conference

Voice input holds significant potential to support people with physical impairments in producing creative visual design outputs, although it is unclear whether well-established interaction methods used for manipulating graphical assets within mainstream ...
Voiceye: A Multimodal Inclusive Development Environment
DIS '20: Proceedings of the 2020 ACM Designing Interactive Systems Conference

People with physical impairments who are unable to use traditional input devices (i.e. mouse and keyboard) are often excluded from technical professions (e.g. web development). Alternative input methods such as eye gaze tracking and speech recognition ...
Interviews and Observation of Blind Software Developers at Work to Understand Code Navigation Challenges
ASSETS '17: Proceedings of the 19th International ACM SIGACCESS Conference on Computers and Accessibility

Integrated Development Environments (IDEs) play an important role in the workflow of many software developers, e.g. providing syntactic highlighting or other navigation aids to support the creation of lengthy codebases. Unfortunately, such complex ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

ICMI '22: Proceedings of the 2022 International Conference on Multimodal Interaction

November 2022

830 pages

ISBN:9781450393904

DOI:10.1145/3536221

Editors:
Raj Tumuluri
Openstream
,
Nicu Sebe
University of Trento
,
Gopal Pingali
Accenture
,
Dinesh Babu Jayagopi
IIIT Bangalore
,
Abhinav Dhall
IIT Ropar
,
Richa Singh
IIT Jodhpur
,
Lisa Anthony
University of Florida
,
Albert Ali Salah
Utrecht University and Boğaziçi University

Copyright © 2022 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 07 November 2022

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

ICMI '22

Sponsor:

SIGCHI

ICMI '22: INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION

November 7 - 11, 2022

Bengaluru, India

Acceptance Rates

Overall Acceptance Rate 321 of 785 submissions, 41%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
159
Total Downloads

Downloads (Last 12 months)45
Downloads (Last 6 weeks)3

Reflects downloads up to 12 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

da Rocha Tde Souza CTeran LMota M(2024)Effective Inclusion of People with Disabilities in Software Development TeamsProceedings of the 18th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement10.1145/3674805.3690749(447-453)Online publication date: 24-Oct-2024
https://dl.acm.org/doi/10.1145/3674805.3690749
Creed CSarcar S(2024)Voice coding experiences for developers with physical impairmentsJournal of Enabling Technologies10.1108/JET-02-2024-0021Online publication date: 22-Oct-2024
https://doi.org/10.1108/JET-02-2024-0021

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Table of Contents