More Web Proxy on the site http://driver.im/

short-paper

Leveraging gaze for potential error prediction in AI-support systems: An exploratory analysis of interaction with a simulated robot

Authors:

Björn Severitt,

Nora Jane Castner,

Olga Lukashova-Sanz,

Siegfried WahlAuthors Info & Claims

ICMI '23 Companion: Companion Publication of the 25th International Conference on Multimodal Interaction

Pages 56 - 60

https://doi.org/10.1145/3610661.3617163

Published: 09 October 2023 Publication History

Abstract

With the increasing prevalence of AI–driven systems that assist humans in daily tasks, ensuring safety and fostering trust is pivotal. Moreover, addressing system errors is important, as conventional methods may be inaccessible and time consuming for people with motor impairments. This study explores implicit communication using eye-tracking features. We conducted a Virtual Reality (VR) experiment where participants used a robotic arm, encountering errors to be prevented through explicit communication. Findings reveal significant differences in eye-tracking features based on error recognition and demonstrate automated error detection feasibility. Using machine learning, we were able to classify when a user intends to stop the system before a potential collision using the eye features (86 – 90% accuracy). This research contributes to understanding eye movement behavior for error detection and offers insights for implementing implicit communication in AI –- driven assistive systems.

References

[1]

H. Abdi and Lynne Williams. 2010. Honestly significant difference (HSD) test. Encyclopedia of Research Design (01 2010), 583–585.

[2]

Sarwar Ali, Shamim Al Mamun, Hisato Fukuda, Antony Lam, Yoshinori Kobayashi, and Yoshonori Kuno. 2018. Smart robotic wheelchair for bus boarding using CNN combined with hough transforms. In Intelligent Computing Methodologies: 14th International Conference, ICIC 2018, Wuhan, China, August 15-18, 2018, Proceedings, Part III 14. Springer, 163–172.

Digital Library

[3]

Nicola C Anderson, Walter F Bischof, Kaitlin EW Laidlaw, Evan F Risko, and Alan Kingstone. 2013. Recurrence quantification analysis of eye movements. Behavior research methods 45 (2013), 842–856.

[4]

Jehangir Arshad, Adan Qaisar, Atta-Ur Rehman, Mustafa Shakir, Muhammad Kamran Nazir, Ateeq Ur Rehman, Elsayed Tag Eldin, Nivin A Ghamry, and Habib Hamam. 2022. Intelligent Control of Robotic Arm Using Brain Computer Interface and Artificial Intelligence. Applied Sciences 12, 21 (2022), 10813.

[5]

Omneya Attallah, Jaidaa Abougharbia, Mohamed Tamazin, and Abdelmonem A Nasser. 2020. A BCI system based on motor imagery for assisting people with motor deficiencies in the limbs. Brain sciences 10, 11 (2020), 864.

[6]

Sean Anthony Byrne, Virmarie Maquiling, Adam Peter Frederick Reynolds, Luca Polonio, Nora Castner, and Enkelejda Kasneci. 2023. Exploring the Effects of Scanpath Feature Engineering for Supervised Image Classification Models. Proceedings of the ACM on Human-Computer Interaction 7, ETRA (2023), 1–18.

Digital Library

[7]

Teng Cao, Feng Wan, Chi Man Wong, Janir Nuno da Cruz, and Yong Hu. 2014. Objective evaluation of fatigue by EEG spectral analysis in steady-state visual evoked potential-based brain-computer interfaces. Biomedical engineering online 13, 1 (2014), 1–13.

[8]

Nora Castner, Thomas C Kuebler, Katharina Scheiter, Juliane Richter, Thérése Eder, Fabian Hüttig, Constanze Keutel, and Enkelejda Kasneci. 2020. Deep semantic gaze embedding and scanpath comparison for expertise classification during OPT viewing. In ACM symposium on eye tracking research and applications. 1–10.

Digital Library

[9]

Mary E Frame, Rik Warren, and Anna M Maresca. 2019. Scanpath comparisons for complex visual search in a naturalistic environment. Behavior Research Methods 51 (2019), 1454–1470.

[10]

Wolfgang Fuhl, Efe Bozkir, Benedikt Hosp, Nora Castner, David Geisler, Thiago C Santini, and Enkelejda Kasneci. 2019. Encodji: encoding gaze data into emoji space for an amusing scanpath classification approach. In Proceedings of the 11th ACM Symposium on Eye Tracking Research & Applications. 1–4.

Digital Library

[11]

Tin Kam Ho. 1995. Random decision forests. In Proceedings of 3rd international conference on document analysis and recognition, Vol. 1. IEEE, 278–282.

[12]

Kenneth Holmqvist, Saga Lee Örbom, Ignace TC Hooge, Diederick C Niehorster, Robert G Alexander, Richard Andersson, Jeroen S Benjamins, Pieter Blignaut, Anne-Marie Brouwer, Lewis L Chuang, 2023. Eye tracking: empirical foundations for a minimal reporting guideline. Behavior research methods 55, 1 (2023), 364–416.

[13]

Rafael Irizarry and Michael Love. [n. d.]. Multiple testing. http://genomicsclass.github.io/book/pages/multiple_testing.html Accessed: 2023-06-30.

[14]

Robert JK Jacob and Keith S Karn. 2003. Eye tracking in human-computer interaction and usability research: Ready to deliver the promises. In The mind’s eye. Elsevier, 573–605.

[15]

Sunil Jacob, Mukil Alagirisamy, Chen Xi, Venki Balasubramanian, Ram Srinivasan, R Parvathi, NZ Jhanjhi, and Sardar MN Islam. 2021. AI and IoT-enabled smart exoskeleton system for rehabilitation of paralyzed people in connected communities. IEEE Access 9 (2021), 80340–80350.

[16]

Min Hun Lee, Daniel P Siewiorek, Asim Smailagic, Alexandre Bernardino, and Sergi Bermúdez i Badia. 2022. Enabling AI and robotic coaches for physical rehabilitation therapy: iterative design and evaluation with therapists and post-stroke survivors. International Journal of Social Robotics (2022), 1–22.

[17]

Olga Lukashova-Sanz, Rajat Agarwala, and Siegfried Wahl. 2022. Context matters during pick-and-place in VR: Impact on search and transport phases. Frontiers in Psychology 13 (2022), 881269.

[18]

Marius Matulis and Carlo Harvey. 2021. A robot arm digital twin utilising reinforcement learning. Computers & Graphics 95 (2021), 106–114.

[19]

Boucetta Lakhdar Nadjib, Chemloul Bilal, and Rebai Karima. 2021. EMG-Based Hand gesture recognition for myoelectric prosthetic hand control. In 2021 International Conference on Artificial Intelligence for Cyber Security Systems and Privacy (AI-CSP). IEEE, 1–6.

[20]

Nafiseh Ghaffar Nia, Ahad Nasab, and Erkan Kaplanoglu. 2022. Reinforcement Learning-Based Grasp Pattern Control of Upper Limb Prosthetics in an AI Platform. In 2022 3rd International Informatics and Software Engineering Conference (IISEC). IEEE, 1–4.

[21]

Yashas Rai and Patrick Le Callet. 2018. Chapter 3 - Visual attention, visual salience, and perceived interest in multimedia applications. In Academic Press Library in Signal Processing, Volume 6, Rama Chellappa and Sergios Theodoridis (Eds.). Academic Press, 113–161. https://doi.org/10.1016/B978-0-12-811889-4.00003-8

[22]

Yuki Sakai, Huimin Lu, Joo-Kooi Tan, and Hyoungseop Kim. 2019. Recognition of surrounding environment from electric wheelchair videos based on modified YOLOv2. Future Generation Computer Systems 92 (2019), 157–161.

Digital Library

[23]

Dario D. Salvucci and Joseph H. Goldberg. 2000. Identifying Fixations and Saccades in Eye-Tracking Protocols. In Proceedings of the 2000 Symposium on Eye Tracking Research & Applications (Palm Beach Gardens, Florida, USA) (ETRA ’00). Association for Computing Machinery, New York, NY, USA, 71–78. https://doi.org/10.1145/355017.355028

Digital Library

[24]

Javier San Agustin, Henrik Skovsgaard, John Paulin Hansen, and Dan Witzner Hansen. 2009. Low-cost gaze interaction: ready to deliver the promises. In CHI’09 Extended Abstracts on Human Factors in Computing Systems. 4453–4458.

[25]

Alexandra Sipatchin, Siegfried Wahl, and Katharina Rifai. 2021. Eye-Tracking for Clinical Ophthalmology with Virtual Reality (VR): A Case Study of the HTC Vive Pro Eye’s Usability. Healthcare 9, 2 (2021). https://doi.org/10.3390/healthcare9020180

[26]

Yvonne Tran, Ashley Craig, Rachel Craig, Rifai Chai, and Hung Nguyen. 2020. The influence of mental fatigue on brain activity: Evidence from a systematic review with meta-analyses. Psychophysiology 57, 5 (2020), e13554.

[27]

Ker-Jiun Wang, Quanbo Liu, Yifan Zhao, Caroline Yan Zheng, Soumya Vhasure, Quanfeng Liu, Prakash Thakur, Mingui Sun, and Zhi-Hong Mao. 2018. Intelligent wearable virtual reality (VR) gaming controller for people with motor disabilities. In 2018 IEEE International Conference on Artificial Intelligence and Virtual Reality (AIVR). IEEE, 161–164.

[28]

Daniel Weber, Enkelejda Kasneci, and Andreas Zell. 2022. Exploiting augmented reality for extrinsic robot calibration and eye-based human-robot collaboration. In 2022 17th ACM/IEEE International Conference on Human-Robot Interaction (HRI). IEEE, 284–293.

Digital Library

[29]

Daniel Weber, Thiago Santini, Andreas Zell, and Enkelejda Kasneci. 2020. Distilling location proposals of unknown objects through gaze information for human-robot interaction. In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 11086–11093.

Digital Library

[30]

Feng Wen, Zixuan Zhang, Tianyiyi He, and Chengkuo Lee. 2021. AI enabled sign language recognition and VR space bidirectional communication using triboelectric smart glove. Nature communications 12, 1 (2021), 5378.

[31]

Walid Zgallai, John Teye Brown, Afnan Ibrahim, Fatma Mahmood, Khulood Mohammad, Maitha Khalfan, Maryam Mohammed, Maryam Salem, and Noof Hamood. 2019. Deep learning AI application to an EEG driven BCI smart wheelchair. In 2019 Advances in Science and Engineering Technology International Conferences (ASET). IEEE, 1–5.

[32]

Chang Zhu, Quan Liu, Wei Meng, Qingsong Ai, and Sheng Q Xie. 2021. An attention-based cnn-lstm model with limb synergy for joint angles prediction. In 2021 IEEE/ASME International Conference on Advanced Intelligent Mechatronics (AIM). IEEE, 747–752.

Digital Library

Cited By

Severitt BLenhart PHosp BCastner NWahl S(2024)Communication breakdown: Gaze-based prediction of system error for AI-assisted robotic arm simulated in VRProceedings of the 2024 Symposium on Eye Tracking Research and Applications10.1145/3649902.3653339(1-7)Online publication date: 4-Jun-2024
https://dl.acm.org/doi/10.1145/3649902.3653339

Index Terms

Leveraging gaze for potential error prediction in AI-support systems: An exploratory analysis of interaction with a simulated robot
1. Human-centered computing
  1. Accessibility
    1. Accessibility technologies
  2. Human computer interaction (HCI)
    1. Interaction paradigms
      1. Virtual reality

Recommendations

Head and gaze control of a telepresence robot with an HMD
ETRA '18: Proceedings of the 2018 ACM Symposium on Eye Tracking Research & Applications

Gaze interaction with telerobots is a new opportunity for wheelchair users with severe motor disabilities. We present a video showing how head-mounted displays (HMD) with gaze tracking can be used to monitor a robot that carries a 360° video camera and ...
Bicycles and Wheelchairs for Locomotion Control of a Simulated Telerobot Supported by Gaze- and Head-Interaction
PETRA '18: Proceedings of the 11th PErvasive Technologies Related to Assistive Environments Conference

We present an interface for control of a telerobot that supports field-of-view panning, mode selections and keyboard typing by head- and gaze-interaction. The utility of the interface was tested by 19 able-bodied participants controlling a virtual ...
Snap, Pursuit and Gain: Virtual Reality Viewport Control by Gaze
CHI '24: Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems

Head-mounted displays let users explore virtual environments through a viewport that is coupled with head movement. In this work, we investigate gaze as an alternative modality for viewport control, enabling exploration of virtual worlds with less head ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

ICMI '23 Companion: Companion Publication of the 25th International Conference on Multimodal Interaction

October 2023

434 pages

ISBN:9798400703218

DOI:10.1145/3610661

Editors:
Elisabeth André
University of Augsburg
,
Mohamed Chetouani
Sorbonne University
,
Dominique Vaufreydaz
Univ. Grenoble Alpes
,
Gale Lucas
USC Institute for Creative Technologies
,
Tanja Schultz
University of Bremen
,
Louis-Philippe Morency
Carnegie Mellon University
,
Alessandro Vinciarelli
University of Glasgow

Copyright © 2023 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 09 October 2023

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Short-paper
Research
Refereed limited

Funding Sources

European Union's Horizon 2020 research and innovation program
Robust Vision: Inference Principles and Neural Mechanisms, TP TRA, project

Conference

ICMI '23

Sponsor:

SIGCHI

ICMI '23: INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION

October 9 - 13, 2023

Paris, France

Acceptance Rates

Overall Acceptance Rate 453 of 1,080 submissions, 42%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1
Total Citations
View Citations
76
Total Downloads

Downloads (Last 12 months)47
Downloads (Last 6 weeks)3

Reflects downloads up to 19 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Severitt BLenhart PHosp BCastner NWahl S(2024)Communication breakdown: Gaze-based prediction of system error for AI-assisted robotic arm simulated in VRProceedings of the 2024 Symposium on Eye Tracking Research and Applications10.1145/3649902.3653339(1-7)Online publication date: 4-Jun-2024
https://dl.acm.org/doi/10.1145/3649902.3653339

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Table of Contents