More Web Proxy on the site http://driver.im/

research-article

Open access

Characterizing joint attention behavior during real world interactions using automated object and gaze detection

Authors:

Pranav Venuprasad,

Tu N. M. Nguyen,

Leanne ChukoskieAuthors Info & Claims

ETRA '19: Proceedings of the 11th ACM Symposium on Eye Tracking Research & Applications

Article No.: 21, Pages 1 - 8

https://doi.org/10.1145/3314111.3319843

Published: 25 June 2019 Publication History

Abstract

Joint attention is an essential part of the development process of children, and impairments in joint attention are considered as one of the first symptoms of autism. In this paper, we develop a novel technique to characterize joint attention in real time, by studying the interaction of two human subjects with each other and with multiple objects present in the room. This is done by capturing the subjects' gaze through eye-tracking glasses and detecting their looks on predefined indicator objects. A deep learning network is trained and deployed to detect the objects in the field of vision of the subject by processing the video feed of the world view camera mounted on the eye-tracking glasses. The looking patterns of the subjects are determined and a real-time audio response is provided when a joint attention is detected, i.e., when their looks coincide. Our findings suggest a trade-off between the accuracy measure (Look Positive Predictive Value) and the latency of joint look detection for various system parameters. For more accurate joint look detection, the system has higher latency, and for faster detection, the detection accuracy goes down.

References

[1]

Leanne Chukoskie, Shengyao Guo, Eric Ho, Yalun Zheng, Qiming Chen, Vivian Meng, John Cao, Nikhita Devgan, Si Wu, and Pamela C. Cosman. 2018. Quantifying Gaze Behavior During Real-World Interactions Using Automated Object, Face, and Fixation Detection. IEEE Transactions on Cognitive and Developmental Systems 10, 4 (2018), 1143--1152.

[2]

Jifeng Dai, Yi Li, Kaiming He, and Jian Sun. 2016. R-FCN: Object Detection via Region-based Fully Convolutional Networks. CoRR abs/1605.06409(2016). arXiv:1605.06409 http://arxiv.org/abs/1605.06409

[3]

John M. Franchak, Kari S. Kretch, and Karen E. Adolph. 2014. See and be seen: Infant-caregiver social looking during locomotor free play. Developmental Science 21, 4 (2014), e12626.

[4]

Kelly Stickles Goods, Eric Ishijima, Ya-Chih Chang, and Connie Kasari. 2013. Preschool Based JASPER Intervention in Minimally Verbal Children with Autism: Pilot RCT. Journal of Autism and Developmental Disorders 43, 5 (01 May 2013), 1050--1056.

[5]

Jasmin Kajopoulos, Alvin Hong Yee Wong, Anthony Wong Chen Yuen, Tran Anh Dung, Tan Yeow Kee, and Agnieszka Wykowska. 2015. Robot-Assisted Training of Joint Attention Skills in Children Diagnosed with Autism. In Social Robotics, Adriana Tapus, Elisabeth André, Jean-Claude Martin, François Ferland, and Mehdi Ammi (Eds.). Springer International Publishing, Cham, 296--305.

[6]

Connie Kasari, Marian Sigman, Peter Mundy, and Nurit Yirmiya. 1990. Affective sharing in the context of joint attention interactions of normal, autistic, and mentally retarded children. Journal of Autism and Developmental Disorders 20, 1 (01 Mar 1990), 87--100.

[7]

Moritz Kassner, William Patera, and Andreas Bulling. 2014. Pupil: An Open Source Platform for Pervasive Eye Tracking and Mobile Gaze-based Interaction. In Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct Publication (UbiComp '14 Adjunct). ACM, New York, NY, USA, 1151--1160.

Digital Library

[8]

Ben Kenward, Felix Koch, Linda Forssman, Julia Brehm, Ida Tidemann, Anett Sundqvist, Carin Marciszko, Tone Hermansen, Mikael Heimann, and Gustaf GredebÃd'ck. 2017. Saccadic Reaction Times in Infants and Adults: Spatiotemporal Factors, Gender, and Interlaboratory Variation. Developmental Psychology 53 (12 2017).

[9]

Tsung-Yi Lin, Michael Maire, Serge J. Belongie, Lubomir D. Bourdev, Ross B. Girshick, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. 2014. Microsoft COCO: Common Objects in Context. CoRR abs/1405.0312 (2014). arXiv:1405.0312 http://arxiv.org/abs/1405.0312

[10]

Nikita Manovich, Boris Sekachev, Andrey Zhavoronkov, Victor Salimonov, Dmitry Sidnev, and Sebastian Yonekura. 2018--2019. Computer Vision Annotation Tool (CVAT). https://github.com/opencv/cvat.git.

[11]

Michael Miller, Leanne Chukoskie, Marla Zinni, Jeanne Townsend, and Doris Trauner. 2014. Dyspraxia, motor function and visual-motor integration in autism. Behavioural Brain Research 269 (2014), 95 -- 102.

[12]

Tanya Paparella and Stephanny F N Freeman. 2015. Methods to improve joint attention in young children with autism: a review. Pediatric Health, Medicine and Therapeutics 2015 (05 2015).

[13]

Sarah M. Rabbitt, Alan E. Kazdin, and Brian Scassellati. 2015. Integrating socially assistive robotics into mental healthcare interventions: Applications and recommendations for expanded use. Clinical Psychology Review 35 (2015), 35 -- 46.

[14]

Joseph Redmon. 2013--2016. Darknet: Open Source Neural Networks in C. http://pjreddie.com/darknet/.

[15]

Joseph Redmon and Ali Farhadi. 2018. YOLOv3: An Incremental Improvement. CoRR abs/1804.02767 (2018). arXiv:1804.02767 http://arxiv.org/abs/1804.02767

[16]

Shaoqing Ren, Kaiming He, Ross B. Girshick, and Jian Sun. 2015. Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks. CoRR abs/1506.01497 (2015). arXiv:1506.01497 http://arxiv.org/abs/1506.01497

[17]

Lauren K. Slone, Drew H. Abney, Jeremy I. Borjon, Chi-hsin Chen, John M. Franchak, Daniel Pearcy, Catalina Suarez-Rivera, Tian Linger Xu, Yayun Zhang, Linda B. Smith, and Chen Yu. 2018. Gaze in Action: Head-mounted Eye Tracking of Children's Dynamic Visual Attention During Naturalistic Behavior. J. Vis. Exp. 141 (2018), e58496.

[18]

Sebastian Thrun, Wolfram Burgard, and Dieter Fox. 2005. Probabilistic Robotics (Intelligent Robotics and Autonomous Agents). The MIT Press.

Digital Library

[19]

Jeanne Townsend, Eric Courchesne, and Brian Egaas. 1996. Slowed orienting of covert visual-spatial attention in autism: Specific deficits associated with cerebellar and parietal abnormality. Development and Psychopathology 8, 3 (1996), 563âĂŞ584.

[20]

Chen Yu and Linda B. Smith. 2016. The Social Origins of Sustained Attention in One-Year-Old Human Infants. Current Biology 26, 9 (2016), 1235 -- 1240.

[21]

Lonnie Zwaigenbaum, Susan Bryson, Tracey Rogers, Wendy Roberts, Jessica Brian, and Peter Szatmari. 2005. Behavioral manifestations of autism in the first year of life. International Journal of Developmental Neuroscience 23, 2 (2005), 143 -- 152. Autism: Modeling Human Brain Abnormalities in Developing Animal Systems.

Cited By

Artiran SBedmutha PCosman P(2024)Analysis of Gaze, Head Orientation, and Joint Attention in Autism With Triadic VR InterviewsIEEE Transactions on Neural Systems and Rehabilitation Engineering10.1109/TNSRE.2024.336372832(759-769)Online publication date: 2024
https://doi.org/10.1109/TNSRE.2024.3363728
Boluk NKose H(2024)Evaluating Gaze Detection for Children with Autism Using the ChildPlay-R Dataset2024 IEEE 18th International Conference on Automatic Face and Gesture Recognition (FG)10.1109/FG59268.2024.10581976(1-5)Online publication date: 27-May-2024
https://doi.org/10.1109/FG59268.2024.10581976
Toma MTurcu CTurcu CVlad STiliute DPascu P(2023)Extended Reality-Based Mobile Application Solutions for the Therapy of Children with Autism Spectrum Disorders: A Systematic Literature Review (Preprint)JMIR Serious Games10.2196/49906Online publication date: 13-Jun-2023
https://doi.org/10.2196/49906
Show More Cited By

Index Terms

Characterizing joint attention behavior during real world interactions using automated object and gaze detection
1. Applied computing
  1. Life and medical sciences
    1. Consumer health
2. Human-centered computing
  1. Human computer interaction (HCI)

Recommendations

Analyzing Gaze Behavior Using Object Detection and Unsupervised Clustering
ETRA '20 Full Papers: ACM Symposium on Eye Tracking Research and Applications

Gaze behavior is important in early development, and atypical gaze behavior is among the first symptoms of autism. Here we describe a system that quantitatively assesses gaze behavior using eye-tracking glasses. Objects in the subject’s field of view ...
Detecting Gaze Towards Eyes in Natural Social Interactions and Its Use in Child Assessment

Eye contact is a crucial element of non-verbal communication that signifies interest, attention, and participation in social interactions. As a result, measures of eye contact arise in a variety of applications such as the assessment of the social ...
Gaze-communicative behavior of stuffed-toy robot with joint attention and eye contact based on ambient gaze-tracking
ICMI '07: Proceedings of the 9th international conference on Multimodal interfaces

This paper proposes a gaze-communicative stuffed-toy robot system with joint attention and eye-contact reactions based on ambient gaze-tracking. For free and natural interaction, we adopted our remote gaze-tracking method. Corresponding to the user's ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

ETRA '19: Proceedings of the 11th ACM Symposium on Eye Tracking Research & Applications

June 2019

623 pages

ISBN:9781450367097

DOI:10.1145/3314111

Conference Chairs:
Krzysztof Krejtz
SWPS University, Poland
,
Bonita Sharif
University of Nebraska-Lincoln

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 25 June 2019

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

National Science Foundation

Conference

ETRA '19

Sponsor:

ETRA '19: 2019 Symposium on Eye Tracking Research and Applications

June 25 - 28, 2019

Colorado, Denver

Acceptance Rates

Overall Acceptance Rate 69 of 137 submissions, 50%

Upcoming Conference

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

11
Total Citations
View Citations
728
Total Downloads

Downloads (Last 12 months)203
Downloads (Last 6 weeks)23

Reflects downloads up to 21 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Artiran SBedmutha PCosman P(2024)Analysis of Gaze, Head Orientation, and Joint Attention in Autism With Triadic VR InterviewsIEEE Transactions on Neural Systems and Rehabilitation Engineering10.1109/TNSRE.2024.336372832(759-769)Online publication date: 2024
https://doi.org/10.1109/TNSRE.2024.3363728
Boluk NKose H(2024)Evaluating Gaze Detection for Children with Autism Using the ChildPlay-R Dataset2024 IEEE 18th International Conference on Automatic Face and Gesture Recognition (FG)10.1109/FG59268.2024.10581976(1-5)Online publication date: 27-May-2024
https://doi.org/10.1109/FG59268.2024.10581976
Toma MTurcu CTurcu CVlad STiliute DPascu P(2023)Extended Reality-Based Mobile Application Solutions for the Therapy of Children with Autism Spectrum Disorders: A Systematic Literature Review (Preprint)JMIR Serious Games10.2196/49906Online publication date: 13-Jun-2023
https://doi.org/10.2196/49906
Lei YHe SKhamis MYe J(2023)An End-to-End Review of Gaze Estimation and its Interactive Applications on Handheld Mobile DevicesACM Computing Surveys10.1145/360694756:2(1-38)Online publication date: 15-Sep-2023
https://dl.acm.org/doi/10.1145/3606947
Artiran SBedmutha PLi ACosman P(2023)Gaze and Head Rotation Analysis in a Triadic VR Job Interview Simulation2023 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct)10.1109/ISMAR-Adjunct60411.2023.00083(381-386)Online publication date: 16-Oct-2023
https://doi.org/10.1109/ISMAR-Adjunct60411.2023.00083
Abeysinghe AArachchige ISamarasinghe PDhanawansa VVelayuthan M(2022)Qualitative Analysis of Automated Visual Tracking of Objects Through Head Pose Estimation2022 4th International Conference on Advancements in Computing (ICAC)10.1109/ICAC57685.2022.10025053(369-374)Online publication date: 9-Dec-2022
https://doi.org/10.1109/ICAC57685.2022.10025053
Dhanawansa VSamarasinghe PGardiner BYogarajah PKarunasena A(2022)The Automated Temporal Analysis of Gaze Following in a Visual Tracking TaskImage Analysis and Processing – ICIAP 202210.1007/978-3-031-06433-3_28(324-336)Online publication date: 15-May-2022
https://doi.org/10.1007/978-3-031-06433-3_28
Shi LCopot CVanlanduit S(2021)GazeEMD: Detecting Visual Intention in Gaze-Based Human-Robot InteractionRobotics10.3390/robotics1002006810:2(68)Online publication date: 30-Apr-2021
https://doi.org/10.3390/robotics10020068
Wu CLiaqat SHelvaci HChcung SChuah COzonoff SYoung G(2021)Machine Learning Based Autism Spectrum Disorder Detection from Videos2020 IEEE International Conference on E-health Networking, Application & Services (HEALTHCOM)10.1109/HEALTHCOM49281.2021.9398924(1-6)Online publication date: 1-Mar-2021
https://doi.org/10.1109/HEALTHCOM49281.2021.9398924
Venuprasad PXu LHuang EGilman APh.D. LCosman P(2020)Analyzing Gaze Behavior Using Object Detection and Unsupervised ClusteringACM Symposium on Eye Tracking Research and Applications10.1145/3379155.3391316(1-9)Online publication date: 2-Jun-2020
https://dl.acm.org/doi/10.1145/3379155.3391316
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents