[go: up one dir, main page]
More Web Proxy on the site http://driver.im/ skip to main content
research-article
Open access

SuperpowerGlass: A Wearable Aid for the At-Home Therapy of Children with Autism

Published: 11 September 2017 Publication History

Abstract

We have developed a system for automatic facial expression recognition running on Google Glass, delivering real-time social cues to children with Autism Spectrum Disorder (ASD). The system includes multiple mechanisms to engage children and their parents, who administer this technology within the home. We completed an at-home design trial with 14 families that used the learning aid over a 3-month period. We found that children with ASD generally respond well to wearing the system at home and opt for the most expressive feedback choice. We further evaluated app usage, facial engagement, and model accuracy. We found that the device can act as a powerful training aid when used periodically in the home, that interactive video content from wearable therapy sessions should be augmented with sufficient context about the content to produce long-term engagement, and that the design of wearable systems for children with ASD should be heavily dependent on the functioning level of the child. We contribute general design implications for developing wearable aids used by children with ASD and other behavioral disorders as well as their parents during at-home parent-administered therapy sessions.

Supplementary Material

washington (washington.zip)
Supplemental movie, appendix, image and software files for, SuperpowerGlass: A Wearable Aid for the At-Home Therapy of Children with Autism

References

[1]
Alex A. Ahmed and Matthew S. Goodwin. Automated Detection of Facial Expressions during Computer-Assisted Instruction in Individuals on the Autism Spectrum. 2017. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI ’17), 6050-6055.
[2]
Clare Allen, Natalie Silove, Katrina Williams, and Paul Hutchins. 2007. Validity of the social communication questionnaire in assessing risk of autism in preschool children with developmental problems. Journal of Autism and Developmental Disorders 37,7: 1272--1278.
[3]
Alberto Battocchi, Fabio Pianesi, Daniel Tomasini, Massimo Zancanaro, Gianluca Esposito, Paola Venuti, A. Ben Sasson, Eynat Gal, and Patrice L. Weiss. 2009. Collaborative Puzzle Game: a tabletop interactive game for fostering collaboration in children with Autism Spectrum Disorders (ASD). In Proceedings of the ACM International Conference on Interactive Tabletops and Surfaces (ITS ‘09), 197-204.
[4]
Kyle D. Bennett, Rangasamy Ramasamy, and Toby Honsberger. The effects of covert audio coaching on teaching clerical skills to adolescents with autism spectrum disorder. 2013. Journal of Autism and Developmental Disorders 43,3: 585-593.
[5]
Centers for Disease Control and Prevention and Centers for Disease Control and Prevention. 2014. Atlanta, GA: National Center on Birth Defects and Developmental Disabilities. Retrieved September 20, 2016 from http://www.cdc.gov/media/releases/2014/p0327-autism-spectrum-disorder.html.
[6]
Chris J. Boyatzis and Reenu Varghese. 1994. Children’s emotional associations with colors. The Journal of genetic psychology 155,1: 77-85.
[7]
LouAnne E. Boyd, Alejandro Rangel, Helen Tomimbang, Andrea Conejo-Toledo, Kanika Patel, Monica Tentori, and Gillian R. Hayes. SayWAT: Augmenting Face-to-Face Conversations for Adults with Autism. 2016 In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI ’16), 4872-4883.
[8]
LouAnne E. Boyd, Xinlong Jiang, and Gillian R. Hayes. ProCom: Designing and Evaluating a Mobile and Wearable System to Support Proximity Awareness for People with Autism. 2017. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI ’17), 2865-2877.
[9]
Navneet Dalal and Bill Triggs. 2005. Histograms of oriented gradients for human detection. In 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR ‘05), 886-893.
[10]
Geraldine Dawson and Karen Burner. 2011. Behavioral interventions in children and adolescents with autism spectrum disorder: a review of recent findings. Current opinion in pediatrics 23,6: 616-620.
[11]
Geraldine Dawson, Sally Rogers, Jeffrey Munson, Milani Smith, Jamie Winter, Jessica Greenson, Amy Donaldson, and Jennifer Varley. 2010. Randomized, controlled trial of an intervention for toddlers with autism: the Early Start Denver Model. Pediatrics 125,1: e17-e23.
[12]
Geraldine Dawson, Sara Jane Webb, and James McPartland. Understanding the nature of face processing impairment in autism: insights from behavioral and electrophysiological studies. Developmental neuropsychology 27,3: 403-424.
[13]
Paul Ekman. 1992. An Argument for Basic Emotions. Cognition and Emotion 6,3-4: 169-200.
[14]
Rana el Kaliouby, Rosalind Picard, and Simon Baron-Cohen. 2006. Affective computing and autism. Annals of the New York Academy of Sciences 1093,1: 228-248.
[15]
Lizbeth Escobedo, David H. Nguyen, LouAnne Boyd, Sen Hirano, Alejandro Rangel, Daniel Garcia-Rosas, Monica Tentori, and Gillian Hayes. 2012. MOSOCO: a mobile assistive tool to support children with autism practicing social skills in real-life situations. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ‘12), 2589-2598.
[16]
Lizbeth Escobedo, Monica Tentori, Eduardo Quintana, Jesus Favela, and Daniel Garcia-Rosas. 2014. Using augmented reality to help children with autism stay focused. IEEE Pervasive Computing 13,1: 38-46.
[17]
David Feil-Seifer and Maja J. Matarić. 2009. Toward socially assistive robotics for augmenting interventions for children with autism spectrum disorders. In Experimental robotics: 201-210.
[18]
Jenny Rose Finkel and Christopher D. Manning. 2009. Hierarchical bayesian domain adaptation. In Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics (pp. 602-610). Association for Computational Linguistics.
[19]
Eynat Gal, Nirit Bauminger, Dina Goren-Bar, Fabio Pianesi, Oliviero Stock, Massimo Zancanaro, and Patrice L. Tamar Weiss. 2009. Enhancing social communication of children with high-functioning autism through a co-located interface. Ai 8 Society 24,1: 75-84.
[20]
Leonardo Giusti, Massimo Zancanaro, Eynat Gal, and Patrice L. Tamar Weiss. 2011. Dimensions of collaboration on a tabletop interface for children with autism spectrum disorder. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ‘11), 3295-3304.
[21]
Ofer Golan, Emma Ashwin, Yael Granader, Suzy McClintock, Kate Day, Victoria Leggett, and Simon Baron-Cohen. 2010. Enhancing emotion recognition in children with autism spectrum conditions: An intervention using animated vehicles with real emotional faces. Journal of autism and developmental disorders 40,3: 269-279.
[22]
Nick Haber, Catalin Voss, Azar Fazel, Terry Winograd, and Dennis Wall. A practical approach to real-time neutral subtraction for facial expression recognition. IEEE Winter Conference on Applications of Computer Vision (WACV), 2016.
[23]
Gillian R. Hayes, Julie A. Kientz, Khai N. Truong, David R. White, Gregory D. Abowd, and Trevor Pering. 2004. Designing capture applications to support the education of children with autism. In International Conference on Ubiquitous Computing (Ubicomp ‘14), 161-178.
[24]
Gillian R. Hayes, Sen Hirano, Gabriela Marcu, Mohamad Monibi, David H. Nguyen, and Michael Yeganyan. 2010. Interactive visual supports for children with autism. Personal and ubiquitous computing 14,7: 663-680.
[25]
Javier Hernandez and Rosalind W. Picard. 2014. SenseGlass: using google glass to sense daily emotions. In Proceedings of the adjunct publication of the 27th annual ACM symposium on User interface software and technology (UIST ‘14), 77-78.
[26]
Sen H. Hirano, Michael T. Yeganyan, Gabriela Marcu, David H. Nguyen, Lou Anne Boyd, and Gillian R. Hayes. 2010. vSked: evaluation of a system to support classroom activities for children with autism. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ‘10), 1633-1642.
[27]
Patricia Howlin, Susan Goode, Jane Hutton, and Michael Rutter. 2004. Adult outcome for children with autism.” Journal of Child Psychology and Psychiatry 45,2: 212-229.
[28]
Warren Jones and Ami Klin. 2013. Attention to eyes is present but in decline in 2-6-month-old infants later diagnosed with autism. Nature 504, 7480: 427-431.
[29]
Michelle R. Kandalaft, Nyaz Didehbani, Daniel C. Krawczyk, Tandra T. Allen, and Sandra B. Chapman. 2013. Virtual reality social cognition training for young adults with high-functioning autism. Journal of autism and developmental disorders 43,1: 34-44.
[30]
Naz Kaya and Helen H. Epps. 2004. Color-emotion associations: Past experience and personal preference. In Proceedings of Color and Paints, Interim Meeting of the International Color Association 5,31.
[31]
Marion Koelle, Matthias Kranz, and Andreas Möller. 2015. Don’t look at me that way!: Understanding User Attitudes Towards Data Glasses Usage. In Proceedings of the International Conference on Human-Computer Interaction with Mobile Devices and Services (MobileHCI ’15), 362-372.
[32]
Rebecca J. Landa, Katherine C. Holman, and Elizabeth Garrett-Mayer. 2007. Social and communication development in toddlers with early and later diagnosis of autism spectrum disorders. Archives of general psychiatry 64,7: 853-864.
[33]
Rebecca J. Landa, Katherine C. Holman, Allison H. O’Neill, and Elizabeth A. Stuart. 2011. Intervention targeting development of socially synchronous engagement in toddlers with autism spectrum disorder: a randomized controlled trial. Journal of Child Psychology and Psychiatry 52,1: 13-21.
[34]
Ole I. Lovaas. Behavioral treatment and normal educational and intellectual functioning in young autistic children. 1987. Journal of consulting and clinical psychology 55,1: 3.
[35]
Patrick Lucey, Jeffrey F. Cohn, Takeo Kanade, Jason Saragih, Zara Ambadar, and Iain Matthews. 2010. The extended cohn-kanade dataset (ck+): A complete dataset for action unit and emotion-specified expression. In 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition-Workshops (CVPR ‘10), 94-101.
[36]
Michael Lyons, Shigeru Akamatsu, Miyuki Kamachi, and Jiro Gyoba. 1998. Coding facial expressions with gabor wavelets. In Proceedings of the third IEEE International Conference on Automatic Face and Gesture Recognition (FG ‘98), 200-205.
[37]
Gabriela Marcu, Anind K. Dey, and Sara Kiesler. 2012. Parent-driven use of wearable cameras for autism support: a field study with families. In Proceedings of the 2012 ACM Conference on Ubiquitous Computing (Ubicomp ‘12), 401-410.
[38]
James C. McPartland, Brian Reichow, and Fred R. Volkmar. 2012. Sensitivity and specificity of proposed DSM-5 diagnostic criteria for autism spectrum disorder. Journal of the American Academy of Child 8 Adolescent Psychiatry 51,4: 368-383.
[39]
David Moore, Yufang Cheng, Paul McGrath, and Norman J. Powell. 2005. Collaborative virtual environment technology for people with autism. Focus on Autism and Other Developmental Disabilities 20,4: 231-243.
[40]
Brenda Smith Myles, Winnie Dunn, Louann Rinner, Taku Hagiwara, Matthew Reese, Abby Huggins, and Stephanie Becker. 2004. Sensory issues in children with Asperger syndrome and autism. Education and training in developmental disabilities: 283-290.
[41]
Letizia Palumbo, Hollie G. Burnett, and Tjeerd Jellema. 2015. Atypical emotional anticipation in high-functioning autism. Molecular autism 6,1: 1.
[42]
Maja Pantic, Michel Valstar, Ron Rademaker, and Ludo Maat. 2005. Web-based database for facial expression analysis. In 2005 IEEE international conference on multimedia and Expo.
[43]
Ji Hyun Park, Bretagne Abirached, and Yan Zhang. 2012. A framework for designing assistive technologies for teaching children with ASDs emotions. In CHI’ 12 Extended Abstracts on Human Factors in Computing Systems (CHI ‘12), 2423-2428).
[44]
Rosalind W. Picard. 2010. Affective computing: from laughter to IEEE. IEEE Transactions on Affective Computing 1,1: 11-17.
[45]
Kaska Porayska-Pomsta, Christopher Frauenberger, Helen Pain, Gnanathusharan Rajendran, Tim Smith, Rachel Menzies, Mary Ellen Foster, Alyssa M Alcorn, S Wass, S Bernadini, Katerina Avramides, Wendy Keay-Bright, Jingying Chen, A Waller, Karen Guldberg, Judith Good, and Oliver Lemon. Developing technology for autism: an interdisciplinary approach. Personal and Ubiquitous Computing 16,2 (2012): 117-127.
[46]
Halley Profita, Reem Albaghli, Leah Findlater, Paul Jaeger, and Shaun K. Kane. 2016. The AT Effect: How Disability Affects the Perceived Social Acceptability of Head-Mounted Display Use. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI ‘16), 4884-4895.
[47]
Gale H. Roid. 2003. Stanford--Binet intelligence scales (5th ed.): Technical manual. Itasca, IL: Riverside Publishing.
[48]
Noah J. Sasson, Amy E. Pinkham, Lauren P. Weittenhiller, Daniel J. Faso, and Claire Simpson. 2015. Context effects on facial affect recognition in schizophrenia and autism: behavioral and eye-tracking evidence. Schizophrenia bulletin: sbv176.
[49]
Arman Savran, Neşe Alyüz, Hamdi Dibeklioğlu, Oya Çeliktutan, Berk Gökberk, Bülent Sankur, and Lale Akarun. 2008. Bosphorus database for 3D face analysis. In European Workshop on Biometrics and Identity Management: 47-56.
[50]
Brian Scassellati, Henny Admoni, and Maja Mataric. 2012. Robots for use in autism research. Annual review of biomedical engineering 14: 275-294.
[51]
Terence Sim, Simon Baker, and Maan Bsat. 2002. The CMU pose, illumination, and expression (PIE) database. In Proceedings of the Fifth IEEE International Conference on Automatic Face and Gesture Recognition (FG ‘02), 46-51.
[52]
Katharina Spiel, Christopher Frauenberger, Eva Hornecker, and Geraldine Fitzpatrick. When Empathy Is Not Enough: Assessing the Experiences of Autistic Children with Technologies. 2017. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI ’17), 2853-2864.
[53]
Katharina Spiel, Laura Malinverni, Judith Good, and Christopher Frauenberger. Participatory Evaluation with Autistic Children. 2017. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI ’17), 5755-5766.
[54]
Xiaoyang Tan and Bill Triggs. 2010. Enhanced local texture feature sets for face recognition under difficult lighting conditions. IEEE transactions on image processing 19,6: 1635-1650.
[55]
Alea C. Teeters. 2007. Use of a wearable camera system in conversation: Toward a companion tool for social-emotional learning in autism. Doctoral dissertation, Massachusetts Institute of Technology, Cambridge, MA.
[56]
Patricia Valdez and Albert Mehrabian. 1994. Effects of color on emotions. Journal of experimental psychology: General 123,4: 394.
[57]
Catalin Voss, Peter Washington, Nick Haber, Aaron Kline, Jena Daniels, Azar Fazel, Titas De, Beth McCarthy, Carl Feinstein, Terry Winograd, and Dennis Wall. 2016. Superpower glass: delivering unobtrusive real-time social cues in wearable systems. In Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct (Ubicomp ‘16), 1218-1226.
[58]
Peter Washington, Catalin Voss, Nick Haber, Serena Tanaka, Jena Daniels, Carl Feinstein, Terry Winograd, and Dennis Wall. 2016. A Wearable Social Interaction Aid for Children with Autism. In Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI ‘16), 2348-2354.
[59]
Kristin Williams, Karyn Moffatt, Denise McCall, and Leah Findlater. 2015. Designing conversation cues on a head-worn display to support persons with aphasia. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI ‘15), 231-240.
[60]
Jean Xavier, Violaine Vignaud, Rosa Ruggiero, Nicolas Bodeau, David Cohen, and Laurence Chaby. 2015. A Multidimensional Approach to the Study of Emotion Recognition in Autism Spectrum Disorders. Frontiers in psychology 6.
[61]
Michael T. Yeganyan, Meg Cramer, Lou Anne Boyd, and Gillian R. Hayes. 2010. vSked: an interactive visual schedule system for use in classrooms for children with autism. In Proceedings of the 9th International Conference on Interaction Design and Children (IDC ‘10), 319-322.
[62]
Lijun Yin, Xiaochen Chen, Yi Sun, Tony Worm, and Michael Reale. 2008. A high-resolution 3D dynamic facial expression database. In 8th Proceedings on the IEEE International Conference On Automatic Face 8 Gesture Recognition (FG’08), 1-6).

Cited By

View all
  • (2024)Personalization of Affective Models Using Classical Machine Learning: A Feasibility StudyApplied Sciences10.3390/app1404133714:4(1337)Online publication date: 6-Feb-2024
  • (2024)Audio-Based Emotion Recognition Using Self-Supervised Learning on an Engineered Feature SpaceAI10.3390/ai50100115:1(195-207)Online publication date: 17-Jan-2024
  • (2024)Using #ActuallyAutistic on Twitter for Precision Diagnosis of Autism Spectrum Disorder: Machine Learning StudyJMIR Formative Research10.2196/526608(e52660)Online publication date: 14-Feb-2024
  • Show More Cited By

Index Terms

  1. SuperpowerGlass: A Wearable Aid for the At-Home Therapy of Children with Autism

    Recommendations

    Comments

    Please enable JavaScript to view thecomments powered by Disqus.

    Information & Contributors

    Information

    Published In

    cover image Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies
    Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies  Volume 1, Issue 3
    September 2017
    2023 pages
    EISSN:2474-9567
    DOI:10.1145/3139486
    Issue’s Table of Contents
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 11 September 2017
    Accepted: 01 June 2017
    Revised: 01 May 2017
    Received: 01 February 2017
    Published in IMWUT Volume 1, Issue 3

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. Autism
    2. Behavior Therapy
    3. Wearable Computing

    Qualifiers

    • Research-article
    • Research
    • Refereed

    Funding Sources

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)525
    • Downloads (Last 6 weeks)46
    Reflects downloads up to 12 Dec 2024

    Other Metrics

    Citations

    Cited By

    View all
    • (2024)Personalization of Affective Models Using Classical Machine Learning: A Feasibility StudyApplied Sciences10.3390/app1404133714:4(1337)Online publication date: 6-Feb-2024
    • (2024)Audio-Based Emotion Recognition Using Self-Supervised Learning on an Engineered Feature SpaceAI10.3390/ai50100115:1(195-207)Online publication date: 17-Jan-2024
    • (2024)Using #ActuallyAutistic on Twitter for Precision Diagnosis of Autism Spectrum Disorder: Machine Learning StudyJMIR Formative Research10.2196/526608(e52660)Online publication date: 14-Feb-2024
    • (2024)Digitally Diagnosing Multiple Developmental Delays Using Crowdsourcing Fused With Machine Learning: Protocol for a Human-in-the-Loop Machine Learning StudyJMIR Research Protocols10.2196/5220513(e52205)Online publication date: 8-Feb-2024
    • (2024)A Comparison of Personalized and Generalized Approaches to Emotion Recognition Using Consumer Wearable Devices: Machine Learning StudyJMIR AI10.2196/521713(e52171)Online publication date: 10-May-2024
    • (2024)Playlogue: Dataset and Benchmarks for Analyzing Adult-Child Conversations During PlayProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36997758:4(1-34)Online publication date: 21-Nov-2024
    • (2024)EMooly: Supporting Autistic Children in Collaborative Social-Emotional Learning with Caregiver Participation through Interactive AI-infused and AR ActivitiesProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36997388:4(1-36)Online publication date: 21-Nov-2024
    • (2024)Body-Based Augmented Reality Feedback During ConversationsProceedings of the ACM on Human-Computer Interaction10.1145/36764918:MHCI(1-22)Online publication date: 24-Sep-2024
    • (2024)TwIPS: A Large Language Model Powered Texting Application to Simplify Conversational Nuances for Autistic UsersProceedings of the 26th International ACM SIGACCESS Conference on Computers and Accessibility10.1145/3663548.3675633(1-18)Online publication date: 27-Oct-2024
    • (2024)CookAR: Affordance Augmentations in Wearable AR to Support Kitchen Tool Interactions for People with Low VisionProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3654777.3676449(1-16)Online publication date: 13-Oct-2024
    • Show More Cited By

    View Options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Login options

    Full Access

    Media

    Figures

    Other

    Tables

    Share

    Share

    Share this Publication link

    Share on social media