[go: up one dir, main page]
More Web Proxy on the site http://driver.im/ skip to main content
research-article

Augmented reality-based video-modeling storybook of nonverbal facial cues for children with autism spectrum disorder to improve their perceptions and judgments of facial expressions and emotions

Published: 01 February 2016 Publication History

Abstract

Autism spectrum disorders (ASD) are characterized by a reduced ability to understand the emotions of other people. Increasing evidence indicates that children with ASD might not recognize or understand crucial nonverbal behaviors, which likely causes them to ignore nonverbal gestures and social cues, like facial expressions, that usually aid social interaction. We used an augmented reality (AR)-based video modeling (VM) storybook (ARVMS) to strengthen and attract the attention of children with ASD to nonverbal social cues because they have difficulty adjusting and switching their attentional focus. In this research, AR has multiple functions: it extends the social features of the story, but it also restricts attention to the most important parts of the videos.Evidence-based research shows that AR attracts the attention of children with ASD. However, few studies have combined AR with VM to train children with ASD to mimic facial expressions and emotions to improve their social skills. In addition, we used markerless natural tracking to teach the children to recognize patterns as they focused on the stable visual image printed in the storybook and then extended their attention to an animation of the story. After the three-phase (baseline, intervention, and maintenance) test data had been collected, the results showed that ARVMS intervention provided an augmented visual indicator which had effectively attracted and maintained the attention of children with ASD to nonverbal social cues and helped them better understand the facial expressions and emotions of the storybook characters. ARVMS increased focus on specific nonverbal cues to prompt social-emotional reciprocity judgments.ARVMS was entertaining and attracted the attention of children with ASD.ARVMS increased virtual visual hints overlaid on the storybook for children with ASD to learn.

References

[1]
American Psychiatric Association, Diagnostic and statistical manual of mental disorders, American Psychiatric Association, Washington, DC, 2000.
[2]
A.L. Apple, F. Billingsley, I.S. Schwartz, Effects of video modeling alone and autism, IEEE Pervasive Computing, 14 (2005) 42-50.
[3]
J.B. Axe, C.J. Evans, Using video modeling to teach children with PDD-NOS to respond to facial expressions, Research in Autism Spectrum Disorders, 6 (2012) 1176-1185.
[4]
K.M. Ayres, J. Langone, Intervention and instruction with video for students with autism: a review of the literature, Education and Training in Developmental Disabilities, 40 (2005) 183-196.
[5]
R.T. Azuma, A'survey of augmented reality, Presence-Teleoperators and Virtual Environments, 6 (1997) 355-385.
[6]
S. Baron-Cohen, M.K. Belmonte, Autism: a window onto the development of the social and the analytic brain, Annual Review of Neuroscience, 28 (2005) 109-126.
[7]
S. Baron-Cohen, A.M. Leslie, U. Frith, Does the autistic child have a "theory of mind"?, Cognition, 21 (1985) 37-46.
[8]
S. Bellini, J. Akullian, A¿meta-analysis of video modeling and video self-modeling interventions for children and adolescents with autism spectrum disorders, Exceptional Children, 73 (2007) 264-287.
[9]
A. Blum-Dimaya, S.A. Reeve, K.F. Reeve, H. Hoch, Teaching children with autism to play a video game using activity schedules and game-embedded simultaneous video modeling, Education and Treatment of Children, 33 (2010) 351-370.
[10]
S. Boelte, J. Hallmayer, Autism spectrum conditions: FAQs on autism, Asperger syndrome, and atypical autism answered by international experts, Hogrefe Publishing, Ashland, OH, 2013.
[11]
L. Bonanni, C.H. Lee, T. Selker, Attention-based design of augmented reality interfaces (Poster), CHI '05 Extended Abstracts on Human Factors in Computing Systems, Portland, OR, USA, 2005.
[12]
T. Cardon, T. Azuma, Deciphering single-subject research design and autism, 2011.
[13]
F. Castelli, Understanding emotions from standardized facial expressions in autism and normal development, Autism, 9 (2005) 428-449.
[14]
H.Y. Chang, H.K. Wu, Y.S. Hsu, Integrating a mobile augmented reality activity to contextualize student learning of a socioscientific issue, British Journal of Educational Technology, 44 (2013) E95-E99.
[15]
M.H. Charlop-Christy, S. Daneshvar, Using video modeling to teach perspective taking to children with autism, Journal of Positive Behavior Interventions, 5 (2003) 12-21.
[16]
M.H. Charlop-Christy, L. Le, K.A. Freeman, A¿comparison of video modeling with in¿vivo modeling for teaching children with autism, Journal of Autism and Developmental Disorders, 30 (2000) 537-552.
[17]
C.H. Chen, C.C. Su, Developing an augmented painting interface for enhancing children painting experience, International Journal of Digital Content Technology and its Applications, 5 (2011) 319-327.
[18]
C.H. Chen, F.G. Wu, S.K. Zhung, Design and augmented reality teaching system with concept mapping technique, in: Proceedings of the 53rd annual conference of JSSD, 2006, pp. 250-251.
[19]
D. Cihak, C. Fahrenkrog, K.M. Ayres, C. Smith, The use of video modeling via a video iPod and a system of least prompts to improve transitional behaviors for students with autism spectrum disorders in the general education classroom, Journal of Positive Behavior Interventions, 12 (2010) 103-115.
[20]
J.N. Constantino, S.A. Davis, R.D. Todd, M.K. Schindler, M.M. Gross, S.L. Brophy, W. Reich, Validation of a brief quantitative measure of autistic traits: comparison of the social responsiveness scale with the autism diagnostic interview-revised, Journal of Autism and Developmental Disorders, 33 (2003) 427-433.
[21]
Á. Di Serio, M.B. Ibáñez, C.D. Kloos, Impact of an augmented reality system on students' motivation for a visual art course, Computers & Education, 68 (2013) 586-596.
[22]
A. Dünser, E. Hornecker, An observational study of children interacting with an augmented story book, in: Proceedings of Edutainment 2007, CUHK, Hong Kong, 2007.
[23]
Durham University News, Children with autism could miss out on non-verbal cues to social interaction, 2013. https://www.dur.ac.uk/news/newsitem/?itemno=18572
[24]
L.M. Elder, L.C. Caterino, J. Chao, D. Shacknai, G. De Simone, The efficacy of social skills treatment for children with Asperger syndrome. Education & treatment of children, West Virginia University Press, 2006.
[25]
M. Elsabbagh, A. Volein, K. Holmboe, L. Tucker, G. Csibra, S. Baron-Cohen, Visual orienting in the early broader autism phenotype: disengagement and facilitation, Journal of Child Psychology and Psychiatry and Allied Disciplines, 50 (2009) 637-642.
[26]
L. Escobedo, M. Tentori, E. Quintana, J. Favela, D. Garcia-Rosas, Using augmented reality to help children with autism stay focused, IEEE Pervasive Computing, 13 (2014) 38-46.
[27]
A. Facoetti, M. Ruffino, A. Peru, P. Paganoni, L. Chelazzi, Sluggish engagement and disengagement of non-spatial attention in dyslexic children, Cortex, 44 (2008) 1221-1233.
[28]
T. Falck-Ytter, S. Bolte, G. Gredeback, Eye tracking in early autism research, Journal of Neurodevelopmental Disorders, 5 (2013) 28.
[29]
T. Grandin, Thinking in pictures, expanded edition: my life with autism, Vintage, New York, 2006.
[30]
J. Hamm, C.G. Kohler, R.C. Gur, R. Verma, Automated facial action coding system for dynamic analysis of facial expressions in neuropsychiatric disorders, Journal of Neuroscience Methods, 200 (2011) 237-256.
[31]
L.V. Ibanez, D.S. Messinger, L. Newell, B. Lambert, M. Sheskin, Visual disengagement in the infant siblings of children with an autism spectrum disorder (ASD), Autism, 12 (2008) 473-485.
[32]
B. Ingersoll, A. Dvortcsak, C. Whalen, D. Sikora, The effects of developmental, social-pragmatic language intervention on rate of expressive language production in young children with autistic spectrum disorders, Focus on Autism and Other Developmental Disabilities, 20 (2005) 213-222.
[33]
B. Ingersoll, A. Dvortcsak, C. Whalen, D. Sikora, The effects of a developmental, social-pragmatic language intervention on rate of expressive language production in young children with autistic spectrum disorders, Focus on Autism and Other Developmental Disabilities, 20 (2005) 213-222.
[34]
C.H. Kennedy, Single case designs for educational research. Chapter 9: A-B-A-B designs, 2005.
[35]
Y. Kikuchi, A. Senju, H. Akechi, Y. Tojo, H. Osanai, T. Hasegawa, Atypical disengagement from faces and its modulation by the control of eye fixation in children with autism spectrum disorder, Journal of Autism and Developmental Disorders, 41 (2011) 629-645.
[36]
K. Koldewyn, S. Weigelt, N. Kanwisher, Y.H. Jiang, Multiple object tracking in autism spectrum disorders, Journal of Autism and Developmental Disorders, 43 (2013) 1394-1405.
[37]
L. Krasny, B.J. Williams, S. Provencal, S. Ozonoff, Social skills interventions for the autism spectrum: essential ingredients and a model curriculum, Child and Adolescent Psychiatric Clinics of North America, 12 (2003) 107-122.
[38]
R. Landry, S.E. Bryson, Impaired disengagement of attention in young children with autism, Journal of Child Psychology and Psychiatry, 45 (2004) 1115-1122.
[39]
L. Maione, P. Mirenda, Effects of video modeling and video feedback on peer-directed social language skills of a child with autism, Journal of Positive Behavior Interventions, 8 (2006) 106-118.
[40]
M.P. Martins, S.L. Harris, Teaching children with autism to respond to joint attention initiations, Child & Family Behavior Therapy, 28 (2006) 51-68.
[41]
R.A. Mason, J.B. Ganz, Seeing is believing: video self-modeling for people with autism and other developmental disabilities, Focus on Autism and Other Developmental Disabilities, 26 (2011).
[42]
J.C. McPartland, S.J. Webb, B. Keehn, G. Dawson, Patterns of visual attention to faces and objects in autism spectrum disorder, Journal of Autism and Developmental Disorders, 41 (2011) 148-157.
[43]
P. Mundy, M. Sigman, J. Ungerer, T. Sherman, Defining the social deficits of autism: the contribution of non-verbal communication measures, Journal of Child Psychology and Psychiatry, 27 (1986) 657-669.
[44]
C.K. Nikopoulos, M. Keenan, Using video modeling to teach complex social sequences to children with autism, Journal of Autism and Developmental Disorders, 37 (2007) 678-693.
[45]
S.L. Odom, L. Collet-Klingenberg, S.J. Rogers, D.D. Hatton, Evidence-based practices in interventions for children and youth with autism spectrum disorders, Preventing School Failure, 54 (2010) 275-282.
[46]
Z. Pan, A.D. Cheok, H. Yang, J. Zhu, J. Shi, Virtual reality and mixed reality for virtual learning environments, Computers & Graphics, 30 (2006) 20-28.
[47]
J. Plavnick, S. Ferreri, Single-case experimental designs in educational research: a methodology for causal analyses in teaching and learning, Educational Psychology Review, 25 (2013) 549-569.
[48]
L.G. Portney, M.P. Watkins, Foundations of clinical research: Applications to practice, Prentice Hall Health, Upper Saddle River, NJ, 2009.
[49]
C. Ryan, C. Ni Charragain, Teaching emotion recognition skills to children with autism, Journal of Autism and Developmental Disorders, 40 (2010) 1505-1511.
[50]
A. Senju, M.H. Johnson, Atypical eye contact in autism: Models, mechanisms and development, Neuroscience and Biobehavioral Reviews, 33 (2009) 1204-1214.
[51]
M. Sherer, K.L. Pierce, S. Paredes, K.L. Kisacky, B. Ingersoll, L. Schreibman, Enhancing conversation skills in children with autism via video technology - which is better, "self" or "other" as a model?, Behavior Modification, 25 (2001) 140-158.
[52]
R. Shipley-Benamou, J.R. Lutzker, M. Taubman, Teaching daily living skills to children with autism through instructional video modeling, Journal of Positive Behavior Interventions, 4 (2002) 165-175.
[53]
S. Siegel, N.J. Castellan, Nonparametric Statistics for the behavioral Sciences, McGraw-Hill, Boston, 1988.
[54]
J. Sigafoos, M. O'Reilly, B. de la Cruz, How to use video modeling and video prompting, PRO-ED, Austin, TX, 2007.
[55]
A. Smith, Cognitive empathy and emotional empathy in human behavior and evolution, Psychological Record, 56 (2006) 3-21.
[56]
E. Sucksmith, C. Allison, S. Baron-Cohen, B. Chakrabarti, R.A. Hoekstra, Empathy and emotion recognition in people with autism, first-degree relatives, and controls, Neuropsychologia, 51 (2013) 98-105.
[57]
M. Tentori, L. Escobedo, G. Balderas, A¿Smart environment for children with autism, IEEE Pervasive Computing, 14 (2015) 42-50.
[58]
A.S. Tetreault, D.C. Lerman, Teaching social skills to children with autism using point-of-view video modeling, Education and Treatment of Children, 33 (2010) 395-419.
[59]
J.N. Van der Geest, C. Kemner, G. Camfferman, M.N. Verbaten, H. van Engeland, Eye movements, visual attention, and autism: a saccadic reaction time study using the gap and overlap paradigm, Biological Psychiatry, 50 (2001) 614-619.
[60]
S.W. White, K. Keonig, L. Scahill, Social skills development in children with autism spectrum disorders: A review of the intervention research, Journal of Autism and Developmental Disorders, 37 (2007) 1858-1868.
[61]
R.L. Young, M. Posselt, Using the transporters DVD as a learning tool for children with autism spectrum disorders (ASD), Journal of Autism and Developmental Disorders, 42 (2012) 984-991.

Cited By

View all
  • (2024)EMooly: Supporting Autistic Children in Collaborative Social-Emotional Learning with Caregiver Participation through Interactive AI-infused and AR ActivitiesProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36997388:4(1-36)Online publication date: 21-Nov-2024
  • (2024)XR technologies to enhance the emotional skills of people with autism spectrum disorderComputers and Graphics10.1016/j.cag.2024.103942121:COnline publication date: 1-Jun-2024
  • (2024)Enabling Technology Integrated Learning for Autistic Children Using Augmented Reality Based Cognitive RehabilitationSN Computer Science10.1007/s42979-023-02495-55:1Online publication date: 5-Jan-2024
  • Show More Cited By

Recommendations

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Computers in Human Behavior
Computers in Human Behavior  Volume 55, Issue PA
February 2016
604 pages

Publisher

Elsevier Science Publishers B. V.

Netherlands

Publication History

Published: 01 February 2016

Author Tags

  1. Augmented reality (AR)
  2. Nonverbal social cues
  3. Video modeling
  4. Visual indicator
  5. Visual supports

Qualifiers

  • Research-article

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)0
  • Downloads (Last 6 weeks)0
Reflects downloads up to 11 Dec 2024

Other Metrics

Citations

Cited By

View all
  • (2024)EMooly: Supporting Autistic Children in Collaborative Social-Emotional Learning with Caregiver Participation through Interactive AI-infused and AR ActivitiesProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36997388:4(1-36)Online publication date: 21-Nov-2024
  • (2024)XR technologies to enhance the emotional skills of people with autism spectrum disorderComputers and Graphics10.1016/j.cag.2024.103942121:COnline publication date: 1-Jun-2024
  • (2024)Enabling Technology Integrated Learning for Autistic Children Using Augmented Reality Based Cognitive RehabilitationSN Computer Science10.1007/s42979-023-02495-55:1Online publication date: 5-Jan-2024
  • (2024)HoloAAC: A Mixed Reality AAC Application for People with Expressive Language DifficultiesVirtual, Augmented and Mixed Reality10.1007/978-3-031-61047-9_20(304-324)Online publication date: 29-Jun-2024
  • (2023)I Am an Earphone and I Can Hear My User’s Face: Facial Landmark Tracking Using Smart EarphonesACM Transactions on Internet of Things10.1145/36144385:1(1-29)Online publication date: 16-Dec-2023
  • (2023)Leveraging Extended Reality for Autistic Individuals: A Scoping Review of Technical Features and Technology AffordancesIEEE Transactions on Learning Technologies10.1109/TLT.2022.319743016:1(133-149)Online publication date: 1-Feb-2023
  • (2023)Designing virtual reality tools for students with Autism Spectrum Disorder: A systematic reviewEducation and Information Technologies10.1007/s10639-022-11545-z28:8(9557-9605)Online publication date: 12-Jan-2023
  • (2022)Recognition of Emotions for People with AutismInternational Journal of Computer Games Technology10.1155/2022/67380682022Online publication date: 15-Jan-2022
  • (2022)Players and Performance: Opportunities for Social Interaction with Augmented Tabletop Games at Centres for Children with AutismProceedings of the ACM on Human-Computer Interaction10.1145/35677166:ISS(161-184)Online publication date: 14-Nov-2022
  • (2022)Designing a Customizable Picture-Based Augmented Reality Application For Therapists and Educational Professionals Working in Autistic ContextsProceedings of the 24th International ACM SIGACCESS Conference on Computers and Accessibility10.1145/3517428.3544884(1-16)Online publication date: 23-Oct-2022
  • Show More Cited By

View Options

View options

Login options

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media