[go: up one dir, main page]
More Web Proxy on the site http://driver.im/ skip to main content
10.1145/3290605.3300657acmconferencesArticle/Chapter ViewAbstractPublication PageschiConference Proceedingsconference-collections
research-article

Detecting Visuo-Haptic Mismatches in Virtual Reality using the Prediction Error Negativity of Event-Related Brain Potentials

Published: 02 May 2019 Publication History

Abstract

Designing immersion is the key challenge in virtual reality; this challenge has driven advancements in displays, rendering and recently, haptics. To increase our sense of physical immersion, for instance, vibrotactile gloves render the sense of touching, while electrical muscle stimulation (EMS) renders forces. Unfortunately, the established metric to assess the effectiveness of haptic devices relies on the user's subjective interpretation of unspecific, yet standardized, questions.
Here, we explore a new approach to detect a conflict in visuo-haptic integration (e.g., inadequate haptic feedback based on poorly configured collision detection) using electroencephalography (EEG). We propose analyzing event-related potentials (ERPs) during interaction with virtual objects. In our study, participants touched virtual objects in three conditions and received either no haptic feedback, vibration, or vibration and EMS feedback. To provoke a brain response in unrealistic VR interaction, we also presented the feedback prematurely in 25% of the trials.
We found that the early negativity component of the ERP (so called prediction error) was more pronounced in the mismatch trials, indicating we successfully detected haptic conflicts using our technique. Our results are a first step towards using ERPs to automatically detect visuo-haptic mismatches in VR, such as those that can cause a loss of the user's immersion.

References

[1]
S. Allin, Y. Matsuoka, and R. Klatzky. 2002. Measuring just noticeable differences for haptic force feedback: implications for rehabilitation. In Proceedings 10th Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems. HAPTICS 2002. 299--302.
[2]
Mortaja AlQassab, Adam Gomes, Maria Karam, David Wang, Zhechen Du, Orion Bruckman, and Richard Bustos. 2016. The Impact of Tactile Sensations on Virtual Reality Impairment. In Universal Access in Human-Computer Interaction. Interaction Techniques and Environments (Lecture Notes in Computer Science), Margherita Antona and Constantine Stephanidis (Eds.). Springer International Publishing, 231--240.
[3]
Yazmir B. 2018. Neural Correlates of User-initiated Motor Success and Failure - A Brain-Computer Interface Perspective. Neuroscience (2018).
[4]
Christopher C. Berger, Mar Gonzalez-Franco, Eyal Ofek, and Ken Hinckley. 2018. The uncanny valley of haptics. Science Robotics (2018).
[5]
Chris Berka, Daniel J. Levendowski, Michelle N. Lumicao, Alan Yau, Gene Davis, Vladimir T. Zivkovic, Richard E. Olmstead, Patrice D. Tremoulet, and Patrick L. Craven. 2007. EEG correlates of task engagement and mental workload in vigilance, learning, and memory tasks. Aviation, Space, and Environmental Medicine 78, 5 Suppl (May 2007), B231--244.
[6]
Benjamin Blankertz, Steven Lemm, Matthias Treder, Stefan Haufe, and Klaus-Robert Mueller. 2011. Single-trial analysis and classification of ERP components -- A tutorial. NeuroImage 56, 2 (May 2011), 814--825.
[7]
Christoph W. Borst and Richard A. Volz. 2005. Evaluation of a Haptic Mixed Reality System for Interactions with a Virtual Control Panel. Presence: Teleoperators and Virtual Environments 14 (2005), 677--696. 7https://intheon.io, last accessed 17/09/2018
[8]
Frederick P. Brooks. 1999. What's Real About Virtual Reality? IEEE Comput. Graph. Appl. 19, 6 (Nov. 1999), 16--27.
[9]
Andres (Andres Alejandre) Calvo. 2017. A body-grounded kinesthetic haptic device for virtual reality. Thesis. Massachusetts Institute of Technology. http://dspace.mit.edu/handle/1721.1/112394
[10]
R. Chavarriaga, Z. Khaliliardali, L. Gheorghe, I. Iturrate, and J. D.R. Millán. 2015. EEG-based decoding of error-related brain activity in a real-world driving task. Journal of Neural Engineering (2015).
[11]
Andy Clark. 2013. Whatever next? Predictive brains, situated agents, and the future of cognitive science. Behavioral and Brain Sciences 36, 3 (June 2013), 181--204.
[12]
Michael G.H Coles, Marten K Scheffers, and Clay B Holroyd. 2001. Why is there an ERN/Ne on correct trials? Response representations, stimulus-related components, and the theory of error-processing. Biological Psychology 56, 3 (June 2001), 173--189.
[13]
Emanuel Donchin and Michael G.H. Coles. 1988. Is the P300 component a manifestation of context updating? Behavioral and Brain Sciences (1988). arXiv:1511.04103
[14]
Katharina Emmerich and Maic Masuch. 2016. The Influence of Virtual Agents on Player Experience and Performance. In Proceedings of the 2016 Annual Symposium on Computer-Human Interaction in Play (CHI PLAY '16). ACM, New York, NY, USA, 10--21.
[15]
G. GarcÃa-Valle, M. Ferre, J. BreÃosa, and D. Vargas. 2018. Evaluation of Presence in Virtual Environments: Haptic Vest and User's Haptic Skills. IEEE Access 6 (2018), 7224--7233.
[16]
Ceenu George, Manuel Demmler, and Heinrich Hussmann. 2018. Intelligent Interruptions for IVR: Investigating the Interplay Between Presence, Workload and Attention. In Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems (CHI EA '18). ACM, New York, NY, USA, Article LBW511, 6 pages.
[17]
Mar Gonzalez-Franco and Jaron Lanier. 2017. Model of Illusions and Virtual Reality. Frontiers in Psychology 8 (2017), 1125.
[18]
Klaus Gramann, Friederike U Hohlefeld, Lukas Gehrke, and Marius Klug. 2018. Heading computation in the human retrosplenial complex during full-body rotation. bioRxiv (2018). arXiv:https://www.biorxiv.org/content/early/2018/09/14/417972.full.pdf
[19]
Xiaochi Gu, Yifei Zhang, Weize Sun, Yuanzhe Bian, Dao Zhou, and Per Ola Kristensson. 2016. Dexmo: An Inexpensive and Lightweight Mechanical Exoskeleton for Motion Capture and Force Feedback in VR. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16). ACM, New York, NY, USA, 1991--1995.
[20]
Clay B. Holroyd, Kaivon L. Pakzad-Vaezi, and Olave E. Krigolson. 2008. The feedback correct-related positivity: Sensitivity of the event-related brain potential to unexpected positive feedback. Psychophysiology 45, 5 (Sept. 2008), 688--697.
[21]
Dhruv Jain, Misha Sra, Jingru Guo, Rodrigo Marques, Raymond Wu, Justin Chiu, and Chris Schmandt. 2016. Immersive Scuba Diving Simulator Using Virtual Reality. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology (UIST '16). ACM, New York, NY, USA, 729--739.
[22]
Hwan Kim, Minhwan Kim, and Woohun Lee. 2016. HapThimble: A Wearable Haptic Device Towards Usable Virtual Touch Screen. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16). ACM, New York, NY, USA, 3694--3705.
[23]
A. Lehmann, C. Geiger, B. Woldecke, and J. Stocklein. 2009. Design and evaluation of 3D content with wind output. In 2009 IEEE Symposium on 3D User Interfaces. 151--152.
[24]
Yu Liao, Klaus Gramann, Wenfeng Feng, Gedeon O. Deák, and Hong Li. 2011. This ought to be good: Brain activity accompanying positive and negative expectations and outcomes. Psychophysiology (2011).
[25]
Pedro Lopes and Patrick Baudisch. 2013. Muscle-propelled Force Feedback: Bringing Force Feedback to Mobile Devices. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '13). ACM, New York, NY, USA, 2577--2580.
[26]
Pedro Lopes, Alexandra Ion, and Patrick Baudisch. 2015. Impacto: Simulating Physical Impact by Combining Tactile Stimulation with Electrical Muscle Stimulation. In Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology (UIST '15). ACM, New York, NY, USA, 11--19.
[27]
Pedro Lopes, Sijing You, Lung-Pan Cheng, Sebastian Marwecki, and Patrick Baudisch. 2017. Providing Haptics to Walls & Heavy Objects in Virtual Reality by Means of Electrical Muscle Stimulation. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI '17). ACM, New York, NY, USA, 1471--1482.
[28]
Pedro Lopes, Sijing You, Alexandra Ion, and Patrick Baudisch. 2018. Adding Force Feedback to Mixed Reality Experiences and Games Using Electrical Muscle Stimulation. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI '18). ACM, New York, NY, USA, Article 446, 13 pages.
[29]
Steven J. Luck. 2014. An Introduction to the Event-Related Potential Technique (2nd edition edition ed.). A Bradford Book, Cambridge, Massachusetts.
[30]
Masahiro Mori. 1970. The uncanny valley. Energy 7, 4 (1970), 33--35.
[31]
Vadim V. Nikulin, Jewgeni Kegeles, and Gabriel Curio. 2010. Miniaturized electroencephalographic scalp electrode for optimal wearing comfort. Clinical Neurophysiology: Official Journal of the International Federation of Clinical Neurophysiology 121, 7 (July 2010), 1007--1014.
[32]
Paul L. Nunez and Ramesh Srinivasan. 2006. Electric Fields of the Brain: The Neurophysics of EEG (2nd ed edition ed.). Oxford University Press, Oxford ; New York.
[33]
Robert Oostenveld and Peter Praamstra. 2001. The five percent electrode system for high-resolution EEG and ERP measurements. Clinical Neurophysiology 112, 4 (April 2001), 713--719.
[34]
Gonçalo Padrao, Mar Gonzalez-Franco, Maria V. Sanchez-Vives, Mel Slater, and Antoni Rodriguez-Fornells. 2016. Violating body movement semantics: Neural signatures of self-generated and external-generated errors. NeuroImage (2016). arXiv:1003.4074
[35]
Jason Palmer, Ken Kreutz-Delgado, and Scott Makeig. 2011. AMICA: An Adaptive Mixture of Independent Component Analyzers with Shared Components. (01 2011).
[36]
John Polich. 2007. Updating P300: An Integrative Theory of P3a and P3b. Clinical neurophysiology : official journal of the International Federation of Clinical Neurophysiology 118, 10 (Oct. 2007), 2128--2148.
[37]
Helena Pongrac. 2008. Vibrotactile perception: examining the coding of vibrations and the just noticeable difference under various conditions. Multimedia Systems 13, 4 (Jan. 2008), 297--307.
[38]
Priscilla Ramsamy, Adrian Haffegee, Ronan Jamieson, and Vassil Alexandrov. 2006. Using Haptics to Improve Immersion in Virtual Environments. In Computational Science -- ICCS 2006 (Lecture Notes in Computer Science), Vassil N. Alexandrov, Geert Dick van Albada, Peter M. A. Sloot, and Jack Dongarra (Eds.). Springer Berlin Heidelberg, 603--609.
[39]
H. Reckter, C. Geiger, J. Singer, and S. Streuber. 2009. Iterative design and test of a multimodal experience. In 2009 IEEE Symposium on 3D User Interfaces. 99--102.
[40]
Andres F. Salazar-Gomez, Joseph Delpreto, Stephanie Gil, Frank H. Guenther, and Daniela Rus. 2017. Correcting robot mistakes in real time using EEG signals. In Proceedings - IEEE International Conference on Robotics and Automation.
[41]
F.-A. Savoie, F. ThÃnault, K. Whittingstall, and P.-M. Bernier. 2018. Visuomotor Prediction Errors Modulate EEG Activity Over Parietal Cortex. Scientific Reports 8, 1 (Aug. 2018), 12513.
[42]
Thomas Schubert, Frank Friedmann, and Holger Regenbrecht. 2001. The Experience of Presence: Factor Analytic Insights. Presence: Teleoperators and Virtual Environments 10, 3 (June 2001), 266--281.
[43]
Avinash Singh, Tim Chen, Yu-Feng Cheng, Jung-Tai King, Li-Wei Ko, Klaus Gramann, and Chin-Teng Lin. 2018. Visual Appearance Modulates Prediction Error in Virtual Reality. IEEE Access PP (May 2018), 1--1.
[44]
Mel Slater. 1999. Measuring Presence: A Response to the Witmer and Singer Presence Questionnaire. Presence: Teleoperators and Virtual Environments 8, 5 (Oct. 1999), 560--565.
[45]
Mel Slater and Martin Usoh. 1993. Representations Systems, Perceptual Position, and Presence in Immersive Virtual Environments. Presence 2 (Jan. 1993), 221--233.
[46]
Melissa K. Stern and James H. Johnson. 2010. Just Noticeable Difference. In The Corsini Encyclopedia of Psychology. American Cancer Society, 1--2.
[47]
Yijun Wang and Tzyy ping Jung. 2013. Improving Brain-Computer Interfaces Using Independent Component Analysis. In In Towards Practical Brain-Computer Interfaces. 67--83.
[48]
Jan R. Wiersema, Jacob J. van der Meere, and Herbert Roeyers. 2007. Developmental changes in error monitoring: An event-related potential study. Neuropsychologia (2007).
[49]
Frank Wilcoxon. 1945. Individual Comparisons by Ranking Methods. Biometrics Bulletin 1, 6 (1945), 80--83. http://www.jstor.org/stable/
[50]
Bob G. Witmer and Michael J. Singer. 1998. Measuring Presence in Virtual Environments: A Presence Questionnaire. Presence: Teleoper. Virtual Environ. 7, 3 (June 1998), 225--240.
[51]
Yasuyoshi Yokokohji, Ralph L. Hollis, and Takeo Kanade. 1996. "What You can See Is What You can Feel." - Development of a Visual/Haptic Interface to Virtual Environment. In In Proceedings of the IEEE 1996 Virtual Reality Annual International Symposium. 46--53.
[52]
Beste F. Yuksel, Kurt B. Oleson, Lane Harrison, Evan M. Peck, Daniel Afergan, Remco Chang, and Robert JK Jacob. 2016. Learn Piano with BACh: An Adaptive Learning Interface That Adjusts Task Difficulty Based on Brain State. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16). ACM, New York, NY, USA, 5372--5384.
[53]
Thorsten O. Zander, Lena M. Andreessen, Angela Berg, Maurice Bleuel, Juliane Pawlitzki, Lars Zawallich, Laurens R. Krol, and Klaus Gramann. 2017. Evaluation of a Dry EEG System for Application of Passive BrainComputer Interfaces in Autonomous Driving. Frontiers in Human Neuroscience 11 (Feb. 2017).
[54]
Thorsten O. Zander, Christian Kothe, Sabine Jatzev, and Matti Gaertner. 2010. Enhancing Human-Computer Interaction with Input from Active and Passive Brain-Computer Interfaces. In Brain-Computer Interfaces, Desney S. Tan and Anton Nijholt (Eds.). Springer London, London, 181--199.
[55]
Thorsten O. Zander, Laurens R. Krol, Niels P. Birbaumer, and Klaus Gramann. 2016. Neuroadaptive technology enables implicit cursor control based on medial prefrontal cortex activity. Proceedings of the National Academy of Sciences 113, 52 (Dec. 2016), 14898--14903.
[56]
Yuhang Zhao, Cynthia L. Bennett, Hrvoje Benko, Edward Cutrell, Christian Holz, Meredith Ringel Morris, and Mike Sinclair. 2018. Enabling People with Visual Impairments to Navigate Virtual Reality with a Haptic and Auditory Cane Simulation. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI '18). ACM, New York, NY, USA, Article 116, 14 pages.

Cited By

View all
  • (2024)An Examination of Ultrasound Mid-air Haptics for Enhanced Material and Temperature Perception in Virtual EnvironmentsProceedings of the ACM on Human-Computer Interaction10.1145/36764888:MHCI(1-21)Online publication date: 24-Sep-2024
  • (2024)Decoding Realism of Virtual Objects: Exploring Behavioral and Ocular Reactions to Inaccurate Interaction FeedbackACM Transactions on Computer-Human Interaction10.1145/366034531:3(1-21)Online publication date: 18-Apr-2024
  • (2024)Predicting the Limits: Tailoring Unnoticeable Hand Redirection Offsets in Virtual Reality to Individuals' Perceptual BoundariesProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3654777.3676425(1-13)Online publication date: 13-Oct-2024
  • Show More Cited By

Index Terms

  1. Detecting Visuo-Haptic Mismatches in Virtual Reality using the Prediction Error Negativity of Event-Related Brain Potentials

      Recommendations

      Comments

      Please enable JavaScript to view thecomments powered by Disqus.

      Information & Contributors

      Information

      Published In

      cover image ACM Conferences
      CHI '19: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems
      May 2019
      9077 pages
      ISBN:9781450359702
      DOI:10.1145/3290605
      Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

      Sponsors

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      Published: 02 May 2019

      Permissions

      Request permissions for this article.

      Check for updates

      Author Tags

      1. eeg
      2. elecrical muscle stimulation
      3. erp
      4. force feedback
      5. prediction error
      6. virtual reality

      Qualifiers

      • Research-article

      Conference

      CHI '19
      Sponsor:

      Acceptance Rates

      CHI '19 Paper Acceptance Rate 703 of 2,958 submissions, 24%;
      Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

      Upcoming Conference

      CHI 2025
      ACM CHI Conference on Human Factors in Computing Systems
      April 26 - May 1, 2025
      Yokohama , Japan

      Contributors

      Other Metrics

      Bibliometrics & Citations

      Bibliometrics

      Article Metrics

      • Downloads (Last 12 months)201
      • Downloads (Last 6 weeks)10
      Reflects downloads up to 03 Mar 2025

      Other Metrics

      Citations

      Cited By

      View all
      • (2024)An Examination of Ultrasound Mid-air Haptics for Enhanced Material and Temperature Perception in Virtual EnvironmentsProceedings of the ACM on Human-Computer Interaction10.1145/36764888:MHCI(1-21)Online publication date: 24-Sep-2024
      • (2024)Decoding Realism of Virtual Objects: Exploring Behavioral and Ocular Reactions to Inaccurate Interaction FeedbackACM Transactions on Computer-Human Interaction10.1145/366034531:3(1-21)Online publication date: 18-Apr-2024
      • (2024)Predicting the Limits: Tailoring Unnoticeable Hand Redirection Offsets in Virtual Reality to Individuals' Perceptual BoundariesProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3654777.3676425(1-13)Online publication date: 13-Oct-2024
      • (2024)Exposing Movement Correlates of Presence Experience in Virtual Reality Using Parametric Maps2024 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW)10.1109/VRW62533.2024.00073(374-380)Online publication date: 16-Mar-2024
      • (2024)Exploring Depth-based Perception Conflicts in Virtual Reality through Error-Related Potentials2024 IEEE Conference Virtual Reality and 3D User Interfaces (VR)10.1109/VR58804.2024.00097(774-784)Online publication date: 16-Mar-2024
      • (2024)Modeling the Intent to Interact With VR Using Physiological FeaturesIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2023.330878730:8(5893-5900)Online publication date: Aug-2024
      • (2024)Effects of Congruent Multisensory Feedback on the Perception and Performance of Virtual Reality Hand-Retargeted InteractionIEEE Access10.1109/ACCESS.2024.345051212(119789-119802)Online publication date: 2024
      • (2024)Evaluating Visual-Spatiotemporal Co-Registration of a Physics-Based Virtual Reality Haptic InterfaceIEEE Access10.1109/ACCESS.2024.339118612(57017-57032)Online publication date: 2024
      • (2024)Prediction of cognitive conflict during unexpected robot behavior under different mental workload conditions in a physical human–robot collaborationJournal of Neural Engineering10.1088/1741-2552/ad249421:2(026010)Online publication date: 19-Mar-2024
      • (2024)Representing scentsInternational Journal of Human-Computer Studies10.1016/j.ijhcs.2024.103357192:COnline publication date: 1-Dec-2024
      • Show More Cited By

      View Options

      Login options

      View options

      PDF

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader

      HTML Format

      View this article in HTML Format.

      HTML Format

      Figures

      Tables

      Media

      Share

      Share

      Share this Publication link

      Share on social media