Aircraft Cockpit Interaction in Virtual Reality with Visual, Auditive, and Vibrotactile Feedback
Article No.: 445, Pages 420 - 443
Abstract
Safety-critical interactive spaces for supervision and time-critical control tasks are usually characterized by many small displays and physical controls, typically found in control rooms or automotive, railway, and aviation cockpits.
Using Virtual Reality (VR) simulations instead of a physical system can significantly reduce the training costs of these interactive spaces without risking real-world accidents or occupying expensive physical simulators.
However, the user's physical interactions and feedback methods must be technologically mediated. Therefore, we conducted a within-subjects study with 24 participants and compared performance, task load, and simulator sickness during training of authentic aircraft cockpit manipulation tasks. The participants were asked to perform these tasks inside a VR flight simulator (VRFS) for three feedback methods (acoustic, haptic, and acoustic+haptic) and inside a physical flight simulator (PFS) of a commercial airplane cockpit.
The study revealed a partial equivalence of VRFS and PFS, control-specific differences input elements, irrelevance of rudimentary vibrotactile feedback, slower movements in VR, as well as a preference for PFS.
References
[1]
Andrea F. Abate, Mariano Guida, Paolo Leoncini, Michele Nappi, and Stefano Ricciardi. 2009. A haptic-based approach to virtual training for aerospace industry. Journal of Visual Languages & Computing, 20, 5 (2009), 318–325. issn:1045926X https://doi.org/10.1016/j.jvlc.2009.07.003
[2]
Gae Ae Ryu and Kwan-Hee Yoo. 2020. Key Factors for Reducing Motion Sickness in 360° Virtual Reality Scene: Extended Abstract. In The 25th International Conference on 3D Web Technology (Web3D ’20). Association for Computing Machinery, New York, NY, USA. Article 28, 2 pages. isbn:9781450381697 https://doi.org/10.1145/3424616.3424723
[3]
Bruno Araujo, Ricardo Jota, Varun Perumal, Jia Xian Yao, Karan Singh, and Daniel Wigdor. 2016. Snake Charmer. In TEI ’16, Saskia Bakker, Caroline Hummels, and Brygg Ullmer (Eds.). ACM, New York. 218–226. isbn:9781450335829 https://doi.org/10.1145/2839462.2839484
[4]
Turgay Aslandere, Daniel Dreyer, Frieder Pantkratz, and Rene Schubotz. 2014. A generic virtual reality flight simulator. In Virtuelle und Erweiterte Realität, 11. Workshop der GI-Fachgruppe VR/AR. Shaker Verlag, Bremen, Germany. 1–13.
[5]
Stefan Auer, Jens Gerken, Harald Reiterer, and Hans-Christian Jetter. 2021. Comparison Between Virtual Reality and Physical Flight Simulators for Cockpit Familiarization. In Mensch Und Computer 2021 (MuC ’21). Association for Computing Machinery, New York, NY, USA. 378–392. isbn:9781450386456 https://doi.org/10.1145/3473856.3473860
[6]
Myroslav Bachynskyi and Jörg Müller. 04212020. Dynamics of Aimed Mid-air Movements. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, Regina Bernhaupt, Florian ’Floyd’ Mueller, David Verweij, Josh Andres, Joanna McGrenere, Andy Cockburn, Ignacio Avellino, Alix Goguey, Pernille Bjørn, Shengdong Zhao, Briane Paul Samson, and Rafal Kocielnik (Eds.). ACM, New York, NY, USA. 1–12. isbn:9781450367080 https://doi.org/10.1145/3313831.3376194
[7]
Pradipta Biswas and Jeevithashree DV. 2018. Eye Gaze Controlled MFD for Military Aviation. In 23rd International Conference on Intelligent User Interfaces (IUI ’18). ACM, New York, NY, USA. 79–89. isbn:978-1-4503-4945-1 https://doi.org/10.1145/3172944.3172973
[8]
James C Byers. 1989. Traditional and raw task load index (TLX) correlations: Are paired comparisons necessary? Advances in Industrial Ergonomics and Safety l: Taylor and Francis., 1, 1 (1989), 481–485.
[9]
Tom Carter, Sue Ann Seah, Benjamin Long, Bruce Drinkwater, and Sriram Subramanian. 2013. UltraHaptics: multi-point mid-air haptic feedback for touch surfaces. In Proceedings of the 26th annual ACM symposium on User interface software and technology - UIST ’13. ACM Press, St. Andrews, Scotland, United Kingdom. 505–514. isbn:978-1-4503-2268-3 https://doi.org/10.1145/2501988.2502018
[10]
Sarah Creem-Regehr, K. Myszkowski, Bobby Bodenheimer, Betty J. Mohler, Bernhard Riecke, and Stephen N. Spencer. 2008. Proceedings APGV 2008: Symposium on Applied Perception in Graphics and Visualization Los Angeles, California, August 9-10, 2008. ACM Press, New York. isbn:9781595939814
[11]
Daniele Sportillo, Alexis Paljic, and Luciano Ojeda. 2019. HRI’19: The 14th ACM/IEEE International Conference on Human-Robot Interaction : March 11-14, 2019, Daegu, South Korea. IEEE, Piscataway, NJ. isbn:9781538685556
[12]
Kai-Uwe Doer, Jens Schiefel, and W. Kubbat. 2001. Virtual Cockpit Simulation for Pilot Training. University of Darmstadt (Germany).
[13]
Tafadzwa Joseph Dube, Yuan Ren, Hannah Limerick, I. Scott MacKenzie, and Ahmed Sabbir Arif. 2022. Push, Tap, Dwell, and Pinch: Evaluation of Four Mid-air Selection Methods Augmented with Ultrasonic Haptic Feedback. Proceedings of the ACM on Human-Computer Interaction, 6, ISS (2022), 207–225. https://doi.org/10.1145/3567718
[14]
Joseph Dumas, Andrew Novobilski, Dawn Ellis, and Mark Paschal. 2002. VR on a Budget: Developing a Flight Simulator in a Small Institution with off-the-Shelf Hardware and Open Source Software. J. Comput. Sci. Coll., 18, 2 (2002), Dec., 138–142. issn:1937-4771
[15]
Alex Girdler and Orestis Georgiou. 2020. Mid-Air Haptics in Aviation–creating the sensation of touch where there is nothing but thin air. ArXiv, abs/2001.01445 (2020).
[16]
Jerome N. Gregoire, Celeste M. Alfes, Andrew P. Reimer, and Mary F. Terhaar. 2017. Flying Lessons for Clinicians: Developing System 2 Practice. Air Medical Journal, 36, 3 (2017), May, 135–137. issn:1067-991X https://doi.org/10.1016/j.amj.2017.02.003
[17]
Xiaochi Gu, Yifei Zhang, Weize Sun, Yuanzhe Bian, Dao Zhou, and Per Ola Kristensson. 05072016. Dexmo. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, Jofish Kaye, Allison Druin, Cliff Lampe, Dan Morris, and Juan Pablo Hourcade (Eds.). ACM, New York, NY, USA. 1991–1995. isbn:9781450333627 https://doi.org/10.1145/2858036.2858487
[18]
Sandra G. Hart and Lowell E. Staveland. 1988. Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research. In Advances in Psychology. 52, Elsevier, Amsterdam. 139–183. isbn:978-0-444-70388-0 https://doi.org/10.1016/S0166-4115(08)62386-9
[19]
Hans-Christian Jetter, Roman Rädle, Tiare Feuchtner, Christoph Anthes, Judith Friedl, and Clemens Nylandsted Klokmose. 2020. "In VR, Everything is Possible!": Sketching and Simulating Spatially-Aware Interactive Spaces in Virtual Reality. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (CHI ’20). Association for Computing Machinery, New York, NY, USA. 1–16. isbn:9781450367080 https://doi.org/10.1145/3313831.3376652
[20]
Hans-Christian Jetter, Harald Reiterer, and Florian Geyer. 2014. Blended Interaction: Understanding Natural Human—Computer Interaction in Post-WIMP Interactive Spaces. Personal Ubiquitous Comput., 18, 5 (2014), jun, 1139–1158. issn:1617-4909 https://doi.org/10.1007/s00779-013-0725-4
[21]
Richard Joyce and Stephen K. Robinson. 2019. Evaluation of a Virtual Reality Environment for Cockpit Design. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 63, 1 (2019), 2328–2332. issn:2169-5067 https://doi.org/10.1177/1071181319631309
[22]
Robert S Kennedy, Julie M Drexler, Daniel E Compton, Kay M Stanney, D Susan Lanham, and Deborah L Harm. 2003. Configural Scoring of Simulator Sickness, Cybersickness and Space Adaptation Syndrome. In Virtual and adaptive environments, Lawrence J. Hettinger and Michael W. Haas (Eds.). Lawrence Erlbaum Associates Publishers, Mahwah, N.J. 247–278. isbn:978-0-8058-3107-8 https://doi.org/10.1201/9781410608888.ch12
[23]
Robert S. Kennedy, Norman E. Lane, Kevin S. Berbaum, and Michael G. Lilienthal. 1993. Simulator Sickness Questionnaire: An Enhanced Method for Quantifying Simulator Sickness. The International Journal of Aviation Psychology, 3, 3 (1993), 203–220. https://doi.org/10.1207/s15327108ijap0303_3
[24]
Helge Lenz and Daniela Schmid. 08.09.2019 - 12.09.2019. Simulation Platform for Reduced Crew Operations — A Case Study. In 2019 IEEE/AIAA 38th Digital Avionics Systems Conference (DASC). IEEE, San Diego, USA. 1–7. isbn:978-1-7281-0649-6 https://doi.org/10.1109/DASC43569.2019.9081747
[25]
Wang Lijing, Xiang Wei, He Xueli, Sun Xiaohui, Yu Jinhai, Zhou Lin, and Sun Gaoyong. 2009. The virtual evaluation of the Ergonomics layout in Aircraft cockpit. In 2009 IEEE 10th International Conference on Computer-Aided Industrial Design And Conceptual Design. IEEE, Wenzhou, China. 1438–1442. https://doi.org/10.1109/CAIDCD.2009.5375353
[26]
Thomas Longridge, Paul Ray, Edward M. Boothe, and Judith S. Burki-Cohen. 1996. Initiative towards more affordable flight simulators for U.S. commuter airline training. Royal Aeronautical Society, USA.
[27]
R.G. Menendez and J.E. Bernard. 2001. Flight simulation in synthetic environments. IEEE Aerospace and Electronic Systems Magazine, 16, 9 (2001), Sept., 19–23. issn:08858985 https://doi.org/10.1109/62.949532
[28]
Mathias Moehring and Bernd Froehlich. 2005. Pseudo-Physical Interaction with a Virtual Car Interior in Immersive Environments. In Eurographics Symposium on Virtual Environments, Erik Kjems and Roland Blach (Eds.). The Eurographics Association, Aire-la-Ville, CH. 185–189. isbn:978-3-905674-06-4 issn:1727-530X https://doi.org/10.2312/EGVE/IPT_EGVE2005/181-189
[29]
W. F. Moroney, D. W. Biers, F. T. Eggemeier, and J. A. Mitchell. 1992. A comparison of two scoring procedures with the NASA task load index in a simulated flight task. In Proceedings of the IEEE 1992 National Aerospace and Electronics Conference. IEEE, Dayton, USA. 734–740 vol.2. https://doi.org/10.1109/NAECON.1992.220513
[30]
Matthias Oberhauser, Reinhard Braunstingl, Daniel Dreyer, and Ioana Victoria Koglbauer. 2016. Pilots’ Interaction with Hardware Controls in a Virtual Reality Flight Simulator. In Proceedings of the 32nd Conference of the European Association for Aviation Psychology. European Association for Aviation Psychology, Cascais,Portugal.
[31]
Matthias Oberhauser and Daniel Dreyer. 2017. A virtual reality flight simulator for human factors engineering. Cognition, Technology & Work, 19, 2 (2017), 263–277. issn:1435-5558 https://doi.org/10.1007/s10111-017-0421-7
[32]
Matthias Oberhauser, Daniel Dreyer, Reinhard Braunstingl, and Ioana Koglbauer. 2018. What’s Real About Virtual Reality Flight Simulation?: Comparing the Fidelity of a Virtual Reality With a Conventional Flight Simulation Environment. Aviation Psychology and Applied Human Factors, 8, 1 (2018), March, 22–34. issn:2192-0923, 2192-0931 https://doi.org/10.1027/2192-0923/a000134
[33]
STEFANO PAPA, ANTONIO LANZOTTI, GIUSEPPE Di GIRONIMO, and ALESSIO BALSAMO. 2018. A NEW INTERACTIVE RAILWAY VIRTUAL SIMULATOR FOR TESTING PREVENTIVE SAFETY. In Computers in Railways XVI : Railway Engineering Design and Operation, G. Passerini, J. M. Mera, N. Tomii, and P. Tzieropoulos (Eds.) (WIT Transactions on The Built Environment). WIT PressSouthampton UK, Lisbon, Portugal. 367–378. https://doi.org/10.2495/CR180331
[34]
Marietta Papadatou-Pastou, Eleni Ntolka, Judith Schmitz, Maryanne Martin, Marcus R. Munafò, Sebastian Ocklenburg, and Silvia Paracchini. 2020. Human handedness: A meta-analysis. Psychological bulletin, 146, 6 (2020), 481–524. https://doi.org/10.1037/bul0000229
[35]
Randy Pausch, Thomas Crea, and Matthew Conway. 1992. A Literature Survey for Virtual Environments: Military Flight Simulator Visual Systems and Simulator Sickness. Presence: Teleoperators and Virtual Environments, 1, 3 (1992), 344–363. https://doi.org/10.1162/pres.1992.1.3.344 arxiv:https://doi.org/10.1162/pres.1992.1.3.344.
[36]
Vsevolod Peysakhovich, Louis Monnier, Mélanie Gornet, and Stéphane Juaneda. 2020-03. Virtual reality vs. real-life training to learn checklists for light aircraft. In Eye-Tracking in Aviation. Proceedings of the 1st International Workshop (ETAVI 2020). ISAE-SUPAERO, Université de Toulouse; Institute of Cartography and Geoinformation (IKG), ETH Zurich, Toulouse; Zurich. 47 – 53. https://doi.org/10.3929/ethz-b-000407648
[37]
Pornthep Preechayasomboon and Eric Rombokas. 2021. Haplets: Finger-Worn Wireless and Low-Encumbrance Vibrotactile Haptic Feedback for Virtual and Augmented Reality. Frontiers in Virtual Reality, 2 (2021), 1–15. https://doi.org/10.3389/frvir.2021.738613
[38]
Rebekka S. Renner, Boris M. Velichkovsky, and Jens R. Helmert. 2013. The perception of egocentric distances in virtual environments - A review. Comput. Surveys, 46, 2 (2013), 1–40. issn:0360-0300 https://doi.org/10.1145/2543581.2543590
[39]
Savern Reweti. 2014. PC-based aviation training devices for pilot training in visual flight rules procedures : development, validation and effectiveness. Massey University.
[40]
Andrew Robinson, Katerina Mania, and Philippe Perey. 2004. Flight simulation: research challenges and user assessments of fidelity. In Proceedings of the 2004 ACM SIGGRAPH international conference on Virtual Reality continuum and its applications in industry - VRCAI ’04. ACM Press, Singapore. 261. isbn:978-1-58113-884-9 https://doi.org/10.1145/1044588.1044644
[41]
J. M. Rolfe and K. J. Staples. 1997. Flight simulation (repr ed.). Cambridge Univ. Press, Cambridge. isbn:0521357519
[42]
S. Rustamov, E. Gasimov, R. Hasanov, S. Jahangirli, E. Mustafayev, and D. Usikov. 2018. Speech recognition in flight simulator. IOP Conference Series: Materials Science and Engineering, 459 (2018), 012005. https://doi.org/10.1088/1757-899X/459/1/012005
[43]
Shantanu A. Satpute, Janet R. Canady, Roberta L. Klatzky, and George D. Stetten. 2020. FingerSight: A Vibrotactile Wearable Ring for Assistance With Locating and Reaching Objects in Peripersonal Space. IEEE transactions on haptics, 13, 2 (2020), 325–333. https://doi.org/10.1109/TOH.2019.2945561
[44]
Jens Schiefele, Oliver Albert, and Kai Uwe Doerr. 1998. IFR flight simulation in a distributed virtual environment. In Modeling and Simulating Sensory Response for Real and Virtual Environments (SPIE Proceedings). SPIE, Bellingham, United States. 111. https://doi.org/10.1117/12.317567
[45]
Daniela Schmid and Neville A. Stanton. 2019. Exploring Bayesian analyses of a small-sample-size factorial design in human systems integration: the effects of pilot incapacitation. Human-Intelligent Systems Integration, 1, 2-4 (2019), 71–88. issn:2524-4876 https://doi.org/10.1007/s42454-020-00012-0
[46]
Samuel B. Schorr and Allison M. Okamura. 2017. Fingertip Tactile Devices for Virtual Object Manipulation and Exploration. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, Gloria Mark, Susan Fussell, Cliff Lampe, m.c. schraefel, Juan Pablo Hourcade, Caroline Appert, and Daniel Wigdor (Eds.). ACM, New York, NY, USA. 3115–3119. isbn:9781450346559 https://doi.org/10.1145/3025453.3025744
[47]
Alexander Smith, Benjamin Ward-Cherrier, Appolinaire Etoundi, and Martin J. Pearson. 23.10.2022 - 27.10.2022. Feeling the Pressure: The Influence of Vibrotactile Patterns on Feedback Perception. In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, Kyoto, Japan. 634–640. isbn:978-1-6654-7927-1 https://doi.org/10.1109/IROS47612.2022.9981594
[48]
Georg Stevenson, Andreas Riener, and Alois Ferscha. 2013. “TactiGlove” – A Guidance System to Effectively Find Hidden Spots in 3D Space. In Mobile Computing, Applications, and Services (Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering, Vol. 110). Springer Berlin Heidelberg, Berlin/Heidelberg. 80–99. isbn:978-3-642-36631-4 https://doi.org/10.1007/978-3-642-36632-1_5
[49]
Peter Thomas, Pradipta Biswas, and Patrick Langdon. 2015. State-of-the-Art and Future Concepts for Interaction in Aircraft Cockpits. In Universal Access in Human-Computer Interaction. Access to Interaction, Margherita Antona and Constantine Stephanidis (Eds.). Springer International Publishing, Cham. 538–549. isbn:978-3-319-20681-3
[50]
Kelvin Valentino, Kevin Christian, and Endra Joelianto. 2017. Virtual reality flight simulator. Internetworking Indonesia Journal, 9, 1 (2017), 21–25.
[51]
Danniel Varona-Marin, Jan A. Oberholzer, Edward Tse, and Stacey D. Scott. 2018. Post-Meeting Curation of Whiteboard Content Captured with Mobile Devices. In Proceedings of the 2018 ACM International Conference on Interactive Surfaces and Spaces (ISS ’18). Association for Computing Machinery, New York, NY, USA. 43–54. isbn:9781450356947 https://doi.org/10.1145/3279778.3279782
[52]
John Vince. 1993. 10 - Virtual Reality Techniques in Flight Simulation. In Virtual Reality Systems, R.A. Earnshaw, M.A. Gigante, and H. Jones (Eds.). Academic Press, Boston. 135 – 141. isbn:978-0-12-227748-1 https://doi.org/10.1016/B978-0-12-227748-1.50018-4
[53]
Bernhard Weber, Simon Schätzle, Thomas Hulin, Carsten Preusche, and Barbara Deml. 2011. Evaluation of a vibrotactile feedback device for spatial guidance. In 2011 IEEE World Haptics Conference. IEEE, Istanbul, Turkey. 349–354. https://doi.org/10.1109/WHC.2011.5945511
[54]
Eric Whitmire, Hrvoje Benko, Christian Holz, Eyal Ofek, and Mike Sinclair. 2018. Haptic Revolver: Touch, Shear, Texture, and Shape Rendering on a Reconfigurable Virtual Reality Controller. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems - CHI ’18. ACM Press, Montreal QC, Canada. 1–12. isbn:978-1-4503-5620-6 https://doi.org/10.1145/3173574.3173660
[55]
Guy Williams, Ken Lawrence, and Richard Weeks. 2004. Reconfigurable Flight Simulators in Modeling and Simulation. In AIAA Modeling and Simulation Technologies Conference and Exhibit (Guidance, Navigation, and Control and Co-located Conferences). American Institute of Aeronautics and Astronautics, Providence, Rhode Island. https://doi.org/10.2514/6.2004-5459
[56]
Chou Wusheng, Wang Tianmiao, and Hu Lei. 2003. Design of data glove and arm type haptic interface. In 11th Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems. IEEE Computer Society, Los Alamitos, Calif. 422–427. isbn:0-7695-1890-7 https://doi.org/10.1109/HAPTIC.2003.1191332
[57]
Binbin Yang, Xiaojun Xia, Shuai Wang, and Lanqing Ye. 2021. Development of flight simulation system based on leap motion controller. Procedia Computer Science, 183 (2021), 794–800. issn:18770509 https://doi.org/10.1016/j.procs.2021.02.131
[58]
I. Yavrucuk, E. Kubali, and O. Tarimci. 2011. A low cost flight simulator using virtual reality tools. IEEE Aerospace and Electronic Systems Magazine, 26, 4 (2011), April, 10–14. https://doi.org/10.1109/MAES.2011.5763338
[59]
L. Yu, Z. Xu, and L. Xin. 2019. Research on Reconfigurable Method of Cockpit Simulation for Virtual Training System. In 2019 IEEE 3rd Information Technology, Networking, Electronic and Automation Control Conference (ITNEC). IEEE 3rd Information Technology, Networking, Electronic and Automation Control Conference (ITNEC), Chengdu China. 2007–2011. https://doi.org/10.1109/ITNEC.2019.8729557
Index Terms
- Aircraft Cockpit Interaction in Virtual Reality with Visual, Auditive, and Vibrotactile Feedback
Recommendations
Comparison Between Virtual Reality and Physical Flight Simulators for Cockpit Familiarization
MuC '21: Proceedings of Mensch und Computer 2021Airlines and flying schools use high-end physical flight simulators (PFS) to reduce costs and risks of pilot training. However, such PFS with full-scale cockpits have very high acquisition and operation costs. In contrast, recent consumer-grade and off-...
HUD symbology evaluation in a virtual reality flight simulation
HCI-Aero '14: Proceedings of the International Conference on Human-Computer Interaction in AerospaceIn this paper, a virtual reality flight simulator for the evaluation of cockpit concepts is described. This simulator consists of Virtual Reality equipment (head/finger tracking and Head-Mounted Display), commercial flight simulation software, and ...
Aircraft Landing Gear Simulation Using Multidomain Modeling Technology
ICM '11: Proceedings of the 2011 International Conference of Information Technology, Computer Engineering and Management Sciences - Volume 01In order to accurately analyze the performance and influence of landing gear system, this paper presents a high fidelity plant model of an aircraft landing gear to inclusion into a full aircraft flight simulation. The use of domain specific modeling ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
December 2023
482 pages
Copyright © 2023 Owner/Author.
This work is licensed under a Creative Commons Attribution 4.0 International License.
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 01 November 2023
Published in PACMHCI Volume 7, Issue ISS
Check for updates
Author Tags
Qualifiers
- Research-article
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 811Total Downloads
- Downloads (Last 12 months)563
- Downloads (Last 6 weeks)73
Reflects downloads up to 08 Mar 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in