Abstract
This paper investigated human self-motion perception through the visual and vestibular sensory systems under the context of virtual reality (VR) and 4D. Consistently with general 4D riding applications, we designed and used sinusoidal motions as stimuli, which resembled simple roller coaster rides moving in three directions of pitch, surge, and heave. Based on the Bayesian integration model, we experimented to determine the uncertainty involved in the two sensory systems and their relative contributions. We factored in small sensory discrepancies between the visual and vestibular cues and visually noticeable obstacles that could distract viewers. We found that the vestibular system contributed more dominantly to the perception in the ratio of 7:3 than the vision, demonstrating vestibular capture. We also discovered that the visual scenes that contain eye-catching elements and pure optical flows can hamper self-motion perception while increasing the perceptual uncertainty. Our findings can serve as a basis for designing motion effects for VR and 4D applications, especially in situations where multiple sensory systems are stimulated simultaneously.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.References
Alais D, Burr D (2019) Cue combination within a bayesian framework. In: Multisens. Processes. Springer, p 9–31. https://doi.org/10.1007/978-3-030-10461-0_2
Angelaki DE, Klier EM, Snyder LH (2009) A vestibular sensation: probabilistic approaches to spatial perception. Neuron 64(4):448–461. https://doi.org/10.1016/j.neuron.2009.11.010
BANDAI NAMCO amusement (2021) VR zone. https://vrzone-pic.com/
Butler JS, Smith ST, Campos JL et al (2010) Bayesian integration of visual and vestibular signals for heading. J Vis 10(11):23–23. https://doi.org/10.1167/10.11.23
Butler JS, Campos JL, Bülthoff HH et al (2011) The role of stereo vision in visual-vestibular integration. Seeing Perceiving 24(5):453–470. https://doi.org/10.1163/187847511X588070
Chandler P, Sweller J (1991) Cognitive load theory and the format of instruction. Cogn Instr 8(4):293–332. https://doi.org/10.1163/187847511X588070
Covaci A, Zou L, Tal I et al (2018) Is multimedia multisensorial?A review of mulsemedia systems. ACM Comput Surv 51(5):1–35. https://doi.org/10.1145/3233774
DeAngelis GC, Angelaki DE (2012) Visual–vestibular integration for self-motion perception. In: Murray MM, Mark TW (eds) The neural bases of multisensory processes. Taylor & Francis, Milton Park. https://doi.org/10.1201/9781439812174
DiCiccio TJ, Efron B (1996) Bootstrap confidence intervals. Stat Sci 11(3):189–228. https://doi.org/10.1214/ss/1032280214
Ernst MO, Banks MS (2002) Humans integrate visual and haptic information in a statistically optimal fashion. Nature 415(6870):429–433. https://doi.org/10.1038/415429a
Ernst MO, Bülthoff HH (2004) Merging the senses into a robust percept. Trends Cogn Sci 8(4):162–169. https://doi.org/10.1016/j.tics.2004.02.00
Fernandez C, Goldberg JM (1976) Physiology of peripheral neurons innervating otolith organs of the squirrel monkey I response to static tilts and to long-duration centrifugal force. J Neurophysiol 39(5):970–984. https://doi.org/10.1152/jn.1976.39.5.970
Fetsch CR, Turner AH, DeAngelis GC et al (2009) Dynamic reweighting of visual and vestibular cues during self-motion perception. J Neurosci 29(49):15601–15612. https://doi.org/10.1523/JNEUROSCI.2574-09.2009
Gibson JJ (1950) The Perception of the Visual World. Houghton Mifflin. https://doi.org/10.2307/1419017
Grabherr L, Nicoucar K, Mast FW et al (2008) Vestibular thresholds for yaw rotation about an earth-vertical axis as a function of frequency. Exp Brain Res 186(4):677–681. https://doi.org/10.1007/s00221-008-1350-8
Han S, Yun G, Choi S (2021) Camera space synthesis of motion effects emphasizing a moving object in 4D films. In: Proc. IEEE Conf. Virtual Real. 3D User Interfaces. IEEE, Manhattan, New York, U.S., pp 670–678. https://doi.org/10.1109/vr50410.2021.00093
Hillis JM, Ernst MO, Banks MS et al (2002) Combining sensory information: mandatory fusion within, but not between, senses. Science 298(5598):1627–1630. https://doi.org/10.1126/science.1075396
Hinde SJ, Smith TJ, Gilchrist ID (2018) Does narrative drive dynamic attention to a prolonged stimulus? Cogn Res: Princ Implic 3(1):1–12. https://doi.org/10.1186/s41235-018-0140-5
ter Horst AC, Koppen M, Selen LP et al (2015) Reliability-based weighting of visual and vestibular cues in displacement estimation. PloS ONE 10(12):e0145015. https://doi.org/10.1371/journal.pone.0145015
Hou H, Gu Y (2020) Multisensory integration for self-motion perception. Ref Module Neurosci Biobehav Psychol. https://doi.org/10.1163/22134808-00002527
Hutchings JB (1998) Color in plants, animals and man. In: Nassau K (ed) Color for science, art and technology. Elsevier, Amsterdam, pp 221–246. https://doi.org/10.1016/s1387-6783(98)80011-1
Klein SA (2001) Measuring, estimating, and understanding the psychometric function: a commentary. Percept Psychophys 63(8):1421–1455. https://doi.org/10.3758/BF03194552
Landy MS, Maloney LT, Johnston EB et al (1995) Measurement and modeling of depth cue combination: in defense of weak fusion. Vis Res 35(3):389–412. https://doi.org/10.1016/0042-6989(94)00176-m
Lappe M, Bremmer F, Van den Berg A (1999) Perception of self-motion from visual flow. Trends Cogn Sci 3(9):329–336. https://doi.org/10.1016/s1364-6613(99)01364-9
Lawther A, Griffin MJ (1987) Prediction of the incidence of motion sickness from the magnitude, frequency, and duration of vertical oscillation. J Acoust Soc Am 82(3):957–966. https://doi.org/10.1121/1.395295
Lee EC, Heo H, Park KR (2010) The comparative measurements of eyestrain caused by 2d and 3d displays. IEEE Trans Consum Electron 56(3):1677–1683. https://doi.org/10.1109/TCE.2010.5606312
Lee J, Han B, Choi S (2016) Interactive motion effects design for a moving object in 4d films. In: Proceedings of the 28th ACM symposium on virtual reality software and technology. ACM, New York, NY, USA, pp 219–228. https://doi.org/10.1145/2993369.2993389
Lee J, Han B, Choi S (2016) Motion effects synthesis for 4D films. IEEE Trans Vis Comput Graph 22(10):2300–2314. https://doi.org/10.1109/tvcg.2015.2507591
Lee J, Park J, Choi S (2021) Absolute and differential thresholds of motion effects in cardinal directions. In: Proceedings of the 28th ACM symposium on virtual reality software and technology. https://doi.org/10.1145/3489849.3489870
Lim B, Han S, Choi S (2021) Image-based texture styling for motion effect rendering. In: Proceedings of the 28th ACM symposium on virtual reality software and technology. https://doi.org/10.1145/3489849.3489854
Liu JS, Chen R (1998) Sequential monte carlo methods for dynamic systems. J Am Stat Assoc 93(443):1032–1044. https://doi.org/10.2307/2669847
Meijer D, Noppeney U (2020) Computational models of multisensory integration. In: Sathian K, Ramachandran VS (eds) Multisensory perception: from laboratory to clinic. Elsevier, chap 5, p 113–133. https://doi.org/10.1016/b978-0-12-812492-5.00005-x
Oruç İ, Maloney LT, Landy MS (2003) Weighted linear cue combination with possibly correlated error. Vis Res 43(23):2451–2468. https://doi.org/10.1016/s0042-6989(03)00435-8
Parkhurst D, Law K, Niebur E (2002) Modeling the role of salience in the allocation of overt visual attention. Vis Res 42(1):107–123. https://doi.org/10.1016/S0042-6989(01)00250-4
Pridmore RW (2020) Complementary colors: a literature review. Color Res Appl 46(2):482–488. https://doi.org/10.1002/col.22576
Prsa M, Gale S, Blanke O (2012) Self-motion leads to mandatory cue fusion across sensory modalities. J Neurophysiol 108(8):2282–2291. https://doi.org/10.1152/jn.00439.2012
Reason JT, Brand JJ (1975) Motion sickness. Academic Press, Cambridge. ISBN: 978-0125840507
Rohde M, van Dam LC, Ernst MO (2016) Statistically optimal multisensory cue integration: a practical tutorial. Multisens Res 29(4–5):279–317. https://doi.org/10.1163/22134808-00002510
Sato Y, Toyoizumi T, Aihara K (2007) Bayesian inference explains perception of unity and ventriloquism aftereffect: identification of common sources of audiovisual stimuli. Neural Comput 19(12):3335–3355. https://doi.org/10.1162/neco.2007.19.12.3335
Seber GA, Wild CJ (2003) Nonlinear regression. Wiley, New York,. ISBN: 978-0471471356
Shin S, Yoo B, Han S (2014) A framework for automatic creation of motion effects from theatrical motion pictures. Multimed Syst 20(3):327–346. https://doi.org/10.1007/s00530-013-0322-4
Solimini AG (2013) Are there side effects to watching 3d movies? a prospective crossover observational study on visually induced motion sickness. PloS One 8(2):e56160. https://doi.org/10.1371/journal.pone.0056160
Triesch J, Ballard DH, Jacobs RA (2002) Fast temporal dynamics of visual cue integration. Perception 31(4):421–434. https://doi.org/10.1068/p3314
Watanabe K, Shimojo S (1998) Attentional modulation in perception of visual motion events. Perception 27(9):1041–1054. https://doi.org/10.1068/p271041
Welch RB (1978) Perceptual modification: adapting to altered sensory environments. Academic Press, Cambridge ISBN: 978-1483274782
Wilder JD, Kowler E, Schnitzer BS et al (2009) Attention during active visual tasks: counting, pointing, or simply looking. Vis Res 49(9):1017–1031. https://doi.org/10.1016/j.visres.2008.04.032
Witten IB, Knudsen EI (2005) Why seeing is believing: merging auditory and visual worlds. Neuron 48(3):489–496. https://doi.org/10.1016/j.neuron.2005.10.020
Yakubovich S, Israeli-Korn S, Halperin O et al (2020) Visual self-motion cues are impaired yet overweighted during visual-vestibular integration in Parkinson’s disease. Brain Commun. https://doi.org/10.1093/braincomms/fcaa035
Sn Yang, Schlieski T, Selmins B et al (2012) Stereoscopic viewing and reported perceived immersion and symptoms. Optom Vis Sci 89(7):1068–1080. https://doi.org/10.1097/OPX.0b013e31825da430
Yuille AL, Bülthoff HH (1996) Bayesian decision theory and psychophysics. Cambridge University Press, Cambridge, p 123–161. https://doi.org/10.1017/cbo9780511984037.006
Yun G, Lee H, Han S, et al (2021) Improving viewing experiences of first-person shooter gameplays with automatically-generated motion effects. In: Proceedings conference on human factors in computing systems . ACM, 320. https://doi.org/10.1145/3411764.3445358
Zhou Y, Tapaswi M, Fidler S (2018) Now you shake me: Towards automatic 4d cinema. In: Proceedings of the IEEE computer society conference on computer vision and pattern recognition IEEE, Manhattan, New York, U.S., pp 7425–7434. https://doi.org/10.1109/cvpr.2018.00775
Funding
This study was funded by Samsung Research Funding and Incubation Center of Samsung Electronics under a grant (SRFC-IT1802-05).
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of Interest
The authors declare that they have no conflict of interest.
Ethical approval
All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki Declaration and its later amendments or comparable ethical standards.
Informed consent
Informed consent was obtained from all individual participants included in the study.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary Information
Below is the link to the electronic supplementary material.
Supplementary file 1 (mp4 137949 KB)
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Lee, J., Han, S.H. & Choi, S. Sensory cue integration of visual and vestibular stimuli: a case study for 4D rides. Virtual Reality 27, 1671–1683 (2023). https://doi.org/10.1007/s10055-023-00762-7
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10055-023-00762-7