Cited By
View all- Namrouti HSik-Lányi CGuzsvinecz T(2025)Exploring measurement tools for color perception in virtual reality: A systematic reviewDisplays10.1016/j.displa.2024.10293787(102937)Online publication date: Apr-2025
Sound and Colour: Evaluating Auditory-Visual Tests in Virtual Reality and Traditional Desktop Settings
Article No.: 21, Pages 1 - 5
Abstract
In this pilot study, we used a balanced A/B testing method to evaluate how an environment—a Virtual Reality (VR) head-mounted display (HMD) versus a traditional desktop—affects the performance of 20 participants on a musical pitch-colour association test. We aimed to discern the influence of testing environments on musical pitch-colour associations. Our findings revealed no significant difference in performance between the VR HMD and the traditional desktop conditions. Previous studies highlight VR’s potential, but the results of our investigation—focusing on limited immersive and presence qualities—suggest that working in VR as opposed to a standard desktop environment has no significant influence on musical pitch-colour association test results. This outcome prompts a further investigation into other inherent VR characteristics, such as spatial audio, natural environment simulations, and interactive object manipulation, which could potentially enhance the effectiveness of auditory-visual association tests. This study is the first in a series of studies to explore how VR technologies can be exploited to augment multi-modal testing, with a fully immersive musical pitch-colour association test envisioned.
References
[1]
J Asher, M Aitken, N Farooqi, S Kurmani, and S Baroncohen. 2006. Diagnosing and Phenotyping Visual Synaesthesia: A Preliminary Evaluation of the Revised Test of Genuineness (TOG-R). Cortex 42, 2 (2006), 137–146. https://doi.org/10.1016/S0010-9452(08)70337-X
[2]
Monica Billger, Ilona Heldal, Beata Stahre, and Kristian Renstrom. 2004. Perception of Color and Space in Virtual Reality: A Comparison between a Real Room and Virtual Reality Models. In Electronic Imaging 2004, Bernice E. Rogowitz and Thrasyvoulos N. Pappas (Eds.). SPIE, San Jose, CA, 90. https://doi.org/10.1117/12.526986
[3]
David Brang and V. S. Ramachandran. 2011. Survival of the Synesthesia Gene: Why Do People Hear Colors and Taste Words?PLoS Biology 9, 11 (Nov. 2011), e1001205. https://doi.org/10.1371/journal.pbio.1001205
[4]
D.A. Carmichael, M.P. Down, R.C. Shillcock, D.M. Eagleman, and J. Simner. 2015. Validating a Standardised Test Battery for Synesthesia: Does the Synesthesia Battery Reliably Detect Synesthesia?Consciousness and Cognition 33 (May 2015), 375–385. https://doi.org/10.1016/j.concog.2015.02.001
[5]
Caroline Curwen. 2018. Music-Colour Synaesthesia: Concept, Context and Qualia. Consciousness and Cognition 61 (May 2018), 94–106. https://doi.org/10.1016/j.concog.2018.04.005
[6]
Ophelia Deroy and Charles Spence. 2013. Why We Are Not All Synesthetes (Not Even Weakly So). Psychonomic Bulletin & Review 20, 4 (Aug. 2013), 643–664. https://doi.org/10.3758/s13423-013-0387-2
[7]
Francisco Díaz-Barrancas, Halina Cwierz, Pedro J. Pardo, Ángel Luis Pérez, and María Isabel Suero. 2020. Spectral Color Management in Virtual Reality Scenes. Sensors 20, 19 (Oct. 2020), 5658. https://doi.org/10.3390/s20195658
[8]
David M. Eagleman, Arielle D. Kagan, Stephanie S. Nelson, Deepak Sagaram, and Anand K. Sarma. 2007. A Standardized Test Battery for the Study of Synesthesia. Journal of Neuroscience Methods 159, 1 (Jan. 2007), 139–145. https://doi.org/10.1016/j.jneumeth.2006.07.012
[9]
Francesca R. Farina, Kevin J. Mitchell, and Richard A. P. Roche. 2017. Synaesthesia Lost and Found: Two Cases of Person- and Music-Colour Synaesthesia. European Journal of Neuroscience 45, 3 (Feb. 2017), 472–477. https://doi.org/10.1111/ejn.13492
[10]
Raquel Gil Rodríguez, Florian Bayer, Matteo Toscani, Dar’ya Guarnera, Giuseppe Claudio Guarnera, and Karl R. Gegenfurtner. 2022. Colour Calibration of a Head Mounted Display for Colour Vision Research Using Virtual Reality. SN Computer Science 3, 1 (Jan. 2022), 22. https://doi.org/10.1007/s42979-021-00855-7
[11]
Aviva I. Goller, Leun J. Otten, and Jamie Ward. 2009. Seeing Sounds and Hearing Colors: An Event-related Potential Study of Auditory–Visual Synesthesia. Journal of Cognitive Neuroscience 21, 10 (Oct. 2009), 1869–1881. https://doi.org/10.1162/jocn.2009.21134
[12]
Stefan Gradl, Markus Wirth, Nico Mächtlinger, Romina Poguntke, Andrea Wonner, Nicolas Rohleder, and Bjoern M. Eskofier. 2019. The Stroop Room: A Virtual Reality-Enhanced Stroop Test. In 25th ACM Symposium on Virtual Reality Software and Technology. ACM, Parramatta NSW Australia, 1–12. https://doi.org/10.1145/3359996.3364247
[13]
Judith Greene and Manuela D’Oliveira. 2000. Learning to use statistical tests in psychology (2. ed., repr ed.). Open University Press, Buckingham.
[14]
Kate Hevner. 1936. Experimental Studies of the Elements of Expression in Music. The American Journal of Psychology 48, 2 (April 1936), 246. https://doi.org/10.2307/1415746 jstor:1415746
[15]
Athena Hosek. 2023. Music in VR: Exploring Creative Expression and Virtually “Tangible” Learning - ProQuest. https://www.proquest.com/docview/2819952937?fromopenview=true&pq-origsite=gscholar&sourcetype=Dissertations%20&%20Theses.
[16]
Donielle Johnson, Carrie Allison, and Simon Baron-Cohen. 2013. The Prevalence of Synesthesia. Oxford University Press, Oxford. https://doi.org/10.1093/oxfordhb/9780199603329.013.0001
[17]
Simon Lacey, Margaret Martinez, Kelly McCormick, and K. Sathian. 2016. Synesthesia Strengthens Sound-Symbolic Cross-Modal Correspondences. European Journal of Neuroscience 44, 9 (Nov. 2016), 2716–2721. https://doi.org/10.1111/ejn.13381
[18]
Pontus Larsson, Daniel Västfjäll, and Mendel Kleiner. 2002. Auditory-Visual Interaction in Real and Virtual Rooms. 6.
[19]
Daniel Martin, Sandra Malpica, Diego Gutierrez, Belen Masia, and Ana Serrano. 2022. Multimodality in VR: A Survey. Comput. Surveys (Jan. 2022), 3508361. https://doi.org/10.1145/3508361
[20]
Anabela Marto, Alexandrino Gonçalves, Miguel Melo, and Maximino Bessa. 2022. A Survey of Multisensory VR and AR Applications for Cultural Heritage. Computers & Graphics 102 (Feb. 2022), 426–440. https://doi.org/10.1016/j.cag.2021.10.001
[21]
Meta. 2022. Color and Brightness Mastering Guide | Oculus Developers. https://developer.oculus.com/resources/color-brightness-mastering/.
[22]
Meta. 2022. Color Management in Meta Quest Headsets | Oculus Developers. https://developer.oculus.com/resources/color-management-guide/.
[23]
Patrick O’Toole, Donald Glowinski, and Maurizio Mancini. 2019. Understanding Chromaesthesia by Strengthening Auditory -Visual-Emotional Associations. In 2019 8th International Conference on Affective Computing and Intelligent Interaction (ACII). IEEE, Cambridge, United Kingdom, 1–7. https://doi.org/10.1109/ACII.2019.8925465
[24]
S. E. Palmer, K. B. Schloss, Z. Xu, and L. R. Prado-Leon. 2013. Music-Color Associations Are Mediated by Emotion. Proceedings of the National Academy of Sciences 110, 22 (May 2013), 8836–8841. https://doi.org/10.1073/pnas.1212562110
[25]
Cesare V. Parise and Charles Spence. 2012. Audiovisual Crossmodal Correspondences and Sound Symbolism: A Study Using the Implicit Association Test. Experimental Brain Research 220, 3-4 (Aug. 2012), 319–333. https://doi.org/10.1007/s00221-012-3140-6
[26]
Thomas D. Parsons and Amanda S. Phillips. 2016. Virtual Reality for Psychological Assessment in Clinical Practice.Practice Innovations 1, 3 (Sept. 2016), 197–217. https://doi.org/10.1037/pri0000028
[27]
Robert Plutchik. 2001. The Nature of Emotions: Human Emotions Have Deep Evolutionary Roots, a Fact That May Explain Their Complexity and Provide Tools for Clinical Practice. American Scientist 89, 4 (2001), 344–350.
[28]
Romina Poguntke, Markus Wirth, and Stefan Gradl. 2019. Same Same but Different: Exploring the Effects of the Stroop Color Word Test in Virtual Reality. In Human-Computer Interaction – INTERACT 2019, David Lamas, Fernando Loizides, Lennart Nacke, Helen Petrie, Marco Winckler, and Panayiotis Zaphiris (Eds.). Vol. 11747. Springer International Publishing, Cham, 699–708. https://doi.org/10.1007/978-3-030-29384-0_42
[29]
Noam Sagiv and Jamie Ward. 2006. Chapter 15 Crossmodal Interactions: Lessons from Synesthesia. In Progress in Brain Research. Vol. 155. Elsevier, 259–271. https://doi.org/10.1016/S0079-6123(06)55015-0
[30]
Arthur Silva Bastos, Renata Faria Gomes, Clemilson Costa dos Santos, and José Gilvan Rodrigues Maia. 2018. Synesthesia: A Study on Immersive Features of Electronic Games. Journal on Interactive Systems 9, 2 (Aug. 2018), 1. https://doi.org/10.5753/jis.2018.700
[31]
Sean Soraghan. 2014. Animating Timbre - A User Study. In ICMC|SMC. Athens, Greece.
[32]
Ferrinne Spector and Daphne Maurer. 2009. Synesthesia: A New Approach to Understanding the Development of Perception.Developmental Psychology 45, 1 (Jan. 2009), 175–189. https://doi.org/10.1037/a0014171
[33]
Charles Spence. 2011. Crossmodal Correspondences: A Tutorial Review. Attention, Perception, & Psychophysics 73, 4 (May 2011), 971–995. https://doi.org/10.3758/s13414-010-0073-7
[34]
Joseph Timoney, Behnam Faghih, Anthony Gibney, Benjamin Korady, and Gareth Young. 2018. Singing-Blocks: Considerations for a Virtual Reality Game to Create Chords and Progressions. (Aug. 2018), 1–13.
[35]
Fernando Trujano, Mina Khan, and Pattie Maes. 2018. ARPiano Efficient Music Learning Using Augmented Reality. In Innovative Technologies and Learning, Ting-Ting Wu, Yueh-Min Huang, Rustam Shadiev, Lin Lin, and Andreja Istenič Starčič (Eds.). Vol. 11003. Springer International Publishing, Cham, 3–17. https://doi.org/10.1007/978-3-319-99737-7_1
[36]
Crétien van Campen. 2010. The Hidden Sense: Synesthesia in Art and Science. MIT Press, Cambridge, Mass.
[37]
Crétien van Campen and Clara Froger. 2003. Personal Profiles of Color Synesthesia: Developing a Testing Method for Artists and Scientists. Leonardo 36, 4 (Aug. 2003), 291–294. https://doi.org/10.1162/002409403322258709
[38]
J Ward, B Huckstep, and E Tsakanikos. 2006. Sound-Colour Synaesthesia: To What Extent Does It Use Cross-Modal Mechanisms Common to Us All?Cortex 42, 2 (2006), 264–280. https://doi.org/10.1016/S0010-9452(08)70352-6
[39]
Michael Wiesing, Hendrik W. Steinkönig, Simone Vossel, Gereon R. Fink, and Ralph Weidner. 2022. Transferring Paradigms from Physical to Virtual Reality: Can Reaction Time Effects Be Replicated in a Virtual Setting?Preprint. PsyArXiv. https://doi.org/10.31234/osf.io/kmsnz
[40]
Roy Williams, Simone Gumtau, and Jenny Mackness. 2015. Synesthesia: From Cross-Modal to Modality-Free Learning and Knowledge. Leonardo 48, 1 (Feb. 2015), 48–54. https://doi.org/10.1162/LEON_a_00937
[41]
Dongrui Wu, Christopher G. Courtney, Brent J. Lance, Shrikanth S. Narayanan, Michael E. Dawson, Kelvin S. Oie, and Thomas D. Parsons. 2010. Optimal Arousal Identification and Classification for Affective Computing Using Physiological Signals: Virtual Reality Stroop Task. IEEE Transactions on Affective Computing 1, 2 (July 2010), 109–118. https://doi.org/10.1109/T-AFFC.2010.12
Index Terms
- Sound and Colour: Evaluating Auditory-Visual Tests in Virtual Reality and Traditional Desktop Settings
Recommendations
Chromaesthetica: A Virtual Reality App for Learning Musical Theory Using Sound-Colour Cross-Modal Associations
MUM '24: Proceedings of the International Conference on Mobile and Ubiquitous MultimediaThis demo paper presents a Virtual Reality (VR) application designed to explore innovative methods of learning basic music theory through sound-colour cross-modal associations. Leveraging the immersive capabilities of the latest head-mounted displays, the ...
Virtual Reality Sickness and Challenges Behind Different Technology and Content Settings
AbstractVR sickness (Cybersickness) presents an important challenge in virtual reality environments. We present the results of a study on the effects of VR technology and VR video content type on VR sickness and on autonomous nervous system of the user. ...
Extending Virtual Reality Display Wall Environments Using Augmented Reality
SUI '19: Symposium on Spatial User InteractionTwo major form factors for virtual reality are head-mounted displays and large display environments such as CAVE®and the LCD-based successor CAVE2®. Each of these has distinct advantages and limitations based on how they’re used. This work explores ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
June 2024
578 pages
Copyright © 2024 Owner/Author.
This work is licensed under a Creative Commons Attribution International 4.0 License.
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 03 June 2024
Check for updates
Author Tags
Qualifiers
- Short-paper
- Research
- Refereed limited
Funding Sources
Conference
AVI 2024
AVI 2024: International Conference on Advanced Visual Interfaces 2024
June 3 - 7, 2024
Arenzano, Genoa, Italy
Acceptance Rates
AVI '24 Paper Acceptance Rate 21 of 82 submissions, 26%;
Overall Acceptance Rate 128 of 490 submissions, 26%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- View Citations1Total Citations
- 215Total Downloads
- Downloads (Last 12 months)215
- Downloads (Last 6 weeks)41
Reflects downloads up to 01 Jan 2025
Other Metrics
Citations
Cited By
View all- Namrouti HSik-Lányi CGuzsvinecz T(2025)Exploring measurement tools for color perception in virtual reality: A systematic reviewDisplays10.1016/j.displa.2024.10293787(102937)Online publication date: Apr-2025
View Options
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderHTML Format
View this article in HTML Format.
HTML FormatLogin options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in