Abstract
Although Music is considered the field of emotions and moods, every composition has a structure: beginning, development, climax and conclusion. In classical music, there are many genres, but each piece always tells a story. Even more, most classical compositions follow the familiar structure of Aristotelian drama. Classical music narrative can take a number of forms: as music analysis on structures of the form (e.g., Sonata form with contrasting themes), emotional narrative (usually miniatures dedicated to one mood range, e.g., the genre of Prelude) or a program narrative according to composer’s notes (e.g., Berlioz’s Symphonie Fantastique). Generally speaking, perception of music is multisensory, so when music tells a story, it does so in a multimodal fashion. In this position paper, we present our investigation into visualisation of music narrative based on different forms of music narratives, and then show how visuals developed by cross-modal associations and synaesthesia art can be used to construct music narrative that is interactive. Our ultimate aim is to develop the approach into AR/VR visualisations of classical music with a strong narrative content.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Koenitz, H., Ferri, G., Haahr, M., Sezen, D., Sezen, T.I., (eds.) Interactive digital narrative: history, theory and practice. Routledge (2015)
Russo, F.: Multisensory processing in music. In: Thaut, M.H., Hodges, D.A., (eds.) The Oxford Handbook of Music and the Brain, Oxford (2019)
Maus, F.: Music as Narrative (PhD dissertation, Princeton) (1990)
Velasco, C., Obrist, M.: Multisensory experiences: where the senses meet technology. Oxford University Press, Oxford (2020)
Lee, C.: Original Creators: Oskar Fischinger, The Father Of Visual Music. Creators (2012). https://creators.vice.com/en_us/article/pgzn97/original-creators-oskar-fischinger-the-father-of-visual-music
Gillies, C.: WBUR iLab- Mary Ellen Bute
Greckel, W.: Visualisation in the performance of classical music: a new challenge. Q. J. Music Teach. Learn. (1992), Accessed: Feb. 04, 2018. http://www-usr.rider.edu/~vrme/v16n1/volume3/visions/winter6
Bailes, F.: Musical imagery and the temporality of consciousness. In: Music and Consciousness, vol. 2, pp. 271–285. Oxford University Press, Oxford (2019)
Timmers, R., Gravot, R.: Music as a multimodal experience. Psychomusicology: Music, Mind Brain 26(N2) (2016)
Van Campen, C. .: The Hidden Sense. Synesthesia in Art and Science. The MIT Press, Cambridge (2010)
Ramachandran, V.S., Hubbard, E.M.: Synaesthesia- a window Into perception, thought and language. J. Conscious. Stud. 8(12), 3–34 (2001)
Ward, J.: Synaesthesia as a Model System for Understanding Variation in the Human Mind and Brain. Sussex University (2020). https://files.osf.io/v1/resources/87fuj/providers/osfstorage/5fd1eff80694b7009ef3600e?action=download&direct&version=1
Haverkamp, M.: Synesthetic Design: Handbook for a Multi-Sensory Approach. Birkhäuser (2012)
Geraeds, M.T., Kruijff, C.V., de Mooij, J,: Augmented Concert (2017). https://www.behance.net/gallery/50955905/Augmented-Concert
Niemann, W.: Brahms. Schuster & Loeffler Verlag.; German Language edition (1 Jan. 1920)
Ward, J.: The frog who croaked blue: synesthesia and the mixing of the senses. Routledge, London and New York (2008)
Bor, D., Rothen, N., Schwartzman, D., Clayton, S., Seth, A.: Adults can be trained to acquire synesthetic experiences. Sci. Rep. 4, (2014). https://www.nature.com/articles/srep07089
Parise, C., Spence, C.: Audiovisual crossmodal correspondences and sound symbolism: A study using the implicit association test. Exp. Brain Res. 220(3–4), 319–333 (2012)
Galeyev, B.M.: The Nature and Functions of Synesthesia in Music. Leonardo 40(3), 285–288 (2007)
Rudenko, S.: Synesthete composer A. Scriabin: Reception History and Archetypal Imagery of Late Piano Miniatures” PhD dissertation (2018). http://digibug.ugr.es
Morrison, S.: Scriabin and Theurgy. In: Russian Opera and the Symbolist Movement, p. 184 University of California Press (2002)
Gawboy, A.M., Townsend, J.: Scriabin and the possible. Society for Music Theory 18, (2012)
Philipp, G., Skrjabin, A.: Ausgewählte Klavierwerke, Leipzig: Edition Peters (1971) Plate E.P. 12588
Garcia, S.: Scriabin’s Symbolist Plot Archetype in the Late Piano Sonatas. University California Press, vol. 23 (2000)
Steynberg, A.J. Using Reverse Interactive Audio Systems (RIAS) to Direct Attention in Virtual Reality Narrative Practices: A Case Study. In: International Conference on Interactive Digital Storytelling, pp. 353–356. Springer, Cham, 2020
Tikka, P., Vuori, R., Kaipainen, M.: Narrative logic of enactive cinema: Obsession. Digital Creativity 17(4), 205–212 (2006)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Rudenko, S., McDonnell, M., Layden, T., Haahr, M. (2022). Exploring Classical Music Narratives Through Multimodality in AR/VR Experiences. In: Vosmeer, M., Holloway-Attaway, L. (eds) Interactive Storytelling. ICIDS 2022. Lecture Notes in Computer Science, vol 13762. Springer, Cham. https://doi.org/10.1007/978-3-031-22298-6_24
Download citation
DOI: https://doi.org/10.1007/978-3-031-22298-6_24
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-22297-9
Online ISBN: 978-3-031-22298-6
eBook Packages: Computer ScienceComputer Science (R0)