[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to main content
Springer Nature Link
Log in

Multimodality: Exploring Sensibility and Sense-Making Beyond the Metaverse

  • Conference paper
  • First Online:
Virtual, Augmented and Mixed Reality (HCII 2023)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 14027))

Included in the following conference series:

Abstract

Immersive technologies are increasingly integrated into creative practice and various aspects of everyday life as augmented and virtual reality (AR/VR). These extended reality (XR) technologies are particularly relevant for researchers exploring human-computer relationships where users intend to interact with 3D objects and real-time multimedia in a spatial context. XR is being researched, developed, and deployed for communicating and building knowledge around what is being called the metaverse. The present trajectory for creating metaverse experiences strongly tends toward attaining heightened realism. Here we discuss the various factors of multimodality that challenge the idea that research in interaction for mixed realities should narrowly converge on or limit itself to developing applications intended to captivate users in these realist environments. We emphasize advancements in multimodality to further experimentation with human sensory modalities. To this end, we discuss the recent aesthetic theory proposed by Fuller and Weizman (2021), which is oriented to the attunement of the senses to further our capacities for sensing and sense-making. Building on Fuller and Weizman’s references to notions of memory, synaesthesia, kinaesthetics, and chronesthesia, we consider multimodality within mixed realities beyond realism to raise users’ sensibility and discernment toward both virtual and physical environments via new forms of experimentation with the senses.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
£29.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
GBP 19.95
Price includes VAT (United Kingdom)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
GBP 79.50
Price includes VAT (United Kingdom)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
GBP 99.99
Price includes VAT (United Kingdom)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Appel, G., Grewal, L., Hadi, R., Stephen, A.T.: The future of social media in marketing. J. Acad. Mark. Sci. 48(1), 79–95 (2020)

    Article  Google Scholar 

  2. Ball, M.: The Metaverse: And How It Will Revolutionize Everything. Liveright Publishing, New York (2022)

    Book  Google Scholar 

  3. Barlow, J.P.: A declaration of the independence of cyberspace. Duke L. Tech. Rev. 18, 5 (2019)

    Google Scholar 

  4. Bartle, R.A.: Designing Virtual Worlds. New Riders, Indianapolis (2004)

    Google Scholar 

  5. Blascovich, J., Loomis, J., Beall, A.C., Swinth, K.R., Hoyt, C.L., Bailenson, J.N.: Immersive virtual environment technology as a methodological tool for social psychology. Psychol. Inq. 13(2), 103–124 (2002)

    Article  Google Scholar 

  6. Borrego, A., Latorre, J., Alcañiz, M., Llorens, R.: Embodiment and presence in virtual reality after stroke: a comparative study with healthy subjects. Front. Neurol. 10, 1061 (2019)

    Article  Google Scholar 

  7. Bowman, D.A., McMahan, R.P.: Virtual reality: how much immersion is enough? Computer 40(7), 36–43 (2007)

    Article  Google Scholar 

  8. Burdea, G.C., Coiffet, P.: Virtual Reality Technology. Wiley, Hoboken (2003)

    Book  Google Scholar 

  9. Davies, C.: OSMOSE: notes on being in immersive virtual space. Digital Creativity (1998)

    Google Scholar 

  10. Dawkins, O., Young, G.W.: Engaging Place with mixed realities: sharing multisensory experiences of place through community-generated digital content and multimodal interaction. In: Chen, J.Y.C., Fragomeni, G. (eds.) HCII 2020. LNCS, vol. 12191, pp. 199–218. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-49698-2_14

    Chapter  Google Scholar 

  11. DiPietro, J.C., Black, E.W.: Visual analysis of avatars in gaming environments. In: Handbook of Research on Effective Electronic Gaming in Education, pp. 606–620. IGI Global (2009)

    Google Scholar 

  12. Friedman, D., Pizarro, R., Or-Berkers, K., Neyret, S., Pan, X., Slater, M.: A method for generating an illusion of backwards time travel using immersive virtual reality—an exploratory study. Front. Psychol. 5, 943 (2014)

    Article  Google Scholar 

  13. Fuller, M., Weizman, E.: Investigative Aesthetics: Conflicts and Commons in the Politics of Truth. Verso Books, New York (2021)

    Google Scholar 

  14. Gibson, W.: Burning Chrome. Hachette, UK (2017)

    Google Scholar 

  15. Grau, O.: Virtual Art: From Illusion to Immersion. MIT press, Cambridge (2004)

    Google Scholar 

  16. Hsieh, M.C., Lee, J.J.: Preliminary study of VR and AR applications in medical and healthcare education. J. Nurs. Health. Stud. 3(1), 1 (2018)

    Article  Google Scholar 

  17. Iriye, H., St. Jacques, P.L.: Memories for third-person experiences in immersive virtual reality. Sci. Rep. 11(1), 4667 (2021)

    Google Scholar 

  18. Konishi, Y., Hanamitsu, N., Outram, B., Minamizawa, K., Mizuguchi, T., Sato, A.: Synesthesia suit: the full body immersive experience. In: SIGGRAPH 2016 VR Village, pp. 1–1. ACM (2016)

    Google Scholar 

  19. Kose, A., Tepljakov, A., Astapov, S., Draheim, D., Petlenkov, E., Vassiljeva, K.: Towards a synesthesia laboratory: real-time localization and visualization of a sound source for virtual reality applications. J. Commun. Softw. Syst. 14(1), 112–120 (2018)

    Google Scholar 

  20. Krueger, M.W.: Environmental technology: making the real world virtual. Commun. ACM 36(7), 36–37 (1993)

    Article  Google Scholar 

  21. Lavoie, R., Main, K., King, C., King, D.: Virtual experience, real consequences: the potential negative emotional consequences of virtual reality gameplay. Virtual Reality 25, 69–81 (2021)

    Article  Google Scholar 

  22. Lillemose, J., Kryger, M.: The (re) invention of cyberspace. Kunstkritikk Nordic Art Review (2015). https://kunstkritikk.com/the-reinvention-of-cyberspace/

  23. Liu, V., Chilton, L.B.: Design guidelines for prompt engineering text-to-image generative models. In: Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems, pp. 1–23 (2022)

    Google Scholar 

  24. Lugmayr, A., Lim, Y.J., Hollick, J., Khuu, J., Chan, F.: Financial data visualization in 3D on immersive virtual reality displays. In: Mehandjiev, N., Saadouni, B. (eds.) FinanceCom 2018. LNBIP, vol. 345, pp. 118–130. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-19037-8_8

    Chapter  Google Scholar 

  25. Mestre, D.R.: Cave versus head-mounted displays: ongoing thoughts. Electron. Imaging 29, 31–35 (2017)

    Article  Google Scholar 

  26. Mikropoulos, T.A., Natsis, A.: Educational virtual environments: a ten-year review of empirical research (1999–2009). Comput. Educ. 56(3), 769–780 (2011)

    Article  Google Scholar 

  27. Milgram, P., Kishino, F.: A taxonomy of mixed reality visual displays. IEICE Trans. Inf. Syst. 77(12), 1321–1329 (1994)

    Google Scholar 

  28. Mystakidis, S.: Metaverse. Encyclopedia 2(1), 486–497 (2022)

    Article  Google Scholar 

  29. Optale, G., et al.: Controlling memory impairment in elderly adults using virtual reality memory training: a randomized controlled pilot study. Neurorehabil. Neural Repair 24(4), 348–357 (2010)

    Article  Google Scholar 

  30. Outram, B.I.: Synesthesia audio-visual interactive-sound and music visualization in virtual reality with orbital observation and navigation. In: 2016 IEEE International Workshop on Mixed Reality Art (MRA), pp. 7–8. IEEE (2016)

    Google Scholar 

  31. O’dwyer, N., Zerman, E., Young, G.W., Smolic, A., Dunne, S., Shenton, H.: Volumetric video in augmented reality applications for museological narratives: a user study for the long room in the library of trinity college Dublin. J. Comput. Cult. Heritage (JOCCH) 14(2), 1–20 (2021)

    Google Scholar 

  32. Pacella, D.: NFT 2.0: Building fan communities and creating user engagement (2021). https://www.verizon.com/about/news/nft-20-building-fan-communities-and-creating-user-engagement

  33. Pimentel, K., Teixeira, K.: Virtual Reality Through the New Looking Glass. CUMINCAD (1993)

    Google Scholar 

  34. Poeschl, S., Doering, N.: The German VR simulation realism scale-psychometric construction for virtual reality applications with virtual humans. Annu. Rev. Cyberther. Telemed. 11, 33–37 (2013)

    Google Scholar 

  35. Qian, Y.: 3d multi-user virtual environments: promising directions for science education. Sci. Educ. 18(2), 25–29 (2009)

    Google Scholar 

  36. Ross, M.: Virtual reality’s new synesthetic possibilities. Telev. New Media 21(3), 297–314 (2020)

    Article  Google Scholar 

  37. Seth, A., Vance, J.M., Oliver, J.H.: Virtual reality for assembly methods prototyping: a review. Virtual Reality 15, 5–20 (2011)

    Article  Google Scholar 

  38. Shedroff, N., Noessel, C.: Make it So: Interaction Design Lessons from Science Fiction. Rosenfeld Media, New York (2012)

    Book  Google Scholar 

  39. Slater, M., Sanchez-Vives, M.V.: Enhancing our lives with immersive virtual reality. Front. Robot. AI 3, 74 (2016)

    Article  Google Scholar 

  40. Stephenson, N.: Snow Crash: A Novel. Spectra, California (2003)

    Google Scholar 

  41. Suddendorf, T., Addis, D.R., Corballis, M.C.: Mental time travel and shaping of the human mind, pp. 344–354. M. Bar, Karnataka (2011)

    Google Scholar 

  42. Sutherland, I.E., et al.: The ultimate display. In: Proceedings of the IFIP Congress, vol. 2, pp. 506–508. New York (1965)

    Google Scholar 

  43. Tulving, E.: Memory and consciousness. Can. Psychol./Psychol. Can. 26(1), 1 (1985)

    Article  Google Scholar 

  44. Vázquez, C., Xia, L., Aikawa, T., Maes, P.: Words in motion: kinesthetic language learning in virtual reality. In: 2018 IEEE 18th International Conference on Advanced Learning Technologies (ICALT), pp. 272–276. IEEE (2018)

    Google Scholar 

  45. Wais, P.E., Arioli, M., Anguera-Singla, R., Gazzaley, A.: Virtual reality video game improves high-fidelity memory in older adults. Sci. Rep. 11(1), 2552 (2021)

    Article  Google Scholar 

  46. Wang, X., Young, G.W., Mc Guckin, C., Smolic, A.: A systematic review of virtual reality interventions for children with social skills deficits. In: 2021 IEEE International Conference on Engineering, Technology & Education (TALE), pp. 436–443. IEEE (2021)

    Google Scholar 

  47. Ward, J.: Synesthesia. Annu. Rev. Psychol. 64, 49–75 (2013)

    Article  Google Scholar 

  48. Witmer, B.G., Singer, M.J.: Measuring presence in virtual environments: a presence questionnaire. Presence 7(3), 225–240 (1998)

    Article  Google Scholar 

  49. Young, G.W., O’Dwyer, N., Smolic, A.: Volumetric video as a novel medium for creative storytelling. In: Immersive Video Technologies, pp. 591–607. Elsevier (2023)

    Google Scholar 

Download references

Acknowledgments

This publication has emanated from research conducted with the financial support of the Science Foundation Ireland (SFI) under Grant Number 15/RP/2776, EPSRC-SFI grant number EP/T03324X/1, European Research Council (ERC) via the Data Stories project grant no. 101052998, and the Horizon Europe Framework Program (HORIZON), under the grant agreement 101070109.

Author information

Authors and Affiliations

  1. TRANSMIXR, School of Computer Science and Statistics, Trinity College Dublin, Dublin, Ireland

    Gareth W. Young

  2. Data Stories, Maynooth University Social Sciences Institute, Maynooth University, Maynooth, Ireland

    Oliver Dawkins

Authors
  1. Gareth W. Young
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Oliver Dawkins
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Gareth W. Young .

Editor information

Editors and Affiliations

  1. U.S. Army Research Laboratory, Aberdeen Proving Ground, MD, USA

    Jessie Y. C. Chen

  2. U.S. Army Combat Capabilities Development Command Soldier Center, Orlando, FL, USA

    Gino Fragomeni

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Young, G.W., Dawkins, O. (2023). Multimodality: Exploring Sensibility and Sense-Making Beyond the Metaverse. In: Chen, J.Y.C., Fragomeni, G. (eds) Virtual, Augmented and Mixed Reality. HCII 2023. Lecture Notes in Computer Science, vol 14027. Springer, Cham. https://doi.org/10.1007/978-3-031-35634-6_22

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-35634-6_22

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-35633-9

  • Online ISBN: 978-3-031-35634-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation