More Web Proxy on the site http://driver.im/

research-article

Comparing the Effects of Visual, Haptic, and Visuohaptic Encoding on Memory Retention of Digital Objects in Virtual Reality

Authors:

Lucas Siqueira Rodrigues,

Timo Torsten Schmidt,

Johann Habakuk Israel,

John Nyakatura,

Thomas KoschAuthors Info & Claims

NordiCHI '24: Proceedings of the 13th Nordic Conference on Human-Computer Interaction

Article No.: 16, Pages 1 - 13

https://doi.org/10.1145/3679318.3685349

Published: 13 October 2024 Publication History

Abstract

Although Virtual Reality (VR) has undoubtedly improved human interaction with 3D data, users still face difficulties retaining important details of complex digital objects in preparation for physical tasks. To address this issue, we evaluated the potential of visuohaptic integration to improve the memorability of virtual objects in immersive visualizations. In a user study (N=20), participants performed a delayed match-to-sample task where they memorized stimuli of visual, haptic, or visuohaptic encoding conditions. We assessed performance differences between these encoding modalities through error rates and response times. We found that visuohaptic encoding significantly improved memorization accuracy compared to unimodal visual and haptic conditions. Our analysis indicates that integrating haptics into immersive visualizations enhances the memorability of digital objects. We discuss its implications for the optimal encoding design in VR applications that assist professionals who need to memorize and recall virtual objects in their daily work.

Supplemental Material

MP4 File

Presentation video

Download
170.25 MB

References

[1]

Amir Amedi, Rafael Malach, Talma Hendler, Sharon Peled, and Ehud Zohary. 2001. Visuo-haptic object-related activation in the ventral visual pathway. Nature Neuroscience 4, 3 (March 2001), 324–330. https://doi.org/10.1038/85201

[2]

R.S. Avila and L.M. Sobierajski. 1996. A haptic interaction method for volume visualization. In Proceedings of Seventh Annual IEEE Visualization ’96. ACM, San Francisco, CA, USA, 197–204,. https://doi.org/10.1109/VISUAL.1996.568108

[3]

Alan Baddeley, Vivien Lewis, and Giuseppe Vallar. 1984. Exploring the Articulatory Loop. The Quarterly Journal of Experimental Psychology Section A 36, 2 (May 1984), 233–252. https://doi.org/10.1080/14640748408402157

[4]

Alan D. Baddeley and Graham Hitch. 1974. Working Memory. In Psychology of Learning and Motivation. Vol. 8. Elsevier, 47–89. https://doi.org/10.1016/S0079-7421(08)60452-1

[5]

Jakki Bailey, Jeremy N Bailenson, and Andrea Stevenson Won. 2012. Presence and Memory: Immersive Virtual Reality Effects on Cued Recall. (2012).

[6]

Florence Bara, Edouard Gentaz, and Pascale Colé. 2007. Haptics in learning to read with children from low socio‐economic status families. British Journal of Developmental Psychology 25, 4 (Nov. 2007), 643–663. https://doi.org/10.1348/026151007X186643

[7]

Guillermo Bernal, Nelson Hidalgo, Conor Russomanno, and Pattie Maes. 2022. Galea: A physiological sensing system for behavioral research in Virtual Environments. In 2022 IEEE Conference on Virtual Reality and 3D User Interfaces (VR). 66–76. https://doi.org/10.1109/VR51125.2022.00024

[8]

Lonni Besançon, Anders Ynnerman, Daniel F. Keefe, Lingyun Yu, and Tobias Isenberg. 2021. The State of the Art of Spatial Interfaces for 3D Visualization. Computer Graphics Forum 40, 1 (Feb. 2021), 293–326. Number: 1.

[9]

Petter Bivall, Shaaron Ainsworth, and Lena A. E. Tibell. 2011. Do haptic representations help complex molecular learning?Science Education 95, 4 (July 2011), 700–719. https://doi.org/10.1002/sce.20439

[10]

Rafal Bogacz, Eric Brown, Jeff Moehlis, Philip Holmes, and Jonathan D. Cohen. 2006. The physics of optimal decision making: A formal analysis of models of performance in two-alternative forced-choice tasks.Psychological Review 113, 4 (2006), 700–765. https://doi.org/10.1037/0033-295X.113.4.700

[11]

Barbara M. Brooks. 1999. The Specificity of Memory Enhancement During Interaction with a Virtual Environment. Memory 7, 1 (Jan. 1999), 65–78. https://doi.org/10.1080/741943713

[12]

Frederick P. Brooks, Ming Ouh-Young, James J. Batter, and P. Jerome Kilpatrick. 1990. Project GROPE - Haptic displays for scientific visualization. ACM SIGGRAPH Computer Graphics 24, 4 (Sept. 1990), 177–185. https://doi.org/10.1145/97880.97899

Digital Library

[13]

Steve Bryson. 1996. Virtual Reality in Scientific Visualization. COMMUNICATIONS OF THE ACM 39, 5 (1996).

[14]

Fabio Buttussi and Luca Chittaro. 2023. Acquisition and retention of spatial knowledge through virtual reality experiences: Effects of VR setup and locomotion technique. International Journal of Human-Computer Studies 177 (Sept. 2023), 103067. https://doi.org/10.1016/j.ijhcs.2023.103067

Digital Library

[15]

Kelly Caine. 2016. Local Standards for Sample Size at CHI. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. ACM, San Jose California USA, 981–992. https://doi.org/10.1145/2858036.2858498

Digital Library

[16]

Zaira Cattaneo and Tomaso Vecchi. 2008. Supramodality effects in visual and haptic spatial processes.Journal of Experimental Psychology: Learning, Memory, and Cognition 34, 3 (2008), 631–642. https://doi.org/10.1037/0278-7393.34.3.631

[17]

Francesco Chiossi, Thomas Kosch, Luca Menghini, Steeven Villa, and Sven Mayer. 2023. SensCon: Embedding Physiological Sensing into Virtual Reality Controllers. Proc. ACM Hum.-Comput. Interact. 7, MHCI, Article 223 (sep 2023), 32 pages. https://doi.org/10.1145/3604270

Digital Library

[18]

Matt Craddock and Rebecca Lawson. 2009. Size-Sensitive Perceptual Representations Underlie Visual and Haptic Object Recognition. PLoS ONE 4, 11 (Nov. 2009), e8009. https://doi.org/10.1371/journal.pone.0008009

[19]

Carolina Cruz-Neira, Daniel J. Sandin, and Thomas A. DeFanti. 1993. Surround-screen projection-based virtual reality: the design and implementation of the CAVE. In Proceedings of the 20th annual conference on Computer graphics and interactive techniques. ACM, Anaheim CA, 135–142. https://doi.org/10.1145/166117.166134

Digital Library

[20]

John A. Cunningham, Imran A. Rahman, Stephan Lautenschlager, Emily J. Rayfield, and Philip C.J. Donoghue. 2014. A virtual world of paleontology. Trends in Ecology & Evolution 29, 6 (June 2014), 347–357. https://doi.org/10.1016/j.tree.2014.04.004 Number: 6.

[21]

Thomas A. Daniel, Jeffrey S. Katz, and Jennifer L. Robinson. 2016. Delayed match-to-sample in working memory: A BrainMap meta-analysis. Biological Psychology 120 (Oct. 2016), 10–20. https://doi.org/10.1016/j.biopsycho.2016.07.015

[22]

Geneviève Desmarais, Melissa Meade, Taylor Wells, and Mélanie Nadeau. 2017. Visuo-haptic integration in object identification using novel objects. Attention, Perception, & Psychophysics 79, 8 (Nov. 2017), 2478–2498. https://doi.org/10.3758/s13414-017-1382-x

[23]

Mathilde R. Desselle, Ross A. Brown, Allan R. James, Mark J. Midwinter, Sean K. Powell, and Maria A. Woodruff. 2020. Augmented and Virtual Reality in Surgery. Computing in Science & Engineering 22, 3 (May 2020), 18–26. https://doi.org/10.1109/MCSE.2020.2972822

[24]

Randolph D. Easton, Anthony J. Greene, and Kavitha Srinivas. 1997. Transfer between vision and haptics: Memory for 2-D patterns and 3-D objects. Psychonomic Bulletin & Review 4, 3 (Sept. 1997), 403–410. https://doi.org/10.3758/BF03210801

[25]

Randolph D. Easton, Kavitha Srinivas, and Anthony J. Greene. 1997. Do vision and haptics share common representations? Implicit and explicit memory within and between modalities.Journal of Experimental Psychology: Learning, Memory, and Cognition 23, 1 (1997), 153–163. https://doi.org/10.1037/0278-7393.23.1.153

[26]

Gerald Edelman. 1987. Neural Darwinism. New Perspectives Quarterly 21, 3 (1987).

[27]

Johannes Engelkamp. 1995. Visual imagery and enactment of actions in memory. British Journal of Psychology 86, 2 (May 1995), 227–240. https://doi.org/10.1111/j.2044-8295.1995.tb02558.x

[28]

Johannes Engelkamp and H. D. Zimmer. 1994. Human memory: a multimodal approach. Hogrefe & Huber Publishers, Seattle.

[29]

Mark O. Ernst, Christoph Lange, and Fiona N. Newell. 2007. Multisensory recognition of actively explored objects.Canadian Journal of Experimental Psychology / Revue canadienne de psychologie expérimentale 61, 3 (Sept. 2007), 242–253. https://doi.org/10.1037/cjep2007025

[30]

Balázs Faludi, Esther I. Zoller, Nicolas Gerig, Azhar Zam, Georg Rauter, and Philippe C. Cattin. 2019. Direct Visual and Haptic Volume Rendering of Medical Data Sets for an Immersive Exploration in Virtual Reality. In Medical Image Computing and Computer Assisted Intervention – MICCAI 2019, Dinggang Shen, Tianming Liu, Terry M. Peters, Lawrence H. Staib, Caroline Essert, Sean Zhou, Pew-Thian Yap, and Ali Khan (Eds.). Vol. 11768. Springer International Publishing, Cham, 29–37. http://link.springer.com/10.1007/978-3-030-32254-0_4

[31]

Nina Gaissert and Christian Wallraven. 2012. Categorizing natural objects: a comparison of the visual and the haptic modalities. Experimental Brain Research 216, 1 (Jan. 2012), 123–134. https://doi.org/10.1007/s00221-011-2916-4

[32]

Tara Renee Ghazi and Susan Courtney. 2022. Bimodal Spatial and Temporal Influences on Working Memory Performance. preprint. PsyArXiv. https://doi.org/10.31234/osf.io/492vw

[33]

George Ghinea, Frederic Andres, and Stephen R. Gulliver (Eds.). 2012. Multisensory presence in virtual reality: Possibilities and limitations. IGI Global. https://doi.org/10.4018/978-1-60960-821-7

[34]

James J. Gibson. 1962. Observations on active touch. Psychological Review 69, 6 (Nov. 1962), 477–491. https://doi.org/10.1037/h0046962

[35]

D. R. Godden and A. D. Baddeley. 1975. Context-dependent memory in two natural environments: On land and underwater. British Journal of Psychology 66, 3 (Aug. 1975), 325–331. https://doi.org/10.1111/j.2044-8295.1975.tb01468.x

[36]

Christian Grefkes, Peter H. Weiss, Karl Zilles, and Gereon R. Fink. 2002. Crossmodal Processing of Object Features in Human Anterior Intraparietal Cortex. Neuron 35, 1 (July 2002), 173–184. https://doi.org/10.1016/S0896-6273(02)00741-9

[37]

Martin Grunwald (Ed.). 2008. Human haptic perception: basics and applications. Birkhäuser, Basel Boston Berlin.

[38]

D. Hamilton, J. McKechnie, E. Edgerton, and C. Wilson. 2021. Immersive virtual reality as a pedagogical tool in education: a systematic literature review of quantitative learning outcomes and experimental design. Journal of Computers in Education 8, 1 (March 2021), 1–32. https://doi.org/10.1007/s40692-020-00169-2

[39]

Karin L. Harman, G.Keith Humphrey, and Melvyn A. Goodale. 1999. Active manual control of object views facilitates visual recognition. Current Biology 9, 22 (Nov. 1999), 1315–1318. https://doi.org/10.1016/S0960-9822(00)80053-6

[40]

Stephenie A. Harrison and Frank Tong. 2009. Decoding reveals the contents of visual working memory in early visual areas. Nature 458, 7238 (April 2009), 632–635. https://doi.org/10.1038/nature07832

[41]

Carrie Heeter. 1992. Being There: The Subjective Experience of Presence. Presence: Teleoperators and Virtual Environments 1, 2 (Jan. 1992), 262–271. https://doi.org/10.1162/pres.1992.1.2.262

[42]

H. Iwata and H. Noma. 1993. Volume haptization. In Proceedings of 1993 IEEE Research Properties in Virtual Reality Symposium. IEEE Comput. Soc. Press, San Jose, CA, USA, 16–23. https://doi.org/10.1109/VRAIS.1993.378268

[43]

Thomas W. James, G.Keith Humphrey, Joseph S. Gati, Philip Servos, Ravi S. Menon, and Melvyn A. Goodale. 2002. Haptic study of three-dimensional objects activates extrastriate visual areas. Neuropsychologia 40, 10 (Jan. 2002), 1706–1714. https://doi.org/10.1016/S0028-3932(02)00017-9

[44]

Yvonne Jansen, Pierre Dragicevic, and Jean-Daniel Fekete. 2013. Evaluating the efficiency of physical visualizations. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, Paris France, 2593–2602. https://doi.org/10.1145/2470654.2481359

Digital Library

[45]

Gunnar Jansson and Anna Ivås. 2001. Can the efficiency of a haptic display be increased by short-time practice in exploration? In Haptic Human-Computer Interaction, Gerhard Goos, Juris Hartmanis, Jan van Leeuwen, Stephen Brewster, and Roderick Murray-Smith (Eds.). Vol. 2058. Springer Berlin Heidelberg, Berlin, Heidelberg, 88–97. https://doi.org/10.1007/3-540-44589-7_10 Series Title: Lecture Notes in Computer Science.

[46]

Bill Jones. 1981. The Developmental Significance of Cross-Modal Matching. In Intersensory Perception and Sensory Integration, Richard D. Walk and Herbert L. Pick (Eds.). Springer US, Boston, MA, 109–136. https://doi.org/10.1007/978-1-4615-9197-9_4

[47]

Martin Juttner, Erol Osman, and Ingo Rentschler. 2001. When touch forms vision - object recognition as a function of polysensory prior knowledge. Ph. D. Dissertation. Neuroscience Research Institute, School of Life & Health Sciences, Aston University, United Kingdom.

[48]

Amanda L. Kaas, Hanneke I. van Mier, Johan Lataster, Mirella Fingal, and Alexander T. Sack. 2007. The effect of visuo-haptic congruency on haptic spatial matching. Experimental Brain Research 183, 1 (Oct. 2007), 75–85. https://doi.org/10.1007/s00221-007-1026-9

[49]

Kapil Kadam, Sameer Sahasrabudhe, and Sridhar Iyer. 2012. Improvement of Mental Rotation Ability Using Blender 3-D. In 2012 IEEE Fourth International Conference on Technology for Education. IEEE, Hyderabad, India, 60–66. https://doi.org/10.1109/T4E.2012.28

Digital Library

[50]

Solène Kalenine, Leatitia Pinet, and Edouard Gentaz. 2011. The visual and visuo-haptic exploration of geometrical shapes increases their recognition in preschoolers. International Journal of Behavioral Development 35, 1 (Jan. 2011), 18–26. https://doi.org/10.1177/0165025410367443

[51]

Tanja Kassuba, Corinna Klinge, Cordula Hölig, Brigitte Röder, and Hartwig R. Siebner. 2013. Vision holds a greater share in visuo-haptic object recognition than touch. NeuroImage 65 (Jan. 2013), 59–68. https://doi.org/10.1016/j.neuroimage.2012.09.054

[52]

Roberta L. Klatzky, Susan J. Lederman, and Victoria A. Metzger. 1985. Identifying objects by touch: An “expert system”. Perception & Psychophysics 37, 4 (July 1985), 299–302. https://doi.org/10.3758/BF03211351

[53]

Elif Hilal Korkut and Elif Surer. 2023. Visualization in virtual reality: a systematic review. Virtual Reality 27, 2 (June 2023), 1447–1480. https://doi.org/10.1007/s10055-023-00753-8

Digital Library

[54]

Thomas Kosch, Jakob Karolus, Havy Ha, and Albrecht Schmidt. 2019. Your skin resists: exploring electrodermal activity as workload indicator during manual assembly. In Proceedings of the ACM SIGCHI Symposium on Engineering Interactive Computing Systems (Valencia, Spain) (EICS ’19). Association for Computing Machinery, New York, NY, USA, Article 8, 5 pages. https://doi.org/10.1145/3319499.3328230

Digital Library

[55]

Thomas Kosch, Jakob Karolus, Johannes Zagermann, Harald Reiterer, Albrecht Schmidt, and Paweł W. Woźniak. 2023. A Survey on Measuring Cognitive Workload in Human-Computer Interaction. Comput. Surveys 55, 13s, Article 283 (2023), 39 pages. https://doi.org/10.1145/3582272

Digital Library

[56]

Thomas Kosch, Albrecht Schmidt, Simon Thanheiser, and Lewis L. Chuang. 2020. One does not Simply RSVP: Mental Workload to Select Speed Reading Parameters using Electroencephalography. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI ’20). Association for Computing Machinery, New York, NY, USA, 1–13. https://doi.org/10.1145/3313831.3376766

Digital Library

[57]

Matthias Kraus, Karsten Klein, Johannes Fuchs, Daniel Keim, Falk Schreiber, and Michael Sedlmair. 2021. The Value of Immersive Visualization. IEEE Computer Graphics and Applications 41, 4 (July 2021), 125–132. https://doi.org/10.1109/MCG.2021.3075258

[58]

Julian Kreimeier, Sebastian Hammer, Daniel Friedmann, Pascal Karg, Clemens Bühner, Lukas Bankel, and Timo Götzelmann. 2019. Evaluation of different types of haptic feedback influencing the task-based presence and performance in virtual reality. In Proceedings of the 12th ACM International Conference on PErvasive Technologies Related to Assistive Environments. ACM, Rhodes Greece, 289–298. https://doi.org/10.1145/3316782.3321536

Digital Library

[59]

Eric Krokos, Catherine Plaisant, and Amitabh Varshney. 2019. Virtual memory palaces: immersion aids recall. Virtual Reality 23, 1 (March 2019), 1–15. https://doi.org/10.1007/s10055-018-0346-3

Digital Library

[60]

Stacey Kuznetsov, Anind K. Dey, and Scott E. Hudson. 2009. The Effectiveness of Haptic Cues as an Assistive Technology for Human Memory. In Pervasive Computing, Hideyuki Tokuda, Michael Beigl, Adrian Friday, A. J. Bernheim Brush, and Yoshito Tobe (Eds.). Vol. 5538. Springer Berlin Heidelberg, Berlin, Heidelberg, 168–175. https://doi.org/10.1007/978-3-642-01516-8_12 Series Title: Lecture Notes in Computer Science.

Digital Library

[61]

Patrick Kyllonen and Jiyun Zu. 2016. Use of Response Time for Measuring Cognitive Ability. Journal of Intelligence 4, 4 (Nov. 2016), 14. https://doi.org/10.3390/jintelligence4040014

[62]

Simon Lacey and Christine Campbell. 2006. Mental representation in visual/haptic crossmodal memory: evidence from interference effects. Quarterly Journal of Experimental Psychology 59, 2 (Feb. 2006), 361–376. https://doi.org/10.1080/17470210500173232

[63]

Simon Lacey, Christine Campbell, and K Sathian. 2007. Vision and Touch: Multiple or Multisensory Representations of Objects?Perception 36, 10 (Oct. 2007), 1513–1521. https://doi.org/10.1068/p5850

[64]

Simon Lacey and K. Sathian. 2011. Multisensory object representation: Insights from studies of vision and touch. In Progress in Brain Research. Vol. 191. Elsevier, Amsterdam, 165–176. https://doi.org/10.1016/B978-0-444-53752-2.00006-0

[65]

Simon Lacey and K. Sathian. 2014. Visuo-haptic multisensory object recognition, categorization, and representation. Frontiers in Psychology 5 (July 2014). https://doi.org/10.3389/fpsyg.2014.00730

[66]

Christophe Lalanne and Jean Lorenceau. 2004. Crossmodal integration for perception and action. Journal of Physiology-Paris 98, 1-3 (Jan. 2004), 265–279. https://doi.org/10.1016/j.jphysparis.2004.06.001

[67]

Maarten H. Lamers and Maik Lanen. 2021. Changing Between Virtual Reality and Real-World Adversely Affects Memory Recall Accuracy. Frontiers in Virtual Reality 2 (March 2021), 602087. https://doi.org/10.3389/frvir.2021.602087

[68]

Joseph J. LaViola, Ernst Kruijff, Ryan P. McMahan, Doug Bowman, and Ivan P. Poupyrev. 2017. 3D User Interfaces: Theory and Practice (2 ed.). Addison-Wesley, Boston.

[69]

D.A. Lawrence, L.Y. Pao, C.D. Lee, and R.Y. Novoselov. 2004. Synergistic visual/haptic rendering modes for scientific visualization. IEEE Computer Graphics and Applications 24, 6 (Nov. 2004), 22–30. https://doi.org/10.1109/MCG.2004.60

Digital Library

[70]

Susan J Lederman and Roberta L Klatzky. 1987. Hand movements: A window into haptic object recognition. Cognitive Psychology 19, 3 (July 1987), 342–368. https://doi.org/10.1016/0010-0285(87)90008-9

[71]

Susan J. Lederman, Roberta L. Klatzky, Cynthia Chataway, and Craig D. Summers. 1990. Visual mediation and the haptic recognition of two-dimensional pictures of common objects. Perception & Psychophysics 47, 1 (Jan. 1990), 54–64. https://doi.org/10.3758/BF03208164

[72]

Susan J Lederman, Craig Summers, and Roberta L Klatzky. 1996. Cognitive Salience of Haptic Object Properties: Role of Modality-Encoding Bias. Perception 25, 8 (Aug. 1996), 983–998. https://doi.org/10.1068/p250983

[73]

David J. Lewkowicz and Robert Lickliter. 1994. The development of intersensory perception: comparative perspectives. L. Erlbaum, Hillsdale, N.J.OCLC: 844923031.

[74]

Yu Liu and Stephen D. Laycock. 2009. A Haptic System for Drilling into Volume Data with Polygonal Tools. Theory and Practice of Computer Graphics (2009), 8 pages. https://doi.org/10.2312/LOCALCHAPTEREVENTS/TPCG/TPCG09/009-016 Artwork Size: 8 pages ISBN: 9783905673715 Publisher: The Eurographics Association.

[75]

Jack M. Loomis. 1982. Analysis of tactile and visual confusion matrices. Perception & Psychophysics 31, 1 (Jan. 1982), 41–52. https://doi.org/10.3758/BF03206199

[76]

Jack M Loomis, Roberta L Klatzky, and Susan J Lederman. 1991. Similarity of Tactual and Visual Picture Recognition with Limited Field of View. Perception 20, 2 (April 1991), 167–177. https://doi.org/10.1068/p200167

[77]

Neal Madras and Alan D. Sokal. 1988. The pivot algorithm: A highly efficient Monte Carlo method for the self-avoiding walk. Journal of Statistical Physics 50, 1-2 (Jan. 1988), 109–186. https://doi.org/10.1007/BF01022990

[78]

Arthur Maneuvrier, Leslie Marion Decker, Hadrien Ceyte, Philippe Fleury, and Patrice Renaud. 2020. Presence Promotes Performance on a Virtual Spatial Cognition Task: Impact of Human Factors on Virtual Reality Assessment. Frontiers in Virtual Reality 1 (Oct. 2020), 571713. https://doi.org/10.3389/frvir.2020.571713

[79]

Thomas H. Massie and Ken Salisbury. 1994. The PHANTOM Haptic Interface- A Device for Probing Virtual Objects. Proceedings of the ASME Winter Annual Meeting, Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems (1994), 6.

[80]

Jon McCormack, Jonathan C. Roberts, Benjamin Bach, Carla Dal Sasso Freitas, Takayuki Itoh, Christophe Hurter, and Kim Marriott. 2018. Multisensory Immersive Analytics. In Immersive Analytics, Kim Marriott, Falk Schreiber, Tim Dwyer, Karsten Klein, Nathalie Henry Riche, Takayuki Itoh, Wolfgang Stuerzlinger, and Bruce H. Thomas (Eds.). Vol. 11190. Springer International Publishing, Cham, 57–94. https://doi.org/10.1007/978-3-030-01388-2_3 Series Title: Lecture Notes in Computer Science.

[81]

B. H. McCormick. 1988. Visualization in scientific computing. ACM SIGBIO Newsletter 10, 1 (March 1988), 15–21. https://doi.org/10.1145/43965.43966

Digital Library

[82]

Andreas H Meier, Chantal L Rawn, and Thomas M Krummel. 2001. Virtual Reality: Surgical Application— Challenge for The New Millennium. Journal of the American College of Surgeons 192, 3 (March 2001), 372–384. https://doi.org/10.1016/S1072-7515(01)00769-4

[83]

E. Mendez, S. Yoshida, H. Noma, R.W. Lindeman, Y. Yanagida, S. Masaki, and K. Hosaka. 2005. Haptic-Assisted Guidance System for Navigating Volumetric Data Sets. In First Joint Eurohaptics Conference and Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems. IEEE, Pisa, Italy, 531–534.

[84]

George Miller, Galanter Eugene, and karl Pribram. 1968. Plans and the Structure of Behaviour. In Systems research for behavioral science. Routledge, London. OCLC: 1000604284.

[85]

Paul Miller and Julian Richards. 1995. The good, the bad, and the downright misleading: archaeological adoption of computer visualization. (1995).

[86]

A. Miquée, C. Xerri, C. Rainville, J.-L. Anton, B. Nazarian, M. Roth, and Y. Zennou-Azogui. 2008. Neuronal substrates of haptic shape encoding and matching: A functional magnetic resonance imaging study. Neuroscience 152, 1 (March 2008), 29–39. https://doi.org/10.1016/j.neuroscience.2007.12.021

[87]

Jules Moloney, Branka Spehar, Anastasia Globa, and Rui Wang. 2018. The affordance of virtual reality to enable the sensory representation of multi-dimensional data for immersive analytics: from experience to insight. Journal of Big Data 5, 1 (Dec. 2018), 53. https://doi.org/10.1186/s40537-018-0158-z

[88]

Guillaume Moreau. 2013. Visual Immersion Issues in Virtual Reality: A Survey. In 2013 26th Conference on Graphics, Patterns and Images Tutorials. IEEE, Arequipa, Peru, 6–14. https://doi.org/10.1109/SIBGRAPI-T.2013.9

Digital Library

[89]

Fiona N. Newell. 2010. Visuo-haptic Perception of Objects and Scenes. In Multisensory Object Perception in the Primate Brain, Jochen Kaiser and Marcus Johannes Naumer (Eds.). Springer New York, New York, NY, 251–271. https://doi.org/10.1007/978-1-4419-5615-6_14

[90]

Fiona N. Newell, Marc O. Ernst, Bosco S. Tjan, and Heinrich H. Bülthoff. 2001. Viewpoint Dependence in Visual and Haptic Object Recognition. Psychological Science 12, 1 (Jan. 2001), 37–42. https://doi.org/10.1111/1467-9280.00307

[91]

Karljohan Lundin Palmerius and Camilla Forsell. 2009. The impact of feedback design in haptic volume visualization. In World Haptics 2009 - Third Joint EuroHaptics conference and Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems. IEEE, Salt Lake City, UT, USA, 154–159. https://doi.org/10.1109/WHC.2009.4810829

Digital Library

[92]

Tommaso Palombi, Federica Galli, Francesco Giancamilli, Monica D’Amico, Fabio Alivernini, Luigi Gallo, Pietro Neroni, Marco Predazzi, Giuseppe De Pietro, Fabio Lucidi, Antonio Giordano, and Andrea Chirico. 2023. The role of sense of presence in expressing cognitive abilities in a virtual reality task: an initial validation study. Scientific Reports 13, 1 (Aug. 2023), 13396. https://doi.org/10.1038/s41598-023-40510-0

[93]

P.J. Passmore, C.F. Nielsen, W.J. Cosh, and A. Darzi. 2001. Effects of viewing and orientation on path following in a medical teleoperation environment. In Proceedings IEEE Virtual Reality 2001. IEEE Comput. Soc, Yokohama, Japan, 209–215. https://doi.org/10.1109/VR.2001.913788

[94]

Randy Pausch, Dennis Proffitt, and George Williams. 1997. Quantifying immersion in virtual reality. In Proceedings of the 24th annual conference on Computer graphics and interactive techniques - SIGGRAPH ’97. ACM Press, Not Known, 13–18. https://doi.org/10.1145/258734.258744

Digital Library

[95]

W. A. Phillips and D. F. M. Christie. 1977. Interference with Visualization. Quarterly Journal of Experimental Psychology 29, 4 (Nov. 1977), 637–650. https://doi.org/10.1080/14640747708400638

[96]

Csaba Pinter, Andras Lasso, Saleh Choueib, Mark Asselin, Jean-Christophe Fillion-Robin, Jean-Baptiste Vimort, Ken Martin, Matthew A. Jolley, and Gabor Fichtinger. 2020. SlicerVR for Medical Intervention Training and Planning in Immersive Virtual Reality. IEEE Transactions on Medical Robotics and Bionics 2, 2 (May 2020), 108–117. https://doi.org/10.1109/TMRB.2020.2983199

[97]

Paul Reilly and S. P. Q. Rahtz. 1992. Visualizing spatial data: the importance of Geographic Information Systems. In Archaeology and the information age: a global perspective. Routledge, London. OCLC: 54663493.

[98]

He Ren and Eva Hornecker. 2021. Comparing Understanding and Memorization in Physicalization and VR Visualization. In Proceedings of the Fifteenth International Conference on Tangible, Embedded, and Embodied Interaction. ACM, Salzburg Austria, 1–7. https://doi.org/10.1145/3430524.3442446

Digital Library

[99]

Irvin Rock and Jack Victor. 1964. Vision and Touch: An Experimentally Created Conflict between the Two Senses. Science 143, 3606 (Feb. 1964), 594–596. https://doi.org/10.1126/science.143.3606.594

[100]

Lucas Siqueira Rodrigues, John Nyakatura, Stefan Zachow, and Johann Habakuk Israel. 2022. An Immersive Virtual Paleontology Application. In 13th International Conference on Human Haptic Sensing and Touch Enabled Computer Applications, EuroHaptics 2022. Springer International Publishing, Berlin, Heidelberg, 478 – 481. https://doi.org/10.1007/978-3-031-06249-0

Digital Library

[101]

Lucas Siqueira Rodrigues, Felix Riehm, Stefan Zachow, and Johann Habakuk Israel. 2023. VoxSculpt: An Open-Source Voxel Library for Tomographic Volume Sculpting in Virtual Reality. In 2023 9th International Conference on Virtual Reality (ICVR). 515–523. https://doi.org/10.1109/ICVR57957.2023.10169420 ISSN: 2331-9569.

[102]

Ranulfo Romo, Carlos D. Brody, Adrián Hernández, and Luis Lemus. 1999. Neuronal correlates of parametric working memory in the prefrontal cortex. Nature 399, 6735 (June 1999), 470–473. https://doi.org/10.1038/20939

[103]

Joan Sol Roo, Jean Basset, Pierre-Antoine Cinquin, and Martin Hachet. 2018. Understanding Users’ Capability to Transfer Information between Mixed and Virtual Reality: Position Estimation across Modalities and Perspectives. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. ACM, Montreal QC Canada, 1–12. https://doi.org/10.1145/3173574.3173937

Digital Library

[104]

Diego C. Ruspini, Krasimir Kolarov, and Oussama Khatib. 1997. The haptic display of complex graphical environments. In Proceedings of the 24th annual conference on Computer graphics and interactive techniques - SIGGRAPH ’97. ACM Press, Not Known, 345–352. https://doi.org/10.1145/258734.258878

Digital Library

[105]

Annie Rydström. [n. d.]. The effect of haptic feedback in visual-manual human-machine interaction. ([n. d.]).

[106]

David Saffo, Sara Di Bartolomeo, Tarik Crnovrsanin, Laura South, Justin Raynor, Caglar Yildirim, and Cody Dunne. 2024. Unraveling the Design Space of Immersive Analytics: A Systematic Review. IEEE Transactions on Visualization and Computer Graphics 30, 1 (Jan. 2024), 495–506. https://doi.org/10.1109/TVCG.2023.3327368 Conference Name: IEEE Transactions on Visualization and Computer Graphics.

Digital Library

[107]

Anahita Sanandaji, Cindy Grimm, Ruth West, and Christopher A. Sanchez. 2023. Developing and Validating a Computer-Based Training Tool for Inferring 2D Cross-Sections of Complex 3D Structures. Human Factors: The Journal of the Human Factors and Ergonomics Society 65, 3 (May 2023), 508–528. https://doi.org/10.1177/00187208211018110

[108]

Timo Torsten Schmidt and Felix Blankenburg. 2018. Brain regions that retain the spatial layout of tactile stimuli during working memory – A ‘tactospatial sketchpad’?NeuroImage 178 (Sept. 2018), 531–539. https://doi.org/10.1016/j.neuroimage.2018.05.076

[109]

Timo Torsten Schmidt, Yuan-hao Wu, and Felix Blankenburg. 2017. Content-Specific Codes of Parametric Vibrotactile Working Memory in Humans. The Journal of Neuroscience 37, 40 (Oct. 2017), 9771–9777. https://doi.org/10.1523/JNEUROSCI.1167-17.2017

[110]

Katie Seaborn, Bernhard E. Riecke, and Alissa N. Antle. 2010. Exploring the interplay of visual and haptic modalities in a pattern-matching task. In 2010 IEEE International Symposium on Haptic Audio Visual Environments and Games. IEEE, Phoenix, AZ, USA, 1–6. https://doi.org/10.1109/HAVE.2010.5623997

[111]

Nikhil Shetty, Aashish Chaudhary, Daniel Coming, William R. Sherman, Patrick O’Leary, Eric T. Whiting, and Simon Su. 2011. Immersive ParaView: A community-based, immersive, universal scientific visualization application. In 2011 IEEE Virtual Reality Conference. IEEE, Singapore, Singapore, 239–240.

Digital Library

[112]

Yeon Soon Shin, Rolando Masís-Obando, Neggin Keshavarzian, Riya Dáve, and Kenneth A. Norman. 2021. Context-dependent memory effects in two immersive virtual reality environments: On Mars and underwater. Psychonomic Bulletin & Review 28, 2 (April 2021), 574–582. https://doi.org/10.3758/s13423-020-01835-3

[113]

R.M. Simpson, J.J. LaViola, D.H. Laidlaw, A.S. Forsberg, and A. Van Dam. 2000. Immersive VR for scientific visualization: a progress report. IEEE Computer Graphics and Applications 20, 6 (Dec. 2000), 26–52. https://doi.org/10.1109/38.888006

Digital Library

[114]

Lucas Siqueira Rodrigues, John Nyakatura, Stefan Zachow, and Johann Habakuk Israel. 2023. Design Challenges and Opportunities of Fossil Preparation Tools and Methods. In Proceedings of the 20th International Conference on Culture and Computer Science: Code and Materiality. ACM, Lisbon Portugal, 1–10. https://doi.org/10.1145/3623462.3623470

Digital Library

[115]

Linda Smith and Michael Gasser. 2005. The Development of Embodied Cognition: Six Lessons from Babies. Artificial Life 11, 1-2 (Jan. 2005), 13–29. https://doi.org/10.1162/1064546053278973

Digital Library

[116]

S. Adam Smith and Neil W. Mulligan. 2021. Immersion, presence, and episodic memory in virtual reality environments. Memory 29, 8 (Sept. 2021), 983–1005. https://doi.org/10.1080/09658211.2021.1953535

[117]

Charles Spence and Jon Driver (Eds.). 2004. Crossmodal Space and Crossmodal Attention. Oxford University Press. https://doi.org/10.1093/acprof:oso/9780198524861.001.0001

[118]

W. David Stahlman, Mercedes McWaters, Erica Christian, Eric Knapp, Andrea Fritch, and Jennifer R. Mailloux. 2018. Overshadowing between visual and tactile stimulus elements in an object recognition task. Behavioural Processes 157 (Dec. 2018), 102–105. https://doi.org/10.1016/j.beproc.2018.08.008

[119]

Detlev Stalling, Malte Westerhoff, and Hans-Christian Hege. 2005. Amira: A Highly Interactive System for Visual Data Analysis. In Visualization Handbook. Elsevier, 749–767. https://doi.org/10.1016/B978-012387582-2/50040-X

[120]

Barry E. Stein and M. Alex Meredith. 1993. The merging of the senses. MIT Press, Cambridge, Mass London.

[121]

Barry E. Stein and Terrence R. Stanford. 2008. Multisensory integration: current issues from the perspective of the single neuron. Nature Reviews Neuroscience 9, 4 (April 2008), 255–266. https://doi.org/10.1038/nrn2331

[122]

Steven A Wall and William S Harwin. 2000. Quantification of the Effects of Haptic Feedback During a Motor Skills Task in a Simulated Environment. PURS 2000: 2nd PHANToM Users Research Symposium (2000).

[123]

Maarten W. A. Wijntjes, Robert Volcic, Sylvia C. Pont, Jan J. Koenderink, and Astrid M. L. Kappers. 2009. Haptic perception disambiguates visual perception of 3D shape. Experimental Brain Research 193, 4 (March 2009), 639–644. https://doi.org/10.1007/s00221-009-1713-9

[124]

Katrin Wolf, Marco Kurzweg, Yannick Weiss, Stephen Brewster, and Albrecht Schmidt. 2022. Visuo-Haptic Interaction. In Proceedings of the 2022 International Conference on Advanced Visual Interfaces. ACM, Frascati, Rome Italy, 1–4. https://doi.org/10.1145/3531073.3535260

Digital Library

[125]

Andrew T. Woods and Fiona N. Newell. 2004. Visual, haptic and cross-modal recognition of objects and scenes. Journal of Physiology-Paris 98, 1-3 (Jan. 2004), 147–159. https://doi.org/10.1016/j.jphysparis.2004.03.006

[126]

Maryjane Wraga, Sarah H. Creem-Regehr, and Dennis R. Proffitt. 2004. Spatial updating of virtual displays during self- and display rotation. Memory & Cognition 32, 3 (April 2004), 399–415. https://doi.org/10.3758/BF03195834

[127]

David E. Wright and Ian Dennis. 1999. Exploiting the speed-accuracy trade-off. In Learning and individual differences: Process, trait, and content determinants., Phillip L. Ackerman, Patrick C. Kyllonen, and Richard D. Roberts (Eds.). American Psychological Association, Washington, 231–248. https://doi.org/10.1037/10315-010

[128]

Yuan-hao Wu, Işıl Uluç, Timo Torsten Schmidt, Kathrin Tertel, Evgeniya Kirilina, and Felix Blankenburg. 2018. Overlapping frontoparietal networks for tactile and visual parametric working memory representations. NeuroImage 166 (Feb. 2018), 325–334. https://doi.org/10.1016/j.neuroimage.2017.10.059

[129]

Andro Zangaladze, Charles M. Epstein, Scott T. Grafton, and K. Sathian. 1999. Involvement of visual cortex in tactile discrimination of orientation. Nature 401, 6753 (Oct. 1999), 587–590. https://doi.org/10.1038/44139

[130]

C.B. Zilles and J.K. Salisbury. 1995. A constraint-based god-object method for haptic display. In Proceedings 1995 IEEE/RSJ International Conference on Intelligent Robots and Systems. Human Robot Interaction and Cooperative Robots, Vol. 3. IEEE Comput. Soc. Press, Pittsburgh, PA, USA, 146–151. https://doi.org/10.1109/IROS.1995.525876

Cited By

Siqueira Rodrigues LSchmidt TNyakatura JZachow SIsrael JKosch T(2024)Assessing the Effects of Sensory Modality Conditions on Object Retention across Virtual Reality and Projected Surface Display EnvironmentsProceedings of the ACM on Human-Computer Interaction10.1145/36981378:ISS(255-282)Online publication date: 24-Oct-2024
https://dl.acm.org/doi/10.1145/3698137

Index Terms

Comparing the Effects of Visual, Haptic, and Visuohaptic Encoding on Memory Retention of Digital Objects in Virtual Reality
1. Human-centered computing
  1. Human computer interaction (HCI)

Recommendations

Assessing the Effects of Sensory Modality Conditions on Object Retention across Virtual Reality and Projected Surface Display Environments

Haptic feedback reportedly enhances human interaction with 3D data, particularly improving the retention of mental representations of digital objects in immersive settings. However, the effectiveness of visuohaptic integration in promoting object ...
Haptic Fidelity Framework: Defining the Factors of Realistic Haptic Feedback for Virtual Reality
CHI '22: Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems

Providing haptic feedback in virtual reality to make the experience more realistic has become a strong focus of research in recent years. The resulting haptic feedback systems differ greatly in their technologies, feedback possibilities, and overall ...
Enabling Reusable Haptic Props for Virtual Reality by Hand Displacement
MuC '21: Proceedings of Mensch und Computer 2021

Virtual Reality (VR) enables compelling visual experiences. However, providing haptic feedback is still challenging. Previous work suggests utilizing haptic props to overcome such limitations and presents evidence that props could function as a single ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Other conferences

NordiCHI '24: Proceedings of the 13th Nordic Conference on Human-Computer Interaction

October 2024

1236 pages

ISBN:9798400709661

DOI:10.1145/3679318

Copyright © 2024 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 13 October 2024

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Funding Sources

Deutsche Forschungsgemeinschaft

Conference

NordiCHI 2024

NordiCHI 2024: Nordic Conference on Human-Computer Interaction

October 13 - 16, 2024

Uppsala, Sweden

Acceptance Rates

Overall Acceptance Rate 379 of 1,572 submissions, 24%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1
Total Citations
View Citations
46
Total Downloads

Downloads (Last 12 months)46
Downloads (Last 6 weeks)23

Reflects downloads up to 06 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Siqueira Rodrigues LSchmidt TNyakatura JZachow SIsrael JKosch T(2024)Assessing the Effects of Sensory Modality Conditions on Object Retention across Virtual Reality and Projected Surface Display EnvironmentsProceedings of the ACM on Human-Computer Interaction10.1145/36981378:ISS(255-282)Online publication date: 24-Oct-2024
https://dl.acm.org/doi/10.1145/3698137

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Table of Contents