More Web Proxy on the site http://driver.im/

research-article

Towards Tangible Cultural Heritage Experiences—Enriching VR-based Object Inspection with Haptic Feedback

Authors:

Stefan Krumpen,

Reinhard Klein,

Michael WeinmannAuthors Info & Claims

ACM Journal on Computing and Cultural Heritage (JOCCH), Volume 15, Issue 1

Article No.: 19, Pages 1 - 17

https://doi.org/10.1145/3470470

Published: 14 December 2021 Publication History

Abstract

VR/AR technology is a key enabler for new ways of immersively experiencing cultural heritage artifacts based on their virtual counterparts obtained from a digitization process. In this article, we focus on enriching VR-based object inspection by additional haptic feedback, thereby creating tangible cultural heritage experiences. For this purpose, we present an approach for interactive and collaborative VR-based object inspection and annotation. Our system supports high-quality 3D models with accurate reflectance characteristics while additionally providing haptic feedback regarding shape features of the object based on a 3D printed replica. The digital object model in terms of a printable representation of the geometry as well as reflectance characteristics are stored in a compact and streamable representation on a central server, which streams the data to remotely connected users/clients. The latter can jointly perform an interactive inspection of the object in VR with additional haptic feedback through the 3D printed replica. Evaluations regarding system performance, visual quality of the considered models, as well as insights from a user study indicate an improved interaction, assessment, and experience of the considered objects.

Supplementary Material

krumpen (krumpen.zip)

Supplemental movie, appendix, image and software files for, Towards Tangible Cultural Heritage Experiences—Enriching VR-based Object Inspection with Haptic Feedback

Download
64.98 KB

References

[1]

M. K. Bekele, R. Pierdicca, E. Frontoni, E. S. Malinverni, and J. Gain. 2018. A survey of augmented, virtual, and mixed reality for cultural heritage. J. Comput. Cult. Herit. 11, 2 (2018), 7:1–7:36.

Digital Library

[2]

W. Benjamin. 1935. The work of art in the age of mechanical reproduction (written 1935 as Das Kunstwerk im Zeitalter Seiner Technischen Reproduzierbarkeit).

[3]

A. Bonci, P. Clini, R. Martin, M. Pirani, R. Quattrini, and A. Raikov. 2018. Collaborative intelligence cyber-physical system for the valorization and re-use of cultural heritage. J. Inf. Technol. Construct. 23 (2018), 305–323.

[4]

J. Carmigniani, B. Furht, M. Anisetti, P. Ceravolo, E. Damiani, and M. Ivkovic. 2011. Augmented reality technologies, systems and applications. Multim. Tools Applic. 51, 1 (2011), 341–377.

Digital Library

[5]

Y. Collet and C. Turner. 2016. Smaller and Faster Data Compression with Zstandard. Retrieved from https://code.fb.com/core-data/smaller-and-faster-data-compression-with-zstandard/.

[6]

F. D’Agnano, C. Balletti, F. Guerra, and P. Vernier. 2015. Tooteko: A case study of augmented reality for an accessible cultural heritage. digitization, 3D printing and sensors for an audio-tactile experience. Int. Arch. Photogram. Rem. Sens. Spatial Inf. Sci. 40, 5 (2015), 207.

[7]

K. J. Dana, S. K. Nayar, B. van Ginneken, and J. J. Koenderink. 1997. Reflectance and texture of real-world surfaces. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 151–157.

Digital Library

[8]

W. Dangxiao, G. Yuan, L. Shiyi, Z. Yuru, X. Weiliang, and X. Jing. 2019. Haptic display for virtual reality: Progress and challenges. Virt. Real. Intell. Hardw. 1, 2 (2019), 136–162.

[9]

P. Debevec. 2006. Image-based lighting. In Proceedings of the ACM SIGGRAPH 2006 Courses.

Digital Library

[10]

T. G. Dulecha, F. A. Fanni, F. Ponchio, F. Pellacini, and A. Giachetti. 2020. Neural reflectance transformation imaging. Vis. Comput. 36, 10 (2020), 2161–2174.

Digital Library

[11]

J. Filip, R. Vavra, M. Haindl, P. Zid, M. Krupika, and V. Havran. 2013. BRDF slices: Accurate adaptive anisotropic appearance acquisition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1468–1473.

Digital Library

[12]

A. Franzluebbers and K. Johnsen. 2018. Performance benefits of high-fidelity passive haptic feedback in virtual reality training. In Proceedings of the Symposium on Spatial User Interaction. 16–24.

Digital Library

[13]

A. Giachetti, I. M. Ciortan, C. Daffara, G. Marchioro, R. Pintus, and E. Gobbetti. 2018. A novel framework for highlight reflectance transformation imaging. Comput. Vis. Image Underst. 168 (2018), 118–131.

Digital Library

[14]

E. Graeme, M. Kirk, and M. Tom. 2010. Archaeological applications of polynomial texture mapping: Analysis, conservation and representation. J. Archaeol. Sci. 37, 8 (2010), 2040–2050.

[15]

M. Haindl and J. Filip. 2007. Extreme compression and modeling of bidirectional texture function. IEEE Trans. Pattern Anal. Mach. Intell. 29, 10 (2007), 1859–1865.

Digital Library

[16]

M. Haindl and J. Filip. 2013. Visual Texture: Accurate Material Appearance Measurement, Representation and Modeling. Springer.

Digital Library

[17]

A. Hanus, M. Hoover, A. Lim, and J. Miller. 2019. A collaborative virtual reality escape room with passive haptics. In Proceedings of the IEEE Conference on Virtual Reality and 3D User Interfaces (VR). 1413–1414.

[18]

V. Havran, J. Filip, and K. Myszkowski. 2010. Bidirectional texture function compression based on multi-level vector quantization. In Computer Graphics Forum, Vol. 29. The Eurographics Association, 175–190.

[19]

V. Havran, J. Hošek, Š. Němcová, J. Čáp, and J. Bittner. 2017. Lightdrum-Portable light stage for accurate BTF measurement on site. Sensors 17, 3 (2017), 423.

[20]

T. Hawkins, J. Cohen, and P. Debevec. 2001. A photometric approach to digitizing cultural artifacts. In Proceedings of the Conference on Virtual Reality, Archeology, and Cultural Heritage. 333–342.

Digital Library

[21]

T. Hawkins, P. Einarsson, and P. E. Debevec. 2005. A dual light stage. Rend. Techniq. 5 (2005), 91–98.

Digital Library

[22]

M. Holroyd, J. Lawrence, and T. Zickler. 2010. A coaxial optical scanner for synchronous acquisition of 3D geometry and surface reflectance. ACM Trans. Graph. 29, 4 (2010), 99:1–99:12.

Digital Library

[23]

M. Holroyd, Ja. Lawrence, and T. Zickler. 2010. A coaxial optical scanner for synchronous acquisition of 3D geometry and surface reflectance. ACM Trans. Graph. 29, 4 (2010), 1–12.

Digital Library

[24]

R. D. Joyce and S. Robinson. 2017. Passive haptics to enhance virtual reality simulations. In Proceedings of the AIAA Modeling and Simulation Technologies Conference. 1313.

[25]

J. Köhler, T. Nöll, G. Reis, and D. Stricker. 2013. A full-spherical device for simultaneous geometry and reflectance acquisition. In Proceedings of the IEEE Workshop on Applications of Computer Vision (WACV). 355–362.

Digital Library

[26]

M. L. Koudelka, S. Magda, P. N. Belhumeur, and D. J. Kriegman. 2003. Acquisition, compression, and synthesis of bidirectional texture functions. In Proceedings of the 3rd International Workshop on Texture Analysis and Synthesis (Texture’03). 59–64.

[27]

S. Krumpen, M. Weinmann, and R. Klein. 2017. OctreeBTFs—A compact, seamless and distortion-free reflectance representation. Comput. Graph. 68 (2017), 21–31.

Digital Library

[28]

K. S. Ladefoged and C. B. Madsen. 2019. Spatially-varying diffuse reflectance capture using irradiance map rendering for image-based modeling applications. In Proceedings of the IEEE International Symposium on Mixed and Augmented Reality (ISMAR). 46–54.

[29]

H. P. A. Lensch, J. Kautz, M. Goesele, W. Heidrich, and H.-P. Seidel. 2003. Image-based reconstruction of spatial appearance and geometric detail. ACM Trans. Graph. 22, 2 (2003), 234–257.

Digital Library

[30]

R. W. Lindeman, J. L. Sibert, and J. K. Hahn. 1999. Hand-held windows: Towards effective 2D interaction in immersive virtual environments. In Proceedings IEEE Virtual Reality (Cat. No. 99CB36316). IEEE, 205–212.

Digital Library

[31]

T. Malzbender, D. Gelb, and H. Wolters. 2001. Polynomial texture maps. In Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques. 519–528.

Digital Library

[32]

P. Milgram and F. Kishino. 1994. A taxonomy of mixed reality visual displays. IEICE Trans. Inf. Syst. 77, 12 (1994), 1321–1329.

[33]

M. R. Mine, F. P. Brooks, and C. H. Sequin. 1997. Moving objects in space: Exploiting proprioception in virtual-environment interaction. In Proceedings of the 24th Annual Conference on Computer Graphics and Interactive Techniques. 19–26.

Digital Library

[34]

M. Mudge, T. Malzbender, A. Chalmers, R. Scopigno, J. Davis, O. Wang, P. Gunawardane, M. Ashley, M. Doerr, A. Proenca, and J. Barbosa. 2008. Image-based empirical information acquisition, scientific reliability, and long-term digital preservation for the natural sciences and cultural heritage. In Eurographics 2008 - Tutorials. The Eurographics Association.

[35]

M. Mudge, T. Malzbender, C. Schroer, and M. Lum. 2006. New reflection transformation imaging methods for rock art and multiple-viewpoint display. In Proceedings of the 7th International Conference on Virtual Reality, Archaeology and Intelligent Cultural Heritage. 195–202.

Digital Library

[36]

R. Nagao, K. Matsumoto, T. Narumi, T. Tanikawa, and M. Hirose. 2018. Ascending and descending in virtual reality: Simple and safe system using passive haptics. IEEE Trans. Visual. Comput. Graph. 24, 4 (2018), 1584–1593.

Digital Library

[37]

G. Nam, J. H. Lee, D. Gutierrez, and M. H Kim. 2018. Practical SVBRDF acquisition of 3D objects with unstructured flash photography. ACM Trans. Graph. 37, 6 (2018), 1–12.

Digital Library

[38]

F. E. Nicodemus, J. C. Richmond, J. J. Hsia, I. W Ginsberg, T. Limperis et al. 1977. Geometrical Considerations and Nomenclature for Reflectance. Vol. 160.

[39]

T. Nöll, J. Köhler, G. Reis, and D. Stricker. 2013. Faithful, compact and complete digitization of cultural heritage using a full-spherical scanner. In Proceedings of the Digital Heritage International Congress (DigitalHeritage). 15–22.

[40]

T. Nöll, J. Köhler, G. Reis, and D. Stricker. 2015. Fully automatic, omnidirectional acquisition of geometry and appearance in the context of cultural heritage preservation. J. Comput. Cult. Herit. 8, 1 (2015), 1–28.

Digital Library

[41]

G. Palma, M. Callieri, M. Dellepiane, and R. Scopigno. 2012. A statistical method for SVBRDF approximation from video sequences in general lighting conditions. Comput. Graph. Forum (Proceedings of the Eurographics Symposium on Rendering) 31, 4 (2012), 1491–1500.

Digital Library

[42]

J. S. Pierce, B. C. Stearns, and R. Pausch. 1999. Voodoo dolls: Seamless interaction at multiple scales in virtual environments. In Proceedings of the Symposium on Interactive 3D Graphics. 141–145.

Digital Library

[43]

F. Ponchio, M. Corsini, and R. Scopigno. 2018. A compact representation of relightable images for the web. In Proceedings of the 23rd International ACM Conference on 3D Web Technology.

Digital Library

[44]

PresenterMedia. 2019. PowerPoint Templates, 3D Animations, and Clipart. Retrieved from https://presentermedia.com/.

[45]

G. Rainer, A. Ghosh, W. Jakob, and T. Weyrich. 2020. Unified neural encoding of BTFs. Comput. Graph. Forum 39, 2 (2020), 167–178.

[46]

G. Rainer, W. Jakob, A. Ghosh, and T. Weyrich. 2019. Neural BTF compression and interpolation. In Computer Graphics Forum, Vol. 38. The Eurographics Asscociation, 235–244.

[47]

R. Ruiters and R. Klein. 2009. BTF compression via sparse tensor decomposition. In Computer Graphics Forum, Vol. 28. The Eurographics Asscociation, 1181–1188.

Digital Library

[48]

P. Santos, M. Ritz, R. Tausch, H. Schmedt, R. Monroy, A. De Stefano, O. Posniak, C. Fuhrmann, and D. W. Fellner. 2014. CultLab3D: On the verge of 3D mass digitization. In Proceedings of the Eurographics Workshop on Graphics and Cultural Heritage. 65–73.

Digital Library

[49]

C. Schwartz, R. Ruiters, and R. Klein. 2013. Level-of-detail streaming and rendering using bidirectional sparse virtual texture functions. Comput. Graph. Forum (Proc. of Pacific Graphics) 32, 7 (2013), 345–354.

[50]

C. Schwartz, R. Ruiters, M. Weinmann, and R. Klein. 2013. WebGL-based streaming and presentation of objects with bidirectional texture functions. J. Comput. Cult. Herit. 6, 3 (2013), 11:1–11:21.

Digital Library

[51]

C. Schwartz, R. Sarlette, M. Weinmann, M. Rump, and R. Klein. 2014. Design and implementation of practical bidirectional texture function measurement devices focusing on the developments at the university of bonn. Sensors 14, 5 (2014), 7753–7819.

[52]

C. Schwartz, M. Weinmann, R. Ruiters, and R. Klein. 2011. Integrated high-quality acquisition of geometry and appearance for cultural heritage. In Proceedings of the International Symposium on Virtual Reality, Archaeology and Intelligent Cultural Heritage (VAST). 25–32.

Digital Library

[53]

Y. Tsai, K. Fang, W. Lin, and Z. Shih. 2010. Modeling bidirectional texture functions with multivariate spherical radial basis functions. IEEE Transactions on Pattern Analysis and Machine Intelligence 33, 7 (2010), 1356–1369.

Digital Library

[54]

D. Tsetserukou, A. Neviarouskaya, H. Prendinger, N. Kawakami, M. Ishizuka, and S. Tachi. 2009. Enhancing mediated interpersonal communication through affective haptics. In Intelligent Technologies for Interactive Entertainment. 246–251.

[55]

D. Tsetserukou, A. Neviarouskaya, H. Prendinger, N. Kawakami, and S. Tachi. 2009. Affective haptics in emotional communication. In 2009 3rd International Conference on Affective Computing and Intelligent Interaction and Workshops. 1–6.

[56]

B. Tunwattanapong, G. Fyffe, P. Graham, J. Busch, X. Yu, A. Ghosh, and P. Debevec. 2013. Acquiring reflectance and shape from continuous spherical harmonic illumination. ACM Transactions on Graphics (TOG) 32, 4 (2013), 1–12.

Digital Library

[57]

M. Weinmann, F. Langguth, M. Goesele, and R. Klein. 2016. Advances in geometry and reflectance acquisition. In Eurographics 2016 Tutorials.

Digital Library

[58]

M. Weinmann, C. Schwartz, R. Ruiters, and R. Klein. 2011. A multi-camera, multi-projector super-resolution framework for structured light. In Proceedings of the International Conference on 3D Imaging, Modeling, Processing, Visualization and Transmission (3DIMPVT). 397–404.

Digital Library

[59]

T. Weyrich, J. Lawrence, H. P. A. Lensch, S. Rusinkiewicz, and T. Zickler. 2009. Principles of appearance acquisition and representation. Foundations and Trends in Computer Graphics and Vision 4, 2 (2009), 75–191.

Digital Library

[60]

H. Wu, J.e Dorsey, and H. Rushmeier. 2011. A sparse parametric mixture model for BTF compression, editing and rendering. In Computer Graphics Forum, Vol. 30. 465–473.

[61]

Q. Zhao. 2009. A survey on virtual reality. Science in China Series F: Information Sciences 52, 3 (2009), 348–400.

Cited By

Mubarrat SChowdhury SFernandes A(2024)Evaluating Visual-Spatiotemporal Co-Registration of a Physics-Based Virtual Reality Haptic InterfaceIEEE Access10.1109/ACCESS.2024.339118612(57017-57032)Online publication date: 2024
https://doi.org/10.1109/ACCESS.2024.3391186
Jia SChi OMartinez SLu L(2023)When “Old” Meets “New”: Unlocking the Future of Innovative Technology Implementation in Heritage TourismJournal of Hospitality & Tourism Research10.1177/10963480231205767Online publication date: 3-Nov-2023
https://doi.org/10.1177/10963480231205767
Kobeisse S(2023)Hands on the Past: Towards a Conceptual Framework for Developing and Evaluating Tangible AR Interfaces for Historical ArtefactsProceedings of the 35th Australian Computer-Human Interaction Conference10.1145/3638380.3638445(340-349)Online publication date: 2-Dec-2023
https://dl.acm.org/doi/10.1145/3638380.3638445
Show More Cited By

Index Terms

Towards Tangible Cultural Heritage Experiences—Enriching VR-based Object Inspection with Haptic Feedback
1. Computing methodologies
  1. Computer graphics
    1. Graphics systems and interfaces
      1. Perception
      2. Virtual reality
    2. Rendering
      1. Reflectance modeling

Recommendations

The Effect of Body-Based Haptic Feedback on Player Experience During VR Gaming
Virtual, Augmented and Mixed Reality
Abstract
As the interest in virtual reality (VR) technology as a game console has rapidly grown over the past few years, many new technologies are also being developed to further enhance the VR gaming experience. One of these technologies is haptic ...
A meta-analysis of recent studies on haptic feedback enhancement in immersive-augmented reality
ICVR 2018: Proceedings of the 4th International Conference on Virtual Reality

In this study, we address a research direction on haptic feedback-driven object presence in immersive Augmented Reality (AR). We reviewed 30 current studies in regard of sensory feedback in domain of Virtual Reality (VR), Augmented Virtuality (AV), AR ...
Haptic feedback with HaptoBend: utilizing shape-change to enhance virtual reality
SUI '17: Proceedings of the 5th Symposium on Spatial User Interaction

HaptoBend¹ is an original device for providing general passive haptic feedback (PHF) in virtual reality (VR). The importance of haptic feedback in VR is apparent through improvements in presence [3] and user performance [2]. However, most general haptic ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Journal on Computing and Cultural Heritage

Journal on Computing and Cultural Heritage Volume 15, Issue 1

February 2022

348 pages

ISSN:1556-4673

EISSN:1556-4711

DOI:10.1145/3505194

Editor:
Franco Niccolucci
VAST-LAB at PIN, University of Florence, Italy

Issue’s Table of Contents

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 14 December 2021

Accepted: 01 June 2021

Revised: 01 May 2021

Received: 01 February 2021

Published in JOCCH Volume 15, Issue 1

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Refereed

Funding Sources

NFDI4Culture - Consortium for research data on material and immaterial cultural heritage

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

9
Total Citations
View Citations
1,109
Total Downloads

Downloads (Last 12 months)340
Downloads (Last 6 weeks)46

Reflects downloads up to 13 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Mubarrat SChowdhury SFernandes A(2024)Evaluating Visual-Spatiotemporal Co-Registration of a Physics-Based Virtual Reality Haptic InterfaceIEEE Access10.1109/ACCESS.2024.339118612(57017-57032)Online publication date: 2024
https://doi.org/10.1109/ACCESS.2024.3391186
Jia SChi OMartinez SLu L(2023)When “Old” Meets “New”: Unlocking the Future of Innovative Technology Implementation in Heritage TourismJournal of Hospitality & Tourism Research10.1177/10963480231205767Online publication date: 3-Nov-2023
https://doi.org/10.1177/10963480231205767
Kobeisse S(2023)Hands on the Past: Towards a Conceptual Framework for Developing and Evaluating Tangible AR Interfaces for Historical ArtefactsProceedings of the 35th Australian Computer-Human Interaction Conference10.1145/3638380.3638445(340-349)Online publication date: 2-Dec-2023
https://dl.acm.org/doi/10.1145/3638380.3638445
Roumeliotis MMania K(2023)A Lightweight Haptic Feedback Glove Employing Normal Indentation, Lateral Skin Stretch and both Softness and Hardness Rendering2023 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct)10.1109/ISMAR-Adjunct60411.2023.00113(525-530)Online publication date: 16-Oct-2023
https://doi.org/10.1109/ISMAR-Adjunct60411.2023.00113
Kim KKwon OYu J(2023)Evaluation of an HMD-Based Multisensory Virtual Museum Experience for Enhancing Sense of PresenceIEEE Access10.1109/ACCESS.2023.331113511(100295-100308)Online publication date: 2023
https://doi.org/10.1109/ACCESS.2023.3311135
Basu APaul SGhosh SDas SChanda BBhagvati CSnasel V(2023)Digital Restoration of Cultural Heritage With Data-Driven Computing: A SurveyIEEE Access10.1109/ACCESS.2023.328063911(53939-53977)Online publication date: 2023
https://doi.org/10.1109/ACCESS.2023.3280639
Mudička ŠKapica R(2022)Digital Heritage, the Possibilities of Information Visualisation through Extended Reality ToolsHeritage10.3390/heritage60100066:1(112-131)Online publication date: 22-Dec-2022
https://doi.org/10.3390/heritage6010006
Lazna RBarvir RVondrakova ABrus J(2022)Creating a Haptic 3D Model of Wenceslas Hill in OlomoucApplied Sciences10.3390/app12211081712:21(10817)Online publication date: 25-Oct-2022
https://doi.org/10.3390/app122110817
Antonya CButnariu S(2022)Preservation of Cultural Heritage Using Virtual Reality Technologies and Haptic Feedback: A Prototype and Case Study on Antique Carpentry ToolsApplied Sciences10.3390/app1216800212:16(8002)Online publication date: 10-Aug-2022
https://doi.org/10.3390/app12168002

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Full Text

View this article in Full Text.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View full text|Download PDF

View Issue’s Table of Contents