More Web Proxy on the site http://driver.im/

research-article

HySAR: Hybrid Material Rendering by an Optical See-Through Head-Mounted Display with Spatial Augmented Reality Projection

Authors:

Takumi Hamasaki,

Maki SugimotoAuthors Info & Claims

IEEE Transactions on Visualization and Computer Graphics, Volume 24, Issue 4

Pages 1457 - 1466

https://doi.org/10.1109/TVCG.2018.2793659

Published: 01 April 2018 Publication History

Abstract

Spatial augmented reality (SAR) pursues realism in rendering materials and objects. To advance this goal, we propose a hybrid SAR (HySAR) that combines a projector with optical see-through head-mounted displays (OST-HMD). In an ordinary SAR scenario with co-located viewers, the viewers perceive the same virtual material on physical surfaces. In general, the material consists of two components: a view-independent (VI) component such as diffuse reflection, and a view-dependent (VD) component such as specular reflection. The VI component is static over viewpoints, whereas the VD should change for each viewpoint even if a projector can simulate only one viewpoint at one time. In HySAR, a projector only renders the static VI components. In addition, the OST-HMD renders the dynamic VD components according to the viewer's current viewpoint. Unlike conventional SAR, the HySAR concept theoretically allows an unlimited number of co-located viewers to see the correct material over different viewpoints. Furthermore, the combination enhances the total dynamic range, the maximum intensity, and the resolution of perceived materials. With proof-of-concept systems, we demonstrate HySAR both qualitatively and quantitatively with real objects. First, we demonstrate HySAR by rendering synthetic material properties on a real object from different viewpoints. Our quantitative evaluation shows that our system increases the dynamic range by 2.24 times and the maximum intensity by 2.12 times compared to an ordinary SAR system. Second, we replicate the material properties of a real object by SAR and HySAR, and show that HySAR outperforms SAR in rendering VD specular components.

References

[1]

O. Bimber and R. Raskar, Spatial augmented reality: merging real and virtual worlds . CRC press, 2005.

Digital Library

[2]

R. Raskar, G. Welch, K.-L. Low, and D. Bandyopadhyay, "Shader lamps: Animating real objects with image-based illumination," in Rendering Techniques 2001, pp. 89–102, <conf-sponsor>Springer</conf-sponsor>, 2001.

Digital Library

[3]

T. Okazaki, T. Okatani, and K. Deguchi, "A projector-camera system for high-quality synthesis of virtual reflectance on real object surfaces," Information and Media Technologies, vol. Volume 5, no. Issue 2, pp. 691–703, 2010.

[4]

M. Broecker, B.H. Thomas, and R.T. Smith, "Adapting ray tracing to spatial augmented reality," in The 13th IEEE ISMAR, pp. 1–6, <conf-sponsor>IEEE</conf-sponsor>, 2013.

[5]

M. Agrawala, A.C. Beers, I. McDowall, B. Fröhlich, M. Bolas, and P. Hanrahan, "The two-user responsive workbench: support for collaboration through individual views of a shared space," in The 24th ACM SIGGRAPH, pp. 327–332, <conf-sponsor>ACM Press/Addison-Wesley Publishing Co.</conf-sponsor>, 1997.

Digital Library

[6]

O. Bimber, B. Fröhlich, D. Schmalstieg, and L.M. Encarnação, "The virtual showcase," in ACM SIGGRAPH 2006 Courses, p. pp.9, <conf-sponsor>ACM</conf-sponsor>, 2006.

Digital Library

[7]

H. Benko, A.D. Wilson, and F. Zannier, "Dyadic projected spatial augmented reality," in The 27th ACM UIST, pp. 645–655, <conf-sponsor>ACM</conf-sponsor>, 2014.

Digital Library

[8]

S.A. Shafer, "Using color to separate reflection components," Color Research & Application, vol. Volume 10, no. Issue 4, pp. 210–218, 1985.

[9]

D.G. Aliaga, Y.H. Yeung, A. Law, B. Sajadi, and A. Majumder, "Fast high-resolution appearance editing using superimposed projections," ACM ToG, vol. Volume 31, no. Issue 2, p. pp.13, 2012.

Digital Library

[10]

A. Majumder and G. Welch, "Computer graphics optique: Optical superposition of projected computer graphics," in Eurographics Workshop on Rendering, pp. 209–218, 2001.

Digital Library

[11]

T. Amano, "Projection based real-time material appearance manipulation," in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 918–923, 2013.

Digital Library

[12]

A. Bermano, P. Brüschweiler, A. Grundhöfer, D. Iwai, B. Bickel, and M. Gross, "Augmenting physical avatars using projector-based illumination," ACM ToG, vol. Volume 32, no. Issue 6, p. pp.189, 2013.

Digital Library

[13]

O. Bimber and D. Iwai, "Superimposing dynamic range," ACM ToG, vol. Volume 27, pp. 150:1–150:8, 2008.

Digital Library

[14]

C. Siegl, M. Colaianni, L. Thies, J. Thies, M. Zollhöfer, S. Izadi, M. Stamminger, and F. Bauer, "Real-time pixel luminance optimization for dynamic multi-projection mapping," ACM ToG, vol. Volume 34, no. Issue 6, p. pp.237, 2015.

Digital Library

[15]

Y. Kitamura, T. Nakayama, T. Nakashima, and S. Yamamoto, "The illusion hole with polarization filters," in Proceedings of the ACM Symposium on Virtual Reality Software and Technology, VRST '06, (New York, NY, USA), pp. 244–251, <conf-sponsor>ACM</conf-sponsor>, 2006.

Digital Library

[16]

H. Benko, R. Jota, and A. Wilson, "Miragetable: freehand interaction on a projected augmented reality tabletop," in Proceedings of the SIGCHI conference on human factors in computing systems, pp. 199–208, <conf-sponsor>ACM</conf-sponsor>, 2012.

Digital Library

[17]

G. Koutaki, "Binary continuous image decomposition for multi-view display," ACM ToG, vol. Volume 35, no. Issue 4, p. pp.69, 2016.

Digital Library

[18]

T. Amano and K. Minami, "Structural color display on retro-reflective objects," in International Conference on Artificial Reality and Telexistence and Eurographics Symposium on Virtual Environments, pp. 37–44, 2015.

Digital Library

[19]

O. Bimber, A. Grundhöfer, G. Wetzstein, and S. Knödel, "Consistent illumination within optical see-through augmented environments," in The 2nd IEEE/ACM ISMAR, pp. 198–207, <conf-sponsor>IEEE Computer Society</conf-sponsor>, 2003.

Digital Library

[20]

A. Maimone, X. Yang, N. Dierk, A. State, M. Dou, and H. Fuchs, "General-purpose telepresence with head-worn optical see-through displays and projector-based lighting," in 2013 IEEE Virtual Reality, pp. 23–26, <conf-sponsor>IEEE</conf-sponsor>, 2013.

[21]

J. Zhou, I. Lee, B. Thomas, R. Menassa, A. Farrant, and A. Sansome, "In-situ support for automotive manufacturing using spatial augmented reality," International Journal of Virtual Reality, vol. Volume 11, no. Issue 1, 2012.

[22]

H. Benko, E. Ofek, F. Zheng, and A.D. Wilson, "Fove ar: Combining an optically see-through near-eye display with projector-based spatial augmented reality," in The 28th ACM UIST, pp. 129–135, <conf-sponsor>ACM</conf-sponsor>, 2015.

Digital Library

[23]

M.B. Hullin, I. Ihrke, W. Heidrich, T. Weyrich, G. Damberg, and M. Fuchs, "Computational Fabrication and Display of Material Appearance," in Euro-graphics 2013 - State of the Art Reports ( M. Sbert and L. Szirmay-Kalos, eds.), <conf-sponsor>The Eurographics Association</conf-sponsor>, 2013.

[24]

M.B. Hullin, H.P. Lensch, R. Raskar, H.-P. Seidel, and I. Ihrke, "Dynamic display of brdfs," Computer Graphics Forum, vol. Volume 30, no. Issue 2, pp. 475–483, 2011.

[25]

Y. Ochiai, A. Oyama, and K. Toyoshima, "A colloidal display: membrane screen that combines transparency, brdf and 3d volume," in ACM SIGGRAPH 2012 Emerging Technologies, p. pp.2, <conf-sponsor>ACM</conf-sponsor>, 2012.

Digital Library

[26]

L. Miyashita, K. Ishihara, Y. Watanabe, and M. Ishikawa, "Zoematrope: a system for physical material design," ACM ToG, vol. Volume 35, pp. 66:1–66:11, 2016.

Digital Library

[27]

I. Kauvar, S.J. Yang, L. Shi, I. McDowall, and G. Wetzstein, "Adaptive color display via perceptually-driven factored spectral projection," ACM ToG, vol. Volume 34, pp. 165:1–165:10, 2015.

Digital Library

[28]

B. Jones, R. Sodhi, M. Murdock, R. Mehra, H. Benko, A. Wilson, E. Ofek, B. MacIntyre, N. Raghuvanshi, and L. Shapira, "Roomalive: magical experiences enabled by scalable, adaptive projector-camera units," in The 27th ACM UIST, pp. 637–644, <conf-sponsor>ACM</conf-sponsor>, 2014.

Digital Library

[29]

M. Huber, D. Pustka, P. Keitler, F. Echtler, and G. Klinker, "A system architecture for ubiquitous tracking environments," in The 6th IEEE/ACM ISMAR, pp. 1–4, <conf-sponsor>IEEE Computer Society</conf-sponsor>, 2007.

Digital Library

[30]

J. Newman, M. Wagner, M. Bauer, A. MacWilliams, T. Pintaric, D. Beyer, D. Pustka, F. Strasser, D. Schmalstieg, and G. Klinker, "Ubiquitous tracking for augmented reality," in The 3rd IEEE/ACM ISMAR, pp. 192–201, <conf-sponsor>IEEE</conf-sponsor>, 2004.

Digital Library

[31]

H. Hua and S. Liu, "Dual-sensor foveated imaging system," Applied optics, vol. Volume 47, no. Issue 3, pp. 317–327, 2008.

[32]

Y. Itoh, M. Dzitsiuk, T. Amano, and G. Klinker, "Semi-parametric color reproduction method for optical see-through head-mounted displays," IEEE TVCG, vol. Volume 21, no. Issue 11, pp. 1269–1278, 2015.

Digital Library

[33]

T. Yoshida, C. Horii, and K. Sato, "A virtual color reconstruction system for real heritage with light projection," in Proc. International Conference on Virtual Systems and Multimedia, pp. 161–168, 2003.

[34]

A. Grundhöfer and D. Iwai, "Robust, error-tolerant photometric projector compensation," IEEE Transactions on Image Processing, vol. Volume 24, no. Issue 12, pp. 5086–5099, 2015.

[35]

G. Wetzstein and O. Bimber, "Radiometric compensation through inverse light transport," in Proceedings of the 15th Pacific Conference on Computer Graphics and Applications, pp. 391–399, <conf-sponsor>IEEE Computer Society</conf-sponsor>, 2007.

Digital Library

[36]

Y. Itoh, T. Amano, D. Iwai, and G. Klinker, "Gaussian light field: Estimation of viewpoint-dependent blur for optical see-through head-mounted displays," IEEE TVCG, 2016.

[37]

J.F. Blinn, "Models of light reflection for computer synthesized pictures," in ACM SIGGRAPH Computer Graphics, vol. Volume 11, pp. 192–198, <conf-sponsor>ACM</conf-sponsor>, 1977.

Digital Library

Cited By

Wang YLing HHuang B(2024)ViComp: Video Compensation for Projector-Camera SystemsIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2024.337207930:5(2347-2356)Online publication date: 4-Mar-2024
https://dl.acm.org/doi/10.1109/TVCG.2024.3372079
Wang YXie SWang LXu HTabata SIshikawa M(2022)ARSlice: Head-Mounted Display Augmented with Dynamic Tracking and ProjectionJournal of Computer Science and Technology10.1007/s11390-022-2173-y37:3(666-679)Online publication date: 1-Jun-2022
https://dl.acm.org/doi/10.1007/s11390-022-2173-y
Zhai LChen D(2021)Image real-time augmented reality technology based on spatial color and depth consistencyJournal of Real-Time Image Processing10.1007/s11554-020-00988-718:2(369-377)Online publication date: 1-Apr-2021
https://dl.acm.org/doi/10.1007/s11554-020-00988-7

Recommendations

Pipeline rendering: interaction and realism through hardware-based multi-pass rendering
Interactive Approximate Rendering of Reflections, Refractions, and Caustics

Reflections, refractions, and caustics are very important for rendering global illumination images. Although many methods can be applied to generate these effects, the rendering performance is not satisfactory for interactive applications. In this paper,...
Virtual Spherical Gaussian Lights for Real-time Glossy Indirect Illumination

Virtual point lights VPLs are well established for real-time global illumination. However, this method suffers from spiky artifacts and flickering caused by singularities of VPLs, highly glossy materials, high-frequency textures, and discontinuous ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image IEEE Transactions on Visualization and Computer Graphics

IEEE Transactions on Visualization and Computer Graphics Volume 24, Issue 4

April 2018

283 pages

ISSN:1077-2626

Issue’s Table of Contents

Copyright © 2018.

Publisher

IEEE Educational Activities Department

United States

Publication History

Published: 01 April 2018

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 22 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Wang YLing HHuang B(2024)ViComp: Video Compensation for Projector-Camera SystemsIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2024.337207930:5(2347-2356)Online publication date: 4-Mar-2024
https://dl.acm.org/doi/10.1109/TVCG.2024.3372079
Wang YXie SWang LXu HTabata SIshikawa M(2022)ARSlice: Head-Mounted Display Augmented with Dynamic Tracking and ProjectionJournal of Computer Science and Technology10.1007/s11390-022-2173-y37:3(666-679)Online publication date: 1-Jun-2022
https://dl.acm.org/doi/10.1007/s11390-022-2173-y
Zhai LChen D(2021)Image real-time augmented reality technology based on spatial color and depth consistencyJournal of Real-Time Image Processing10.1007/s11554-020-00988-718:2(369-377)Online publication date: 1-Apr-2021
https://dl.acm.org/doi/10.1007/s11554-020-00988-7
Hartmann JYeh YVogel DIqbal SMacLean KChevalier FMueller S(2020)AARProceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology10.1145/3379337.3415849(445-458)Online publication date: 20-Oct-2020
https://dl.acm.org/doi/10.1145/3379337.3415849

View Options

View options

Media

Figures

Other

Tables

View Issue’s Table of Contents