Projection mapping based on BRDF reconstruction from single RGBD image
Article No.: 5, Pages 1 - 8
Abstract
There have been many researches on projection mapping focused on target objects tracking, geometric shape recovering or virtual materials simulating such as clothes. However, few people pay attention to the material of target object which actually influences the visual results of projection. We present a new projection mapping framework based on BRDF reconstruction for the goal of more real projection results by enhancing the effects of Augmented Reality. In the framework, 3D computer vision method is used to reconstruct the BRDF of target object with a single RGBD image. A new algorithm is proposed using two Convolutional Neural Networks(CNN) which can predict both normal map and reflectance map of the target surface simultaneously with the RGBD image. The predicted maps are used to render the content to be projected onto the target object. Our BRDF reconstruction algorithm can recover several materials in one scene correctly in use of just one image. Experimental results show our framework has impressive performance and relatively accurate consequence.
References
[1]
Miika Aittala, Timo Aila, and Jaakko Lehtinen. 2016. Reflectance modeling by neural texture synthesis. ACM Transactions on Graphics (TOG) 35, 4 (2016), 65.
[2]
Nobumichi Asai. 2014. Omote Real-Time Face Tracking and Projection Mapping. https://vimeo.com/103425574/.
[3]
Hirotaka Asayama, Daisuke Iwai, and Kosuke Sato. 2015. Diminishable visual markers on fabricated projection object for dynamic spatial augmented reality. In SIGGRAPH Asia 2015 Emerging Technologies. ACM, 7.
[4]
Deepak Bandyopadhyay, Ramesh Raskar, and Henry Fuchs. 2001. Dynamic shader lamps: Painting on movable objects. In Augmented Reality, 2001. Proceedings. IEEE and ACM International Symposium on. IEEE, 207--216.
[5]
Amit Bermano, Philipp Brüschweiler, Anselm Grundhöfer, Daisuke Iwai, Bernd Bickel, and Markus Gross. 2013. Augmenting physical avatars using projector-based illumination. ACM Transactions on Graphics (TOG) 32, 6 (2013), 189.
[6]
Amit H Bermano, Markus Billeter, Daisuke Iwai, and Anselm Grundhöfer. 2017. Makeup lamps: Live augmentation of human faces via projection. In Computer Graphics Forum, Vol. 36. Wiley Online Library, 311--323.
[7]
Oliver Bimber, Daisuke Iwai, Gordon Wetzstein, and Anselm Grundhöfer. 2008. The Visual Computing of Projector-Camera Systems. In Computer Graphics Forum, Vol. 27. Wiley Online Library, 2219--2245.
[8]
Oliver Bimber and Ramesh Raskar. 2005. Spatial augmented reality: merging real and virtual worlds. CRC press.
[9]
Yasutaka Furukawa and Jean Ponce. 2010. Accurate, dense, and robust multiview stereopsis. IEEE transactions on pattern analysis and machine intelligence 32, 8 (2010), 1362--1376.
[10]
Kouki Hayashi, Yoshitaka Onishi, Kazuko Itoh, Hiroyasu Miwa, and Atsuo Takanishi. 2006. Development and evaluation of face robot to express various face shape. In Robotics and Automation, 2006. ICRA 2006. Proceedings 2006 IEEE International Conference on. IEEE, 481--486.
[11]
Naoto Hieda and Jeremy R Cooperstock. 2015. Digital facial augmentation for interactive entertainment. In Intelligent Technologies for Interactive Entertainment (INTETAIN), 2015 7th International Conference on. IEEE, 9--16.
[12]
Shahram Izadi, David Kim, Otmar Hilliges, David Molyneaux, Richard Newcombe, Pushmeet Kohli, Jamie Shotton, Steve Hodges, Dustin Freeman, Andrew Davison, et al. 2011. KinectFusion: real-time 3D reconstruction and interaction using a moving depth camera. In Proceedings of the 24th annual ACM symposium on User interface software and technology. ACM, 559--568.
[13]
Brett R Jones, Hrvoje Benko, Eyal Ofek, and Andrew D Wilson. 2013. IllumiRoom: peripheral projected illusions for interactive experiences. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 869--878.
[14]
Alex Kendall, Hayk Martirosyan, Saumitro Dasgupta, Peter Henry, Ryan Kennedy, Abraham Bachrach, and Adam Bry. 2017. End-to-end learning of geometry and context for deep stereo regression. CoRR, vol. abs/1703.04309 (2017).
[15]
Koyaku Kobayashi Yamato Noda Hiraoka Nakamura Kuwahara Asai, Sakata Lacroix and Doi Nagashio. 2014. Face hacking. http://projection-mapping.org/face-hacking/.
[16]
Xiao Li, Yue Dong, Pieter Peers, and Xin Tong. 2017. Modeling surface appearance from a single photograph using self-augmented convolutional neural networks. ACM Transactions on Graphics (TOG) 36, 4 (2017), 45.
[17]
Peter Lincoln, Greg Welch, Andrew Nashel, Adrian Ilie, Andrei State, and Henry Fuchs. 2009. Animatronic shader lamps avatars. In Mixed and Augmented Reality, 2009. ISMAR 2009. 8th IEEE International Symposium on. IEEE, 27--33.
[18]
Stephen Lombardi and Ko Nishino. 2012. Reflectance and natural illumination from a single image. In European Conference on Computer Vision. Springer, 582--595.
[19]
Stephen Lombardi and Ko Nishino. 2016. Reflectance and illumination recovery in the wild. IEEE transactions on pattern analysis and machine intelligence 38, 1 (2016), 129--141.
[20]
Shoki Miyagawa, Yoshihiro Fukuhara, Fumiya Narita, Norihiro Ogata, and Shigeo Morishima. 2017. Retexturing under self-occlusion using hierarchical markers. In ACM SIGGRAPH 2017 Posters. ACM, 27.
[21]
David Molyneaux, Shahram Izadi, David Kim, Otmar Hilliges, Steve Hodges, Xiang Cao, Alex Butler, and Hans Gellersen. 2012. Interactive environment-aware handheld projectors for pervasive computing spaces. In International Conference on Pervasive Computing. Springer, 197--215.
[22]
Richard A Newcombe, Dieter Fox, and Steven M Seitz. 2015. Dynamicfusion: Reconstruction and tracking of non-rigid scenes in real-time. In Proceedings of the IEEE conference on computer vision and pattern recognition. 343--352.
[23]
Richard A Newcombe, Shahram Izadi, Otmar Hilliges, David Molyneaux, David Kim, Andrew J Davison, Pushmeet Kohi, Jamie Shotton, Steve Hodges, and Andrew Fitzgibbon. 2011. KinectFusion: Real-time dense surface mapping and tracking. In Mixed and augmented reality (ISMAR), 2011 10th IEEE international symposium on. IEEE, 127--136.
[24]
Geoffrey Oxholm and Ko Nishino. 2016. Shape and reflectance estimation in the wild. IEEE transactions on pattern analysis and machine intelligence 38, 2 (2016), 376--389.
[25]
Jaesik Park, Sudipta N Sinha, Yasuyuki Matsushita, Yu-Wing Tai, and In So Kweon. 2017. Robust multiview photometric stereo using planar mesh parameterization. IEEE transactions on pattern analysis and machine intelligence 39, 8 (2017), 1591--1604.
[26]
Konstantinos Rematas, Tobias Ritschel, Mario Fritz, Efstratios Gavves, and Tinne Tuytelaars. 2016. Deep reflectance maps. In Computer Vision and Pattern Recognition (CVPR), 2016 IEEE Conference on. IEEE, 4508--4516.
[27]
Christoph Resch, Peter Keitler, and Gudrun Klinker. 2014. Sticky projections---a new approach to interactive shader lamp tracking. In Mixed and Augmented Reality (ISMAR), 2014 IEEE International Symposium on. IEEE, 151--156.
[28]
Fabiano Romeiro and Todd Zickler. 2010. Blind reflectometry. In European conference on computer vision. Springer, 45--58.
[29]
Daniel Saakes, Hui-Shyong Yeo, Seung-Tak Noh, Gyeol Han, and Woontack Woo. 2016. Mirror mirror: An on-body t-shirt design system. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. ACM, 6058--6063.
[30]
Hiroaki Santo, Masaki Samejima, Yusuke Sugano, Boxin Shi, and Yasuyuki Matsushita. 2017. Deep Photometric Stereo Network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 501--509.
[31]
Evan Shelhamer, Jonathan T Barron, and Trevor Darrell. 2015. Scene intrinsics and depth from a single image. In Computer Vision Workshop (ICCVW), 2015 IEEE International Conference on. IEEE, 235--242.
[32]
Boxin Shi, Zhe Wu, Zhipeng Mo, Dinglong Duan, Sai-Kit Yeung, and Ping Tan. 2016. A benchmark dataset and evaluation for non-lambertian and uncalibrated photometric stereo. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 3707--3716.
[33]
Christian Siegl, Matteo Colaianni, Lucas Thies, Justus Thies, Michael Zollhöfer, Shahram Izadi, Marc Stamminger, and Frank Bauer. 2015. Real-time pixel luminance optimization for dynamic multi-projection mapping. ACM Transactions on Graphics (TOG) 34, 6 (2015), 237.
[34]
Yichuan Tang, Ruslan Salakhutdinov, and Geoffrey Hinton. 2012. Deep lambertian networks. arXiv preprint arXiv:1206.6445 (2012).
[35]
Tatsunori Taniai and Takanori Maehara. 2018. Neural Photometric Stereo Reconstruction for General Reflectance Surfaces. arXiv preprint arXiv:1802.10328 (2018).
[36]
Tatsunori Taniai, Yasuyuki Matsushita, Yoichi Sato, and Takeshi Naemura. 2017. Continuous 3D Label Stereo Matching using Local Expansion Moves. IEEE Transactions on Pattern Analysis and Machine Intelligence (2017).
[37]
Chun-Po Wang, Noah Snavely, and Steve Marschner. 2011. Estimating dual-scale properties of glossy surfaces from step-edge lighting. ACM Transactions on Graphics (TOG) 30, 6 (2011), 172.
[38]
Junchen Wang, Hideyuki Suenaga, Kazuto Hoshi, Liangjing Yang, Etsuko Kobayashi, Ichiro Sakuma, and Hongen Liao. 2014. Augmented reality navigation with automatic marker-free image registration using 3-D image overlay for dental surgery. IEEE transactions on biomedical engineering 61, 4 (2014), 1295--1304.
[39]
Ting-Chun Wang, Manmohan Chandraker, Alexei A Efros, and Ravi Ramamoorthi. 2016. SVBRDF-invariant shape and reflectance estimation from light-field cameras. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 5451--5459.
[40]
Zexiang Xu, Jannik Boll Nielsen, Jiyang Yu, Henrik Wann Jensen, and Ravi Ramamoorthi. 2016. Minimal BRDF sampling for two-shot near-field reflectance acquisition. ACM Transactions on Graphics (TOG) 35, 6 (2016), 188.
[41]
Feng Zheng, Ryan Schubert, and Greg Weich. 2013. A general approach for closed-loop registration in AR. In Virtual Reality (VR), 2013 IEEE. IEEE, 47--50.
[42]
Yi Zhou, Shuangjiu Xiao, Ning Tang, Zhiyong Wei, and Xu Chen. 2016. Pmomo: projection mapping on movable 3D object. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. ACM, 781--790.
Index Terms
- Projection mapping based on BRDF reconstruction from single RGBD image
Recommendations
Light-field Projection for Tangible Projection Mapping
SIGGRAPH '21: ACM SIGGRAPH 2021 PostersIn the present study, we propose a novel projection mapping using a 3D light-field image as a light source. In recent years, spatial augmented reality has evolved into dynamic projection mapping that extends the target to moving objects. However, ...
BRDF reconstruction from real object using reconstructed geometry of multi-view images
SA '17: SIGGRAPH Asia 2017 Technical BriefsIn this paper, we present an effective approach to reconstruct the bidirectional reflectance distribution function (BRDF) from a real object without any special devices and complex setups. Multi-view images are commonly used to reconstruct the three-...
Realistic 3D projection mapping using polynomial texture maps
SIGGRAPH '16: ACM SIGGRAPH 2016 PostersProjection mapping has been widely used to efficiently visualize real world objects in various areas such as exhibitions, advertisements, and theatrical performances. To represent the projected content in a realistic manner, the appearance of an object ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
December 2018
200 pages
ISBN:9781450360876
DOI:10.1145/3284398
- Conference Chairs:
- Koji Mikami,
- Zhigeng Pan,
- Matt Adcock,
- Daniel Thalmann,
- Program Chairs:
- Xubo Yang,
- Tomoki Itamiya,
- Enhua Wu
Copyright © 2018 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 02 December 2018
Check for updates
Author Tags
Qualifiers
- Research-article
Conference
VRCAI '18
Sponsor:
VRCAI '18: International Conference on Virtual Reality Continuum and its Applications in Industry
December 2 - 3, 2018
Tokyo, Japan
Acceptance Rates
Overall Acceptance Rate 51 of 107 submissions, 48%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 202Total Downloads
- Downloads (Last 12 months)13
- Downloads (Last 6 weeks)1
Reflects downloads up to 31 Dec 2024
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in