More Web Proxy on the site http://driver.im/

research-article

TextureMe: High-Quality Textured Scene Reconstruction in Real Time

Authors:

Seungyong LeeAuthors Info & Claims

ACM Transactions on Graphics (TOG), Volume 41, Issue 3

Article No.: 24, Pages 1 - 18

https://doi.org/10.1145/3503926

Published: 07 March 2022 Publication History

Abstract

Three-dimensional (3D) reconstruction using an RGB-D camera has been widely adopted for realistic content creation. However, high-quality texture mapping onto the reconstructed geometry is often treated as an offline step that should run after geometric reconstruction. In this article, we propose TextureMe, a novel approach that jointly recovers 3D surface geometry and high-quality texture in real time. The key idea is to create triangular texture patches that correspond to zero-crossing triangles of truncated signed distance function (TSDF) progressively in a global texture atlas. Our approach integrates color details into the texture patches in parallel with the depth map integration to a TSDF. It also actively updates a pool of texture patches to adapt TSDF changes and minimizes misalignment artifacts that occur due to camera drift and image distortion. Our global texture atlas representation is fully compatible with conventional texture mapping. As a result, our approach produces high-quality textures without utilizing additional texture map optimization, mesh parameterization, or heavy post-processing. High-quality scenes produced by our real-time approach are even comparable to the results from state-of-the-art methods that run offline.

References

[1]

Fausto Bernardini, Ioana M. Martin, and Holly Rushmeier. 2001. High-quality texture reconstruction from multiple scans. IEEE Transactions on Visualization and Computer Graphics 7, 4 (2001), 318–332.

Digital Library

[2]

Sai Bi, Nima Khademi Kalantari, and Ravi Ramamoorthi. 2017. Patch-based optimization for image-based texture mapping. ACM Transactions on Graphics 36, 4 (2017), 106–101.

Digital Library

[3]

Chris Buehler, Michael Bosse, Leonard McMillan, Steven Gortler, and Michael Cohen. 2001. Unstructured lumigraph rendering. In Proceedings of SIGGRAPH Conference.425–432.

Digital Library

[4]

Nathan A. Carr and John C. Hart. 2002. Meshed atlases for real-time procedural solid texturing. ACM Transactions on Graphics 21, 2 (April 2002), 106–131.

Digital Library

[5]

Anpei Chen, Minye Wu, Yingliang Zhang, Nianyi Li, Jie Lu, Shenghua Gao, and Jingyi Yu. 2018. Deep surface light fields. Proceedings of the ACM on Computer Graphics and Interactive Techniques 1, 1 (2018), 1–17.

Digital Library

[6]

Wei-Chao Chen, Jean-Yves Bouguet, Michael H. Chu, and Radek Grzeszczuk. 2002. Light field mapping: Efficient representation and hardware rendering of surface light fields. ACM Transactions on Graphics 21, 3 (2002), 447–456.

Digital Library

[7]

Sungjoon Choi, Qian-Yi Zhou, and Vladlen Koltun. 2015. Robust reconstruction of indoor scenes. In Proceedings of CVPR. 5556–5565.

[8]

Frederique Crete, Thierry Dolmiere, Patricia Ladret, and Marina Nicolas. 2007. The blur effect: Perception and estimation with a new no-reference perceptual blur metric. In Human Vision and Electronic Imaging XII, Vol. 6492. International Society for Optics and Photonics, 64920I.

[9]

Brian Curless and Marc Levoy. 1996. A volumetric method for building complex models from range images. In Proceedings of SIGGRAPH Conference.303–312.

Digital Library

[10]

Angela Dai, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. 2017a. ScanNet: Richly-annotated 3D reconstructions of indoor scenes. In Proceedings of CVPR. 5828–5839.

[11]

Angela Dai, Matthias Nießner, Michael Zollhöfer, Shahram Izadi, and Christian Theobalt. 2017b. Bundlefusion: Real-time globally consistent 3D reconstruction using on-the-fly surface reintegration. ACM Transactions on Graphics 36, 3 (2017), 24.

Digital Library

[12]

Paul Debevec, Yizhou Yu, and George Borshukov. 1998. Efficient view-dependent image-based rendering with projective texture-mapping. In Eurographics Workshop on Rendering Techniques. 105–116.

[13]

Yanping Fu, Qingan Yan, Long Yang, Jie Liao, and Chunxia Xiao. 2018. Texture mapping for 3D reconstruction with RGB-D sensor. In Proceedings of CVPR. 4645–4653.

[14]

Y. Furukawa, B. Curless, S. M. Seitz, and R. Szeliski. 2010. Towards Internet-scale multi-view stereo. In Proceedings of CVPR. 1434–1441.

[15]

Y. Furukawa and J. Ponce. 2010. Accurate, dense, and robust multiview stereopsis. IEEE Transactions on Pattern Analysis and Machine Intelligence 32, 8 (Aug 2010), 1362–1376.

Digital Library

[16]

Ran Gal, Yonatan Wexler, Eyal Ofek, Hugues Hoppe, and Daniel Cohen-Or. 2010. Seamless montage for texturing models. In Computer Graphics Forum, Vol. 29. 479–486.

[17]

Steven J. Gortler, Radek Grzeszczuk, Richard Szeliski, and Michael F. Cohen. 1996. The lumigraph. In Proceedings of SIGGRAPH Conference.43–54.

Digital Library

[18]

Jingwei Huang, Angela Dai, Leonidas J. Guibas, and Matthias Nießner. 2017. 3Dlite: Towards commodity 3D scanning for content creation. ACM Transactions on Graphics 36, 6 (2017), 203–1.

Digital Library

[19]

Jingwei Huang, Justus Thies, Angela Dai, Abhijit Kundu, Chiyu Jiang, Leonidas J. Guibas, Matthias Nießner, Thomas Funkhouser, et al. 2020. Adversarial texture optimization from RGB-D scans. In Proceedings of CVPR. 1559–1568.

[20]

Junho Jeon, Yeongyu Jung, Haejoon Kim, and Seungyong Lee. 2016. Texture map generation for 3D reconstructed scenes. The Visual Computer 32, 6–8 (2016), 955–965.

Digital Library

[21]

Michael Kazhdan and Hugues Hoppe. 2013. Screened Poisson surface reconstruction. ACM Transactions on Graphics 32, 3, Article 29 (July 2013), 13 pages.

Digital Library

[22]

Joo Ho Lee, Hyunho Ha, Yue Dong, Xin Tong, and Min H. Kim. 2020. TextureFusion: High-quality texture acquisition for real-time RGB-D scanning. In Proceedings of CVPR. 1269–1277.

[23]

Victor Lempitsky and Denis Ivanov. 2007. Seamless mosaicing of image-based texture maps. In Proceedings of CVPR. 1–6.

[24]

Marc Levoy and Pat Hanrahan. 1996. Light field rendering. In Proceedings of SIGGRAPH Conference.31–42.

Digital Library

[25]

Wei Li, Huajun Gong, and Ruigang Yang. 2018. Fast texture mapping adjustment via local/global optimization. IEEE Transactions on Visualization and Computer Graphics 25, 6 (2018), 2296–2303.

[26]

Lingjie Liu, Jiatao Gu, Kyaw Zaw Lin, Tat-Seng Chua, and Christian Theobalt. 2020. Neural sparse voxel fields. In Proceedings of NeurIPS.

[27]

William E. Lorensen and Harvey E. Cline. 1987. Marching cubes: A high resolution 3D surface construction algorithm. In Proceedings of SIGGRAPH Conference.163–169.

Digital Library

[28]

Kok-Lim Low. 2004. Linear least-squares optimization for point-to-plane ICP surface registration. Technical Report TR04-004, Chapel Hill, University of North Carolina,1–3.

[29]

Robert Maier, Kihwan Kim, Daniel Cremers, Jan Kautz, and Matthias Nießner. 2017. Intrinsic3D: High-quality 3D reconstruction by joint appearance and geometry optimization with spatially-varying lighting. In Proceedings of ICCV. 3114–3122.

[30]

Microsoft. 2020. Azure Kinect DK. Retrieved from https://azure.microsoft.com/en-us/services/kinect-dk/Online; accessed 19 Jan 2020.

[31]

Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. 2020. NeRF: Representing scenes as neural radiance fields for view synthesis. In Proceedings of ECCV. 405–421.

Digital Library

[32]

Richard A. Newcombe, Dieter Fox, and Steven M. Seitz. 2015. DynamicFusion: Reconstruction and tracking of non-rigid scenes in real-time. In Proceedings of CVPR. 343–352.

[33]

Richard A. Newcombe, Shahram Izadi, Otmar Hilliges, David Molyneaux, David Kim, Andrew J. Davison, Pushmeet Kohi, Jamie Shotton, Steve Hodges, and Andrew Fitzgibbon. 2011. KinectFusion: Real-time dense surface mapping and tracking. In Proceedings of ISMAR. 127–136.

Digital Library

[34]

Matthias Nießner, Michael Zollhöfer, Shahram Izadi, and Marc Stamminger. 2013. Real-time 3D reconstruction at scale using voxel hashing. ACM Transactions on Graphics 32, 6 (2013), 169.

Digital Library

[35]

J. Park, Q. Zhou, and V. Koltun. 2017. Colored point cloud registration revisited. In Proceedings of ICCV. 143–152.

[36]

Patrick Pérez, Michel Gangnet, and Andrew Blake. 2003. Poisson image editing. In Proceedings of SIGGRAPH Conference.313–318.

Digital Library

[37]

Justus Thies, Michael Zollhöfer, and Matthias Nießner. 2019. Deferred neural rendering: Image synthesis using neural textures. ACM Transactions on Graphics 38, 4 (2019), 1–12.

Digital Library

[38]

Greg Turk and Marc Levoy. 1994. Zippered polygon meshes from range images. In Proceedings of SIGGRAPH Conference.311–318.

Digital Library

[39]

Luiz Velho and Jonas Sossai Jr. 2007. Projective texture atlas construction for 3D photography. The Visual Computer 23, 9 (2007), 621–629.

Digital Library

[40]

Michael Waechter, Mate Beljan, Simon Fuhrmann, Nils Moehrle, Johannes Kopf, and Michael Goesele. 2017. Virtual rephotography: Novel view prediction error for 3D reconstruction. ACM Transactions on Graphics 36, 1, Article 8 (Jan.2017), 11 pages.

Digital Library

[41]

Michael Waechter, Nils Moehrle, and Michael Goesele. 2014. Let there be color! Large-scale texturing of 3D reconstructions. In Proceedings of ECCV. 836–850.

[42]

Thomas Whelan, Stefan Leutenegger, R. Salas-Moreno, Ben Glocker, and Andrew Davison. 2015. ElasticFusion: Dense SLAM without a pose graph. Robotics: Science and Systems. In Proceedings of RSS.

[43]

Thomas Whelan, John McDonald, Michael Kaess, Maurice Fallon, Hordur Johannsson, and John J. Leonard. 2012. Kintinuous: Spatially extended KinectFusion. In Proceedings of RSS’12 Workshop on RGB-D: Advanced Reasoning with Depth Cameras.

[44]

Zexiang Xu, Sai Bi, Kalyan Sunkavalli, Sunil Hadap, Hao Su, and Ravi Ramamoorthi. 2019. Deep view synthesis from sparse photometric images. ACM Transactions on Graphics 38, 4 (2019), 1–13.

Digital Library

[45]

K. Zhou, M. Gong, X. Huang, and B. Guo. 2011. Data-parallel octrees for surface reconstruction. IEEE Transactions on Visualization and Computer Graphics 17, 5 (May 2011), 669–681.

Digital Library

[46]

Qian-Yi Zhou and Vladlen Koltun. 2013. Dense scene reconstruction with points of interest. ACM Transactions on Graphics 32, 4 (2013), 1–8.

Digital Library

[47]

Qian-Yi Zhou and Vladlen Koltun. 2014. Color map optimization for 3D reconstruction with consumer depth cameras. ACM Transactions on Graphics 33, 4 (2014), 155.

Digital Library

[48]

Qian-Yi Zhou, Stephen Miller, and Vladlen Koltun. 2013. Elastic fragments for dense scene reconstruction. In Proceedings of ICCV. 473–480.

Digital Library

Cited By

Liu YXiaoqun W(2024)Survey of texture optimization algorithms for 3D reconstructed scenesJournal of Image and Graphics10.11834/jig.23047829:8(2303-2318)Online publication date: 2024
https://doi.org/10.11834/jig.230478
Fukaya KDaylamani-Zad DAgius H(2024)Evaluation Metrics for Intelligent Generation of Graphical Game Assets: A Systematic Survey-Based FrameworkIEEE Transactions on Pattern Analysis and Machine Intelligence10.1109/TPAMI.2024.339899846:12(7998-8017)Online publication date: Dec-2024
https://doi.org/10.1109/TPAMI.2024.3398998
Kuo GSchiffers FLanman DCossairt OMatsuda N(2023)Multisource HolographyACM Transactions on Graphics10.1145/361838042:6(1-14)Online publication date: 5-Dec-2023
https://dl.acm.org/doi/10.1145/3618380
Show More Cited By

Index Terms

TextureMe: High-Quality Textured Scene Reconstruction in Real Time
1. Computing methodologies
  1. Computer graphics
    1. Image manipulation
      1. Texturing
    2. Shape modeling
      1. Mesh models

Recommendations

High Quality Photometric Reconstruction Using a Depth Camera
CVPR '14: Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition

In this paper we present a depth-guided photometric 3D reconstruction method that works solely with a depth camera like the Kinect. Existing methods that fuse depth with normal estimates use an external RGB camera to obtain photometric information and ...
Depth-map completion for large indoor scene reconstruction
Highlights
- Propose a new depth completion algorithm for MVS depth-maps.
- Use occlusion ...
Abstract
Traditional Multi View Stereo (MVS) algorithms are often difficult to deal with large-scale indoor scene reconstruction, due to the photo-consistency measurement errors in weak textured regions, which are commonly exist in indoor ...
Texture map generation for 3D reconstructed scenes

We present a novel method for generating texture maps for 3D geometric models reconstructed using consumer RGB-D sensors. Our method generates a texture map for a simplified 3D mesh of the reconstructed scene using spatially and temporally sub-sampled ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Transactions on Graphics

ACM Transactions on Graphics Volume 41, Issue 3

June 2022

213 pages

ISSN:0730-0301

EISSN:1557-7368

DOI:10.1145/3517033

Editor:
Carol O'Sullivan
Trinity College Dublin, Ireland

Issue’s Table of Contents

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 07 March 2022

Accepted: 01 December 2021

Revised: 01 November 2021

Received: 01 November 2020

Published in TOG Volume 41, Issue 3

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Refereed

Funding Sources

Ministry of Science and ICT, Korea

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

8
Total Citations
View Citations
2,250
Total Downloads

Downloads (Last 12 months)353
Downloads (Last 6 weeks)25

Reflects downloads up to 05 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Liu YXiaoqun W(2024)Survey of texture optimization algorithms for 3D reconstructed scenesJournal of Image and Graphics10.11834/jig.23047829:8(2303-2318)Online publication date: 2024
https://doi.org/10.11834/jig.230478
Fukaya KDaylamani-Zad DAgius H(2024)Evaluation Metrics for Intelligent Generation of Graphical Game Assets: A Systematic Survey-Based FrameworkIEEE Transactions on Pattern Analysis and Machine Intelligence10.1109/TPAMI.2024.339899846:12(7998-8017)Online publication date: Dec-2024
https://doi.org/10.1109/TPAMI.2024.3398998
Kuo GSchiffers FLanman DCossairt OMatsuda N(2023)Multisource HolographyACM Transactions on Graphics10.1145/361838042:6(1-14)Online publication date: 5-Dec-2023
https://dl.acm.org/doi/10.1145/3618380
Li ZZheng ZLiu YZhou BLiu Y(2023)PoseVocab: Learning Joint-structured Pose Embeddings for Human Avatar ModelingACM SIGGRAPH 2023 Conference Proceedings10.1145/3588432.3591490(1-11)Online publication date: 23-Jul-2023
https://dl.acm.org/doi/10.1145/3588432.3591490
Do JKim SKang J(2023)Seam-Aware Rendering Quality Enhancement Network For Compressed 3D Scene2023 IEEE International Conference on Visual Communications and Image Processing (VCIP)10.1109/VCIP59821.2023.10402698(1-5)Online publication date: 4-Dec-2023
https://doi.org/10.1109/VCIP59821.2023.10402698
Gao YCao YShan Y(2023)SurfelNeRF: Neural Surfel Radiance Fields for Online Photorealistic Reconstruction of Indoor Scenes2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR52729.2023.00019(108-118)Online publication date: Jun-2023
https://doi.org/10.1109/CVPR52729.2023.00019
Liu XLi JLu G(2023)Generating High-Fidelity Texture in RGB-D Reconstruction using Patches Density RegularizationComputer-Aided Design10.1016/j.cad.2023.103516160:COnline publication date: 1-Jul-2023
https://dl.acm.org/doi/10.1016/j.cad.2023.103516
Rebol MPietroszek KRanniger CHood CRutenberg ASikka NLi DGütl C(2022)Mixed Reality Communication for Medical Procedures: Teaching the Placement of a Central Venous Catheter2022 IEEE International Symposium on Mixed and Augmented Reality (ISMAR)10.1109/ISMAR55827.2022.00050(346-354)Online publication date: Oct-2022
https://doi.org/10.1109/ISMAR55827.2022.00050

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Full Text

View this article in Full Text.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View full text|Download PDF

View Issue’s Table of Contents