More Web Proxy on the site http://driver.im/

research-article

Photo-to-shape material transfer for diverse structures

Authors:

Oliver Van Kaick,

Hui HuangAuthors Info & Claims

ACM Transactions on Graphics (TOG), Volume 41, Issue 4

Article No.: 131, Pages 1 - 14

https://doi.org/10.1145/3528223.3530088

Published: 22 July 2022 Publication History

Abstract

We introduce a method for assigning photorealistic relightable materials to 3D shapes in an automatic manner. Our method takes as input a photo exemplar of a real object and a 3D object with segmentation, and uses the exemplar to guide the assignment of materials to the parts of the shape, so that the appearance of the resulting shape is as similar as possible to the exemplar. To accomplish this goal, our method combines an image translation neural network with a material assignment neural network. The image translation network translates the color from the exemplar to a projection of the 3D shape and the part segmentation from the projection to the exemplar. Then, the material prediction network assigns materials from a collection of realistic materials to the projected parts, based on the translated images and perceptual similarity of the materials.

One key idea of our method is to use the translation network to establish a correspondence between the exemplar and shape projection, which allows us to transfer materials between objects with diverse structures. Another key idea of our method is to use the two pairs of (color, segmentation) images provided by the image translation to guide the material assignment, which enables us to ensure the consistency in the assignment. We demonstrate that our method allows us to assign materials to shapes so that their appearances better resemble the input exemplars, improving the quality of the results over the state-of-the-art method, and allowing us to automatically create thousands of shapes with high-quality photorealistic materials. Code and data for this paper are available at https://github.com/XiangyuSu611/TMT.

Supplemental Material

MP4 File

presentation

Transcript for: presentation

ZIP File

supplemental material

Download
9.40 MB

References

[1]

Sean Bell, Paul Upchurch, Noah Snavely, and Kavita Bala. 2013. OpenSurfaces: A Richly Annotated Catalog of Surface Appearance. ACM Trans. on Graphics (Proc. SIGGRAPH) 32, 4 (2013), 111:1--17.

Digital Library

[2]

Sean Bell, Paul Upchurch, Noah Snavely, and Kavita Bala. 2015. Material Recognition in the Wild With the Materials in Context Database. In Proc. IEEE Conf. on Computer Vision & Pattern Recognition.

[3]

Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. 2018. Encoder-decoder with atrous separable convolution for semantic image segmentation. In Proceedings of the European conference on computer vision (ECCV). 801--818.

Digital Library

[4]

Qifeng Chen and Vladlen Koltun. 2017. Photographic Image Synthesis With Cascaded Refinement Networks. In Proc. Int. Conf. on Computer Vision.

[5]

Zhiqin Chen and Hao Zhang. 2019. Learning Implicit Fields for Generative Shape Modeling. In Proc. IEEE Conf. on Computer Vision & Pattern Recognition. 5939--5948.

[6]

Julian Chibane, Aymen Mir, and Gerard Pons-Moll. 2020. Neural Unsigned Distance Fields for Implicit Function Learning. In Advances in Neural Information Processing Systems (NeurIPS).

[7]

Johanna Delanoy, Mathieu Aubry, Phillip Isola, Alexei A. Efros, and Adrien Bousseau. 2018. 3D Sketching Using Multi-View Deep Volumetric Prediction. In Proc. ACM Comput. Graph. Interact. Tech. 21:1--15.

Digital Library

[8]

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition. Ieee, 248--255.

[9]

D. Guarnera, G. C. Guarnera, A. Ghosh, C. Denk, and M. Glencross. 2016. BRDF Representation and Acquisition. In Proc. Eurographics State of the Art Reports. Eurographics Association, 625--650.

[10]

Xian-Feng Han, Hamid Laga, and Mohammed Bennamoun. 2021. Image-Based 3D Object Reconstruction: State-of-the-Art and Trends in the Deep Learning Era. IEEE Trans. Pattern Analysis & Machine Intelligence 43, 5 (2021), 1578--1604.

[11]

Vlastimil Havran, Jiri Filip, and Karol Myszkowski. 2016. Perceptually motivated BRDF comparison using single image. In Computer graphics forum, Vol. 35. Wiley Online Library, 1--12.

[12]

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. 770--778.

[13]

Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. 2017. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems 30 (2017).

[14]

Elad Hoffer and Nir Ailon. 2015. Deep metric learning using triplet network. In International workshop on similarity-based pattern recognition. Springer, 84--92.

[15]

Tao Hu, Geng Lin, Zhizhong Han, and Matthias Zwicker. 2021. Learning to Generate Dense Point Clouds With Textures on Multiple Categories. In Proc. IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). 2170--2179.

[16]

Yiwei Hu, Julie Dorsey, and Holly Rushmeier. 2019. A Novel Framework for Inverse Procedural Texture Modeling. ACM Trans. on Graphics (Proc. SIGGRAPH Asia) 38, 6 (2019), 186:1--14.

[17]

Hui Huang, Ke Xie, Lin Ma, Dani Lischinski, Minglun Gong, Xin Tong, and Daniel Cohen-Or. 2018b. Appearance Modeling via Proxy-to-Image Alignment. ACM Trans. on Graphics 37, 1 (2018), 10:1--15.

Digital Library

[18]

Qixing Huang, Hai Wang, and Vladlen Koltun. 2015. Single-view reconstruction via joint analysis of image and shape collections. ACM Transactions on Graphics (TOG) 34, 4 (2015), 1--10.

Digital Library

[19]

Xun Huang, Ming-Yu Liu, Serge Belongie, and Jan Kautz. 2018a. Multimodal Unsupervised Image-to-image Translation. In Proc. Euro. Conf. on Computer Vision.

Digital Library

[20]

Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A. Efros. 2017. Image-To-Image Translation With Conditional Adversarial Networks. In Proc. IEEE Conf. on Computer Vision & Pattern Recognition.

[21]

Arjun Jain, Thorsten Thormählen, Tobias Ritschel, and Hans-Peter Seidel. 2012. Material memex: Automatic material suggestions for 3d objects. ACM Transactions on Graphics (TOG) 31, 6 (2012), 1--8.

Digital Library

[22]

Mengqi Ji, Juergen Gall, Haitian Zheng, Yebin Liu, and Lu Fang. 2017. SurfaceNet: An End-To-End 3D Neural Network for Multiview Stereopsis. In Proc. Int. Conf. on Computer Vision. 2307--2315.

[23]

Evangelos Kalogerakis, Melinos Averkiou, Subhransu Maji, and Siddhartha Chaudhuri. 2017. 3D Shape Segmentation with Projective Convolutional Networks. In Proc. IEEE Conf. on Computer Vision & Pattern Recognition.

[24]

Manuel Lagunas, Sandra Malpica, Ana Serrano, Elena Garces, Diego Gutierrez, and Belen Masia. 2019a. A Similarity Measure for Material Appearance. ACM Trans. on Graphics (Proc. SIGGRAPH) 38, 4 (2019), 135:1--12.

Digital Library

[25]

Manuel Lagunas, Sandra Malpica, Ana Serrano, Elena Garces, Diego Gutierrez, and Belen Masia. 2019b. A similarity measure for material appearance. ACM Transactions on Graphics (TOG) 38, 4 (2019), 1--12.

Digital Library

[26]

Juncheng Liu, Zhouhui Lian, and Jianguo Xiao. 2017. Auto-Colorization of 3D Models from Images. In SIGGRAPH Asia 2017 Technical Briefs. 15:1--4.

[27]

Aymen Mir, Thiemo Alldieck, and Gerard Pons-Moll. 2020. Learning to Transfer Texture from Clothing Images to 3D Humans. In Proc. IEEE Conf. on Computer Vision & Pattern Recognition. IEEE.

[28]

Kaichun Mo, Shilin Zhu, Angel X Chang, Li Yi, Subarna Tripathi, Leonidas J Guibas, and Hao Su. 2019. Partnet: A large-scale benchmark for fine-grained and hierarchical part-level 3d object understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 909--918.

[29]

Chuong H Nguyen, Tobias Ritschel, Karol Myszkowski, Elmar Eisemann, and Hans-Peter Seidel. 2012. 3D material style transfer. In Computer Graphics Forum, Vol. 31. Wiley Online Library, 431--438.

[30]

Michael Oechsle, Lars Mescheder, Michael Niemeyer, Thilo Strauss, and Andreas Geiger. 2019. Texture fields: Learning texture representations in function space. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 4531--4540.

[31]

Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. 2019. DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation. In Proc. IEEE Conf. on Computer Vision & Pattern Recognition. 165--174.

[32]

Keunhong Park, Konstantinos Rematas, Ali Farhadi, and Steven M Seitz. 2018. PhotoShape: photorealistic materials for large-scale shape collections. ACM Trans. on Graphics (Proc. SIGGRAPH Asia) 37, 6 (2018), 1--12.

Digital Library

[33]

Quang-Hieu Pham, Thanh Nguyen, Binh-Son Hua, Gemma Roig, and Sai-Kit Yeung. 2019. JSIS3D: Joint Semantic-Instance Segmentation of 3D Point Clouds With Multi-Task Pointwise Networks and Multi-Value Conditional Random Fields. In Proc. IEEE Conf. on Computer Vision & Pattern Recognition.

[34]

Amit Raj, Cusuh Ham, Connelly Barnes, Vladimir Kim, Jingwan Lu, and James Hays. 2019. Learning to Generate Textures on 3D Meshes. In Proc. IEEE Conf. on Computer Vision & Pattern Recognition.

[35]

Konstantinos Rematas, Chuong H Nguyen, Tobias Ritschel, Mario Fritz, and Tinne Tuytelaars. 2016. Novel views of objects from a single image. IEEE transactions on pattern analysis and machine intelligence 39, 8 (2016), 1576--1590.

[36]

Thorsten-Walther Schmidt, Fabio Pellacini, Derek Nowrouzezahrai, Wojciech Jarosz, and Carsten Dachsbacher. 2016. State of the Art in Artistic Editing of Appearance, Lighting and Material. Computer Graphics Forum 35, 1 (2016), 216--233.

Digital Library

[37]

Ana Serrano, Diego Gutierrez, Karol Myszkowski, Hans-Peter Seidel, and Belen Masia. 2016. An Intuitive Control Space for Material Appearance. ACM Trans. on Graphics (Proc. SIGGRAPH Asia) 35, 6 (2016), 186:1--12.

[38]

Hao Su, Qixing Huang, Niloy J Mitra, Yangyan Li, and Leonidas Guibas. 2014. Estimating image depth using shape collections. ACM Transactions on Graphics (TOG) 33, 4 (2014), 1--11.

Digital Library

[39]

Tiancheng Sun, Ana Serrano, Diego Gutierrez, and Belen Masia. 2017. Attribute-preserving gamut mapping of measured BRDFs. In Computer graphics forum, Vol. 36. Wiley Online Library, 47--54.

[40]

Tuanfeng Y Wang, Hao Su, Qixing Huang, Jingwei Huang, Leonidas J Guibas, and Niloy J Mitra. 2016. Unsupervised texture transfer from images to model collections. ACM Trans. on Graphics (Proc. SIGGRAPH Asia) 35, 6 (2016), 177--1.

[41]

Jiajun Wu, Yifan Wang, Tianfan Xue, Xingyuan Sun, William T Freeman, and Joshua B Tenenbaum. 2017. MarrNet: 3D Shape Reconstruction via 2.5D Sketches. In Proc. Conf. on Neural Information Processing Systems.

[42]

Qiangeng Xu, Weiyue Wang, Duygu Ceylan, Radomir Mech, and Ulrich Neumann. 2019. Disn: Deep implicit surface network for high-quality single-view 3d reconstruction. arXiv preprint arXiv:1905.10711 (2019).

[43]

Bo Zhang, Mingming He, Jing Liao, Pedro V Sander, Lu Yuan, Amine Bermak, and Dong Chen. 2019. Deep exemplar-based video colorization. In Proc. IEEE Conf. on Computer Vision & Pattern Recognition. 8052--8061.

[44]

Pan Zhang, Bo Zhang, Dong Chen, Lu Yuan, and Fang Wen. 2020. Cross-Domain Correspondence Learning for Exemplar-Based Image Translation. In Proc. IEEE Conf. on Computer Vision & Pattern Recognition. IEEE.

[45]

Jie Zhu, Yanwen Guo, and Han Ma. 2018. A Data-Driven Approach for Furniture and Indoor Scene Colorization. IEEE Trans. Visualization & Computer Graphics 24, 9 (2018), 2473--2486.

Cited By

Gallega RSumi Y(2024)Exploring the use of generative AI for material texturing in 3D interior design spacesFrontiers in Computer Science10.3389/fcomp.2024.14939376Online publication date: 28-Nov-2024
https://doi.org/10.3389/fcomp.2024.1493937
Ru XZhao HWang XWu ZLiu SZhu YFrangi AGurrin CKongkachandra RSchoeffmann KDang-Nguyen DRossetto LSatoh SZhou L(2024)Vector-Aware Anisotropic Gauge Equivariant Mesh Convolution Network for 3D Aneurysm DetectionProceedings of the 2024 International Conference on Multimedia Retrieval10.1145/3652583.3658072(248-256)Online publication date: 30-May-2024
https://dl.acm.org/doi/10.1145/3652583.3658072
Zhang SPeng SXu TYang YChen TXue NShen YBao HHu RZhou X(2024)MaPa: Text-driven Photorealistic Material Painting for 3D ShapesACM SIGGRAPH 2024 Conference Papers10.1145/3641519.3657504(1-12)Online publication date: 13-Jul-2024
https://dl.acm.org/doi/10.1145/3641519.3657504
Show More Cited By

Index Terms

Photo-to-shape material transfer for diverse structures
1. Computing methodologies
  1. Computer graphics
    1. Shape modeling
      1. Mesh geometry models

Recommendations

3D Material Style Transfer

This work proposes a technique to transfer the material style or mood from a guide source such as an image or video onto a target 3D scene. It formulates the problem as a combinatorial optimization of assigning discrete materials extracted from the ...
Unsupervised Multi-modal Medical Image Registration via Invertible Translation
Computer Vision – ECCV 2024
Abstract
In medical imaging, the alignment of multi-modal images plays a critical role in providing comprehensive information for image-guided therapies. Despite its importance, multi-modal image registration poses significant challenges due to the complex ...
Statistical shape analysis of anatomical structures

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Transactions on Graphics

ACM Transactions on Graphics Volume 41, Issue 4

July 2022

1978 pages

ISSN:0730-0301

EISSN:1557-7368

DOI:10.1145/3528223

Issue’s Table of Contents

Copyright © 2022 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 22 July 2022

Published in TOG Volume 41, Issue 4

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

NSFC
GD Natural Science Foundation
DEGP Key Project
Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ)
GD Talent Plan
Shenzhen Science and Technology Program

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

25
Total Citations
View Citations
253
Total Downloads

Downloads (Last 12 months)29
Downloads (Last 6 weeks)3

Reflects downloads up to 02 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Gallega RSumi Y(2024)Exploring the use of generative AI for material texturing in 3D interior design spacesFrontiers in Computer Science10.3389/fcomp.2024.14939376Online publication date: 28-Nov-2024
https://doi.org/10.3389/fcomp.2024.1493937
Ru XZhao HWang XWu ZLiu SZhu YFrangi AGurrin CKongkachandra RSchoeffmann KDang-Nguyen DRossetto LSatoh SZhou L(2024)Vector-Aware Anisotropic Gauge Equivariant Mesh Convolution Network for 3D Aneurysm DetectionProceedings of the 2024 International Conference on Multimedia Retrieval10.1145/3652583.3658072(248-256)Online publication date: 30-May-2024
https://dl.acm.org/doi/10.1145/3652583.3658072
Zhang SPeng SXu TYang YChen TXue NShen YBao HHu RZhou X(2024)MaPa: Text-driven Photorealistic Material Painting for 3D ShapesACM SIGGRAPH 2024 Conference Papers10.1145/3641519.3657504(1-12)Online publication date: 13-Jul-2024
https://dl.acm.org/doi/10.1145/3641519.3657504
Guerrero-Viu JHasan MRoullier AHarikumar MHu YGuerrero PGutiérrez DMasia BDeschaintre V(2024)TexSliders: Diffusion-Based Texture Editing in CLIP SpaceACM SIGGRAPH 2024 Conference Papers10.1145/3641519.3657444(1-11)Online publication date: 13-Jul-2024
https://dl.acm.org/doi/10.1145/3641519.3657444
Zheng ZGao XPan ZLi WWang PWang GWu K(2024)Visual-Preserving Mesh RepairIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2023.334882930:9(6586-6597)Online publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1109/TVCG.2023.3348829
Mu TShen MLai YHu S(2024)Learning Virtual View Selection for 3D Scene Semantic SegmentationIEEE Transactions on Image Processing10.1109/TIP.2024.342195233(4159-4172)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.1109/TIP.2024.3421952
Shu ZWu TShen JXin SLiu L(2024)Semi-Supervised 3D Shape Segmentation via Self RefiningIEEE Transactions on Image Processing10.1109/TIP.2024.337420033(2044-2057)Online publication date: 12-Mar-2024
https://dl.acm.org/doi/10.1109/TIP.2024.3374200
Fan LHe FSi TFan RYe CLi B(2024)MBA: Backdoor Attacks Against 3D Mesh ClassifierIEEE Transactions on Information Forensics and Security10.1109/TIFS.2023.334664419(2127-2142)Online publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1109/TIFS.2023.3346644
Zhang SMa JWu JRitchie DAgrawala M(2023)Editing Motion Graphics Video via Motion Vectorization and TransformationACM Transactions on Graphics10.1145/361831642:6(1-13)Online publication date: 5-Dec-2023
https://dl.acm.org/doi/10.1145/3618316
Yan KLuan FHašan MGroueix TDeschaintre VZhao S(2023)PSDR-Room: Single Photo to Scene using Differentiable RenderingSIGGRAPH Asia 2023 Conference Papers10.1145/3610548.3618165(1-11)Online publication date: 10-Dec-2023
https://dl.acm.org/doi/10.1145/3610548.3618165
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Issue’s Table of Contents