[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to main content
Springer Nature Link
Log in

MatTrans: Material Reflectance Property Estimation of Complex Objects with Transformer

  • Conference paper
  • First Online:
Computational Visual Media (CVM 2024)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 14592))

Included in the following conference series:

Abstract

Material Reflectance Property Estimation of an object is challenging and it can be used in realistic rendering to make the appearance of objects realistic. Current research focuses primarily on the near-planar objects, with little attention paid to complex-shaped objects. In this paper, we propose a method called MatTrans to estimate geometry and material reflectance properties with Transformer. Specifically, a Transformer Encoder module is designed to fuse local and global information for each material property respectively, and then a cascaded network with residual learning is introduced to estimate the geometry and reflectance properties of any 3D object surface from a single image. Extensive experiments validate that our method brings a clear improvement over previous methods for single-shot capture of spatially varying BRDFs.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
£29.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
GBP 19.95
Price includes VAT (United Kingdom)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
GBP 79.50
Price includes VAT (United Kingdom)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
GBP 99.99
Price includes VAT (United Kingdom)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Aittala, M., Weyrich, T., Lehtinen, J., et al.: Two-shot SVBRDF capture for stationary materials. ACM Trans. Graph. 34(4), 110–1 (2015)

    Article  Google Scholar 

  2. Baek, S.H., Jeon, D.S., Tong, X., Kim, M.H.: Simultaneous acquisition of polarimetric SVBRDF and normals. ACM Trans. Graph. 37(6), 1–268 (2018)

    Article  Google Scholar 

  3. Bi, S., Xu, Z., Sunkavalli, K., Kriegman, D., Ramamoorthi, R.: Deep 3d capture: geometry and reflectance from sparse multi-view images. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5960–5969 (2020)

    Google Scholar 

  4. Cheng, B., Zhao, J., Duan, F.: Material reflectance property estimation of complex objects using an attention network. In: 2022 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW), pp. 632–633. IEEE (2022)

    Google Scholar 

  5. Cook, R.L., Torrance, K.E.: A reflectance model for computer graphics. ACM SIGGRAPH Comput. Graph. 15(3), 307–316 (1981)

    Article  Google Scholar 

  6. Cook, R.L., Torrance, K.E.: A reflectance model for computer graphics. ACM Trans. Graph. (ToG) 1(1), 7–24 (1982)

    Article  Google Scholar 

  7. Deschaintre, V., Aittala, M., Durand, F., Drettakis, G., Bousseau, A.: Single-image SVBRDF capture with a rendering-aware deep network. ACM Trans. Graph. (ToG) 37(4), 1–15 (2018)

    Article  Google Scholar 

  8. Dong, Y., Chen, G., Peers, P., Zhang, J., Tong, X.: Appearance-from-motion: recovering spatially varying surface reflectance under unknown lighting. ACM Trans. Graph. (TOG) 33(6), 1–12 (2014)

    Article  Google Scholar 

  9. Dosovitskiy, A., et al.: An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 (2020)

  10. Gao, D., Li, X., Dong, Y., Peers, P., Xu, K., Tong, X.: Deep inverse rendering for high-resolution SVBRDF estimation from an arbitrary number of images. ACM Trans. Graph. 38(4), 1–134 (2019)

    Article  Google Scholar 

  11. Goodfellow, I., et al.: Generative adversarial nets, in ‘advances in neural information processing systems 27’, Curran Associates (2014)

    Google Scholar 

  12. Guo, J., et al.: Highlight-aware two-stream network for single-image SVBRDF acquisition. ACM Trans. Graph. (TOG) 40(4), 1–14 (2021)

    Article  Google Scholar 

  13. Guo, Y., Smith, C., Hašan, M., Sunkavalli, K., Zhao, S.: MaterialGAN: reflectance capture using a generative SVBRDF model. arXiv preprint arXiv:2010.00114 (2020)

  14. Hasselgren, J., Hofmann, N., Munkberg, J.: Shape, light, and material decomposition from images using monte Carlo rendering and denoising. Adv. Neural. Inf. Process. Syst. 35, 22856–22869 (2022)

    Google Scholar 

  15. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)

    Google Scholar 

  16. Ho, J., Jain, A., Abbeel, P.: Denoising diffusion probabilistic models. Adv. Neural. Inf. Process. Syst. 33, 6840–6851 (2020)

    Google Scholar 

  17. Holroyd, M., Lawrence, J., Zickler, T.: A coaxial optical scanner for synchronous acquisition of 3d geometry and surface reflectance. ACM Trans. Graph. (TOG) 29(4), 1–12 (2010)

    Article  Google Scholar 

  18. Kang, K., Chen, Z., Wang, J., Zhou, K., Wu, H.: Efficient reflectance capture using an autoencoder. ACM Trans. Graph. 37(4), 127–1 (2018)

    Article  Google Scholar 

  19. Karras, T., Laine, S., Aittala, M., Hellsten, J., Lehtinen, J., Aila, T.: Analyzing and improving the image quality of StyleGAN. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8110–8119 (2020)

    Google Scholar 

  20. Lagarde, S.: Spherical gaussian approximation for Blinn-Phong, Phong and Fresnel. Random Thoughts Graphics in Games blog, 3 June 2012

    Google Scholar 

  21. Li, X., Dong, Y., Peers, P., Tong, X.: Modeling surface appearance from a single photograph using self-augmented convolutional neural networks. ACM Trans. Graph. (ToG) 36(4), 1–11 (2017)

    Google Scholar 

  22. Li, Z., Shafiei, M., Ramamoorthi, R., Sunkavalli, K., Chandraker, M.: Inverse rendering for complex indoor scenes: shape, spatially-varying lighting and SVBRDF from a single image. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2475–2484 (2020)

    Google Scholar 

  23. Li, Z., Sunkavalli, K., Chandraker, M.: Materials for masses: SVBRDF acquisition with a single mobile phone image. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11207, pp. 74–90. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01219-9_5

    Chapter  Google Scholar 

  24. Li, Z., Xu, Z., Ramamoorthi, R., Sunkavalli, K., Chandraker, M.: Learning to reconstruct shape and spatially-varying reflectance from a single image. ACM Trans. Graph. (TOG) 37(6), 1–11 (2018)

    Article  Google Scholar 

  25. Luan, F., Zhao, S., Bala, K., Dong, Z.: Unified shape and SVBRDF recovery using differentiable monte Carlo rendering. In: Computer Graphics Forum, vol. 40, pp. 101–113. Wiley Online Library (2021)

    Google Scholar 

  26. Munkberg, J., et al.: Extracting triangular 3d models, materials, and lighting from images. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8280–8290 (2022)

    Google Scholar 

  27. Nam, G., Lee, J.H., Gutierrez, D., Kim, M.H.: Practical SVBRDF acquisition of 3d objects with unstructured flash photography. ACM Trans. Graph. (TOG) 37(6), 1–12 (2018)

    Article  Google Scholar 

  28. Riviere, J., Peers, P., Ghosh, A.: Mobile surface reflectometry. In: ACM SIGGRAPH 2014 Posters, pp. 1–1 (2014)

    Google Scholar 

  29. Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-resolution image synthesis with latent diffusion models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10684–10695 (2022)

    Google Scholar 

  30. Ronneberger, O., Fischer, P., Brox, T.: U-net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-24574-4_28

    Chapter  Google Scholar 

  31. Sang, S., Chandraker, M.: Single-shot neural relighting and SVBRDF estimation. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12364, pp. 85–101. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58529-7_6

    Chapter  Google Scholar 

  32. Schlick, C.: An inexpensive BRDF model for physically-based rendering. In: Computer Graphics Forum, vol. 13, pp. 233–246. Wiley Online Library (1994)

    Google Scholar 

  33. Tunwattanapong, B., et al.: Acquiring reflectance and shape from continuous spherical harmonic illumination. ACM Trans. Graph. (TOG) 32(4), 1–12 (2013)

    Article  Google Scholar 

  34. Vaswani, A., et al.: Attention is all you need. In: Advances in Neural Information Processing Systems, vol. 30 (2017)

    Google Scholar 

  35. Vecchio, G., et al.: Controlmat: a controlled generative approach to material capture. arXiv preprint arXiv:2309.01700 (2023)

  36. Vecchio, G., Sortino, R., Palazzo, S., Spampinato, C.: Matfuse: controllable material generation with diffusion models. arXiv preprint arXiv:2308.11408 (2023)

  37. Walter, B., Marschner, S.R., Li, H., Torrance, K.E.: Microfacet models for refraction through rough surfaces. In: Proceedings of the 18th Eurographics Conference on Rendering Techniques, pp. 195–206 (2007)

    Google Scholar 

  38. Wang, X., Girshick, R., Gupta, A., He, K.: Non-local neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7794–7803 (2018)

    Google Scholar 

  39. Wu, H., Wang, Z., Zhou, K.: Simultaneous localization and appearance estimation with a consumer RGB-D camera. IEEE Trans. Visual Comput. Graph. 22(8), 2012–2023 (2015)

    Article  Google Scholar 

  40. Xia, R., Dong, Y., Peers, P., Tong, X.: Recovering shape and spatially-varying surface reflectance under unknown illumination. ACM Trans. Graph. (TOG) 35(6), 1–12 (2016)

    Article  Google Scholar 

  41. Xu, Z., Nielsen, J.B., Yu, J., Jensen, H.W., Ramamoorthi, R.: Minimal BRDF sampling for two-shot near-field reflectance acquisition. ACM Trans. Graph. (TOG) 35(6), 1–12 (2016)

    Google Scholar 

  42. Zhang, L., Rao, A., Agrawala, M.: Adding conditional control to text-to-image diffusion models. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3836–3847 (2023)

    Google Scholar 

  43. Zhao, Y., Wang, B., Xu, Y., Zeng, Z., Wang, L., Holzschuch, N.: Joint SVBRDF recovery and synthesis from a single image using an unsupervised generative adversarial network. In: EGSR (DL), pp. 53–66 (2020)

    Google Scholar 

  44. Zhou, X., Kalantari, N.K.: Adversarial single-image SVBRDF estimation with hybrid training. In: Computer Graphics Forum, vol. 40, pp. 315–325. Wiley Online Library (2021)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. School of Artificial Intelligence, Beijing Normal University, 100875, Beijing, China

    Liping Wu, Bin Cheng, Wentao Chao & Fuqing Duan

  2. School of Data Science and Software Engineering, Qingdao University, 266071, Qingdao, China

    Juli Zhao

Authors
  1. Liping Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Bin Cheng
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Wentao Chao
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Juli Zhao
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Fuqing Duan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Liping Wu .

Editor information

Editors and Affiliations

  1. Victoria University of Wellington, Wellington, New Zealand

    Fang-Lue Zhang

  2. Ben-Gurion University, Be'er Sheva, Israel

    Andrei Sharf

Rights and permissions

Reprints and permissions

Copyright information

© 2024 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Wu, L., Cheng, B., Chao, W., Zhao, J., Duan, F. (2024). MatTrans: Material Reflectance Property Estimation of Complex Objects with Transformer. In: Zhang, FL., Sharf, A. (eds) Computational Visual Media. CVM 2024. Lecture Notes in Computer Science, vol 14592. Springer, Singapore. https://doi.org/10.1007/978-981-97-2095-8_11

Download citation

  • DOI: https://doi.org/10.1007/978-981-97-2095-8_11

  • Published:

  • Publisher Name: Springer, Singapore

  • Print ISBN: 978-981-97-2094-1

  • Online ISBN: 978-981-97-2095-8

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation