[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to main content
Springer Nature Link
Log in

Collaborative Control for Geometry-Conditioned PBR Image Generation

  • Conference paper
  • First Online:
Computer Vision – ECCV 2024 (ECCV 2024)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 15071))

Included in the following conference series:

Abstract

Graphics pipelines require physically-based rendering (PBR) materials, yet current 3D content generation approaches are built on RGB models. We propose to model the PBR image distribution directly, avoiding photometric inaccuracies in RGB generation and the inherent ambiguity in extracting PBR from RGB. As existing paradigms for cross-modal fine-tuning are not suited for PBR generation due to both a lack of data and the high dimensionality of the output modalities, we propose to train a new PBR model that is tightly linked to a frozen RGB model using a novel cross-network communication paradigm. As the base RGB model is fully frozen, the proposed method retains its general performance and remains compatible with e.g. IPAdapters for that base model.

S. Vainer and S. Donné—Equal Contributions.

M. Boss—Stability AI, work done while at Unity Technologies.

Core Technical Contributions.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
£29.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
GBP 19.95
Price includes VAT (United Kingdom)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
GBP 49.99
Price includes VAT (United Kingdom)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
GBP 64.99
Price includes VAT (United Kingdom)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    Our intuition as to why a fixed environment map is beneficial is that it makes the RGB model’s internal states more consistent to interpret, and makes the control problem of projecting to \(\textrm{Im}(f)\) simpler. Early in training, generated sample quality can be boosted significantly by applying the foreground mask to the RGB estimate for the first few timesteps; a rough projection to bring the estimate much closer to \(\textrm{Im}(f)\). After longer training, this is no longer necessary, as the PBR branch is capable enough to restrict the RGB branch to \(\textrm{Im}(f)\).

  2. 2.

    https://polyhaven.com/a/studio_small_08.

References

  1. Achiam, J., et al.: GPT-4 technical report. arXiv preprint arXiv:2303.08774 (2023)

  2. Ba, J.L., Kiros, J.R., Hinton, G.E.: Layer normalization. stat 1050, 21 (2016)

    Google Scholar 

  3. Burley, B.: Physically based shading at disney. ACM Trans. Graph. (SIGGRAPH) (2012)

    Google Scholar 

  4. Cao, T., Kreis, K., Fidler, S., Sharp, N., Yin, K.: TexFusion: synthesizing 3D textures with text-guided image diffusion models. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4169–4181 (2023)

    Google Scholar 

  5. Chambon, T., Heitz, E., Belcour, L.: Passing multi-channel material textures to a 3-channel loss. In: ACM SIGGRAPH 2021 Talks, pp. 1–2 (2021)

    Google Scholar 

  6. Chen, D.Z., Siddiqui, Y., Lee, H.Y., Tulyakov, S., Nießner, M.: Text2Tex: text-driven texture synthesis via diffusion models. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 18558–18568 (2023)

    Google Scholar 

  7. Chen, R., Chen, Y., Jiao, N., Jia, K.: Fantasia3D: disentangling geometry and appearance for high-quality text-to-3D content creation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 22246–22256 (2023)

    Google Scholar 

  8. Cook, R.L., Torrance, K.E.: A reflectance model for computer graphics. ACM Trans. Graph. (ToG) 1(1), 7–24 (1982)

    Article  Google Scholar 

  9. Deitke, M., et al.: Objaverse: a universe of annotated 3D objects. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13142–13153 (2023)

    Google Scholar 

  10. Zavadski, D., Feiden, J.F., Rother, C.: ControlNet-XS: designing an efficient and effective architecture for controlling text-to-image diffusion models (2023)

    Google Scholar 

  11. Du, X., Kolkin, N., Shakhnarovich, G., Bhattad, A.: Intrinsic LoRA: a generalist approach for discovering knowledge in generative models. In: Synthetic Data for Computer Vision Workshop, CVPR 2024 (2024)

    Google Scholar 

  12. Duan, Y., Guo, X., Zhu, Z.: DiffusionDepth: diffusion denoising approach for monocular depth estimation. arXiv preprint arXiv:2303.05021 (2023)

  13. Foong, T.Y., Kotyan, S., Mao, P.Y., Vargas, D.V.: The challenges of image generation models in generating multi-component images. arXiv preprint arXiv:2311.13620 (2023)

  14. Goodfellow, I., et al.: Generative adversarial networks. Commun. ACM 63(11), 139–144 (2020)

    Article  MathSciNet  Google Scholar 

  15. Guo, P., et al.: StableDreamer: taming noisy score distillation sampling for text-to-3D. arXiv preprint arXiv:2312.02189 (2023)

  16. Hessel, J., Holtzman, A., Forbes, M., Bras, R.L., Choi, Y.: CLIPScore: a reference-free evaluation metric for image captioning. arXiv preprint arXiv:2104.08718 (2021)

  17. Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., Hochreiter, S.: GANs trained by a two time-scale update rule converge to a local Nash equilibrium. In: Advances in Neural Information Processing Systems, vol. 30 (2017)

    Google Scholar 

  18. Ho, J., Jain, A., Abbeel, P.: Denoising diffusion probabilistic models. In: Advances in Neural Information Processing Systems, vol. 33, pp. 6840–6851 (2020)

    Google Scholar 

  19. Hu, E.J., et al.: LoRA: low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685 (2021)

  20. Hu, L.: Animate anyone: consistent and controllable image-to-video synthesis for character animation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8153–8163 (2024)

    Google Scholar 

  21. Huang, T., et al.: DreamControl: control-based text-to-3D generation with 3D self-prior. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5364–5373 (2024)

    Google Scholar 

  22. Huang, Y., Wang, J., Shi, Y., Qi, X., Zha, Z.J., Zhang, L.: DreamTime: an improved optimization strategy for text-to-3D content creation. arXiv preprint arXiv:2306.12422 (2023)

  23. Jayasumana, S., Ramalingam, S., Veit, A., Glasner, D., Chakrabarti, A., Kumar, S.: Rethinking FID: towards a better evaluation metric for image generation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9307–9315 (2024)

    Google Scholar 

  24. Jin, Z., Shen, X., Li, B., Xue, X.: Training-free diffusion model adaptation for variable-sized text-to-image synthesis. In: Advances in Neural Information Processing Systems, vol. 36 (2024)

    Google Scholar 

  25. Karras, T., Aila, T., Laine, S., Lehtinen, J.: Progressive growing of GANs for improved quality, stability, and variation. arXiv preprint arXiv:1710.10196 (2017)

  26. Karras, T., et al.: Alias-free generative adversarial networks. In: Advances in Neural Information Processing Systems, vol. 34, pp. 852–863 (2021)

    Google Scholar 

  27. Karras, T., Laine, S., Aila, T.: A style-based generator architecture for generative adversarial networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4401–4410 (2019)

    Google Scholar 

  28. Karras, T., Laine, S., Aittala, M., Hellsten, J., Lehtinen, J., Aila, T.: Analyzing and improving the image quality of StyleGAN. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8110–8119 (2020)

    Google Scholar 

  29. Ke, B., Obukhov, A., Huang, S., Metzger, N., Daudt, R.C., Schindler, K.: Repurposing diffusion-based image generators for monocular depth estimation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9492–9502 (2024)

    Google Scholar 

  30. Kingma, D.P., Welling, M.: Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 (2013)

  31. Knodt, J., Gao, X.: Consistent mesh diffusion. arXiv preprint arXiv:2312.00971 (2023)

  32. Le, C., Hetang, C., Cao, A., He, Y.: EucliDreamer: fast and high-quality texturing for 3D models with stable diffusion depth. arXiv preprint arXiv:2311.15573 (2023)

  33. Lee, H.Y., Tseng, H.Y., Yang, M.H.: Exploiting diffusion prior for generalizable dense prediction. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7861–7871 (2024)

    Google Scholar 

  34. Li, X., et al.: Advances in 3D generation: a survey. arXiv preprint arXiv:2401.17807 (2024)

  35. Liang, Y., Yang, X., Lin, J., Li, H., Xu, X., Chen, Y.: LucidDreamer: towards high-fidelity text-to-3D generation via interval score matching. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6517–6526 (2024)

    Google Scholar 

  36. Lin, S., Liu, B., Li, J., Yang, X.: Common diffusion noise schedules and sample steps are flawed. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 5404–5411 (2024)

    Google Scholar 

  37. Liu, F., Wu, D., Wei, Y., Rao, Y., Duan, Y.: Sherpa3D: boosting high-fidelity text-to-3D generation via coarse 3D prior. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20763–20774 (2024)

    Google Scholar 

  38. Liu, R., Wu, R., Van Hoorick, B., Tokmakov, P., Zakharov, S., Vondrick, C.: Zero-1-to-3: zero-shot one image to 3D object. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9298–9309 (2023)

    Google Scholar 

  39. Liu, Y.T., Guo, Y.C., Luo, G., Sun, H., Yin, W., Zhang, S.H.: PI3D: efficient text-to-3D generation with pseudo-image diffusion. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19915–19924 (2024)

    Google Scholar 

  40. Liu, Z., et al.: UniDream: unifying diffusion priors for relightable text-to-3D generation. arXiv preprint arXiv:2312.08754 (2023)

  41. Long, X., et al.: Wonder3D: single image to 3D using cross-domain diffusion. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9970–9980 (2024)

    Google Scholar 

  42. Ma, B., Deng, H., Zhou, J., Liu, Y.S., Huang, T., Wang, X.: GeoDream: disentangling 2D and geometric priors for high-fidelity and consistent 3D generation. arXiv preprint arXiv:2311.17971 (2023)

  43. Ma, Y., et al.: X-Dreamer: creating high-quality 3D content by bridging the domain gap between text-to-2D and text-to-3D generation. arXiv preprint arXiv:2312.00085 (2023)

  44. Mou, C., et al.: T2I-Adapter: learning adapters to dig out more controllable ability for text-to-image diffusion models. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 38, pp. 4296–4304 (2024)

    Google Scholar 

  45. Van den Oord, A., Kalchbrenner, N., Espeholt, L., Vinyals, O., Graves, A., et al.: Conditional image generation with PixelCNN decoders. In: Advances in Neural Information Processing Systems, vol. 29 (2016)

    Google Scholar 

  46. Poole, B., Jain, A., Barron, J.T., Mildenhall, B.: DreamFusion: text-to-3D using 2D diffusion. In: The Eleventh International Conference on Learning Representations (2023)

    Google Scholar 

  47. Raj, A., et al.: DreamBooth3D: subject-driven text-to-3D generation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2349–2359 (2023)

    Google Scholar 

  48. Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-resolution image synthesis with latent diffusion models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10684–10695 (2022)

    Google Scholar 

  49. Salimans, T., Goodfellow, I., Zaremba, W., Cheung, V., Radford, A., Chen, X.: Improved techniques for training GANs. In: Advances in Neural Information Processing Systems, vol. 29 (2016)

    Google Scholar 

  50. Sarkar, A., Mai, H., Mahapatra, A., Lazebnik, S., Forsyth, D.A., Bhattad, A.: Shadows don’t lie and lines can’t bend! Generative models don’t know projective geometry... for now. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 28140–28149 (2024)

    Google Scholar 

  51. Sartor, S., Peers, P.: MatFusion: a generative diffusion model for SVBRDF capture. In: SIGGRAPH Asia 2023 Conference Papers, SA 2023. ACM (2023). https://doi.org/10.1145/3610548.3618194

  52. Schuhmann, C., et al.: LAION-5B: an open large-scale dataset for training next generation image-text models. In: Advances in Neural Information Processing Systems, vol. 35, pp. 25278–25294 (2022)

    Google Scholar 

  53. Sharma, P., et al.: Alchemist: parametric control of material properties with diffusion models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 24130–24141 (2024)

    Google Scholar 

  54. Shi, R., et al.: Zero123++: a single image to consistent multi-view diffusion base model. arXiv preprint arXiv:2310.15110 (2023)

  55. Shi, Y., Wang, P., Ye, J., Mai, L., Li, K., Yang, X.: MVDream: multi-view diffusion for 3D generation. In: The Twelfth International Conference on Learning Representations (2024)

    Google Scholar 

  56. Sohl-Dickstein, J., Weiss, E., Maheswaranathan, N., Ganguli, S.: Deep unsupervised learning using nonequilibrium thermodynamics. In: International Conference on Machine Learning, pp. 2256–2265. PMLR (2015)

    Google Scholar 

  57. Song, J., Meng, C., Ermon, S.: Denoising diffusion implicit models. In: International Conference on Learning Representations (2020)

    Google Scholar 

  58. Stan, G.B.M., et al.: LDM3D: latent diffusion model for 3D. arXiv preprint arXiv:2305.10853 (2023)

  59. Subias, J.D., Lagunas, M.: In-the-wild material appearance editing using perceptual attributes. In: Computer Graphics Forum, vol. 42, pp. 333–345. Wiley Online Library (2023)

    Google Scholar 

  60. Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., Wojna, Z.: Rethinking the inception architecture for computer vision. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2818–2826 (2016)

    Google Scholar 

  61. Tang, B., Wang, J., Wu, Z., Zhang, L.: Stable score distillation for high-quality 3D generation. arXiv preprint arXiv:2312.09305 (2023)

  62. Tang, J., Ren, J., Zhou, H., Liu, Z., Zeng, G.: DreamGaussian: generative gaussian splatting for efficient 3D content creation. In: The Twelfth International Conference on Learning Representations (2024)

    Google Scholar 

  63. Huggingface zerodiffusion model weights v0.9. https://huggingface.co/drhead. https://huggingface.co/drhead/ZeroDiffusion. Accessed 08 Feb 2024

  64. Van Den Oord, A., Kalchbrenner, N., Kavukcuoglu, K.: Pixel recurrent neural networks. In: International Conference on Machine Learning, pp. 1747–1756. PMLR (2016)

    Google Scholar 

  65. Vecchio, G., et al.: ControlMat: a controlled generative approach to material capture. arXiv preprint arXiv:2309.01700 (2023)

  66. Wang, P., et al.: SteinDreamer: variance reduction for text-to-3D score distillation via stein identity. arXiv preprint arXiv:2401.00604 (2023)

  67. Wang, Z., Li, M., Chen, C.: LucidDreaming: controllable object-centric 3D generation. arXiv preprint arXiv:2312.00588 (2023)

  68. Wang, Z., et al.: ProlificDreamer: high-fidelity and diverse text-to-3D generation with variational score distillation. In: Advances in Neural Information Processing Systems, vol. 36 (2024)

    Google Scholar 

  69. Wu, H., et al.: Q-Align: teaching LMMs for visual scoring via discrete text-defined levels. In: Forty-First International Conference on Machine Learning (2024)

    Google Scholar 

  70. Wu, T., et al.: HyperDreamer: hyper-realistic 3D content generation and editing from a single image. In: SIGGRAPH Asia 2023 Conference Papers, pp. 1–10 (2023)

    Google Scholar 

  71. Wu, Z., Zhou, P., Yi, X., Yuan, X., Zhang, H.: Consistent3D: towards consistent high-fidelity text-to-3D generation with deterministic sampling prior. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9892–9902 (2024)

    Google Scholar 

  72. Xu, X., Lyu, Z., Pan, X., Dai, B.: MATLABER: material-aware text-to-3D via latent BRDF auto-encoder. arXiv preprint arXiv:2308.09278 (2023)

  73. Ye, H., Zhang, J., Liu, S., Han, X., Yang, W.: IP-Adapter: text compatible image prompt adapter for text-to-image diffusion models. arXiv preprint arXiv:2308.06721 (2023)

  74. Yeh, Y.Y., et al.: TextureDreamer: image-guided texture synthesis through geometry-aware diffusion. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4304–4314 (2024)

    Google Scholar 

  75. Youwang, K., Oh, T.H., Pons-Moll, G.: Paint-it: text-to-texture synthesis via deep convolutional texture map optimization and physically-based rendering. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4347–4356 (2024)

    Google Scholar 

  76. Yu, K., Liu, J., Feng, M., Cui, M., Xie, X.: Boosting3D: high-fidelity image-to-3D by boosting 2D diffusion prior to 3D prior with progressive learning. arXiv preprint arXiv:2311.13617 (2023)

  77. Zeng, X., et al.: Paint3D: paint anything 3D with lighting-less texture diffusion models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4252–4262 (2024)

    Google Scholar 

  78. Zhang, J., et al.: Repaint123: fast and high-quality one image to 3D generation with progressive controllable 2D repainting. arXiv preprint arXiv:2312.13271 (2023)

  79. Zhang, L., Rao, A., Agrawala, M.: Adding conditional control to text-to-image diffusion models. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3836–3847 (2023)

    Google Scholar 

  80. Zhou, L., Shih, A., Meng, C., Ermon, S.: DreamPropeller: supercharge text-to-3D generation with parallel sampling. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4610–4619 (2024)

    Google Scholar 

  81. Zhuang, J., Wang, C., Lin, L., Liu, L., Li, G.: DreamEditor: text-driven 3D scene editing with neural fields. In: SIGGRAPH Asia 2023 Conference Papers, pp. 1–10 (2023)

    Google Scholar 

Download references

Acknowledgements

This work was supported fully by Unity Technologies, without external funding. We would like to thank the reviewers for their valuable feedback and suggestions.

Author information

Authors and Affiliations

  1. Unity Technologies, San Francisco, USA

    Shimon Vainer, Mathias Parger, Konstantin Kutsy, Dante De Nigris, Ciara Rowles, Nicolas Perony & Simon Donné

  2. San Francisco, USA

    Mark Boss

Authors
  1. Shimon Vainer
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mark Boss
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Mathias Parger
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Konstantin Kutsy
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Dante De Nigris
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Ciara Rowles
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Nicolas Perony
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Simon Donné
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Shimon Vainer .

Editor information

Editors and Affiliations

  1. University of Birmingham, Birmingham, UK

    Aleš Leonardis

  2. University of Trento, Trento, Italy

    Elisa Ricci

  3. Technical University of Darmstadt, Darmstadt, Germany

    Stefan Roth

  4. Princeton University, Princeton, NJ, USA

    Olga Russakovsky

  5. Czech Technical University in Prague, Prague, Czech Republic

    Torsten Sattler

  6. École des Ponts ParisTech, Marne-la-Vallée, France

    Gül Varol

1 Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 12385 KB)

Supplementary material 2 (pdf 318 KB)

Rights and permissions

Reprints and permissions

Copyright information

© 2025 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Vainer, S. et al. (2025). Collaborative Control for Geometry-Conditioned PBR Image Generation. In: Leonardis, A., Ricci, E., Roth, S., Russakovsky, O., Sattler, T., Varol, G. (eds) Computer Vision – ECCV 2024. ECCV 2024. Lecture Notes in Computer Science, vol 15071. Springer, Cham. https://doi.org/10.1007/978-3-031-72624-8_8

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-72624-8_8

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-72623-1

  • Online ISBN: 978-3-031-72624-8

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation