More Web Proxy on the site http://driver.im/

Article

3D Gaussian Parametric Head Model

Authors:

Yebin LiuAuthors Info & Claims

Computer Vision – ECCV 2024: 18th European Conference, Milan, Italy, September 29 – October 4, 2024, Proceedings, Part XXXV

Pages 129 - 147

https://doi.org/10.1007/978-3-031-72761-0_8

Published: 30 September 2024 Publication History

Abstract

Creating high-fidelity 3D human head avatars is crucial for applications in VR/AR, telepresence, digital human interfaces, and film production. Recent advances have leveraged morphable face models to generate animated head avatars from easily accessible data, representing varying identities and expressions within a low-dimensional parametric space. However, existing methods often struggle with modeling complex appearance details, e.g., hairstyles and accessories, and suffer from low rendering quality and efficiency. This paper introduces a novel approach, 3D Gaussian Parametric Head Model, which employs 3D Gaussians to accurately represent the complexities of the human head, allowing precise control over both identity and expression. Additionally, it enables seamless face portrait interpolation and the reconstruction of detailed head avatars from a single image. Unlike previous methods, the Gaussian model can handle intricate details, enabling realistic representations of varying appearances and complex expressions. Furthermore, this paper presents a well-designed training framework to ensure smooth convergence, providing a guarantee for learning the rich content. Our method achieves high-quality, photo-realistic rendering with real-time efficiency, making it a valuable contribution to the field of parametric head models.

References

[1]

An, S., Xu, H., Shi, Y., Song, G., Ogras, U.Y., Luo, L.: Panohead: geometry-aware 3d full-head synthesis in 360deg. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 20950–20959 (2023)

[2]

Blanz, V., Vetter, T.: A morphable model for the synthesis of 3d faces. In: 26th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH 1999), pp. 187–194. ACM Press (1999)

[3]

Bühler, M.C., et al.: Preface: a data-driven volumetric prior for few-shot ultra high-resolution face synthesis. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3402–3413 (2023)

[4]

Bulat, A., Tzimiropoulos, G.: How far are we from solving the 2d & 3d face alignment problem? (and a dataset of 230,000 3d facial landmarks). In: International Conference on Computer Vision (2017)

[5]

Cao, C., et al.: Authentic volumetric avatars from a phone scan. ACM Trans. Graph. 41(4) (2022)

[6]

Cao C, Weng Y, Zhou S, Tong Y, and Zhou K Facewarehouse: a 3d facial expression database for visual computing IEEE Trans. Visualizat. Comput. Graph. 2014 20 413-425

Digital Library

[7]

Chan, E., Monteiro, M., Kellnhofer, P., Wu, J., Wetzstein, G.: pi-gan: periodic implicit generative adversarial networks for 3d-aware image synthesis. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5795–5805 (2020)

[8]

Chan, E.R., et al.: Efficient geometry-aware 3D generative adversarial networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 16102–16112 (2022)

[9]

Chen, X., Deng, Y., Wang, B.: Mimic3d: thriving 3d-aware gans via 3d-to-2d imitation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) (2023)

[10]

Chen, Y., et al.: Monogaussianavatar: monocular gaussian point-based head avatar. In: ACM SIGGRAPH 2023 Conference Proceedings (2024)

[11]

Deng, Y., Yang, J., Xiang, J., Tong, X.: Gram: generative radiance manifolds for 3d-aware image generation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10663–10673 (2021)

[12]

Gafni, G., Thies, J., Zollhofer, M., Niessner, M.: Dynamic neural radiance fields for monocular 4d facial avatar reconstruction. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8645–8654 (2021)

[13]

Gao, X., Zhong, C., Xiang, J., Hong, Y., Guo, Y., Zhang, J.: Reconstructing personalized semantic facial nerf models from monocular video. ACM Trans. Graph. 41(6) (2022)

[14]

Gerig, T., et al.: Morphable face models - an open framework. In: 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018), pp. 75–82 (2017)

[15]

Giebenhain, S., Kirschstein, T., Georgopoulos, M., Rünz, M., Agapito, L., Nießner, M.: Learning neural parametric head models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2023)

[16]

Giebenhain, S., Kirschstein, T., Georgopoulos, M., Rünz, M., Agapito, L., Nießner, M.: Mononphm: dynamic head reconstruction from monocular videos. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2024)

[17]

Grassal, P.W., Prinzler, M., Leistner, T., Rother, C., Nießner, M., Thies, J.: Neural head avatars from monocular rgb videos. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 18632–18643 (2022)

[18]

Gu, J., Liu, L., Wang, P., Theobalt, C.: Stylenerf: a style-based 3d aware generator for high-resolution image synthesis. In: International Conference on Learning Representations (2022)

[19]

Hong, Y., Peng, B., Xiao, H., Liu, L., Zhang, J.: Headnerf: a real-time nerf-based parametric head model. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 20374–20384 (2022)

[20]

Hu, L., et al.: Gaussianavatar: towards realistic human avatar modeling from a single video via animatable 3d gaussians. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2024)

[21]

Kerbl, B., Kopanas, G., Leimkühler, T., Drettakis, G.: 3d gaussian splatting for real-time radiance field rendering. ACM Trans. Graph. 42(4) (2023)

[22]

Khakhulin, T., Sklyarova, V., Lempitsky, V., Zakharov, E.: Realistic one-shot mesh-based head avatars. In: Proceedings of the European Conference on Computer Vision (ECCV) (2022)

[23]

Kirschstein, T., Giebenhain, S., Nießner, M.: Diffusionavatars: deferred diffusion for high-fidelity 3d head avatars. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2024)

[24]

Kirschstein, T., Qian, S., Giebenhain, S., Walter, T., Nießner, M.: Nersemble: multi-view radiance field reconstruction of human heads. ACM Trans. Graph. 42(4) (2023)

[25]

Li, T., Bolkart, T., Black, M.J., Li, H., Romero, J.: Learning a model of facial shape and expression from 4d scans. ACM Trans. Graph. 36(6) (2017)

[26]

Li, X., De Mello, S., Liu, S., Nagano, K., Iqbal, U., Kautz, J.: Generalizable one-shot neural head avatar. In: NeurIPS (2023)

[27]

Li, Z., Zheng, Z., Wang, L., Liu, Y.: Animatable gaussians: learning pose-dependent gaussian maps for high-fidelity human avatar modeling. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2024)

[28]

Lin, C.Z., et al.: Single-shot implicit morphable faces with consistent texture parameterization. In: ACM SIGGRAPH 2023 Conference Proceedings (2023)

[29]

Lin, S., Ryabtsev, A., Sengupta, S., Curless, B., Seitz, S., Kemelmacher-Shlizerman, I.: Real-time high-resolution background matting. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2021)

[30]

Lombardi, S., Simon, T., Schwartz, G., Zollhoefer, M., Sheikh, Y., Saragih, J.: Mixture of volumetric primitives for efficient neural rendering. ACM Trans. Graph. 40(4) (2021)

[31]

Loper, M., Mahmood, N., Romero, J., Pons-Moll, G., Black, M.J.: SMPL: a skinned multi-person linear model. ACM Trans. Graph. 34(6), 248:1–248:16 (2015)

[32]

Luiten, J., Kopanas, G., Leibe, B., Ramanan, D.: Dynamic 3d gaussians: tracking by persistent dynamic view synthesis. In: 3DV (2024)

[33]

Ma, S., Weng, Y., Shao, T., Zhou, K.: 3d gaussian blendshapes for head avatar animation. In: ACM SIGGRAPH 2023 Conference Proceedings (2024)

[34]

Mildenhall B, Srinivasan PP, Tancik M, Barron JT, Ramamoorthi R, and Ng R Vedaldi A, Bischof H, Brox T, and Frahm J-M NeRF: representing scenes as neural radiance fields for view synthesis Computer Vision – ECCV 2020 2020 Cham Springer 405-421

Digital Library

[35]

Or-El, R., Luo, X., Shan, M., Shechtman, E., Park, J.J., Kemelmacher-Shlizerman, I.: Stylesdf: high-resolution 3d-consistent image and geometry generation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 13493–13503 (2021)

[36]

Pavlakos, G., et al.: Expressive body capture: 3D hands, face, and body from a single image. In: Proceedings IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10975–10985 (2019)

[37]

Qian, S., Kirschstein, T., Schoneveld, L., Davoli, D., Giebenhain, S., Nießner, M.: Gaussianavatars: photorealistic head avatars with rigged 3d gaussians. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2024)

[38]

Qin, M., Liu, Y., Xu, Y., Zhao, X., Liu, Y., Wang, H.: High-fidelity 3d head avatars reconstruction through spatially-varying expression conditioned neural radiance field. In: AAAI Conference on Artificial Intelligence (2023)

[39]

Saito, S., Schwartz, G., Simon, T., Li, J., Nam, G.: Relightable gaussian codec avatars. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2024)

[40]

Shao, Z., et al.: SplattingAvatar: realistic real-time human avatars with mesh-embedded gaussian splatting. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2024)

[41]

Shen, T., Gao, J., Yin, K., Liu, M.Y., Fidler, S.: Deep marching tetrahedra: a hybrid representation for high-resolution 3d shape synthesis. Adv. Neural Inf. Process. Syst. (NeurIPS) (2021)

[42]

Sun J, Wang X, Shi Y, Wang L, Wang J, and Liu Y Ide-3d: interactive disentangled editing for high-resolution 3d-aware portrait synthesis ACM Trans. Graph. (TOG) 2022 41 6 1-10

Digital Library

[43]

Sun, J., et al.: Next3d: generative neural texture rasterization for 3d-aware head avatars. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2023)

[44]

Wang, D., Chandran, P., Zoss, G., Bradley, D., Gotardo, P.: Morf: morphable radiance fields for multiview neural head modeling. In: ACM SIGGRAPH 2022 Conference Proceedings. SIGGRAPH ’22, Association for Computing Machinery, New York (2022)

[45]

Wang, J., Xie, J.C., Li, X., Xu, F., Pun, C.M., Gao, H.: Gaussianhead: high-fidelity head avatars with learnable gaussian derivation (2024)

[46]

Wang, K., et al.: Mead: a large-scale audio-visual dataset for emotional talking-face generation. In: Proceedings of the European Conference on Computer Vision (ECCV) (2020)

[47]

Wang, L., Chen, Z., Yu, T., Ma, C., Li, L., Liu, Y.: Faceverse: a fine-grained and detail-controllable 3d face morphable model from a hybrid dataset. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2022)

[48]

Wu, G., et al.: 4d gaussian splatting for real-time dynamic scene rendering (2024)

[49]

Wu, S., et al.: Ganhead: towards generative animatable neural head avatars. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 437–447 (2023)

[50]

Wu, Y., Deng, Y., Yang, J., Wei, F., Qifeng, C., Tong, X.: Anifacegan: animatable 3d-aware face image generation for video avatars. Adv. Neural Inf. Process. Syst. (2022)

[51]

Wu, Y., et al.: Aniportraitgan: animatable 3d portrait generation from 2d image collections. In: SIGGRAPH Asia 2023 Conference Proceedings (2023)

[52]

Xiang, J., Yang, J., Deng, Y., Tong, X.: Gram-hd: 3d-consistent image generation at high resolution with generative radiance manifolds. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 2195–2205 (2022)

[53]

Xiang, J., Gao, X., Guo, Y., Zhang, J.: Flashavatar: high-fidelity head avatar with efficient gaussian embedding. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2024)

[54]

Xu, Y., et al.: Gaussian head avatar: ultra high-fidelity head avatar via dynamic gaussians. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2024)

[55]

Xu, Y., Wang, L., Zhao, X., Zhang, H., Liu, Y.: Avatarmav: fast 3d head avatar reconstruction using motion-aware neural voxels. In: ACM SIGGRAPH 2023 Conference Proceedings (2023)

[56]

Xu, Y., et al.: Latentavatar: learning latent expression code for expressive neural head avatar. In: ACM SIGGRAPH 2023 Conference Proceedings (2023)

[57]

Yang, Z., Yang, H., Pan, Z., Zhu, X., Zhang, L.: Real-time photorealistic dynamic scene representation and rendering with 4d gaussian splatting (2023)

[58]

Yang, Z., Gao, X., Zhou, W., Jiao, S., Zhang, Y., Jin, X.: Deformable 3d gaussians for high-fidelity monocular dynamic scene reconstruction (2023)

[59]

Yenamandra, T., et al.: i3dmm: deep implicit 3d morphable model of human heads. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2021)

[60]

Zhang, R., Isola, P., Efros, A.A., Shechtman, E., Wang, O.: The unreasonable effectiveness of deep features as a perceptual metric. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 586–595 (2018)

[61]

Zhao, X., Wang, L., Sun, J., Zhang, H., Suo, J., Liu, Y.: Havatar: high-fidelity head avatar via facial model conditioned neural radiance field. ACM Trans. Graph. (2023)

[62]

Zheng, Y., Abrevaya, V.F., Bühler, M.C., Chen, X., Black, M.J., Hilliges, O.: I m avatar: implicit morphable head avatars from videos. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 13535–13545 (2022)

[63]

Zheng, Y., Yifan, W., Wetzstein, G., Black, M.J., Hilliges, O.: Pointavatar: deformable point-based head avatars from videos. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2023)

[64]

Zhuang, Y., Zhu, H., Sun, X., Cao, X.: Mofanerf: morphable facial neural radiance field. In: Proceedings of the European Conference on Computer Vision (ECCV) (2022)

[65]

Zielonka, W., Bolkart, T., Thies, J.: Instant volumetric head avatars (2023)

Index Terms

3D Gaussian Parametric Head Model
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. Interaction paradigms
2. Social and professional topics
  1. Professional topics
    1. History of computing

Index terms have been assigned to the content through auto-classification.

Recommendations

3D Gaussian Blendshapes for Head Avatar Animation
SIGGRAPH '24: ACM SIGGRAPH 2024 Conference Papers

We introduce 3D Gaussian blendshapes for modeling photorealistic head avatars. Taking a monocular video as input, we learn a base head model of neutral expression, along with a group of expression blendshapes, each of which corresponds to a basis ...
Gaussian mixture 3D morphable face model

A Gaussian Mixture 3DMM (GM-3DMM) which models the global population as a mixture of Gaussian subpopulations.A ESO-based model selection strategy for GM-3DMM fitting.A GM-3DMM-based face recognition framework by fusing multiple experts, which has ...
HAvatar: High-fidelity Head Avatar via Facial Model Conditioned Neural Radiance Field
The problem of modeling an animatable 3D human head avatar under lightweight setups is of significant importance but has not been well solved. Existing 3D representations either perform well in the realism of portrait images synthesis or the accuracy of ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Guide Proceedings

Computer Vision – ECCV 2024: 18th European Conference, Milan, Italy, September 29 – October 4, 2024, Proceedings, Part XXXV

Sep 2024

561 pages

ISBN:978-3-031-72760-3

DOI:10.1007/978-3-031-72761-0

Editors:
Aleš Leonardis
University of Birmingham, Birmingham, UK
,
Elisa Ricci
https://ror.org/05trd4x28University of Trento, Trento, Italy
,
Stefan Roth
Technical University of Darmstadt, Darmstadt, Hessen, Germany
,
Olga Russakovsky
Princeton University, Palo Alto, CA, USA
,
Torsten Sattler
Czech Technical University in Prague, Prague, Czech Republic
,
Gül Varol
École des Ponts ParisTech, Marne-la-Vallée, France

© The Author(s), under exclusive license to Springer Nature Switzerland AG 2025.

Publisher

Springer-Verlag

Berlin, Heidelberg

Publication History

Published: 30 September 2024

Author Tags

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

0
Total Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 11 Dec 2024

Other Metrics

View Author Metrics

Citations

View Options

View options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents