More Web Proxy on the site http://driver.im/

research-article

Open access

SoftDECA: Computationally Efficient Physics-Based Facial Animations

Authors:

Nicolas Wagner,

Ulrich SchwaneckeAuthors Info & Claims

MIG '23: Proceedings of the 16th ACM SIGGRAPH Conference on Motion, Interaction and Games

Article No.: 11, Pages 1 - 11

https://doi.org/10.1145/3623264.3624439

Published: 15 November 2023 Publication History

All formats PDF

Abstract

Facial animation on computationally weak systems is still mostly dependent on linear blendshape models. However, these models suffer from typical artifacts such as loss of volume, self-collisions, or erroneous soft tissue elasticity. In addition, while extensive effort is required to personalize blendshapes, there are limited options to simulate or manipulate physical and anatomical properties once a model has been crafted. Finally, second-order dynamics can only be represented to a limited extent.

For decades, physics-based facial animation has been investigated as an alternative to linear blendshapes but is still cumbersome to deploy and results in high computational cost at runtime. We propose SoftDECA, an approach that provides the benefits of physics-based simulation while being as effortless and fast to use as linear blendshapes. SoftDECA is a novel hypernetwork that efficiently approximates a FEM-based facial simulation while generalizing over the comprehensive DECA model of human identities, facial expressions, and a wide range of material properties that can be locally adjusted without re-training. Along with SoftDECA, we introduce a pipeline for creating the needed high-resolution training data. Part of this pipeline is a novel layered head model that densely positions the biomechanical anatomy within a skin surface while avoiding self-intersections.

Supplementary Material

Supp. Text and Video (MIG_SUP-2.pdf)

Download
115.12 KB

MP4 File (MIG_2023_video_6_1.mp4)

Update to video

Download
44.00 MB

MP4 File (MIG_2023_video_6.mp4)

Supp. Text and Video

Download
62.01 MB

References

[1]

Jascha Achenbach, Robert Brylka, Thomas Gietzen, Katja zum Hebel, Elmar Schömer, Ralf Schulze, Mario Botsch, and Ulrich Schwanecke. 2018. A multilinear model for bidirectional craniofacial reconstruction. In Proceedings of the Eurographics Workshop on Visual Computing for Biology and Medicine. 67–76.

Digital Library

[2]

Dicko Ali-Hamadi, Tiantian Liu, Benjamin Gilles, Ladislav Kavan, François Faure, Olivier Palombi, and Marie-Paule Cani. 2013. Anatomy transfer. ACM transactions on graphics (TOG) 32, 6 (2013), 1–8.

[3]

ShahRukh Athar, Zexiang Xu, Kalyan Sunkavalli, Eli Shechtman, and Zhixin Shu. 2022. RigNeRF: Fully Controllable Neural 3D Portraits. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 20364–20373.

[4]

Michael Bao, Matthew Cong, Stéphane Grabli, and Ronald Fedkiw. 2019. High-quality face capture using anatomical muscles. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 10802–10811.

[5]

Vincent Barrielle, Nicolas Stoiber, and Cédric Cagniart. 2016. Blendforces: A dynamic framework for facial animation. In Computer Graphics Forum, Vol. 35. Wiley Online Library, 341–352.

[6]

Thabo Beeler and Derek Bradley. 2014. Rigid stabilization of facial expressions. ACM Transactions on Graphics (TOG) 33, 4 (2014), 1–9.

Digital Library

[7]

Bernd Bickel, Manuel Lang, Mario Botsch, Miguel A Otaduy, and Markus H Gross. 2008. Pose-Space Animation and Transfer of Facial Details. In Symposium on Computer Animation. 57–66.

[8]

Mario Botsch and Leif Kobbelt. 2005. Real-time shape editing using radial basis functions. In Computer graphics forum, Vol. 24. Blackwell Publishing, Inc Oxford, UK and Boston, USA, 611–621.

[9]

Mario Botsch, Robert Sumner, Mark Pauly, and Markus Gross. 2006. Deformation transfer for detail-preserving surface editing. In Vision, Modeling & Visualization. Citeseer, 357–364.

[10]

Sofien Bouaziz, Sebastian Martin, Tiantian Liu, Ladislav Kavan, and Mark Pauly. 2014. Projective dynamics: Fusing constraint projections for fast simulation. ACM transactions on graphics (TOG) 33, 4 (2014), 1–11.

Digital Library

[11]

Derek Bradley, Wolfgang Heidrich, Tiberiu Popa, and Alla Sheffer. 2010. High resolution passive facial performance capture. In ACM SIGGRAPH 2010 papers. 1–10.

Digital Library

[12]

Christopher Brandt, Elmar Eisemann, and Klaus Hildebrandt. 2018. Hyper-reduced projective dynamics. ACM Transactions on Graphics (TOG) 37, 4 (2018), 1–13.

Digital Library

[13]

Chen Cao, Tomas Simon, Jin Kyu Kim, Gabe Schwartz, Michael Zollhoefer, Shun-Suke Saito, Stephen Lombardi, Shih-En Wei, Danielle Belko, Shoou-I Yu, 2022. Authentic volumetric avatars from a phone scan. ACM Transactions on Graphics (TOG) 41, 4 (2022), 1–19.

Digital Library

[14]

Dan Casas and Miguel A Otaduy. 2018. Learning nonlinear soft-tissue dynamics for interactive avatars. Proceedings of the ACM on Computer Graphics and Interactive Techniques 1, 1 (2018), 1–15.

Digital Library

[15]

Byungkuk Choi, Haekwang Eom, Benjamin Mouscadet, Stephen Cullingford, Kurt Ma, Stefanie Gassel, Suzi Kim, Andrew Moffat, Millicent Maier, Marco Revelant, 2022. Animatomy: an Animator-centric, Anatomically Inspired System for 3D Facial Modeling, Animation and Transfer. In SIGGRAPH Asia 2022 Conference Papers. 1–9.

Digital Library

[16]

Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. 2014. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555 (2014).

Digital Library

[17]

Matthew Cong and Ronald Fedkiw. 2019. Muscle-based facial retargeting with anatomical constraints. In ACM SIGGRAPH 2019 Talks. 1–2.

Digital Library

[18]

Matthew Deying Cong. 2016. Art-directed muscle simulation for high-end facial animation. Stanford University.

[19]

Yao Feng, Haiwen Feng, Michael J Black, and Timo Bolkart. 2021. Learning an animatable detailed 3D face model from in-the-wild images. ACM Transactions on Graphics (ToG) 40, 4 (2021), 1–13.

Digital Library

[20]

Stephan J Garbin, Marek Kowalski, Virginia Estellers, Stanislaw Szymanowicz, Shideh Rezaeifar, Jingjing Shen, Matthew Johnson, and Julien Valentin. 2022. VolTeMorph: Realtime, Controllable and Generalisable Animation of Volumetric Representations. arXiv preprint arXiv:2208.00949 (2022).

[21]

Thomas Gietzen, Robert Brylka, Jascha Achenbach, Katja Zum Hebel, Elmar Schömer, Mario Botsch, Ulrich Schwanecke, and Ralf Schulze. 2019. A method for automatic forensic facial reconstruction based on dense statistics of soft tissue thickness. PloS one 14, 1 (2019), e0210257.

[22]

Benjamin Gilles, Lionel Reveret, and Dinesh K Pai. 2010. Creating and animating subject-specific anatomical models. In Computer Graphics Forum, Vol. 29. Wiley Online Library, 2340–2351.

[23]

Philip-William Grassal, Malte Prinzler, Titus Leistner, Carsten Rother, Matthias Nießner, and Justus Thies. 2022. Neural head avatars from monocular RGB videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 18653–18664.

[24]

David Ha, Andrew Dai, and Quoc V Le. 2016. Hypernetworks. arXiv preprint arXiv:1609.09106 (2016).

[25]

Daniel Holden, Bang Chi Duong, Sayantan Datta, and Derek Nowrouzezahrai. 2019. Subspace neural physics: Fast data-driven interactive simulation. In Proceedings of the 18th annual ACM SIGGRAPH/Eurographics Symposium on Computer Animation. 1–12.

Digital Library

[26]

Alexandru Eugen Ichim, Sofien Bouaziz, and Mark Pauly. 2015. Dynamic 3D avatar creation from hand-held video input. ACM Transactions on Graphics (ToG) 34, 4 (2015), 1–14.

Digital Library

[27]

Alexandru-Eugen Ichim, Petr Kadleček, Ladislav Kavan, and Mark Pauly. 2017. Phace: Physics-based face modeling and animation. ACM Transactions on Graphics (TOG) 36, 4 (2017), 1–14.

Digital Library

[28]

Alexandru Eugen Ichim, Ladislav Kavan, Merlin Nimier-David, and Mark Pauly. 2016. Building and animating user-specific volumetric face rigs. In Symposium on Computer Animation. 107–117.

[29]

Petr Kadleček, Alexandru-Eugen Ichim, Tiantian Liu, Jaroslav Křivánek, and Ladislav Kavan. 2016. Reconstructing personalized anatomical models for physics-based body animation. ACM Transactions on Graphics (TOG) 35, 6 (2016), 1–13.

Digital Library

[30]

Petr Kadleček and Ladislav Kavan. 2019. Building accurate physics-based face models from data. Proceedings of the ACM on Computer Graphics and Interactive Techniques 2, 2 (2019), 1–16.

Digital Library

[31]

Tero Karras, Samuli Laine, and Timo Aila. 2019. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 4401–4410.

[32]

Marilyn Keller, Silvia Zuffi, Michael J Black, and Sergi Pujades. 2022. OSSO: Obtaining Skeletal Shape from Outside. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 20492–20501.

[33]

Martin Komaritzan and Mario Botsch. 2018. Projective skinning. Proceedings of the ACM on Computer Graphics and Interactive Techniques 1, 1 (2018), 1–19.

Digital Library

[34]

Martin Komaritzan, Stephan Wenninger, and Mario Botsch. 2021. Inside Humans: Creating a Simple Layered Anatomical Model from Human Surface Scans. Frontiers in Virtual Reality 2 (2021), 694244.

[35]

Yeara Kozlov, Derek Bradley, Moritz Bächer, Bernhard Thomaszewski, Thabo Beeler, and Markus Gross. 2017. Enriching facial blendshape rigs with physical simulation. In Computer Graphics Forum, Vol. 36. Wiley Online Library, 75–84.

[36]

Juho Lee, Yoonho Lee, Jungtaek Kim, Adam Kosiorek, Seungjin Choi, and Yee Whye Teh. 2019. Set transformer: A framework for attention-based permutation-invariant neural networks. In International conference on machine learning. PMLR, 3744–3753.

[37]

John P Lewis, Ken Anjyo, Taehyun Rhee, Mengjie Zhang, Frederic H Pighin, and Zhigang Deng. 2014. Practice and theory of blendshape facial models.Eurographics (State of the Art Reports) 1, 8 (2014), 2.

[38]

John P Lewis, Jonathan Mooser, Zhigang Deng, and Ulrich Neumann. 2005. Reducing blendshape interference by selected motion attenuation. In Proceedings of the 2005 symposium on Interactive 3D graphics and games. 25–29.

Digital Library

[39]

Hao Li, Thibaut Weise, and Mark Pauly. 2010. Example-based facial rigging. Acm transactions on graphics (tog) 29, 4 (2010), 1–6.

[40]

Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J. Black. 2015. SMPL: A Skinned Multi-Person Linear Model. ACM Trans. Graphics (Proc. SIGGRAPH Asia) 34, 6 (Oct. 2015), 248:1–248:16.

Digital Library

[41]

Nadia Maalin, Sophie Mohamed, Robin SS Kramer, Piers L Cornelissen, Daniel Martin, and Martin J Tovée. 2021. Beyond BMI for self-estimates of body size and shape: A new method for developing stimuli correctly calibrated for body composition. Behavior Research Methods 53, 3 (2021), 1308–1321.

[42]

Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. 2021. Nerf: Representing scenes as neural radiance fields for view synthesis. Commun. ACM 65, 1 (2021), 99–106.

Digital Library

[43]

Frederic I Parke. 1991. Control parameterization for facial animation. In Computer Animation’91. Springer, 3–14.

[44]

Cristian Romero, Dan Casas, Maurizio M Chiaramonte, and Miguel A Otaduy. 2022. Contact-centric deformation learning. ACM Transactions on Graphics (TOG) 41, 4 (2022), 1–11.

Digital Library

[45]

Shunsuke Saito, Zi-Ye Zhou, and Ladislav Kavan. 2015. Computational bodybuilding: Anatomically-based modeling of human bodies. ACM Transactions on Graphics (TOG) 34, 4 (2015), 1–12.

Digital Library

[46]

Igor Santesteban, Elena Garces, Miguel A Otaduy, and Dan Casas. 2020. SoftSMPL: Data-driven Modeling of Nonlinear Soft-tissue Dynamics for Parametric Humans. In Computer Graphics Forum, Vol. 39. Wiley Online Library, 65–75.

[47]

Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini. 2008. The graph neural network model. IEEE transactions on neural networks 20, 1 (2008), 61–80.

[48]

Robert Schleicher, Marlies Nitschke, Jana Martschinke, Marc Stamminger, Björn M Eskofier, Jochen Klucken, and Anne D Koelewijn. 2021. BASH: Biomechanical Animated Skinned Human for Visualization of Kinematics and Muscle Activity. In VISIGRAPP (1: GRAPP). 25–36.

[49]

Eftychios Sifakis, Igor Neverov, and Ronald Fedkiw. 2005. Automatic determination of facial muscle activations from sparse motion capture marker data. In ACM SIGGRAPH 2005 Papers. 417–425.

Digital Library

[50]

Steven L Song, Weiqi Shi, and Michael Reed. 2020. Accurate face rig approximation with deep differential subspace reconstruction. ACM Transactions on Graphics (TOG) 39, 4 (2020), 34–1.

Digital Library

[51]

Jente L Spille, Martin Grunwald, Sven Martin, and Stephanie M Mueller. 2021. Stop touching your face! A systematic review of triggers, characteristics, regulatory functions and neuro-physiology of facial self touch. Neuroscience & Biobehavioral Reviews 128 (2021), 102–116.

[52]

Sangeetha Grama Srinivasan, Qisi Wang, Junior Rojas, Gergely Klár, Ladislav Kavan, and Eftychios Sifakis. 2021. Learning active quasistatic physics-based models from data. ACM Transactions on Graphics (TOG) 40, 4 (2021), 1–14.

Digital Library

[53]

Robert W Sumner and Jovan Popović. 2004. Deformation transfer for triangle meshes. ACM Transactions on graphics (TOG) 23, 3 (2004), 399–405.

[54]

Lingchen Yang, Byungsoo Kim, Gaspard Zoss, Baran Gözcü, Markus Gross, and Barbara Solenthaler. 2022. Implicit neural representation for physics-driven actuated soft bodies. ACM Transactions on Graphics (TOG) 41, 4 (2022), 1–10.

Digital Library

[55]

Li Zhang, Noah Snavely, Brian Curless, and Steven M Seitz. 2008. Spacetime faces: High-resolution capture for modeling and animation. In Data-Driven 3D Facial Animation. Springer, 248–276.

[56]

Yufeng Zheng, Victoria Fernández Abrevaya, Marcel C Bühler, Xu Chen, Michael J Black, and Otmar Hilliges. 2022. Im avatar: Implicit morphable head avatars from videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 13545–13555.

[57]

Wojciech Zielonka, Timo Bolkart, and Justus Thies. 2023. Instant volumetric head avatars. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 4574–4584.

Cited By

Hou JWeng DZhao ZLi YZhou J(2024)Neutral Facial Rigging from Limited Spatiotemporal MeshesElectronics10.3390/electronics1313244513:13(2445)Online publication date: 21-Jun-2024
https://doi.org/10.3390/electronics13132445
Day SXiao ZChaudhry EHooker MZhu XChang JIglesias AYou LZhang J(2024)Improving Realism of Facial Interpolation and Blendshapes with Analytical Partial Differential Equation-Represented PhysicsAxioms10.3390/axioms1303018513:3(185)Online publication date: 12-Mar-2024
https://doi.org/10.3390/axioms13030185
Zhu CJoslin C(2024)Fabrig: A Cloth-Simulated Transferable 3D Face ParameterizationSIGGRAPH Asia 2024 Conference Papers10.1145/3680528.3687695(1-10)Online publication date: 3-Dec-2024
https://dl.acm.org/doi/10.1145/3680528.3687695
Show More Cited By

Index Terms

SoftDECA: Computationally Efficient Physics-Based Facial Animations
1. Computing methodologies
  1. Computer graphics
    1. Animation
      1. Physical simulation
  2. Machine learning
    1. Machine learning approaches
      1. Neural networks

Recommendations

SparseSoftDECA — Efficient high-resolution physics-based facial animation from sparse landmarks
Abstract
Facial animation on computationally limited systems still heavily relies on linear blendshape models. Nonetheless, these models exhibit common issues like volume loss, self-collisions, and inaccuracies in soft tissue elasticity. Furthermore, ...
Highlights
- Computationally efficient physics-based facial animations.
- Localized material adaptions.
- Anatomically plausible personalized facial animations from tracked landmarks.
- Data-driven procedure for fitting the biomechanics anatomy ...
Graphical abstract

Display Omitted
Facial retargeting with automatic range of motion alignment

While facial capturing focuses on accurate reconstruction of an actor's performance, facial animation retargeting has the goal to transfer the animation to another character, such that the semantic meaning of the animation remains. Because of the ...
Example-based data optimization for facial simulation
SIGGRAPH '16: ACM SIGGRAPH 2016 Posters

Digital characters are common in modern films visual effects and the demand for digital actors has increased during the past few years. The success of digitally created actors is related to their believability and, in particular, the realism of the ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

MIG '23: Proceedings of the 16th ACM SIGGRAPH Conference on Motion, Interaction and Games

November 2023

224 pages

ISBN:9798400703935

DOI:10.1145/3623264

Editors:
Julien Pettré
Inria, France
,
Barbara Solenthaler
ETH Zurich, Switzerland
,
Rachel McDonnell
TCD, Ireland
,
Christopher Peters
KTH, Sweden

Copyright © 2023 Owner/Author.

This work is licensed under a Creative Commons Attribution International 4.0 License.

Sponsors

SIGGRAPH: ACM Special Interest Group on Computer Graphics and Interactive Techniques

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 15 November 2023

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

MIG '23

Sponsor:

SIGGRAPH

MIG '23: The 16th ACM SIGGRAPH Conference on Motion, Interaction and Games

November 15 - 17, 2023

Rennes, France

Acceptance Rates

Overall Acceptance Rate -9 of -9 submissions, 100%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

10
Total Citations
View Citations
644
Total Downloads

Downloads (Last 12 months)575
Downloads (Last 6 weeks)47

Reflects downloads up to 19 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Hou JWeng DZhao ZLi YZhou J(2024)Neutral Facial Rigging from Limited Spatiotemporal MeshesElectronics10.3390/electronics1313244513:13(2445)Online publication date: 21-Jun-2024
https://doi.org/10.3390/electronics13132445
Day SXiao ZChaudhry EHooker MZhu XChang JIglesias AYou LZhang J(2024)Improving Realism of Facial Interpolation and Blendshapes with Analytical Partial Differential Equation-Represented PhysicsAxioms10.3390/axioms1303018513:3(185)Online publication date: 12-Mar-2024
https://doi.org/10.3390/axioms13030185
Zhu CJoslin C(2024)Fabrig: A Cloth-Simulated Transferable 3D Face ParameterizationSIGGRAPH Asia 2024 Conference Papers10.1145/3680528.3687695(1-10)Online publication date: 3-Dec-2024
https://dl.acm.org/doi/10.1145/3680528.3687695
Yang LZoss GChandran PGross MSolenthaler BSifakis EBradley D(2024)Learning a Generalized Physical Face Model From DataACM Transactions on Graphics10.1145/365818943:4(1-14)Online publication date: 19-Jul-2024
https://dl.acm.org/doi/10.1145/3658189
Zhu CJoslin C(2024)A review of motion retargeting techniques for 3D character facial animationComputers & Graphics10.1016/j.cag.2024.104037123(104037)Online publication date: Oct-2024
https://doi.org/10.1016/j.cag.2024.104037
Wagner NSchwanecke UBotsch M(2024)AnaConDaR: Anatomically-Constrained Data-Adaptive Facial RetargetingComputers & Graphics10.1016/j.cag.2024.103988122(103988)Online publication date: Aug-2024
https://doi.org/10.1016/j.cag.2024.103988
Nicolas WUlrich SMario B(2024)SparseSoftDECA — Efficient high-resolution physics-based facial animation from sparse landmarksComputers & Graphics10.1016/j.cag.2024.103903119(103903)Online publication date: Apr-2024
https://doi.org/10.1016/j.cag.2024.103903
Büchner TPenzel NGuntinas-Lichius ODenzler J(2024)Facing Asymmetry - Uncovering the Causal Link Between Facial Symmetry and Expression Classifiers Using Synthetic InterventionsComputer Vision – ACCV 202410.1007/978-981-96-0911-6_26(443-464)Online publication date: 8-Dec-2024
https://doi.org/10.1007/978-981-96-0911-6_26
Milojevic APeter DHuber NAzevedo LLatyshev ASailer IGross MThomaszewski BSolenthaler BGözcü B(2024)AutoSkull: Learning-Based Skull Estimation for Automated PipelinesMedical Image Computing and Computer Assisted Intervention – MICCAI 202410.1007/978-3-031-72104-5_11(109-118)Online publication date: 7-Oct-2024
https://dl.acm.org/doi/10.1007/978-3-031-72104-5_11
Zhu CJoslin C(2024)A Facial Motion Retargeting Pipeline for Appearance Agnostic 3D CharactersComputer Animation and Virtual Worlds10.1002/cav.7000135:6Online publication date: 19-Nov-2024
https://doi.org/10.1002/cav.70001

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents