More Web Proxy on the site http://driver.im/

research-article

Free access

Realtime facial animation with on-the-fly correctives

Authors:

Chris BreglerAuthors Info & Claims

ACM Transactions on Graphics (TOG), Volume 32, Issue 4

Article No.: 42, Pages 1 - 10

https://doi.org/10.1145/2461912.2462019

Published: 21 July 2013 Publication History

Abstract

We introduce a real-time and calibration-free facial performance capture framework based on a sensor with video and depth input. In this framework, we develop an adaptive PCA model using shape correctives that adjust on-the-fly to the actor's expressions through incremental PCA-based learning. Since the fitting of the adaptive model progressively improves during the performance, we do not require an extra capture or training session to build this model. As a result, the system is highly deployable and easy to use: it can faithfully track any individual, starting from just a single face scan of the subject in a neutral pose. Like many real-time methods, we use a linear subspace to cope with incomplete input data and fast motion. To boost the training of our tracking model with reliable samples, we use a well-trained 2D facial feature tracker on the input video and an efficient mesh deformation algorithm to snap the result of the previous step to high frequency details in visible depth map regions. We show that the combination of dense depth maps and texture features around eyes and lips is essential in capturing natural dialogues and nuanced actor-specific emotions. We demonstrate that using an adaptive PCA model not only improves the fitting accuracy for tracking but also increases the expressiveness of the retargeted character.

Supplementary Material

ZIP File (a42-li.zip)

Supplemental material.

Download
241.02 MB

MP4 File (tp089.mp4)

Download
293.21 MB

References

[1]

Alexander, O., Rogers, M., Lambeth, W., Chiang, M., and Debevec, P. 2009. The digital Emily project: photo-real facial modeling and animation. In ACM SIGGRAPH 2009 Courses, 12:1--12:15.

Digital Library

[2]

Beeler, T., Hahn, F., Bradley, D., Bickel, B., Beardsley, P., Gotsman, C., Sumner, R. W., and Gross, M. 2011. High-quality passive facial performance capture using anchor frames. ACM Trans. Graph. 30 (August), 75:1--75:10.

Digital Library

[3]

Bickel, B., Lang, M., Botsch, M., Otaduy, M. A., and Gross, M. 2008. Pose-space animation and transfer of facial details. In Proceedings of the 2008 ACM SIGGRAPH/Eurographics Symposium on Computer Animation.

Digital Library

[4]

Black, M. J., and Yacoob, Y. 1995. Tracking and recognizing rigid and non-rigid facial motions using local parametric models of image motion. In Proceedings of the Fifth International Conference on Computer Vision, IEEE Computer Society, Washington, DC, USA, ICCV '95, 374--.

Digital Library

[5]

Blanz, V., and Vetter, T. 1999. A morphable model for the synthesis of 3d faces. In Proceedings of ACM Siggraph 99, ACM Press/Addison-Wesley Publishing Co., 187--194.

Digital Library

[6]

Borshukov, G., Piponi, D., Larsen, O., Lewis, J. P., and Tempelaar-Lietz, C. 2005. Universal capture - image-based facial animation for "the matrix reloaded". In ACM SIGGRAPH 2005 Courses.

Digital Library

[7]

Botsch, M., and Sorkine, O. 2008. On linear variational surface deformation methods. IEEE Transactions on Visualization and Computer Graphics 14, 1 (Jan.), 213--230.

Digital Library

[8]

Bradley, D., Heidrich, W., Popa, T., and Sheffer, A. 2010. High resolution passive facial performance capture. ACM Trans. Graph. 29 (July), 41:1--41:10.

Digital Library

[9]

Bregler, C., and Omohundro, S. 1994. Surface learning with applications to lipreading. Advances in neural information processing systems, 43--43.

[10]

Bregler, C., Covell, M., and Slaney, M. 1997. Video rewrite: Driving visual speech with audio. In Proceedings of Computer graphics and interactive techniques.

Digital Library

[11]

Chai, J.-x., Xiao, J., and Hodgins, J. 2003. Vision-based control of 3d facial animation. In Proceedings of the 2003 ACM SIGGRAPH/Eurographics symposium on Computer animation, Eurographics Association, Aire-la-Ville, Switzerland, Switzerland, SCA '03, 193--206.

Digital Library

[12]

Chandrasekaran, S., Manjunath, B. S., Wang, Y.-F., Winkeler, J., and Zhang, H. 1997. An eigenspace update algorithm for image analysis. CVGIP: Graphical Model and Image Processing, 5, 321--332.

Digital Library

[13]

Chuang, E., and Bregler, C. 2002. Performance driven facial animation using blendshape interpolation. Tech. rep., Stanford University.

[14]

Collins, R., Liu, Y., and Leordeanu, M. 2005. Online selection of discriminative tracking features. Pattern Analysis and Machine Intelligence, IEEE Transactions on 27, 10, 1631--1643.

Digital Library

[15]

Cootes, T. F., Edwards, G. J., and Taylor, C. J. 1998. Active appearance models. In IEEE Transactions on Pattern Analysis and Machine Intelligence, Springer, 484--498.

Digital Library

[16]

Covell, M., and Bregler, C. 1996. Eigen-points. In Image Processing, 1996. Proceedings., International Conference on, vol. 3, IEEE, 471--474.

[17]

Decarlo, D., and Metaxas, D. 2000. Optical flow constraints on deformable models with applications to face tracking. Int. J. Comput. Vision 38, 2 (July), 99--127.

Digital Library

[18]

Dempster, A., Laird, N., and Rubin, D. 1977. Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Society. Series B (Methodological), 1--38.

[19]

Ekman, P., and Friesen, W. 1978. Facial Action Coding System: A Technique for the Measurement of Facial Movement. Consulting Psychologists Press, Palo Alto.

[20]

Essa, I., Basu, S., Darrell, T., and Pentland, A. 1996. Modeling, tracking and interactive animation of faces and heads using input from video. In Proceedings of the Computer Animation, IEEE Computer Society, CA '96, 68--.

Digital Library

[21]

Furukawa, Y., and Ponce, J. 2009. Dense 3d motion capture for human faces. In 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2009), 20--25 June 2009, Miami, Florida, USA, IEEE, 1674--1681.

[22]

Fyffe, G., Hawkins, T., Watts, C., Ma, W.-C., and Debevec, P. 2011. Comprehensive facial performance capture. In Eurographics 2011.

[23]

Grabner, H., Leistner, C., and Bischof, H. 2008. Semi-supervised on-line boosting for robust tracking. In Proceedings of the 10th European Conference on Computer Vision: Part I, Springer-Verlag, Berlin, Heidelberg, ECCV '08, 234--247.

Digital Library

[24]

Gu, M., and Eisenstat, S. C. 1993. A Stable and Fast Algorithm for Updating the Singular Value Decomposition. Tech. Rep. YALEU/DCS/RR-966, Yale University, New Haven, CT.

[25]

Guenter, B., Grimm, C., Wood, D., Malvar, H., and Pighin, F. 1998. Making faces. In Proceedings of SIGGRAPH '98, ACM, 55--66.

Digital Library

[26]

ImageMetrics. 2012. Image metrics live driver SDK http://www.image-metrics.com/livedriver/.

[27]

Kalal, Z., Matas, J., and Mikolajczyk, K. 2009. Online learning of robust object detectors during unstable tracking. In In International Conference on Computer Vision.

[28]

Kirby, M., and Sirovich, L. 1990. Application of the karhunen-loeve procedure for the characterization of human faces. Pattern Analysis and Machine Intelligence, IEEE Transactions on 12, 1, 103--108.

Digital Library

[29]

Li, H., Roivainen, P., and Forcheimer, R. 1993. 3-d motion estimation in model-based facial image coding. IEEE Transactions on PAMI 15, 6, 545--555.

Digital Library

[30]

Li, H., Adams, B., Guibas, L. J., and Pauly, M. 2009. Robust single-view geometry and motion reconstruction. ACM Transactions on Graphics (Proceedings SIGGRAPH Asia 2009) 28, 5.

Digital Library

[31]

Li, H., Weise, T., and Pauly, M. 2010. Example-based facial rigging. ACM Transactions on Graphics (Proceedings SIGGRAPH 2010) 29, 3 (July).

Digital Library

[32]

Paysan, P., Knothe, R., Amberg, B., Romdhani, S., and Vetter, T. 2009. A 3d face model for pose and illumination invariant face recognition.

[33]

Pearson, K. 1901. Liii. on lines and planes of closest fit to systems of points in space. The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science 2, 11, 559--572.

[34]

Pighin, F., and Lewis, J. P. 2006. Performance-driven facial animation. In ACM SIGGRAPH 2006 Courses, ACM, New York, NY, USA, SIGGRAPH '06.

[35]

Pighin, F. H., Szeliski, R., and Salesin, D. 1999. Resynthesizing Facial Animation through 3D Model-based Tracking. In Proc. 7th International Conference on Computer Vision, Kerkyra, Greece, 143--150.

[36]

Roweis, S. 1998. EM algorithms for pca and spca. In in Advances in Neural Information Processing Systems, MIT Press, 626--632.

Digital Library

[37]

Rusinkiewicz, S., and Levoy, M. 2001. Efficient variants of the icp algorithm. In International Conference on 3-D Digital Imaging and Modeling.

[38]

Rusinkiewicz, S., Hall-Holt, O., and Levoy, M. 2002. Real-time 3D model acquisition. ACM Transactions on Graphics (Proc. SIGGRAPH) 21, 3 (July), 438--446.

Digital Library

[39]

Saragih, J. M., Lucey, S., and Cohn, J. F. 2011. Deformable model fitting by regularized landmark mean-shift. Int. J. Comput. Vision 91, 2 (Jan.), 200--215.

Digital Library

[40]

Skocaj, D., and Leonardis, A. 2003. Weighted and robust incremental method for subspace learning. In Computer Vision, 2003. Proceedings. Ninth IEEE International Conference on, IEEE, 1494--1501.

Digital Library

[41]

Sugimoto, T., Fukushima, M., and Ibaraki, T. 1995. A parallel relaxation method for quadratic programming problems with interval constraints. Journal of Computational and Applied Mathematics 60, 12, 219--236.

Digital Library

[42]

Sumner, R. W., and Popović, J. 2004. Deformation transfer for triangle meshes. ACM Trans. Graph. 23, 3 (Aug.), 399--405.

Digital Library

[43]

Valgaerts, L., Wu, C., Bruhn, A., Seidel, H.-P., and Theobalt, C. 2012. Lightweight binocular facial performance capture under uncontrolled lighting. In ACM Transactions on Graphics (Proceedings of SIGGRAPH Asia 2012), vol. 31.

Digital Library

[44]

Vlasic, D., Brand, M., Pfister, H., and Popović, J. 2005. Face transfer with multilinear models. In ACM SIGGRAPH 2005 Papers, ACM, New York, NY, USA, SIGGRAPH '05, 426--433.

Digital Library

[45]

Weise, T., Li, H., Gool, L. V., and Pauly, M. 2009. Face/off: Live facial puppetry. In Proceedings of the 2009 ACM SIGGRAPH/Eurographics Symposium on Computer animation.

Digital Library

[46]

Weise, T., Bouaziz, S., Li, H., and Pauly, M. 2011. Real-time performance-based facial animation. ACM Transactions on Graphics (Proceedings SIGGRAPH 2011) 30, 4 (July).

Digital Library

[47]

Welch, G., and Bishop, G. 1995. An introduction to the kalman filter. Tech. rep., Chapel Hill, NC, USA.

Digital Library

[48]

Williams, L. 1990. Performance-driven facial animation. SIGGRAPH Comput. Graph. 24, 4 (Sept.), 235--242.

Digital Library

[49]

Zhang, S., and Huang, P. 2004. High-resolution, real-time 3d shape acquisition. In Proceedings of the 2004 Conference on Computer Vision and Pattern Recognition Workshop (CVPRW'04) Volume 3 - Volume 03, IEEE Computer Society, Washington, DC, USA, CVPRW '04, 28--.

Digital Library

[50]

Zhang, L., Snavely, N., Curless, B., and Seitz, S. M. 2004. Spacetime faces: High-resolution capture for modeling and animation. In ACM Annual Conference on Computer Graphics, 548--558.

Digital Library

Cited By

Sirola MKoskinen MPolvinen TPihlatie M(2024)Tracing State Structure for Ecological Processes in Soil Including Greenhouse Gas Exchange with Lower AtmosphereSensors10.3390/s2411350724:11(3507)Online publication date: 29-May-2024
https://doi.org/10.3390/s24113507
Li YNumerow LThomaszewski BCoros S(2024)Differentiable Geodesic Distance for Intrinsic Minimization on Triangle MeshesACM Transactions on Graphics10.1145/365812243:4(1-14)Online publication date: 19-Jul-2024
https://dl.acm.org/doi/10.1145/3658122
Bednarik JWood EChoutas VBolkart TWang DWu CBeeler T(2024)Learning to Stabilize FacesComputer Graphics Forum10.1111/cgf.1503843:2Online publication date: 24-Apr-2024
https://doi.org/10.1111/cgf.15038
Show More Cited By

Index Terms

Realtime facial animation with on-the-fly correctives
1. Computing methodologies
  1. Computer graphics
    1. Animation
    2. Graphics systems and interfaces
2. Human-centered computing
  1. Human computer interaction (HCI)
    1. Interaction devices
      1. Graphics input devices

Recommendations

Online modeling for realtime facial animation

We present a new algorithm for realtime face tracking on commodity RGB-D sensing devices. Our method requires no user-specific training or calibration, or any other form of manual assistance, thus enabling a range of new applications in performance-...
Realtime performance-based facial animation

This paper presents a system for performance-based character animation that enables any user to control the facial expressions of a digital avatar in realtime. The user is recorded in a natural environment using a non-intrusive, commercially available ...
High fidelity facial animation capture and retargeting with contours
SCA '13: Proceedings of the 12th ACM SIGGRAPH/Eurographics Symposium on Computer Animation

Human beings are naturally sensitive to subtle cues in facial expressions, especially in areas of the eyes and mouth. Current facial motion capture methods fail to accurately reproduce motions in those areas due to multiple limitations. In this paper, ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Transactions on Graphics

ACM Transactions on Graphics Volume 32, Issue 4

July 2013

1215 pages

ISSN:0730-0301

EISSN:1557-7368

DOI:10.1145/2461912

Issue’s Table of Contents

Copyright © 2013 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 21 July 2013

Published in TOG Volume 32, Issue 4

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

253
Total Citations
View Citations
1,724
Total Downloads

Downloads (Last 12 months)42
Downloads (Last 6 weeks)5

Reflects downloads up to 11 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Sirola MKoskinen MPolvinen TPihlatie M(2024)Tracing State Structure for Ecological Processes in Soil Including Greenhouse Gas Exchange with Lower AtmosphereSensors10.3390/s2411350724:11(3507)Online publication date: 29-May-2024
https://doi.org/10.3390/s24113507
Li YNumerow LThomaszewski BCoros S(2024)Differentiable Geodesic Distance for Intrinsic Minimization on Triangle MeshesACM Transactions on Graphics10.1145/365812243:4(1-14)Online publication date: 19-Jul-2024
https://dl.acm.org/doi/10.1145/3658122
Bednarik JWood EChoutas VBolkart TWang DWu CBeeler T(2024)Learning to Stabilize FacesComputer Graphics Forum10.1111/cgf.1503843:2Online publication date: 24-Apr-2024
https://doi.org/10.1111/cgf.15038
Pan YTan SCheng SLin QZeng ZMitchell K(2024)Expressive Talking AvatarsIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2024.337204730:5(2538-2548)Online publication date: 4-Mar-2024
https://dl.acm.org/doi/10.1109/TVCG.2024.3372047
Retsinas GFilntisis PDaněček RAbrevaya VRoussos ABolkarr TMaragos P(2024)3D Facial Expressions through Analysis-by-Neural-Synthesis2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR52733.2024.00241(2490-2501)Online publication date: 16-Jun-2024
https://doi.org/10.1109/CVPR52733.2024.00241
Tran MChang DSiniukov MSoleymani M(2024)DIM: Dyadic Interaction Modeling for Social Behavior GenerationComputer Vision – ECCV 202410.1007/978-3-031-72913-3_27(484-503)Online publication date: 2-Dec-2024
https://doi.org/10.1007/978-3-031-72913-3_27
Ming XLi JLing JZhang LXu F(2024)High-Quality Mesh Blendshape Generation from Face Videos via Neural Inverse RenderingComputer Vision – ECCV 202410.1007/978-3-031-72897-6_7(106-125)Online publication date: 2-Dec-2024
https://doi.org/10.1007/978-3-031-72897-6_7
Kumar SNandini Arkam MChaturvedi S(2024)A Comparative Overview of Deep Learning Aided Image GenerationProceedings of 4th International Conference on Artificial Intelligence and Smart Energy10.1007/978-3-031-61471-2_2(18-34)Online publication date: 12-Jun-2024
https://doi.org/10.1007/978-3-031-61471-2_2
Wu YUmetani N(2023)Two-Way Coupling of Skinning Transformations and Position Based DynamicsProceedings of the ACM on Computer Graphics and Interactive Techniques10.1145/36069306:3(1-18)Online publication date: 24-Aug-2023
https://dl.acm.org/doi/10.1145/3606930
Pan YZhang RWang JDing YMitchell KEl Saddik AMei TCucchiara RBertini MTobon Vallejo DAtrey PHossain M(2023)Real-time Facial Animation for 3D Stylized Character with Emotion DynamicsProceedings of the 31st ACM International Conference on Multimedia10.1145/3581783.3613803(6851-6859)Online publication date: 26-Oct-2023
https://dl.acm.org/doi/10.1145/3581783.3613803
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

Media

Figures

Other

Tables

View Issue’s Table of Contents