More Web Proxy on the site http://driver.im/

research-article

High-quality passive facial performance capture using anchor frames

Authors:

Paul Beardsley,

Robert W. Sumner,

Markus GrossAuthors Info & Claims

ACM Transactions on Graphics (TOG), Volume 30, Issue 4

Article No.: 75, Pages 1 - 10

https://doi.org/10.1145/2010324.1964970

Published: 25 July 2011 Publication History

Abstract

We present a new technique for passive and markerless facial performance capture based on anchor frames. Our method starts with high resolution per-frame geometry acquisition using state-of-the-art stereo reconstruction, and proceeds to establish a single triangle mesh that is propagated through the entire performance. Leveraging the fact that facial performances often contain repetitive subsequences, we identify anchor frames as those which contain similar facial expressions to a manually chosen reference expression. Anchor frames are automatically computed over one or even multiple performances. We introduce a robust image-space tracking method that computes pixel matches directly from the reference frame to all anchor frames, and thereby to the remaining frames in the sequence via sequential matching. This allows us to propagate one reconstructed frame to an entire sequence in parallel, in contrast to previous sequential methods. Our anchored reconstruction approach also limits tracker drift and robustly handles occlusions and motion blur. The parallel tracking and mesh propagation offer low computation times. Our technique will even automatically match anchor frames across different sequences captured on different occasions, propagating a single mesh to all performances.

Supplementary Material

Supplemental material. (a75-beeler.zip)

Download
48.92 MB

References

[1]

Alexander, O., Rogers, M., Lambeth, W., Chiang, M., and Debevec, P. 2009. The digital Emily project: photoreal facial modeling and animation. In ACM SIGGRAPH Courses, 1--15.

[2]

Anuar, N., and Guskov, I. 2004. Extracting animated meshes with adaptive motion estimation. In Proc. Vision, Modeling, and Visualization, 63--71.

[3]

Beeler, T., Bickel, B., Sumner, R., Beardsley, P., and Gross, M. 2010. High-quality single-shot capture of facial geometry. ACM Trans. Graphics (Proc. SIGGRAPH), 40.

[4]

Bickel, B., Botsch, M., Angst, R., Matusik, W., Otaduy, M., Pfister, H., and Gross, M. 2007. Multi-scale capture of facial geometry and motion. ACM Trans. Graphics (Proc. SIGGRAPH), 33.

[5]

Blanz, V., Basso, C., Vetter, T., and Poggio, T. 2003. Reanimating faces in images and video. Computer Graphics Forum (Proc. Eurographics) 22, 3, 641--650.

[6]

Bradley, D., Popa, T., Sheffer, A., Heidrich, W., and Boubekeur, T. 2008. Markerless garment capture. ACM Trans. Graphics (Proc. SIGGRAPH), 99.

[7]

Bradley, D., Heidrich, W., Popa, T., and Sheffer, A. 2010. High resolution passive facial performance capture. ACM Trans. Graphics (Proc. SIGGRAPH), 41.

[8]

DeCarlo, D., and Metaxas, D. 1996. The integration of optical flow and deformable models with applications to human face shape and motion estimation. In Proc. CVPR, 231--238.

Digital Library

[9]

Ekman, P., and Friesen, W. 1978. The facial action coding system: A technique for the measurement of facial movement. In Consulting Psychologists.

[10]

Essa, I., Basu, S., Darrell, T., and Pentland, A. 1996. Modeling, tracking and interactive animation of faces and heads using input from video. In Proc. Computer Animation, 68.

Digital Library

[11]

Furukawa, Y., and Ponce, J. 2009. Dense 3D motion capture for human faces. In Proc. CVPR, 1674--1681.

[12]

Fyffe, G., Hawkins, T., Watts, C., Ma, W.-C., and Debevec, P. 2011. Comprehensive facial performance capture. Comp. Graphics Forum (Proc. Eurographics) 30, 2, 425--434.

[13]

Guenter, B., Grimm, C., Wood, D., Malvar, H., and Pighin, F. 1998. Making faces. In Comp. Graphics, 55--66.

[14]

Hernández, C., and Vogiatzis, G. 2010. Self-calibrating a real-time monocular 3D facial capture system. In Proceedings International Symposium on 3D Data Processing, Visualization and Transmission (3DPVT).

[15]

Kraevoy, V., and Sheffer, A. 2004. Cross-parameterization and compatible remeshing of 3D models. ACM Trans. Graph. 23, 861--869.

Digital Library

[16]

Li, H., Roivainen, P., and Forcheimer, R. 1993. 3-D motion estimation in model-based facial image coding. IEEE Trans. Pattern Anal. Mach. Intell. 15, 6, 545--555.

Digital Library

[17]

Lin, I.-C., and Ouhyoung, M. 2005. Mirror mocap: Automatic and efficient capture of dense 3D facial motion parameters from video. The Visual Computer 21, 6, 355--372.

[18]

Lowe, D. G. 2004. Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vision 60, 2, 91--110.

Digital Library

[19]

Ma, W.-C., Hawkins, T., Peers, P., Chabert, C.-F., Weiss, M., and Debevec, P. 2007. Rapid acquisition of specular and diffuse normal maps from polarized spherical gradient illumination. In Eurographics Symposium on Rendering, 183--194.

[20]

Ma, W.-C., Jones, A., Chiang, J.-Y., Hawkins, T., Frederiksen, S., Peers, P., Vukovic, M., Ouhyoung, M., and Debevec, P. 2008. Facial performance synthesis using deformation-driven polynomial displacement maps. ACM Trans. Graphics (Proc. SIGGRAPH Asia) 27, 5, 121.

Digital Library

[21]

Pighin, F. H., Szeliski, R., and Salesin, D. 1999. Resynthesizing facial animation through 3D model-based tracking. In Proc. ICCV, 143--150.

[22]

Popa, T., South-Dickinson, I., Bradley, D., Sheffer, A., and Heidrich, W. 2010. Globally consistent space-time reconstruction. Comp. Graphics Forum (Proc. SGP), 1633--1642.

[23]

Rav-Acha, A., Kohli, P., Rother, C., and Fitzgibbon, A. 2008. Unwrap mosaics: A new representation for video editing. ACM Trans. Graphics (Proc. SIGGRAPH), 17.

[24]

Sharf, A., Alcantara, D. A., Lewiner, T., Greif, C., Sheffer, A., Amenta, N., and Cohen-Or, D. 2008. Space-time surface reconstruction using incompressible flow. ACM Trans. Graphics 27, 110.

Digital Library

[25]

Sumner, R. W., and Popović, J. 2004. Deformation transfer for triangle meshes. ACM Trans. Graphics 23, 399--405.

Digital Library

[26]

Wand, M., Adams, B., Ovsjanikov, M., Berner, A., Bokeloh, M., Jenke, P., Guibas, L., Seidel, H.-P., and Schilling, A. 2009. Efficient reconstruction of nonrigid shape and motion from real-time 3D scanner data. ACM Trans. Graph. 28, 2, 1--15.

Digital Library

[27]

Wang, Y., Huang, X., Lee, C.-S., Zhang, S., Li, Z., Samaras, D., Metaxas, D., Elgammal, A., and Huang, P. 2004. High resolution acquisition, learning and transfer of dynamic 3-D facial expressions. Comp. Graphics Forum 23, 3, 677--686.

[28]

Williams, L. 1990. Performance-driven facial animation. In Computer Graphics (Proc. SIGGRAPH), vol. 24, 235--242.

[29]

Wilson, C. A., Ghosh, A., Peers, P., Chiang, J.-Y., Busch, J., and Debevec, P. 2010. Temporal upsampling of performance geometry using photometric alignment. ACM Trans. Graphics 29, 2.

Digital Library

[30]

Winkler, T., Hormann, K., and Gotsman, C. 2008. Mesh massage. The Visual Computer 24, 775--785.

Digital Library

[31]

Zhang, L., Snavely, N., Curless, B., and Seitz, S. M. 2004. Spacetime faces: High resolution capture for modeling and animation. ACM Trans. Graphics 23, 3, 548--558.

Digital Library

Cited By

Tian ZLiang BFang HWeng D(2024)Dynamic 4D facial capture pipeline with appearance driven progressive retopology based on optical flowOptics Express10.1364/OE.52984632:18(31830)Online publication date: 19-Aug-2024
https://doi.org/10.1364/OE.529846
Ha HHwang IMonzon NCho JKim DBaek SMuñoz AGutierrez DKim M(2024)Polarimetric BSSRDF Acquisition of Dynamic FacesACM Transactions on Graphics10.1145/368776743:6(1-11)Online publication date: 19-Dec-2024
https://dl.acm.org/doi/10.1145/3687767
Bai SWang TLi CVenkatesh ASimon TCao CSchwartz GSaragih JSheikh YWei S(2024)Universal Facial Encoding of Codec Avatars from VR HeadsetsACM Transactions on Graphics10.1145/365823443:4(1-22)Online publication date: 19-Jul-2024
https://dl.acm.org/doi/10.1145/3658234
Show More Cited By

Index Terms

High-quality passive facial performance capture using anchor frames

Recommendations

High resolution passive facial performance capture

We introduce a purely passive facial capture approach that uses only an array of video cameras, but requires no template facial geometry, no special makeup or markers, and no active lighting. We obtain initial geometry using multi-view stereo, and then ...
High-quality passive facial performance capture using anchor frames
SIGGRAPH '11: ACM SIGGRAPH 2011 papers

We present a new technique for passive and markerless facial performance capture based on anchor frames. Our method starts with high resolution per-frame geometry acquisition using state-of-the-art stereo reconstruction, and proceeds to establish a ...
High Quality Binocular Facial Performance Capture from Partially Blurred Image Sequence
CADGRAPHICS '13: Proceedings of the 2013 International Conference on Computer-Aided Design and Computer Graphics

Existing methods on passive facial performance capture assume that the input images are well captured. They merely consider how to deal with motion blurred images in the input sequence, which is very common in the image capture process. This paper ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Transactions on Graphics

ACM Transactions on Graphics Volume 30, Issue 4

July 2011

829 pages

ISSN:0730-0301

EISSN:1557-7368

DOI:10.1145/2010324

Issue’s Table of Contents

Copyright © 2011 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 25 July 2011

Published in TOG Volume 30, Issue 4

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

243
Total Citations
View Citations
2,833
Total Downloads

Downloads (Last 12 months)40
Downloads (Last 6 weeks)8

Reflects downloads up to 24 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Tian ZLiang BFang HWeng D(2024)Dynamic 4D facial capture pipeline with appearance driven progressive retopology based on optical flowOptics Express10.1364/OE.52984632:18(31830)Online publication date: 19-Aug-2024
https://doi.org/10.1364/OE.529846
Ha HHwang IMonzon NCho JKim DBaek SMuñoz AGutierrez DKim M(2024)Polarimetric BSSRDF Acquisition of Dynamic FacesACM Transactions on Graphics10.1145/368776743:6(1-11)Online publication date: 19-Dec-2024
https://dl.acm.org/doi/10.1145/3687767
Bai SWang TLi CVenkatesh ASimon TCao CSchwartz GSaragih JSheikh YWei S(2024)Universal Facial Encoding of Codec Avatars from VR HeadsetsACM Transactions on Graphics10.1145/365823443:4(1-22)Online publication date: 19-Jul-2024
https://dl.acm.org/doi/10.1145/3658234
Zou KFaisan SYu BValette SSeo H(2024)4D Facial Expression Diffusion ModelACM Transactions on Multimedia Computing, Communications, and Applications10.1145/365345521:1(1-23)Online publication date: 16-Dec-2024
https://dl.acm.org/doi/10.1145/3653455
Cong MLan LFedkiw R(2024)Local Geometric Indexing of High Resolution Data for Facial Reconstruction From Sparse MarkersIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2023.328949530:8(5289-5298)Online publication date: Aug-2024
https://doi.org/10.1109/TVCG.2023.3289495
Saito SSchwartz GSimon TLi JNam G(2024)Relightable Gaussian Codec Avatars2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR52733.2024.00021(130-141)Online publication date: 16-Jun-2024
https://doi.org/10.1109/CVPR52733.2024.00021
Kabadayi BZielonka WBhatnagar BPons-Moll GThies J(2024)GAN-Avatar: Controllable Personalized GAN-based Human Head Avatar2024 International Conference on 3D Vision (3DV)10.1109/3DV62453.2024.00058(882-892)Online publication date: 18-Mar-2024
https://doi.org/10.1109/3DV62453.2024.00058
Peng SZhu XYi DQian CLei Z(2024)Formulating facial mesh tracking as a differentiable optimization problem: a backpropagation-based solutionVisual Intelligence10.1007/s44267-024-00054-x2:1Online publication date: 19-Jul-2024
https://doi.org/10.1007/s44267-024-00054-x
Ming XLi JLing JZhang LXu F(2024)High-Quality Mesh Blendshape Generation from Face Videos via Neural Inverse RenderingComputer Vision – ECCV 202410.1007/978-3-031-72897-6_7(106-125)Online publication date: 2-Dec-2024
https://doi.org/10.1007/978-3-031-72897-6_7
Bharadwaj SZheng YHilliges OBlack MAbrevaya V(2023)FLARE: Fast Learning of Animatable and Relightable Mesh AvatarsACM Transactions on Graphics10.1145/361840142:6(1-15)Online publication date: 5-Dec-2023
https://doi.org/10.1145/3618401
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents