DOI: 10.1145/2010324.1964972. Realtime performance-based facial animation

Realtime performance-based facial animation

10.1145/2010324.1964972

Crossref journal-article

Association for Computing Machinery (ACM)

ACM Transactions on Graphics (320)

Abstract

This paper presents a system for performance-based character animation that enables any user to control the facial expressions of a digital avatar in realtime. The user is recorded in a natural environment using a non-intrusive, commercially available 3D sensor. The simplicity of this acquisition device comes at the cost of high noise levels in the acquired data. To effectively map low-quality 2D images and 3D depth maps to realistic facial expressions, we introduce a novel face tracking algorithm that combines geometry and texture registration with pre-recorded animation priors in a single optimization. Formulated as a maximum a posteriori estimation in a reduced parameter space, our method implicitly exploits temporal coherence to stabilize the tracking. We demonstrate that compelling 3D facial dynamics can be reconstructed in realtime without the use of face markers, intrusive lighting, or complex scanning hardware. This makes our system easy to deploy and facilitates a range of new applications, e.g. in digital gameplay or social interactions.

Bibliography

Weise, T., Bouaziz, S., Li, H., & Pauly, M. (2011). Realtime performance-based facial animation. ACM Transactions on Graphics, 30(4), 1â10.

Authors 4

Thibaut Weise (first)
Sofien Bouaziz (additional)
Hao Li (additional)
Mark Pauly (additional)

References 40 Referenced 329

10.1145/1667239.1667251
10.1145/1778765.1778777
Black M. J. and Yacoob Y. 1995. Tracking and recognizing rigid and non-rigid facial motions using local parametric models of image motion. In ICCV 374--381. (10.5555/839277.840053)
10.1145/311535.311556
10.1145/1198555.1198596
10.1145/1778765.1778778
Chai J. X. Xiao J. and Hodgins J. 2003. Vision-based control of 3d facial animation. In SCA. (10.5555/846276.846304)
Chuang E. and Bregler C. 2002. Performance driven facial animation using blendshape interpolation. Tech. rep. Stanford University.
10.1109/34.927467
10.5555/524467.796028
DeCarlo D. and Metaxas D. 1996. The integration of optical flow and deformable models with applications to human face shape and motion estimation. In CVPR. (10.5555/794190.794604)
10.1023/A:1008122917811
Ekman P. and Friesen W. 1978. Facial Action Coding System: A Technique for the Measurement of Facial Movement. Consulting Psychologists Press. (10.1037/t27734-000)
10.5555/791215.791488
Furukawa Y. and Ponce J. 2009. Dense 3d motion capture for human faces. In CVPR. (10.1109/CVPRW.2009.5206868)
10.1145/1015706.1015755
Guenter, B., Grimm, C., Wood, D., Malvar, H., and Pighin, F. 1993. Making faces. IEEE Computer Graphics and Applications 13, 6--8. / IEEE Computer Graphics and Applications / Making faces by Guenter B. (1993)
10.1145/1477926.1477927
10.5555/1272690.1272712
10.1109/34.216724
10.1145/1618452.1618521
10.1145/1778765.1778769
10.1007/s00371-005-0291-5
10.1109/TVCG.2010.23
Lu P. Nocedal J. Zhu C. Byrd R. H. and Byrd R. H. 1994. A limited-memory algorithm for bound constrained optimization. SIAM Journal on Scientific Computing.
10.5555/2383847.2383873
McLachlan G. J. and Krishnan T. 1996. The EM Algorithm and Extensions. Wiley-Interscience.
10.1145/882262.882269
10.1145/1185657.1185856
Pighin, F., Szeliski, R., and Salesin, D. 1999. Resynthesizing facial animation through 3d model-based tracking. ICCV 1, 143--150. / ICCV / Resynthesizing facial animation through 3d model-based tracking by Pighin F. (1999)
Roberts S. 1959. Control chart tests based on geometric moving averages. In Technometrics 239250. (10.2307/1266443)
10.1111/1467-9868.00196
Tipping M. E. and Bishop C. M. 1999. Mixtures of probabilistic principal component analyzers. Neural Computation 11. 10.1162/089976699300016728 (10.1162/089976699300016728)
Viola P. and Jones M. 2001. Rapid object detection using a boosted cascade of simple features. In CVPR.
Weise T. Leibe B. and Gool L. V. 2008. Accurate and robust registration for in-hand modeling. In CVPR. (10.1109/CVPR.2008.4587832)
Weise T. Li H. Gool L. V. and Pauly M. 2009. Face/off: Live facial puppetry. In SCA. 10.1145/1599470.1599472 (10.1145/1599470.1599472)
Williams L. 1990. Performance-driven facial animation. In Comp. Graph. (Proc. SIGGRAPH 90). 10.1145/97879.97906 (10.1145/97879.97906)
10.1145/1731047.1731055
10.5555/1032634.1032919
10.1145/1015706.1015759

Dates

Type	When
Created	14 years ago (July 26, 2011, 10:17 a.m.)
Deposited	2 months, 1 week ago (June 18, 2025, 7:06 a.m.)
Indexed	3 weeks, 3 days ago (Aug. 2, 2025, 12:35 a.m.)
Issued	14 years, 1 month ago (July 1, 2011)
Published	14 years, 1 month ago (July 1, 2011)
Published Online	14 years, 1 month ago (July 25, 2011)
Published Print	14 years, 1 month ago (July 1, 2011)

Funders 1

Swiss National Science Foundation 10.13039/501100001711 Schweizerischer Nationalfonds zur Förderung der Wissenschaftlichen Forschung
Region: Europe
pri (Trusts, charities, foundations (both public and private))
Labels10
1. Schweizerischer Nationalfonds
2. Swiss National Science Foundation
3. Fonds National Suisse de la Recherche Scientifique
4. Fondo Nazionale Svizzero per la Ricerca Scientifica
5. Fonds National Suisse
6. Fondo Nazionale Svizzero
7. Schweizerische Nationalfonds
8. SNF
9. SNSF
10. FNS
Awards1
1. 20PA21L 129607

BibTeX

@article{Weise_2011, title={Realtime performance-based facial animation}, volume={30}, ISSN={1557-7368}, url={http://dx.doi.org/10.1145/2010324.1964972}, DOI={10.1145/2010324.1964972}, number={4}, journal={ACM Transactions on Graphics}, publisher={Association for Computing Machinery (ACM)}, author={Weise, Thibaut and Bouaziz, Sofien and Li, Hao and Pauly, Mark}, year={2011}, month=jul, pages={1–10} }

JSON

{
  "indexed": {
    "date-parts": [
      [
        2025,
        8,
        2
      ]
    ],
    "date-time": "2025-08-02T04:35:28Z",
    "timestamp": 1754109328428,
    "version": "3.41.0"
  },
  "reference-count": 40,
  "publisher": "Association for Computing Machinery (ACM)",
  "issue": "4",
  "license": [
    {
      "start": {
        "date-parts": [
          [
            2011,
            7,
            1
          ]
        ],
        "date-time": "2011-07-01T00:00:00Z",
        "timestamp": 1309478400000
      },
      "content-version": "vor",
      "delay-in-days": 0,
      "URL": "https://www.acm.org/publications/policies/copyright_policy#Background"
    }
  ],
  "funder": [
    {
      "DOI": "10.13039/501100001711",
      "name": "Swiss National Science Foundation",
      "doi-asserted-by": "publisher",
      "award": [
        "20PA21L 129607"
      ],
      "id": [
        {
          "id": "10.13039/501100001711",
          "id-type": "DOI",
          "asserted-by": "publisher"
        }
      ]
    }
  ],
  "content-domain": {
    "domain": [
      "dl.acm.org"
    ],
    "crossmark-restriction": true
  },
  "published-print": {
    "date-parts": [
      [
        2011,
        7
      ]
    ]
  },
  "abstract": "<jats:p>This paper presents a system for performance-based character animation that enables any user to control the facial expressions of a digital avatar in realtime. The user is recorded in a natural environment using a non-intrusive, commercially available 3D sensor. The simplicity of this acquisition device comes at the cost of high noise levels in the acquired data. To effectively map low-quality 2D images and 3D depth maps to realistic facial expressions, we introduce a novel face tracking algorithm that combines geometry and texture registration with pre-recorded animation priors in a single optimization. Formulated as a maximum a posteriori estimation in a reduced parameter space, our method implicitly exploits temporal coherence to stabilize the tracking. We demonstrate that compelling 3D facial dynamics can be reconstructed in realtime without the use of face markers, intrusive lighting, or complex scanning hardware. This makes our system easy to deploy and facilitates a range of new applications, e.g. in digital gameplay or social interactions.</jats:p>",
  "DOI": "10.1145/2010324.1964972",
  "type": "journal-article",
  "created": {
    "date-parts": [
      [
        2011,
        7,
        26
      ]
    ],
    "date-time": "2011-07-26T14:17:46Z",
    "timestamp": 1311689866000
  },
  "page": "1-10",
  "update-policy": "https://doi.org/10.1145/crossmark-policy",
  "source": "Crossref",
  "is-referenced-by-count": 329,
  "title": "Realtime performance-based facial animation",
  "prefix": "10.1145",
  "volume": "30",
  "author": [
    {
      "given": "Thibaut",
      "family": "Weise",
      "sequence": "first",
      "affiliation": [
        {
          "name": "EPFL"
        }
      ]
    },
    {
      "given": "Sofien",
      "family": "Bouaziz",
      "sequence": "additional",
      "affiliation": [
        {
          "name": "EPFL"
        }
      ]
    },
    {
      "given": "Hao",
      "family": "Li",
      "sequence": "additional",
      "affiliation": [
        {
          "name": "EPFL"
        }
      ]
    },
    {
      "given": "Mark",
      "family": "Pauly",
      "sequence": "additional",
      "affiliation": [
        {
          "name": "EPFL"
        }
      ]
    }
  ],
  "member": "320",
  "published-online": {
    "date-parts": [
      [
        2011,
        7,
        25
      ]
    ]
  },
  "reference": [
    {
      "key": "e_1_2_2_1_1",
      "doi-asserted-by": "publisher",
      "DOI": "10.1145/1667239.1667251"
    },
    {
      "key": "e_1_2_2_2_1",
      "doi-asserted-by": "publisher",
      "DOI": "10.1145/1778765.1778777"
    },
    {
      "key": "e_1_2_2_3_1",
      "doi-asserted-by": "publisher",
      "unstructured": "Black M. J. and Yacoob Y. 1995. Tracking and recognizing rigid and non-rigid facial motions using local parametric models of image motion. In ICCV 374--381.",
      "DOI": "10.5555/839277.840053"
    },
    {
      "key": "e_1_2_2_4_1",
      "doi-asserted-by": "publisher",
      "DOI": "10.1145/311535.311556"
    },
    {
      "key": "e_1_2_2_5_1",
      "doi-asserted-by": "publisher",
      "DOI": "10.1145/1198555.1198596"
    },
    {
      "key": "e_1_2_2_6_1",
      "doi-asserted-by": "publisher",
      "DOI": "10.1145/1778765.1778778"
    },
    {
      "key": "e_1_2_2_7_1",
      "doi-asserted-by": "publisher",
      "unstructured": "Chai J. X. Xiao J. and Hodgins J. 2003. Vision-based control of 3d facial animation. In SCA.",
      "DOI": "10.5555/846276.846304"
    },
    {
      "key": "e_1_2_2_8_1",
      "unstructured": "Chuang E. and Bregler C. 2002. Performance driven facial animation using blendshape interpolation. Tech. rep. Stanford University."
    },
    {
      "key": "e_1_2_2_9_1",
      "doi-asserted-by": "publisher",
      "DOI": "10.1109/34.927467"
    },
    {
      "key": "e_1_2_2_10_1",
      "doi-asserted-by": "publisher",
      "DOI": "10.5555/524467.796028"
    },
    {
      "key": "e_1_2_2_11_1",
      "doi-asserted-by": "publisher",
      "unstructured": "DeCarlo D. and Metaxas D. 1996. The integration of optical flow and deformable models with applications to human face shape and motion estimation. In CVPR.",
      "DOI": "10.5555/794190.794604"
    },
    {
      "key": "e_1_2_2_12_1",
      "doi-asserted-by": "publisher",
      "DOI": "10.1023/A:1008122917811"
    },
    {
      "key": "e_1_2_2_13_1",
      "doi-asserted-by": "crossref",
      "unstructured": "Ekman P. and Friesen W. 1978. Facial Action Coding System: A Technique for the Measurement of Facial Movement. Consulting Psychologists Press.",
      "DOI": "10.1037/t27734-000"
    },
    {
      "key": "e_1_2_2_14_1",
      "doi-asserted-by": "publisher",
      "DOI": "10.5555/791215.791488"
    },
    {
      "key": "e_1_2_2_15_1",
      "doi-asserted-by": "crossref",
      "unstructured": "Furukawa Y. and Ponce J. 2009. Dense 3d motion capture for human faces. In CVPR.",
      "DOI": "10.1109/CVPRW.2009.5206868"
    },
    {
      "key": "e_1_2_2_16_1",
      "doi-asserted-by": "publisher",
      "DOI": "10.1145/1015706.1015755"
    },
    {
      "key": "e_1_2_2_17_1",
      "first-page": "6",
      "article-title": "Making faces",
      "volume": "13",
      "author": "Guenter B.",
      "year": "1993",
      "unstructured": "Guenter, B., Grimm, C., Wood, D., Malvar, H., and Pighin, F. 1993. Making faces. IEEE Computer Graphics and Applications 13, 6--8.",
      "journal-title": "IEEE Computer Graphics and Applications"
    },
    {
      "key": "e_1_2_2_18_1",
      "doi-asserted-by": "publisher",
      "DOI": "10.1145/1477926.1477927"
    },
    {
      "key": "e_1_2_2_19_1",
      "doi-asserted-by": "publisher",
      "DOI": "10.5555/1272690.1272712"
    },
    {
      "key": "e_1_2_2_20_1",
      "doi-asserted-by": "publisher",
      "DOI": "10.1109/34.216724"
    },
    {
      "key": "e_1_2_2_21_1",
      "doi-asserted-by": "publisher",
      "DOI": "10.1145/1618452.1618521"
    },
    {
      "key": "e_1_2_2_22_1",
      "doi-asserted-by": "publisher",
      "DOI": "10.1145/1778765.1778769"
    },
    {
      "key": "e_1_2_2_23_1",
      "doi-asserted-by": "publisher",
      "DOI": "10.1007/s00371-005-0291-5"
    },
    {
      "key": "e_1_2_2_24_1",
      "doi-asserted-by": "publisher",
      "DOI": "10.1109/TVCG.2010.23"
    },
    {
      "key": "e_1_2_2_25_1",
      "unstructured": "Lu P. Nocedal J. Zhu C. Byrd R. H. and Byrd R. H. 1994. A limited-memory algorithm for bound constrained optimization. SIAM Journal on Scientific Computing."
    },
    {
      "key": "e_1_2_2_26_1",
      "doi-asserted-by": "publisher",
      "DOI": "10.5555/2383847.2383873"
    },
    {
      "key": "e_1_2_2_27_1",
      "unstructured": "McLachlan G. J. and Krishnan T. 1996. The EM Algorithm and Extensions. Wiley-Interscience."
    },
    {
      "key": "e_1_2_2_28_1",
      "doi-asserted-by": "publisher",
      "DOI": "10.1145/882262.882269"
    },
    {
      "key": "e_1_2_2_29_1",
      "doi-asserted-by": "publisher",
      "DOI": "10.1145/1185657.1185856"
    },
    {
      "key": "e_1_2_2_30_1",
      "first-page": "143",
      "article-title": "Resynthesizing facial animation through 3d model-based tracking",
      "volume": "1",
      "author": "Pighin F.",
      "year": "1999",
      "unstructured": "Pighin, F., Szeliski, R., and Salesin, D. 1999. Resynthesizing facial animation through 3d model-based tracking. ICCV 1, 143--150.",
      "journal-title": "ICCV"
    },
    {
      "key": "e_1_2_2_31_1",
      "doi-asserted-by": "crossref",
      "unstructured": "Roberts S. 1959. Control chart tests based on geometric moving averages. In Technometrics 239250.",
      "DOI": "10.2307/1266443"
    },
    {
      "key": "e_1_2_2_32_1",
      "doi-asserted-by": "publisher",
      "DOI": "10.1111/1467-9868.00196"
    },
    {
      "key": "e_1_2_2_33_1",
      "doi-asserted-by": "publisher",
      "unstructured": "Tipping M. E. and Bishop C. M. 1999. Mixtures of probabilistic principal component analyzers. Neural Computation 11. 10.1162/089976699300016728",
      "DOI": "10.1162/089976699300016728"
    },
    {
      "key": "e_1_2_2_34_1",
      "unstructured": "Viola P. and Jones M. 2001. Rapid object detection using a boosted cascade of simple features. In CVPR."
    },
    {
      "key": "e_1_2_2_35_1",
      "doi-asserted-by": "crossref",
      "unstructured": "Weise T. Leibe B. and Gool L. V. 2008. Accurate and robust registration for in-hand modeling. In CVPR.",
      "DOI": "10.1109/CVPR.2008.4587832"
    },
    {
      "key": "e_1_2_2_36_1",
      "doi-asserted-by": "publisher",
      "unstructured": "Weise T. Li H. Gool L. V. and Pauly M. 2009. Face/off: Live facial puppetry. In SCA. 10.1145/1599470.1599472",
      "DOI": "10.1145/1599470.1599472"
    },
    {
      "key": "e_1_2_2_37_1",
      "doi-asserted-by": "publisher",
      "unstructured": "Williams L. 1990. Performance-driven facial animation. In Comp. Graph. (Proc. SIGGRAPH 90). 10.1145/97879.97906",
      "DOI": "10.1145/97879.97906"
    },
    {
      "key": "e_1_2_2_38_1",
      "doi-asserted-by": "publisher",
      "DOI": "10.1145/1731047.1731055"
    },
    {
      "key": "e_1_2_2_39_1",
      "doi-asserted-by": "publisher",
      "DOI": "10.5555/1032634.1032919"
    },
    {
      "key": "e_1_2_2_40_1",
      "doi-asserted-by": "publisher",
      "DOI": "10.1145/1015706.1015759"
    }
  ],
  "container-title": "ACM Transactions on Graphics",
  "original-title": [],
  "language": "en",
  "link": [
    {
      "URL": "https://dl.acm.org/doi/10.1145/2010324.1964972",
      "content-type": "unspecified",
      "content-version": "vor",
      "intended-application": "text-mining"
    },
    {
      "URL": "https://dl.acm.org/doi/pdf/10.1145/2010324.1964972",
      "content-type": "unspecified",
      "content-version": "vor",
      "intended-application": "similarity-checking"
    }
  ],
  "deposited": {
    "date-parts": [
      [
        2025,
        6,
        18
      ]
    ],
    "date-time": "2025-06-18T11:06:23Z",
    "timestamp": 1750244783000
  },
  "score": 1,
  "resource": {
    "primary": {
      "URL": "https://dl.acm.org/doi/10.1145/2010324.1964972"
    }
  },
  "subtitle": [],
  "short-title": [],
  "issued": {
    "date-parts": [
      [
        2011,
        7
      ]
    ]
  },
  "references-count": 40,
  "journal-issue": {
    "issue": "4",
    "published-print": {
      "date-parts": [
        [
          2011,
          7
        ]
      ]
    }
  },
  "alternative-id": [
    "10.1145/2010324.1964972"
  ],
  "URL": "http://dx.doi.org/10.1145/2010324.1964972",
  "relation": {},
  "ISSN": [
    "0730-0301",
    "1557-7368"
  ],
  "subject": [],
  "container-title-short": "ACM Trans. Graph.",
  "published": {
    "date-parts": [
      [
        2011,
        7
      ]
    ]
  },
  "assertion": [
    {
      "value": "2011-07-25",
      "order": 3,
      "name": "published",
      "label": "Published",
      "group": {
        "name": "publication_history",
        "label": "Publication History"
      }
    }
  ]
}