This is a repository for organizing papers, codes and other resources related to the topic of Avatar (talking-face and talking-body).

If you have any suggestions (missing papers, new papers, key researchers or typos), please feel free to edit and pull a request.

• 2024.09.07: add ASR and TTS tool
• 2024.08.24: add backgrounds for image/video generations
• 2024.08.24: re-organize paper list with table formating
• 2024.08.24: add works about full-body avatar synthesis

• Main paper list
• Researchers list
• Toolbox for avatar
• Add paper link
• Add paper notes
• Add codes if have
• Add project page if have
• Datasets and metrics
• Related links

1. NVIDIA Research
   • Neural rendering models for human generation: vid2vid NeurIPS'18, fs-vid2vid NeurIPS'19, EG3D CVPR'22;
   • Talking-face synthesis: face-vid2vid CVPR'21, Implicit NeurIPS'22, SPACE ICCV'23, One-shot Neural Head Avatar arXiv'23;
   • Talking-body synthesis: DreamPose ICCV'23;
   • Face enhancement (relighting, restoration, etc): Lumos SIGGRAPH Asia 2022, RANA ICCV'23;
   • Authorized use of synthetic videos: Avatar Fingerprinting arXiv'23;
2. Aliaksandr Siarohin @ Snap Research
   • Neural rendering models for human generation (focus on flow-based generative models): Unsupervised-Volumetric-Animation CVPR'23, 3DAvatarGAN CVPR'23, 3D-SGAN ECCV'22, Articulated-Animation CVPR'21, Monkey-Net CVPR'19, FOMM NeurIPS'19;
3. Ziwei Liu @ Nanyang Technological University
   • Talking-face synthesis: StyleSync CVPR'23, AV-CAT SIGGRAPH Asia 2022, StyleGANX ICCV'23, StyleSwap ECCV'22, PC-AVS CVPR'21, Speech2Talking-Face IJCAI'21, VToonify SIGGRAPH Asia 2022;
   • Talking-body synthesis: MotionDiffuse arXiv'22;
   • Face enhancement (relighting, restoration, etc): Relighting4D ECCV'22;
4. Xiaodong Cun @ Tencent AI Lab:
   • Talking-face synthesis: StyleHEAT ECCV'22, VideoReTalking SIGGRAPH Asia'22, ToolTalking ICCV'23, DPE CVPR'23, CodeTalker CVPR'23, SadTalker CVPR'23;
   • Talking-body synthesis: LivelySpeaker ICCV'23;

5. Max Planck Institute for Informatics:
   • 3D face models (e.g., 3DMM): FLAME SIGGRAPH Asia 2017;

Conference	Paper	Affiliation	Codebase	Notes
CVPR 2021	Function4D: Real-time Human Volumetric Capture from Very Sparse Consumer RGBD Sensors	Tsinghua University	Dataset
ECCV 2022	HuMMan: Multi-Modal 4D Human Dataset for Versatile Sensing and Modeling	Shanghai Artificial Intelligence Laboratory	Dataset
SIGGRAPH 2023	AvatarReX: Real-time Expressive Full-body Avatars	Tsinghua University	Dataset
arXiv 2024	A Survey on 3D Human Avatar Modeling - From Reconstruction to Generation	The University of Hong Kong
arXiv 2024	From Audio to Photoreal Embodiment: Synthesizing Humans in Conversations	Meta Reality Labs Research	Code	conversational avatar
CVPR 2024	Animatable Gaussians: Learning Pose-dependent Gaussian Maps for High-fidelity Human Avatar Modeling	Tsinghua Univserity	Code
CVPR 2024	4K4D: Real-Time 4D View Synthesis at 4K Resolution	Zhejiang University	Code	real-time synthesis with 3DGS

Conference	Paper	Affiliation	Codebase	Training Code	Notes
MM 2020	Wav2Lip: Accurately Lip-sync Videos to Any Speech	The International Institute of Islamic Thought (IIIT), India	Code	✅	most accurate lip-sync model, bad video quality 96*96, pre-trained on ~180 hours video data from LRS2
MM 2021	Imitating Arbitrary Talking Style for Realistic Audio-Driven Talking Face Synthesis	Tsinghua University	Code,
CVPR 2021	Pose-Controllable Talking Face Generation by Implicitly Modularized Audio-Visual Representation	The Chinese University of Hong Kong	Code		contrastive learning on audio-lip
ICCV 2021	PIRenderer: Controllable Portrait Image Generation via Semantic Neural Rendering	Peking University	Code
ECCV 2022	StyleHEAT: One-Shot High-Resolution Editable Talking Face Generation via Pre-trained StyleGAN	Tsinghua University	Code		High-fidenity synthesis via StyleGAN
SIGGRAPH Asia 2022	VideoReTalking: Audio-based Lip Synchronization for Talking Head Video Editing In the Wild	Xidian University	Code
AAAI 2023	DINet: Deformation Inpainting Network for Realistic Face Visually Dubbing on High Resolution Video	Virtual Human Group, Netease Fuxi AI Lab	Code	✅	accurate lip-sync and high-quality synthesis (256*256)
CVPR 2023	SadTalker: Learning Realistic 3D Motion Coefficients for Stylized Audio-Driven Single Image Talking Face Animation	Xi'an Jiaotong University	Code , Note
arXiv 2023	DreamTalk: When Expressive Talking Head Generation Meets Diffusion Probabilistic Models	Tsinghua University	Code,		diffusion
		Tencent TMElyralab	MuseTalk: Real-Time High Quality Lip Synchorization with Latent Space Inpainting
arXiv 2024	LivePortrait: Efficient Portrait Animation with Stitching and Retargeting Control	Kuaishou Technology	Code		face reenactment with micro-expression
arXiv 2024	EchoMimic: Lifelike Audio-Driven Portrait Animations through Editable Landmark Conditions	Ant Group	Code		accurate lip-sync on Chinese speakers, diffusion, pre-trained on 540 hours cleaned video data (collected from internet)
arXiv 2024	Hallo: Hierarchical Audio-Driven Visual Synthesis for Portrait Image Animation	Fudan University	Code,	✅	accurate lip-sync, diffusion, pre-trained on 264 hours of cleaned video data (155 hours from internet and 9 hours from HDTF)
[arXiv 2024]	Loopy: Taming Audio-Driven Portrait Avatar with Long-Term Motion Dependency	Zhejiang University and ByteDance			expressive animation driven by audio only, pre-trained on 160 hours of cleaned video data (collected from internet)

Conference	Paper	Affiliation	Codebase	Notes
ICCV 2021	AD-NeRF: Audio Driven Neural Radiance Fields for Talking Head Synthesis	University of Science and Technology of China	Code
ECCV 2022	Learning Dynamic Facial Radiance Fields for Few-Shot Talking Head Synthesis	Tsinghua University	Code
ICLR 2023	GeneFace: Generalized and High-Fidelity Audio-Driven 3D Talking Face Synthesis	Zhejiang University	Code
ICCV 2023	Efficient Region-Aware Neural Radiance Fields for High-Fidelity Talking Portrait Synthesis	Beihang University	Code
arXiv 2023	GeneFace++: Generalized and Stable Real-Time Audio-Driven 3D Talking Face Generation	Zhejiang University	Code
CVPR 2024	SyncTalk: The Devil is in the Synchronization for Talking Head Synthesi	Renmin University of China	Code
ECCV 2024	TalkingGaussian: Structure-Persistent 3D Talking Head Synthesis via Gaussian Splatting	Beihang University	Code

Conference	Paper	Affiliation	Codebase	Notes
NeurIPS 2018	Video-to-Video Synthesis	NVIDIA	Code
ICCV 2019	Everybody Dance Now	UC Berkeley	Code
arXiv 2023	Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animation	Alibaba Group	Code
CVPR 2024	MagicAnimate: Temporally Consistent Human Image Animation using Diffusion Model	National University of Singapore	Code
arXiv 2024	Champ: Controllable and Consistent Human Image Animation with 3D Parametric Guidance	Nanjing University	Code
Github repo	MuseV: Infinite-length and High Fidelity Virtual Human Video Generation with Visual Conditioned Parallel Denoising	Tencent TMElyralab	Code
Github repo	MusePose: a Pose-Driven Image-to-Video Framework for Virtual Human Generation	Tencent	Code ⭐
arXiv 2024	ControlNeXt: Powerful and Efficient Control for Image and Video Generation	The Chinese University of Hong Kong	Code	stable video diffusion
[arXiv 2024]	CyberHost: Taming Audio-driven Avatar Diffusion Model with Region Codebook Attention	Zhejiang University and ByteDance		pre-trained on 200 hours video data and more than 10k unique identities

1. A general toolbox for AIGC, including common metrics and models https://github.com/open-mmlab/mmagic
2. face3d: Python tools for processing 3D face https://github.com/yfeng95/face3d
3. 3DMM model fitting using Pytorch https://github.com/ascust/3DMM-Fitting-Pytorch
4. OpenFace: a facial behavior analysis toolkit https://github.com/TadasBaltrusaitis/OpenFace
5. autocrop: Automatically detects and crops faces from batches of pictures https://github.com/leblancfg/autocrop
6. OpenPose: Real-time multi-person keypoint detection library for body, face, hands, and foot estimation https://github.com/CMU-Perceptual-Computing-Lab/openpose
7. GFPGAN: Practical Algorithm for Real-world Face Restoration https://github.com/TencentARC/GFPGAN
8. CodeFormer: Robust Blind Face Restoration https://github.com/sczhou/CodeFormer
9. metahuman-stream: Real time interactive streaming digital human https://github.com/lipku/metahuman-stream
10. EasyVolcap: a PyTorch library for accelerating neural volumetric video research https://github.com/zju3dv/EasyVolcap
11. 3D Model in gradio https://www.gradio.app/guides/how-to-use-3D-model-component

1. BELLE-2/Belle-whisper-large-v3-zh https://huggingface.co/BELLE-2/Belle-whisper-large-v3-zh
2. SenseVoice (multilingual) https://github.com/FunAudioLLM/SenseVoice 👍👍

1. CosyVoice, Alibaba Tongyi SpeechTeam https://github.com/FunAudioLLM/CosyVoice 👍👍
2. FireRedTTS, FireReadTeam https://github.com/FireRedTeam/FireRedTTS
3. GPT-SoVITS https://github.com/RVC-Boss/GPT-SoVITS?tab=readme-ov-file

1. Mini-Omni, Tsinghua University https://github.com/gpt-omni/mini-omni
2. Speech To Speech, HuggingFace https://github.com/huggingface/speech-to-speech

If you are interested in avatar and digital human, we would also like to recommend you to check out other related collections:

• awesome digital human https://github.com/weihaox/awesome-digital-human