Shuai Tan
·
Biao Gong†
·
Xiang Wang
·
Shiwei Zhang
Dandan Zheng
·
Ruobing Zheng
·
Kecheng Zheng
·
Jingdong Chen
·
Ming Yang
Ant Group | Tongyi Lab
This repository is the official implementation of paper "Animate-X: Universal Character Image Animation with Enhanced Motion Representation". Animate-X is a universal animation framework based on latent diffusion models for various character types (collectively named X), including anthropomorphic characters.
|
|
- [2025.02.11] 🔥 Animate-X has been accepted by ICLR 2025!
- [2024.12.20] 🔥 We release our Animate-X inference codes.
- [2024.12.10] 🔥 We release our Animate-X CKPT checkpoints.
- [2024.10.14] 🔥 Our paper is in public on arxiv.
animatex_intro.mp4 |
animatex_comp.mp4 |
animatex_teaser.mp4 |
animatex_longvideo.mp4 |
Install with conda:
conda env create -f environment.yaml
conda activate animate-xDownload Animate-X checkpoints and put all files in checkpoints dir, which should be like:
./checkpoints/
|---- animate-x.pth
|---- dw-ll_ucoco_384.onnx
|---- open_clip_pytorch_model.bin
|---- v2-1_512-ema-pruned.ckpt
└---- yolox_l.onnx
The default inputs are a image (.jpg) and a dance video (.mp4). The default output is a 32-frame video (.mp4) with 768x512 resolution, which will be saved in ./results dir.
- pre-process the video.
python process_data.py \ --source_video_paths data/videos \ --saved_pose_dir data/saved_pkl \ --saved_pose data/saved_pose \ --saved_frame_dir data/saved_frames - run Animate-X.
python inference.py --cfg configs/Animate_X_infer.yaml
Some key parameters in the .yaml configuration file are described as follows. For example, users can adjust the max_frames or sampling interval of the dance video to generate videos of varying durations or speeds.
-
max_frames: Number of frames (default as 32) in the generated video (fps: 8).- If you want to generage longer video with more frames, you should modify
-
max_framesas the number of frames -
seq_leninUNetas the number of frames + 1
-
- We take 96 frames as an example, and the config should be:
{ max_frames: 96 # 1. modify `max_frames` as the number of frames (e.g. 96) ...... UNet: { ...... 'use_sim_mask': False, 'seq_len': 97, # 2. modify `seq_len` in `UNet` as the number of frames + 1 (e.g. 97 = 96 + 1) } }
- If you want to generage longer video with more frames, you should modify
-
round: The number of times each test case is generated. -
test_list_path: The input paths for all test cases, for example:[ [2, "data/images/1.jpg", "data/saved_pose/dance_1","data/saved_frames/dance_1","data/saved_pkl/dance_1.pkl", 14], [2, "data/images/4.png", "data/saved_pose/dance_1","data/saved_frames/dance_1","data/saved_pkl/dance_1.pkl", 14], ...... ]-
2indicates that 1 frame is sampled from every 2 frames of the reference dance video to be used as input for the model. -
"data/images/1.jpg"indicates the path to the reference image. -
"data/saved_pose/dance_1"indicates the path to the saved pose images. (output byprocess_data.py,$I^p$ , keypoints visualization) -
"data/saved_frames/dance_1"indicates the path to the saved frames from the driven video. (output byprocess_data.py) -
"data/saved_pkl/dance_1.pkl"indicates the path to the saved pose keypoints. (output byprocess_data.py,$p^d$ , DWPose) -
14indicates the random seed.
-
-
log_dir: path to the generated animation videos, e.g.,./results.
✔ Some tips:
Although Animate-x does not rely on strict pose alignment and we did not perform any manual alignment operations for all the results in the paper, we cannot guarantee that all cases are perfect. Therefore, users can perform handmade pose alignment operations themselves, e.g, applying the overall x/y translation and scaling on the pose skeleton of each frame to align with the position of the subject in the reference image. (put in
data/saved_pose)
Our implementation is based on UniAnimate, MimicMotion, and MusePose. Thanks for their remarkable contribution and released code! If we missed any open-source projects or related articles, we would like to complement the acknowledgement of this specific work immediately.
This repository is released under the Apache-2.0 license as found in the LICENSE file.
If you find this codebase useful for your research, please use the following entry.
@article{AnimateX2025,
title={Animate-X: Universal Character Image Animation with Enhanced Motion Representation},
author={Tan, Shuai and Gong, Biao and Wang, Xiang and Zhang, Shiwei and Zheng, Dandan and Zheng, Ruobing and Zheng, Kecheng and Chen, Jingdong and Yang, Ming},
journal={ICLR 2025},
year={2025}
}
@article{Mimir2025,
title={Mimir: Improving Video Diffusion Models for Precise Text Understanding},
author={Tan, Shuai and Gong, Biao and Feng, Yutong and Zheng, Kecheng and Zheng, Dandan and Shi, Shuwei and Shen, Yujun and Chen, Jingdong and Yang, Ming},
journal={arXiv preprint arXiv:2412.03085},
year={2025}
}