Code of the paper "Exploring Vision Transformers for 3D Human Motion-Language Models with Motion Patches" (CVPR 2024).
- Python 3.11
- PyTorch 2.0.1+
Using Poetry (Recommended)
poetry installconda create -n MoPa python=3.11
conda activate MoPa
conda install pytorch==2.0.1 torchvision==0.15.2 torchaudio==2.0.2 pytorch-cuda=11.8 -c pytorch -c nvidia
pip install -r requirements.txtDownload HumanML3D Dataset and KIT-ML Dataset from the repository of HumanML3D. Unzip and locate them in the data folder.
The whole directory should be look like this:
MotionPatches
│ README.md
│ requirements.txt
| ...
|
└───conf
└───scripts
└───...
│
└───data
└───HumanML3D
| └───new_joint_vecs
| └───new_joints
| └───...
│
└───KIT-ML
└───new_joint_vecs
└───new_joints
└───...
Then calculate the mean and variance of each dataset by:
python scripts/cal_mean_var.pyDownload pre-trained model from huggingface and put them in checkpoints/pretrained/.
Evaluate the model with HumanML3D via retrieval:
python scripts/test.py dataset=HumanML3D exp_name=pretrainedEvaluate the model with KIT-ML via retrieval:
python scripts/test.py dataset=KIT-ML exp_name=pretrainedUsing scripts/test_batch.py can get the result of small batches with 32 samples.
Train the model with HumanML3D:
python scripts/train.py dataset=HumanML3DTrain the model with KIT-ML:
python scripts/train.py dataset=KIT-MLEvaluate the model with HumanML3D via retrieval:
python scripts/test.py dataset=HumanML3DEvaluate the model with KIT-ML via retrieval:
python scripts/test.py dataset=KIT-MLUsing scripts/test_batch.py can get the result of small batches with 32 samples.
@InProceedings{yu2024exploring,
title={Exploring Vision Transformers for 3D Human Motion-Language Models with Motion Patches},
author={Yu, Qing and Tanaka, Mikihiro and Fujiwara, Kent},
booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
year={2024}
}
Additionally, this repository contains third-party software. Refer NOTICE.txt for more details and follow the terms and conditions of their use.