This repository provides code for the GEM pipeline. It includes a high-quality regressor, which can be thought of as a self-contained Gaussian Avatar, as it is fully controllable using the FLAME model. The output from this regressor is then compressed into the GEM model. You can use either the CNN regressor or the distilled GEM version of it in your experiments.
To install GEM, first clone the repository and then create the environment using the following commands:
git clone --recursive https://github.com/Zielon/GEM.git
cd GEM
After the repository is cloned with all the submodules, we can create the environment and install the libraries using install.sh.
yes Y | conda create -n gem python=3.9
conda activate gem
./install.sh
We used VHAP for tracking the Nersemble dataset. Importantly, our pipeline requires removing garments and the neck. Follow the render_masks.py file to render an alpha mask without the neck. To remove garments, you can use an off-the-shelf segmentation network like Face Parsing.
We used FLAME2023 and FLAME2020 in our experiments. Follow the download instructions and place the model under:
gem/data/FLAME2023/flame2023.pkl
gem/lib/F3DMM/FLAME2023/assets/flame2023.pkl
gem/lib/F3DMM/FLAME2020/assets/FLAME_2020.pkl
We additionally release all trained models for three actors. You can simply use the script ./checkpoints.sh to automatically download them into the experiments folder. You will find all the checkpoints and the results of the test scripts there.
Our pipeline requires running a few scripts:
- Generating PCA for input meshes
- Training the Gaussian texture regressor
- Building GEM
- Training the mapping network for coefficients
See run.sh for reference how the pipeline is executed.
Below, you can see how each of these steps is executed.
conda activate gem
export VOL_TRACKER_PATH=$PWD# PCA for input meshes
python pca_mesh.py# Training the Gaussian texture regressor
python train.py configs/nersemble/253/default.yml
# Testing the Gaussian texture regressor
python test.py configs/nersemble/253/default.ymlSee GEM configuration.
# Building GEM
python pca_gauss.py configs/nersemble/253/default.yml
# Visualize obtained GEM
python pca_viewer.py configs/nersemble/253/default.yml# Training the mapping network for coefficients
python train.py configs/regressor/253/regressor_gem.yml
# Testing the mapping network for coefficients
python test.py configs/regressor/253/regressor_gem.yml
# Reeanacment
python test.py configs/regressor/253/regressor_gem.yml configs/regressor/306/regressor_gem.ymlIf you use this project in your research please cite GEM:
@proceedings{zielonka2025gem,
title={Gaussian Eigen Models for Human Heads},
author = {Zielonka, Wojciech and Bolkart, Timo and Beeler Thabo, and Thies, Justus},
booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
month = {June},
year={2025},
}