Project Page | Paper | Video | Data
- [12/01/2024] Initial code release
- Release code of EgoGaussian
- Release 3DGS-ready egocentric data we processed from EPIC-KITCHENS, HOI4D, and EPIC Fields. Please also consider citing their great works if you use this subset 🤗
- Upload pre-trained checkpoints for quick evaluation and visualization
- EgoGaussian viewer
- Pipeline optimization
- Tutorial for running EgoGaussian on customized data
The setup should be very similar to the original 3D Gaussian Splatting except we used a modified version of differential gaussian rasterization with support of depth & alpha rendering. We will release the requirements.txt later.
The full EgoGaussian pipelie consists of 4 main stages corresponding to different scripts under trainers
- Static object & background initialization
- Coarse object pose estimation
- Fine-tuning object pose & shape
- Fine-tuning full dynamic scene
You can use the following script to run a full EgoGaussian pipeline from scratch on the provided data.
sbatch train.shYou can also skip the training and directly reproducing the results of Table 1 in our paper and videos on the webpage by running the following script with the checkpoints we provide.
DATA_TYPE=EK # or HOI
DATA_NAME=P03_03 # or Video0
RUN_NAME=full
python eval.py \
--source_path ${DATA_DIR}/${DATA_TYPE}/${DATA_NAME} \
--out_root ${OUT_DIR} \
--data_type ${DATA_TYPE} \
--video ${DATA_NAME} \
--run_name ${RUN_NAME} \Our implementation is heavily based on the original 3D Gaussian Splatting. We thank the authors for their revolutionary work and open-source contributions.
If you find our paper useful, please cite us:
@misc{zhang2024egogaussiandynamicsceneunderstanding,
title={EgoGaussian: Dynamic Scene Understanding from Egocentric Video with 3D Gaussian Splatting},
author={Daiwei Zhang and Gengyan Li and Jiajie Li and Mickaël Bressieux and Otmar Hilliges and Marc Pollefeys and Luc Van Gool and Xi Wang},
year={2024},
eprint={2406.19811},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2406.19811},
}