We present MagicColor, a diffusion-based framework for multi-instance sketch colorization. Previous methods can only achieve multi-instance sketch colorization step by step, which is time-consuming and inaccurate. In contrast, our method has the capability of coloring sketches while maintaining consistency, making multi-instance sketch colorization easier.
Given a set of reference instances and corresponding line art images, our approach enables coloring sketches while maintaining consistency across multiple instances. Compared to traditional methods, our approach significantly improves coloring efficiency.
conda create -n MagicColor python=3.8
pip install -r requirements.txt
git clone https://github.com/YinHan-Zhang/MagicColor.git
cd MagicColor
Use tools to automatically extract masks:
git clone https://github.com/IDEA-Research/Grounded-Segment-Anything.git
Install Grounded-Segment-Anything see here -> install guideline
mv automatic_label.py Grounded-Segment-Anything/
cd Grounded-Segment-Anything
mkdir ckpt
# move sam/ram/groundingdino weight to ckpt dir
Then, you can train the model on your dataset:
python automatic_label.py \
--config GroundingDINO/groundingdino/config/GroundingDINO_SwinT_OGC.py \
--ram_checkpoint ./ckpt/ram_swin_large_14m.pth \
--grounded_checkpoint ./ckpt/groundingdino_swint_ogc.pth \
--sam_checkpoint ./ckpt/sam_vit_h_4b8939.pth \
--data_dir ./data \
--output_dir ./data_res \
--box_threshold 0.15 \
--text_threshold 0.15 \
--iou_threshold 0.15 \
--device "cuda"
bash download.sh
Dataset Format:
data/
├── dir_name1
├──masks/
├── mask_1.png # instance mask
├── mask_2.png
├── ...
├── mask_n.png
├── dir_name1.jpg # origin image 1
├── dir_name2
├──masks/
├── mask_1.png # instance mask
├── mask_2.png
├── ...
├── mask_n.png
├── dir_name2.jpg # origin image 2
then,
cd scripts
bash multi_ref_train.sh
Dataset Format:
data/
├── dir_name
├──masks/
├── mask_1.png # reference mask
├── mask_2.png
├── ...
├── mask_n.png
├── dir_name_1.jpg # reference instance
├── dir_name_2.jpg
├── ...
├── dir_name_n.jpg
├── dir_name.jpg # sketch image
then,
cd scripts
bash multi_ref_infer.sh # modify input_data_dir
run the script:
cd inference
python gradio_app.py
Due to the limitation of computing resources and data, the amount of data for model training is limited. If you have enough computing resources, you can train the model yourself.
Thanks for the reference contributions of these works:
- MangaNinjia
- ColorizeDiffusion
- DreamBooth