We propose Conceptrol, a training-free method that boosts zero-shot personalized image generation across Stable Diffusion / SDXL / FLUX without additional training / data / models.
[03/2025] Code and paper are publicly available.
TL;DR: Conceptrol is a free lunch that elicits the personalized ability of zero-shot adapter by transforming image condition to visual specification contrained by textual concept, even outperforming fine-tuning methods.
CLICK for the full abstract
Personalized image generation with text-to-image diffusion models generates unseen images based on reference image content. Zero-shot adapter methods such as IP-Adapter and OminiControl are especially interesting because they do not require test-time fine-tuning. However, they struggle to balance preserving personalized content and adherence to the text prompt. We identify a critical design flaw resulting in this performance gap: current adapters inadequately integrate personalization images with the textual descriptions. The generated images, therefore, replicate the personalized content rather than adhere to the text prompt instructions. Yet the base text-to-image has strong conceptual understanding capabilities that can be leveraged.We propose Conceptrol, a simple yet effective framework that enhances zero-shot adapters without adding computational overhead. Conceptrol constrains the attention of visual specification with a textual concept mask that improves subject-driven generation capabilities. It achieves as much as 89% improvement on personalization benchmarks over the vanilla IP-Adapter and can even outperform fine-tuning approaches such as Dreambooth LoRA. The source code will be released upon acceptance.
conda create -n conceptrol python=3.10
conda activate conceptrol
pip install -r requirements.txtpip install gradio
gradio gradio_src/app.py| Model Name | Link |
|---|---|
| Stable Diffusion 1.5 | stable-diffusion-v1-5/stable-diffusion-v1-5 |
| Realistic Vision V5.1 | SG161222/Realistic_Vision_V5.1_noVAE |
| Stable Diffusion XL-1024 | stabilityai/stable-diffusion-xl-base-1.0 |
| Animagine XL v4.0 | cagliostrolab/animagine-xl-4.0 |
| Realistic Vision XL V5.0 | SG161222/RealVisXL_V5.0 |
| FLUX-schnell | black-forest-labs/FLUX.1-schnell |
| Adapter Name | Link |
|---|---|
| IP-Adapter | h94/IP-Adapter |
| OminiControl | Yuanshi/OminiControl |
If you found this repository/our paper useful, please consider citing:
@article{he2025conceptrol,
title={Conceptrol: Concept Control of Zero-shot Personalized Image Generation},
author={Qiyuan He and Angela Yao},
journal={arXiv preprint arXiv:2503.06568},
year={2025}
}We thank the following repositories for their great work: