RAD: Realistic Anonymization of Images Using Stable Diffusion
Authors: 
Simon Malm, Viktor Rönnbäck, Amanda Håkansson, Minh-ha Le, Karol Wojtulewicz, 
Niklas CarlssonAuthors Info & Claims
Pages 193 - 211
Abstract
Many of the deep learning models currently driving the advancements in computer vision, expected to transform our society, require extensive training data. However, privacy regulations require explicit consent or anonymization of personal data, and traditional anonymization methods degrade data quality, thus hindering model performance. To address this challenge, we introduce the Realistic Anonymization using Diffusion (RAD) framework, which uses Stable Diffusion and ControlNet to produce high-quality synthetic images. RAD's three-step pipeline maintains contextual integrity and data utility, achieving superior image quality compared to previous GAN-based methods. We evaluated RAD's privacy preservation and data utility through face recognition accuracy, a segmentation task, and human assessment. RAD anonymized faces in 95.5% of cases, with high photo-realism ratings from human evaluators. Segmentation tasks on both original and anonymized images showed minimal performance drop, confirming RAD's high utility. Our analysis also identifies the strengths and weaknesses of using Stable Diffusion for full-body anonymization in various conditions. In summary, our work advances the understanding of high-utility anonymized data generation, and demonstrates that RAD can effectively balance privacy and utility.
References
[1]
Stability AI. [n.,d.]. Stable Diffusion Launch Announcement. https://stability.ai/news/stable-diffusion-announcement
[2]
Hamed Alqahtani, Manolya Kavakli-Thorne, and Gulshan Kumar. 2021. Applications of Generative Adversarial Networks (GANs): An Updated Review. Archives of Computational Methods in Engineering, Vol. 28, 2 (01 3 2021), 525--552. https://doi.org/10.1007/s11831-019-09388-y
[3]
Thangapavithraa Balaji, Patrick Blies, Georg Göri, Raphael Mitsch, Marcel Wasserer, and Torsten Schön. 2021. Temporally coherent video anonymization through GAN inpainting. CoRR, Vol. abs/2106.02328 (2021). https://doi.org/10.48550/arXiv.2106.02328
[4]
Pascal Birnstill, Daoyuan Ren, and Jürgen Beyerer. 2015. A User Study on Anonymization Techniques for Smart Video Surveillance. In Proceedings of the 12th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS). 1--6. https://doi.org/10.1109/AVSS.2015.7301805
[5]
Karla Brkic, Ivan Sikiric, Tomislav Hrkac, and Zoran Kalafatic. 2017. I Know That Person: Generative Full Body and Face De-identification of People in Images. In Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). 1319--1328. https://doi.org/10.1109/CVPRW.2017.173
[6]
Tim Brooks, Bill Peebles, Connor Homes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Wing Yin Ng, Ricky Wang, and Aditya Ramesh. 2024. Video generation models as world simulators. https://openai.com/research/video-generation-models-as-world-simulators
[7]
Nicolas Carlini, Jamie Hayes, Milad Nasr, Matthew Jagielski, Vikash Sehwag, Florian Tramer, Borja Balle, Daphne Ippolito, and Eric Wallace. 2023. Extracting training data from diffusion models. In USENIX Security Symposium. 5253--5270.
[8]
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. 2024. The Cityscapes Dataset for Semantic Urban Scene Understanding. In Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 3213--3223. https://doi.org/10.1109/CVPR.2016.350
[9]
Florinel-Alin Croitoru, Vlad Hondru, Radu Tudor Ionescu, and Mubarak Shah. 2023. Diffusion models in vision: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence (2023). https://doi.org/10.1109/TPAMI.2023.3261988
[10]
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2020. Generative adversarial networks. Commun. ACM, Vol. 63, 11 (10 2020), 139--144. https://doi.org/10.1145/3422622
[11]
Simon Hanisch, Julian Todt, Jose Patino, Nicholas Evans, and Thorsten Strufe. 2024. A False Sense of Privacy: Towards a Reliable Evaluation Methodology for the Anonymization of Biometric Data. Proceedings on Privacy Enhancing Technologies (PoPETs), Vol. 2024 (2024), 116--132. Issue 1. https://doi.org/10.56553/popets-2024-0008
[12]
Huggingface. [n.,d.]. Diffusers. https://huggingface.co/docs/diffusers/en/index
[13]
Håkon Hukkelås, Rudolf Mester, and Frank Lindseth. 2019. Deepprivacy: A generative adversarial network for face anonymization. In International Symposium on Visual Computing. Springer, 565--578. https://doi.org/10.1007/978--3-030--33720--9_44
[14]
Håkon Hukkelås and Frank Lindseth. 2022. DeepPrivacy2: Towards Realistic Full-Body Anonymization. In Proceedings of the 2022 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). https://doi.org/10.1109/WACV56688.2023.00138
[15]
Håkon Hukkelås and Frank Lindseth. 2023. Does Image Anonymization Impact Computer Vision Training?. In Proceedings of the 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). 140--150. https://doi.org/10.1109/CVPRW59228.2023.00019
[16]
Håkon Hukkelås and Frank Lindseth. 2024. Synthesizing Anyone, Anywhere, in Any Pose. In Proceedings of the 2024 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). 4023--4034. https://doi.org/10.1109/WACV57701.2024.00399
[17]
Håkon Hukkelås, Morten Smebye, Rudolf Mester, and Frank Lindseth. 2023. Realistic Full-Body Anonymization with Surface-Guided GANs. In Proceedings of the 2023 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). 1430--1440. https://doi.org/10.1109/WACV56688.2023.00148
[18]
Mohamed Khamis, Habiba Farzand, Marija Mumm, and Karola Marky. 2022. DeepFakes for Privacy: Investigating the Effectiveness of State-of-the-Art Privacy-Enhancing Face Obfuscation Methods. In Proceedings of the 2022 International Conference on Advanced Visual Interfaces (AVI). Article 21, 5 pages. https://doi.org/10.1145/3531073.3531125
[19]
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick. 2023. Segment Anything. In Proceedings of the 2023 IEEE/CVF International Conference on Computer Vision (ICCV). 3992--4003. https://doi.org/10.1109/ICCV51070.2023.00371
[20]
Marvin Klemp, Kevin Rösch, Royden Wagner, Jannik Quehl, and Martin Lauer. 2023. LDFA: Latent Diffusion Face Anonymization for Self-driving Applications. In Proceedings of the 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). 3199--3205. https://doi.org/10.1109/CVPRW59228.2023.00322
[21]
Kuno Kurzhals. 2023. Privacy in Eye Tracking Research with Stable Diffusion. In Proceedings of the 2023 Symposium on Eye Tracking Research and Applications (ETRA). Article 70, 7 pages. https://doi.org/10.1145/3588015.3589842
[22]
Minh Ha Le and Niklas Carlsson. 2023. StyleID: Identity Disentanglement for Anonymizing Faces. Proceedings on Privacy Enhancing Technologies (PoPETs), Vol. 2023 (2023), 264--278. Issue 1. https://doi.org/10.56553/popets-2023-0016
[23]
Yifang Li, Nishant Vishwamitra, Bart P. Knijnenburg, Hongxin Hu, and Kelly Caine. 2017. Blur vs. Block: Investigating the Effectiveness of Privacy-Enhancing Obfuscation for Images. In Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). 1343--1351. https://doi.org/10.1109/CVPRW.2017.176
[24]
Maxim Maximov, Ismail Elezi, and Laura Leal-Taixé. 2020. CIAGAN: Conditional Identity Anonymization Generative Adversarial Networks. In Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 5446--5455. https://doi.org/10.1109/CVPR42600.2020.00549
[25]
Ricardo Mendes and Jo ao P. Vilela. 2017. Privacy-preserving data mining: methods, metrics, and applications. IEEE Access, Vol. 5 (2017), 10562--10582. https://doi.org/10.1109/Access.2017.2706947
[26]
Anton Milan, Laura Leal-Taixé, Ian D. Reid, Stefan Roth, and Konrad Schindler. 2016. MOT16: A Benchmark for Multi-Object Tracking. CoRR, Vol. abs/1603.00831 (2016). https://doi.org/10.48550/arXiv.1603.00831
[27]
Council of European Union. [n.,d.]. Regulation (EU) 2016/679 of the European Parliament and of the Council. https://eur-lex.europa.eu/legal-content/EN/TXT/PDF/?uri=CELEX:32016R0679
[28]
Anastasiia Pika, Moe T. Wynn, Stephanus Budiono, Arthur H.M. ter Hofstede, Wil M.P. van der Aalst, and Hajo A. Reijers. 2020. Privacy-Preserving Process Mining in Healthcare. International Journal of Environmental Research and Public Health, Vol. 17, 5 (2020). https://doi.org/10.3390/ijerph17051612
[29]
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. 2023. SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis. arXiv preprint (2023). https://doi.org/10.48550/arXiv.2307.01952
[30]
Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. 2016. You Only Look Once: Unified, Real-Time Object Detection. In Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 779--788. https://doi.org/10.1109/CVPR.2016.91
[31]
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. 2022. High-Resolution Image Synthesis with Latent Diffusion Models. In Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 10674--10685. https://doi.org/10.1109/CVPR52688.2022.01042
[32]
Will Rowan, Patrik Huber, Nick Pears, and Andrew Keeling. 2023. Fake It Without Making It: Conditioned Face Generation for Accurate 3D Face Reconstruction. arXiv preprint (2023). https://doi.org/10.48550/arXiv.2307.13639
[33]
Sefik Ilkin Serengil and Alper Ozpinar. 2020. LightFace: A Hybrid Deep Face Recognition Framework. In Proceedings of the 2020 Innovations in Intelligent Systems and Applications Conference (ASYU). IEEE, 23--27. https://doi.org/10.1109/ASYU50717.2020.9259802
[34]
Zhaoyang Xia, Carol Neidle, and Dimitris N. Metaxas. 2023. DiffSLVA: Harnessing Diffusion Models for Sign Language Video Anonymization. arXiv preprint (2023). https://doi.org/10.48550/arXiv.2311.16060
[35]
Shuai Yang, Yifan Zhou, Ziwei Liu, and Chen Change Loy. 2023. Rerender A Video: Zero-Shot Text-Guided Video-to-Video Translation. In Proceedings of SIGGRAPH Asia 2023. Article 95, 11 pages. https://doi.org/10.1145/3610548.3618160
[36]
Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. 2023. Adding Conditional Control to Text-to-Image Diffusion Models. In Proceedings of the 2023 IEEE/CVF International Conference on Computer Vision (ICCV). 3813--3824. https://doi.org/10.1109/ICCV51070.2023.00355
[37]
Jingxing Zhou and Jürgen Beyerer. 2022. Impacts of Data Anonymization on Semantic Segmentation. In Proceedings of the 2022 IEEE Intelligent Vehicles Symposium (IV). 997--1004. https://doi.org/10.1109/IV51971.2022.9827262
Index Terms
- RAD: Realistic Anonymization of Images Using Stable Diffusion
Recommendations
IMR based Anonymization for Privacy Preservation in Data Mining
KMO '16: Proceedings of the The 11th International Knowledge Management in Organizations Conference on The changing face of Knowledge Management Impacting SocietyPrivacy Preserving Data Mining (PPDM) is a data mining research area that aims to protect individual's personal information from unsolicited or unauthorized disclosure. Privacy relates to personal information that a person would not wish others to know ...
An evolutionary feature set decomposition based anonymization for classification workloads: Privacy Preserving Data Mining
Privacy has become an important concern while publishing micro data about a population. The emerging area called privacy preserving data mining (PPDM) focus on individual privacy without compromising data mining results. An adversarial exploitation of ...
Identities Anonymization in Dynamic Social Networks
ICDM '11: Proceedings of the 2011 IEEE 11th International Conference on Data MiningPrivacy in social network data publishing is always an important concern. Nowadays most prior privacy protection techniques focus on static social networks. However, there are additional privacy disclosures in dynamic social networks due to the ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
November 2024
219 pages
Copyright © 2023 Owner/Author.
This work is licensed under a Creative Commons Attribution International 4.0 License.
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 21 November 2024
Check for updates
Author Tags
Qualifiers
- Research-article
Funding Sources
- Wallenberg AI, Autonomous Systems and Software Program (WASP)
Conference
CCS '24
Sponsor:
CCS '24: ACM SIGSAC Conference on Computer and Communications Security
October 14 - 18, 2024
UT, Salt Lake City, USA
Acceptance Rates
Overall Acceptance Rate 106 of 355 submissions, 30%
Upcoming Conference
CCS '25
- Sponsor:
- sigsac
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 54Total Downloads
- Downloads (Last 12 months)54
- Downloads (Last 6 weeks)54
Reflects downloads up to 12 Dec 2024
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in