More Web Proxy on the site http://driver.im/

research-article

DECORAIT - DECentralized Opt-in/out Registry for AI Training

Authors:

Andrew Gilbert,

Alexander Black,

John CollomosseAuthors Info & Claims

CVMP '23: Proceedings of the 20th ACM SIGGRAPH European Conference on Visual Media Production

Article No.: 4, Pages 1 - 10

https://doi.org/10.1145/3626495.3626506

Published: 30 November 2023 Publication History

Abstract

We present DECORAIT; a decentralized registry through which content creators may assert their right to opt in or out of AI training and receive rewards for their contributions. Generative AI (GenAI) enables images to be synthesized using AI models trained on vast amounts of data scraped from public sources. Model and content creators who may wish to share their work openly without sanctioning its use for training are thus presented with a data governance challenge. Further, establishing the provenance of GenAI training data is important to creatives to ensure fair recognition and reward for their such use. We report a prototype of DECORAIT, which explores hierarchical clustering and a combination of on/off-chain storage to create a scalable decentralized registry to trace the provenance of GenAI training data to determine training consent and reward creatives who contribute that data. DECORAIT combines distributed ledger technology (DLT) with visual fingerprinting, leveraging the emerging C2PA (Coalition for Content Provenance and Authenticity) standard to create a secure, open registry through which creatives may express consent and data ownership for GenAI.

References

[1]

J. Aythora 2020. Multi-stakeholder Media Provenance Management to Counter Synthetic Media Risks in News Publishing. In Proc. Intl. Broadcasting Convention (IBC).

[2]

K. Balan, S. Agarwal, S. Jenni, A. Parsons, A. Gilbert, and J. Collomosse. 2023. EKILA: Synthetic Media Provenance and Attribution for Generative Art. In Proc. CVPR Workshop on Media Forensics.

[3]

J. Benet. 2014. IPFS - Content Addressed, Versioned, P2P File System. arxiv:1407.3561 [cs.NI]

[4]

A. Bharati, D. Moreira, P. Flynn, A. de Rezende Rocha, K. Bowyer, and W. Scheirer. 2021. Transformation-Aware Embeddings for Image Provenance. IEEE Trans. Info. Forensics and Sec. 16 (2021), 2493–2507.

[5]

S. Bhujel and Y. Rahulamathavan. 2022. A Survey: Security, Transparency, and Scalability Issues of NFT’s and Its Marketplaces. J. Sensors. MDPI. 22, 22 (2022).

[6]

A. Black, T. Bui, H. Jin, V. Swaminathan, and J. Collomosse. 2021. Deep Image Comparator: Learning To Visualize Editorial Change. In Proc. CVPR Workshop on Media Forensics. 972–980.

[7]

T. Bui, S. Agarwal, N. Yu, and J. Collomosse. 2023. RoSteALS: Robust Steganography using Autoencoder Latent Space. In Proc. CVPR WS.

[8]

T. Bui, D. Cooper, J. Collomosse, M. Bell, A. Green, J. Sheridan, J. Higgins, A. Das, J. Keller, and O. Thereaux. 2020. Tamper-proofing Video with Hierarchical Attention Autoencoder Hashing on Blockchain. IEEE Trans. Multimedia (TMM) 22, 11 (2020), 2858–2872.

[9]

T. Bui, D. Cooper, J. Collomosse, M. Bell, A. Green, J. Sheridan, J. Higgins, A. Das, J. Keller, O. Thereaux, and A. Brown. [n. d.]. ARCHANGEL: Tamper-proofing Video Archives using Temporal Content Hashes on the Blockchain. In Proc. CVPR Workshop on Computer Vision, AI and Blockchain, year = 2019.

[10]

N. Carlini, J. Hayes, M. Nasr, M. Jagielski, V. Sehwag, F. Tramèr, B. Balle, D. Ippolito, and E. Wallace. 2023. Extracting Training Data from Diffusion Models. arxiv:2301.13188 [cs.CR]

[11]

T. Chen, S. Kornblith, M. Norouzi, and G. Hinton. 2020. A simple framework for contrastive learning of visual representations. In International conference on machine learning. PMLR, 1597–1607.

[12]

Coalition for Content Provenance and Authenticity. 2021. Technical Specification v1.3. Technical Report. C2PA. https://c2pa.org/

[13]

J. Collomosse, T. Bui, A. Brown, J. Sheridan, A. Green, M. Bell, J. Fawcett, J. Higgins, and O. Thereaux. 2018. ARCHANGEL: Trusted Archives of Digital Public Documents. In Proc. ACM Doc.Eng.

[14]

P. Meenakshi Devi, M. Venkatesan, and K. Duraiswamy. 2019. A Fragile Watermarking scheme for Image Authentication with Tamper Localization Using Integer Wavelet transform. J. Computer Science 5, 11 (2019), 831–837.

[15]

P. Dhariwal and A. Nichol. 2021. Diffusion models beat GANs on image synthesis. NeuIPS 34 (2021), 8780–8794.

[16]

J. Fairfield. 2021. Tokenized: The Law of Non-Fungible Tokens and Unique Digital Property. Indiana Law Journal (2021).

[17]

H. Gilbert and H. Handschuh. 2003. Security Analysis of SHA-256 and Sisters. In Proc. Selected Areas in Cryptography (SAC).

[18]

J. D. Harris and B. Waggoner. 2019. Decentralized and Collaborative AI on Blockchain. In IEEE Intl. Conf. on Blockchain. IEEE.

[19]

D. Hendrycks and T. Dietterich. 2019. Benchmarking Neural Network Robustness to Common Corruptions and Perturbations. Proceedings of the International Conference on Learning Representations (2019).

[20]

J. Ho, A. Jain, and P. Abbeel. 2020. Denoising diffusion probabilistic models. NIPS 33 (2020), 6840–6851.

[21]

C. Holmes. 2018. Distributed ledger technologies for public good. Tech. Rep., UK Gov. Office for Science 1 (2018), 1–33.

[22]

N. Kumari, B. Zhang, R. Zhang, E. Shechtman, and J. Zhu. 2023. Multi-Concept Customization of Text-to-Image Diffusion. arxiv:2212.04488 [cs.CV]

[23]

V. L. Lemieux. 2016. Blockchain Technology for Recordkeeping: Help or Hype?Technical Report. U. British Columbia.

[24]

S. Nakamoto. 2008. Bitcoin: A peer-to-peer electronic cash system. http://www.bitcoin.org/bitcoin.pdf

[25]

A. Narayanan, J. Bonneau, E. Felten, A. Miller, and S. Goldfeder. 2016. Bitcoin and Cryptocurrency Technologies: A Comprehensive Introduction. U. Princeton.

[26]

E. Nguyen, T. Bui, V. Swaminathan, and J. Collomosse. 2021. OSCAR-Net: Object-centric Scene Graph Attention for Image Attribution. In Proc. ICCV.

[27]

OpenAI. [n. d.]. Introducing Chat-GPT. https://openai.com/blog/chatgpt. Accessed: 2023.

[28]

D. Podell, Z. English, K. Lacey, A. Blattmann, T. Dockhorn, J. Müller, J. Penna, and R. Rombach. 2023. SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis. arxiv:2307.01952 [cs.CV]

[29]

A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen. 2022. Hierarchical Text-Conditional Image Generation with CLIP Latents. https://doi.org/10.48550/ARXIV.2204.06125

[30]

R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. 2021. High-Resolution Image Synthesis with Latent Diffusion Models. arxiv:2112.10752 [cs.CV]

[31]

L. Rosenthol, A. Parsons, E. Scouten, J. Aythora, B. MacCormack, P. England, M. Levallee, J. Dotan, 2020. Content Authenticity Initiative (CAI): Setting the Standard for Content Attribution. Technical Report. Adobe Inc.

[32]

N. Ruiz, Y. Li, V. Jampani, Y. Pritch, M. Rubinstein, and K. Aberman. 2022. DreamBooth: Fine Tuning Text-to-image Diffusion Models for Subject-Driven Generation. (2022).

[33]

C. Saharia, W. Chan, S. Saxena, L. Li, J. Whang, E. Denton, S. Kamyar Ghasemipour, B. Karagol Ayan, S. Mahdavi, R. Gontijo Lopes, T. Salimans, J. Ho, D. J. Fleet, and M. Norouzi. 2022. Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding. arXiv preprint arXiv:2205.11487 (2022).

[34]

C. Schuhmann, R. Vencu, R. Beaumont,. Kaczmarczyk, C. Mullis, A. Katta, T. Coombes, J. Jitsev, and A. Komatsuzaki. 2021. LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs. arxiv:2111.02114 [cs.CV]

[35]

J. Shi, W. Xiong, Z. Lin, and H. Joon Jung. 2023. InstantBooth: Personalized Text-to-Image Generation without Test-Time Finetuning. arxiv:2304.03411 [cs.CV]

[36]

G. Somepalli, V. Singla, M. Goldblum, J. Geiping, and T. Goldstein. 2022. Diffusion Art or Digital Forgery? Investigating Data Replication in Diffusion Models. arxiv:2212.03860 [cs.LG]

[37]

G. Somepalli, V. Singla, M. Goldblum, J. Geiping, and T. Goldstein. 2023. Understanding and Mitigating Copying in Diffusion Models. arxiv:2305.20086 [cs.LG]

[38]

Stablity.ai. [n. d.]. Stable Diffusion Public Release. https://stability.ai/blog/stable-diffusion-public-release. Accessed: 2023.

[39]

G. Tolias, R. Sicre, and H. Jégou. 2015. Particular object retrieval with integral max-pooling of CNN activations. arXiv preprint arXiv:1511.05879 (2015).

[40]

M. Walport. 2015. Distributed Ledgers: Beyond Blockchain. Technical Report. UK Government.

[41]

N. Yu, V. Skripniuk, D. Chen, L. Davis, and M. Fritz. 2021. Responsible Disclosure of Generative Models Using Scalable Fingerprinting. In Proc. ICLR.

Cited By

Balan KGilbert ACollomosse J(2024)PDFed: Privacy-Preserving and Decentralized Asynchronous Federated Learning for Diffusion ModelsProceedings of 21st ACM SIGGRAPH Conference on Visual Media Production10.1145/3697294.3697306(1-9)Online publication date: 18-Nov-2024
https://dl.acm.org/doi/10.1145/3697294.3697306
Liddell FTallyn EMorgan EBalan KDisley MKoterwas TDixon BMoruzzi CCollomosse JElsden C(2024)ORAgen: Exploring the Design of Attribution through Media TokenisationCompanion Publication of the 2024 ACM Designing Interactive Systems Conference10.1145/3656156.3663693(229-233)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1145/3656156.3663693
Collomosse JParsons A(2024)To Authenticity, and Beyond! Building Safe and Fair Generative AI Upon the Three Pillars of ProvenanceIEEE Computer Graphics and Applications10.1109/MCG.2024.338016844:3(82-90)Online publication date: May-2024
https://doi.org/10.1109/MCG.2024.3380168

Index Terms

DECORAIT - DECentralized Opt-in/out Registry for AI Training

Recommendations

Integrating Content Authenticity with DASH Video Streaming
MMSys '24: Proceedings of the 15th ACM Multimedia Systems Conference

The importance of content authenticity and provenance has significantly increased in the digital era, due to the rampant spread of misinformation, which makes it necessary to build safe and trustworthy systems. To this effect, the Coalition for Content ...
Multi-label co-training
IJCAI'18: Proceedings of the 27th International Joint Conference on Artificial Intelligence

Multi-label learning aims at assigning a set of appropriate labels to multi-label samples. Although it has been successfully applied in various domains in recent years, most multi-label learning methods require sufficient labeled training samples, ...
Tri-Training: Exploiting Unlabeled Data Using Three Classifiers

In many practical data mining applications, such as Web page classification, unlabeled training examples are readily available, but labeled ones are fairly expensive to obtain. Therefore, semi-supervised learning algorithms such as co-training have ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Other conferences

CVMP '23: Proceedings of the 20th ACM SIGGRAPH European Conference on Visual Media Production

November 2023

112 pages

ISBN:9798400704260

DOI:10.1145/3626495

Editors:
Marco Volino
University of Surrey, UK
,
Armin Mustafa
University of Surrey, UK
,
Peter Vangorp
Utrecht University, Netherlands

Copyright © 2023 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 30 November 2023

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Badges

Best Paper

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

CVMP '23

CVMP '23: European Conference on Visual Media Production

November 30 - December 1, 2023

London, United Kingdom

Acceptance Rates

Overall Acceptance Rate 40 of 67 submissions, 60%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
88
Total Downloads

Downloads (Last 12 months)84
Downloads (Last 6 weeks)8

Reflects downloads up to 12 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Balan KGilbert ACollomosse J(2024)PDFed: Privacy-Preserving and Decentralized Asynchronous Federated Learning for Diffusion ModelsProceedings of 21st ACM SIGGRAPH Conference on Visual Media Production10.1145/3697294.3697306(1-9)Online publication date: 18-Nov-2024
https://dl.acm.org/doi/10.1145/3697294.3697306
Liddell FTallyn EMorgan EBalan KDisley MKoterwas TDixon BMoruzzi CCollomosse JElsden C(2024)ORAgen: Exploring the Design of Attribution through Media TokenisationCompanion Publication of the 2024 ACM Designing Interactive Systems Conference10.1145/3656156.3663693(229-233)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1145/3656156.3663693
Collomosse JParsons A(2024)To Authenticity, and Beyond! Building Safe and Fair Generative AI Upon the Three Pillars of ProvenanceIEEE Computer Graphics and Applications10.1109/MCG.2024.338016844:3(82-90)Online publication date: May-2024
https://doi.org/10.1109/MCG.2024.3380168

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Table of Contents