Misinformation, Fraud, and Stereotyping: Towards a Typology of Harm Caused by Deepfakes
Pages 533 - 538
Abstract
Scholars, politicians, and journalists have raised alarm over the potential for AI-generated photos, video, and audio - often referred to as deepfakes - to reduce trust in one another and our institutions. Despite these clarion calls, little empirical work exists on how deepfakes are being used to harm individuals outside of non-consensual intimate imagery (NCII). This research provides a preliminary analysis of 50 wide-ranging incidents of deepfake harm. We find that the most common types of harm are relational, systemic, financial, and emotional. Apart from AI-generated NCII, the most prevalent uses of deepfakes to cause harm were instances of mis- and disinformation, fraud, and misrepresentation of or stereotyping about marginalized groups (e.g., women and racial minorities). We concluded with recommendations for future work and discuss potential challenges in identifying, quantifying, and preventing harm caused by deepfakes both online and off.
References
[1]
Shruti Agarwal, Hany Farid, Yuming Gu, Mingming He, Koki Nagano, and Hao Li. 2019. Protecting World Leaders Against Deep Fakes. In CVPR workshops, Vol. 1. 38.
[2]
Henry Ajder, Giorgio Patrini, Francesco Cavalli, and Laurence Cullen. 2019. The state of deepfakes: Landscape, threats, and impact. Amsterdam: Deeptrace, Vol. 27 (2019).
[3]
Michael Atleson. [n.,d.]. Chatbots, deepfakes, and voice clones: AI deception for sale. https://www.ftc.gov/business-guidance/blog/2023/03/chatbots-deepfakes-voice-clones-ai-deception-sale
[4]
Sander Beckers, Hana Chockler, and Joseph Y Halpern. 2022. Quantifying harm. arXiv preprint arXiv:2209.15111 (2022).
[5]
Virginia Braun and Victoria Clarke. 2012. Thematic analysis.American Psychological Association.
[6]
Luke J Chang, Bradley B Doll, Mascha vant Wout, Michael J Frank, and Alan G Sanfey. 2010. Seeing is believing: Trustworthiness as a dynamic belief. Cognitive psychology, Vol. 61, 2 (2010), 87--105.
[7]
Bobby Chesney and Danielle Citron. 2019. Deep fakes: A looming challenge for privacy, democracy, and national security. Calif. L. Rev., Vol. 107 (2019), 1753.
[8]
Danielle Keats Citron and Mary Anne Franks. 2014. Criminalizing revenge porn. Wake Forest L. Rev., Vol. 49 (2014), 345.
[9]
Arman Cohan, Sydney Young, Andrew Yates, and Nazli Goharian. 2017. Triaging content severity in online mental health forums. Journal of the Association for Information Science and Technology, Vol. 68, 11 (2017), 2675--2689.
[10]
Kevin Collier. [n.,d.]. Deepfake scams have arrived: Fake videos spread on Facebook, TikTok and Youtube. NBC News ( [n.,d.]). https://www.nbcnews.com/tech/tech-news/deepfake-scams-arrived-fake-videos-spread-facebook-tiktok-youtube-rcna101415
[11]
Surjit Singh Dhooper. 1991. Toward an effective response to the needs of Asian-Americans. Journal of Multicultural Social Work, Vol. 1, 2 (1991), 65--82.
[12]
Don Fallis. 2021. The epistemic threat of deepfakes. Philosophy & Technology, Vol. 34, 4 (2021), 623--643.
[13]
Emily Flitter and Stacy Cowley. [n.,d.]. Voice Deepfakes Are Coming for Your Bank Balance. ( [n.,d.]). https://www.nytimes.com/2023/08/30/business/voice-deepfakes-bank-scams.html
[14]
Tarleton Gillespie. 2020. Platforms throw content moderation at every problem. (2020).
[15]
Sam Gregory. 2022. Deepfakes, misinformation and disinformation and authenticity infrastructure responses: Impacts on frontline witnessing, distant witnessing, and civic journalism. Journalism, Vol. 23, 3 (2022), 708--729.
[16]
Jeffrey T Hancock and Jeremy N Bailenson. 2021. The social impact of deepfakes., 149--152 pages.
[17]
Joey Chiao-Yin Hsiao, Sylvia Darling, and Tawanna R Dillahunt. 2023. How Recent Migrants Develop Trust Through Community Commerce: The Emergence of Sociotechnical Adaptation. Proceedings of the ACM on Human-Computer Interaction, Vol. 7, CSCW1 (2023), 1--24.
[18]
Caroline Jack. 2017. Lexicon of lies: Terms for problematic information. (2017).
[19]
Nina Jankowicz. [n.,d.]. I Shouldnt Have to Accept Being in Deepfake Porn. https://www.theatlantic.com/ideas/archive/2023/06/deepfake-porn-ai-misinformation/674475/ Section: Ideas.
[20]
Jialun Aaron Jiang, Morgan Klaus Scheuerman, Casey Fiesler, and Jed R Brubaker. 2021. Understanding international perceptions of the severity of harmful content online. PloS one, Vol. 16, 8 (2021), e0256762.
[21]
Darin EW Johnson. 2019. Russian election interference and race-baiting. Colum. J. Race & L., Vol. 9 (2019), 191.
[22]
Deborah G Johnson and Nicholas Diakopoulos. 2021. What to do about deepfakes. Commun. ACM, Vol. 64, 3 (2021), 33--35.
[23]
Kristjan Kikerpill. 2020. Choose your stars and studs: the rise of deepfake designer porn. Porn Studies, Vol. 7, 4 (2020), 352--356.
[24]
Yiyi Li and Ying Xie. 2020. Is a picture worth a thousand words? An empirical study of image content and social media engagement. Journal of Marketing Research, Vol. 57, 1 (2020), 1--19.
[25]
Jacqueline Low. 2019. A pragmatic definition of the concept of theoretical saturation. Sociological focus, Vol. 52, 2 (2019), 131--139.
[26]
Alice E Marwick and Danah Boyd. 2011. I tweet honestly, I tweet passionately: Twitter users, context collapse, and the imagined audience. New media & society, Vol. 13, 1 (2011), 114--133.
[27]
Momina Masood, Mariam Nawaz, Khalid Mahmood Malik, Ali Javed, Aun Irtaza, and Hafiz Malik. 2023. Deepfakes generation and detection: State-of-the-art, open challenges, countermeasures, and way forward. Applied intelligence, Vol. 53, 4 (2023), 3974--4026.
[28]
Sean McGregor. 2021. Preventing repeated real world AI failures by cataloging incidents: The AI incident database. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 35. 15458--15463.
[29]
Pulak Mehta, Gauri Jagatap, Kevin Gallagher, Brian Timmerman, Progga Deb, Siddharth Garg, Rachel Greenstadt, and Brendan Dolan-Gavitt. 2023. Can Deepfakes Be Created on a Whim?. In Companion Proceedings of the ACM Web Conference 2023 (Austin, TX, USA) (WWW '23 Companion). Association for Computing Machinery, New York, NY, USA, 1324--1334. https://doi.org/10.1145/3543873.3587581
[30]
Yisroel Mirsky and Wenke Lee. 2021. The creation and detection of deepfakes: A survey. ACM Computing Surveys (CSUR), Vol. 54, 1 (2021), 1--41.
[31]
Sarah Nguyen, Rachel Kuo, Madhavi Reddi, Lan Li, and Rachel E Moran. 2022. Studying mis-and disinformation in Asian diasporic communities: The need for critical transnational research beyond Anglocentrism. Harvard Kennedy School Misinformation Review (2022).
[32]
George Onofrei, Raffaele Filieri, and Lorraine Kennedy. 2022. Social media interactions, purchase intention, and behavioural engagement: The mediating role of source and content factors. Journal of Business Research, Vol. 142 (2022), 100--112.
[33]
Daniel K Pryce and Joselyne L Chenane. 2021. Trust and confidence in police officers and the institution of policing: The views of African Americans in the American South. Crime & Delinquency, Vol. 67, 6--7 (2021), 808--838.
[34]
Alvaro Puig. [n.,d.]. Scammers use AI to enhance their family emergency schemes. https://consumer.ftc.gov/consumer-alerts/2023/03/scammers-use-ai-enhance-their-family-emergency-schemes
[35]
Morgan Klaus Scheuerman, Jialun Aaron Jiang, Casey Fiesler, and Jed R Brubaker. 2021. A framework of severity for harmful content online. Proceedings of the ACM on Human-Computer Interaction, Vol. 5, CSCW2 (2021), 1--33.
[36]
Shawn Shan, Jenna Cryan, Emily Wenger, Haitao Zheng, Rana Hanocka, and Ben Y Zhao. 2023. Glaze: Protecting artists from style mimicry by text-to-image models. arXiv preprint arXiv:2302.04222 (2023).
[37]
Renee Shelby, Shalaleh Rismani, Kathryn Henne, AJung Moon, Negar Rostamzadeh, Paul Nicholas, N'Mah Yilla-Akbari, Jess Gallegos, Andrew Smart, Emilio Garcia, et al. 2023. Sociotechnical harms of algorithmic systems: Scoping a taxonomy for harm reduction. In Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society. 723--741.
[38]
Larry Stillman, Marlien Herselman, Mario Marais, Mmamakanye Pitse Boshomane, Paul Plantinga, and Sheelagh Walton. 2012. Digital Doorway: Social-Technical Innovation for High-Needs Communities. The Electronic Journal of Information Systems in Developing Countries, Vol. 50, 1 (2012), 1--18.
[39]
Violet Turri and Rachel Dzombak. 2023. Why We Need to Know More: Exploring the State of AI Incident Documentation Practices. In Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society. 576--583.
[40]
Luisa Verdoliva. 2020. Media forensics and deepfakes: an overview. IEEE Journal of Selected Topics in Signal Processing, Vol. 14, 5 (2020), 910--932.
[41]
Jessica Wolfendale. 2007. My avatar, my self: Virtual harm and attachment. Ethics and information technology, Vol. 9 (2007), 111--119.
Index Terms
- Misinformation, Fraud, and Stereotyping: Towards a Typology of Harm Caused by Deepfakes
Recommendations
Gender stereotyping over instant messenger: The effects of gender and context
Researchers have observed that contrary to the high expectations for online communications, gender stereotyping has not disappeared in this medium, and is sometimes even accentuated. This study aimed to understand the effect of gender over instant ...
Cyberbullying: Eliciting harm without consequence
Students (N=260; M age=12.88years) reported their involvement in cyberbullying as well as their perceptions of the likelihood of cyberbully behavior eliciting harm, being reported, and the initiator receiving consequences. Also, students' scores on the ...
Towards Understanding Substance Abuse Misinformation in YouTube Videos
CSCW '23 Companion: Companion Publication of the 2023 Conference on Computer Supported Cooperative Work and Social ComputingTrigger warning: texts and figures contain substance abuse, addiction, and mental illness
YouTube is increasingly being utilized to acquire information about substance addiction and treatment experiences. However, the questionable quality and reliability ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
November 2024
755 pages
ISBN:9798400711145
DOI:10.1145/3678884
- General Chairs:
- Rosta Farzan,
- Claudia López,
- Program Chairs:
- Daniel Cardoso Llach,
- Daniele Quercia,
- Maryam Mustafa,
- Shuo Niu,
- Marisol Wong-Villacrés
Copyright © 2024 Owner/Author.
This work is licensed under a Creative Commons Attribution-ShareAlike International 4.0 License.
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 13 November 2024
Check for updates
Author Tags
Qualifiers
- Research-article
Funding Sources
Conference
CSCW '24
Sponsor:
CSCW '24: The 27th ACM Conference on Computer-Supported Cooperative Work and Social Computing
November 9 - 13, 2024
San Jose, Costa Rica
Acceptance Rates
Overall Acceptance Rate 2,235 of 8,521 submissions, 26%
Upcoming Conference
CSCW '25
- Sponsor:
- sigchi
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 158Total Downloads
- Downloads (Last 12 months)158
- Downloads (Last 6 weeks)104
Reflects downloads up to 04 Jan 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in