More Web Proxy on the site http://driver.im/

research-article

Auditing radicalization pathways on YouTube

Authors:

Manoel Horta Ribeiro,

Raphael Ottoni,

Virgílio A. F. Almeida,

Wagner Meira, Jr.Authors Info & Claims

FAT* '20: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency

Pages 131 - 141

https://doi.org/10.1145/3351095.3372879

Published: 27 January 2020 Publication History

Abstract

Non-profits, as well as the media, have hypothesized the existence of a radicalization pipeline on YouTube, claiming that users systematically progress towards more extreme content on the platform. Yet, there is to date no substantial quantitative evidence of this alleged pipeline. To close this gap, we conduct a large-scale audit of user radicalization on YouTube. We analyze 330,925 videos posted on 349 channels, which we broadly classified into four types: Media, the Alt-lite, the Intellectual Dark Web (I.D.W.), and the Alt-right. According to the aforementioned radicalization hypothesis, channels in the I.D.W. and the Alt-lite serve as gateways to fringe far-right ideology, here represented by Alt-right channels. Processing 72M+ comments, we show that the three channel types indeed increasingly share the same user base; that users consistently migrate from milder to more extreme content; and that a large percentage of users who consume Alt-right content now consumed Alt-lite and I.D.W. content in the past. We also probe YouTube's recommendation algorithm, looking at more than 2M video and channel recommendations between May/July 2019. We find that Alt-lite content is easily reachable from I.D.W. channels, while Alt-right videos are reachable only through channel recommendations. Overall, we paint a comprehensive picture of user radicalization on YouTube.

Supplementary Material

PDF File (p131-ribeiro-supp.pdf)

Supplemental material.

Download
1.70 MB

References

[1]

[n. d.]. Media Bias/Fact Check - Search and Learn the Bias of News Media. https://mediabiasfactcheck.com/

[2]

ADL. [n. d.]. Glossary Terms Alt-Right. https://www.adl.org/resources/glossary-terms/alt-right

[3]

ADL. 2019. From Alt Right to Alt Lite: Naming the Hate. https://web.archive.org/web/20190422202936/https://www.adl.org/resources/backgrounders/from-alt-right-to-alt-lite-naming-the-hate

[4]

Swati Agarwal and Ashish Sureka. 2014. A Focused Crawler for Mining Hate and Extremism Promoting Videos on YouTube. In Proceedings of the 25th ACM Conference on Hypertext and Social Media. ACM.

Digital Library

[5]

Monica Anderson and Jingjing Jiang. 2018. Teens, Social Media & Technology 2018. Technical Report. Pew Research Center. https://www.pewinternet.org/2018/05/31/teens-social-media-technology-2018/

[6]

Andywarski. 2018. Richard Spencer, Styx and Sargon Have a Chat - Andy and JF moderate. https://web.archive.org/web/20190616134137/https://www.youtube.com/watch?v=UiUH-tWHbr8

[7]

Anonymous. [n. d.]. The Intellectual Dark Web. https://web.archive.org/web/20190407170300/http://intellectualdark.website/

[8]

David C. Atkinson. 2018. Charlottesville and the alt-right: a turning point? Politics, Groups, and Identities (2018).

[9]

Khaled A. Beydoun. 2018. US liberal Islamophobia is rising - and more insidious than rightwing bigotry | Khaled A Beydoun. The Guardian (2018). https://www.theguardian.com/commentisfree/2018/may/26/us-liberal-islamophobia-rising-more-insidious

[10]

Tyler Bridges. 2018. "Alt-Lite" Bloggers and the Conservative Ecosystem. https://shorensteincenter.org/alt-lite-bloggers-conservative-ecosystem/

[11]

Paul Covington, Jay Adams, and Emre Sargin. 2016. Deep Neural Networks for YouTube Recommendations. In Proceedings of the 10th ACM Conference on Recommender Systems (RecSys '16). ACM.

Digital Library

[12]

James Davidson, Benjamin Liebald, Junning Liu, Palash Nandy, Taylor Van Vleet, Ullas Gargi, Sujoy Gupta, Yu He, Mike Lambert, Blake Livingston, and Dasarathi Sampath. 2010. The YouTube Video Recommendation System. In Proceedings of the Fourth ACM Conference on Recommender Systems. ACM.

Digital Library

[13]

Nicholas Diakopoulos. 2014. Algorithmic Accountability Reporting: On the Investigation of Black Boxes. (2014).

[14]

Benjamin G. Edelman and Michael Luca. 2014. Digital Discrimination: The Case of Airbnb.com. Technical Report. Social Science Research Network. https://papers.ssrn.com/abstract=2377353

[15]

Lisa W. Foderaro. 2018. Alexandria Ocasio-Cortez Likens $10,000 Debate Offer by Conservative Columnist to Catcalling. The New York Times (2018). https://www.nytimes.com/2018/08/10/nyregion/alexandria-ocasio-cortez-debate-catcalling-ben-shapiro.html

[16]

T. Giannakopoulos, A. Pikrakis, and S. Theodoridis. 2010. A Multimodal Approach to Violence Detection in Video Sharing Sites. In 2010 20th International Conference on Pattern Recognition.

[17]

Rosie Gray. 2015. How 2015 Fueled The Rise Of The Freewheeling, White Nationalist Alt- Movement. https://www.buzzfeednews.com/article/rosiegray/how-2015-fueled-the-rise-of-the-freewheeling-white-nationali

[18]

Keegan Hankes and Alex Amend. 2018. The Alt-Right is Killing People. https://www.splcenter.org/20180205/alt-right-killing-people

[19]

Aniko Hannak, Piotr Sapiezynski, Arash Molavi Kakhki, Balachander Krishnamurthy, David Lazer, Alan Mislove, and Christo Wilson. 2013. Measuring Personalization of Web Search. In Proceedings of the 22Nd International Conference on World Wide Web. ACM.

Digital Library

[20]

Aniko Hannak, Gary Soeller, David Lazer, Alan Mislove, and Christo Wilson. 2014. Measuring Price Discrimination and Steering on E-commerce Web Sites. In Proceedings of the 2014 Conference on Internet Measurement Conference (IMC '14). ACM.

Digital Library

[21]

Mathew Ingram. 2018. Most Americans say they have lost trust in the media. Columbia Journalism Review (2018). https://www.cjr.org/the_media_today/trust-in-media-down.php

[22]

Brendan Joel Kelly. 2017. Lauren Southern: The alt-right's Canadian dog whistler. https://www.splcenter.org/hatewatch/2017/11/07/lauren-southern-alt-right%E2%80%99s-canadian-dog-whistler

[23]

Max Kutner. 2016. Roosh V's journey from pickup artist to right-wing provocateur. Newsweek (2016). https://www.newsweek.com/2016/10/21/roosh-v-pickup-artist-right-wing-provocateur-509319.html

[24]

Rebecca Lewis. 2018. Alternative influence: Broadcasting the reactionary right on YouTube. Technical Report. Data and Society.

[25]

Cristina Lopez G. 2019. Stefan Molyneux is MAGA Twitter's favorite white nationalist. https://www.mediamatters.org/people/stefan-molyneux

[26]

Tim Lott. 2017. Jordan Peterson and the transgender wars. https://life.spectator.co.uk/2017/09/jordan-peterson-and-the-transgender-wars/

[27]

Alex Mann, Kevin Nguyen, and Katherine Gregory. 2019. 'Emperor Cottrell': Accused Christchurch shooter had celebrated rise of the Australian far-right.

[28]

Andrew Marantz. 2017. The Alt-Right Branding War Has Torn the Movement in Two. (2017). https://www.newyorker.com/news/news-desk/the-alt-right-branding-war-has-torn-the-movement-in-two

[29]

Clark McCauley and Sophia Moskalenko. 2008. Mechanisms of Political Radicalization: Pathways Toward Terrorism. Terrorism and Political Violence (2008).

[30]

Angela Nagle. 2017. Kill All Normies: Online Culture Wars From 4Chan And Tumblr To Trump And The Alt-Right.

[31]

Newman Nic, Richard Fletcher, Antonis Kalogeropoulos, David AL Levy, and Rasmus Kleis Nielsen. 2018. Reuters Institute Digital News Report 2018. Technical Report. Reuters Institute for the Study of Journalism.

[32]

Raphael Ottoni, Evandro Cunha, Gabriel Magno, Pedro Bernardina, Wagner Meira Jr., and Virgílio Almeida. 2018. Analyzing Right-wing YouTube Channels: Hate, Violence and Discrimination. In Proceedings of the 10th ACM Conference on Web Science (WebSci '18). ACM.

Digital Library

[33]

Derek O'Callaghan, Derek Greene, Maura Conway, Joe Carthy, and Pádraig Cunningham. 2015. Down the (White) Rabbit Hole: The Extreme Right and Online Recommender Systems. Social Science Computer Review (2015).

[34]

Kostantinos Papadamou, Antonis Papasavva, Savvas Zannettou, Jeremy Blackburn, Nicolas Kourtellis, Ilias Leontiadis, Gianluca Stringhini, and Michael Sirivianos. 2019. Disturbed YouTube for Kids: Characterizing and Detecting Inappropriate Videos Targeting Young Children. arXiv:1901.07046 [cs] (2019). http://arxiv.org/abs/1901.07046

[35]

PowerfulJRE. 2017. Sargon of Akkad - Joe Rogan Experience #979. https://web.archive.org/web/20190616134332/https://www.youtube.com/watch?v=xrBCsLsSD2E

[36]

Kevin Roose. 2019. The Making of a YouTube Radical. The New York Times (2019). https://www.nytimes.com/interactive/2019/06/08/technology/youtube-radical.html, https://www.nytimes.com/interactive/2019/06/08/technology/youtube-radical.html

[37]

Roosh V. 2016. I Do Not Disavow Richard Spencer. https://web.archive.org/web/20190331073037/https://www.rooshv.com/i-do-not-disavow-richard-spencer

[38]

Andrew Sellars. 2016. Defining Hate Speech. SSRN Scholarly Paper. Social Science Research Network. https://papers.ssrn.com/abstract=2882244

[39]

Ashish Sureka, Ponnurangam Kumaraguru, Atul Goyal, and Sidharth Chhabra. 2010. Mining YouTube to Discover Extremist Videos, Users and Hidden Communities. In Information Retrieval Technology (Lecture Notes in Computer Science). Springer Berlin Heidelberg.

[40]

The Rubin Report. 2018. Eric Weinstein: The Future of The Intellectual Dark Web. https://www.youtube.com/watch?v=tUl7-SvntQ4

[41]

Zeynep Tufekci. 2018. YouTube, the Great Radicalizer. The New York Times (2018). https://www.nytimes.com/2018/03/10/opinion/sunday/youtube-politics-radical.html

[42]

Bari Weiss and Damon Winter. 2018. Meet the Renegades of the Intellectual Dark Web. The New York Times (2018). https://www.nytimes.com/2018/05/08/opinion/intellectual-dark-web.html

[43]

Savvas Zannettou, Tristan Caulfield, Jeremy Blackburn, Emiliano De Cristofaro, Michael Sirivianos, Gianluca Stringhini, and Guillermo Suarez-Tangil. 2018. On the Origins of Memes by Means of Fringe Web Communities. In Proceedings of the Internet Measurement Conference 2018. ACM.

Digital Library

[44]

Savvas Zannettou, Tristan Caulfield, Emiliano De Cristofaro, Nicolas Kourtelris, Ilias Leontiadis, Michael Sirivianos, Gianluca Stringhini, and Jeremy Blackburn. 2017. The Web Centipede: Understanding How Web Communities Influence Each Other Through the Lens of Mainstream and Alternative News Sources. In Proceedings of the 2017 Internet Measurement Conference. ACM.

Digital Library

Cited By

Brett C(2025)“We have watched the woke monster destroy everything”: gender and affect in the construction of wokeness on reactionary YouTubeFeminist Media Studies10.1080/14680777.2024.2446303(1-17)Online publication date: 3-Jan-2025
https://doi.org/10.1080/14680777.2024.2446303
Lin YHuang P(2025)Digital traps: How technology fuels nomophobia and insomnia in Taiwanese college studentsActa Psychologica10.1016/j.actpsy.2024.104674252(104674)Online publication date: Feb-2025
https://doi.org/10.1016/j.actpsy.2024.104674
Ditlhokwa G(2024)Unlocking Monetization Potential in the Age of YouTube Algorithmic Bias: An Analysis of Botswana FilmmakingThe Future of Television and Video Industry10.5772/intechopen.113306Online publication date: 12-Jun-2024
https://doi.org/10.5772/intechopen.113306
Show More Cited By

Index Terms

Auditing radicalization pathways on YouTube
1. Human-centered computing
  1. Collaborative and social computing
    1. Empirical studies in collaborative and social computing

Recommendations

Analyzing Right-wing YouTube Channels: Hate, Violence and Discrimination
WebSci '18: Proceedings of the 10th ACM Conference on Web Science

As of 2018, YouTube, the major online video sharing website, hosts multiple channels promoting right-wing content. In this paper, we observe issues related to hate, violence and discriminatory bias in a dataset containing more than 7,000 videos and 17 ...
Tweeting propaganda, radicalization and recruitment: Islamic state supporters multi-sided twitter networks
dg.o '15: Proceedings of the 16th Annual International Conference on Digital Government Research

Islamic State (IS) terrorist networks in Syria and Iraq pose threats to national security. IS' exploitation of social media and digital strategy plays a key role in its global dissemination of propaganda, radicalization, and recruitment. However, ...
Topic modelling of racist and xenophobic YouTube comments. Analyzing hate speech against migrants and refugees spread through YouTube in Spanish
TEEM'21: Ninth International Conference on Technological Ecosystems for Enhancing Multiculturality (TEEM'21)

Racist and xenophobic hate speech transmitted online is of increasing concern, as social media provide the most suitable channel for its rapid, uncontrolled and massive spread. However, although this type of hate has been studied on diverse social ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

FAT* '20: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency

January 2020

895 pages

ISBN:9781450369367

DOI:10.1145/3351095

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

ACM: Association for Computing Machinery

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 27 January 2020

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Atmosphere
Capes
INCT-Cyber
Google Research Award
Fapemig
CNPq
MASWEB

Conference

FAT* '20

Sponsor:

ACM

FAT* '20: Conference on Fairness, Accountability, and Transparency

January 27 - 30, 2020

Barcelona, Spain

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

215
Total Citations
View Citations
9,523
Total Downloads

Downloads (Last 12 months)699
Downloads (Last 6 weeks)109

Reflects downloads up to 04 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Brett C(2025)“We have watched the woke monster destroy everything”: gender and affect in the construction of wokeness on reactionary YouTubeFeminist Media Studies10.1080/14680777.2024.2446303(1-17)Online publication date: 3-Jan-2025
https://doi.org/10.1080/14680777.2024.2446303
Lin YHuang P(2025)Digital traps: How technology fuels nomophobia and insomnia in Taiwanese college studentsActa Psychologica10.1016/j.actpsy.2024.104674252(104674)Online publication date: Feb-2025
https://doi.org/10.1016/j.actpsy.2024.104674
Ditlhokwa G(2024)Unlocking Monetization Potential in the Age of YouTube Algorithmic Bias: An Analysis of Botswana FilmmakingThe Future of Television and Video Industry10.5772/intechopen.113306Online publication date: 12-Jun-2024
https://doi.org/10.5772/intechopen.113306
Paolo GGonzalez-Billandon JKégl BSalakhutdinov RKolter ZHeller KWeller AOliver NScarlett JBerkenkamp F(2024)PositionProceedings of the 41st International Conference on Machine Learning10.5555/3692070.3693669(39493-39508)Online publication date: 21-Jul-2024
https://dl.acm.org/doi/10.5555/3692070.3693669
Bachaud L(2024)La manosphère anglophone : tour d’horizon et revue de la littératureRevue française des sciences de l’information et de la communication10.4000/11ubk28Online publication date: 2024
https://doi.org/10.4000/11ubk
Marino EBenitez-Baleato JRibeiro A(2024)The Polarization Loop: How Emotions Drive Propagation of Disinformation in Online Media—The Case of Conspiracy Theories and Extreme Right Movements in Southern EuropeSocial Sciences10.3390/socsci1311060313:11(603)Online publication date: 5-Nov-2024
https://doi.org/10.3390/socsci13110603
Capelos TSalmela MTalalakina ACotena O(2024)Ressentiment in the Manosphere: Conceptions of Morality and Avenues for Resistance in the Incel Hatred PipelinePhilosophies10.3390/philosophies90200369:2(36)Online publication date: 13-Mar-2024
https://doi.org/10.3390/philosophies9020036
Bronner G(2024)The Empire of Beliefs and Digital Worlds : What Does Contemporary Science Say?The Tocqueville Review10.3138/ttr.45.2.1145:2(11-34)Online publication date: 1-Dec-2024
https://doi.org/10.3138/ttr.45.2.11
Bondi TRafieian OYao a(2024)Privacy and Polarization: An Inference-Based FrameworkSSRN Electronic Journal10.2139/ssrn.4641822Online publication date: 2024
https://doi.org/10.2139/ssrn.4641822
Epstein R(2024)The Video Manipulation Effect (VME): A quantification of the possible impact that the ordering of YouTube videos might have on opinions and voting preferencesPLOS ONE10.1371/journal.pone.030303619:11(e0303036)Online publication date: 20-Nov-2024
https://doi.org/10.1371/journal.pone.0303036
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents