More Web Proxy on the site http://driver.im/

research-article

Automated influence and the challenge of cognitive security

Authors:

Sarah Rajtmajer,

Daniel SusserAuthors Info & Claims

HotSoS '20: Proceedings of the 7th Symposium on Hot Topics in the Science of Security

Article No.: 8, Pages 1 - 9

https://doi.org/10.1145/3384217.3385615

Published: 21 September 2020 Publication History

Abstract

Advances in AI are powering increasingly precise and widespread computational propaganda, posing serious threats to national security. The military and intelligence communities are starting to discuss ways to engage in this space, but the path forward is still unclear. These developments raise pressing ethical questions, about which existing ethics frameworks are silent. Understanding these challenges through the lens of "cognitive security," we argue, offers a promising approach.

References

[1]

Mike Ananny. 2016. Toward an Ethics of Algorithms: Convening, Observation, Probability, and Timeliness. Science, Technology, Human Values 41, 1 (Jan 2016), 93--117.

[2]

Solon Barocas and Andrew D. Selbst. 2016. Big Data's Disparate Impact. California Law Review 104 (2016), 671--732.

[3]

Laurie R. Blank. 2017. Media Warfare, Propaganda, and the Law of War. Cambridge University Press, 88--103.

[4]

Hal Brands. 2016. Paradoxes of the Gray Zone. Technical Report. Foreign Policy Research Institute.

[5]

Matt Chessen. 2017. The MADCOM Future. Technical Report. The Atlantic Council. http://www.atlanticcouncil.org/images/publications/The_MADCOM_Future_RW_0926.pdf

[6]

CISA. 2018. Foreign Influence Taxonomy. Technical Report. US Cybersecurity and Infrastructure Security Agency. https://www.dhs.gov/sites/default/files/publications/19_0717_cisa_foreign-influence-taxonomy.pdf

[7]

CISA. 2019. The #Protect2020 Initiative. Technical Report. US Cybersecurity and Infrastructure Security Agency. https://www.dhs.gov/cisa/protect2020

[8]

CISA. 2019. The War on Pineapple. Technical Report. US Cybersecurity and Infrastructure Security Agency. https://www.dhs.gov/sites/default/files/publications/19_0717_cisa_the-war-on-pineapple-understanding-foreign-interference-in-5-steps_0.pdf

[9]

CoFR. 2018. Putin's Asymmetric Assault on Democracy in Russia and Europe: Implications for U.S. National Security. Technical Report. US Senate Committee on Foreign Relations. https://www.foreign.senate.gov/imo/media/doc/SPrt_115-21.pdf

[10]

Jack Corrigan. 2017. Social media is 'First Tool' of 21st-Century Warfare, Lawmaker Says. Nextgov.com (Sep 2017). https://www.nextgov.com/cio-briefing/2017/09/social-media-first-tool-21st-century-warfare-lawmaker-says/141379/

[11]

David Danks and Joseph H. Danks. 2017. Beyond Machines: Humans in Cyber-operations, Espionage, and Conflict. Oxford University Press, 177--198.

[12]

Randall R. Dipert. 2010. The Ethics of Cyberwarfare. Journal of Military Ethics 9, 4 (Dec. 2010), 384--410.

[13]

Randall R. Dipert. 2016. Distinctive Ethical Issues of Cyberwarfare. Oxford University Press, 56--72.

[14]

DoD. 2015. Cyber Strategy. Technical Report. US DoD. http://www.defense.gov/home/features/2015/0415_cyber-strategy/Final_2015_DoD_CYBER_STRATEGY_for_web.pdf

[15]

DoD. 2018. Cyber Strategy. Technical Report. US DoD. https://media.defense.gov/2018/Sep/18/2002041658/-1/-1/1/CYBER_STRATEGY_SUMMARY_FINAL.PDF

[16]

Batya Friedman and Helen Nissenbaum. 1996. Bias in Computer Systems. ACM Transactions on Information Systems 14, 3 (Jul 1996), 330--347.

Digital Library

[17]

Vijay Gadde and Yoel Roth. 2018. Enabling further research of information operations on Twitter. https://blog.twitter.com/en_us/topics/company/2018/enabling-further-research-of-information-operations-on-twitter.html

[18]

Adam Goldman. 2018. Justice Dept. Accuses Russians of Interfering in Midterm Elections. https://www.nytimes.com/2018/10/19/us/politics/russia-interference-midterm-elections.html?module=inline

[19]

Michael Gross and Tamar Meisels. 2017. Introduction. In Soft War: The Ethics of Unarmed Conflict. Cambridge University Press.

[20]

Michael L. Gross. 2017. Proportionate Self-Defense in Unarmed Conflict. Cambridge University Press, 217--232.

[21]

Matthew Hindman and Vlad Barash. 2018. Disinformation, "Fake News" and Influence Campaigns on Twitter. 62 pages. https://kf-site-production.s3.amazonaws.com/media_elements/files/000/000/238/original/KF-DisinformationReport-final2.pdf

[22]

Peter Holley. 2017. YouTube is tricking people who search for ISIS videos. https://www.washingtonpost.com/news/innovations/wp/2017/07/24/youtube-is-tricking-people-who-search-for-isis-videos/

[23]

Charlotte Jee. 2019. Twitter and Facebook won't remove false Trump campaign ads about Biden. MIT Technology Review (Oct 2019). https://www.technologyreview.com/f/614549/twitter-and-facebook-wont-remove-false-trump-campaign-ads-about-biden/

[24]

Kate Klonick. 2018. The New Governors: The People, Rules, and Processes Governing Online Speech. Harvard Law Review 131 (2018), 73.

[25]

Seth Lazar. 2017. War. Stanford Encyclopedia of Philosophy (2017). https://plato.stanford.edu/archives/spr2017/entries/war/

[26]

Kalev Leetaru. 2017. The backfire effect and why Facebook's 'Fake News' warning gets it all wrong. https://www.forbes.com/sites/kalevleetaru/2017/03/23/the-backfire-effect-and-why-facebooks-fake-news-warning-gets-it-all-wrong/

[27]

Cristiano Lima. 2019. Facebook, Twitter take down disinformation campaigns linked to Iran, Russia, Venezuela. https://politi.co/2G1gNQG

[28]

Jon R. Lindsay and Erik Gertzke (Eds.). 2019. Cross-Domain Deterrence. Oxford University press.

[29]

Ryan Lucas. 2018. New Reports Detail Expansive Russia Disinformation Scheme Targeting U.S. (Dec 2018). https://www.npr.org/2018/12/17/677390345/new-reports-detail-expansive-russia-disinformation-scheme-targeting-u-s

[30]

Alice Marwick and Rebecca Lewis. 2017. Media Manipulation and Disinformation Online. 106 pages.

[31]

M. Mazarr, A. Casey, A. Demus, S. Harold, L. Matthews, N. Beauchamp-Mustafaga, and J. Sladden. 2019. Hostile Social Manipulation: Present Realities and Emerging Trends. (2019). https://www.rand.org/pubs/research_reports/RR2713.html

[32]

Brent Daniel Mittelstadt, Patrick Allo, Mariarosaria Taddeo, Sandra Wachter, and Luciano Floridi. 2016. The Ethics of Algorithms. Big Data Society 3, 2 (Dec 2016).

[33]

MWI. 2018. Rule of Law in the Gray Zone. Technical Report. Modern War Institute. https://mwi.usma.edu/rule-law-gray-zone/

[34]

Anthony Nadler, Matthew Crain, and Joan Donovan. 2018. Weaponizing the Digital Influence Machine: The Political Perils of Online Ad Tech. 47 pages. https://datasociety.net/wp-content/uploads/2018/10/DS_Digital_Influence_Machine.pdf

[35]

Helen Nissenbaum. 2005. Where Computer Security Meets National Security. Ethics and Information Technology 7, 2 (Jun 2005), 61--73.

Digital Library

[36]

Eli Pariser. 2011. The Filter Bubble: What the Internet Is Hiding from You. Penguin Press. http://search.ebscohost.com/login.aspx?direct=true&scope=site&db=nlebk&db=nlabk&AN=1118322

[37]

Frank Pasquale. 2015. The Black Box Society: The Secret Algorithms That Control Money and Information. Harvard University Press.

[38]

Neil Postman. 2006. Amusing Ourselves to Death: Public Discourse in the Age of Show Business (20th anniversary ed ed.). Penguin Books.

[39]

Associated Press. 2020. U.S. military adopts new ethics principles for using AI in war. https://www.latimes.com/world-nation/story/2020-02-24/pentagon-adopts-new-ethical-principles-for-using-ai-in-war

[40]

Elspeth Reeve. 2013. Americans finally have access to American propaganda. https://www.theatlantic.com/politics/archive/2013/07/americans-finally-have-access-american-propaganda/313305/

[41]

Heather M. Roff and David Danks. 2018. "Trust but Verify": The Difficulty of Trusting Autonomous Weapons Systems. Journal of Military Ethics 17, 1 (Jan 2018), 2--20.

[42]

Kevin Roose. 2018. Facebook Grapples With a Maturing Adversary in Election Meddling. https://www.nytimes.com/2018/08/01/technology/facebook-trolls-midterm-elections.html?module=inline

[43]

Gabe Rottman. 2012. New government "propaganda" bill a positive step for First Amendment. https://www.aclu.org/blog/free-speech/new-government-propaganda-bill-positive-step-first-amendment

[44]

Cheyney Ryan. 2016. Pacifism. In The Oxford Handbook of Ethics of War, Seth Lazar and Helen Frowe (Eds.). Oxford University Press, Oxford.

[45]

Peter Schoomaker. 2005. Field Manual No. 3-05.30: Psychological Operations. Technical Report. US Army. https://fas.org/irp/doddir/army/fm3-05-30.pdf

[46]

Deepa Seetharaman. 2017. Facebook drowns out faek news with more information. https://www.wsj.com/articles/facebook-drowns-out-fake-news-with-more-information-1501754403

[47]

Chengcheng Shao, Giovanni Luca Ciampaglia, Onur Varol, Kai-Cheng Yang, Alessandro Flammini, and Filippo Menczer. 2018. The spread of low-credibility content by social bots. Nature Communications 9, 1 (Nov 2018).

[48]

Daniel J. Solove. 2011. Nothing to Hide: The False Tradeoff Between Privacy and Security. Yale University Press.

[49]

Colin Stretch. 2017. Social Media Influence in the 2016 U.S. Election. Hearing before the United States Senate, Committee on the Judiciary, Subcommittee on Crime and Terrorism.

[50]

Daniel Susser, Beate Roessler, and Helen Nissenbaum. 2018. Online Manipulation: Hidden Influences in a Digital World. SSRN (2018). https://papers.ssrn.com/abstract=3306006

[51]

Mariarosaria Taddeo. 2012. Information Warfare: A Philosophical Perspective. Philosophy & Technology 25, 1 (March 2012), 105--120.

[52]

Mariarosaria Taddeo. 2016. Just Information Warfare. Topoi 35, 1 (April 2016), 213--224.

[53]

Christopher Telley. 2018. The Influence Machine: Automated Information Operations as a Strategic Defeat Mechanism. Technical Report 121. The Institute of Land Warfare, Arlington, VA. 23 pages.

[54]

Catherine Theohary. 2018. Information Warfare: Issues for Congress. https://fas.org/sgp/crs/natsec/R45142.pdf

[55]

Mac Thornberry. 2012. The Smith-Mundt Modernization Act of 2012 (H.R. 5736). https://www.govinfo.gov/content/pkg/BILLS-112hr5736ih/pdf/BILLS-112hr5736ih.pdf

[56]

Jared Tracy. 2018. 100 years of subterfuge: the history of Army psychological operations. https://www.army.mil/article/199431/100_years_of_subterfuge_the_history_of_army_psychological_operations

[57]

Zeynep Tufekci. 2014. Engineering the Public: Big Data, Surveillance and Computational Politics. First Monday 19, 7 (Jul 2014). http://firstmonday.org/ojs/index.php/fm/article/view/4901

[58]

USSOCOM. 2018. Defining Gray Zone Challenges. Technical Report. US Special Operations Command.

[59]

Siva Vaidhyanathan. 2018. Antisocial Media: How Facebook Disconnects Us and Undermines Democracy. Oxford University Press.

[60]

Rand Waltzman. April 2017. The Weaponization of Information: The Need for Cognitive Security.

[61]

Michael Walzer. 2006. Just and Unjust Wars: A Moral Argument with Historical Illustrations (4th ed ed.). Basic Books, New York. OCLC: ocm71165547.

[62]

Tim Wu. 2016. The Attention Merchants: The Epic Scramble to Get Inside Our Heads (first edition ed.). Alfred A. Knopf.

[63]

Karen Yeung. 2017. Hypernudge: Big Data as a Mode of Regulation by Design. Information, Communication Society 20, 1 (Jan 2017), 118--136.

[64]

Frederik J. Zuiderveen Borgesius, Damian Trilling, Judith Möller, Balázs Bodó, Claes H. De Vreese, and Natali Helberger. 2016. Should We Worry About Filter Bubbles? Internet Policy Review (2016).

Cited By

Wang XLi JSrivatsavaya ERajtmajer S(2023)Evidence of inter-state coordination amongst state-backed information operationsScientific Reports10.1038/s41598-023-34245-113:1Online publication date: 12-May-2023
https://doi.org/10.1038/s41598-023-34245-1
Susser DGrimaldi VFourcade MKuipers BLazar SMulligan D(2021)Measuring Automated Influence: Between Empirical Evidence and Ethical ValuesProceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society10.1145/3461702.3462532(242-253)Online publication date: 21-Jul-2021
https://dl.acm.org/doi/10.1145/3461702.3462532
Grady CRajtmajer SDennis L(2021)When Smart Systems Fail: The Ethics of Cyber–Physical Critical Infrastructure RiskIEEE Transactions on Technology and Society10.1109/TTS.2021.30586052:1(6-14)Online publication date: Mar-2021
https://doi.org/10.1109/TTS.2021.3058605

Index Terms

Automated influence and the challenge of cognitive security
1. Security and privacy
  1. Human and societal aspects of security and privacy
2. Social and professional topics

Recommendations

Disinformation Warfare: Understanding State-Sponsored Trolls on Twitter and Their Influence on the Web
WWW '19: Companion Proceedings of The 2019 World Wide Web Conference

Over the past couple of years, anecdotal evidence has emerged linking coordinated campaigns by state-sponsored actors with efforts to manipulate public opinion on the Web, often around major political events, through dedicated accounts, or “trolls.” ...
Did State-Sponsored Trolls Shape the 2016 US Presidential Election Discourse? Quantifying Influence on Twitter
Security and Privacy in Social Networks and Big Data
Abstract
It is a widely accepted fact that state-sponsored Twitter accounts operated during the 2016 US presidential election, spreading millions of tweets with misinformation and inflammatory political content. Whether these social media campaigns of the ...
Unraveling the Web of Disinformation: Exploring the Larger Context of State-Sponsored Influence Campaigns on Twitter
RAID '24: Proceedings of the 27th International Symposium on Research in Attacks, Intrusions and Defenses

Social media platforms offer unprecedented opportunities for connectivity and exchange of ideas; however, they also serve as fertile grounds for the dissemination of disinformation. Over the years, there has been a rise in state-sponsored campaigns ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Other conferences

HotSoS '20: Proceedings of the 7th Symposium on Hot Topics in the Science of Security

September 2020

189 pages

ISBN:9781450375610

DOI:10.1145/3384217

General Chair:
Perry Alexander
The University of Kansas
,
Program Chairs:
Drew Davidson
The University of Kansas
,
Baek-Young Choi
University of Missouri, Kansas City

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 21 September 2020

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

HotSoS '20

HotSoS '20: Hot Topics in the Science of Security

September 21 - 23, 2020

Kansas, Lawrence

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
175
Total Downloads

Downloads (Last 12 months)32
Downloads (Last 6 weeks)5

Reflects downloads up to 16 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Wang XLi JSrivatsavaya ERajtmajer S(2023)Evidence of inter-state coordination amongst state-backed information operationsScientific Reports10.1038/s41598-023-34245-113:1Online publication date: 12-May-2023
https://doi.org/10.1038/s41598-023-34245-1
Susser DGrimaldi VFourcade MKuipers BLazar SMulligan D(2021)Measuring Automated Influence: Between Empirical Evidence and Ethical ValuesProceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society10.1145/3461702.3462532(242-253)Online publication date: 21-Jul-2021
https://dl.acm.org/doi/10.1145/3461702.3462532
Grady CRajtmajer SDennis L(2021)When Smart Systems Fail: The Ethics of Cyber–Physical Critical Infrastructure RiskIEEE Transactions on Technology and Society10.1109/TTS.2021.30586052:1(6-14)Online publication date: Mar-2021
https://doi.org/10.1109/TTS.2021.3058605

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents