[go: up one dir, main page]
More Web Proxy on the site http://driver.im/ skip to main content
10.1145/3375627.3375807acmconferencesArticle/Chapter ViewAbstractPublication PagesaiesConference Proceedingsconference-collections
research-article

When Trusted Black Boxes Don't Agree: Incentivizing Iterative Improvement and Accountability in Critical Software Systems

Published: 07 February 2020 Publication History

Abstract

Software increasingly plays a key role in regulated areas like housing, hiring, and credit, as well as major public functions such as criminal justice and elections. It is easy for there to be unintended defects with a large impact on the lives of individuals and society as a whole. Preventing, finding, and fixing software defects is a key focus of both industrial software development efforts as well as academic research in software engineering. In this paper, we discuss flaws in the larger socio-technical decision-making processes in which critical black-box software systems are developed, deployed, and trusted. We use criminal justice software, specifically probabilistic genotyping (PG) software, as a concrete example. We describe how PG software systems, designed to do the same job, produce different results. We highlight the under-appreciated impact of changes in key parameters and the disparate impact that one such parameter can have on different racial/ethnic groups. We propose concrete changes to the socio-technical decision-making processes surrounding the use of PG software that could be used to incentivize iterative improvements in the accuracy, fairness, reliability, and accountability of these systems.

References

[1]
2017. Post-conviction hearing in the matter of People v. Robinson, (ordering DNA testing on certain samples). 147 A.D.3d 784 (2d Dept. 2017).
[2]
Joy Buolamwini and Timnit Gebru. 2018. Gender shades: Intersectional accuracy disparities in commercial gender classification. In Conference on fairness, accountability and transparency. 77--91.
[3]
John M Butler, Richard Schoske, Peter M Vallone, JanetteWRedman, Margaret C Kline, et al. 2003. Allele frequencies for 15 autosomal STR loci on US Caucasian, African American, and Hispanic populations. Journal of forensic sciences 48, 4 (2003), 908--911.
[4]
P. Garofano, D. Caneparo, G. D'Amico, M. Vincenti, and E. Alladioa. 2015. An alternative application of the consensus method to DNA typing interpretation for Low Template-DNA mixtures. Genetics Supplement Series 5 (2015). https://doi.org/10.1016/j.fsigss.2015.09.168
[5]
Timnit Gebru, Jamie Morgenstern, Briana Vecchione, JenniferWortman Vaughan, Hanna M. Wallach, Hal Daumé III, and Kate Crawford. 2018. Datasheets for Datasets. CoRR abs/1803.09010 (2018). arXiv:1803.09010 http://arxiv.org/abs/1803.09010
[6]
Lauren Kirchner. 2017. Thousands of Criminal Cases in New York Relied on Disputed DNA Testing Techniques. https://www.propublica.org/article/thousandsof- criminal-cases-in-new-york-relied-on-disputed-dna-testing-techniques. (September 2017).
[7]
Steven P Lund and Hari Iyer. 2017. Likelihood Ratio as Weight of Forensic Evidence: A Closer Look. Journal of Research of National Institute of Standards and Technology 122, 27 (2017). https://doi.org/10.6028/jres.122.027
[8]
Jeanna Matthews, Marzieh Babaeianjelodar, Stephen Lorenz, Abigail Matthews, Mariama Njie, Nathaniel Adams, Dan Krane, Jessica Goldthwaite, and Clinton Hughes. 2019. The Right To Confront Your Accusers: Opening the Black Box of Forensic DNA Software. In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society (AIES '19). ACM, New York, NY, USA, 321--327. https://doi.org/10.1145/3306618.3314279
[9]
Adele A Mitchell, Jeannie Tamariz, Kathleen O'Connell, Nubia Ducasse, Zoran Budimlija, Mechthild Prinz, and Theresa Caragine. 2012. Validation of a DNA mixture statistics tool incorporating allelic drop-out and drop-in. Forensic Science International: Genetics 6, 6 (2012), 749--761.
[10]
Marco Túlio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016. "Why Should I Trust You?": Explaining the Predictions of Any Classifier. CoRR abs/1602.04938 (2016). arXiv:1602.04938 http://arxiv.org/abs/1602.04938
[11]
Supreme Court Kings County New York 2015. People v. Collins. 15 N.Y.S.3d 564 (Sup.Ct. Kings Co. 2015).
[12]
H. Swaminathan, M. Qureshi, C. Grgicak, K. Duffy, and D. Lun. 2018. Four model variants within a continuous forensic DNA mixture interpretation framework: Effects on evidential inference and reporting. PLOS One 13, 11 (2018). https://doi.org/10.1371/journal.pone.0207599
[13]
Rebecca Wexler. 2018. Life, liberty, and trade secrets: Intellectual property in the criminal justice system. Stan. L. Rev. 70 (2018), 1343.
[14]
R. Wyden, C. Booker, and Y. Clarke. 2019. The Algorithmic Accountability Act.

Cited By

View all
  • (2024)(Beyond) Reasonable Doubt: Challenges that Public Defenders Face in Scrutinizing AI in CourtProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3641902(1-19)Online publication date: 11-May-2024
  • (2023)Unfairness Is Everywhere, so What to Do? An Interview With Jeanna MatthewsIEEE Software10.1109/MS.2023.330572240:6(135-138)Online publication date: 1-Nov-2023
  • (2023)Metamorphic Testing and Debugging of Tax Preparation SoftwareProceedings of the 45th International Conference on Software Engineering: Software Engineering in Society10.1109/ICSE-SEIS58686.2023.00019(138-149)Online publication date: 17-May-2023
  • Show More Cited By

Recommendations

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences
AIES '20: Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society
February 2020
439 pages
ISBN:9781450371100
DOI:10.1145/3375627
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 07 February 2020

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. algorithmic accountability
  2. criminal justice software
  3. disparate impact
  4. probabilistic genotyping
  5. software verification

Qualifiers

  • Research-article

Conference

AIES '20
Sponsor:

Acceptance Rates

Overall Acceptance Rate 61 of 162 submissions, 38%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)21
  • Downloads (Last 6 weeks)0
Reflects downloads up to 06 Jan 2025

Other Metrics

Citations

Cited By

View all
  • (2024)(Beyond) Reasonable Doubt: Challenges that Public Defenders Face in Scrutinizing AI in CourtProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3641902(1-19)Online publication date: 11-May-2024
  • (2023)Unfairness Is Everywhere, so What to Do? An Interview With Jeanna MatthewsIEEE Software10.1109/MS.2023.330572240:6(135-138)Online publication date: 1-Nov-2023
  • (2023)Metamorphic Testing and Debugging of Tax Preparation SoftwareProceedings of the 45th International Conference on Software Engineering: Software Engineering in Society10.1109/ICSE-SEIS58686.2023.00019(138-149)Online publication date: 17-May-2023
  • (2022)Adversarial Scrutiny of Evidentiary Statistical SoftwareProceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency10.1145/3531146.3533228(1733-1746)Online publication date: 21-Jun-2022
  • (2021)Modeling and Guiding the Creation of Ethical Human-AI TeamsProceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society10.1145/3461702.3462573(469-479)Online publication date: 21-Jul-2021
  • (2020)Applying Algorithmic Accountability Frameworks with Domain-specific Codes of EthicsProceedings of the 2020 ACM-IMS on Foundations of Data Science Conference10.1145/3412815.3416897(83-91)Online publication date: 19-Oct-2020
  • (2020)What can forensic probabilistic genotyping software developers learn from significant non‐forensic software failures?WIREs Forensic Science10.1002/wfs2.13983:2Online publication date: 5-Oct-2020
  • (undefined)Adversarial Scrutiny of Evidentiary Statistical SoftwareSSRN Electronic Journal10.2139/ssrn.4107017

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media