More Web Proxy on the site http://driver.im/

research-article

Open access

Data-Centric Factors in Algorithmic Fairness

Authors:

Elliott AshAuthors Info & Claims

AIES '22: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society

Pages 396 - 410

https://doi.org/10.1145/3514094.3534147

Published: 27 July 2022 Publication History

Abstract

Notwithstanding the widely held view that data generation and data curation processes are prominent sources of bias in machine learning algorithms, there is little empirical research seeking to document and understand the specific data dimensions affecting algorithmic unfairness. Contra the previous work, which has focused on modeling using simple, small-scale benchmark datasets, we hold the model constant and methodically intervene on relevant dimensions of a much larger, more diverse dataset. For this purpose, we introduce a new dataset on recidivism in 1.5 million criminal cases from courts in the U.S. state of Wisconsin, 2000-2018. From this main dataset, we generate multiple auxiliary datasets to simulate different kinds of biases in the data. Focusing on algorithmic bias toward different race/ethnicity groups, we assess the relevance of training data size, base rate difference between groups, representation of groups in the training data, temporal aspects of data curation, including race/ethnicity or neighborhood characteristics as features, and training separate classifiers by race/ethnicity or crime type. We find that these factors often do influence fairness metrics holding the classifier specification constant, without having a corresponding effect on accuracy metrics. The methodology and the results in the paper provide a useful reference point for a data-centric approach to studying algorithmic fairness in recidivism prediction and beyond.

Supplementary Material

MP4 File (Data_Centric_Factors.mp4)

In this video, Naman Goel gives a brief introduction to the AIES-22 paper "Data-Centric Factors in Algorithmic Fairness". Joint work with Nianyun Li and Elliott Ash.

Download
110.64 MB

References

[1]

Nil-Jana Akpinar, Manish Nagireddy, Logan Stapleton, Hao-Fei Cheng, Haiyi Zhu, Steven Wu, and Hoda Heidari. 2022. A Sandbox Tool to Bias (Stress)-Test Fairness Algorithms. arXiv preprint arXiv:2204.10233 (2022).

[2]

David Arnold, Will Dobbie, and Crystal S Yang. 2018. Racial bias in bail decisions. The Quarterly Journal of Economics, Vol. 133, 4 (2018), 1885--1932.

[3]

David Arnold, Will S Dobbie, and Peter Hull. 2020. Measuring racial discrimination in bail decisions. Technical Report. National Bureau of Economic Research.

[4]

Elliott Ash, Sam Asher, Aditi Bhowmick, Sandeep Bhupatiraju, Daniel Chen, Tanaya Devi, Christoph Goessmann, Paul Novosad, and Bilal Siddiqi. 2021. In-group bias in the Indian judiciary: Evidence from 5 million criminal cases. Technical Report. Working paper, August.

[5]

Elliott Ash, Daniel L Chen, and Suresh Naidu. 2019. Ideas have consequences: The impact of law and economics on american justice. Center for Law & Economics Working Paper Series, Vol. 4 (2019).

[6]

Elliott Ash, Sergio Galletta, Dominik Hangartner, Yotam Margalit, and Matteo Pinna. 2020. The effect of Fox News on health behavior during COVID-19. Available at SSRN 3636762 (2020).

[7]

Carolyn Ashurst, Ryan Carey, Silvia Chiappa, and Tom Everitt. 2022. Why Fair Labels Can Yield Unfair Predictions: Graphical Conditions for Introduced Unfairness. arXiv preprint arXiv:2202.10816 (2022).

[8]

Michelle Bao, Angela Zhou, Samantha Zottola, Brian Brubach, Sarah Desmarais, Aaron Horowitz, Kristian Lum, and Suresh Venkatasubramanian. 2021. It's COMPASlicated: The Messy Relationship between RAI Datasets and Algorithmic Fairness Benchmarks. (2021).

[9]

Solon Barocas and Andrew D. Selbst. 2016. Big data's disparate impact. California Law Review 104 (2016), 671--732. https://doi.org/10.15779/Z38BG31

[10]

Richard Berk. 2009. The role of race in forecasts of violent crime. Race and social problems, Vol. 1, 4 (2009), 231--242.

[11]

Richard Berk, Hoda Heidari, Shahin Jabbari, Michael Kearns, and Aaron Roth. 2021. Fairness in criminal justice risk assessments: The state of the art. Sociological Methods & Research, Vol. 50, 1 (2021), 3--44.

[12]

Alex Beutel, Jilin Chen, Tulsee Doshi, Hai Qian, Allison Woodruff, Christine Luu, Pierre Kreitmann, Jonathan Bischof, and Ed H Chi. 2019. Putting fairness principles into practice: Challenges, metrics, and improvements. In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society. 453--459.

Digital Library

[13]

Alex Beutel, Jilin Chen, Zhe Zhao, and Ed H Chi. 2017. Data Decisions and Theoretical Implications when Adversarially Learning Fair Representations. FATML (2017).

[14]

Arpita Biswas, Marta Kolczynska, Saana Rantanen, and Polina Rozenshtein. 2020. The role of in-group bias and balanced data: A comparison of human and machine recidivism risk predictions. In Proceedings of the 3rd ACM SIGCAS Conference on Computing and Sustainable Societies. 97--104.

Digital Library

[15]

Joy Buolamwini and Timnit Gebru. 2018. Gender shades: Intersectional accuracy disparities in commercial gender classification. In Conference on Fairness, Accountability, and Transparency (FAT*).

[16]

Aylin Caliskan, Joanna J. Bryson, and Arvind Narayanan. 2017. Semantics derived automatically from language corpora contain human-like biases. Science, Vol. 356, 6334 (2017), 183--186. https://doi.org/10.1126/science.aal4230

[17]

Irene Chen, Fredrik D Johansson, and David Sontag. 2018. Why is my classifier discriminatory? Advances in Neural Information Processing Systems, Vol. 31 (2018).

[18]

Tianqi Chen and Carlos Guestrin. 2016. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining. 785--794.

Digital Library

[19]

Silvia Chiappa and William S Isaac. 2018. A causal Bayesian networks viewpoint on fairness. In IFIP International Summer School on Privacy and Identity Management. Springer, 3--20.

[20]

Alexandra Chouldechova. 2017. Fair prediction with disparate impact: A study of bias in recidivism prediction instruments. Big Data, Vol. 5, 2 (2017), 153--163. https://doi.org/10.1089/big.2016.0047

[21]

Sam Corbett-Davies, Emma Pierson, Avi Feller, Sharad Goel, and Aziz Huq. 2017. Algorithmic decision making and the cost of fairness. In International Conference on Knowledge Discovery and Data Mining (KDD). https://doi.org/10.1145/3097983.3098095

Digital Library

[22]

Amanda Coston, Ashesh Rambachan, and Alexandra Chouldechova. 2021. Characterizing fairness over the set of good models under selective labels. In International Conference on Machine Learning. PMLR, 2144--2155.

[23]

Bo Cowgill and Catherine E Tucker. 2020. Algorithmic fairness and economics. Columbia Business School Research Paper (2020).

[24]

Alexander D'Amour, Hansa Srinivasan, James Atwood, Pallavi Baljekar, D Sculley, and Yoni Halpern. 2020. Fairness is not static: deeper understanding of long term fairness via simulation studies. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. 525--534.

Digital Library

[25]

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition. Ieee, 248--255.

[26]

Frances Ding, Moritz Hardt, John Miller, and Ludwig Schmidt. 2021. Retiring adult: New datasets for fair machine learning. Advances in Neural Information Processing Systems, Vol. 34 (2021).

[27]

Alessandro Fabris, Stefano Messina, Gianmaria Silvello, and Gian Antonio Susto. 2022. Algorithmic Fairness Datasets: the Story so Far. arXiv preprint arXiv:2202.01711 (2022).

[28]

Michael Feldman, Sorelle A Friedler, John Moeller, Carlos Scheidegger, and Suresh Venkatasubramanian. 2015. Certifying and removing disparate impact. In proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining. 259--268.

Digital Library

[29]

Sorelle A Friedler, Carlos Scheidegger, and Suresh Venkatasubramanian. 2021. The (im) possibility of fairness: Different value systems require different mechanisms for fair decision making. Commun. ACM, Vol. 64, 4 (2021), 136--143.

Digital Library

[30]

Talia B Gillis. 2020. False dreams of algorithmic fairness: The case of credit pricing. Available at SSRN 3571266 (2020).

[31]

Naman Goel, Alfonso Amayuelas, Amit Deshpande, and Amit Sharma. 2021. The Importance of Modeling Data Missingness in Algorithmic Fairness: A Causal Perspective. In Proceedings of the AAAI Conference on Artificial Intelligence.

[32]

Naman Goel and Boi Faltings. 2019. Crowdsourcing with fairness, diversity and budget constraints. In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society. 297--304.

Digital Library

[33]

Moritz Hardt and Eric Price. 2016. Equality of opportunity in supervised learning. In Advances in Neural Information Processing Systems (NIPS).

[34]

Moritz Hardt, Eric Price, and Nati Srebro. 2016. Equality of opportunity in supervised learning. Advances in neural information processing systems, Vol. 29 (2016).

Digital Library

[35]

Hans Hofmann. 1994. UCI Statlog (German Credit Data) Data Set. (1994).

[36]

Kim Steven Hunt and Robert Dumville. 2016. Recidivism among federal offenders: A comprehensive overview. United States Sentencing Commission.

[37]

Ranae Jabri. 2019. Predictive Power at What Cost? Economic and Racial Justice of Data-Driven Algorithms. Economic and Racial Justice of Data-Driven Algorithms (July 1, 2019) (2019).

[38]

HV Jagadish, Francesco Bonchi, Tina Eliassi-Rad, Lise Getoor, Krishna Gummadi, and Julia Stoyanovich. 2019. The responsibility challenge for data. In Proceedings of the 2019 International Conference on Management of Data. 412--414.

Digital Library

[39]

Jeff Larson, Surya Mattu, Lauren Kirchner and Julia Angwin. 2016. How We Analyzed the COMPAS Recidivism Algorithm. https://www.propublica.org/article/how-we-analyzed-the-compas-recidivism-algorithm

[40]

Eun Seo Jo and Timnit Gebru. 2020. Lessons from archives: Strategies for collecting sociocultural data in machine learning. In Proceedings of the 2020 conference on fairness, accountability, and transparency. 306--316.

Digital Library

[41]

Nathan Kallus and Angela Zhou. 2018. Residual unfairness in fair machine learning from prejudiced data. In International Conference on Machine Learning. PMLR, 2439--2448.

[42]

Divyansh Kaushik, Eduard Hovy, and Zachary Lipton. 2020. Learning The Difference That Makes A Difference With Counterfactually-Augmented Data. In International Conference on Learning Representations.

[43]

Jon Kleinberg, Himabindu Lakkaraju, Jure Leskovec, Jens Ludwig, and Sendhil Mullainathan. 2018. Human decisions and machine predictions. The quarterly journal of economics, Vol. 133, 1 (2018), 237--293.

[44]

Jon Kleinberg, Sendhil Mullainathan, and Manish Raghavan. 2017. Inherent trade-offs in the fair determination of risk scores. In Conference on Innovations in Theoretical Computer Science (ITCS).

[45]

Ronny Kohavi and Barry Becker. 1996. UCI ADULT Data Set. (1996).

[46]

Rodrigo L. Cardoso, Wagner Meira Jr, Virgilio Almeida, and Mohammed J. Zaki. 2019. A framework for benchmarking discrimination-aware models in machine learning. In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society. 437--444.

[47]

Himabindu Lakkaraju, Jon Kleinberg, Jure Leskovec, Jens Ludwig, and Sendhil Mullainathan. 2017. The selective labels problem: Evaluating algorithmic predictions in the presence of unobservables. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 275--284.

Digital Library

[48]

J. Larson, S. Mattu, L. Kirchner, and J. Angwin. 2016. https://github.com/propublica/compas-analysis, 2016. (2016).

[49]

Klas Leino, Matt Fredrikson, Emily Black, Shayak Sen, and Anupam Datta. 2019. Feature-Wise Bias Amplification. In International Conference on Learning Representations (ICLR).

[50]

Amanda Levendowski. 2018. How copyright law can fix artificial intelligence's implicit bias problem. Wash. L. Rev., Vol. 93 (2018), 579.

[51]

Claire SH Lim, Bernardo S Silveira, and James M Snyder. 2016. Do judge characteristics matter? ethnicity, gender, and partisanship in texas state trial courts. American Law and Economics Review, Vol. 18, 2 (2016), 302--357.

[52]

Kristian Lum and William Isaac. 2016. To predict and serve? Significance, Vol. 13, 5 (2016), 14--19.

[53]

Ninareh Mehrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan. 2021. A survey on bias and fairness in machine learning. ACM Computing Surveys (CSUR), Vol. 54, 6 (2021), 1--35.

Digital Library

[54]

Fatemeh Nargesian, Abolfazl Asudeh, and HV Jagadish. 2021. Tailoring data source distributions for fairness-aware data integration. Proceedings of the VLDB Endowment, Vol. 14, 11 (2021), 2519--2532.

Digital Library

[55]

Andrew Ng. 2021. MLOps: From Model-Centric to Data-Centric AI. DeepLearning AI (2021).

[56]

Amandalynne Paullada, Inioluwa Deborah Raji, Emily M Bender, Emily Denton, and Alex Hanna. 2021. Data and its (dis) contents: A survey of dataset development and use in machine learning research. Patterns, Vol. 2, 11 (2021), 100336.

[57]

Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. 2011. Scikit-learn: Machine learning in Python. the Journal of machine Learning research, Vol. 12 (2011), 2825--2830.

[58]

Inioluwa Deborah Raji, Emily M Bender, Amandalynne Paullada, Emily Denton, and Alex Hanna. 2021. AI and the Everything in the Whole Wide World Benchmark. (2021).

[59]

Ashesh Rambachan and Jonathan Roth. 2020. Bias In, Bias Out? Evaluating the Folk Wisdom. In 1st Symposium on Foundations of Responsible Computing.

[60]

Anna Rogers. 2021. Changing the World by Changing the Data. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 2182--2194.

[61]

Morgan Klaus Scheuerman, Alex Hanna, and Emily Denton. 2021. Do datasets have politics? Disciplinary values in computer vision dataset development. Proceedings of the ACM on Human-Computer Interaction, Vol. 5, CSCW2 (2021), 1--37.

Digital Library

[62]

Viktor Schlegel, Goran Nenadic, and Riza Batista-Navarro. 2020. Beyond leaderboards: A survey of methods for revealing weaknesses in natural language inference data and models. arXiv preprint arXiv:2005.14709 (2020).

[63]

David Sculley, Jasper Snoek, Alex Wiltschko, and Ali Rahimi. 2018. Winner's curse? On pace, progress, and empirical rigor. (2018).

[64]

Agnieszka Słowik and Léon Bottou. 2021. Algorithmic bias and data bias: Understanding the relation between distributionally robust optimization and data curation. arXiv preprint arXiv:2106.09467 (2021).

[65]

A TORRALBA. 2011. Unbiased Look at Dataset Bias. Proc. of IEEE Computer Vision and Pattern Recognition, 2011 (2011), 1521--1528.

[66]

Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Andrew Ilyas, and Aleksander Madry. 2020. From imagenet to image classification: Contextualizing progress on benchmarks. In International Conference on Machine Learning. PMLR, 9625--9635.

[67]

Tianlu Wang, Jieyu Zhao, Mark Yatskar, Kai-Wei Chang, and Vicente Ordonez. 2019. Balanced datasets are not enough: Estimating and mitigating gender bias in deep image representations. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 5310--5319.

[68]

Michael Wick, Jean-Baptiste Tristan, et al. 2019. Unlocking fairness: a trade-off revisited. Advances in neural information processing systems, Vol. 32 (2019).

[69]

Kaiyu Yang, Klint Qinami, Li Fei-Fei, Jia Deng, and Olga Russakovsky. 2020. Towards fairer datasets: Filtering and balancing the distribution of the people subtree in the imagenet hierarchy. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. 547--558.

Digital Library

[70]

Gal Yona, Amirata Ghorbani, and James Zou. 2021. Who's Responsible? Jointly Quantifying the Contribution of the Learning Algorithm and Data. In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society. 1034--1041.

Digital Library

[71]

Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2017. Men also like shopping: Reducing gender bias amplification using corpus-level constraints. arXiv preprint arXiv:1707.09457 (2017).

Cited By

Bhardwaj EGujral HWu SZogheib CMaharaj TBecker C(2024)Machine learning data practices through a data curation lens: An evaluation frameworkProceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency10.1145/3630106.3658955(1055-1067)Online publication date: 3-Jun-2024
https://dl.acm.org/doi/10.1145/3630106.3658955
Rosado Gomez ACalderón Benavides MAngélica LLattanzi SMuñoz Medina AAkoglu LGionis AVassilvitskii S(2024)Framework for Bias Detection in Machine Learning Models: A Fairness ApproachProceedings of the 17th ACM International Conference on Web Search and Data Mining10.1145/3616855.3635731(1152-1154)Online publication date: 4-Mar-2024
https://dl.acm.org/doi/10.1145/3616855.3635731
Ash EGoel NLi NMarangon CSun POh ANaumann TGloberson ASaenko KHardt MLevine S(2023)WCLDProceedings of the 37th International Conference on Neural Information Processing Systems10.5555/3666122.3666676(12626-12643)Online publication date: 10-Dec-2023
https://dl.acm.org/doi/10.5555/3666122.3666676
Show More Cited By

Index Terms

Data-Centric Factors in Algorithmic Fairness

Recommendations

Algorithmic Fairness and Structural Injustice: Insights from Feminist Political Philosophy
AIES '22: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society

Data-driven predictive algorithms are widely used to automate and guide high-stake decision making such as bail and parole recommendation, medical resource distribution, and mortgage allocation. Nevertheless, harmful outcomes biased against vulnerable ...
Algorithmic Unfairness through the Lens of EU Non-Discrimination Law: Or Why the Law is not a Decision Tree
FAccT '23: Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency

Concerns regarding unfairness and discrimination in the context of artificial intelligence (AI) systems have recently received increased attention from both legal and computer science scholars. Yet, the degree of overlap between notions of algorithmic ...
Scaffolding Children’s Sensemaking around Algorithmic Fairness
IDC '23: Proceedings of the 22nd Annual ACM Interaction Design and Children Conference

Prior research has investigated children’s perceptions of algorithmic bias, but provides little guidance on engaging children in conversations on algorithmic bias that center their agency and well-being. To address this, we developed discussions and ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

AIES '22: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society

July 2022

939 pages

ISBN:9781450392471

DOI:10.1145/3514094

General Chairs:
Vincent Conitzer
Duke University & University of Oxford
,
John Tasioulas
University of Oxford
,
Program Chairs:
Matthias Scheutz
Tufts University
,
Ryan Calo
University of Washington
,
Martina Mara
Johannes Kepler University Linz
,
Annette Zimmermann
University of York

Copyright © 2022 Owner/Author.

This work is licensed under a Creative Commons Attribution International 4.0 License.

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 27 July 2022

Check for updates

Author Tags

Qualifiers

Research-article

Conference

AIES '22

Sponsor:

SIGAI

AIES '22: AAAI/ACM Conference on AI, Ethics, and Society

May 19 - 21, 2021

Oxford, United Kingdom

Acceptance Rates

Overall Acceptance Rate 61 of 162 submissions, 38%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

6
Total Citations
View Citations
686
Total Downloads

Downloads (Last 12 months)283
Downloads (Last 6 weeks)22

Reflects downloads up to 09 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Bhardwaj EGujral HWu SZogheib CMaharaj TBecker C(2024)Machine learning data practices through a data curation lens: An evaluation frameworkProceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency10.1145/3630106.3658955(1055-1067)Online publication date: 3-Jun-2024
https://dl.acm.org/doi/10.1145/3630106.3658955
Rosado Gomez ACalderón Benavides MAngélica LLattanzi SMuñoz Medina AAkoglu LGionis AVassilvitskii S(2024)Framework for Bias Detection in Machine Learning Models: A Fairness ApproachProceedings of the 17th ACM International Conference on Web Search and Data Mining10.1145/3616855.3635731(1152-1154)Online publication date: 4-Mar-2024
https://dl.acm.org/doi/10.1145/3616855.3635731
Ash EGoel NLi NMarangon CSun POh ANaumann TGloberson ASaenko KHardt MLevine S(2023)WCLDProceedings of the 37th International Conference on Neural Information Processing Systems10.5555/3666122.3666676(12626-12643)Online publication date: 10-Dec-2023
https://dl.acm.org/doi/10.5555/3666122.3666676
Black ENaidu RGhani RRodolfa KHo DHeidari H(2023)Toward Operationalizing Pipeline-aware ML Fairness: A Research Agenda for Developing Practical Guidelines and ToolsEquity and Access in Algorithms, Mechanisms, and Optimization10.1145/3617694.3623259(1-11)Online publication date: 30-Oct-2023
https://doi.org/10.1145/3617694.3623259
Singh HChunara R(2023)Measures of Disparity and their Efficient EstimationProceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society10.1145/3600211.3604697(927-938)Online publication date: 8-Aug-2023
https://dl.acm.org/doi/10.1145/3600211.3604697
Gallese CScantamburlo TManzoni LNobile M(2023)Investigating Semi-Automatic Assessment of Data Sets Fairness by Means of Fuzzy Logic2023 IEEE Conference on Computational Intelligence in Bioinformatics and Computational Biology (CIBCB)10.1109/CIBCB56990.2023.10264913(1-10)Online publication date: 29-Aug-2023
https://doi.org/10.1109/CIBCB56990.2023.10264913

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents