More Web Proxy on the site http://driver.im/

research-article

Datasheets for Datasets help ML Engineers Notice and Understand Ethical Issues in Training Data

Author:

Karen L. BoydAuthors Info & Claims

Proceedings of the ACM on Human-Computer Interaction, Volume 5, Issue CSCW2

Article No.: 438, Pages 1 - 27

https://doi.org/10.1145/3479582

Published: 18 October 2021 Publication History

Abstract

The social computing community has demonstrated interest in the ethical issues sometimes produced by machine learning (ML) models, like violations of privacy, fairness, and accountability. This paper discovers what kinds of ethical considerations machine learning engineers recognize, how they build understanding, and what decisions they make when working with a real-world dataset. In particular, it illustrates ways in which Datasheets for Datasets, an accountability intervention designed to help engineers explore unfamiliar training data, scaffolds the process of issue discovery, understanding, and ethical decision-making. Participants were asked to review an intentionally ethically problematic dataset and asked to think aloud as they used it to solve a given ML problem. Out of 23 participants, 11 were given a Datasheet they could use while completing the task. Participants were ethically sensitive enough to identify concerns in the dataset; participants who had a Datasheet did open and refer to it; and those with Datasheets mentioned ethical issues during the think-aloud earlier and more often than than those without. The think-aloud protocol offered a grounded description of how participants recognized, understood, and made a decision about ethical problems in an unfamiliar dataset. The method used in this study can test other interventions that claim to encourage recognition, promote understanding, and support decision-making among technologists.

Supplementary Material

ZIP File (v5cscw438aux.zip)

The zip contains the Datasheet offered to some participants as described in the study. It was originally presented as a Google doc, but is offered here as an rtf.

Download
7.50 KB

References

[1]

Ishmael P. Akaah. "Differences in research ethics judgments between male and female marketing professionals". In: Journal of Business Ethics 8.5 (May 1989), pp. 375--381. issn: 0167--4544, 1573-0697. url: http://link.springer.com/10.1007/BF00381729 (visited on 12/07/2019).

[2]

Cecilia Aragon et al. "Developing a Research Agenda for Human-Centered Data Science". In: Proceedings of the 19th ACM Conference on Computer Supported Cooperative Work and Social Computing Companion. CSCW '16 Companion. New York, NY, USA: Association for Computing Machinery, Feb. 2016, pp. 529--535. isbn: 978--1--4503--3950--6. url: https://doi.org/10.1145/2818052.2855518 (visited on 04/03/2021).

Digital Library

[3]

Solon Barocas and Helen Nissenbaum. "On notice: The trouble with Notice and Consent". In: Proceedings of the Engaging Data Forum: The First International Forum on the Application and Management of Personal Electronic Information. 2009. url: http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2567409 (visited on 06/24/2016).

[4]

Solon Barocas and Andrew D. Selbst. "Big Data's Disparate Impact". In: SSRN Electronic Journal (2016). issn: 1556--5068. url: https://www.ssrn.com/abstract=2477899 (visited on 02/27/2019).

[5]

Emily M. Bender and Batya Friedman. "Data Statements for Natural Language Processing: Toward Mitigating System Bias and Enabling Better Science". In: Transactions of the Association for Computational Linguistics 6 (Dec. 2018), pp. 587--604. issn: 2307--387X. url: https://www.mitpressjournals.org/doi/abs/10.1162/tacl_a_00041 (visited on 02/28/2019).

[6]

Niels van Berkel et al. "Crowdsourcing Perceptions of Fair Predictors for Machine Learning: A Recidivism Case Study". en. In: Proceedings of the ACM on Human-Computer Interaction 3.CSCW (Nov. 2019), pp. 1--21. issn: 2573-0142. url: https://dl.acm.org/doi/10.1145/3359130 (visited on 03/27/2021).

Digital Library

[7]

Lawrence Blum. Moral Perception and Particularity. 1991. url: www.jstor.org/stable/2381661 (visited on 12/02/2019).

[8]

Tolga Bolukbasi et al. "Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings". In: Advances in Neural Information Processing Systems 29. Ed. by D. D. Lee et al. Curran Associates, Inc., 2016, pp. 4349--4357. url: http://papers.nips.cc/paper/6228-man-is-to-computer-programmer-as-woman-is-to-homemaker-debiasing-word-embeddings.pdf (visited on 01/08/2019).

[9]

Karen Boyd and Katie Shilton. "Adapting Ethical Sensitivity as a Construct to Study Technology Design Teams (forthcoming)". In: Proceedings of the 2022 ACM Conference on Supporting Groupwork (2022).

[10]

Joy Buolamwini. CODED BIAS. CODED BIAS. url: https://www.codedbias.com (visited on 09/29/2020).

[11]

Joy Buolamwini and Timnit Gebru. "Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification". In: Proceedings of the 1st Conference on Fairness, Accountability and Transparency (2018), p. 15.

[12]

Jenna Burrell et al. "When Users Control the Algorithms: Values Expressed in Practices on Twitter". In: Proceedings of the ACM on Human-Computer Interaction 3.CSCW (Nov. 2019), 138:1--138:20. url: https://doi.org/10.1145/3359240 (visited on 04/03/2021).

Digital Library

[13]

Aylin Caliskan, Joanna J. Bryson, and Arvind Narayanan. "Semantics derived automatically from language corpora contain human-like biases". In: Science 356.6334 (Apr. 14, 2017), pp. 183--186. issn: 0036--8075, 1095--9203. arXiv: 1608.07187. url: http://arxiv.org/abs/1608.07187 (visited on 01/08/2019).

[14]

Stevie Chancellor, Eric P. S. Baumer, and Munmun De Choudhury. "Who is the "Human" in Human-Centered Machine Learning: The Case of Predicting Mental Health from Social Media". In: Proceedings of the ACM on Human-Computer Interaction 3. CSCW (Nov. 2019), 147:1--147:32. url: https://doi.org/10.1145/3359249 (visited on 04/03/2021).

Digital Library

[15]

Henriikka Clarkeburn. "A Test for Ethical Sensitivity in Science". In: Journal of Moral Education 31.4 (Dec. 2002), pp. 439--453. issn: 03057240. url: http://search.ebscohost.com/login.aspx?direct=true&db=asn&AN=8633678&site=ehost-live (visited on 12/13/2019).

[16]

Michael P Coyne, Dawn W Massey, and Jay C Thibodeau. "Raising Students' Ethical Sensivity with a Value Relevance Approach". In: Advances in Accounting Education: teaching and Curriculum Innovation 7 (2005).

[17]

Peggy Desautels. "Gestalt Shifts in Moral Perception". In: Mind and Morals: Essays on Cognitive Science and Ethics. Library Catalog: www.semanticscholar.org. 1996, pp. 129--143.

[18]

Nicholas Diakopoulos. "Algorithmic Accountability". In: Digital Journalism 3.3 (May 2015), pp. 398--415. issn: 2167-0811. url: https://doi.org/10.1080/21670811.2014.976411 (visited on 02/28/2019).

[19]

Benjamin H. Dotger. ""I Had No Idea": Developing Dispositional Awareness and Sensitivity through a Cross-Professional Pedagogy". In: Teaching and Teacher Education: An International Journal of Research and Studies 26.4 (May 2010), pp. 805--812. issn: 0742-051X. (Visited on 03/06/2020).

[20]

K. Anders Ericsson and Herbert A. Simon. Protocol analysis: Verbal reports as data. Protocol analysis: Verbal reports as data. Cambridge, MA, US: The MIT Press, 1984. 426 pp. isbn: 978-0--262--55012--3.

[21]

Melanie Feinberg. "Material Vision". In: Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing. CSCW '17. New York, NY, USA: Association for Computing Machinery, Feb. 2017, pp. 604--617. isbn: 978--1--4503--4335-0. url: https://doi.org/10.1145/2998181.2998204 (visited on 04/03/2021).

Digital Library

[22]

Samantha Fowler, Dana Zeidler, and Troy Sadler. "Moral Sensitivity in the Context of Socioscientific Issues in High School Science Students". In: International Journal of Science Education 31.2 (2009), pp. 279--296.

[23]

Batya Friedman and David Hendry. "The envisioning cards: a toolkit for catalyzing humanistic and technical imaginations". In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. CHI '12. Austin, Texas, USA: Association for Computing Machinery, May 5, 2012, pp. 1145--1148. isbn: 978--1--4503--1015--4. url: http://doi.org/10.1145/2207676.2208562 (visited on 08/06/2020).

Digital Library

[24]

Timnit Gebru et al. "Datasheets for Datasets". In: arXiv:1803.09010 [cs] (Mar. 23, 2018). arXiv: 1803.09010. url: http://arxiv.org/abs/1803.09010 (visited on 02/28/2019).

[25]

Nina Grgic-Hlaca et al. "Human Perceptions of Fairness in Algorithmic Decision Making: A Case Study of Criminal Risk Prediction". In: Proceedings of the 2018 World Wide Web Conference. WWW '18. Republic and Canton of Geneva, CHE: International World Wide Web Conferences Steering Committee, Apr. 2018, pp. 903--912. isbn: 978--1--4503--5639--8. url: https://doi.org/10.1145/3178876.3186138 (visited on 04/03/2021).

Digital Library

[26]

Thomas Hellström, Virginia Dignum, and Suna Bensch. "Bias in Machine Learning -- What is it Good for?" In: arXiv:2004.00686 [cs] (Sept. 2020). arXiv: 2004.00686. url: http://arxiv.org/abs/2004.00686 (visited on 07/02/2021).

[27]

Sarah Holland et al. "The Dataset Nutrition Label: A Framework To Drive Higher Data Quality Standards". In: arXiv:1805.03677 [cs] (May 9, 2018). arXiv: 1805.03677. url: http://arxiv.org/abs/1805.03677 (visited on 02/28/2019).

[28]

Kenneth Holstein et al. "Improving fairness in machine learning systems: What do industry practitioners need?" In: arXiv:1812.05239 [cs] (Dec. 12, 2018). arXiv: 1812.05239. url: http://arxiv.org/abs/1812.05239 (visited on 03/14/2019).

Digital Library

[29]

Ben Hutchinson et al. "Towards Accountability for Machine Learning Datasets: Practices from Software Engineering and Infrastructure". en. In: Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency. Virtual Event Canada: ACM, Mar. 2021, pp. 560--575. isbn: 978--1--4503--8309--7. url: https://dl.acm.org/doi/10.1145/3442188.3445918 (visited on 07/02/2021).

Digital Library

[30]

Suzy Jagger. "Ethical sensitivity: A Foundation for Moral Judgment". In: Journal of Business Ethics Education 1 (Jan. 1, 2011), pp. 13--30.

[31]

Thomas M. Jones. "Ethical Decision Making by Individuals in Organizations: An Issue-Contingent Model". In: The Academy of Management Review 16.2 (Apr. 1991), p. 366. issn: 03637425. url: http://www.jstor.org/stable/258867?origin=crossref (visited on 05/31/2018).

[32]

Jeff Larson Julia Angwin. Machine Bias. ProPublica. May 23, 2016. url: https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing (visited on 04/13/2018).

[33]

Tero Karras, Samuli Laine, and Timo Aila. "A Style-Based Generator Architecture for Generative Adversarial Networks". In: arXiv:1812.04948 [cs, stat] (Mar. 29, 2019). arXiv: 1812.04948. url: http://arxiv.org/abs/1812.04948 (visited on 07/24/2020).

[34]

Niki Kilbertus et al. "Blind Justice: Fairness with Encrypted Sensitive Attributes". In: arXiv:1806.03281 [cs, stat] (June 8, 2018). arXiv: 1806.03281. url: http://arxiv.org/abs/1806.03281 (visited on 01/12/2019).

[35]

Elizabeth Anne Kinsella. "Practitioner Reflection and Judgement as Phronesis". In: Phronesis as Professional Knowledge: Practical Wisdom in the Professions. Ed. by Elizabeth Anne Kinsella and Allan Pitman. Professional Practice and Education: A Diversity of Voices. Rotterdam: SensePublishers, 2012, pp. 35--52. isbn: 978--94--6091--731--8. url: https://doi.org/10.1007/978--94--6091--731--8_3 (visited on 09/28/2020).

[36]

Paul B. de Laat. "Algorithmic Decision-Making Based on Machine Learning from Big Data: Can Transparency Restore Accountability?" en. In: Philosophy & Technology 31.4 (Dec. 2018), pp. 525--541. issn: 2210--5433, 2210--5441. url: http://link.springer.com/10.1007/s13347-017-0293-z (visited on 07/02/2021).

[37]

Reed Larson and Mihaly Csikszentmihalyi. "The Experience Sampling Method". In: Flow and the Foundations of Positive Psychology: The Collected Works of Mihaly Csikszentmihalyi. Ed. by Mihaly Csikszentmihalyi. Dordrecht: Springer Netherlands, 2014, pp. 21--34. isbn: 978--94-017--9088--8. 94- 017- 9088- 8_2. url: https://doi.org/10.1007/978--94-017--9088--8_2 (visited on 09/29/2020).

[38]

Min Kyung Lee and Su Baykal. "Algorithmic Mediation in Group Decisions: Fairness Perceptions of Algorithmically Mediated vs. Discussion-Based Social Division". en. In: Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing. Portland Oregon USA: ACM, Feb. 2017, pp. 1035--1048. isbn: 978--1--4503--4335-0. url: https://dl.acm.org/doi/10.1145/2998181.2998230 (visited on 03/27/2021).

Digital Library

[39]

Rebecca Lind. "Ethical Sensitivity in Viewer Evaluations of a TV News Investigative Report". In: Human Communication Research 23.4 (June 1997), pp. 535--561. issn: 03603989. url: https://academic.oup.com/hcr/article/23/4/535--561/4564970 (visited on 04/09/2020).

[40]

Rebecca Ann Lind and Tammy Swenson-Lepper. "Measuring Sensitivity to Conflicts of Interest: A Preliminary Test of Method". In: Science and Engineering Ethics 19.1 (Mar. 1, 2013), pp. 43--62. issn: 1471--5546. url: https://doi.org/10.1007/s11948-011--9319--6 (visited on 02/07/2020).

[41]

Kim Lützén, Gun Nordström, and Mats Evertzon. "Moral Sensitivity in Nursing Practice". In: Scandinavian Journal of Caring Sciences 9.3 (1995), pp. 131--138. issn: 1471--6712. 6712.1995.tb00403.x. url: http://onlinelibrary.wiley.com/doi/abs/10.1111/j.1471--6712.1995.tb00403.x (visited on 12/12/2019).

[42]

Ninareh Mehrabi et al. "A Survey on Bias and Fairness in Machine Learning". In: arXiv:1908.09635 [cs] (Sept. 2019). arXiv: 1908.09635. url: http://arxiv.org/abs/1908.09635 (visited on 07/02/2021).

[43]

Jacob Metcalf et al. "Algorithmic Impact Assessments and Accountability: The Co-construction of Impacts". In: Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency. FAccT '21. New York, NY, USA: Association for Computing Machinery, Mar. 2021, pp. 735--746. isbn: 978--1--4503--8309--7. url: https://doi.org/10.1145/3442188.3445935 (visited on 04/03/2021).

Digital Library

[44]

Margaret Mitchell et al. "Model Cards for Model Reporting". In: Proceedings of the Conference on Fairness, Accountability, and Transparency (2019), pp. 220--229. arXiv: 1810.03993. url: http://arxiv.org/abs/1810.03993 (visited on 03/27/2019).

Digital Library

[45]

Michael Muller et al. "Interrogating Data Science". In: Conference Companion Publication of the 2020 on Computer Supported Cooperative Work and Social Computing. CSCW '20 Companion. New York, NY, USA: Association for Computing Machinery, Oct. 2020, pp. 467--473. isbn: 978--1--4503--8059--1. url: https://doi.org/10.1145/3406865.3418584 (visited on 04/03/2021).

Digital Library

[46]

Liisa Myyry and Klaus Helkama. "The Role of Value Priorities and Professional Ethics Training in Moral Sensitivity". In: Journal of Moral Education 31.1 (Mar. 1, 2002), pp. 35--50. issn: 0305--7240. url: https://doi.org/10.1080/03057240120111427 (visited on 02/27/2020).

[47]

Helen Nissenbaum. "Privacy as Contextual Integrity". In: Washington Law Review 79 (2004), p. 119. url: https://heinonline.org/HOL/Page?handle=hein.journals/washlr79&id=129&div=&collection=.

[48]

Cathy O'Neil. Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy. Crown, 2016. 274 pp. isbn: 978-0--553--41881--1.

[49]

Robin R Radtke. "The Effects of Gender and Setting on Accountants' Ethically Sensitive Decisions". en. In: Journal of Business Ethics 24.4 (2000), pp. 299--312.

[50]

Dillon Reisman et al. Algorithmic impact assessments: A practical framework for public agency accountability. en. Tech. rep. AI Now Institute, 2018.

[51]

James Rest et al. Postconventional moral thinking: A neo-Kohlbergian approach. Postconventional moral thinking: A neo-Kohlbergian approach. Pages: ix, 229. Mahwah, NJ, US: Lawrence Erlbaum Associates Publishers, 1999. ix, 229. isbn: 978-0--8058--3285--3.

[52]

James R. Rest. "A Psychologist Looks at the Teaching of Ethics". In: The Hastings Center Report 12.1 (1982), pp. 29--36. issn: 0093-0334. url: http://www.jstor.org/stable/3560621 (visited on 07/23/2019).

[53]

Nripsuta Ani Saxena et al. "How Do Fairness Definitions Fare? Examining Public Attitudes Towards Algorithmic Definitions of Fairness". In: Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society. AIES '19. New York, NY, USA: Association for Computing Machinery, Jan. 2019, pp. 99--106. isbn: 978--1--4503--6324--2. url: https://doi.org/10.1145/3306618.3314248 (visited on 04/03/2021).

Digital Library

[54]

Allen Schmaltz. "On the Utility of Lay Summaries and AI Safety Disclosures: Toward Robust, Open Research Oversight". In: Proceedings of the Second ACL Workshop on Ethics in Natural Language Processing. New Orleans, Louisiana, USA: Association for Computational Linguistics, June 2018, pp. 1--6. url: http://www.aclweb.org/anthology/W18-0801 (visited on 02/28/2019).

[55]

Andrew D. Selbst. "Disparate Impact in Big Data Policing". In: SSRN Electronic Journal (2017). issn: 1556--5068. url: https://www.ssrn.com/abstract=2819182 (visited on 02/28/2019).

[56]

Hetan Shah. "Algorithmic accountability". en. In: Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences 376.2128 (Sept. 2018), p. 20170362. issn: 1364--503X, 1471--2962. url: https://royalsocietypublishing.org/doi/10.1098/rsta.2017.0362 (visited on 07/02/2021).

[57]

Katie Shilton. "Four billion little brothers?: Privacy, mobile phones, and ubiquitous data collection". In: Communications of the ACM 52.11 (2009), pp. 48--53. url: http://dl.acm.org/citation.cfm?id=1592778 (visited on 08/18/2016).

Digital Library

[58]

Ben Shneiderman. "Opinion: The dangers of faulty, biased, or malicious algorithms requires independent oversight". In: Proceedings of the National Academy of Sciences 113.48 (Nov. 29, 2016), pp. 13538--13540. issn: 0027--8424, 1091--6490. url: http://www.pnas.org/lookup/doi/10.1073/pnas.1618211113 (visited on 02/28/2019).

[59]

Megha Srivastava, Hoda Heidari, and Andreas Krause. "Mathematical Notions vs. Human Perception of Fairness: A Descriptive Approach to Fairness for Machine Learning". In: Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. KDD '19. New York, NY, USA: Association for Computing Machinery, July 2019, pp. 2459--2468. isbn: 978--1--4503--6201--6. url: https://doi.org/10.1145/3292500.3330664 (visited on 04/03/2021).

Digital Library

[60]

Tammy Swenson-Lepper. "Ethical Sensitivity for Organizational Communication Issues: Examining Individual and Organizational Differences". In: Journal of Business Ethics 59.3 (July 1, 2005), pp. 205--231. issn: 1573-0697. url: https://doi.org/10.1007/s10551-005--2925-y (visited on 12/06/2019).

[61]

Linda Klebe Trevino et al. "Managing Ethics and Legal Compliance: What Works And What Hurts". In: California Management Review 41.2 (Jan. 1, 1999), pp. 131--151. issn: 0008--1256, 2162--8564. . 2307 / 41165990. url: http://cmr.ucpress.edu/content/41/2/131 (visited on 04/10/2017).

[62]

Joaquin Vanschoren. OpenML. OpenML: exploring machine learning better, together. url: https://www.openml.org (visited on 04/15/2019).

[63]

Michael Veale, Max Van Kleek, and Reuben Binns. "Fairness and Accountability Design Needs for Algorithmic Support in High-Stakes Public Sector Decision-Making". en. In: (2018), p. 14.

[64]

S. Verma and J. Rubin. "Fairness Definitions Explained". In: 2018 IEEE/ACM International Workshop on Software Fairness (FairWare). May 2018, pp. 1--7.

[65]

Jessica Vitak, Katie Shilton, and Zahra Ashktorab. "Beyond the Belmont Principles: Ethical Challenges, Practices, and Beliefs in the Online Data Research Community". In: Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing. CSCW '16. New York, NY, USA: Association for Computing Machinery, Feb. 2016, pp. 941--953. isbn: 978--1--4503--3592--8. url: https://doi.org/10.1145/2818048.2820078 (visited on 04/03/2021).

Digital Library

[66]

Ruotong Wang, F. Maxwell Harper, and Haiyi Zhu. "Factors Influencing Perceived Fairness in Algorithmic Decision-Making: Algorithm Outcomes, Development Procedures, and Individual Differences". In: Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. CHI '20. New York, NY, USA: Association for Computing Machinery, Apr. 2020, pp. 1--14. isbn: 978--1--4503--6708-0. url: https://doi.org/10.1145/3313831.3376813 (visited on 04/03/2021).

Digital Library

[67]

Kathryn Weaver. "Ethical Sensitivity: State of Knowledge and Needs for Further Research". In: Nursing Ethics 14.2 (Mar. 2007), pp. 141--155. issn: 0969--7330, 1477-0989. url: http://journals.sagepub.com/doi/10.1177/0969733007073694 (visited on 09/09/2019).

[68]

Kathryn Weaver, Janice Morse, and Carl Mitcham. "Ethical sensitivity in professional practice: concept analysis". In: Journal of Advanced Nursing (2008). url: https://onlinelibrary-wiley-com.proxy-um.researchport.umd.edu/doi/full/10.1111/j.1365--2648.2008.04625.x (visited on 09/18/2019).

[69]

Richmond Y Wong and Nick Merrill. "Engaging Speculative Practices to Probe Values & Ethics in Sociotechnical Systems". In: Workshop at iConference. 2019, p. 4.

[70]

Richmond Y. Wong et al. "Beyond Checklist Approaches to Ethics in Design". In: Conference Companion Publication of the 2020 on Computer Supported Cooperative Work and Social Computing. CSCW '20 Companion. New York, NY, USA: Association for Computing Machinery, Oct. 2020, pp. 511--517. isbn: 978--1--4503--8059--1. url: https://doi.org/10.1145/3406865.3418590 (visited on 04/03/2021).

Digital Library

[71]

Richmond Y. Wong et al. "Eliciting Values Reflections by Engaging Privacy Futures Using Design Workbooks". In: Proceedings of the ACM on Human-Computer Interaction 1 (CSCW Dec. 6, 2017), 111:1--111:26. url: http://doi.org/10.1145/3134746 (visited on 08/06/2020).

Digital Library

[72]

Ke Yang et al. "A Nutritional Label for Rankings". In: Proceedings of the 2018 International Conference on Management of Data - SIGMOD '18 (2018), pp. 1773--1776. arXiv: 1804.07890. url: http://arxiv.org/abs/1804.07890 (visited on 02/28/2019).

Digital Library

[73]

Zliobaitè, Indrè and Custers, Bart. "Using sensitive personal data may be necessary for avoiding discrimination in data-driven decision models". In: Artificial Intelligence and Law 24.2 (June 1, 2016), pp. 183--201. issn: 1572--8382. url: https://doi.org/10.1007/s10506-016--9182--5 (visited on 10/08/2020).

Cited By

Muralikumar MMcDonald D(2025)An Emerging Design Space of How Tools Support Collaborations in AI Design and DevelopmentProceedings of the ACM on Human-Computer Interaction10.1145/37011819:GROUP(1-28)Online publication date: 10-Jan-2025
https://dl.acm.org/doi/10.1145/3701181
Longpre SMahari RObeng-Marnu NBrannon WSouth TGero KPentland SKabbara JSalakhutdinov RKolter ZHeller KWeller AOliver NScarlett JBerkenkamp F(2024)Position: data authenticity, consent, & provenance for AI are all brokenProceedings of the 41st International Conference on Machine Learning10.5555/3692070.3693398(32711-32725)Online publication date: 21-Jul-2024
https://dl.acm.org/doi/10.5555/3692070.3693398
M RPankaj D(2024)Social-sum-Mal: A Dataset for Abstractive Text Summarization in MalayalamACM Transactions on Asian and Low-Resource Language Information Processing10.1145/369610723:11(1-20)Online publication date: 21-Nov-2024
https://dl.acm.org/doi/10.1145/3696107
Show More Cited By

Index Terms

Datasheets for Datasets help ML Engineers Notice and Understand Ethical Issues in Training Data
1. Human-centered computing
  1. Collaborative and social computing
    1. Collaborative and social computing design and evaluation methods
    2. Collaborative and social computing theory, concepts and paradigms
      1. Computer supported cooperative work
  2. Human computer interaction (HCI)
    1. HCI design and evaluation methods
      1. User studies

Recommendations

Designing Up with Value-Sensitive Design: Building a Field Guide for Ethical ML Development
FAccT '22: Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency

If “studying up,” or researching powerful actors in a social system, can offer insight into the workings and effects of power in social systems, this paper argues that “designing up” will give researchers and designers a tool to intervene. This paper ...
Ethical Sensitivity in Machine Learning Development
CSCW '20 Companion: Companion Publication of the 2020 Conference on Computer Supported Cooperative Work and Social Computing

Despite a great deal of attention to developing ethical mitigations for Machine Learning (ML) training data and models, we don't yet know how these interventions will be adopted by those who curate data and use them to train ML models. Will they help ML ...
Ethical responsibilities: the smart card engineer
ETHICS '14: Proceedings of the IEEE 2014 International Symposium on Ethics in Engineering, Science, and Technology

Albert Einstein in one of his philosophical notes stated: "I do not believe in immortality of the individual, and I consider ethics to be an exclusively human concern with no superhuman authority behind it". Arguably, ethics does not necessarily mean ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Proceedings of the ACM on Human-Computer Interaction

Proceedings of the ACM on Human-Computer Interaction Volume 5, Issue CSCW2

CSCW2

October 2021

5376 pages

EISSN:2573-0142

DOI:10.1145/3493286

Editor:
Jeff Nichols
Apple Inc., United States

Issue’s Table of Contents

Copyright © 2021 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 18 October 2021

Published in PACMHCI Volume 5, Issue CSCW2

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Badges

Honorable Mention

Author Tags

Qualifiers

Research-article

Funding Sources

National Science Foundation

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

38
Total Citations
View Citations
911
Total Downloads

Downloads (Last 12 months)260
Downloads (Last 6 weeks)30

Reflects downloads up to 09 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Muralikumar MMcDonald D(2025)An Emerging Design Space of How Tools Support Collaborations in AI Design and DevelopmentProceedings of the ACM on Human-Computer Interaction10.1145/37011819:GROUP(1-28)Online publication date: 10-Jan-2025
https://dl.acm.org/doi/10.1145/3701181
Longpre SMahari RObeng-Marnu NBrannon WSouth TGero KPentland SKabbara JSalakhutdinov RKolter ZHeller KWeller AOliver NScarlett JBerkenkamp F(2024)Position: data authenticity, consent, & provenance for AI are all brokenProceedings of the 41st International Conference on Machine Learning10.5555/3692070.3693398(32711-32725)Online publication date: 21-Jul-2024
https://dl.acm.org/doi/10.5555/3692070.3693398
M RPankaj D(2024)Social-sum-Mal: A Dataset for Abstractive Text Summarization in MalayalamACM Transactions on Asian and Low-Resource Language Information Processing10.1145/369610723:11(1-20)Online publication date: 21-Nov-2024
https://dl.acm.org/doi/10.1145/3696107
Widder DDabbish LHerbsleb JMartelaro N(2024)Power and Play: Investigating "License to Critique" in Teams' AI Ethics DiscussionsProceedings of the ACM on Human-Computer Interaction10.1145/36869388:CSCW2(1-23)Online publication date: 8-Nov-2024
https://dl.acm.org/doi/10.1145/3686938
Gray CChivukula SJohns JWill MObi ILi Z(2024)Languaging Ethics in Technology PracticeACM Journal on Responsible Computing10.1145/36564681:2(1-15)Online publication date: 20-Jun-2024
https://dl.acm.org/doi/10.1145/3656468
Tan MLee HWang DSubramonyam H(2024)Is a Seat at the Table Enough? Engaging Teachers and Students in Dataset Specification for ML in EducationProceedings of the ACM on Human-Computer Interaction10.1145/36373588:CSCW1(1-32)Online publication date: 26-Apr-2024
https://dl.acm.org/doi/10.1145/3637358
Bhardwaj EGujral HWu SZogheib CMaharaj TBecker C(2024)Machine learning data practices through a data curation lens: An evaluation frameworkProceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency10.1145/3630106.3658955(1055-1067)Online publication date: 3-Jun-2024
https://dl.acm.org/doi/10.1145/3630106.3658955
Berman GGoyal NMadaio M(2024)A Scoping Study of Evaluation Practices for Responsible AI Tools: Steps Towards Effectiveness EvaluationsProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642398(1-24)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642398
McMillan-Major ABender EFriedman B(2024)Data Statements: From Technical Concept to Community PracticeACM Journal on Responsible Computing10.1145/35947371:1(1-17)Online publication date: 20-Mar-2024
https://dl.acm.org/doi/10.1145/3594737
Ataguba GRiyadh MOkaiyeto SDaniel JXiao HOrji R(2024)Ethical Practices for Collecting Ground-Truth Food Datasets: A Systematic Review2024 IEEE Conference on Artificial Intelligence (CAI)10.1109/CAI59869.2024.00105(530-536)Online publication date: 25-Jun-2024
https://doi.org/10.1109/CAI59869.2024.00105
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents