Code-ifying the Law: How Disciplinary Divides Afflict the Development of Legal Software
Article No.: 398, Pages 1 - 37
Abstract
Proponents of legal automation believe that translating the law into code can improve the legal system. However, research and reporting suggest that legal software systems often contain flawed translations of the law, resulting in serious harms such as terminating children's healthcare and charging innocent people with fraud. Efforts to identify and contest these mistranslations after they arise treat the symptoms of the problem, but fail to prevent them from emerging. Meanwhile, existing recommendations to improve the development of legal software remain untested, as there is little empirical evidence about the translation process itself. In this paper, we investigate the behavior of fifteen teams---nine composed of only computer scientists and six of computer scientists and legal experts---as they attempt to translate a bankruptcy statute into software. Through an interpretative qualitative analysis, we characterize a significant epistemic divide between computer science and law and demonstrate that this divide contributes to errors, misunderstandings, and policy distortions in the development of legal software. Even when development teams included legal experts, communication breakdowns meant that the resulting tools predominantly presented incorrect legal advice and adopted inappropriately harsh legal standards. Study participants did not recognize the errors in the tools they created. We encourage policymakers and researchers to approach legal software with greater skepticism, as the disciplinary divide between computer science and law creates an endemic source of error and mistranslation in the production of legal software.
References
[1]
Abdi Aidid and Benjamin Alarie. 2023. The Legal Singularity: How Artificial Intelligence Can Make Law Radically Better. University of Toronto Press.
[2]
Julia Angwin, Jeff Larson, Surya Mattu, and Lauren Kirchner. 2016. Machine Bias. ProPublica (2016). https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing Retrieved September 23, 2024 from
[3]
Azer Bestavros, Stacey Dogan, Paul Ohm, and Andrew Sellars. 2022. Bridging the Computer Science--Law Divide. Technical Report. Georgetown Law, Institute for Technology Law & Policy.
[4]
David A. Binder and Paul Bergman. 2003. Taking Lawyering Skills Training Seriously. Clinical Law Review, Vol. 10 (2003), 191--220. https://doi.org/10.2139/ssrn.470903
[5]
Anna Bobkowska and Magdalena Kowalska. 2010. On Efficient Collaboration Between Lawyers and Software Engineers When Transforming Legal Regulations to Law-related Requirements. In 2010 2nd International Conference on Information Technology. IEEE, 105--109.
[6]
Valerie Braithwaite. 2020. Beyond the Bubble that is Robodebt: How Governments that Lose Integrity Threaten Democracy. Australian Journal of Social Issues, Vol. 55, 3 (2020), 242--259. https://doi.org/10.1002/ajs4.122
[7]
Travis D. Breaux and Thomas Norton. 2022. Legal Accountability as Software Quality: A U.S. Data Processing Perspective. In 2022 IEEE 30th International Requirements Engineering Conference (RE). IEEE, 101--113. https://doi.org/10.1109/RE54965.2022.00016
[8]
Noah Bunnell. 2020. Remedying Public-Sector Algorithmic Harms: The Case for Local and State Regulation via Independent Agency. Columbia Journal of Law & Social Problems, Vol. 54 (2020), 261--303.
[9]
Ryan Calo and Danielle Keats Citron. 2021. The Automated Administrative State: A Crisis of Legitimacy. Emory Law Journal, Vol. 70 (2021), 797--845.
[10]
Terry Carney. 2019. Robo-Debt Illegality: The Seven Veils of Failed Guarantees of the Rule of Law? Alternative Law Journal, Vol. 44, 1 (2019), 4--10. https://doi.org/10.1177/1037969X18815913
[11]
Christian Chessman. 2017. A “Source” of Error: Computer Code, Criminal Defendants, and the Constitution. California Law Review (2017), 179--228. https://doi.org/10.2139/ssrn.2707101
[12]
Danielle Keats Citron. 2008. Technological Due Process. Washington University Law Review, Vol. 85, 6 (2008), 1249--1313.
[13]
Jennifer Cobbe. 2020. Legal Singularity and the Reflexivity of Law. In Is Law Computable? Critical Perspectives on Law and Artificial Intelligence. Ed. Simon Deakin and Christopher Markou, Hart Publishing, 107--134. https://doi.org/10.2139/ssrn.3858474
[14]
Mark Cohen. 2020. The FT Innovative Lawyers-Global Legal Hackathon: Meeting The Challenge Of Law In The Digital Age. Forbes (2020). https://www.forbes.com/sites/markcohen1/2020/05/26/the-ft-innovative-lawyers-global-legal-hackathon-meeting-the-challenge-of-law-in-the-digital-age/ Retrieved September 23, 2024 from
[15]
Colin Lecher. 2018. What Happens When an Algorithm Cuts Your Health Care. The Verge (2018). https://www.theverge.com/2018/3/21/17144260/healthcare-medicaid-algorithm-arkansas-cerebral-palsy Retrieved September 23, 2024 from
[16]
Joseph Dainow. 1966. The Civil Law and the Common Law: Some Points of Comparison. The American Journal of Comparative Law, Vol. 15, 3 (1966), 419--435. https://doi.org/10.2307/838275
[17]
David Freeman Engstrom, Daniel E. Ho, Catherine M Sharkey, and Mariano-Florentino Cuéllar. 2020. Government by Algorithm: Artificial Intelligence in Federal Administrative Agencies. NYU School of Law, Public Law Research Paper 20--54 (2020). https://doi.org/10.2139/ssrn.3551505
[18]
Nel Escher and Nikola Banovic. 2020. Exposing Error in Poverty Management Technology: A Method for Auditing Government Benefits Screening Tools. In Proceedings of the ACM on Human-Computer Interaction (CSCW1, Vol. 4). Association for Computing Machinery, New York, NY, 1--20. https://doi.org/10.1145/3392874
[19]
Virginia Eubanks. 2018. Automating Inequality: How High-Tech Tools Profile, Police, and Punish the Poor. St. Martin's Press.
[20]
Matthew S. Farina. 2021. Schoolbooks and Shackles: The Undue Hardship Standard and Treatment of Student Debt at Bankruptcy. Boston College Law Review, Vol. 62, 5 (2021), 1620--1664.
[21]
Katherine Fink. 2018. Opening the Government's Black Boxes: Freedom of Information and Algorithmic Accountability. Information, Communication & Society, Vol. 21, 10 (2018), 1453--1471. https://doi.org/10.1080/1369118X.2017.1330418
[22]
Kenneth R Fleischmann. 2006. Boundary Objects with Agency: A Method for Studying the Design--Use Interface. The Information Society, Vol. 22, 2 (2006), 77--87. https://doi.org/10.1080/01972240600567188
[23]
Michael Genesereth. 2015. Computational Law: The Cop in the Backseat. White Paper, CodeX: The Center for Legal Informatics Stanford University (2015).
[24]
Daniel Goldsworthy. 2019. Dworkin's Dream: Towards a Singularity of Law. Alternative Law Journal, Vol. 44, 4 (2019), 286--290. https://doi.org/10.1177/1037969X19875825
[25]
Ben Green. 2022. The Flaws of Policies Requiring Human Oversight of Government Algorithms. Computer Law & Security Review, Vol. 45 (2022), 105681. https://doi.org/10.1016/j.clsr.2022.105681
[26]
Ben Green and Salomé Viljoen. 2020. Algorithmic Realism: Expanding the Boundaries of Algorithmic Thought. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. ACM, Barcelona Spain, 19--31. https://doi.org/10.1145/3351095.3372840
[27]
Alexandria Hansen, Jim Gribble, Amber Moran, Eric Hansen, and Danielle Harlow. 2021. Making Computer Science Accessible. Science and Children, Vol. 58, 5 (2021), 80--86.
[28]
HelloPrenup. 2023. Online Prenuptial Agreements - Affordable Prenups. https://helloprenup.com/ Retrieved September 23, 2024 from
[29]
Mireille Hildebrandt. 2020. Code-Driven Law: Freezing the Future and Scaling the Past. In Is Law Computable? Critical Perspectives on Law and Artificial Intelligence. Ed. Simon Deakin and Christopher Markou, Hart Publishing, 67--84. https://doi.org/10.2139/ssrn.3522079
[30]
Mireille Hildebrandt. 2023. Grounding Computational `Law' in Legal Education and Professional Legal Training. In Research Handbook on Law and Technology. Ed. Bartosz Bro'ek, Olia Kanevskaia, and Przemys'aw Pa'ka, Elgar Publishing, 99--127. https://doi.org/10.4337/9781803921327.00014
[31]
Dominique Hogan-Doran. 2017. Computer Says “No”: Automation, Algorithms and Artificial Intelligence in Government Decision-Making. Judicial Review: Selected Conference Papers: Journal of the Judicial Commission of New South Wales, Vol. 13, 3 (2017), 345--382.
[32]
Immi. 2023. Do You Know Your Path to Legal Immigration Status? https://www.immi.org/en Retrieved September 23, 2024 from
[33]
Jason Iuliano. 2012. An Empirical Assessment of Student Loan Discharges and the Undue Hardship Standard. The American Bankruptcy Law Journal, Vol. 86, 3 (2012), 495--526. https://doi.org/10.2139/ssrn.1894445
[34]
Tech For Justice. 2017. Tech For Justice Hackathon Veterans - Project Gallery. https://tech-for-justice-veterans.devpost.com/project-gallery Retrieved September 23, 2024 from
[35]
David Kennedy and William W. Fisher. 2007. The Canon of American Legal Thought. Princeton University Press. https://doi.org/10.2307/j.ctv39x51k
[36]
Marc Lauritsen and Quinten Steenhuis. 2019. Substantive Legal Software Quality: A Gathering Storm?. In Proceedings of the Seventeenth International Conference on Artificial Intelligence and Law (ICAIL '19). Association for Computing Machinery, New York, NY, 52--62. https://doi.org/10.1145/3322640.3326706
[37]
Brian Leiter. 1999. Positivism, Formalism, Realism. Columbia Law Review, Vol. 99 (1999), 1138--1164. https://doi.org/10.2307/1123484
[38]
Victor Li. 2017. Chatbot to Help Victims of Military Sex Trauma Wins Tech for Justice Hackathon Event. ABA Journal (2017). https://www.abajournal.com/news/article/chatbot_to_help_victims_of_military_sex_trauma_wins_tech_for_justice_hackat Retrieved September 23, 2024 from
[39]
Thomas Lodato and Carl DiSalvo. 2018. Institutional Constraints: the Forms and Limits of Participatory Design in the Public Realm. In Proceedings of the 15th Participatory Design Conference: Full Papers-Volume 1. 1--12. https://doi.org/10.1145/3210586.3210595
[40]
Nathaniel Love and Michael Genesereth. 2005. Computational Law. In Proceedings of the 10th International Conference on Artificial Intelligence and Law (ICAIL '05). Association for Computing Machinery, New York, NY, 205--209. https://doi.org/10.1145/1165485.1165517
[41]
Steve McConnell. 2004. Code Complete. Pearson Education.
[42]
Mona Nikidehaghani, Jane Andrew, and Corinne Cortese. 2022. Algorithmic Accountability: Robodebt and the Making of Welfare Cheats. Accounting, Auditing & Accountability Journal, Vol. 36, 2 (2022), 677--711. https://doi.org/10.1108/AAAJ-02--2022--5666
[43]
Don Norman. 2013. The Design of Everyday Things: Revised and Expanded Edition. Basic Books.
[44]
Law Help NY. 2023. I am Serving Active Duty and Facing Eviction. What Can I Do? https://www.lawhelpny.org/resource/letter-to-landlord-eviction Retrieved September 23, 2024 from
[45]
House of Delegates. 2020. Model Rules of Professional Conduct. American Bar Association. https://www.americanbar.org/groups/professional_responsibility/publications/model_rules_of_professional_conduct Retrieved September 23, 2024 from
[46]
Colorado Department of Health Care Policy & Financing. 2024. Care and Case Management System. https://hcpf.colorado.gov/care-case-management-system Retrieved September 23, 2024 from
[47]
The IEEE Global Initiative on Ethics of Autonomous and Intelligent Systems. 2019. Ethically Aligned Design: A Vision for Prioritizing Human Well-being with Autonomous and Intelligent Systems. IEEE (2019).
[48]
Frank Pasquale. 2016. Automating the Professions: Utopian Pipe Dream or Dystopian Nightmare. Los Angeles Review of Books (2016).
[49]
Frank Pasquale. 2019. A Rule of Persons, Not Machines: The Limits of Legal Automation. The George Washington Law Review, Vol. 87 (2019), 1--55.
[50]
Jeanne Frazier Price. 2013. Wagging, Not Barking: Statutory Definitions. Cleveland State Law Review, Vol. 60, 4 (2013), 999--1055.
[51]
In re Long. 2003. 322 F.3d 549 (2003).
[52]
Tapani Rinta-Kahila, Ida Someh, Nicole Gillespie, Marta Indulska, and Shirley Gregor. 2023. Managing Unintended Consequences of Algorithmic Decision-Making: The Case of Robodebt. Journal of Information Technology Teaching Cases (2023), 165--171. https://doi.org/10.1177/20438869231165538
[53]
Andrea Roth. 2016. Trial by Machine. Georgetown Law Journal, Vol. 104, 5 (2016), 1245--1305.
[54]
Andrea L Roth. 2017. Machine Testimony. Yale Law Journal, Vol. 126, 1 (2017), 1973--2053.
[55]
Pierre Schlag. 1985. Rules and Standards. UCLA Law Review, Vol. 33 (1985), 379--430.
[56]
Suzanne J. Schmitz and Alice M. Noble-Allgire. 2011. Reinvigorating the 1L Curriculum: Sequenced Writing Across the Curriculum Assignments as the Foundation for Producing Practice-Ready Law Graduates. Southern Illinois University Law Journal, Vol. 36 (2011), 287--315. https://doi.org/10.2139/ssrn.1879618
[57]
Lisa A. Shay, Woodrow Hartzog, John Nelson, and Gregory Conti. 2016. Do Robots Dream of Electric Laws? An Experiment in the Law as Algorithm. In Robot Law. Edward Elgar Publishing, 274--305. https://doi.org/10.4337/9781783476732.00020
[58]
Gilien Silsby. 2018. Global Legal Hackathon: USC Law Prof. Challenges Hackers to Tackle `Problems Worth Solving'. USC Gould School of Law (2018). https://gould.usc.edu/news/global-legal-hackathon-usc-law-prof-challenges-hackers-to-tackle-problems-worth-solving/ Retrieved September 23, 2024 from
[59]
Susan Leigh Star and James R Griesemer. 1989. Institutional Ecology, `Translations' and Boundary Objects: Amateurs and Professionals in Berkeley's Museum of Vertebrate Zoology, 1907--39. Social Studies of Science, Vol. 19, 3 (1989), 387--420. https://doi.org/10.1177/030631289019003001
[60]
Esther Surden. 2017. At CourtHack 2017, Hackers Present Solutions For Real-World Justice System Problems. New Jersey Tech Weekly (2017). https://njtechweekly.com/at-courthack-2017-hackers-present-solutions-for-real-world-justice-system-problems/ Retrieved September 23, 2024 from
[61]
Harry Surden. 2010. The Variable Determinacy Thesis. Columbia Science and Technology Law Review, Vol. 12, 1 (2010). https://doi.org/10.7916/stlr.v12i0.3949
[62]
Richard Susskind. 2008. The End of Lawyers? Rethinking the Nature of Legal Services. Oxford University Press.
[63]
Iddo Tavory and Stefan Timmermans. 2014. Abductive Analysis: Theorizing Qualitative Research. University of Chicago Press.
[64]
Intuit TurboTax. 2023. File Taxes Online, Tax Filing Made Easy. https://turbotax.intuit.com/ Retrieved September 23, 2024 from
[65]
Thomas v. Dep't of Educ. (In re Thomas). 2019. 931 F.3d 449 (2019).
[66]
Brunner v. New York State Higher Educ. Serv. 1987. 831 F.2d 395 (1987).
[67]
Ivory v. United States (In re Ivory). 2001. 269 B.R. 890 (2001).
[68]
Erkson v. U.S. Dept. of Ed. 2018. 582 B.R. 542 (2018).
[69]
Smith v. U.S. Dept. of Ed. 2018. 582 B.R. 556 (2018).
[70]
Sarah Valentine. 2019. Impoverished Algorithms: Misguided Governments, Flawed Technologies, and Social Control. Fordham Urban Law Journal, Vol. 46 (2019), 364--427.
[71]
Ari Ezra Waldman. 2018. Designing Without Privacy. Houston Law Review, Vol. 55 (2018), 659--727.
[72]
Nyu Wang and Michael Yuan Tian. 2022. `Intelligent Justice': AI Implementations in China's Legal Systems. In Artificial Intelligence and Its Discontents: Critiques from the Social Sciences and Humanities. Springer, 197--222. https://doi.org/10.1007/978--3-030--88615--8_10
[73]
Ran Wang. 2020. Legal Technology in Contemporary USA and China. Computer Law & Security Review, Vol. 39 (2020). https://doi.org/10.1016/j.clsr.2020.105459
[74]
Quicken Willmaker. 2024. Estate Planning You Can Trust. https://www.willmaker.com/ Retrieved September 23, 2024 from
[75]
Stephanie Wykstra. 2020. Government's Use of Algorithm Serves Up False Fraud Charges. Undark (2020). https://undark.org/2020/06/01/michigan-unemployment-fraud-algorithm/ Retrieved September 23, 2024 from
Index Terms
- Code-ifying the Law: How Disciplinary Divides Afflict the Development of Legal Software
Recommendations
Legal issues surrounding monitoring during network research
IMC '07: Proceedings of the 7th ACM SIGCOMM conference on Internet measurementThis work was motivated by a discussion that two of the coauthors (computer science professors) had with the other coauthor (a law professor and a former computer crime Trial Attorney at the U.S. Department of Justice), in which it was pointed out that ...
Lessons in Copyright Activism: K-12 Education and the DMCA 1201 Exemption Rulemaking Process
Digital learning is being transformed by changes in copyright law. This article discusses the author's personal journey as a copyright education activist through two rounds of rulemaking proceedings before the Copyright Office concerning the anti-...
The use of legal software by non-lawyers and the perils of unauthorised practice of law charges in the United States: a review of Jayson Reynoso decision
This paper critically reviews the judgment of the United States Court of Appeals for the Ninth Circuit In re: Jayson Reynoso: Frankfort Digital Services et al., v. Sara L. Kistler, United States Trustee et al. (2007) 447 F.3d 1117. The appellants, who ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
November 2024
5177 pages
Copyright © 2024 Owner/Author.
This work is licensed under a Creative Commons Attribution International 4.0 License.
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 08 November 2024
Published in PACMHCI Volume 8, Issue CSCW2
Check for updates
Author Tags
Qualifiers
- Research-article
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 253Total Downloads
- Downloads (Last 12 months)253
- Downloads (Last 6 weeks)253
Reflects downloads up to 17 Dec 2024
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in