More Web Proxy on the site http://driver.im/

research-article

Provenance-Enabled Explainable AI

Authors:

Benjamin E. UjcichAuthors Info & Claims

Proceedings of the ACM on Management of Data, Volume 2, Issue 6

Article No.: 250, Pages 1 - 27

https://doi.org/10.1145/3698826

Published: 20 December 2024 Publication History

Abstract

Machine learning (ML) algorithms have advanced significantly in recent years, progressively evolving into artificial intelligence (AI) agents capable of solving complex, human-like intellectual challenges. Despite the advancements, the interpretability of these sophisticated models lags behind, with many ML architectures remaining "black boxes" that are too intricate and expansive for human interpretation. Recognizing this issue, there has been a revived interest in the field of explainable AI (XAI) aimed at explaining these opaque ML models. However, XAI tools often suffer from being tightly coupled with the underlying ML models and are inefficient due to redundant computations. We introduce provenance-enabled explainable AI (PXAI). PXAI decouples XAI computation from ML models through a provenance graph that tracks the creation and transformation of all data within the model. PXAI improves XAI computational efficiency by excluding irrelevant and insignificant variables and computation in the provenance graph. Through various case studies, we demonstrate how PXAI enhances computational efficiency when interpreting complex ML models, confirming its potential as a valuable tool in the field of XAI.

References

[1]

Somak Aditya, Yezhou Yang, and Chitta Baral. 2018. Explicit Reasoning over End-to-End Neural Architectures for Visual Question Answering. In AAAI.

[2]

Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus-Robert Muller, and Wojciech Samek. 2015. On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PloS one 10, 7 (2015), e0130140.

[3]

Stephen H. Bach, Matthias Broecheler, Bert Huang, and Lise Getoor. 2015. Hinge-Loss Markov Random Fields and Probabilistic Soft Logic. Journal of Machine Learning Research 18 (2015), 109:1--109:67.

[4]

Nathalie Baracaldo, Bryant Chen, Heiko Ludwig, and Jaehoon Amir Safavi. 2017. Mitigating Poisoning Attacks on Machine Learning Models: A Data Provenance Based Approach. In Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security (Dallas, Texas, USA) (AISec '17). Association for Computing Machinery, New York, NY, USA, 103--110. https://doi.org/10.1145/3128572.3140450

Digital Library

[5]

Alejandro Barredo Arrieta, Natalia Díaz-Rodríguez, Javier Del Ser, Adrien Bennetot, Siham Tabik, Alberto Barbado, Salvador Garcia, Sergio Gil-Lopez, Daniel Molina, Richard Benjamins, Raja Chatila, and Francisco Herrera. 2020. Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI. Information Fusion 58 (2020), 82--115. https://doi.org/10.1016/j.inffus.2019.12.012

Digital Library

[6]

Peter Buneman, Sanjeev Khanna, and Tan Wang-Chiew. 2001. Why and where: A characterization of data provenance. In International conference on database theory. Springer, 316--330.

Digital Library

[7]

Rich Caruana, Yin Lou, Johannes Gehrke, Paul Koch, Marc Sturm, and Noemie Elhadad. 2015. Intelligible Models for HealthCare: Predicting Pneumonia Risk and Hospital 30-Day Readmission. In Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (Sydney, NSW, Australia) (KDD '15). Association for Computing Machinery, New York, NY, USA, 1721--1730. https://doi.org/10.1145/2783258.2788613

Digital Library

[8]

Adriane P. Chapman, H. V. Jagadish, and Prakash Ramanan. 2008. Efficient provenance storage. In Proceedings of the 2008 ACM SIGMOD International Conference on Management of Data (Vancouver, Canada) (SIGMOD '08). Association for Computing Machinery, New York, NY, USA, 993--1006. https://doi.org/10.1145/1376616.1376715

Digital Library

[9]

Daniel Crankshaw, Xin Wang, Guilio Zhou, Michael J. Franklin, Joseph E. Gonzalez, and Ion Stoica. 2017. Clipper: A Low-Latency Online Prediction Serving System. In 14th USENIX Symposium on Networked Systems Design and Implementation (NSDI 17). USENIX Association, Boston, MA, 613--627. https://www.usenix.org/conference/nsdi17/ technical-sessions/presentation/crankshaw

[10]

Susanne Dandl, Christoph Molnar, Martin Binder, and Bernd Bischl. 2020. Multi-objective counterfactual explanations. In Parallel Problem Solving from Nature--PPSN XVI: 16th International Conference, PPSN 2020, Leiden, The Netherlands, September 5--9, 2020, Proceedings, Part I. Springer, 448--469.

[11]

Li Deng. 2012. The mnist database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine 29, 6 (2012), 141--142.

[12]

Daniel Deutch, Amir Gilad, and Yuval Moskovitch. 2015. Selective provenance for datalog programs using top-k queries. Proc. VLDB Endow. 8, 12 (aug 2015), 1394--1405. https://doi.org/10.14778/2824032.2824039

Digital Library

[13]

Hailun Ding, Juan Zhai, Dong Deng, and Shiqing Ma. 2023. The Case for Learned Provenance Graph Storage Systems. In 32nd USENIX Security Symposium (USENIX Security 23). USENIX Association, Anaheim, CA, 3277--3294. https://www.usenix.org/conference/usenixsecurity23/presentation/ding-hailun-provenance

[14]

Pedro Domingos, Dominik Jain, Stanley Kok, Daniel Lowd, Lily Mihalkova, Hoifung Poon, Matthew Richardson, Parag Singla, Marc Sumner, and Jue Wang. [n. d.]. Alchemy - Open Source AI. http://alchemy.cs.washington.edu/alchemy1. html

[15]

Pedro Domingos and Daniel Lowd. 2009. Markov Logic: An Interface Layer for Artificial Intelligence (1st ed.). Morgan and Claypool Publishers.

[16]

Pedro Domingos and Daniel Lowd. 2019. Unifying logical and statistical AI with Markov logic. Commun. ACM 62, 7 (2019), 74--83.

Digital Library

[17]

Jaimie Drozdal, Justin Weisz, Dakuo Wang, Gaurav Dass, Bingsheng Yao, Changruo Zhao, Michael Muller, Lin Ju, and Hui Su. 2020. Trust in AutoML: exploring information needs for establishing trust in automated machine learning systems. In Proceedings of the 25th International Conference on Intelligent User Interfaces (Cagliari, Italy) (IUI '20). Association for Computing Machinery, New York, NY, USA, 297--307. https://doi.org/10.1145/3377325.3377501

Digital Library

[18]

Patrick Forré and Joris M Mooij. 2017. Markov properties for graphical models with cycles and latent variables. arXiv preprint arXiv:1710.08775 (2017).

[19]

Víctor Garcia Satorras and Max Welling. 2021. Neural Enhanced Belief Propagation on Factor Graphs. In Proceedings of The 24th International Conference on Artificial Intelligence and Statistics (Proceedings of Machine Learning Research, Vol. 130), Arindam Banerjee and Kenji Fukumizu (Eds.). PMLR, 685--693. https://proceedings.mlr.press/v130/garciasatorras21a. html

[20]

Antonio A. Ginart, Melody Y. Guan, Gregory Valiant, and James Zou. 2019. Making AI Forget You: Data Deletion in Machine Learning. Curran Associates Inc., Red Hook, NY, USA.

[21]

Alex Goldstein, Adam Kapelner, Justin Bleich, and Emil Pitkin. 2015. Peeking inside the black box: Visualizing statistical learning with plots of individual conditional expectation. journal of Computational and Graphical Statistics 24, 1 (2015), 44--65.

[22]

Todd J. Green, Grigoris Karvounarakis, and Val Tannen. 2007. Provenance Semirings. In Proceedings of the Twenty-Sixth ACM SIGMOD-SIGACT-SIGART Symposium on Principles of Database Systems (Beijing, China) (PODS '07). Association for Computing Machinery, New York, NY, USA, 31--40. https://doi.org/10.1145/1265530.1265535

Digital Library

[23]

David Gunning and David Aha. 2019. DARPA's explainable artificial intelligence (XAI) program. AI magazine 40, 2 (2019), 44--58.

[24]

Wajih Ul Hassan, Mohammad A. Noureddine, Pubali Datta, and Adam Bates. 2020. OmegaLog: High-Fidelity Attack Investigation via Transparent Multi-layer Log Analysis. Proceedings 2020 Network and Distributed System Security Symposium (2020). https://api.semanticscholar.org/CorpusID:211268590

[25]

Lisa Anne Hendricks, Zeynep Akata, Marcus Rohrbach, Jeff Donahue, Bernt Schiele, and Trevor Darrell. 2016. Generating visual explanations. In Computer Vision--ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11--14, 2016, Proceedings, Part IV 14. Springer, 3--19.

[26]

Andreas Holzinger, Randy Goebel, Ruth Fong, Taesup Moon, Klaus-Robert Mueller, and Wojciech Samek. 2022. xxAIbeyond explainable artificial intelligence. In xxAI-Beyond Explainable AI: International Workshop, Held in Conjunction with ICML 2020, July 18, 2020, Vienna, Austria, Revised and Extended Papers. Springer, 3--10.

[27]

Die Hu, Dan Feng, Yulai Xie, Gongming Xu, Xinrui Gu, and Darrell Long. 2020. Efficient Provenance Management via Clustering and Hybrid Storage in Big Data Environments. IEEE Transactions on Big Data 6, 4 (2020), 792--803. https://doi.org/10.1109/TBDATA.2019.2907116

[28]

Been Kim, Rajiv Khanna, and Oluwasanmi O Koyejo. 2016. Examples are not enough, learn to criticize! criticism for interpretability. Advances in neural information processing systems 29 (2016).

[29]

Daphne Koller and Nir Friedman. 2009. Probabilistic graphical models: principles and techniques. MIT press.

Digital Library

[30]

Kyu Hyung Lee, Xiangyu Zhang, and Dongyan Xu. 2013. High Accuracy Attack Provenance via Binary-based Execution Partition. In 20th Annual Network and Distributed System Security Symposium, NDSS 2013, San Diego, California, USA, February 24--27, 2013. The Internet Society. https://www.ndss-symposium.org/ndss2013/high-accuracy-attackprovenance- binary-based-execution-partition

[31]

Yunseong Lee, Alberto Scolari, Byung-Gon Chun, Marco Domenico Santambrogio, Markus Weimer, and Matteo Interlandi. 2018. {PRETZEL}: Opening the black box of machine learning prediction serving systems. In 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI 18). 611--626.

[32]

H-A Loeliger. 2004. An introduction to factor graphs. IEEE Signal Processing Magazine 21, 1 (2004), 28--41.

[33]

Scott M Lundberg and Su-In Lee. 2017. A unified approach to interpreting model predictions. Advances in neural information processing systems 30 (2017).

[34]

Shiqing Ma, Yousra Aafer, Zhaogui Xu, Wen-Chuan Lee, Juan Zhai, Yingqi Liu, and Xiangyu Zhang. 2017. LAMP: Data Provenance for Graph Based Machine Learning Algorithms through Derivative Computation. In Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering (Paderborn, Germany) (ESEC/FSE 2017). Association for Computing Machinery, New York, NY, USA, 786--797. https://doi.org/10.1145/3106237.3106291

Digital Library

[35]

Stephen Macke, Hongpu Gong, Doris Jung-Lin Lee, Andrew Head, Doris Xin, and Aditya Parameswaran. 2021. Fine-Grained Lineage for Safer Notebook Interactions. Proc. VLDB Endow. 14, 6 (feb 2021), 1093--1101. https: //doi.org/10.14778/3447689.3447712

Digital Library

[36]

Tova Milo, Yuval Moskovitch, and Brit Youngmann. 2020. Contribution Maximization in Probabilistic Datalog. In 2020 IEEE 36th International Conference on Data Engineering (ICDE). 817--828. https://doi.org/10.1109/ICDE48307.2020.00076

[37]

Ramaravind K Mothilal, Amit Sharma, and Chenhao Tan. 2020. Explaining machine learning classifiers through diverse counterfactual explanations. In Proceedings of the 2020 conference on fairness, accountability, and transparency. 607--617.

Digital Library

[38]

Kevin Murphy, Yair Weiss, and Michael I Jordan. 2013. Loopy belief propagation for approximate inference: An empirical study. arXiv preprint arXiv:1301.6725 (2013).

[39]

Rohan Paris. [n. d.]. Credit Score Classification. https://www.kaggle.com/datasets/parisrohan/credit-score-classification

[40]

Arnab Phani, Benjamin Rath, and Matthias Boehm. 2021. LIMA: Fine-Grained Lineage Tracing and Reuse in Machine Learning Systems. In Proceedings of the 2021 International Conference on Management of Data (Virtual Event, China) (SIGMOD '21). Association for Computing Machinery, New York, NY, USA, 1426--1439. https://doi.org/10.1145/3448016. 3452788

Digital Library

[41]

João Felipe Pimentel, Leonardo Murta, Vanessa Braganholo, and Juliana Freire. 2017. noWorkflow: a tool for collecting, analyzing, and managing provenance from python scripts. Proc. VLDB Endow. 10, 12 (aug 2017), 1841--1844. https: //doi.org/10.14778/3137765.3137789

Digital Library

[42]

Joelle Pineau, Philippe Vincent-Lamarre, Koustuv Sinha, Vincent Larivière, Alina Beygelzimer, Florence d'Alché Buc, Emily Fox, and Hugo Larochelle. 2021. Improving Reproducibility in Machine Learning Research (a Report from the NeurIPS 2019 Reproducibility Program). J. Mach. Learn. Res. 22, 1, Article 164 (jan 2021), 20 pages.

[43]

Inc. Preferred Networks. 2021. Overview of PyTorch Autograd Engine. https://pytorch.org/blog/overview-of-pytorchautograd- engine

[44]

Atul Rawal, James McCoy, Danda B. Rawat, Brian M. Sadler, and Robert St. Amant. 2022. Recent Advances in Trustworthy Explainable Artificial Intelligence: Status, Challenges, and Perspectives. IEEE Transactions on Artificial Intelligence 3, 6 (2022), 852--866. https://doi.org/10.1109/TAI.2021.3133846

[45]

Christopher Ré and Dan Suciu. 2008. Approximate lineage for probabilistic databases. In PVLDB. 797--808.

[46]

Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016. " Why should i trust you?" Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining. 1135--1144.

Digital Library

[47]

Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2018. Anchors: High-precision model-agnostic explanations. In Proceedings of the AAAI conference on artificial intelligence, Vol. 32.

[48]

Matthew Richardson and Pedro Domingos. 2006. Markov Logic Networks. Mach. Learn. 62, 1--2 (Feb. 2006), 107--136.

Digital Library

[49]

Wojciech Samek, Grégoire Montavon, Sebastian Lapuschkin, Christopher J. Anders, and Klaus-Robert Muller. 2021. Explaining Deep Neural Networks and Beyond: A Review of Methods and Applications. Proc. IEEE 109, 3 (2021), 247--278. https://doi.org/10.1109/JPROC.2021.3060483

[50]

Sebastian Schelter, Joos-Hendrik Böse, Johannes Kirschnick, Thoralf Klein, and Stephan Seufert. 2017. Automatically tracking metadata and provenance of machine learning experiments. In NeurIPS 2017. https://www.amazon.science/ publications/automatically-tracking-metadata-and-provenance-of-machine-learning-experiments

[51]

Jeremy Siek, Lie-Quan Lee, and Andrew Lumsdaine. [n. d.]. The Boost Graph Library (BGL). https://www.boost.org/ doc/libs/1_80_0/libs/graph/doc/index.html

[52]

Parag Singla and Pedro M Domingos. 2008. Lifted First-Order Belief Propagation. In AAAI, Vol. 8. 1094--1099.

Digital Library

[53]

Congzheng Song and Vitaly Shmatikov. 2019. Auditing data provenance in text-generation models. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 196--206.

Digital Library

[54]

Evan R Sparks, Shivaram Venkataraman, Tomer Kaftan, Michael J Franklin, and Benjamin Recht. 2017. Keystoneml: Optimizing pipelines for large-scale advanced analytics. In 2017 IEEE 33rd international conference on data engineering (ICDE). IEEE, 535--546.

[55]

Gabriele Tolomei, Fabrizio Silvestri, Andrew Haines, and Mounia Lalmas. 2017. Interpretable predictions of treebased ensembles via actionable feature tweaking. In Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining. 465--474.

Digital Library

[56]

Paul Voigt and Axel Von dem Bussche. 2017. The eu general data protection regulation (gdpr). A Practical Guide, 1st Ed., Cham: Springer International Publishing 10, 3152676 (2017), 10--5555.

[57]

Sandra Wachter, Brent Mittelstadt, and Chris Russell. 2017. Counterfactual explanations without opening the black box: Automated decisions and the GDPR. Harv. JL & Tech. 31 (2017), 841.

[58]

Shaobo Wang, Hui Lyu, Jiachi Zhang, Chenyuan Wu, Xinyi Chen, Wenchao Zhou, Boon Thau Loo, Susan B. Davidson, and Chen Chen. 2020. Provenance for Probabilistic Logic Programs. In Extending Database Technology. 145--156.

[59]

Yinjun Wu, Val Tannen, and Susan B Davidson. 2020. Priu: A provenance-based approach for incrementally updating regression models. In Proceedings of the 2020 ACM SIGMOD International Conference on Management of Data. 447--462.

Digital Library

[60]

Doris Xin, Stephen Macke, Litian Ma, Jialin Liu, Shuchen Song, and Aditya Parameswaran. 2018. HELIX: Holistic Optimization for Accelerating Iterative Machine Learning. Proc. VLDB Endow. 12, 4 (dec 2018), 446--460. https: //doi.org/10.14778/3297753.3297763

Digital Library

[61]

Hao Yuan, Jiliang Tang, Xia Hu, and Shuiwang Ji. 2020. XGNN: Towards Model-Level Explanations of Graph Neural Networks. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (Virtual Event, CA, USA) (KDD '20). Association for Computing Machinery, New York, NY, USA, 430--438. https: //doi.org/10.1145/3394486.3403085

Digital Library

[62]

Luisa M Zintgraf, Taco S Cohen, Tameem Adel, and Max Welling. 2017. Visualizing deep neural network decisions: Prediction difference analysis. arXiv preprint arXiv:1702.04595 (2017).

Cited By

Wang HXie WMa JLi DLi YCai JKankanhalli MPrabhakaran BBoll SSubramanian RZheng LSingh VCesar PXie LXu D(2024)FedSLS: Exploring Federated Aggregation in Saliency Latent SpaceProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3681278(7182-7190)Online publication date: 28-Oct-2024
https://dl.acm.org/doi/10.1145/3664647.3681278

Index Terms

Provenance-Enabled Explainable AI

Recommendations

Methods and standards for research on explainable artificial intelligence: Lessons from intelligent tutoring systems
Abstract
The DARPA Explainable Artificial Intelligence (AI) (XAI) Program focused on generating explanations for AI programs that use machine learning techniques. This article highlights progress during the DARPA Program (2017‐2021) relative to research ...

Lessons learned in the work on intelligent tutoring systems that apply to system design in Explainable AI. image image
On the role of knowledge graphs in explainable AI

The current hype of Artificial Intelligence (AI) mostly refers to the success of machine learning and its sub-domain of deep learning. However, AI is also about other areas, such as Knowledge Representation and Reasoning, or Distributed AI, i.e., areas ...
Explainable AI and Fuzzy Logic Systems
Theory and Practice of Natural Computing
Abstract
The recent advances in computing power coupled with the rapid increases in the quantity of available data has led to a resurgence in the theory and applications of Artificial Intelligence (AI). However, the use of complex AI algorithms like Deep ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Proceedings of the ACM on Management of Data

Proceedings of the ACM on Management of Data Volume 2, Issue 6

SIGMOD

December 2024

792 pages

EISSN:2836-6573

DOI:10.1145/3709598

Editor:
Divyakant Agrawal
University of California, Santa Barbara, United States

Issue’s Table of Contents

Copyright © 2024 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 20 December 2024

Published in PACMMOD Volume 2, Issue 6

Permissions

Request permissions for this article.

Request Permissions

Author Tags

Qualifiers

Research-article

Funding Sources

National Science Foundation

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1
Total Citations
View Citations
28
Total Downloads

Downloads (Last 12 months)28
Downloads (Last 6 weeks)28

Reflects downloads up to 03 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Wang HXie WMa JLi DLi YCai JKankanhalli MPrabhakaran BBoll SSubramanian RZheng LSingh VCesar PXie LXu D(2024)FedSLS: Exploring Federated Aggregation in Saliency Latent SpaceProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3681278(7182-7190)Online publication date: 28-Oct-2024
https://dl.acm.org/doi/10.1145/3664647.3681278

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents