A Roadmap of Explainable Artificial Intelligence: Explain to Whom, When, What and How?
Authors: 
Ziming Wang, Changwu Huang, Xin YaoAuthors Info & Claims
Article No.: 20, Pages 1 - 40
Abstract
Explainable artificial intelligence (XAI) has gained significant attention, especially in AI-powered autonomous and adaptive systems (AASs). However, a discernible disconnect exists among research efforts across different communities. The machine learning community often overlooks “explaining to whom,” while the human-computer interaction community has examined various stakeholders with diverse explanation needs without addressing which XAI methods meet these requirements. Currently, no clear guidance exists on which XAI methods suit which specific stakeholders and their distinct needs. This hinders the achievement of the goal of XAI: providing human users with understandable interpretations. To bridge this gap, this article presents a comprehensive XAI roadmap. Based on an extensive literature review, the roadmap summarizes different stakeholders, their explanation needs at different stages of the AI system lifecycle, the questions they may pose, and existing XAI methods. Then, by utilizing stakeholders’ inquiries as a conduit, the roadmap connects their needs to prevailing XAI methods, providing a guideline to assist researchers and practitioners to determine more easily which XAI methodologies can meet the specific needs of stakeholders in AASs. Finally, the roadmap discusses the limitations of existing XAI methods and outlines directions for future research.
1 Introduction
With the rapid development of AI and its growing role in autonomous and adaptive systems (AASs), its impact and application continue to expand, encompassing areas such as face recognition [56], medical diagnosis [37], product recommendation [71], and so on. However, the widespread use of AI, particularly in AASs, presents many ethical challenges, particularly regarding the challenges and risks associated with opaque and unexplainable AI models [92]. Many prevalent machine learning (ML) models are regarded as “black box” models, implying that the decisions made by AI systems lack explainability. This lack of explainability poses significant trust issues for users, especially in critical areas such as healthcare, finance, autonomous driving, and law, where AASs are increasingly adopted. Therefore, explainable AI (XAI) has received a lot of attention in recent years [10, 80], with the goal of providing human users with appropriate explanations that they can understand [10]. Although the article talks about AI systems, almost all discussions are equally applicable to AASs. This is because almost all AASs use AI techniques to a certain extent. Thus, explainability is also essential for decisions made by autonomous systems, especially if the decision will interact with human beings or the natural environment.
To achieve this, in the ML community, many XAI methods have been proposed to interpret the ML model's decision or behavior [10, 80]. However, current research in XAI has encountered criticism, with the metaphorical notion of “inmates running the asylum” [142], highlighting a significant gap between existing XAI methods and the needs of real-world users [120]. This gap arises from the fact that the development of XAI methods is largely based on researchers’ assumptions rather than a thorough understanding of stakeholders’ specific needs [58]. In other words, ML researchers frequently create new XAI methods (i.e., how to explain) without considering the intended audience, simply assuming that the recipients of explanations are either “users” or “AI experts” [22, 72, 205]. Consequently, there have been increasing calls in the ML community to follow a human-centered approach in XAI research [120, 187], especially in AASs that take into account the stakeholders to whom explanations are provided.
In contrast, within the human-computer interaction (HCI) community, a number of studies have independently considered various stakeholders associated with XAI methods (i.e., explain to whom) [47, 67, 72, 94, 115, 121, 123, 138, 140, 145, 171, 179, 182, 210], the stages in the AI system lifecycle at which explanations are provided (i.e., when to explain) [47, 179, 205], and the questions or inquires that stakeholders could pose (i.e., what to explain) [120, 123, 145]. Despite these efforts to explore XAI in different dimensions, critiques have emerged regarding the breadth and depth of such research. Suresh et al. [205] pointed out that the existing research is too coarse in its delineation of stakeholders, while Bingley et al. [24] emphasized the need to focus further on the stakeholders’ need for explanations. Further, Bhatt et al. [22] found that the vast majority of work only identifies what different stakeholders there are, ignoring their specific needs. In addition, the existing research is incomplete in its consideration of the stages of explanation and the questions that stakeholders may ask. Overall, existing studies are unclear about which appropriate XAI methods stakeholders should choose to provide explanations to meet their needs, leaving a gap between achieving the goal of XAI, i.e., to provide human users with appropriate explanations that are easy for them to understand. This issue is particularly pressing in AASs, where appropriate and accurate explanations can directly impact trust and system reliability. Therefore, this article improves existing reviews of XAI by providing a more refined and comprehensive roadmap that links different pillars of XAI together, especially within the context of AASs. In particular, we consider nine different stakeholders and their respective needs for explanations, four stages of explanations that span the entire AI system lifecycle, and nine questions that stakeholders might ask.
To the best of our knowledge, while prior studies have individually delved into issues of explaining to whom, when, what, or how, there exists a conspicuous gap in the research that consolidates all four dimensions together in a single framework, with only limited efforts made to explore the interactions between pairs of these dimensions [47, 120, 205]. As observed by Vermeire et al. [218], there is a lack of thorough and sufficient cross-referencing of relevant studies on XAI from the ML perspective (relevant studies from the ML community) and from the user perspective (relevant studies from the HCI community). This leads to a lack of clarity among stakeholders regarding the availability of suitable XAI methods that can effectively address their needs, affecting the practical utilization of XAI methods [218] and thus hindering the achievement of the goal of XAI, which is to provide human users with explanations that they can understand [10]. Therefore, this article emphasizes the creation of a roadmap that connects stakeholders’ needs with XAI methods, utilizing “what to explain” questions as a bridge to holistically examine the four facets of explaining to whom, when, what, and how. This roadmap provides a guideline to help various stakeholders in AASs select the appropriate XAI method to meet their needs for explanation, so as to achieve the goal of XAI [10].
The main contributions of this article are summarized as follows:
(1)
We introduce a comprehensive ML ecosystem with a fine-grained division of nine different stakeholders. Through an extensive literature review, we have clarified each stakeholder's needs for explanations at different stages of the AI system lifecycle, forming a crucial part of the roadmap.
(2)
A new type of “what to explain” question is proposed to complement the existing work in the literature, further enriching the roadmap.
(3)
By using “what to explain” questions as a bridge to integrate the four dimensions of to whom, when, what, and how to explain, we have established a connection between stakeholders’ needs and the existing XAI methods. This connection forms the core of the roadmap, providing a guideline to help stakeholders choose the appropriate XAI method to meet their needs for explanation. It also enables us to see more clearly and precisely what has been done or is missing in the existing literature.
(4)
Two case studies are provided to illustrate how the roadmap and its guidance can be used to assist stakeholders in AASs in selecting appropriate XAI methods to meet their explanation needs.
(5)
Through our structured and systematic roadmap, major gaps in the current literature are identified from the point of view of the AI system lifecycle, enabling readers to understand in more detail the potential challenges of meeting the goals of XAI in real-world applications of AI and AASs.
The rest of this article is organized as follows. Section 2 describes the integrated structure of our roadmap. Section 3 presents the ML ecosystem proposed in this article, considering various stakeholders and their needs at different stages of the AI system lifecycle. Section 4 introduces the nine “what to explain” questions considered in this article. Section 5 summarizes the 10 categories of XAI methods considered in this article. Section 6 uses the “what to explain” questions as a bridge to connect stakeholders’ needs with XAI methods, thus providing a guideline within our roadmap to help stakeholders select the appropriate XAI method to meet their needs for explanation. Section 7 presents two case studies to demonstrate the practical application of this roadmap and guideline. Section 8 discusses research gaps identified through our study and presents some future research directions. Section 9 concludes the article.
2 Four-Pillar Structure of Our Roadmap
In the realm of XAI, to achieve the goals of XAI, there are four key questions [47, 115]: (1) Whom are the intended audiences or stakeholders for the explanation, and what are their specific needs [72, 171, 210]? (2) When in the AI system lifecycle to provide explanations to the stakeholders [47, 179, 205]? (3) What inquiries will stakeholders pose in their quest for explanations? What do they want [120, 123, 145]? (4) How should these questions be effectively answered [10, 74, 80, 125, 175, 257]? While each of these questions has been independently studied by researchers, there has been a lack of a comprehensive overview that addresses all four questions together and analyzes their interconnections, particularly in the context of AASs.
To bridge this gap, we have taken a holistic approach by summarizing these questions as the four central pillars of XAI, which are critical to applying XAI in a way that meets stakeholders’ needs. These pillars provide the foundation for a complete view of XAI, as illustrated in Figure 1. For each of these pillars, researchers have conducted relevant studies, which are described below individually.
Fig. 1.
2.1 Explain to Whom
Within the HCI community, there are many research efforts that consider different stakeholders [47, 67, 72, 94, 115, 121, 123, 138, 140, 145, 171, 179, 182, 210] in the context of AI and AASs, including AI experts, decision makers, regulators, users, and so on. However, such work has suffered some criticism. Among them, Suresh et al. [205] pointed out that many works have a very coarse granularity of stakeholder categorization. Bhatt et al. [22] found that the vast majority of the work only considered what different stakeholders there are, without clarifying the specific needs of each stakeholder for explanation. In response to these critiques, through extensive literature research, this article considers nine types of stakeholders at a fine-grained level and specifies each stakeholder's need for explanation. For example, the general end-users of autonomous vehicles may need to explain how the system makes navigation decisions in order to build appropriate trust.
2.2 When to Explain
A small number of XAI-related studies have considered when to provide explanations. Among them, Rosenfeld et al. [179] were the first to divide explanations into before the task, continuing explanations throughout the task, and after the task, based on when the explanation is presented. Suresh et al. [205] considered four stages: development, deployment, immediate usage, and downstream impact and linked them to goals and objectives distilled from stakeholders’ needs. Dhanorkar et al. [47] divided the AI system lifecycle into three stages: model development, model validation during proof of concept and model in-production, and related them to AI experts as well as subject matter experts and their motivation for explainability. However, the AI system lifecycles considered in the literature [205] and [47] are incomplete, as they commence from the modeling stage, disregarding the preceding stages. To address this gap, this article considers a more comprehensive AI system lifecycle model proposed in the literature [6, 63, 237].
2.3 What to Explain
The researchers considered various questions that stakeholders might ask to a black-box ML model [120, 123, 145]. Lim and Dey [123] considered five types of questions that end-users of the system might ask. Mohseni et al. [145] reviewed six “what to explain” questions commonly employed in the design of XAI systems. Inspired by the work of Lim and Dey [123], Liao et al. [120] represented users’ needs for explainability in terms of questions a user might ask. Through interviews with 20 designers and user experience practitioners involved in various AI products, they [120] developed an algorithm-based library of XAI questions. However, the questions that may be asked by the stakeholders considered in the existing research are mainly focused on seeking explanations for the decision-making outcomes of the ML model, while neglecting to seek insights within the ML model, such as what representations or knowledge a neural network has learned. This article introduces a new “what to explain” question, focusing on insights into the ML model's internals, as a complement to the existing work, considering a total of nine different “what to explain” questions in a more comprehensive way. For example, in an adaptive recommender system, the general end-users may want to know why the system made a particular recommendation.
2.4 How to Explain
In order to meet stakeholders’ needs for explanations, researchers in the ML community have proposed various XAI approaches, including feature attribution explanation (FAE) [130], partial dependence plot (PDP) [68], counterfactual (CF) [223] , and others. In this article, we consider a total of ten classes of XAI methods in terms of how to explain, based on several widely recognized survey papers [10, 74, 80, 125, 175, 257].
While there are papers addressing to whom, when, what, or how to explain separately, as mentioned previously, no existing work has ever considered all four pillars in a single framework and discussed interplays among the four pillars. A few studies considered two pillars only. Suresh et al. [205] and Dhanorkar et al. [47] considered stakeholders’ needs at different stages of explanation (when), while Liao et al. [120] related the questions that stakeholders would ask (what) to several XAI methods (how). However, the lack of a comprehensive overview encompassing all four pillars has hindered practical applications of XAI methods, as it is unclear what stakeholders’ needs for explanation are, what questions they might ask, as well as which XAI methods stakeholders should use and when to use them to meet their specific needs for explanation.
To fill this gap, this article constructs a roadmap that considers all four pillars of XAI simultaneously in order to present a holistic picture of “when to explain, what to explain, to whom, to meet what need, and how.” Our roadmap involves identifying the stakeholders and their needs for explanations throughout different stages of the AI system lifecycle, determining the questions they would pose to address their needs, and assessing the existing XAI methods that can offer corresponding answers. By establishing these connections among the four pillars, we can gain a comprehensive understanding of the inter-relationships among the four pillars and provide a guideline to assist stakeholders in selecting the appropriate XAI method to meet their needs for explanation.
3 When to Explain, to Whom, and to Meet What Need
In this section, we first introduce the four stages of the AI system lifecycle [6, 63, 237] that are taken into account in this article. Subsequently, we present the nine distinct classes of stakeholders considered and outline their respective needs for explanation at different stages of the AI system lifecycle, thereby establishing the foundation of our roadmap.
3.1 AI System Lifecycle under Consideration
Based on the proposed AI system lifecycle models [6, 63, 237], this article considers four major stages relevant to AI and AASs:
Model Requirement Analysis: At this initial stage, the meticulous definition of project goals and identification of desired functionalities are paramount [83]. For example, in an autonomous vehicle project, stakeholders might specify safety as a primary goal, influencing the selection of algorithms and models. In addition, selecting an appropriate formulation for the specific problem is important [83].
Data Acquisition and Understanding: This stage entails the discovery and integration of relevant datasets. It involves essential tasks such as data cleaning, elimination of inaccuracies or noise, annotation of ground truth labels, comprehensive exploration, and comprehension of the dataset, as well as rigorous quality analysis and evaluation [6, 63]. For example, in an autonomous vehicle application, understanding and processing data from various sensors are vital for accurate navigation.
Deployment, Monitoring, and Interaction: This stage requires tracking the behavior of the ML model to ensure that it performs as expected and to monitor for errors that may occur during the ML model's execution [6, 83]. For example, in an autonomous vehicle, real-time monitoring is crucial to adjust its driving behavior based on road conditions and obstacles. In addition, many stakeholders may interact with the ML model at this stage.3.2 Identify Stakeholders and Their Needs for Explanation at Different Stages
We consider nine stakeholders who directly or indirectly interact with the ML system, forming the ML ecosystem, as shown in Figure 2. From Figure 2, we can see that all stakeholders, except for the decision makers, the subjects of decision, and the data subjects, engage in direct two-way interactions with the ML system. Decision makers interact bidirectionally with both model operators and subjects of decision, receiving the ML system's decision results from the ML model operators and conveying the final decision outcome to the subjects of decision. Data subjects, on the other hand, only engage with the ML system in a unidirectional manner, where the ML system receives data from the data subjects, but there is no reciprocal interaction. Thus, the arrow between the data subjects and the ML system in Figure 2 represents a unidirectional flow. In the rest of this section, we will delve into each class of stakeholders individually, outlining their specific needs for explanation at different stages of the AI system lifecycle.
Fig. 2.
3.2.1 AI Theory Experts.
AI theory experts generally refer to the theoretical researchers who drive the development of AI. Their primary focus lies in exploring the properties, internal information, and logic of black-box models, particularly deep neural networks (DNNs), rather than developing specific applications [94, 171, 181].
■
When to explain: Modeling AI; Deployment, Monitoring and Interaction.
—
Need: Insight and understanding of the internal logic of complex ML models. Complex black-box models undermine people's trust in them by making it impossible to understand their decision logic due to their opaqueness, which largely affects their application and development [79, 167]. AI theory experts aim to leverage XAI to reveal what black-box models have learned, to better understand the decision logic of black box models and their properties, and thus to gain new insights and knowledge about how black box models learn from data [242].
3.2.2 AI Development Experts.
AI development experts primarily refer to professionals involved in building AI applications. AI development experts have different responsibilities as well as the need for explanations at different stages of the AI system lifecycle [108, 137], which are described below.
■
When to explain: Model Requirements Analysis.
—
Need: Comparing analysis of multiple ML models. In the model requirements analysis stage, AI development experts need to select a suitable ML model from a set of candidate models [65]. Solely depending on performance metrics, such as accuracy or F1 score, often proves inadequate for making an informed choice [204]. Instead, they strive to gain deeper insights into the ML models by comprehending their execution details and underlying logic [155]. This deeper comprehension aids in providing guidance for choosing the most suitable model for a given problem. XAI methods can provide insight into models to help AI development experts compare models and architectures [181, 183].
■
When to explain: Data Acquisition and Understanding.
—
Need 1: Insight and understanding of datasets. An ML model is usually built from data, and gaining insight and improving understanding of the dataset is instrumental in building more effective ML models [208]. Therefore, before modeling, AI development experts seek to gain insights from datasets to enhance their comprehension of datasets. Through the application of XAI methods, they can analyze data distribution, data quality, and feature relationships within the data [102].
—
Need 2: Analyzing potential errors, noise, and bias in the dataset. Gudivada et al. [78] emphasized the criticality of high-quality datasets in the development of ML models. However, datasets often contain noise, bias, and even mislabeled samples [201]. Therefore, AI development experts can improve the quality of datasets by using XAI methods to detect and identify potential errors, noise, biases, and other issues in the datasets [141].
■
When to explain: Modeling AI.
—
Need 1: Assisting with feature selection. Feature engineering involves transforming raw data into new representations to address specific data problems [49], thus improving the generalization ability of ML models. This step is often regarded as a time-consuming and technically challenging step [154, 225]. AI development experts can analyze and gain deeper insight into the data and models using XAI methods to guide feature selection.
—
Need 2: Optimizing model architecture and hyper-parameters. As deep learning continues to grow in popularity, identifying optimal combinations of model architecture and hyper-parameters has become increasingly crucial for building high-performing models [225]. AI development experts can leverage XAI methods to guide the tuning of model architectures and hyper-parameter settings, ultimately enhancing model performance.
—
Need 3: Checking the model's decisions. AI development experts possess a natural inclination to delve deeper into the ML model's decision-making process rather than just analyzing its outputs. They strive to comprehend the overall logic inside the constructed model and how it arrives at specific decisions [47]. Therefore, AI development experts rely on XAI methods to interpret the ML model in order to assess whether its behavior aligns with expectations [115]. This understanding is critical in safety-critical AASs like autonomous vehicles.
—
Need 4: Guiding model debugging and error refinement. During the modeling stage, AI development experts continuously debug and iterate to enhance the ML model's performance [30]. They aspire to understand the reasons behind the ML model's underperformance in certain challenging scenarios [22], as well as identify potential biases and discrimination within the ML model. XAI methods offer valuable assistance in diagnosing model issues and can be employed to mitigate bias, ensuring social responsibility and fairness [73].
■
When to explain: Deployment, Monitoring and Interaction.
—
Need 1: Adjusting the ML model to meet the user's expectations and needs. The performance of AI systems sometimes defies user expectations, and this mismatch of expectations becomes a barrier to HCI [153]. In instances where the ML model's decision result does not align with the user's desired outcome, XAI methods can assist AI development experts in adapting the ML model to better match user expectations [47].
—
Need 2: Assessing the impact of dataset shift. After deploying a model, there may be a significant shift in the distribution of predicted data compared to the training data [7, 32], which greatly affects the accuracy and reliability of trained ML models [231]. AI development experts utilize XAI methods to analyze and understand the effects of dataset shifts on the model's decision-making capabilities [22].
3.2.3 Model Validation Team.
The model validation team typically consists of enterprise owners/managers [18] and/or a group of senior experts in the relevant field [72]. Their responsibilities involve reviewing and validating the training data and the ML model to assess the compatibility of the training data with their proprietary data and to determine if the ML model aligns with the specific needs of their enterprise. Additionally, they evaluate whether the ML model meets regulatory requirements in terms of privacy and security [47], ultimately making decisions on whether to trust and deploy the ML model in systems such as autonomous vehicles or adaptive healthcare solutions.
■
When to explain: Data Acquisition and Understanding.
—
Need: Evaluating data suitability. The model validation team requires comprehensive knowledge of the training data to evaluate its compatibility with their proprietary data [47]. Therefore, they rely on XAI methods to gain insights into the training data as well as compare and analyze them along with their proprietary data.
■
When to explain: Modeling AI.
—
Need 1: Reviewing the ML model's decision logic. Through the use of XAI methods, the model validation team seeks to enhance their global and local understanding of the ML model. Their goal is to effectively review and validate whether the ML model's decision logic aligns with the pertinent domain expertise [72, 221].
—
Need 2: Determining compliance with regulations. A paper by the UK's Financial Conduct Authority stated that boardrooms must learn to tackle some significant issues emerging from AI, particularly in terms of ethics, accountability, transparency, and responsibility [64]. The model validation team, especially business owners/managers, relies on XAI methods to review models’ compliance with relevant legal regulations, such as requirements for ethics, privacy, security, and fairness[47].
3.2.4 Model Operators.
Model operators [210] refer to the objects that interact directly with the ML model/system, such as loan officers [2, 11], medical professionals [91], and operators of autonomous systems like drones or self-driving cars.
■
When to explain: Deployment, Monitoring and Interaction.
—
Need: Ensuring correct and efficient interaction. Model operators interact directly with the ML model/system and can utilize XAI methods to help them better understand the ML model and its decisions, so that they can interact with the ML model correctly and efficiently [210].
3.2.5 Decision Makers.
After obtaining the information passed from the model operators, the decision makers are responsible for making the final decision with the aid of model output and their own experience.
■
When to explain: Deployment, Monitoring and Interaction.
—
Need 1: Comprehending specific model decisions. While obtaining the decision results of the ML model, comprehending the rationale behind specific decisions enhances decision makers’ confidence and trust [72]. This understanding allows decision makers to integrate their knowledge with the ML model's information effectively, enabling them to make informed decisions [60, 210].
—
Need 2: Deepening overall understanding of the ML model and improving decisions. Decision makers leverage XAI methods to grasp the decision boundaries, classification thresholds, and decision logic of black box models. This deeper understanding empowers them to deepen their overall understanding of the ML models and refine and improve their own decision logic [72].
3.2.6 Regulators.
Regulators safeguard the interests of the subjects of decision and general end-users [138]. They review models for compliance with relevant standards or regulations, including reviewing models for interpretability and fairness, as well as the allocation of responsibility in the event of an accident, and so on.
■
When to explain: Deployment, Monitoring and Interaction.
—
Need 1: Ancillary model review. In addition to interpretability and transparency, relevant standards or regulations make relevant requirements for other aspects of AI systems. For example, the EU High Level Expert Group on AI has established guidelines for trustworthy AI, which outline requirements for AI safety, robustness, privacy protection, and fairness [41]. XAI methods can provide explanations to assist in reviewing whether AI systems fulfill the stipulations of relevant regulations [2, 73].
—
Need 2: Assisting in apportioning responsibility. When errors or accidents occur, regulators are responsible for attributing accountability [192]. However, the inherent complexity and lack of transparency in AI systems can make this task daunting [144]. XAI methods can assist in apportioning responsibility [53] by revealing the causes of errors and mishaps through techniques such as attribution.
3.2.7 Data Subjects.
The General Data Protection Regulation (GDPR) defines a data subject as “who can be identified, directly or indirectly, in particular by reference to an identifier such as a name, an identification number, location data, an online identifier or to one or more factors specific to the physical, physiological, genetic, mental, economic, cultural or social identity of that natural person” [214]. In this article, data subjects are considered objects whose personal data are utilized for model training but do not directly interact with the ML model.
■
When to explain: Modeling AI; Deployment, Monitoring and Interaction.
—
Need: Protecting personal data information. Data subjects seek to understand the types of information utilized by the ML model and how such information influences the ML model's decisions, along with the potential implications [205, 210]. This understanding is crucial for protecting their lawful rights concerning personal data, as outlined in the relevant legal frameworks [70].
3.2.8 Subjects of Decision.
Subjects of decision are those who are affected by the decisions made by decision makers [210]. Although they do not interact directly with the ML system, they are directly affected by its output [138], such as loan applicants or patients in healthcare AASs.
■
When to explain: Deployment, Monitoring and Interaction.
—
Need 1: Understanding the decision and how to maintain or change it. The subjects of decision seek to comprehend how the ML model makes a particular decision and how they can maintain or alter the outcome to align with their expectations.
—
Need 2: Examining bias. Subjects of decision aim to ensure that the decisions obtained are fair [52] and ethical [97]. However, AI systems often exhibit biases, discrimination, and information asymmetry [118] and are difficult to capture due to the opaqueness of the system. XAI methods can assist in tracking and exposing biases and discrimination that occur in the decision-making process of the AI system [30, 73], and even contribute to their mitigation to some extent [115].
3.2.9 General End-Users.
General end-users are individuals who directly interact with AI systems, as distinguished from the subjects of decision. They include individuals who use smart end-products (e.g., smart homes, social media, and e-commerce) in their daily lives.
■
When to explain: Deployment, Monitoring and Interaction.
—
Need 1: Establishing appropriate trust and building mental models. Establishing appropriate levels of trust is crucial for general end-users to engage in positive and satisfactory interactions with the AI system [235]. However, the complexity of the ML model leads to frequent misunderstandings and gaps between it and the user's mental model [185], making it difficult for the user to understand the ML model's behavior and decisions [164, 219]. As a result, general end-users are increasingly demanding transparency in the AI system they rely on daily [33]. XAI methods provide explanations that enable general end-users to develop appropriate trust in the system [89], thus helping them to build accurate “mental models” [233].
4 What to Explain
Some researchers have pointed out that explanation is actually an answer to a question [120, 142, 161]. Therefore, this section aims to elucidate the types of questions that stakeholders may pose to meet their needs. These questions, referred to as “what to explain” questions, serve as a crucial bridge within our roadmap, connecting stakeholders’ needs with appropriate XAI methods.
As we discussed in Section 2.3, previous research has considered various “what to explain” questions [120, 122, 123, 145]. However, the questions that stakeholders might ask that have been considered in existing research have focused on seeking explanations for the decision-making outcomes of ML models. None of them have considered questions about the inner working mechanisms of an ML model, which is of interest to AI theory experts, AI development experts, and other curious or inquisitive stakeholders. To address this gap, this article introduces a new “what to explain” question, called “What Explanations.” Together with existing questions considered by the literature [120, 122, 123, 145] already, we consider a more comprehensive set of nine questions as follows:
(1)
How Explanations: How does the system work as a whole? How Explanations seek to gain a comprehensive understanding of the system, encompassing its overall logical structure and decision boundaries. For example, how does the autonomous driving system work?
(2)
Why Explanations: Why does the system make a particular decision? Why Explanations aim to comprehend the factors influencing the system's decisions for a given input. They involve understanding the features of the input data, the underlying logic of the ML model, specific examples, and related aspects. For example, why did the car decide to slow down at this specific moment?
(3)
Why-not Explanations: Why does the system not make a particular decision? Why-not Explanations typically ask why the system does not produce a desired output for a specific input, particularly when it deviates from stakeholders’ expectations. The goal is to uncover the reasons for the discrepancies between the system's decisions and stakeholder's expectations. For example, why did the vehicle not take an alternate route when encountering heavy traffic?
(4)
What Explanations: What happens inside the system? What Explanations usually inquire how the system works internally, what potential information the ML model has learned, and to understand the ML model's internal representations. Compared to other questions, such as How Explanations or Why Explanations, which focus on asking how the ML model makes decisions, What Explanations focuses on understanding what knowledge the ML model has learned, such as what a particular structure or neuron has learned or represented. For example, what has the AI system learned internally about recognizing obstacles?
(5)
What-if Explanations: What would the system do if the input changes? What-if Explanations aim to understand how different input data impact the system's decisions. For example, what if a pedestrian suddenly steps in front of the autonomous car, how would it react?
(6)
What-else Explanations: What else are the similar instances? What-else Explanations provide stakeholders with instances that have similar inputs. For example, what else are similar scenarios where autonomous vehicles also make decisions to turn?
(7)
How-to Explanations: How to let the system make another particular decision? How-to Explanations explore how to modify the inputs to obtain different outputs that are consistent with the user's expectations. For example, how to change the environment to allow autonomous vehicles to choose a faster route?
(8)
How-still Explanations: How much of a perturbation can there be while maintaining the same decision? How-still Explanations help stakeholders grasp the extent to which inputs or models can be altered while still obtaining consistent predictive results. For example, how much variation in road conditions can the vehicle withstand while still making the same decisions?
(9)
Data Explanations: Ask for information about the data. Data Explanations involve seeking potential information about both the training and input data, including features, types, ranges, sizes, distributions, potential biases, and disentangled representations. For example, what biases exist in the training data of the autonomous vehicle system?
The above nine types of questions have summarized potential needs from different stakeholders for XAI, particularly in AASs. As can be seen, the purposes of explanations are very different. As a result, the representations and contents of the explanations would be different. The methods for generating the explanations will be different as well. Although there have been several excellent review papers on XAI methods [10, 74, 80, 125, 175, 257], none has clearly stated which XAI methods are used for answering which types of questions at what stages of the AI system lifecycle. Section 6 of this article will fill in this gap within our roadmap.
5 How to Explain
Having thoroughly examined the needs of various stakeholders for explanations and the “what to explain” questions, this section introduces 10 categories of widely used XAI methods. This is a crucial step in our roadmap, as it sets the stage for linking stakeholders’ needs with XAI methods through “what to explain” questions in Section 6. The XAI methods considered in this article were identified through a thorough review of existing popular XAI survey papers [10, 74, 80, 125, 175, 257]. These considered methods broadly encompass the techniques discussed in these surveys, ensuring a comprehensive and complete analysis. It is important to note that our objective here is not to provide an exhaustive review of existing XAI methods. For a more detailed description of these methods and their further categorization, we refer readers to the literature [10, 74, 80, 125, 175, 257]. Our primary objective here is to introduce the ten major categories of XAI methods so that we can discuss, as part of our roadmap, which class of XAI methods are best in addressing which types of questions to fulfill what needs in the next section.
(1)
Decision Trees (DTs) and Decision Rules (DRs). DTs and DRs are considered together in this article due to their similar interpretability properties [257]. They are categorized into global and local explanations based on the scope of the explanation. Global DTs [28, 42, 99, 112] and DRs [43, 98, 190, 211] approximate the explained ML model through techniques such as distillation and rule extraction. Local DTs [256] and DRs [177, 227], on the other hand, serve as surrogate models to explain the model's decisions on a small subset of instances.
(2)
FAE. FAE provides the importance of each feature [228] and can be divided into global and local explanations. Global FAE describes the overall significance of each feature used by the ML model [120], with existing methods including [84, 88, 128, 129, 209, 220]. Local FAE focuses on how much each input feature contributes to the ML model's output for a specific data point [21]. Existing methods include [13, 14, 45, 130, 132, 158, 176, 189, 193, 195], and so on.
(3)
PDP. PDP provides a visual representation of how prediction results vary with different feature values [80] and can be categorized into global and local explanations. Global PDP describes the impact of a set of input features of interest on the prediction results of the ML model as a whole. Classic methods in this category include [9, 68]. Local PDP emphasizes more on the dependency between features and model predictions on a specific sample, with classic methods including [23, 75].
(4)
CF. The CF method considered in this article are divided into CF explanations (CFE) and CF instance (CFI). CFE yields different predicted outcomes (usually the desired outcome) by perturbing the inputs to the ML model [48, 146, 217, 223]. In contrast, CFI provides instances with similar inputs but different prediction results than the instance being explained [116, 120, 151].
(5)
(6)
(7)
(8)
(9)
Association Explanation (AE). AE aims to establish associations between inputs [5], internal model structures (e.g., neurons) [51], or outputs [117] with human-understandable conceptual objects (e.g., natural language, images) by modifying the ML model's architecture or incorporating regularization terms [175]. They are divided into input AE (IpAE), internal AE (ItAE), and output AE (OAE).
In order to facilitate the mapping between stakeholders’ needs and XAI methods using the “what to explain” questions as a bridge, we discuss the relationship between the nine “what to explain” questions described in Section 4 and the 10 XAI methods. Specifically, we identify which XAI methods can address the respective questions. This mapping forms a critical component of our roadmap. It is worth mentioning that a similar mapping was attempted before [120], although only six “what to explain” questions and five XAI methods were considered. We have extended the previous work [120] and produced a more comprehensive result as given in Table 1, which lays the groundwork for employing “what to explain” questions as a bridge between stakeholders’ needs and XAI methods in the next section.
Table 1.
| “What to Explain” Question | XAI Methods |
|---|---|
| How explanations | Global DT, Global DR, Global FAE, GE |
| Why explanations | Local DT, Local DR, Local FAE, PE, TE, IpAE, OAE, Global DT, Global DR, CF, GE, ItAE |
| Why-not explanations | Local FAE, CF, Global DT, Global DR |
| What explanations | MV, GE, ItAE |
| What-if explanations | PDP, Global DT, Global DR, GE |
| What-else explanations | PE, CFI |
| How to be that explanations | CF, Local PDP, Global DT, Global DR |
| How to still be this explanations | Local DT, Local DR, PDP, Global DT, Global DR, PE |
| Data explanations | PE, EDA |
Table 1. The Mapping between the “What to Explain” Questions and XAI Methods, Which Serves as an Extension and Refinement of the Related Work in the Literature [120]
A bolded font XAI method means that the method can answer the corresponding “what to explain” question well, while a regular font XAI method may be able to answer the corresponding “what to explain” question, but the answer may not be intuitive or comprehensive.
6 Bridging Stakeholders’ Needs and XAI Methods: A Roadmap and Guideline
In this section, we establish a connection between the stakeholders’ needs outlined in Section 3 and the XAI methods discussed in Section 5 using the “what to explain” questions from Section 4 as a bridge. This connection forms a central component of our roadmap. Specifically, we first relate the stakeholders’ needs to the “what to explain” questions by considering what questions stakeholders are likely to ask in order to satisfy a specific need. Table 1 is then used to determine which XAI methods can answer specific questions and thus satisfy the corresponding needs of the stakeholders. This roadmap provides a guideline to assist stakeholders in selecting the appropriate XAI method to satisfy their need for explanation. Finally, we provide some examples of existing research to illustrate how the corresponding XAI methods can be used to meet stakeholders’ needs. For clarity, we reiterate the full name of each XAI method and its corresponding abbreviation, as shown in Table 2.
Table 2.
| Full Name | Abbreviation |
|---|---|
| Decision tree | DT |
| Decision rule | DR |
| Feature attribution explanation | FAE |
| Partial dependence plot | PDP |
| Counterfactual explanations/instance | CFE/CFI |
| Prototype explanation | PE |
| Text explanation | TE |
| Model visualization | MV |
| Graph explanation | GE |
| Input/Internal/output association explanation | IpAE/ItAE/OAE |
| Exploratory data analysis | EDA |
In the following, if an XAI method is underlined, it indicates that there has been research using that particular XAI method to meet the corresponding needs. The rest of this section will be organized around different stakeholders, their needs for explanations at different stages of the AI system lifecycle, and how their needs were met by existing XAI methods.
6.1 AI Theory Experts
■
When to explain: Modeling AI; Deployment, Monitoring and Interaction.
—
Need: Insight and understanding of the internal logic of complex ML models.
–
What to explain: What Explanations.
–
Corresponding XAI methods: MV, GE, ItAE.
–
Examples: Several existing works have focused on providing insights into black-box ML models by visualizing neurons using various techniques such as activation maximization [61, 62, 242], deconvolution [244], guided backpropagation [199], and construction and-or graph [250, 251, 252, 253, 254]. For example, Yosinski et al. [242] showed the internal features of each layer in a convolutional neural network (CNN) through an activation-maximization-based MV method, thus helping researchers build valuable intuitions about how CNNs work. There is also work that facilitates the understanding of information learned by each neuron in a neural network by associating neurons with human-understandable semantic concepts [17, 255].
6.2 AI Development Experts
■
When to explain: Model Requirements Analysis.
—
Need: Comparing analysis of multiple ML models.
–
What to explain: How, Why, What Explanations.
–
Corresponding XAI methods: DT, DR, FAE, GE, PE, TE, AE, CF, MV.
–
Examples: Most of the existing studies compared multiple models by FAE methods [65, 103, 139, 155, 160]. For example, Narkar et al. [155] developed Model LineUpper, a tool that integrates several XAI and visualization techniques, including global and local FAE, to support data scientists manually selecting the final model from dozens of candidates. Mostafa et al. [150] assisted in selecting a better CNN model through an MV explanation method called guided backpropagation by visualizing the neurons in the CNN.
■
When to explain: Data Acquisition and Understanding.
—
Need 1: Insight and Understanding of Datasets.
–
What to explain: Data Explanations.
–
Corresponding XAI methods: PE, EDA.
–
Examples: Limited research has focused on utilizing XAI methods to comprehend and gain valuable insights into datasets. DSouza et al. [55] employed EDA to analyze and derive insights from the COVID-19 dataset, providing a better understanding of the impacts of the pandemic in relation to the variables and labels within the dataset. Gurumoorthy et al. [82] employed a PE approach to analyze the entire dataset by selecting a representative set of examples and associated weights from the dataset in order to capture the underlying distribution of the entire dataset.
—
Need 2: Analyzing potential errors, noise, and bias in the dataset.
–
What to explain: Data Explanations.
–
Corresponding XAI methods: PE, EDA.
–
Examples: Few studies have applied XAI methods to evaluate the quality of datasets. One notable example is Schmidt [186] that analyzed gender bias in 14 million teaching assessments on the RateMyProfessors.com Web site through EDA.
■
When to explain: Modeling AI.
—
Need 1: Assisting with feature selection.
–
What to explain: How, Why Explanations.
–
Corresponding XAI methods: DT, DR, FAE, GE, PE, TE, AE, CF.
–
Examples: The FAE method quantifies the contribution of each input feature to the decision-making process of an ML model [21], making it a popular tool among researchers for feature selection. For example, Ribeiro et al. [176] improved the performance of the ML model by removing unimportant features, based on the interpretation provided by LIME. In [20], DNNs were evaluated using LRP to identify information in the dataset that hindered the training of deep learning models, and removing this information improved the F1 score by 20\(\%\). Yang et al. [240] used FAE for feature selection and improved fairness by eliminating features that lead to unfair decision-making. In addition, many related works [100, 135, 212, 243] used the FAE method based on the Shapley value for feature selection.
—
Need 2: Optimizing model architecture and hyper-parameters.
–
What to explain: How, Why, What Explanations.
–
Corresponding XAI methods: DT, DR, FAE, GE, PE, TE, AE, CF, MV.
–
Examples: Researchers have leveraged XAI methods, including MV [150, 238, 244], and FAE [81], to assist in fine-tuning the ML model's architecture and hyper-parameters. As an example, Zeiler and Fergus [244] provided insight into the functionality of the ML model's intermediate feature layer and the operation of the classifier by visualizing the ML model and adjusting its architecture accordingly, ultimately improving the ML model's performance.
—
Need 3: Checking the model's decisions.
–
What to explain: How, Why, Why-not, What, What-if, What-else, How-to, How-still Explanations.
–
Corresponding XAI methods: DT, DR, FAE, GE, PE, TE, AE, CF, MV, PDP.
–
Examples: Researchers have utilized various XAI methods to understand and review black box models, including FAE [26], AE [51], DR [143], DT [173], MV [126], TE [50], and PDP [111], among others. For example, Bojarski et al. [26] used a local FAE method to explain what a DNN learned with end-to-end training and how it makes decisions to drive a car automatically. Liu et al. [126] introduced DGMTracker, a visualization tool that analyzes the training process of deep generative models by visualizing dataflow and examining neuron contributions, providing valuable insights for AI development experts to comprehend and diagnose the training process. Krause et al. [111] developed Prospector, an interactive visual analytics system that provides interactive partial dependency diagnostics through a PDP approach, thereby assisting data scientists in analyzing how features affect the overall prediction.
—
Need 4: Guiding model debugging and error refinement.
–
What to explain: How, Why, Why-not, What Explanations.
–
Corresponding XAI methods: DT, DR, FAE, GE, PE, TE, AE, CF, MV.
–
Examples: Many existing studies have employed XAI methods, including FAE [15, 20, 34, 69, 76, 163, 222] and AE [51], for debugging models and mitigating biases. For example, Palatnik et al. [163] used various FAE methods, including LIME [176], GradCAM [189], and SmoothGrad [196], to analyze the COVID CT-Scan classifier and successfully identify biases within the ML model. Dong et al. [51] proposed an AE method that associates each neuron with a topic, enhancing the interpretability of DNNs. Several other works [27, 114, 188, 207] have incorporated AI experts into the training loop through human-in-the-loop-based explanatory debugging and used their feedback on the provided explanations to modify and refine the ML model.
■
When to explain: Deployment, Monitoring and Interaction.
—
Need 1: Adjusting the ML model to meet the user's expectations and needs.
–
What to explain: Why, Why-not, What-if, How-to Explanations.
–
Corresponding XAI methods: DT, DR, Local FAE, PE, TE, AE, CF, GE, PDP.
–
Examples: Researchers have used various explanation methods, such as FAE [69, 178], CF [29], and AE [51], to adapt the ML model to better align with stakeholders’ expectations and requirements. For example, Rieger et al. [178] proposed the contextual decomposition explanation penalization (CDEP). When the ML model incorrectly assigns importance to certain features (contrary to stakeholders’ expectations or anticipations), CDEP can correct these errors by explaining the insertion of domain knowledge into the ML model. Brandao and Magazzeni [29] minimally modified the map by CFE in path planning, providing stakeholders with suggestions for map changes that can lead to the desired path becoming optimal. Some other studies adopted a human-in-the-loop interpretive debugging approach [114, 188, 207]. In this iterative process, the system initially explains its predictions to the user, who then provides necessary corrections. This iterative collaboration process enables the ML model to be refined and aligned with the user's expectations.
—
Need 2: Assessing the impact of dataset shift.
–
What to explain: How, Why, Data Explanations.
–
Corresponding XAI methods: DT, DR, FAE, GE, PE, TE, AE, CF, EDA.
–
Examples: Currently, there is little work on using XAI methods to detect dataset shifts. Among them, Wang et al. [229] used the FAE method combined with two-sample detection to assist in detecting harmful dataset shifts. Masegosa et al. [136] proposed a method for EDA of stream data based on probabilistic graphical models to explore and analyze financial customer data, capturing the sources of different conceptual drifts.
6.3 Model Validation Team
■
When to explain: Data Acquisition and Understanding.
—
Need: Evaluating data suitability.
–
What to explain: Data Explanations.
–
Corresponding XAI methods: PE, EDA.
–
Examples: EDA enables the examination of data distribution, and evaluation of data quality and structure using analytical techniques such as visualization [110], thus helping the model validation teams gain insight into the training data to determine its suitability in relation to their proprietary data.
■
When to explain: Modeling AI.
—
Need 1: Reviewing the ML model's decision logic.
–
What to explain: How, Why, Why-not, What-if, What-else, How-to, How-still Explanations.
–
Corresponding XAI methods: DT, DR, FAE, GE, PE, TE, AE, CF, PDP.
–
Examples: Gerlings et al. [72] noted that a group of experienced hospital professionals and medical annotators reviewed the output of the ML model through the XAI methods to ensure consistency with their expertise. This thorough evaluation resulted in the removal of two out of the original six pathologies, as their performance did not meet the required standards. Christianto et al. [39] proposed a smart interpretable model framework that requires little AI knowledge. This framework generates a set of fuzzy IF-THEN rules along with their corresponding membership functions, enabling subject matter experts to gain comprehensive insights into the ML model through these rules and sample interpretations.
—
Need 2: Determining compliance with regulations.
–
What to explain: How, Why, Why-not, What-if, What-else Explanations.
–
Corresponding XAI methods: DT, DR, FAE, GE, PE, TE, AE, CF, PDP.
–
Examples: Existing research has applied XAI methods to meet legal requirements for model interpretability [109, 169], to assess the fairness of models and detect bias in models [149, 165, 234, 247], to evaluate the security of models [156], and so on. For example, to ensure compliance with relevant laws such as GDPR, Kolyshkina and Simoff [109] defined the interpretability of ML solutions and proposed CRISP-ML. CRISP-ML is a detailed step-by-step methodology capable of creating the necessary level of interpretable guidance at each stage of the solution building process, guiding companies to have the ability to interpret their algorithms’ decisions to meet relevant regulations. Wich et al. [234] employed XAI methods to visualize biases in politically biased datasets, thus helping to build unbiased datasets or to de-bias them through the interpreted results. Nguyen and Choo [156] proposed a human-in-the-loop framework that integrated security analysts and forensic investigators into the human-computer loop by leveraging XAI methods to assist in vulnerability detection, investigation, and mitigation, ultimately enhancing the security of the ML model.
6.4 Model Operators
■
When to explain: Deployment, Monitoring and Interaction.
—
Need: Ensuring correct and efficient interaction.
–
What to explain: How, Why, What-if, What-else Explanations.
–
Corresponding XAI methods: DT, DR, FAE, GE, PE, TE, AE, CF, PDP.
–
Examples: Zhang et al. [246] introduced and discussed how to use XAI methods (including global DTs, local FAE, PDP, local DRs, and CF) to audit practitioners, guiding them to interact properly with AI tools and meet audit documentation and audit evidence standards. Furthermore, Zhang et al. [249] used the proposed Deep-SHAP method in a power system emergency control scenario, enabling operators to comprehend the underlying mechanisms of deep reinforcement learning models and enhancing their efficiency in interacting with the ML models.
6.5 Decision Makers
■
When to explain: Deployment, Monitoring and Interaction.
—
Need 1: Comprehending specific model decisions.
–
What to explain: Why, Why-not, What-else Explanations.
–
Corresponding XAI methods: DT, DR, Local FAE, PE, TE, AE, CF, GE.
–
Examples: Existing studies have primarily focused on using local FAE methods to help decision makers understand specific model decisions [1, 44, 59, 131, 166, 180, 184, 202, 226, 239, 241]. For example, Lundberg et al. [131] developed an ML system capable of predicting the risk of hypoxemia while explaining risk factors during anesthesia in real-time. It was demonstrated experimentally that the system's explanations were broadly consistent with the prior knowledge of anesthesiologists and that the explanations improved anesthesiologist performance. Monteath and Sheh [147] proposed a new XAI approach that used global DTs to provide incremental decision support for medical diagnosis. Their approach promotes collaboration between AI systems and subject matter experts, facilitating mutual information exchange and joint decision-making. Their system is able to guide physicians in determining which test results are most useful based on available data. Kavya et al. [104] extracted rules from random forests and presented them in the IF-THEN format to aid clinical diagnosticians when comprehending model decisions.
—
Need 2: Deepening overall understanding of the ML model and improving decisions.
–
What to explain: How Explanations.
–
Corresponding XAI methods: Global DT, Global DR, Global FAE, GE.
–
Examples: Existing work that helped decision makers understand the overall logic of the ML model focused on using global FAE methods [1, 35, 166, 224, 226]. Among them, Wang et al. [226] used a global FAE to extract significant features from intrusion detection systems. By illustrating the association between feature values and various types of attacks, their method enabled security experts to gain a deeper understanding of intrusion detection system design. Rad et al. [173] proposed a new XAI method to extract surrogate DTs from black-box models to explain the temporal importance of clinical features in predicting kidney graft survival globally, thus helping physicians understand the decision logic behind the black-box models.
6.6 Regulators
■
When to explain: Deployment, Monitoring and Interaction.
—
Need 1: Ancillary model review.
–
What to explain: How, Why, Why-not, What-if, What-else Explanations.
–
Corresponding XAI methods: DT, DR, FAE, GE, PE, TE, AE, CF, PDP.
–
Examples: Adler et al. [3] proposed an XAI method to evaluate the fairness of black box models from the perspective of an external reviewer. They examined the extent to which the black box model utilizes specific sensitive attributes (e.g., skin color, gender) in the dataset. Similar work [149, 165, 234, 247] has also used XAI methods to reveal potential biases in the ML models as well as to assess the fairness of the ML models. However, no existing research has been found that specifically applies XAI methods for evaluating the safety and robustness of the ML models.
—
Need 2: Assisting in apportioning responsibility.
–
What to explain: Why Explanations.
–
Corresponding XAI methods: DT, DR, Local FAE, PE, TE, AE, CF, GE.
–
Examples: Jafta et al. [95] emphasized the potential of XAI methods in understanding the decision-making process of models, identifying relevant variables, and distinguishing between morally right and wrong decisions. These capabilities not only contribute to uncovering the truth but also facilitate accountability and the prevention of future malpractices. Lima et al. [124] explored whether and to what extent explainability can assist in addressing the accountability problem posed by autonomous AI systems. However, despite these discussions, we have not encountered any specific examples or instances where XAI methods have been applied directly to promote accountability.
6.7 Data Subjects
■
When to explain: Modeling AI; Deployment, Monitoring and Interaction.
—
Need: Protecting personal data information.
–
What to explain: How, Why Explanations.
–
Corresponding XAI methods: DT, DR, FAE, GE, PE, TE, AE, CF.
–
Examples: EU regulations grant data subjects the right to obtain an explanation of how the ML model uses their data to make decisions [8]. The GDPR [214] safeguards the personal data of all EU residents, regardless of the place of processing, and it gives EU residents the right to access, rectify, delete, and restrict the processing of their personal data. However, our investigation has not found any relevant research that uses XAI methods to disclose what information about data subjects is used by the ML models in their decisions and how the data of data subjects affect the ML models’ decisions.
6.8 Subjects of Decision
■
When to explain: Deployment, Monitoring and Interaction.
—
Need 1: Understanding the decision and how to maintain or change it.
–
What to explain: Why, Why-not, What-if, What-else, How-to, How-still Explanations.
–
Corresponding XAI methods: DT, DR, Local FAE, PE, TE, AE, CF, GE, PDP.
–
Examples: An example provided in the AIX360 toolkit [12] uses the CF method to explain why a bank customer's loan was denied and what modifications should be made to get the loan approved. The XAI approach proposed in [147] used DTs to provide incremental decision support for medical diagnosis. The system was able to explain how a particular decision was made and trace it directly back to the underlying training data, thus helping patients understand the system's decisions as well as increasing their confidence in the system's decisions.
—
Need 2: Examining bias.
–
What to explain: How, Why, Why-not, What-if Explanations.
–
Corresponding XAI methods: DT, DR, FAE, GE, PE, TE, AE, CF, PDP.
–
Examples: Wang et al. [230] used FAE to portray the decision-making process of the ML model in order to assess the ML model's procedural fairness. Malhi et al. [133] conducted an HCI study to investigate the effect of explanation on the biases introduced by human participants in human-computer decision-making. The final experimental results found that users derived benefits from explanations, had increased trust, and were able to reduce bias in human decision-making. Similar research in assessing model fairness [149, 165, 234, 247] can also be leveraged to assist subjects of the decision in detecting or even mitigating bias and discrimination in the ML models.
6.9 General End-Users
■
When to explain: Deployment, Monitoring and Interaction.
—
Need 1: Establishing appropriate trust and building mental models.
–
What to explain: How, Why, Why-not, What-if, What-else Explanations.
–
Corresponding XAI methods: DT, DR, FAE, GE, PE, TE, AE, CF, PDP.
–
Examples: Researchers have employed various XAI methods to help general end-users develop trust and construct accurate mental models. For example, Kulesza et al. [114] proposed an explanatory debugging approach where the system explains to the user how it made each prediction, and the user then explains to the learning system any necessary corrections that can help the end-user build an appropriate mental model. In [236], interviews and user research conducted in the context of autonomous driving revealed that XAI visualization depicting detected objects and their predicted action significantly enhanced users’ comprehension and trust in the ML model, thereby facilitating users’ mental model development. Van der Waa et al. [215] proposed a framework for an interpretable confidence measure (ICM). They demonstrated through experiments with end-users using self-driving cars that the ICM can provide users with an interpretable confidence value for case-based reasoning about decision outcomes. This can effectively guide users on whether to trust the ML model's decision recommendations.
—
Need 2: Protecting personal data information.
–
What to explain: How, Why Explanations.
–
Corresponding XAI methods: DT, DR, FAE, GE, PE, TE, AE, CF.
–
Examples: Some researchers have developed various types of visualization tools, such as Floodwatch [159] and Behind the Banner [16], to inform users about the data advertisers may collect. However, no research has been conducted to explain how XAI methods can be used to protect users’ private data information.
The above detailed discussions can be summarized by Table 3, which gives a global picture of the relationships among different stakeholders, their needs for explanation at different stages of the AI system lifecycle, different types of explanations, and existing methods that could be used to generate explanations. Table 3 is a vital part of the roadmap and serves as a guideline to help the stakeholders in selecting appropriate XAI methods to fulfill their needs for explanation.
Table 3.
| Explain to Whom | When to Explain | Need | What to Explain | How to Explain | Examples |
|---|---|---|---|---|---|
| AI theory experts |  | Insight and understanding of the internal logic of complex ML models | What | MV, GE, ItAE | [17, 61, 62, 90, 101, 157, 168, 199, 242, 244, 248, 250–255] |
| AI development experts |  | Comparing analysis of multiple ML models | How, why, what | DT, DR, FAE, GE, PE, TE, AE, CF, MV | [65, 103, 139, 150, 155, 160] |
 | Insight and understanding of datasets | Data | PE, EDA | [55, 82] | |
| Analyzing potential errors, noise, and bias in the dataset | Data | PE, EDA | [186] | ||
 | Assisting with feature selection | How, why | DT, DR, FAE, GE, PE, TE, AE, CF | [20, 100, 105, 135, 148, 152, 176, 206, 212, 216, 240, 243] | |
| Optimizing model architecture and hyperparameters | How, why, what | DT, DR, FAE, GE, PE, TE, AE, CF, MV | [77, 81, 150, 197, 203, 238, 244] | ||
| Checking the model's decisions | How, why, why-not, what, what-if, what-else, how-to, how-still | DT, DR, FAE, GE, PE, TE, AE, CF, MV, PDP | [19, 26, 50, 51, 106, 111, 126, 143, 173] | ||
| Guiding model debugging and error refinement | How, why, why-not, what | DT, DR, FAE, GE, PE, TE, AE, CF, MV | [4, 15, 19, 20, 34, 51, 69, 76, 114, 162, 163, 170, 188, 198, 200, 207, 222] | ||
 | Adjusting the ML model to meet the user's expectations and needs | Why, why-not, what-if, how-to | DT, DR, Local FAE, PE, TE, AE, CF, GE, PDP | [22, 29, 51, 69, 114, 178, 188, 207] | |
| Assessing the impact of dataset shift | How, why, data | DT, DR, FAE, GE, PE, TE, AE, CF, EDA | [32, 38, 136, 229] | ||
| Model validation team | | Evaluating data suitability | Data | PE, EDA | [110] |
| Reviewing the ML model's decision logic | How, why, why-not, what-if, what-else, how-to, how-still | DT, DR, FAE, GE, PE, TE, AE, CF, PDP | [39, 46, 113, 127] | |
| Determining compliance with regulations | How, why, why-not, what-if, what-else | DT, DR, FAE, GE, PE, TE, AE, CF, PDP | [109, 149, 156, 165, 169, 234, 247] | ||
| Model operators | | Ensuring correct and efficient interaction | How, why, what-if, what-else | DT, DR, FAE, GE, PE, TE, AE, CF, PDP | [134, 174, 194, 246, 249] |
| Decision makers | | Comprehending specific model decisions | Why, why-not, what-else | DT, DR, Local FAE, PE, TE, AE, CF, GE | [1, 44, 59, 66, 104, 131, 147, 166, 180, 184, 202, 226, 239, 241] |
| Deepening overall understanding of the ML model and improving decisions | How | Global DT, Global DR, Global FAE, GE | [1, 35, 44, 166, 173, 224, 226] | ||
| Regulators | | Ancillary model review | How, why, why-not, what-if, what-else | DT, DR, FAE, GE, PE, TE, AE, CF, PDP | [3, 149, 165, 234, 247] |
| Assisting in apportioning responsibility | Why | DT, DR, Local FAE, PE, TE, AE, CF, GE | None | ||
| Data subjects | | Protecting personal data information | How, why | DT, DR, FAE, GE, PE, TE, AE, CF | None |
| Subjects of decision | | Understanding the decision and how to maintain or change it | Why, why-not, what-if, what-else, how-to, how-still | DT, DR, Local FAE, PE, TE, AE, CF, GE, PDP | [12, 147] |
| Examining bias | How, why, why-not, what-if | DT, DR, FAE, GE, PE, TE, AE, CF, PDP | [40, 133, 149, 165, 230, 234, 247] | ||
| General end-users | | Establishing appropriate trust and building mental models | How, why, why-not, what-if, what-else | DT, DR, FAE, GE, PE, TE, AE, CF, PDP | [25, 54, 86, 114, 172, 215, 232, 236] |
| Protecting personal data information | How, why | DT, DR, FAE, GE, PE, TE, AE, CF | None |
Table 3. A Roadmap's Guideline to Assist Stakeholders in Selecting Appropriate XAI Methods to Meet Their Needs for Explanation
It uses the “what to explain” questions as a bridge to connect the stakeholders’ needs for explanation at different stages with XAI methods. An underline below the XAI method indicates that the XAI method has been used in the given examples to meet the corresponding need of the stakeholder.
Furthermore, from Table 3 we can identify which XAIs are more universally applicable to the vast majority of stakeholders. Specifically, PE is the most generalized XAI method, capable of addressing 21 out of 23 different stakeholder needs, except for challenges in helping AI theory experts and decision makers understand the model. This finding aligns with Humer et al. [93], who compared several XAI methods and found that PE was the most helpful XAI method for human decision-making. Additionally, DT, DR, FAE, CF, TE, GE, and AE are highly versatile XAI methods, each capable of addressing over 75% of these stakeholder needs.
In addition, it is clear from Table 3 what has been accomplished and what research gaps remain, which helps us to gauge the state-of-the-art in XAI in a more comprehensive context. As an example of gaps in research, AI development experts in AASs could benefit from (1) gaining insights into the overall behavior of an ML model through global XAI methods such as Global DT, Global DR, or GE, aiding in model comparison in adaptive environments, (2) assisting in feature selection by identifying important features for decision-making in real-time autonomous systems through GT or DR, and (3) using MV visualization techniques for debugging models, particularly for autonomous systems that need to react to unforeseen changes dynamically.
7 Case Studies
After providing the roadmap to bridge stakeholder needs and existing XAI methods in Section 6, this section demonstrates how the roadmap's guideline (Table 3) can be utilized to assist stakeholders in fulfilling their needs through two hypothetical case studies. Specifically, we focus on autonomous driving and financial loan scenarios within the context of AASs, which are representative of real-world applications [25, 31, 36].
7.1 Autonomous Driving Scenario
In an autonomous driving scenario, the adaptive driving system uses ML models to assist in making decisions such as route optimization, obstacle detection, and vehicle speed adjustments. This scenario involves two primary stakeholders: the vehicle driver (general end-users) and the traffic safety officer (regulators). The following section describes their needs for explanations at a given stage, the questions they may ask, and how the corresponding existing XAI technologies can answer these questions to fulfill their needs. An overview is shown in Figure 3.
Fig. 3.
Stakeholder 1:
—
Whom: General end-users (a vehicle driver).
—
When: Deployment, Monitoring and Interaction.
—
Need: Establishing appropriate trust and building mental models. This vehicle driver expects to understand why the autonomous driving system makes particular decisions, facilitating the development of trust in its operation.
—
What: According to Table 3, when general end-users have the need for “establishing appropriate trust and building mental models,” the questions they may ask include “How,” “Why,” “Why-not,” “What-if,” and “What-else.” In this scenario, we assume that the vehicle driver asks the following question:
(1)
Why does the system make a particular decision?
—
How: Table 3 indicates that eight XAI methods (“DT,” “DR,” “Local FAE,” “PE,” “TE,” “AE,” “CF,” and “GE”) could potentially address the question posed to fulfill that need. In this scenario, we use the following XAI method to answer the “Why” question asked by the vehicle driver:
(1)
Local FAE is used in this scenario to answer the “Why” question. This method helps the vehicle driver understand the autonomous driving system's decisions by highlighting key information, such as lane markings, surrounding vehicles, and obstacles, that influence the system's actions.
Stakeholder 2:
—
Whom: Regulators (a traffic safety officer).
—
When: Deployment, Monitoring and Interaction.
—
Need: Ancillary model review. This traffic safety officer needs the help of XAI technologies to assist him in reviewing the compliance of the autonomous driving system with traffic safety regulations.
—
What: According to Table 3, when regulators have the need for “ancillary model review,” the questions they may ask include “How,” “Why,” “Why-not,” “What-if,” and “What-else.” In this scenario, we assume that the traffic safety officer asks the following question:
(1)
How does the system work as a whole?
—
How: Table 3 indicates that four XAI methods (“Global DT,” “Global DR,” “Global FAE,” and “GE”) could potentially address the question posed to fulfill that need. In this scenario, we use the following XAI method to answer the “How” question asked by the traffic safety officer:
(1)
Global DT is used in this scenario to answer the “How” question. This approach reveals the overall decision-making rules of the autonomous driving system and helps the traffic safety officer understand the system's adherence to traffic rules.
7.2 Financial Loan Scenario
In this financial loan process, the autonomous financial system adapts its evaluation based on external market trends and internal user data, involving two primary stakeholders: bank loan officers (decision makers) and loan customers (subjects of decision). When a customer applies for a loan, the autonomous financial system analyzes factors such as market conditions and the applicant's financial history, providing the bank loan officer with recommendations. With the assistance of these recommendations, the officer arrives at the final decision.
In this scenario, we assume two main stakeholders, their needs for explanations at a given stage, the questions they might ask, and accordingly which existing XAI technologies answer those questions to fulfill their needs. This is described further below, while the overview is shown in Figure 4.
Fig. 4.
Stakeholder 1:
—
Whom: Decision Makers (a bank loan officer).
—
When: Deployment, Monitoring and Interaction.
—
Need: Comprehending specific model decisions. This bank loan officer relies on the adaptive financial system to assist in deciding whether to approve a loan. To make an informed decision, the officer seeks to understand how the system arrives at its conclusions, which is crucial for determining whether to trust the system's recommendations.
—
What: According to Table 3, when decision-makers have the need for “comprehending specific model decisions,” the questions they may ask include “Why,” “Why-not,” and “What-else.” In this scenario, we assume that the bank loan officer asks the following two questions:
(1)
Why does the system make a particular decision?
(2)
What else are the similar instances?
—
How: Table 3 indicates that eight XAI methods (“DT,” “DR,” “Local FAE,” “PE,” “TE,” “AE,” “CF,” and “GE”) could potentially address the questions posed to fulfill that need. In this scenario, we use the following two XAI methods to answer the “Why” and “What-else” questions asked by the bank loan officer, respectively:
(1)
Local FAE is used in this scenario to answer the “Why” question. The local FAE helps this bank loan officer understand the adaptive financial system's specific decisions by revealing the extent to which each piece of information (input feature) about a given customer contributes to the system's decisions.
(2)
PE is used in this scenario to answer the “What-else” question. The PE provides some decision cases with similar inputs and outputs to the customer being explained, helping this bank loan officer better understand and trust the adaptive financial system's decisions.
Stakeholder 2:
—
Whom: Subjects of Decision (a bank loan customer).
—
When: Deployment, Monitoring and Interaction.
—
Need: Understanding the decision and how to maintain or change it. This bank loan customer's loan application was rejected and he seeks to understand why and how he can improve his chances of approval in the future.
—
What: According to Table 3, when subjects of decision have the need for “understanding the decision and how to maintain or change it,” the questions they may ask include “Why,” “Why-not,” “What-if,” “What-else,” “How-to,” and “How-still.” In this scenario, we assume that the bank loan customer asks the following question:
(1)
How to let the system make another particular decision?
—
How: Table 3 indicates that nine XAI methods (“DT,” “DR,” “Local FAE,” “PE,” “TE,” “AE,” “CF,” “GE,” and “PDP”) could potentially address the questions posed to fulfill that need. In this scenario, we use the following XAI method to answer the “How-to” question asked by the bank loan customer:
(1)
CFE is used in this scenario to answer the “How-to” question. CFE provides explanations by informing the bank loan customer of the factors that led to the denial of his or her loan application, helping the customer to understand the adaptive financial system's decision-making. It also suggests areas for improvement, increasing the likelihood of future loan approvals.
This section demonstrates the practical application of the roadmap's guideline (Table 3) by showcasing how stakeholders can select appropriate XAI methods to meet their needs in real-world scenarios. Using the examples of autonomous driving and financial loan scenarios, we illustrate how general end-users, regulators, decision makers, and subjects of decision can utilize XAI technologies to gain insights into ML systems. This process enhances transparency, builds trust, and supports informed decision-making. The case studies serve as a guide for readers on how to effectively apply the roadmap's guideline (Table 3) to select the appropriate XAI method to meet various stakeholders’ needs for explanation.
8 Discussion
Table 3 has led to a number of important observations that reveal gaps in existing research in achieving the goal of XAI to provide human users with understandable interpretations [10] and point out promising directions for future research. These observations are discussed in this section.
8.1 Imbalanced Consideration of Different Stakeholders
Table 3 has made one thing very clear: While there is much research about XAI methods for some stakeholders (e.g., AI development experts), there is a distinct lack of research to meet the needs of other stakeholders. In fact, no XAI methods have been developed to cater for the needs of regulators, data subjects, and general end-users. This observation aligns with the findings reported elsewhere [22, 96, 205]. However, these stakeholders play a vital role in both the ML ecosystems and AASs. For example, in AASs such as autonomous vehicles, regulators are critical in ensuring compliance with safety standards, while end-users should trust the system's decisions. Therefore, future research efforts should prioritize providing appropriate XAI methods to these stakeholders.
8.2 Imbalanced Consideration of Different Research Directions
If we examine the types of explanations generated by existing XAI methods and the questions they tried to answer, we find that there are relatively few studies of XAI methods in the following four directions.
8.2.1 XAI Methods for Interpretation of Data.
Although many stakeholders express a desire to gain insight into data through XAI methods, including assisting AI development experts to understand data characteristics, size, distribution, noise, and bias before modeling [72], helping AI development experts and model validation teams to check for data shift and its possible impact [22, 47], and assisting in protecting data information of data subjects and general end-users [145, 205, 210], our investigation has revealed a scarcity of studies that utilize XAI methods for the above purposes, particularly when excluding the traditional data analysis technique of EDA.
We find that the aforementioned scarcity can largely be attributed to the limited availability of XAI methods focused on providing insights into the data, as researchers have predominantly focused on designing XAI methods for explaining ML models. Notably, Wang et al. [229] utilized FAE to detect data shifts. However, this approach requires the aid of an ML model and can only be used to detect data shifts that are detrimental to model performance. Another relevant contribution is the ProtoDash approach proposed in [82]. This approach is able to generalize and compactly represent prototype examples of potentially complex data distributions by mining information in datasets and performing prototype selection to convey meaningful insights to humans in domains where simple explanations are difficult to extract. However, overall, the scarcity of XAI methods for data interpretation remains evident. There is a pressing need for the development of XAI methods that enable deeper insights into data.
8.2.2 XAI Methods for Reviewing Model Compliance.
The current regulatory landscape has witnessed governments strengthening and implementing regulations concerning the risks associated with the application of AI systems. For example, the “Ethics Guidelines for Trustworthy AI” [41] developed by the EU's High-Level Expert Group on Artificial Intelligence emphasizes the necessity for trustworthy AI systems to adhere to requirements such as algorithmic fairness, non-discrimination, privacy protection, security, transparency, and interpretability. These requirements are particularly relevant in AASs like autonomous driving or autonomous drones, where non-compliance can pose serious risks. Research has shown the desire of AI development experts to utilize XAI methods to identify and mitigate model bias, while model validation teams and regulators seek XAI methods to assist in reviewing model compliance [47, 210].
Despite the continuous enhancement of regulatory requirements for AI systems and the corresponding practical needs of stakeholders, there remains a paucity of research on how to effectively implement these requirements with the support of XAI methods. This observation was also made in [73], which stated that “the literature greatly suggests that XAI will ease this barrier, however, there has not yet been any significant empirical studies showing how XAI in fact may satisfy regulators need to ensure compliance or undergo audits.” However, there were some preliminary attempts at trying to review the security [156], fairness [3, 149, 165, 234, 247], transparency [57, 109, 169], and so on. of the ML model through XAI methods.
8.2.3 XAI Methods for Accountability.
Much of the literature [10, 94, 95, 124, 145, 191] stated that a major purpose of XAI is to enable accountability in AI, facilitating the attribution of responsibility to human agents involved in its development, deployment, and interaction. For example, in autonomous healthcare systems, understanding who is responsible for a wrong diagnosis is vital. Unfortunately, our survey revealed no relevant research related to assisting or enabling accountability through XAI. Consequently, while the idea of achieving accountability through XAI is highly attractive, the actual feasibility and research progress remain unclear. One important future work would be to clarify whether, how, and to what extent XAI methods can assist in accountability.
8.2.4 XAI Methods to Help Subjects of Decision Understand the ML Model.
An interesting finding from our survey is that although AI experts, subject matter experts, and subjects of decision all express a desire to understand model decisions through XAI methods, there is little research work catering the needs of subjects of decision. For example, in the medical field, where XAI methods are widely used, many studies [44, 59, 180, 202] have focused on assisting physicians in understanding model decisions, but few studies [147] have addressed the needs of patients. This is a very important future research direction if AI is to be applied in areas that affect human lives.
8.3 Imbalance in the Use of Different XAI Methods
Through our comprehensive survey, we have observed that although researchers have proposed numerous XAI methods, there is a wide variation in the frequency of using different XAI methods. In particular, the FAE method [195] emerges as the predominant choice in many studies.
(1)
We counted the distribution of XAI methods utilized in the 118 studies related to applying XAI methods to meet stakeholders’ needs used in Table 3, and the results are shown in Figure 5. From this figure, we can see that the FAE method was employed in a significant 68 of these studies, the second-ranked MV method was used in only 13 cases, and the PE method, which has the potential to meet the needs of the vast majority of stakeholders, was applied in only four cases. Even the sum of the number of applications of all other XAI methods is not as high as FAE. This variation underscores the substantial reliance on the FAE method, with other XAI methods being considerably underutilized in the existing literature.
(2)
Focusing on a specific application scenario that has been extensively researched, namely helping decision makers in “comprehending specific model decisions,” many XAI methods could theoretically fulfill that need, including DT, DR, FAE, PE, TE, GE methods, and so on. However, for a total of 14 relevant papers listed in Table 3, except for one paper that explored the role of XAI from the decision maker's perspective and one paper used the DR method, all remaining 12 papers relied on the FAE method for comprehending model-specific decisions (with one study utilizing both FAE and PDP methods).
Fig. 5.
It is clear that there is a pronounced imbalance in the application of various XAI methods. While the preference for the FAE method may be attributed to its reputation as the most extensively studied explainability method to date [10, 22], it is essential to recognize the importance of incorporating different XAI methods to create a comprehensive explanatory framework because different XAI methods can reveal different aspects of ML models. For example, in AASs, where different methods may be required to explain different aspects of a system's behavior (e.g., safety vs. efficiency in an autonomous vehicle). It would be interesting in the future to see more studies using a diverse set of XAI methods to give a fuller explanation of ML models to different stakeholders.
8.4 Unclear Stakeholders When Discussing XAI Methods
A crucial oversight when discussing XAI methods in the current literature is the lack of clarity on who the appropriate stakeholders are in using these XAI methods [22, 72]. Our survey revealed that many studies focusing on the application of XAI methods have also failed to explicitly clarify their intended stakeholders. For example, there are studies that use existing XAI methods to examine the presence of bias in the ML models and to assess their fairness [149, 165, 234], yet they do not specify which stakeholders their methods are intended for. Yet, it is important to recognize that although AI experts, subject matter experts, external entities, and general end-users all have a need to review and assess the fairness of models, they actually require different explanations because they have different AI expertise and domain-specific knowledge [52, 72, 145, 179]. As a result, some research on XAI methods may struggle to effectively address stakeholders’ needs. It is essential in our future research of XAI methods to be explicitly clear about the stakeholders, their needs at different stages of the AI system lifecycle, and the types of question that the XAI methods would answer. Hopefully this article, especially Table 3, could serve as a roadmap and guideline for people to conduct their future research.
9 Conclusion
Existing review papers on XAI methods tend to focus on techniques and algorithms to generate explanations of ML models, less attention has been paid to key questions such as: Who will use the XAI method? To whom does the method explain to? What types of explanation does the XAI method provide at what stage of the AI system lifecycle? Does the explanation meet a stakeholder's need? What are the XAI methods that have been used to meet a stakeholder's need? and so on. This article gives a more holistic and comprehensive review of existing literature on XAI by focusing on the four pillars in a common framework (Figure 1)—who explains to whom at which stage of the AI system lifecycle about what and for what purposes? The emphasis of this article is not on any individual pillars, but on the essential relationships among them.
First, the article presents an ML ecosystem in which the roles of different stakeholders can be clearly described. Second, the AI system lifecycle is used to clarify different explanation needs at different stages by different stakeholders. Third, different types of explanations are discussed, linking the discussions to different needs by stakeholders at different stages of the AI system lifecycle. Fourth, the article summarizes the XAI methods proposed in the literature. Fifth, the article provides a comprehensive mapping from XAI methods to stakeholders’ needs, capturing the key information in Table 3 and providing a roadmap to assist stakeholders in selecting the appropriate XAI method to meet their needs for explanation. Finally, based on our review, major gaps in the literature and future research directions are pointed out. It is important to clarify the target stakeholders for XAI methods during their design and application. Additionally, future research should prioritize addressing the identified gaps by focusing on specific stakeholders, research directions, and XAI methods that remain under-explored.
References
[1]
Zakaria Abou El Houda, Bouziane Brik, and Lyes Khoukhi. 2022. ‘Why should I trust your IDS?’: An explainable deep learning framework for intrusion detection systems in internet of things networks. IEEE Open Journal of the Communications Society 3 (2022), 1164–1176.
[2]
Amina Adadi and Mohammed Berrada. 2018. Peeking inside the black-box: A survey on explainable artificial intelligence (XAI). IEEE Access 6 (2018), 52138–52160.
[3]
Philip Adler, Casey Falk, Sorelle A. Friedler, Tionney Nix, Gabriel Rybeck, Carlos Scheidegger, Brandon Smith, and Suresh Venkatasubramanian. 2018. Auditing black-box models for indirect influence. Knowledge and Information Systems 54, 1 (2018), 95–122.
[4]
Yongsu Ahn and Yu-Ru Lin. 2019. Fairsight: Visual analytics for fairness in decision making. IEEE Transactions on Visualization and Computer Graphics 26, 1 (2019), 1086–1095.
[5]
David Alvarez Melis and Tommi Jaakkola. 2018. Towards robust interpretability with self-explaining neural networks. In Advances in Neural Information Processing Systems, Vol. 31, 7786–7795.
[6]
Saleema Amershi, Andrew Begel, Christian Bird, Robert DeLine, Harald Gall, Ece Kamar, Nachiappan Nagappan, Besmira Nushi, and Thomas Zimmermann. 2019. Software engineering for machine learning: A case study. In Proceedings of the 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP ’19). IEEE, 291–300.
[7]
Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. 2016. Concrete problems in AI safety. arXiv:1606.06565. Retrieved from http://arxiv.org/abs/1606.06565
[8]
Anna Markella Antoniadi, Yuhan Du, Yasmine Guendouz, Lan Wei, Claudia Mazo, Brett A. Becker, and Catherine Mooney. 2021. Current challenges and future opportunities for XAI in machine learning-based clinical decision support systems: A systematic review. Applied Sciences 11, 11 (2021), 5088.
[9]
Daniel W. Apley and Jingyu Zhu. 2020. Visualizing the effects of predictor variables in black box supervised learning models. Journal of the Royal Statistical Society: Series B (Statistical Methodology) 82, 4 (2020), 1059–1086.
[10]
Alejandro Barredo Arrieta, Natalia Díaz-Rodríguez, Javier Del Ser, Adrien Bennetot, Siham Tabik, Alberto Barbado, Salvador García, Sergio Gil-López, Daniel Molina, Richard Benjamins, et al. 2020. Explainable artificial intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI. Information Fusion 58 (2020), 82–115.
[11]
Vijay Arya, Rachel K. E. Bellamy, Pin-Yu Chen, Amit Dhurandhar, Michael Hind, Samuel C. Hoffman, Stephanie Houde, Q. Vera Liao, Ronny Luss, Aleksandra Mojsilović, et al. 2019. One explanation does not fit all: A toolkit and taxonomy of AI explainability techniques. arXiv:1909.03012. Retrieved from http://arxiv.org/abs/1909.03012
[12]
Vijay Arya, Rachel K. E. Bellamy, Pin-Yu Chen, Amit Dhurandhar, Michael Hind, Samuel C. Hoffman, Stephanie Houde, Q. Vera Liao, Ronny Luss, Aleksandra Mojsilovic, et al. 2020. AI explainability 360: An extensible toolkit for understanding data and machine learning models. Journal of Machine Learning Research 21, 130 (2020), 1–6.
[13]
David Baehrens, Timon Schroeter, Stefan Harmeling, Motoaki Kawanabe, Katja Hansen, and Klaus-Robert Müller. 2010. How to explain individual classification decisions. Journal of Machine Learning Research 11 (2010), 1803–1831.
[14]
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In Proceedings of the International Conference on Learning Representations, 1–11.
[15]
Pieter Barnard, Irene Macaluso, Nicola Marchetti, and Luiz A. DaSilva. 2022. Resource reservation in sliced networks: An explainable artificial intelligence (XAI) approach. In Proceedings of the ICC 2022-IEEE International Conference on Communications. IEEE, 1530–1535.
[16]
John Battelle. 2013. Behind the banner, a visualization of the adtech ecosystem. Retrieved from https://battellemedia.com/archives/2013/05/behind-the-banner-a-visualization-of-the-adtech-ecosystem
[17]
David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. 2017. Network dissection: Quantifying interpretability of deep visual representations. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 6541–6549.
[18]
Vaishak Belle and Ioannis Papantonis. 2021. Principles and practice of explainable machine learning. Frontiers in Big Data 4 (2021), 688969.
[19]
Adrien Bennetot, Jean-Luc Laurent, Raja Chatila, and Natalia Díaz-Rodríguez. 2019. Towards explainable neural-symbolic visual reasoning. In Proceedings of the IJCAI Neural-Symbolic Learning and Reasoning Workshop, 1–6. Retrieved from http://arxiv.org/abs/1909.09065
[20]
Vitor Bento, Manoela Kohler, Pedro Diaz, Leonardo Mendoza, and Marco Aurelio Pacheco. 2021. Improving deep learning performance by using explainable artificial intelligence (XAI) approaches. Discover Artificial Intelligence 1, 1 (2021), 1–11.
[21]
Umang Bhatt, Adrian Weller, and José M. F. Moura. 2021. Evaluating and aggregating feature-based model explanations. In Proceedings of the 29th International Joint Conference on Artificial Intelligence, 3016–3022.
[22]
Umang Bhatt, Alice Xiang, Shubham Sharma, Adrian Weller, Ankur Taly, Yunhan Jia, Joydeep Ghosh, Ruchir Puri, José MF Moura, and Peter Eckersley. 2020. Explainable machine learning in deployment. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, 648–657.
[23]
P. Biecek. 2019. Ceteris paribus plots (what-if plots) for explanations of a single observation. Retrieved from https://github.com/pbiecek/ceterisParibus
[24]
William J. Bingley, Caitlin Curtis, Steven Lockey, Alina Bialkowski, Nicole Gillespie, S. Alexander Haslam, Ryan K. L. Ko, Niklas Steffens, Janet Wiles, and Peter Worthy. 2023. Where is the human in human-centered AI? Insights from developer priorities and user experiences. Computers in Human Behavior 141 (2023), 107617.
[25]
Mariusz Bojarski, Anna Choromanska, Krzysztof Choromanski, Bernhard Firner, Larry J. Ackel, Urs Muller, Phil Yeres, and Karol Zieba. 2018. VisualBackProp: Efficient visualization of CNNs for autonomous driving. In Proceedings of the 2018 IEEE International Conference on Robotics and Automation (ICRA ’18). 4701–4708.
[26]
Mariusz Bojarski, Philip Yeres, Anna Choromanska, Krzysztof Choromanski, Bernhard Firner, Lawrence Jackel, and Urs Muller. 2017. Explaining how a deep neural network trained with end-to-end learning steers a car. arXiv:1704.07911. Retrieved from http://arxiv.org/abs/1704.07911
[27]
Andrea Bontempelli, Stefano Teso, Fausto Giunchiglia, and Andrea Passerini. 2022. Concept-level debugging of part-prototype networks. In Proceedings of the Workshop on Trustworthy Artificial Intelligence as a Part of the ECML/PKDD 22 Program, 1–13
[28]
Olcay Boz. 2002. Extracting decision trees from trained neural networks. In Proceedings of the 8th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 456–461.
[29]
Martim Brandao and Daniele Magazzeni. 2020. Explaining plans at scale: Scalable path planning explanations in navigation meshes using inverse optimization. In Proceedings of the International Conference on Automated Planning and Scheduling, 31, 1 (2020), 56–64.
[30]
Andrea Brennen. 2020. What do people really want when they say they want “explainable AI?” We asked 60 stakeholders. In Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems, 1–7.
[31]
Longbing Cao. 2022. AI in finance: Challenges, techniques, and opportunities. ACM Computing Surveys 55, 3 (2022), 1–38.
[32]
Steffen Castle, Robert Schwarzenberg, and Mohsen Pourvali. 2021. Detecting covariate drift with explanations. In Proceedings of the CCF International Conference on Natural Language Processing and Chinese Computing. Springer, 317–322.
[33]
Larissa Chazette and Kurt Schneider. 2020. Explainability as a non-functional requirement: Challenges and recommendations. Requirements Engineering 25, 4 (2020), 493–514.
[34]
Ching-Ju Chen, Ling-Wei Chen, Chun-Hao Yang, Ya-Yu Huang, and Yueh-Min Huang. 2021. Improving CNN-based pest recognition with a post-hoc explanation of XAI. Retrieved from http://www.researchsquare.com/article/rs-782408/v1
[35]
Dehua Chen, Hongjin Zhao, Jianrong He, Qiao Pan, and Weiliang Zhao. 2021. An causal XAI diagnostic model for breast cancer based on mammography reports. In Proceedings of the 2021 IEEE International Conference on Bioinformatics and Biomedicine (BIBM ’21), 3341–3349.
[36]
Ning Chen, Bernardete Ribeiro, and An Chen. 2016. Financial credit risk assessment: A recent review. Artificial Intelligence Review 45 (2016), 1–23.
[37]
Runpu Chen, Le Yang, Steve Goodison, and Yijun Sun. 2020. Deep-learning approach to identifying cancer subtypes using high-dimensional genomic data. Bioinformatics 36, 5 (2020), 1476–1483.
[38]
Youngwon Choi, Wenxi Yu, Mahesh B. Nagarajan, Pangyu Teng, Jonathan G. Goldin, Steven S. Raman, Dieter R. Enzmann, Grace Hyun J. Kim, and Matthew S. Brown. 2023. Translating AI to clinical practice: Overcoming data shift with explainability. Radiographics 43, 5 (2023), e220105.
[39]
Hotman Christianto, Gary Kee Khoon Lee, Zhou Weigui Jair, Henry Kasim, and Deepu Rajan. 2022. Smart interpretable model (SIM) enabling subject matter experts in rule generation. Expert Systems with Applications 207 (2022), 117945.
[40]
Ching-Hua Chuan, Ruoyu Sun, Shiyun Tian, and Wan-Hsiu Sunny Tsai. 2024. EXplainable artificial intelligence (XAI) for facilitating recognition of algorithmic bias: An experiment from imposed users’ perspectives. Telematics and Informatics 91 (2024), 102135.
[41]
European Commission. 2019. Ethics guidelines for trustworthy AI. Retrieved from https://digital-strategy.ec.europa.eu/en/library/ethics-guidelines-trustworthy-ai
[42]
Mark Craven and Jude Shavlik. 1995. Extracting tree-structured representations of trained networks. In Advances in Neural Information Processing Systems, Vol. 8, 24–30.
[43]
Mark W. Craven and Jude W. Shavlik. 1994. Using sampling and queries to extract rules from trained neural networks. In Proceedings of the 11th International Conference on International Conference on Machine Learning, 37–45.
[44]
Harry Freitas Da Cruz, Boris Pfahringer, Tom Martensen, Frederic Schneider, Alexander Meyer, Erwin Böttinger, and Matthieu-P. Schapranow. 2021. Using interpretability approaches to update “black-box” clinical prediction models: An external validation study in nephrology. Artificial Intelligence in Medicine 111 (2021), 101982.
[45]
Anupam Datta, Shayak Sen, and Yair Zick. 2016. Algorithmic transparency via quantitative input influence: Theory and experiments with learning systems. In Proceedings of the 2016 IEEE Symposium on Security and Privacy (SP ’16), 598–617.
[46]
Omer Deperlioglu, Utku Kose, Deepak Gupta, Ashish Khanna, Fabio Giampaolo, and Giancarlo Fortino. 2022. Explainable framework for Glaucoma diagnosis by image processing and convolutional neural network synergy: Analysis with doctor evaluation. Future Generation Computer Systems 129 (2022), 152–169.
[47]
Shipi Dhanorkar, Christine T. Wolf, Kun Qian, Anbang Xu, Lucian Popa, and Yunyao Li. 2021. Who needs to know what, when?: Broadening the explainable AI (XAI) design space by looking at explanations across the AI lifecycle. In Proceedings of the Designing Interactive Systems Conference 2021, 1591–1602.
[48]
Amit Dhurandhar, Pin-Yu Chen, Ronny Luss, Chun-Chen Tu, Paishun Ting, Karthikeyan Shanmugam, and Payel Das. 2018. Explanations based on the missing: Towards contrastive explanations with pertinent negatives. In Advances in Neural Information Processing Systems, Vol. 31, 592–603.
[49]
Pedro Domingos. 2012. A few useful things to know about machine learning. Communications of the ACM 55, 10 (2012), 78–87.
[50]
Jiqian Dong, Sikai Chen, Mohammad Miralinaghi, Tiantian Chen, Pei Li, and Samuel Labi. 2023. Why did the AI make that decision? Towards an explainable artificial intelligence (XAI) for autonomous driving systems. Transportation Research Part C: Emerging Technologies 156 (2023), 104358.
[51]
Yinpeng Dong, Hang Su, Jun Zhu, and Bo Zhang. 2017. Improving interpretability of deep neural networks with semantic information. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 4306–4314.
[52]
Finale Doshi-Velez and Been Kim. 2017. Towards a rigorous science of interpretable machine learning. arXiv:1702.08608. Retrieved from http://arxiv.org/abs/1702.08608
[53]
Finale Doshi-Velez, Mason Kortz, Ryan Budish, Chris Bavitz, Sam Gershman, David O’Brien, Kate Scott, Stuart Schieber, James Waldo, David Weinberger, et al. 2017. Accountability of AI under the law: The role of explanation. arXiv:1711.01134. Retrieved from http://arxiv.org/abs/1711.01134
[54]
Jeff Druce, Michael Harradon, and James Tittle. 2019. Explainable artificial intelligence (XAI) for increasing user trust in deep reinforcement learning driven autonomous systems. In Proceedings of the NeurIPS 2019 Deep RL Workshop, 1–9. Retrieved from http://arxiv.org/abs/2106.03775
[55]
Joanita DSouza and Senthil Velan S. 2020. Using exploratory data analysis for generating inferences on the correlation of COVID-19 cases. In 2020 11th International Conference on Computing, Communication and Networking Technologies (ICCCNT ’20). IEEE, 1–6.
[56]
Hang Du, Hailin Shi, Dan Zeng, Xiao-Ping Zhang, and Tao Mei. 2020. The elements of end-to-end deep face recognition: A survey of recent advances. ACM Computing Surveys 54, 10s (2020), 1–42.
[57]
Upol Ehsan, Q. Vera Liao, Michael Muller, Mark O. Riedl, and Justin D. Weisz. 2021. Expanding explainability: Towards social transparency in AI systems. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, 1–19.
[58]
Upol Ehsan, Philipp Wintersberger, Q. Vera Liao, Martina Mara, Marc Streit, Sandra Wachter, Andreas Riener, and Mark O. Riedl. 2021. Operationalizing human-centered perspectives in explainable AI. In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems, 1–6.
[59]
Shaker El-Sappagh, Jose M. Alonso, S. M. Islam, Ahmad M. Sultan, and Kyung Sup Kwak. 2021. A multilayer multimodal detection and prediction model based on explainable artificial intelligence for Alzheimer's disease. Scientific Reports 11, 1 (2021), 1–26.
[60]
Mica R. Endsley. 2017. From here to autonomy: Lessons learned from human–automation research. Human Factors 59, 1 (2017), 5–27.
[61]
Dumitru Erhan, Yoshua Bengio, Aaron Courville, and Pascal Vincent. 2009. Visualizing higher-layer features of a deep network. University of Montreal 1341, 3 (2009), 1–13.
[62]
Dumitru Erhan, Aaron Courville, and Yoshua Bengio. 2010. Understanding Representations Learned in Deep Architectures. Technical Report 1355, Department dInformatique et Recherche Operationnelle, University of Montreal, QC, Canada.
[63]
Gary Ericson, William Anton Rohm, Josée Martens, Kent Sharkey, Craig Casey, Beth Harvey, and Nick Schonning. 2017. Team data science process documentation. Retrieved 11 April 2017 from. http://learn.microsoft.com/en-us/azure/architecture/data-science-process/overview
[64]
Magnus Falk. 2019. Artificial intelligence in the boardroom. Retrieved from https://www.fca.org.uk/insight/artificial-intelligence-boardroom
[65]
Fan Fang, Carmine Ventre, Lingbo Li, Leslie Kanthan, Fan Wu, and Michail Basios. 2020. Better model selection with a new definition of feature importance. arXiv:2009.07708. Retrieved from http://arxiv.org/abs/2009.07708
[66]
Juliana Jansen Ferreira and Mateus Monteiro. 2021. The human-AI relationship in decision-making: AI explanation to support people on justifying their decisions. arXiv:2102.05460. Retrieved from http://arxiv.org/abs/2102.05460
[67]
Juliana J. Ferreira and Mateus S. Monteiro. 2020. What are people doing about XAI user experience? A survey on AI explainability research and practice. In Proceedings of the International Conference on Human-Computer Interaction, 56–73.
[68]
Jerome H. Friedman. 2001. Greedy function approximation: A gradient boosting machine. Annals of Statistics 29, 5 (2001), 1189–1232.
[69]
Felix Friedrich, Wolfgang Stammer, Patrick Schramowski, and Kristian Kersting. 2023. A typology for exploring the mitigation of shortcut behaviour. Nature Machine Intelligence 5, 3 (2023), 319–330.
[70]
Detlev Gabel and Tim Hickman. 2019. GDPR handbook: Unlocking the EU general data protection regulation. White & Case Technology Newsflash (2019). Retrieved from https://www.whitecase.com/insight-our-thinking/gdpr-handbook-unlocking-eu-general-data-protection-regulation
[71]
Chen Gao, Tzu-Heng Lin, Nian Li, Depeng Jin, and Yong Li. 2023. Cross-platform item recommendation for online social e-commerce. IEEE Transactions on Knowledge and Data Engineering 35, 2 (2023), 1351–1364.
[72]
Julie Gerlings, Millie Søndergaard Jensen, and Arisa Shollo. 2022. Explainable AI, but explainable to whom? An exploratory case study of XAI in healthcare. In Handbook of Artificial Intelligence in Healthcare. Springer, 169–198.
[73]
Julie Gerlings, Arisa Shollo, and Ioanna Constantiou. 2020. Reviewing the need for explainable artificial intelligence (XAI). arXiv:2012.01007. Retrieved from http://arxiv.org/abs/2012.01007
[74]
Leilani H. Gilpin, David Bau, Ben Z. Yuan, Ayesha Bajwa, Michael Specter, and Lalana Kagal. 2018. Explaining explanations: An overview of interpretability of machine learning. In Proceedings of the 2018 IEEE 5th International Conference on Data Science and Advanced Analytics (DSAA ’18), 80–89.
[75]
Alex Goldstein, Adam Kapelner, Justin Bleich, and Emil Pitkin. 2015. Peeking inside the black box: Visualizing statistical learning with plots of individual conditional expectation. Journal of Computational and Graphical Statistics 24, 1 (2015), 44–65.
[76]
Przemyslaw A. Grabowicz, Nicholas Perello, and Aarshee Mishra. 2022. Marrying fairness and explainability in supervised learning. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, 1905–1916.
[77]
Yulia Grushetskaya, Mike Sips, Reyko Schachtschneider, Mohammadmehdi Saberioon, and Akram Mahan. 2024. HPExplorer: XAI method to explore the relationship between hyperparameters and model performance. In Proceedings of the Joint European Conference on Machine Learning and Knowledge Discovery in Databases. Springer, 319–334.
[78]
Venkat Gudivada, Amy Apon, and Junhua Ding. 2017. Data quality considerations for big data and machine learning: Going beyond data cleaning and transformations. International Journal on Advances in Software 10, 1 (2017), 1–20.
[79]
Riccardo Guidotti, Anna Monreale, Fosca Giannotti, Dino Pedreschi, Salvatore Ruggieri, and Franco Turini. 2019. Factual and counterfactual explanations for black box decision making. IEEE Intelligent Systems 34, 6 (2019), 14–23.
[80]
Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, Franco Turini, Fosca Giannotti, and Dino Pedreschi. 2018. A survey of methods for explaining black box models. ACM Computing Surveys 51, 5 (2018), 1–42.
[81]
Calvin Guillot Suarez. 2022. Human-in-the-Loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master's thesis. Aalto University. School of Electrical Engineering. Retrieved from http://urn.fi/URN:NBN:fi:aalto-202205223354
[82]
Karthik S. Gurumoorthy, Amit Dhurandhar, Guillermo Cecchi, and Charu Aggarwal. 2019. Efficient data representation by selecting prototypes with importance weights. In Proceedings of the 2019 IEEE International Conference on Data Mining (ICDM ’19), 260–269.
[83]
Mark Haakman, Luís Cruz, Hennie Huijgens, and Arie van Deursen. 2021. AI lifecycle models need to be revised. Empirical Software Engineering 26, 5 (2021), 1–29.
[84]
David Hardage and Peyman Najafirad. 2020. Hate and toxic speech detection in the context of covid-19 pandemic using XAI: Ongoing applied research. In Proceedings of the 1st Workshop on NLP for COVID-19 (Part 2) at EMNLP 2020, 1–5.
[85]
Michael Harradon, Jeff Druce, and Brian Ruttenberg. 2018. Causal learning and explanation of deep neural networks via autoencoded activations. arXiv:1802.00541. Retrieved from http://arxiv.org/abs/1802.00541
[86]
Alexander Heimerl, Katharina Weitz, Tobias Baur, and Elisabeth Andre. 2020. Unraveling ML models of emotion with nova: Multi-level explainable AI for non-experts. IEEE Transactions on Affective Computing 13, 3 (2020), 1155–1167.
[87]
Lisa Anne Hendricks, Zeynep Akata, Marcus Rohrbach, Jeff Donahue, Bernt Schiele, and Trevor Darrell. 2016. Generating visual explanations. In Proceedings of the European Conference on Computer Vision. Springer, 3–19.
[88]
Andreas Henelius, Kai Puolamäki, Henrik Boström, Lars Asker, and Panagiotis Papapetrou. 2014. A peek into the black box: Exploring classifiers by randomization. Data Mining and Knowledge Discovery 28, 5 (2014), 1503–1529.
[89]
Robert R. Hoffman, Shane T. Mueller, Gary Klein, and Jordan Litman. 2018. Metrics for explainable AI: Challenges and prospects. arXiv:1812.04608. Retrieved from http://arxiv.org/abs/1812.04608
[90]
Fred Hohman, Haekyu Park, Caleb Robinson, and Duen Horng Polo Chau. 2019. Summit: Scaling deep learning interpretability by visualizing activation and attribution summarizations. IEEE Transactions on Visualization and Computer Graphics 26, 1 (2019), 1096–1106.
[91]
Andreas Holzinger, Chris Biemann, Constantinos S. Pattichis, and Douglas B. Kell. 2017. What do we need to build explainable AI systems for the medical domain? arXiv:1712.09923. Retrieved from http://arxiv.org/abs/1712.09923
[92]
Changwu Huang, Zeqi Zhang, Bifei Mao, and Xin Yao. 2022. An overview of artificial intelligence ethics. IEEE Transactions on Artificial Intelligence 4, 4 (2023), 799–819.
[93]
Christina Humer, Andreas Hinterreiter, Benedikt Leichtmann, Martina Mara, and Marc Streit. 2024. Reassuring, misleading, debunking: Comparing effects of XAI methods on human decisions. ACM Transactions on Interactive Intelligent Systems 14, 3 (2024), 1–36.
[94]
Fatima Hussain, Rasheed Hussain, and Ekram Hossain. 2021. Explainable artificial intelligence (XAI): An engineering perspective. arXiv:2101.03613. Retrieved from http://arxiv.org/abs/2101.03613
[95]
Gandhi Jafta, Alta de Waal, Iena Derks, and Emma Ruttkamp-Bloem. 2022. Evaluation of XAI as an enabler for fairness, accountability and transparency. In Proceedings of the 2nd Southern African Conference for Artificial Intelligence Research, 541–542.
[96]
Helen Jiang and Erwen Senge. 2021. On two XAI cultures: A case study of non-technical explanations in deployed AI system. arXiv:2112.01016. Retrieved from http://arxiv.org/abs/2112.01016
[97]
Anna Jobin, Marcello Ienca, and Effy Vayena. 2019. The global landscape of AI ethics guidelines. Nature Machine Intelligence 1, 9 (2019), 389–399.
[98]
U. Johansson, R. König, and L. Niklasson. 2003. Rule extraction from trained neural networks using genetic programming. In Proceedings of the 13th International Conference on Artificial Neural Networks, 13–16.
[99]
Ulf Johansson and Lars Niklasson. 2009. Evolving decision trees using oracle guides. In Proceedings of the 2009 IEEE Symposium on Computational Intelligence and Data Mining. IEEE, 238–244.
[100]
Neesha Jothi, Wahidah Husain, and Nur’Aini Abdul Rashid. 2021. Predicting generalized anxiety disorder among women using shapley value. Journal of Infection and Public Health 14, 1 (2021), 103–108.
[101]
Minsuk Kahng, Pierre Y. Andrews, Aditya Kalro, and Duen Horng Chau. 2017. ActiVis: Visual exploration of industry-scale deep neural network models. IEEE Transactions on Visualization and Computer Graphics 24, 1 (2017), 88–97.
[102]
Uday Kamath and John Liu. 2021. Explainable Artificial Intelligence: An Introduction to Interpretable Machine Learning. Springer.
[103]
Amany A. Kandeel, Hazem M. Abbas, and Hossam S. Hassanein. 2021. Explainable model selection of a convolutional neural network for driver's facial emotion identification. In Proceedings of the International Conference on Pattern Recognition. Springer, 699–713.
[104]
Ramisetty Kavya, Jabez Christopher, Subhrakanta Panda, and Y. Bakthasingh Lazarus. 2021. Machine learning and XAI approaches for allergy diagnosis. Biomedical Signal Processing and Control 69 (2021), 102681.
[105]
Aditya Khamparia, Deepak Gupta, Ashish Khanna, and Valentina E. Balas. 2022. Biomedical Data Analysis and Processing Using Explainable (XAI) and Responsive Artificial Intelligence (RAI). Springer.
[106]
Nadia Khan, Muhammad Nauman, Ahmad S. Almadhor, Nadeem Akhtar, Abdullah Alghuried, and Adi Alhudhaif. 2024. Guaranteeing correctness in black-box machine learning: A fusion of explainable AI and formal methods for healthcare decision-making. IEEE Access 12, (2024), 90299–90316.
[107]
Jinkyu Kim, Anna Rohrbach, Trevor Darrell, John Canny, and Zeynep Akata. 2018. Textual explanations for self-driving vehicles. In Proceedings of the European Conference on Computer Vision (ECCV ’18), 563–578.
[108]
Miryung Kim, Thomas Zimmermann, Robert DeLine, and Andrew Begel. 2016. The emerging role of data scientists on software development teams. In Proceedings of the 2016 IEEE/ACM 38th International Conference on Software Engineering (ICSE ’16). IEEE, 96–107.
[109]
Inna Kolyshkina and Simeon Simoff. 2019. Interpretability of machine learning solutions in industrial decision engineering. In Proceedings of the Australasian Conference on Data Mining. Springer, 156–170.
[110]
Matthieu Komorowski, Dominic C. Marshall, Justin D. Salciccioli, and Yves Crutain. 2016. Exploratory data analysis. Springer International Publishing, Cham, 185–203.
[111]
Josua Krause, Adam Perer, and Kenney Ng. 2016. Interacting with predictions: Visual inspection of black-box machine learning models. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, 5686–5697.
[112]
R. Krishnan, G. Sivakumar, and P. Bhattacharya. 1999. Extracting decision trees from trained neural networks. Pattern Recognition 32, 12 (1999), 1999–2009.
[113]
Jean-Philippe Kröll, Simon B. Eickhoff, Felix Hoffstaedter, and Kaustubh R. Patil. 2020. Evolving complex yet interpretable representations: Application to Alzheimer's diagnosis and prognosis. In Proceedings of the 2020 IEEE Congress on Evolutionary Computation (CEC ’20). IEEE, 1–8.
[114]
Todd Kulesza, Margaret Burnett, Weng-Keen Wong, and Simone Stumpf. 2015. Principles of explanatory debugging to personalize interactive machine learning. In Proceedings of the 20th International Conference on Intelligent User Interfaces, 126–137.
[115]
Markus Langer, Daniel Oster, Timo Speith, Holger Hermanns, Lena Kästner, Eva Schmidt, Andreas Sesing, and Kevin Baum. 2021. What do we want from explainable artificial intelligence (XAI)?–A stakeholder perspective on XAI and a conceptual model guiding interdisciplinary XAI research. Artificial Intelligence 296 (2021), 103473.
[116]
Thibault Laugel, Marie-Jeanne Lesot, Christophe Marsala, Xavier Renard, and Marcin Detyniecki. 2017. Inverse classification for comparison-based interpretability in machine learning. arXiv:1712.08443. Retrieved from http://arxiv.org/abs/1712.08443
[117]
Tao Lei, Regina Barzilay, and Tommi Jaakkola. 2016. Rationalizing neural predictions. arXiv:1606.04155. Retrieved from http://arxiv.org/abs/1606.04155
[118]
Bruno Lepri, Nuria Oliver, Emmanuel Letouzé, Alex Pentland, and Patrick Vinck. 2018. Fair, transparent, and accountable algorithmic decision-making processes. Philosophy & Technology 31, 4 (2018), 611–627.
[119]
Oscar Li, Hao Liu, Chaofan Chen, and Cynthia Rudin. 2018. Deep learning for case-based reasoning through prototypes: A neural network that explains its predictions. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 32, 3530–3537.
[120]
Q. Vera Liao, Daniel Gruen, and Sarah Miller. 2020. Questioning the AI: Informing design practices for explainable AI user experiences. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, 1–15.
[121]
Q. Vera Liao and Kush R. Varshney. 2021. Human-centered explainable AI (XAI): From algorithms to user experiences. arXiv:2110.10790. Retrieved from http://arxiv.org/abs/2110.10790
[122]
Brian Y. Lim and Anind K. Dey. 2010. Toolkit to support intelligibility in context-aware applications. In Proceedings of the 12th ACM International Conference on Ubiquitous Computing, 13–22.
[123]
Brian Y. Lim, Anind K. Dey, and Daniel Avrahami. 2009. Why and why not explanations improve the intelligibility of context-aware intelligent systems. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 2119–2128.
[124]
Gabriel Lima, Nina Grgić-Hlača, Jin Keun Jeong, and Meeyoung Cha. 2022. The conflict between explainable and accountable decision-making algorithms. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, 2103–2113.
[125]
Pantelis Linardatos, Vasilis Papastefanopoulos, and Sotiris Kotsiantis. 2020. Explainable AI: A review of machine learning interpretability methods. Entropy 23, 1 (2020), 18.
[126]
Mengchen Liu, Jiaxin Shi, Kelei Cao, Jun Zhu, and Shixia Liu. 2017. Analyzing the training processes of deep generative models. IEEE Transactions on Visualization and Computer Graphics 24, 1 (2017), 77–87.
[127]
Jörn Lötsch, Dario Kringel, and Alfred Ultsch. 2021. Explainable artificial intelligence (XAI) in biomedicine: Making AI decisions trustworthy for physicians and patients. BioMedInformatics 2, 1 (2021), 1–17.
[128]
Yin Lou, Rich Caruana, and Johannes Gehrke. 2012. Intelligible models for classification and regression. In Proceedings of the 18th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 150–158.
[129]
Yin Lou, Rich Caruana, Johannes Gehrke, and Giles Hooker. 2013. Accurate intelligible models with pairwise interactions. In Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 623–631.
[130]
Scott M. Lundberg and Su-In Lee. 2017. A unified approach to interpreting model predictions. In Advances in Neural Information Processing Systems, 4768–4777.
[131]
Scott M. Lundberg, Bala Nair, Monica S. Vavilala, Mayumi Horibe, Michael J. Eisses, Trevor Adams, David E. Liston, Daniel King-Wai Low, Shu-Fang Newman, Jerry Kim, et al. 2018. Explainable machine-learning predictions for the prevention of hypoxaemia during surgery. Nature Biomedical Engineering 2, 10 (2018), 749–760.
[132]
Minh-Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Effective approaches to attention-based neural machine translation. arXiv:1508.04025. Retrieved from http://arxiv.org/abs/1508.04025
[133]
Avleen Malhi, Samanta Knapic, and Kary Främling. 2020. Explainable agents for less bias in human-agent decision making. In Proceedings of the International Workshop on Explainable, Transparent Autonomous Agents and Multi-Agent Systems. Springer, 129–146.
[134]
Nicholas Maltbie, Nan Niu, Matthew Van Doren, and Reese Johnson. 2021. XAI tools in the public sector: A case study on predicting combined sewer overflows. In Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 1032–1044.
[135]
Wilson E. Marcílio and Danilo M. Eler. 2020. From explanations to feature selection: Assessing SHAP values as feature selection mechanism. In Proceedings of the 2020 33rd SIBGRAPI Conference on Graphics, Patterns and Images (SIBGRAPI ’20). IEEE, 340–347.
[136]
Andrés R. Masegosa, Ana M. Martínez, Darío Ramos-López, Helge Langseth, Thomas D. Nielsen, and Antonio Salmerón. 2020. Analyzing concept drift: A case study in the financial sector. Intelligent Data Analysis 24, 3 (2020), 665–688.
[137]
Kate Matsudaira. 2015. The science of managing data science. Communications of the ACM 58, 6 (2015), 44–47.
[138]
John A. McDermid, Yan Jia, Zoe Porter, and Ibrahim Habli. 2021. Artificial intelligence explainability: The technical and ethical dimensions. Philosophical Transactions of the Royal Society A 379, 2207 (2021), 20200363.
[139]
Linhao Meng, Stef Van Den Elzen, and Anna Vilanova. 2022. ModelWise: Interactive model comparison for model diagnosis, improvement and selection. Computer Graphics Forum 41 (2022), 97–108.
[140]
Christian Meske, Enrico Bunde, Johannes Schneider, and Martin Gersch. 2022. Explainable artificial intelligence: Objectives, stakeholders, and future research opportunities. Information Systems Management 39, 1 (2022), 53–63.
[141]
Agnieszka Mikołajczyk, Michał Grochowski, and Arkadiusz Kwasigroch. 2021. Towards explainable classifiers using the counterfactual approach-global explanations for discovering bias in data. Journal of Artificial Intelligence and Soft Computing Research 11, 1 (2021), 51–67.
[142]
Tim Miller. 2019. Explanation in artificial intelligence: Insights from the social sciences. Artificial Intelligence 267 (2019), 1–38.
[143]
Yao Ming, Huamin Qu, and Enrico Bertini. 2018. Rulematrix: Visualizing and understanding classifiers with rules. IEEE Transactions on Visualization and Computer Graphics 25, 1 (2018), 342–352.
[144]
Brent Daniel Mittelstadt, Patrick Allo, Mariarosaria Taddeo, Sandra Wachter, and Luciano Floridi. 2016. The ethics of algorithms: Mapping the debate. Big Data & Society 3, 2 (2016), 2053951716679679.
[145]
Sina Mohseni, Niloofar Zarei, and Eric D. Ragan. 2021. A multidisciplinary survey and framework for design and evaluation of explainable AI systems. ACM Transactions on Interactive Intelligent Systems 11, 3–4 (2021), 1–45.
[146]
Christoph Molnar. 2020. Interpretable Machine Learning. Lulu. com.
[147]
Isaac Monteath and Raymond Sheh. 2018. Assisted and incremental medical diagnosis using explainable artificial intelligence. In Proceedings of the 2nd Workshop on Explainable Artificial Intelligence, 104–108.
[148]
Pedro A. Moreno-Sanchez. 2021. An automated feature selection and classification pipeline to improve explainability of clinical prediction models. In Proceedings of the 2021 IEEE 9th International Conference on Healthcare Informatics (ICHI ’21). IEEE, 527–534.
[149]
Edoardo Mosca, Maximilian Wich, and Georg Groh. 2021. Understanding and interpreting the impact of user context in hate speech detection. In Proceedings of the 9th International Workshop on Natural Language Processing for Social Media, 91–102.
[150]
Sakib Mostafa, Debajyoti Mondal, Michael Beck, Christopher Bidinosti, Christopher Henry, and Ian Stavness. 2021. Visualizing feature maps for model selection in convolutional neural networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 1362–1371.
[151]
Ramaravind K. Mothilal, Amit Sharma, and Chenhao Tan. 2020. Explaining machine learning classifiers through diverse counterfactual explanations. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, 607–617.
[152]
F. G. Mourik. 2023. IterSHAP: An XAI Feature Selection Method for Small High-Dimensional Datasets. Master's thesis. University of Twente.
[153]
Shane T. Mueller. 2020. Cognitive anthropomorphism of AI: How humans and computers classify images. Ergonomics in Design 28, 3 (2020), 12–19.
[154]
Fatemeh Nargesian, Horst Samulowitz, Udayan Khurana, Elias B. Khalil, and Deepak S. Turaga. 2017. Learning feature engineering for classification. In Proceedings of the 26th International Joint Conference on Artificial Intelligence, 2529–2535.
[155]
Shweta Narkar, Yunfeng Zhang, Q. Vera Liao, Dakuo Wang, and Justin D. Weisz. 2021. Model lineUpper: Supporting interactive model comparison at multiple levels for autoML. In Proceedings of the 26th International Conference on Intelligent User Interfaces, 170–174.
[156]
Tien N. Nguyen and Raymond Choo. 2021. Human-in-the-loop XAI-enabled vulnerability detection, investigation, and mitigation. In Proceedings of the 2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE ’21). IEEE, 1210–1212.
[157]
Chris Olah, Arvind Satyanarayan, Ian Johnson, Shan Carter, Ludwig Schubert, Katherine Ye, and Alexander Mordvintsev. 2018. The building blocks of interpretability. Distill 3, 3 (2018), e10.
[158]
Julian D. Olden and Donald A. Jackson. 2002. Illuminating the “black box”: A randomization approach for understanding variable contributions in artificial neural networks. Ecological Modelling 154, 1–2 (2002), 135–150.
[159]
Lara O’Reilly. 2014. Here's one way to find out which advertisers are tracking you across the internet. Retrieved from https://www.businessinsider.com/floodwatch-ad-tracking-chrome-extension-2014-10
[160]
Dieudonne N. Ouedraogo. 2021. Interpretable machine learning model selection for breast cancer diagnosis based on k-means clustering. Applied Medical Informatics 43, 3 (2021), 91–102.
[161]
James Overton. 2011. Scientific explanation and computation. In ExaCt, 41–50.
[162]
Frederik Pahde, Maximilian Dreyer, Wojciech Samek, and Sebastian Lapuschkin. 2023. Reveal to revise: An explainable AI life cycle for iterative bias correction of deep models. In Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, 596–606.
[163]
Iam Palatnik de Sousa, Marley M. B. R. Vellasco, and Eduardo Costa da Silva. 2021. Explainable artificial intelligence for bias detection in covid CT-Scan classifiers. Sensors 21, 16 (2021), 5657.
[164]
Raja Parasuraman, Thomas B. Sheridan, and Christopher D. Wickens. 2008. Situation awareness, mental workload, and trust in automation: Viable, empirically supported cognitive engineering constructs. Journal of Cognitive Engineering and Decision Making 2, 2 (2008), 140–160.
[165]
Ankita Ramjibhai Patel, Jaganmohan Chandrasekaran, Yu Lei, Raghu N. Kacker, and D. Richard Kuhn. 2022. A combinatorial approach to fairness testing of machine learning models. In Proceedings of the 2022 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW ’22). IEEE, 94–101.
[166]
Junfeng Peng, Kaiqiang Zou, Mi Zhou, Yi Teng, Xiongyong Zhu, Feifei Zhang, and Jun Xu. 2021. An explainable artificial intelligence framework for the deterioration risk prediction of hepatitis patients. Journal of Medical Systems 45, 5 (2021), 1–9.
[167]
Jeremy Petch, Shuang Di, and Walter Nelson. 2022. Opening the black box: The promise and limitations of explainable machine learning in cardiology. Canadian Journal of Cardiology 38, 2 (2022), 204–213.
[168]
Nicola Pezzotti, Thomas Höllt, Jan Van Gemert, Boudewijn P. F. Lelieveldt, Elmar Eisemann, and Anna Vilanova. 2017. Deepeyes: Progressive visual analytics for designing deep neural networks. IEEE Transactions on Visualization and Computer Graphics 24, 1 (2017), 98–108.
[169]
Anna Polzer, Jürgen Fleiß, Thomas Ebner, Philipp Kainz, Christoph Koeth, and Stefan Thalmann. 2022. Validation of AI-based information systems for sensitive use cases: Using an XAI approach in pharmaceutical engineering. In Proceedings of the 55th Hawaii International Conference on System Sciences, 1500–1509.
[170]
Romila Pradhan, Jiongli Zhu, Boris Glavic, and Babak Salimi. 2022. Interpretable data-based explanations for fairness debugging. In Proceedings of the 2022 International Conference on Management of Data, 247–261.
[171]
Alun Preece, Dan Harborne, Dave Braines, Richard Tomsett, and Supriyo Chakraborty. 2018. Stakeholders in explainable AI. arXiv:1810.00184. Retrieved from http://arxiv.org/abs/1810.00184
[172]
Pearl Pu and Li Chen. 2007. Trust-inspiring explanation interfaces for recommender systems. Knowledge-Based Systems 20, 6 (2007), 542–556.
[173]
Jaber Rad, Karthik K. Tennankore, Amanda Vinson, and Syed Sibte Raza Abidi. 2022. Extracting surrogate decision trees from black-box models to explain the temporal importance of clinical features in predicting kidney graft survival. In Proceedings of the International Conference on Artificial Intelligence in Medicine. Springer, 88–98.
[174]
Enayat Rajabi and Kobra Etminani. 2021. Towards a knowledge graph-based explainable decision support system in Healthcare. Studies in Health Technology and Informatics 281 (2021), 502–503.
[175]
Gabrielle Ras, Ning Xie, Marcel van Gerven, and Derek Doran. 2022. Explainable deep learning: A field guide for the uninitiated. Journal of Artificial Intelligence Research 73 (2022), 329–397.
[176]
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016. ‘Why should I trust you?’ Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 1135–1144.
[177]
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2018. Anchors: High-precision model-agnostic explanations. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 32, 1527–1535.
[178]
Laura Rieger, Chandan Singh, William Murdoch, and Bin Yu. 2020. Interpretations are useful: Penalizing explanations to align neural networks with prior knowledge. In Proceedings of the 37th International Conference on Machine Learning. PMLR, 8116–8126.
[179]
Avi Rosenfeld and Ariella Richardson. 2019. Explainability in human–agent systems. Autonomous Agents and Multi-Agent Systems 33, 6 (2019), 673–705.
[180]
Patrik Sabol, Peter Sinčák, Pitoyo Hartono, Pavel Kočan, Zuzana Benetinová, Alžbeta Blichárová, L’udmila Verbóová, Erika Štammová, Antónia Sabolová-Fabianová, et al. 2020. Explainable classifier for improving the accountability in decision-making for colorectal cancer diagnosis from histopathological images. Journal of Biomedical Informatics 109 (2020), 103523.
[181]
Waddah Saeed and Christian Omlin. 2023. Explainable AI (XAI): A systematic meta-survey of current challenges and future opportunities. Knowledge-Based Systems 263 (2023), 110273.
[182]
Wojciech Samek and Klaus-Robert Müller. 2019. Towards Explainable Artificial Intelligence. Springer, 5–22.
[183]
Wojciech Samek, Thomas Wiegand, and Klaus-Robert Müller. 2017. Explainable artificial intelligence: Understanding, visualizing and interpreting deep learning models. arXiv:1708.08296. Retrieved from http://arxiv.org/abs/1708.08296
[184]
Salih Sarp, Murat Kuzlu, Emmanuel Wilson, Umit Cali, and Ozgur Guler. 2021. The enlightening role of explainable artificial intelligence in chronic wound classification. Electronics 10, 12 (2021), 1406.
[185]
N. B. Sarter and D. D. Woods. 1995. Autonomy, authority, and observability: The evolution of critical automation properties and their impact on man-machine coordination and cooperation. In Proceedings of the 6th IFAC/IFIP/IFORS/IEA Symposium on Analysis, Design, and Evaluation of Man-Machine Systems, 149–152.
[186]
Benajmin Schmidt. 2016. A public exploratory data analysis of gender bias in teaching evaluations. In Proceedings of Workshop on Visualization for the Digital Humanities (Vis4DH ’16), 1–4.
[187]
Tjeerd A. J. Schoonderwoerd, Wiard Jorritsma, Mark A. Neerincx, and Karel Van Den Bosch. 2021. Human-centered XAI: Developing design patterns for explanations of clinical decision support systems. International Journal of Human-Computer Studies 154 (2021), 102684.
[188]
Patrick Schramowski, Wolfgang Stammer, Stefano Teso, Anna Brugger, Franziska Herbert, Xiaoting Shao, Hans-Georg Luigs, Anne-Katrin Mahlein, and Kristian Kersting. 2020. Making deep neural networks right for the right scientific reasons by interacting with their explanations. Nature Machine Intelligence 2, 8 (2020), 476–486.
[189]
Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. 2017. Grad-CAM: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE International Conference on Computer Vision, 618–626.
[190]
Rudy Setiono and Huan Liu. 1995. Understanding neural networks via rule extraction. In Proceedings of the 14th International Joint Conference on Artificial Intelligence, Vol. 1, 480–485.
[191]
Arash Shaban-Nejad, Martin Michalowski, John S. Brownstein, and David L. Buckeridge. 2021. Guest editorial explainable AI: Towards fairness, accountability, transparency and trust in healthcare. IEEE Journal of Biomedical and Health Informatics 25, 7 (2021), 2374–2375.
[192]
Hetan Shah. 2018. Algorithmic accountability. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences 376, 2128 (2018), 20170362.
[193]
Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. 2017. Learning important features through propagating activation differences. In Proceedings of the 34th International Conference on Machine Learning. PMLR, 3145–3153.
[194]
Andrew Silva, Mariah Schrum, Erin Hedlund-Botti, Nakul Gopalan, and Matthew Gombolay. 2023. Explainable artificial intelligence: Evaluating the objective and subjective impacts of XAI on human-agent interaction. International Journal of Human–Computer Interaction 39, 7 (2023), 1390–1404.
[195]
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. 2014. Deep inside convolutional networks: Visualising image classification models and saliency maps. In Proceedings of the Workshop at International Conference on Learning Representations, 1–8. Citeseer.
[196]
Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Viégas, and Martin Wattenberg. 2017. SmoothGrad: Removing noise by adding noise. arXiv:1706.03825. Retrieved from http://arxiv.org/abs/1706.03825
[197]
Peter Sollich. 2002. Bayesian methods for support vector machines: Evidence and predictive class probabilities. Machine Learning 46, 1 (2002), 21–52.
[198]
Thilo Spinner, Daniel Fürst, and Mennatallah El-Assady. 2024. iNNspector: Visual, interactive deep model debugging. arXiv:2407.17998. Retrieved from http://arxiv.org/abs/2407.17998
[199]
Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, and Martin Riedmiller. 2014. Striving for simplicity: The all convolutional net. arXiv:1412.6806. Retrieved from https://arxiv.org/abs/1412.6806
[200]
Wolfgang Stammer, Patrick Schramowski, and Kristian Kersting. 2021. Right for the right concept: Revising neuro-symbolic concepts by interacting with their explanations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 3619–3629.
[201]
Pierre Stock and Moustapha Cisse. 2018. Convnets and imagenet beyond accuracy: Understanding mistakes and uncovering biases. In Proceedings of the European Conference on Computer Vision (ECCV ’18), 498–512.
[202]
Jungyo Suh, Sangjun Yoo, Juhyun Park, Sung Yong Cho, Min Chul Cho, Hwancheol Son, and Hyeon Jeong. 2020. Development and validation of an explainable artificial intelligence-based decision-supporting tool for prostate biopsy. BJU International 126, 6 (2020), 694–703.
[203]
Shimon Sumita, Hiroyuki Nakagawa, and Tatsuhiro Tsuchiya. 2023. Xtune: An XAI-based hyperparameter tuning method for time-series forecasting using deep learning. Retrieved from http://www.researchsquare.com/article/rs-3008932/v1
[204]
Dong Sun, Zezheng Feng, Yuanzhe Chen, Yong Wang, Jia Zeng, Mingxuan Yuan, Ting-Chuen Pong, and Huamin Qu. 2020. DFSeer: A visual analytics approach to facilitate model selection for demand forecasting. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, 1–13.
[205]
Harini Suresh, Steven R. Gomez, Kevin K. Nam, and Arvind Satyanarayan. 2021. Beyond expertise and roles: A framework to characterize the stakeholders of interpretable machine learning and their needs. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, 1–16.
[206]
Jianrong Tao, Yu Xiong, Shiwei Zhao, Yuhong Xu, Jianshi Lin, Runze Wu, and Changjie Fan. 2020. XAI-driven explainable multi-view game cheating detection. In Proceedings of the 2020 IEEE Conference on Games (CoG ’20). IEEE, 144–151.
[207]
Stefano Teso and Kristian Kersting. 2019. Explanatory interactive machine learning. In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society, 239–245.
[208]
Patanamon Thongtanunam, Xin Yang, Norihiro Yoshida, Raula Gaikovina Kula, Ana Erika Camargo Cruz, Kenji Fujiwara, and Hajimu Iida. 2014. ReDA: A web-based visualization tool for analyzing modern code review dataset. In Proceedings of the 2014 IEEE International Conference on Software Maintenance and Evolution. IEEE, 605–608.
[209]
Gabriele Tolomei, Fabrizio Silvestri, Andrew Haines, and Mounia Lalmas. 2017. Interpretable predictions of tree-based ensembles via actionable feature tweaking. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 465–474.
[210]
Richard Tomsett, Dave Braines, Dan Harborne, Alun Preece, and Supriyo Chakraborty. 2018. Interpretable to whom? A role-based model for analyzing interpretable machine learning systems. arXiv:1806.07552. Retrieved from http://arxiv.org/abs/1806.07552
[211]
Geoffrey G. Towell and Jude W. Shavlik. 1993. Extracting refined rules from knowledge-based neural networks. Machine Learning 13, 1 (1993), 71–101.
[212]
Sandhya Tripathi, N. Hemachandra, and Prashant Trivedi. 2020. Interpretable feature subset selection: A Shapley value based approach. In Proceedings of the 2020 IEEE International Conference on Big Data (Big Data ’20). IEEE, 5463–5472.
[213]
Stéphane Tufféry. 2011. Data Mining and Statistics for Decision Making. John Wiley & Sons.
[214]
European Union. 2018. General data protection regulation (GDPR). Retrieved from https://gdpr-info.eu/
[215]
Jasper van der Waa, Tjeerd Schoonderwoerd, Jurriaan van Diggelen, and Mark Neerincx. 2020. Interpretable confidence measures for decision support systems. International Journal of Human-Computer Studies 144 (2020), 102493.
[216]
Corne Van Zyl, Xianming Ye, and Raj Naidoo. 2024. Harnessing eXplainable artificial intelligence for feature selection in time series energy forecasting: A comparative analysis of Grad-CAM and SHAP. Applied Energy 353 (2024), 122079.
[217]
Sahil Verma, John Dickerson, and Keegan Hines. 2024. Counterfactual explanations and algorithmic recourses for machine learning: A review. ACM Computing Surveys 56, 12 (2024), 1–42.
[218]
Tom Vermeire, Thibault Laugel, Xavier Renard, David Martens, and Marcin Detyniecki. 2021. How to choose an explainability method? Towards a methodical implementation of XAI in practice. In Proceedings of the Joint European Conference on Machine Learning and Knowledge Discovery in Databases. Springer, 521–533.
[219]
Trent W. Victor, Emma Tivesten, Pär Gustavsson, Joel Johansson, Fredrik Sangberg, and Mikael Ljung Aust. 2018. Automation expectation mismatch: Incorrect prediction despite eyes on threat and hands on wheel. Human Factors 60, 8 (2018), 1095–1116.
[220]
Marina M.-C. Vidovic, Nico Görnitz, Klaus-Robert Müller, and Marius Kloft. 2016. Feature importance measure for non-linear learning algorithms. arXiv:1611.07567. Retrieved from http://arxiv.org/abs/1611.07567
[221]
Giulia Vilone and Luca Longo. 2020. Explainable artificial intelligence: A systematic review. arXiv:2006.00093. Retrieved from http://arxiv.org/abs/2006.00093
[222]
Klaus Virtanen. 2022. Using XAI tools to detect harmful bias in ML models. Bachelor's Thesis. Umeå University, Department of Computing Science.
[223]
Sandra Wachter, Brent Mittelstadt, and Chris Russell. 2017. Counterfactual explanations without opening the black box: Automated decisions and the GDPR. Harvard Journal of Law & Technology 31 (2017), 841.
[224]
Syed Wali and Irfan Khan. 2021. Explainable AI and random forest based reliable intrusion detection system. DOI:
[225]
Dakuo Wang, Justin D. Weisz, Michael Muller, Parikshit Ram, Werner Geyer, Casey Dugan, Yla Tausczik, Horst Samulowitz, and Alexander Gray. 2019. Human-AI collaboration in data science: Exploring data scientists’ perceptions of automated AI. In Proceedings of the ACM on Human-Computer Interaction, Vol. 3, 1–24.
[226]
Maonan Wang, Kangfeng Zheng, Yanqing Yang, and Xiujuan Wang. 2020. An explainable machine learning framework for intrusion detection systems. IEEE Access 8 (2020), 73127–73141.
[227]
Tong Wang. 2019. Gaining free or low-cost interpretability with interpretable partial substitute. In Proceedings of the International Conference on Machine Learning. PMLR, 6505–6514.
[228]
Ziming Wang, Changwu Huang, Yun Li, and Xin Yao. 2024. Multi-objective feature attribution explanation for explainable machine learning. ACM Transactions on Evolutionary Learning and Optimization 4, 1 (2024), 1–32.
[229]
Ziming Wang, Changwu Huang, and Xin Yao. 2023. Feature attribution explanation to detect harmful dataset shift. In Proceedings of the 2023 International Joint Conference on Neural Networks (IJCNN ’23). IEEE, 1–8.
[230]
Ziming Wang, Changwu Huang, and Xin Yao. 2024. Procedural fairness in machine learning. arXiv:2404.01877. Retrieved from http://arxiv.org/abs/2404.01877
[231]
Geoffrey I Webb, Loong Kuan Lee, François Petitjean, and Bart Goethals. 2017. Understanding concept drift. arXiv:1704.00362. http://arxiv.org/abs/1704.00362
[232]
Katharina Weitz, Dominik Schiller, Ruben Schlagowski, Tobias Huber, and Elisabeth André. 2019. ‘Do you trust me?’ Increasing user-trust by integrating virtual agents in explainable AI interaction design. In Proceedings of the 19th ACM International Conference on Intelligent Virtual Agents, 7–9.
[233]
Adrian Weller. 2017. Challenges for transparency. arXiv:1708.01870. Retrieved from http://arxiv.org/abs/1708.01870
[234]
Maximilian Wich, Jan Bauer, and Georg Groh. 2020. Impact of politically biased data on hate speech classification. In Proceedings of the 4th Workshop on Online Abuse and Harms, 54–64.
[235]
Andrew C. Wicks, Shawn L. Berman, and Thomas M. Jones. 1999. The structure of optimal trust: Moral and strategic implications. Academy of Management Review 24, 1 (1999), 99–116.
[236]
Gesa Wiegand, Matthias Schmidmaier, Thomas Weber, Yuanting Liu, and Heinrich Hussmann. 2019. I drive-you trust: Explaining driving behavior of autonomous cars. In Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems, 1–6.
[237]
Rüdiger Wirth and Jochen Hipp. 2000. CRISP-DM: Towards a standard process model for data mining. In Proceedings of the 4th International Conference on the Practical Applications of Knowledge Discovery and Data Mining, Vol. 1, Manchester, 29–39.
[238]
Kanit Wongsuphasawat, Daniel Smilkov, James Wexler, Jimbo Wilson, Dandelion Mane, Doug Fritz, Dilip Krishnan, Fernanda B. Viégas, and Martin Wattenberg. 2017. Visualizing dataflow graphs of deep learning models in tensorflow. IEEE Transactions on Visualization and Computer Graphics 24, 1 (2017), 1–12.
[239]
Yao Xie, Melody Chen, David Kao, Ge Gao, and Xiang’Anthony’ Chen. 2020. CheXplain: Enabling physicians to explore and understand data-driven, AI-enabled medical imaging analysis. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, 1–13.
[240]
Zhi Yang, Ziming Wang, Changwu Huang, and Xin Yao. 2023. An explainable feature selection approach for fair machine learning. In Proceedings of the International Conference on Artificial Neural Networks. Springer, 75–86.
[241]
Qinghao Ye, Jun Xia, and Guang Yang. 2021. Explainable AI for COVID-19 CT classifiers: An initial comparison study. In Proceedings of the 2021 IEEE 34th International Symposium on Computer-Based Medical Systems (CBMS ’21). IEEE, 521–526.
[242]
Jason Yosinski, Jeff Clune, Anh Nguyen, Thomas Fuchs, and Hod Lipson. 2015. Understanding neural networks through deep visualization. arXiv:1506.06579. Retrieved from http://arxiv.org/abs/1506.06579
[243]
Mohammad Zaeri-Amirani, Fatemeh Afghah, and Sajad Mousavi. 2018. A feature selection method based on shapley value to false alarm reduction in ICUs a genetic-algorithm approach. In Proceedings of the 2018 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC ’18). IEEE, 319–323.
[244]
Matthew D. Zeiler and Rob Fergus. 2014. Visualizing and understanding convolutional networks. In Proceedings of the European Conference on Computer Vision. Springer, 818–833.
[245]
Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. From recognition to cognition: Visual commonsense reasoning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 6720–6731.
[246]
Chanyuan Abigail Zhang, Soohyun Cho, and Miklos Vasarhelyi. 2022. Explainable artificial intelligence (XAI) in auditing. International Journal of Accounting Information Systems 46, (2022), 100572.
[247]
Junzhe Zhang and Elias Bareinboim. 2018. Fairness in decision-making—The causal explanation formula. In Proceedings of the 32nd AAAI Conference on Artificial Intelligence, 2037–2045.
[248]
Jianming Zhang, Sarah Adel Bargal, Zhe Lin, Jonathan Brandt, Xiaohui Shen, and Stan Sclaroff. 2018. Top-down neural attention by excitation backprop. International Journal of Computer Vision 126, 10 (2018), 1084–1102.
[249]
Ke Zhang, Jun Zhang, Pei-Dong Xu, Tianlu Gao, and David Wenzhong Gao. 2021. Explainable AI in deep reinforcement learning models for power system emergency control. IEEE Transactions on Computational Social Systems 9, 2 (2021), 419–427.
[250]
Quanshi Zhang, Ruiming Cao, Ying Nian Wu, and Song-Chun Zhu. 2017. Mining object parts from CNNs via active question-answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 346–355.
[251]
Quanshi Zhang, Ruiming Cao, Feng Shi, Ying Nian Wu, and Song-Chun Zhu. 2018. Interpreting CNN knowledge via an explanatory graph. In Proceedings of the 32nd AAAI Conference on Artificial Intelligence, 4454–4463.
[252]
Quanshi Zhang, Ruiming Cao, Ying Nian Wu, and Song-Chun Zhu. 2017. Growing interpretable part graphs on convnets via multi-shot learning. In Proceedings of 31st the AAAI Conference on Artificial Intelligence, 2898–2906.
[253]
Quanshi Zhang, Jie Ren, Ge Huang, Ruiming Cao, Ying Nian Wu, and Song-Chun Zhu. 2020. Mining interpretable AOG representations from convolutional networks via active question answering. IEEE Transactions on Pattern Analysis and Machine Intelligence 43, 11 (2020), 3949–3963.
[254]
Quanshi Zhang, Xin Wang, Ruiming Cao, Ying Nian Wu, Feng Shi, and Song-Chun Zhu. 2020. Extraction of an explanatory graph to interpret a CNN. IEEE Transactions on Pattern Analysis and Machine Intelligence 43, 11 (2020), 3863–3877.
[255]
Quanshi Zhang, Ying Nian Wu, and Song-Chun Zhu. 2018. Interpretable convolutional neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 8827–8836.
[256]
Quanshi Zhang, Yu Yang, Haotian Ma, and Ying Nian Wu. 2019. Interpreting CNNs via decision trees. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 6261–6270.
[257]
Yu Zhang, Peter Tiňo, Aleš Leonardis, and Ke Tang. 2021. A survey on neural network interpretability. IEEE Transactions on Emerging Topics in Computational Intelligence 5, 5 (2021), 726–742.
Index Terms
- A Roadmap of Explainable Artificial Intelligence: Explain to Whom, When, What and How?
Recommendations
Explainable Artificial Intelligence: Requirements for Explainability
SIGSIM-PADS '22: Proceedings of the 2022 ACM SIGSIM Conference on Principles of Advanced Discrete SimulationTo date, many reasons have been suggested for making explainable artificial intelligence (XAI) models. However, it is unclear when the XAI suggested content is considered an explanation. This paper conducts a survey to determine the requirements for ...
Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI
Highlights- We review concepts related to the explainability of AI methods (XAI).
- We comprehensive analyze the XAI literature organized in two taxonomies.
- We identify future research directions of the XAI field.
- We discuss potential ...
AbstractIn the last few years, Artificial Intelligence (AI) has achieved a notable momentum that, if harnessed appropriately, may deliver the best of expectations over many application sectors across the field. For this to occur shortly in Machine ...
A Review of Taxonomies of Explainable Artificial Intelligence (XAI) Methods
FAccT '22: Proceedings of the 2022 ACM Conference on Fairness, Accountability, and TransparencyThe recent surge in publications related to explainable artificial intelligence (XAI) has led to an almost insurmountable wall if one wants to get started or stay up to date with XAI. For this reason, articles and reviews that present taxonomies of XAI ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
December 2024
179 pages
Copyright © 2024 Copyright held by the owner/author(s).
This work is licensed under a Creative Commons Attribution International 4.0 License.
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 24 November 2024
Online AM: 05 November 2024
Accepted: 21 October 2024
Revised: 20 October 2024
Received: 30 September 2024
Published in TAAS Volume 19, Issue 4
Check for updates
Author Tags
Qualifiers
- Research-article
Funding Sources
- National Natural Science Foundation of China
- Guangdong Provincial Key Laboratory
- Program for Guangdong Introducing Innovative and Entrepreneurial Teams
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 1,058Total Downloads
- Downloads (Last 12 months)1,058
- Downloads (Last 6 weeks)1,058
Reflects downloads up to 14 Dec 2024
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in