Open AccessReview

Explainable Machine Learning in Critical Decision Systems: Ensuring Safe Application and Correctness

Julius Wiggerthale

^*,†

and

Christoph Reich

^*,†

Faculty of Computer Science, Furtwangen University, Robert-Gerwig-Platz 1, 78120 Furtwangen, Germany

Authors to whom correspondence should be addressed.

^†

These authors contributed equally to this work.

AI 2024, 5(4), 2864-2896; https://doi.org/10.3390/ai5040138

Submission received: 4 November 2024 / Revised: 3 December 2024 / Accepted: 9 December 2024 / Published: 11 December 2024

Download

Browse Figures

Figure 1
Overview of paper’s structure with results obtained in steps and connections between steps. "> Figure 2
Development of dataset in different filtering steps: from originally 508 publications, 2 were dropped as duplicates and 422 were dropped because they don’t contribute to answering the RQ. "> Figure 3
Distribution of publication years in final selection of papers: most titles were selected from IEEE Digital Library (blue) and only few from ACM Digital Library (orange); 2023 is year of most frequent publications. "> Figure 4
Keyword cloud for keywords in selected publications; terms related to ML are dominant. "> Figure 5
Histogram of 15 most common keywords in literature selection; deep learning, explainable ai and machine learning are the most common kewords (25–36 counts). "> Figure 6
Classification of most common XAI methods in papers examined in taxonomy of XAI. "> Figure 7
Grouping of purposes pursued by XAI in critical decision systems: support users and model improvement as two big groups of XAI purposes. "> Figure 8
R4VR-framework for enhancing safety by XAI: Developer creates accountable ML model using XAI in the steps Reliability, ßtextitValidation and Verification; user applies model for safe decision in reverse steps Verification, Validation and Reliability. ">

Versions Notes

Abstract

Machine learning (ML) is increasingly used to support or automate decision processes in critical decision systems such as self driving cars or systems for medical diagnosis. These systems require decisions in which human lives are at stake and the decisions should therefore be well founded and very reliable. This need for reliability contrasts with the black-box nature of many ML models, making it difficult to ensure that they always behave as intended. In face of the high stakes involved, the resulting uncertainty is a significant challenge. Explainable artificial intelligence (XAI) addresses the issue by making black-box models more interpretable, often to increase user trust. However, many current XAI applications focus more on transparency and usability than on enhancing safety of ML applications. In this work, we therefore conduct a systematic literature review to examine how XAI can be leveraged to increase safety of ML applications in critical decision systems. We strive to find out for what purposes XAI is currently used in critical decision systems, what are the most common XAI techniques in critical decision systems and how XAI can be harnessed to increase safety of ML applications in critical decision systems. Using the SPAR-4-SLR protocol, we are able to answer these questions and provide a foundational resource for researchers and practitioners seeking to mitigate risks of ML applications. Essentially, we identify promising approaches of XAI which go beyond increasing trust to actively ensure correctness of decisions. Our findings propose a three-layered framework to enhance safety of ML in critical decision systems by means of XAI. The approach consists of Reliability, Validation and Verification. Furthermore, we point out gaps in research and propose future directions of XAI research for enhancing safety of ML applications in critical decision systems.

Keywords:

artificial intelligence; machine learning; explainable artificial intelligence; explainability; critical infrastructure; decision-making

1. Introduction

Machine learning (ML) has transformed our lives in recent years. Automated text translation, text generation and visual inspection are only some examples of tasks where ML is nowadays applied. Even in entities of critical infrastructure, ML models are nowadays widely adopted. Often, they serve to make or support certain decisions [1,2,3].

Critical infrastructures are vital for the functioning of our society [4]. Wrong decisions in any entity of critical infrastructure can have far-reaching consequences for the society as a whole, the economy or individual lives. Therefore, decisions in these entities are often related to ethical and legal issues. Take, for example, a self driving car that does not stop when a pedestrian passes the street. This example illustrates that no wrong decisions should occur.

Given these high stakes, integrating ML into critical infrastructure necessitates careful consideration. This is also recognized by the European Union (EU). In the EU, the EU AI Act came into force in August 2024. The act classifies all AI applications which deal with management and operation of critical infrastructure high risk applications. Such high risk applications have to be assessed before operation and also during their life cycle [5].

However, the far-reaching consequences of decisions are not limited to critical infrastructure. In other fields, such as social life, automated decisions can have big impact as well. For instance, in education, justice or asylum, ethical and legal questions are closely related to automated decision-making. Take, for example, the decision of which school a child is allowed to visit. A wrong decisions can affect the life of the child massively. Besides the immediate effect on a person, the decision can also have long term economic effects. For example, a child which had the potential to fund a successful company could end up receiving social welfare due to a wrong decision. Therefore, the EU AI Act classifies AI in use cases related to education, legal questions or asylum high risk applications as well. In the same way as applications in critical infrastructure, AI systems in these use cases have to be assessed before operation and during their life cycle [5]. All the above mentioned examples highlight that there are several systems where wrong decisions have unacceptable consequences which is why we consider them critical decision systems.

Adoption of ML based systems in critical decision systems entails different problems. One big challenge is the fact that ML-based systems often operate as black-boxes, making it difficult for humans to understand their decision-making process [6]. Consequently, it is challenging to determine how a model decides in each case. Thus, malfunctions cannot be easily detected or ruled out in the first place. Also, the models often lack user trust [7].

Explainable artificial intelligence (XAI) deals with how to make black-box models explainable and interpretable to humans. While XAI is nowadays extensively used to increase trust [8,9], improve ML models [10] and gain knowledge [11], there has been less focus on leveraging XAI to actively enhance safety of ML applications. Safety of ML applications in this context is about raising the certainty that a critical decision is correct. The ultimate goal therefore would be to ensure the correctness of models and thus rule out the possibility of it causing harm to people, society or the economy.

1.1. Aims and Contributions

Due to their black-box nature, application of many ML models for autonomous decision-making remains a risk. Also in decision support, they can cause harm if an user relies on models without deeply questioning their output. Existing literature often addresses the interpretability and transparency of models but falls short of systematically exploring how XAI techniques can be employed to enhance safety of ML applications or ultimately even guarantee the correctness of predictions. This gap is particularly concerning given the high stakes associated with incorrect decisions in critical decision systems. This systematic review paper aims to address the need for research on how to leverage XAI for enhancing safety of ML applications in critical decision systems. To the best of our knowledge, this is the first review to explicitly focus on that aspect of XAI.

In particular, the elaboration is driven by the following research questions (RQ):

RQ 1: For what purposes is XAI currently leveraged in critical decision systems?
RQ 2: What XAI techniques are commonly applied in critical decision systems?
RQ 3: How can XAI increase safety of ML applications in critical decision systems?

By answering the RQ, we aim to provide a foundational resource for researchers and practitioners seeking to mitigate risks associated with ML deployment in critical decision systems. In particular, the paper’s contributions are:

An overview of common XAI applications in critical decision systems
A clustering of XAI applications in critical decision systems
An outline of promising approaches to enhance safety of ML applications in critical decision systems
An identification of research gaps and relevant future directions of research for enhancing safety
A proposal of a conceptual three-layered framework to enhance safety of ML applications in critical decision systems

Regarding the proposed three layered framework reader should notice that the proposal of the framework is not the main aim of the elaboration. At its core, the paper is a systematic literature review striving to identify current directions of research and open challenges in the field of enhancing safety of ML applications in critical decision systems.

1.2. Methods and Structure

In order to answer the RQ, we follow a structured approach. First of all, we will provide a background on XAI in Section 2. This serves to create a common understanding of basic principles in XAI. Beyond that, Section 2 briefly presents some basic principles of critical infrastructures and defines what we consider critical decision systems in course of the elaboration. This is intended to make the field of critical decision systems more tangible.

Afterwards, we present related review papers in Section 3. Thereby, we attempt to point out gaps in research and the unique contributions of our elaboration.

Having laid the general foundation required to understand the elaboration, we points out how we systematically search, filter and organize literature using the SPAR-4-SLR protocol in Section 4. In course of the section, we define the search criteria and keywords and acquire literature based on that. We then filter the literature and examine the publications in more detail. In this course, we attempt to classify the publications regarding different categories. That way, we strive to find out what main streams are currently pursued by researchers. On the one hand, we can answer RQ 2 that way. Beyond that, we also find out what sectors of critical infrastructure are most frequently addressed and what kind of research is mainly conducted.

This in depth analysis of literature allows us to present a structured overview of common use cases of XAI in critical infrastructure in Section 5. Here, we try to identify how XAI is currently used in critical decision systems.

The overview of use cases serves as basis for Section 6. In the course of the Section, we first of all examine the main purpose of XAI in the previously described use cases thus providing an answer to RQ 1. Beyond that, we cluster the use cases. Thereby, we show that XAI is already an important resource for enhancing safety of ML in critical decision systems. Furthermore, the clustering helps us to identify three main goals that are pursued by XAI techniques for enhancing safety of ML applications. Based on these three aims-we refer to them as three layers-we then examine the use cases related to safety of ML in more detail and point out how each of the identified layers can be addressed by XAI. These insights provide an answer to RQ 3.

To make the answer to RQ 3 easier to grasp, we suggest a conceptual three-layered framework for enhancing safety of ML applications integrating different aspects from our findings in Section 6.4. The framework was not validated in real use cases yet. Nevertheless, we decide to present the framework in this early stage of research. On the one hand, the framework can help to facilitate transfer from the theoretical results described in our elaboration to practice. It provides a clear vision of how XAI can be harnessed to enhance safety of ML applications in critical decision systems and thus helps other researcher to define practical approaches. That way, the presentation of the framework can also be a basis for future practical research on the topic and encourage discourse on the question of how XAI can best be harnessed to enhance safety of ML applications in critical decision systems. That way, we hope to accelerate progress in the field of research.

We analyze and discuss our results in more detail in Section 7. On the one hand, we proide a brief summary of the results referring to the RQ again. On the other hand, we point out future directions of research. The paper closes with a conclusion in Section 8. The basic structure and workflow with the results obtained in single steps and the connections between different steps is also visualized in Figure 1.

2. Background

Before we deal with the topic of XAI in critical decision systems, we briefly introduce some facts on XAI as well as critical infrastructure and critical decision systems. This serves to highlight the relevance of correct decisions in critical decision systems. Furthermore the background will help to establish the link between critical infrastructure, critical decision systems and XAI.

2.1. Taxonomy of XAI

XAI is a field of research which deals with making AI systems explainable, meaning to enable humans to understand the reasons behind a prediction [12]. One aspect to consider when applying XAI techniques is the fact that there is often a trade-off between model performance and explainability. More explainable models are usually less complex. Consequently, they achieve less accurate predictive results [13].

There are different ways to categorize XAI methods. First of all, a distinction is made between intrinsic explainability and post-hoc explanations. Intrinsic explainability refers to the property of a ML model of being explainable. Examples of intrinsically explainable models are linear regression, logistic regression or decision trees. For these models, no XAI techniques are required since the prediction rules can be extracted from the model‘s structure or model‘s parameters. In contrast, post-hoc methods uncover the relationship between feature values and predictions [14].

Post-hoc methods can be divided in model-specific and model-agnostic methods. Model-specific methods are only applicable to a specific types of model. For example, class activation mapping (CAM) is mainly applicable for convolutional neural networks (CNNs) [15]. In contrast, model-agnostic methods such as Shapely additive explanations (SHAP) or local interpretable model-agnostic explanations (LIME) can be applied to any model [14]. Furthermore, a distinction is made between local and global explanations. Local explanations such as LIME only explain a certain prediction whereas global explanations such as provided by SHAP explain the overall model behavior [14]. More detailed information on single XAI techniques can be found in [14].

What is of relevance when it comes to selection of an appropriate XAI method is the kind of explanation the method provides. Common explanation types are [14]:

Feature summary statistics: These methods provide statistics for each feature. These statistics can be as simple as the feature importance or more complex, such as pairwise feature correlation strength. One example is SHAP. The method assumes the output of a model as sum of the contribution of all features. SHAP calculates a value which quantifies the contribution of each feature to the model behavior [2].
Visualization of feature summary: These methods provide a visualization of the features’ impacts on the prediction. One example are partial dependency plots (PDP) which show how the average predicted value changes when a feature is changed [16].
Model internals: These methods provide internal structures or rules of a model, for example, the structure of a decision tree or the weights in a linear regression [17].
Data points: These methods provide a data point which shall help to understand which features are important for a prediction. An example are counterfactual explanations where a close datapoint to the sample of interest is provided for which the predicted outcome is different (other class) [18].
Interpretable model: These methods try to train explainable models which approximate the behavior of the black-box model to be explained. The model is trained based on a dataset and the predictions, a black-box model makes on the data. For example, neural networks can be approximated using decision trees [18].

2.2. Critical Infrastructure

In this section, we provide an overview of critical infrastructures. The insights provided mainly refer to definitions and regulations within the EU but the general concepts are similar in other regions as well. The European Commission defines critical infrastructure as:

“Critical entities, as providers of essential services, play an indispensable role in the maintenance of vital societal functions or economic activities in the internal market in an increasingly interdependent Union economy.” [4]

For entities belonging to critical infrastructure, the Directive on the Resilience of Critical Entities accounts. The directive was released by the European Commission in 2023 and applies to 11 sectors [19], namely energy, transport, banking, financial market infrastructure, health, Drinking water, wastewater, digital infrastructure, public administration, space and production, processing and distribution of food.

The purpose of the directive is to strengthen the resilience of entities against external threats such as natural hazards or terrorism, but also against internal threats such as sabotage or health emergencies. The directive addresses member states, as well as entities of critical infrastructure. Entities of critical infrastructure can be any asset, system, organization or network involved in a sector of critical infrastructure. The Directive on the Resilience of Critical Entities entails the following rules [19]:

Every member state has to define a strategy to identify critical entities and carry out risk assessment regularly
Entities belonging to critical infrastructure have to do risk assessment as well and take measures to enhance resilience
If a critical entity provides service in 6 or more member states, it will receive support in risk assessment and taking measures by the European authorities
Member states have to provide support to entities of critical infrastructure. In turn, the commission will offer support to the member states

Regarding the use of AI in critical infrastructure, the EU AI Act applies. According to the act, all AI systems in operation and control of critical infrastructure have to be assessed by the developer before operation and during their life cycle. This assessment serves to demonstrate that the application conforms to the requirements of accountable AI [5].

2.3. Critical Decision Systems

According to the EU AI Act, there are also other applications considered as high risk applications. In particular, the EU AI Act classifies all products falling under the EU’s product safety legislation high risk applications. This includes cars, medical devices, lifts but also toys [19]. Despite that, the act defines sectors where AI applications are considered high risk applications. These sectors are [5]:

Critical infrastructure operation and management
Vocational training and education
Employment, employee management and entry into self-employment
Access to and utilization of basic private and public services and benefits
Execution of processes ensuing compliance with the law
Management of migration, asylum and border controls
Support in interpretation and application of the law

Obviously, there are domains besides those of critical infrastructure where wrong decisions can have far-reaching impact. Therefore, the question arises when a decision is to be seen as critical. The paper at hand cannot provide a final answer to this question. However, in the course of the paper, we consider critical decisions systems as systems where a wrong decision fulfills at least one of the following criteria:

Has the potential to kill a human immediately
Has the potential to save a human
Has an impact on the stability of national or international financial market
Has an impact on the security of supply with goods vital for life
Has the potential to affect the health of a human
Has the potential to affect a human’s opportunities for good and dignified life

This definition of critical decision systems is only to be seen as a framework for the paper at hand. It serves to define a platform for the literature review. Other delimitations are possible and can be useful in different use cases.

3. Related Work

After providing a background on XAI and critical decision systems, we also want to present an overview of related literature reviews. Thereby, we attempt to point out what researchers focused on so far, what research gaps exist and what unique contributions our literature review offers. To date, there is a wide range of literature reviews dealing with XAI. Many of these reviews also focus on specific use cases or domains related to critical decision systems. We present reviews from the four sectors of critical infrastructure that are most commonly represented in the literature (c.f. Section 4.3), namely health, digital infrastructure, energy and transportation. The extensive representation of these domains in existing literature underscores their importance and also provides a robust foundation for identifying effective XAI strategies that could be generalized to other critical sectors. Furthermore, these domains cover scenarios where failures can lead to societal, financial, or individual consequences making them representative for critical decision systems.

3.1. Reviews on XAI in Health Sector

The authors of [20] deal with XAI in the medical sector. They review different publications on XAI in general. Based on this, they transfer the insights to medical contexts. The authors try to identify promising solutions for medical applications and point out gaps as well as open challenges. The review shares some similarities with our review. It focuses on enhancing interpretability for medical applications—a sector of critical infrastructure (c.f. Section 2.2). It highlights the challenges of applying XAI techniques in critical settings like healthcare emphasizing the need for trustworthy and understandable ML models. In contrast to the work in [20], our review takes a broader approach. We analyze XAI across several critical decision systems. Also, our review emphasizes enhancing safety and correctness in these systems, rather than increasing interpretability or trust.

Another review which deals with XAI in medical contexts is presented by Farkhadov et al. in [21]. They try to reveal reasons for distrust in ML based systems. Based on that, they show the relevance of XAI in medical applications. The study provides insights into how trust can be regained through explanations and transparency in healthcare settings. The work by Farkhadov et al. is relevant to our study as it also discusses critical decision systems, with a primary focus on healthcare. Similar to our paper, the paper of Farkhadov et al. emphasizes the value of XAI in high-stakes situations. Unlike Farkhadov et al., who primarily address trust issues, our study extends beyond trust to focus on enhancing safety of ML applications. Furthermore, we take a more holistic approach on critical decision systems in general instead of focusing on the health sector.

3.2. Reviews on XAI in Digital Infrastructure

Besides health, digital infrastructure is frequently addressed by review papers. In [22], Jagatheesaperumal et al. deal with XAI in the internet of things (IoT). They point out that XAI can contribute to more user trust. They illustrate XAI services for IoT and present open questions for future work. Also, the authors address the challenges of explainability in digital infrastructure. For example, they highlight the security issues related to XAI in IoT applications. Digital infrastructure as a domain of critical infrastructure is also considered in our review. However, our review takes a broader focus on critical decision systems in general. Beyond that, we are not primarily concerned about user trust but rather about safety of ML applications which is less emphasized by Jagatheesaperumal et al.

In [23], Zhang et al. deal with XAI in cyber security. In particular, they address defense against cyber attacks, providing a roadmap for navigating XAI literature. The authors highlight that XAI is essential to increase trust in ML based systems. Trust, in turn, is important for several reasons, for example to allow more autonomy of ML based systems. This review is related to the document at hand as it highlights the role of XAI in a critical decision system. Both reviews underline the necessity of explainability for ensuring effective decision-making in critical environments. While the work in [23] focuses on XAI in cyber security, our work goes further by addressing decision systems in critical infrastructure in general. We strive to enhance safety of ML applications in such systems. Contrary, the focus of [23] is primarily on user trust.

3.3. Reviews on XAI in Energy Sector

Another sector that is addressed by reviews on XAI is energy sector. In [24], the authors deal with XAI in the energy sector. The authors review literature belonging to three different categories, namely XAI for critical power grid practices, XAI for forecasting renewable energy production and XAI for building energy management applications. They point out that XAI is gaining relevance in energy sector and propose different use cases based on their insights. For example, they suggest using XAI for enhancing trustworthiness of models for energy storage control making these models applicable in large scale applications. Similar to our work, the authors address a high risk domain. Also, they point out the potential of XAI for making ML models applicable in critical applications. However, in contrast to the work in [24] our work covers a variety of critical decision systems. Also, we aim to enhance safety of ML applications by XAI. This aspect is only briefly addressed by [24] who rather focus on the question of how to increase trust or justify decisions.

3.4. Reviews on XAI in Transport Sector

Also, the transportation sector is addressed by review papers. in [25], Kuznietsov et al. mainly address the topic of autonomous driving. They highlight the potential of XAI to enhance safety and trustworthiness of ML applications in autonomous driving. In particular, they describe five way in which XAI can support autonomous driving, namely interpretable design, interpretable surrogate models, interpretable monitoring, auxiliary explanations and interpretable validation. In a similar way as our work, Kuznietsov et al. pay high attention to the question of how to enhance safety of ML by means of XAI. However our work takes a broader focus by addressing critical decision systems in general. Also, we pursue a goal oriented approach to enhanicng safety of ML applications and focus on the question which general mechanisms can help to enhance safety. In contrast, the work of Kuznietsov et al. focuses on specific techniques and processes that enhance safety of ML applications in critical decision systems.

3.5. Summary of Related Work

In addition to the presented reviews, there is a wide variety of similar review papers dealing with XAI in different sectors of critical infrastructure or certain critical decision systems. Nevertheless, the analysis of related literature reveals significant gaps. The following aspects represent key research gaps:

Focus on trust rather than safety: Many existing works emphasize improving transparency and user trust in ML models. However, the question of how XAI techniques can be systematically leveraged to enhance the safety of ML applications is only a minor focus of researchers.
Lack of analysis across sectors: Many researchers already address individual domains of critical infrastructure or certain critical decision systems. A holistic analysis of XAI across different sectors of critical infrastructure is largely absent, even though these sectors share similar challenges regarding the safety and correctness of decisions.
Lack of methodological foundations: Detailed methodological approaches on how XAI can be used to enhance the safety of ML applications through feedback mechanisms or cross-sectoral standards are missing. This also includes integrating human feedback and adapting XAI techniques to different target audiences and contexts.

In order to address the research gaps, our review takes a holistic view of critical decision systems across multiple domains. Also, the review at hand focuses on the question of how to enhance safety of ML applications through XAI instead of increasing trust. Hence, the review provides a novel perspective on applying XAI in critical decision systems. It is thus a foundational resource for researchers and practitioners seeking to mitigate risks associated with ML deployment in critical decision systems

4. Methodology of Literature Review

With the aims stated in Section 3 in mind, we conduct the systematic literature review following the scientific procedures and rationales for systematic literature reviews (SPAR-4-SLR) protocol [26]. In this course, we conduct the stages assembling, assessing and arranging.

During our research we focus on two major databases, namely IEEE Digital Library and ACM Digital Library. The libraries are selected as they are leading in the field of informatics. Beyond that, they provide peer reviewed articles and cover relevant conferences in the field of AI and ML. In contrast, other databases such as Web of Science or Scopus cover a wider range of disciplines which might offer complementary insights. Preliminary exploration of Web of Science or Scopus indicated coverage of high-level theoretical and interdisciplinary studies. However, these were less aligned with the detailed technical focus required for this review. Nevertheless, it has to be noted that focus on two libraries can lead to neglecting relevant publications from other sources such as Web of Science or Scopus. This is a clear limitation of the elaboration that could be addressed in further research.

4.1. Assembling

The first stage of the SPAR-4-SLR protocol is assembling. This stage consists of two major steps: identification and acquisition [26]. In course of identification, the domain, source type, research questions and source quality criteria are defined. The domain of our research is XAI. The research questions were already defined in Section 1.1.

In the review process, we focus on literature published after 2019. This is due to the fact that the year 2019 marked a pivotal point in XAI research. Different vision papers and guidelines in related areas such as ethical or trustworthy AI were published in 2019. For example the European Commission published guidelines for trustworthy AI [27] and IEEE published the second version of a vision paper on Ethically Aligned Design [28]. These events lead to extended research in the field of XAI in the subsequent years. Furthermore, we limit our research to available literature written in English. The inclusion- and exclusion- criteria are documented in Table 1.

We are aware of the limitations that come with the exclusion criteria. Relevant studies published before 2019 or in other languages than English could be neglected. Furthermore, the focus on literature published in English can potentially lead to exclusion of relevant viewpoints and therefore introduce bias in our findings. These limitations should be addressed in future research.

In the next step, we define a search string based on our RQ. Since the review is focused on XAI, one of the topics of explanations or explainability and the component of AI or ML should be represented by the search string. Furthermore, we need keywords indicating that the publication deals with a critical use case or domain. However, the term critical is not commonly used in document titles. We therefore search for terms related to any kind of critical infrastructure, safety or security. It should be noted that we focus on instances of critical infrastructure in our search. This is due to the fact that this allows us to establish a clearly defined frame for the research. Without this limitation, the search could quickly lose its focus. We are aware of the fact that there are also other domains and fields requiring critical decisions. We leave it up to future research to include such domains in the review process.

Table 2 gives an overview of the terms we used in each category. In our search string, terms within one column are combined with or whereas terms from different columns are combined with and. The search strings we applied can be found attached (c.f. Appendix A and Appendix B). Some terms from our search string, such as IoT or CPS, may seem only marginally related to critical decision systems. However, we found that these are relevant systems for future applications in critical decision systems. Many researchers already deal with the question of how to make these systems applicable in critical decision systems and apply XAI to them.

Using these keywords, we conduct the acquisition-step. We conduct our search on 10 July 2024. During our search in the selected libraries, we find 508 publications, 432 in IEEE Digital Library and 76 in ACM Digital Library.

4.2. Arranging

The next stage of the SPAR-4-SLR-framework is arranging. This stage serves to reduce the amount of publications and consists of two steps, namely organization and purification [26]. First of all, we search for duplicates in our database. Here, we drop two titles. Afterwards, we filter the articles by quality criteria. This includes two main aspects. One aspect is the publication type. For our research, we decide to exclude review papers. The second aspect is the relevance for our research. To make an evaluation regarding this aspect, we read the abstract of each paper and try to answer the following questions:

Is XAI the primary topic of the paper?
Does the paper contribute to answering our RQ?

If one of the questions can’t be answered with yes based on the abstract, we exclude the paper. The latter question, serves to exclude papers that don’t focus on enhancing safety of model application but instead of other factors related to XAI such as trust.

Finally, we identify 84 publications which are potentially relevant for our review. Figure 2 shows how the number of publications developed with the different filtering steps.

From the selected publications, 10 are available in ACM Digital Library and the remainder of 74 publications is available in IEEE Digital Library. Most of the titles were published after 2022. Figure 3 shows how the publication years are distributed for publications from IEEE Digital Library (blue) and ACM Digital Library (orange).

In order to convey a basic idea of the prevalent topics in the literature selected, we create a keyword cloud showing the most common keywords in the selected publications (c.f. Figure 4). To provide a better feeling for the absolute number of counts for single key words, we also create a histogram showing the number of counts for the 15 most common key words (c.f. Figure 5). Figure 4 and Figure 5 show that terms related to ML, such as deep learning, artificial intelligence or explainable AI are prevalent. However, also the term security or LIME (the XAI method) are represented among the most common keywords.

4.3. Assessing

The final stage of the SPAR-4-SLR protocol is assessing. The stage consists of evaluation and reporting [26]. For evaluation, we focus on three aspects:

Type of paper: What type of research is conducted in the paper?
Sector of critical infrastructure: What sector of critical infrastructure (c.f. Section 2.2) is mainly addressed by the researchers?
XAI technique: What XAI techniques are applied in the paper?

For these three aspects, we try to define different categorize and assign each paper to a corresponding category.

4.3.1. Type of Paper

First of all, we try to identify what kind of study is conducted in each paper. Here, we define five categories:

Framework: Researchers propose a new theoretical or practical framework for solving current challenges in XAI. Usually, the framework incorporates a systematic structure and integrates different concepts and components.
Method: Researchers propose a new method or technique to solve specific issues in XAI. In contrast to frameworks, a method it is more technical and tailored to specific use cases.
Empirical study: Researchers collect data to examine certain aspects of topics related to XAI. One example can be a study which examines how users perceive explanations provided by a XAI method.
Use case: Researchers examine a certain use case in XAI. Usually, qualitative methods are applied.
Conceptual work: Researchers develop new concepts or theories. Usually, there is no link to practical use cases.

Table 3 gives an overview of how many publications we find in each category. The Table shows that most publications are either frameworks (31 papers) or methods (28 papers). There are also 15 empirical studies and 8 use cases but only 2 conceptual works. This indicates that researchers work rather practice oriented when it comes to investigating XAI in critical decision systems.

4.3.2. Sector of Critical Infrastructure

In a next step, we define which sector of critical infrastructure is addressed in each paper. Most papers deal with use cases in one certain sector or propose methods and frameworks that mainly address one certain sector. However, some publications also deal with critical infrastructures in general. We therefore define a class general for such publications. We find that only five sectors are represented in our selection, namely health, transport, energy, digital infrastructure, financial market infrastructure and drinking water. In our initial results, there were also use cases from other sectors.

Table 4 gives an overview of how many publications we find in each category. The Table shows that most publications refer to health sector (38 publications). Digital infrastructure and transport are ranked second respectively third (20 and 14 publications). For drinking water as well as financial market infrastructure, there is only one entry in our selection of publications each.

4.3.3. XAI Method

The third aspect for what we try to classify the publications is the XAI method applied in the paper. Thereby, we also provide an answer to RQ 2. Some publications apply several methods whereas others don’t actually use a XAI method but only focus on theoretical reasoning on the topic. We conduct a multi label classification including the most common XAI techniques. The following classes are defined:

Shapely additive explanations (SHAP): The method assumes the output of a model as sum of the contribution of all features. SHAP calculates a value which quantifies the contribution of each feature [14].
Local interpretable model-agnostic explanations (LIME): The method explains a prediction by building a local, explainable surrogate model based on the black-box model’s behavior [14].
Class activation mapping (CAM): The method calculates the gradients of the output with respect to the extracted features or the input via back-propagation. Based on this, attribution scores are estimated. CAM is mainly applicable for CNNs for image classification [15].
Layer wise relevance propagation (LRP): The method breaks down the prediction of a neural network to relevance scores for the single input dimensions of a instance [108].
Decision trees: Decision trees are interpretable models. From the tree structure of the model, decision rules can be extracted [14].
Explain like I am 5 (ELI5): ELI5 is not a single XAI technique but a model-agnostic framework that provides different XAI techniques for generating global and local explanations [109].
Custom: This category includes all papers which propose new XAI techniques.
Others: This category includes all papers which use other methods only represented seldom in the selected publications. Examples are fuzzy logic and integrated gradients (IG).
None: This category is for papers that don’t apply XAI but only theoreticall reason about the topic.

Regarding the class CAM, readers should notice that we also include related methods such as GradCAM and GradCAM++ into this category. Table 5 gives an overview of how many publications we find in each category. It can be seen that SHAP is the most commonly used XAI technique with 32 records. LIME and CAM are ranked second, respectively, third (18 and 8 records).

The most common approaches are also visualized in a taxonomy of XAI methods (c.f. Figure 6). Decision trees are the only inherently explainable models. The other methods provide post-hoc explanations. SHAP, LIME and ELI5 are model-agnostic methods, meaning they can be applied to any model. In contrast, the model-specfic methods LRP and CAM are only applicable to neural networks. More details on the single XAI techniques can be found in [14].

5. Use Cases of XAI in Critical Decision Systems

The above presented overview of paper classifications shows that there is a wide variety of papers in the publications we selected for review. Different research types, domains and XAI techniques are covered by the selection. We now examine the literature in more detail and point out certain use cases of XAI. As mentioned in Section 4.3, the selected papers mainly deal with applications related to health, energy, digital infrastructure and transport. The extensive representation of these domains in existing literature not only underscores their importance but also provides a robust foundation for identifying effective XAI strategies that could be generalized to other critical sectors. We therefore provide an overview of XAI use cases divided by these sectors. Nevertheless, we are aware of the limitations that come with this focus. These limitations could be addressed by future research.

5.1. XAI Use Cases in Health

First of all, we turn towards the health sector. In our paper selection, this is the most common sector. Health is a sector of critical infrastructure (c.f. Section 2.2) and decisions made have massive impact on individuals. For example, wrong diagnosis of serious disease, such as cancer, can significantly reduce the life expectancy of a human due to delayed treatment of the disease [85]. Beyond that, decisions in health emergencies such as the Corona pandemic can have broad social or economic consequences [110]. These examples highlight the critical character of decisions in health sector.

A wide variety of use cases is proposed in health sector. However, the predominant use case is medical diagnosis based on ML. The basic idea here is to train ML models which can detect disease and thus allow physicians for faster or more reliable diagnosis. Duamwan et al., for example, develop a CNN for diagnosing Alzheimer’s disease based on magnetic resonance images. They apply LIME to their model to identify relevant regions in the input images and implement Felzenszwalb’s segmentation algorithm to automatically visualize relevant image segments [65].

In addition to XAI-based diagnostic tools, more and more researchers are also dealing with real time analysis of streamed medical data, for example from smart devices and IoT systems that allow the early identification of changes in health and potential risks. Wu et al. try to predict health changes of an individual based on various sensor data including activity, bed time and motion scheme data. They apply a Sequential Possibilistic Gaussian Mixture Model and explore different XAI techniques, including linguistic explanations based on fuzzy logic and SHAP [84]. The work in [29] has a similar direction. The authors deal with the question of how to integrate XAI in medical IoT networks to make transparent and reliable medical diagnosis.

As such IoT networks become more relevant in medical applications, the question of security arises. Seetharaman et al. explore how to harness XAI for securing medical IoT networks. They propose a framework utilizing autoencoders, contrastive learning and distance based explanations to detect drifting samples in a dataset and securing medical IoT networks that way [106]. The work in [39] deals with identifying malicious traffic in medical IoT systems by ML. They explain the predictions using the methods LIME, SHAP and ELI5 and try to make them accessible to experts as well as non-experts by using a Shapas Monitor interface.

Many use cases in the health sector are based on supervised learning. Shen et al., however, propose an unsupervised approach to detection of fundus disease [35]. The most commonly used input type is image-data and CNNs are the main ML models applied [29,32,37,56]. However, also time series data such as EEG data [111] or even tabular data [94] are sometimes used for training respective models. Besides CNNs, Decision Trees [61] and XGBoost [103] are common ML models. Frequent XAI methods for health use cases include LIME [37,65], SHAP [82] and GradCAM [32,56]. Some researchers also compare different XAI methods in health. For example, ref. [87] finds that SHAP provides better generalizability, stability, and concordance compared to LIME in two case studies. Conversely, ref. [104] states that LIME and GradCAM provide more reliable results.

5.2. XAI Use Cases in Digital Infrastructure

Digital infrastructure is the sector that is second in terms of the number of publications in our selection of papers. A functioning digital infrastructure is vital for our everyday life, economy and security. As pointed out in Section 2.2, digital infrastructure is a sector of critical infrastructure. The work in [22] illustrates the role of IoT in smart cities for disaster management. Obviously, decisions made in disaster management have a big impact on human lives and the society. This example illustrates that decision systems in digital infrastructure can be highly critical.

Many of the papers belonging to this category deal with how to ensure security in digital systems such as CPS or IoT. Krishnaveni et al., for example, present NexGuard, a framework which utilizes Bayesian optimization to improve gated recurrent unit long short term memory (GRU-LSTM) for intrusuion detection in CPS. In addition, the framework incorporates SHAP to highlight the most important features that contribute to a prediction and thus enables better human-machine cooperation [78].

An approach to network protection that goes beyond the mere provision of explanations is proposed in [44]. The authors suggest a framework utilizing LIME and SHAP for providing explanations and CF-SHAP for providing insights into actionable countermeasures for network protection. They show that their framework can efficiently help operators identify measures for network protection.

Another important XAI use case in digital infrastructure is resource allocation. Rezazadeh et al. propose a framework based on deep reinforcement learning to allocate resources in 6G networks under strict level agreements. The authors harness explanation guided learning and show that their approach is superior to previous approaches [73].

In contrast to use cases in health, unsupervised learning is more prevalent in use cases of digital infrastructure [36,75]. Important types of models when it comes to network security are CNNs [55,75], LSTMs [75,78] or decision trees [77]. Especially complex CNNs or LSTMs achieve high performance when analyzing time series data for anomalies [55,75,78]. Besides time series data, tabular data is sometimes used [36,75]. Commonly used XAI methods are SHAP [44,78,93], LIME [44,93], ELI5 [93] and LRP [41].

Kalakoti et al. examine the performance of different XAI methods in botnet detection. They point out that DeepLift performs better in terms of faithfulness, consistency and complexity compared to methods such as LIME, IG, SHAP or saliencies [112].

5.3. XAI Use Cases in Transport

Next, we will turn to the transport sector which is represented in our selection with 14 publications. Transportation is a sector of critical infrastructure (c.f. Section 2.2). Functioning transportation chains are essential for the economy but also for the supply with goods vital for live [4]. Beyond that, autonomous driving is gaining entrance in the filed of transportation. Autonomous driving systems can potentially harm people with wrong decisions [80] and are therefore critical decision systems as well.

Autonomous driving is also one of the most relevant topics addressed by literature from our selection. For example, Das et al. want to receive deeper understanding of autonomous vehicles’ behavior in mixed traffic. To that end, they investigate which factors affect acceleration behavior of autonomous vehicles using regularized stacked LSTMs and the XAI methods SHAP and PDP [16]. In [68], the authors tackle the problem of path planning in autonomous vehicle networks comprising several participants. They propose a framework based on federated learning in which several autonomous vehicles help a new vehicle planning the best trajectory. To identify trustworthy vehicles within the network, the authors propose leveraging XAI. They show that their framework achieves superior performance compared to state of the art methods. Further use cases in autonomous driving include filtering of information in a network of autonomous vehicles to enhance privacy, reduce complexity and optimize computational performance [8] or harnessing XAI to increase safety of pedestrian detection [80].

Another use case in transport is remote steering of unmanned vehicles. Friedrich et al. address the question of how to support pilots of partially autonomous unmanned aerial vehicles. They consider a use case where ML models screen the environment for risk and the pilots only have to take action when the system detects a risk it cannot handle by itself. They propose a human machine interface (HMI) integrating explanations provided by CAM-methods and LRP thus enabling pilots to better assess the situation and make well-founded decisions [42].

Reinforcement learning is an important paradigm in use cases related to transport [68]. Important ML models are LSTMs [16] and CNNs [80]. Commonly applied XAI methods include SHAP [8,16,46,92], LIME [46], PDP [16], CAM [42] and LRP [42].

5.4. XAI Use Cases in Energy

Energy is another sector frequently represented in our selection of publications. In total, there are 9 records related to that sector. Energy is a sector of critical infrastructure (c.f. Section 2.2). Wrong decisions in energy systems can, for example, lead to black out [69] which comes with devastating consequences for our every day live, economy and also medical institutions. Decision systems in the energy sector are therefore highly critical.

One use case of XAI in energy is stability assessment. The authors of [52] propose application of a Bayesian neural network for transient stability analysis and develop Grad SHAP to provide global and local explanations. They show that their proposed method outperforms prior approaches in terms of accuracy and uncertainty quantification. Another approach to this use case is presented by Ren et al. who harness tree based regularization to achieve explainability in transient stability assessment [57].

The work in [98] goes beyond assessment of stability and harnesses XAI to allocate power in networks. The authors find that the proposed model is able to precisely allocate power but their explainability method based on Meijer G-function cannot ensure to find the most representative function for the network.

Another use case in energy is protection of networks and facilities. Paul et al. use XGBoost in combination with LIME to create explainable models for detection of cyber attacks on photovoltaic system [9]. The authors of [69] deal with how to ensure security of voltage stability in power system. They use residual neural networks for predicting voltage stability. From these networks, they create globally explainable security rules by pruning and piecewise linear approximation. These rules are used to create hyperplanes of security boundaries.

Important ML models for XAI use cases in energy are LSTMs [58] and Bayesian neural networks [52]. XAI methods in energy use cases include SHAP [58,99] and LIME [9].

6. Can XAI Be Harnessed to Enhance Safety?

We now turn to the main question motivating the paper: For what purposes is XAI currently leveraged in critical decision systems? (RQ 1) and: How can XAI increase safety of ML applications in critical decision systems? (RQ 3)

6.1. Purposes of XAI in Examined Use Cases

In order to answer the former question, we examine what is the intention of authors in the publications examined in Section 5 and in what way this intention is in alignment with enhancing safety of ML applications.

6.1.1. XAI for Gaining Trust

One major reason for applying XAI techniques in critical decision systems is increasing trust. A majority of authors over all use cases addresses this aspect at least partially [9,29,61,87]. In many cases, user trust is not really relevant for safety. However, some applications such as network threat protection or remote steering of autonomous vehicles may require quick interaction of users. If the users hesitate because they don’t trust the model or want to confirm the model’s prediction on their own, valuable time is lost. Thus, user trust is essential in such use cases in order to ensure safe reaction.

6.1.2. XAI for Model Improvement

Many researchers also examine how XAI can be leveraged to improve models, for example by detecting bias in a model [31,55,96]. Obviously, model improvement is closely related to safety. Only a high performance model can be applied to critical decision systems safely, as every wrong prediction is a potential source of fatal system failure.

6.1.3. XAI for Gaining Insights

Another relevant purpose of XAI in the use cases examined is to gain knowledge. Many authors harness ML to train models applicable to problems that are not understood by research or are only partially understood. Applying XAI techniques to such models can help to get a deeper understanding of the underlying rules [16,58]. This aspect has no clear relation to safety of ML applications in critical decision systems. However, the extracted knowledge can enable human experts to make better decisions in critical situations. For example, a model trained to predict Alzheimer’s disease from images could use markers that are currently unknown to physicians. When XAI techniques can reveal these markers, physicians could potentially detect the disease more precisely or earlier, thus making more informed decisions.

6.1.4. Clustering Use Cases

When considering the use cases mentioned above and the main purpose they pursue, we find two major fields: Supporting users and enhancing models. We therefore try to identify certain purposes of XAI that fit in these groups. The results are shown in Figure 7.

The clustering of use cases clearly shows that there are two main target groups of explanations. While explanations used for model improvement mainly address developers, explanations which serve to support users have to address the end user of a ML model. End users can, for example, be physicians, passengers of autonomous vehicles, operators of unmanned vehicles or operators in systems for network protection. Different authors point out the challenges arising in the latter case. On the one hand, the explanations provided by many common XAI techniques are only hard to understand by most end users [86]. On the other hand, many common XAI techniques require high computing resources and are only hardly scalable to real world applications [75].

6.2. Enhancing Safety

We now turn towards publications dealing with the question of how XAI can be harnessed to enhance safety of ML applications, thus answering RQ 3.

6.2.1. Reliability of Decisions

We first examine how researchers leverage XAI to increase reliability of models. There are several publications dealing with the question of how to make ML models reliable by harnessing XAI techniques. One obvious method is using XAI for model improvement. As pointed out in Section 6.1, several authors deal with this question. Some authors try to reduce bias [31,55,96], others see the potential to enhance overall predictive performance [18,70] and still others try to increase robustness [18,82].

Also the work in [18] addresses the question of reliability in ML models. The authors examine the question how explainability affects safety of ML in healthcare. They present a conceptual model showing that safety is the product of several influences. In particular, they mention robustness, human machine interface, data management, performance, other safety specific controls and explainability as the six dimensions affecting safety of ML. Beyond this concept, they point out how explainability can support safety in every phase of the ML lifecycle. They state that during data management-although XAI is not applicable here-prototypes can already help to better understand and process data. During model selection, they propose to include explainability as one factor for assessment of models. In model training, they highlight the potential of XAI in model debugging and for enhancing robustness. During model evaluation, they suggest to check if model explanations are conform with domain knowledge. Furthermore, they recommend to use counterfactual explanations in order to assess robustness of a model. During model application, XAI mainly serves for decision support in their concept.

6.2.2. Validation

A step beyond enhancing reliability by XAI is validating a model’s functionality. Of course, validation of models is a crucial step when it comes to transfer of models to practice. It is common practice to validate that a model is not overift. Obviously, this is essential for allowing safe application of the ML model. Some researchers, however, take the concept of validating models a step further.

Liu and Müller examine lane keeping in autonomous driving and propose a three-step approach to evaluate and validate the predictions of a deep neural network (DNN). First of all, they calculate a saliency map using LRP. The saliency map represents the most important visual features. Afterwards, they assess if the features the model relies on are reliable using pixel flipping. Finally, they use a physical model of the vehicle in order to calculate the trajectory in the near feature. The trajectory can be displayed and validated by humans. Liu and Müller state that a plausible trajectory corresponds to good model decisions [60].

6.2.3. Verification

Finally, we turn towards the last method to enhance safety of ML applications we find during the review. Verifying a model’s functioning, thus ensuring its correct behavior. For the theoretical case that we can verify a model for all possible inputs, we can ensure correctness and thus apply the model with more certainty. However, this is obviously not possible. Nevertheless, we find some approaches which strive to verify models in certain boundaries.

In [2], the authors use formal methods to ensure the correctness of cancer classification made by a black-box model. In particular, they transform the decision logic of the black-box model into Coloured Petri-Nets (CP-Nets), allowing formal verification of the correctness of each decision.

To do so, they pursue a three step approach. First of all, a decision tree is derived from a black-box model using a synthetic dataset. Afterwards, to model the decision tree, CP-Nets are created. In the CP-Net, each state (class of cancer, malign or benign) is represented by a token. Finally, the decision-making process is evaluated using state space analysis. This state space analysis allows to verify the correctness of the decision by checking if a token reaches the correct place in the CP-Net. If this is the case, prediction and actual label are identical and the decision is considered correct.

Sherry et al. consider formal methods as approach to designing a flight guidance and control system based on XAI. They propose to leverage XAI to build a formal situation-goal-behavior model from the black-box model. They state that these models can be evaluated by model analysis and simulation [96].

6.2.4. General Perspectives on XAI for Enhancing Safety

As we showed, there are different layers and components of ensuring safety. Definitely verifying a model could potentially be sufficient to apply ML models safely in critical decision systems. However, since it is currently not possible to definitively verify a model, different components, functionalities and layers have to come together to ensure safe ML application in critical decision systems. This is also recognized by some authors.

An overreaching approach to enhancing safety of ML applications in critical decision systems is provided in [105]. In course of Horizon Europe, the authors present the project SAFEXPLAIN. The project strives to enable certification of autonomous AI systems in critical industries including automotive, space and railway. In the publication, they present a concept which shall help them to make steps towards certifying deep learning models in critical industries. They pursue a comprehensive approach comprising different strains of action. They put functional safety requirements in the center of their approach. In particular they plan considering how training-data determines the model as first measure. Beyond that, they strive to develop new techniques enhancing safety, traceability, and robustness of models. Furthermore, they want to create new methods which are based on functional safety requirements. Finally, they plan to develop platforms enhancing predictability of deep learning solutions.

To accomplish these objectives, they present a methodology based on 4 pillars. First of all, they will design safety certifications that take into account the nature of deep learning models. Secondly, they strive to design new deep learning solutions that are aware of functional safety. Also, they plan to adapt deep learning software to functional safety requirements. Furthermore, they want to create platforms that allow enhanced predictability and analyzability. Finally, they will test their solutions in industrial use cases allowing them to assess and refine them.

Thus, in summary their approach comprises different means to enhance reliability of models. The models created that way are then validated in real use cases to create realistic operation conditions.

6.3. Beyond Enhancing Safety of ML Applications

Obviously, the currently available methods are limited when it comes to enhancing safety of ML applications. It will be possible to create very reliable models, validate them under very realistic conditions and maybe even verify them in certain boundaries. However, a little risk will always remain. We therefore present some thoughts on what is necessary to apply ML in critical decision systems anyway.

6.3.1. Uncertainty Quantification

One important aspect is keeping a human in the loop of the decision cycle. For many use cases, such as medical decisions, this will probably remain the standard in near future. In such cases, ML models should serve as a basis for decision-making. In order to ensure that human decision makers don’t rely blindly on a prediction, uncertainty quantification is an important measure. The work in [80] tries to address uncertainty quantification in pedestrian recognition in autonomous driving. The authors present a methodology based on distance metrics that allows to quantify how certain a model is. This approach is based on the requirements of DO-178C-a norm for software development-which requires that high level requirements are broken down into low level requirements. To comply with this demand, they break down the DNN into layers and their learnable parameters. This allows for identification of individual components with individual requirements. The overall DNN has to meet a high level requirement and the components have to meet low level requirements. In particular, they want to achieve a structured representation of the latent space (high level requirement) which is then classified. For structuring the latent space, they use prototypes. This approach allows to apply distance metrics and thus quantify how close the latent representation of a given input is to the latent representation of a pedestrian.

6.3.2. Regulation and Certification of ML

Another important measure beyond ensuring safety is regulating application of ML in critical decision systems. In face of the fact that no absolute guarantees can be provided when applying ML models, it is of utmost importance that clear responsibilities are defined by law. Every stakeholder in ML applications has to be aware of his rights and obligations.

There are already different general regulations all over the world such as the EU AI Act (c.f. Section 2). Beyond that, domain specific requirements have to be met in many use cases in order to achieve certification of a product. In health, for example, regulation 2017/745/EU Annex II(4.) and 2017/746/EU Annex II(4.) prescribe that manufacturers of medical devices have to comply with general safety and performance requirements [113].

Also, several authors deal with the question of how to certify a ML system. Sutthithatip et al. deal with XAI in aerospace. They define three roles, namely the system creators, guarantors and interpreters. They suppose that guarantors should be third party individuals or organizations that are responsible for certifying a ML system [54].

The already mentioned project SAFEXPLAIN is focused on how to achieve ML certification in critical industrial applications as well. The authors propose the already described approach (c.f. Section 6.2) [105].

6.4. The R4VR-Framework to Enhance Safety of ML Applications

The presented results show that there are basically three general mechanisms to increase safety of ML models by XAI. We therefore propose a conceptual three-layered framework for enhancing safety of ML models in critical decision systems by means of XAI.

Currently, it is only a conceptual framework that was not validated in certain use cases. Nevertheless, we decide to share the framework at the early stage of research. First of all, we believe that the description of explicit steps that can be token to enhance safety of ML applications supports researchers who try to transfer our results to practice. Furthermore, the framework can be the basis for further research on the question of how XAI can be harnessed to enhance safety of ML applications in critical decision systems. Both of the mentioned aspects can lead to increased discourse in the field of XAI for enhancing safety and thereby accelerate progress of research.

The framework addresses developers as well as end users of ML models. Beyond that, it can support companies meeting the requirements of the EU AI Act. In its first phase, it shall support developers to create highly accountable ML models by application of XAI. Once the model is developed, the framework enters its second phase in which it shall help end users to make safe decisions using the ML model and XAI. In course of the framework, the developer undergoes the layers Reliability, Validation and Verification. The end user undergoes these layers reversely: Verification, Validation and Reliability. We therefore call the framework R4VR-framework.

In course of the framework, the developer has to increase Reliability of the model firstly. Reliability refers to the property of performing consistently well. Obviously, a ML model which fulfills this requirement without any limitations can be applied safely in any application. As the literature review showed, XAI can help to increase reliability of ML models by increasing overall performance, reducing bias or enhancing robustness.

However, ML models will never achieve perfect reliability. In real world application, situations will usually occur one day that were not represented in the training-data and where the model cannot reliably predict the correct answer. Therefore, Validation is the second layer in our framework. Validation of ML models is regularly done using test-data that was not used during training. However, for actually increasing safety of ML applications, we propose to conduct validation in real world scenarios with a human in the loop. To do so, the ML model should be applied in real world scenarios and the predictions should be explained by appropriate XAI techniques. Furthermore, the human in the loop should be informed when the model’s prediction is uncertain. For example, a confidence score the model automatically generates for each prediction could be used for that purpose. When the model makes wrong decisions, the user is able to intervene. That way, the model immediately receives a new datapoint which includes the correct behavior for the situation it handled incorrectly. Beyond that, the human will also provide an explanation of why he acted the way he did. This will provide the model with additional information supporting the process of re-training.

Based on the validation, we propose Verification of the ML model by means of XAI-the last layer of our proposed framework. Definite verification of ML models is-to the best of our knowledge-not possible. However, validation in real world scenarios reveals important insights in situations where the model may be unreliable. Retraining based on the new data generated using our validation approach hopefully enhanced the model’s performance in these situations. For every situation, where the model was previously uncertain or predicted wrongly, methods for verification should be applied. For example, local surrogate models could be created and examined using formal methods. That way, developers could be enabled to specify a permissible range where safe application of the model is possible.

The readily developed ML model as well as the permissible range and the XAI techniques are then handed over to the end user. As the end user applies the ML model, our framework enters the second phase. In this phase, the end user applies the ML model and XAI techniques in decision systems. Given the model, the XAI technique and the permissible range, he has to get to a decision. In this process, we propose to conduct the process of Reliability, Validation and Verification reversely. Given the decision situation, the end user first has to verify if his situation falls into the approved range of the ML model.

If this is the case, he can apply the model and the XAI technique. His task is then to validate the prediction based on the XAI output. Finally, he has to ensure the prediction is reliable. To do so, he should compare the prediction with his experience. Based on these three layers, the user can make the final decision.

The basic workflow of the R4VR-framework as described above is shown in Figure 8. The framework can be a powerful tool for harnessing XAI in critical decision systems. It entails two steps of model improvement. That way, highly reliable models can be created. Beyond that, the R4VR-framework provides the end user with valuable guidelines for getting to a decision. Following these guidelines ensures that the model is approved for the decision situation and that the model’s prediction is questioned by the end user.

How the R4VR-Framework Compares to the EU AI Act

To illustrate how a company can apply the R4VR-framework to comply with the EU AI Act, we consider an example process:

First of all, the company develops a ML model for a critical decision systems. Here, the framework emphasizes creating highly accurate and robust models, aligning with the EU AI Act’s requirement for technical robustness and accuracy. Developers iteratively refine the model, using a combination of real world data and synthetic scenarios to ensure high performance across diverse contexts.

In the next step, the company employs the framework’s iterative testing process. This process involves deploying the model in real world environments, collecting feedback, and iterating on the model to improve its performance. This approach ensures the model is validated under conditions that simulate its actual use. Thereby, the EU AI Act’s requirement for a quality management system can be met.

Also the demand for risk management throughout the ML model’s lifecycle can be met by the R4VR-framework. The company defines a permissible range of use cases, ensuring the model operates safely within specific parameters. Additionally, XAI is used both for model improvement and to support decision-making during deployment, thereby mitigating risks associated with inappropriate or unintended use. Usage of XAI also ensures human oversight, a key requirement of the EU AI Act. Explanations enable the end user to interpret the model’s outputs. For instance, a doctor using the diagnostic model can view explanations of its predictions to validate their plausibility, ensuring informed decision-making.

Beyond that, the company is enabled to technically document the model development process in a structured way following the framework’s structure. The documentation is complemented by clear user instructions, enabling end users to understand the model’s capabilities and limitations. For example, a permissible range might highlight that the model is suitable for diagnosing specific medical conditions.

One area where the framework requires additional measures is data governance, particularly when retraining a model using data generated through a human-in-the-loop process. To address this, the company implements specific protocols for data management, ensuring compliance with data governance requirements in the EU AI Act. These protocols might include robust data anonymization, traceability, and validation.

Hence, by following the R4VR-framework, the company demonstrates a clear path to meet the EU AI Act’s requirements. The framework acts as a practical guide for building safe and compliant ML systems. We will therefore deal with implementation of the framework in real use cases in future research in order to find out how it works in practice.

7. Analysis and Discussion

In this section, we would like to briefly analyze and discuss the main findings of the review.

7.1. Summary of Results

The systematic literature review at hand was motivated by three main RQ to explore the role of XAI in critical decision systems. Our findings reveal that XAI serves two primary purposes: supporting users through trust-building and decision guidance, and enhancing model safety by mitigating bias, improving robustness, and increasing accuracy. The most frequently applied XAI techniques include the model-agnostic methods SHAP and LIME, alongside CAM-based methods tailored to neural networks. These techniques are predominantly employed in critical sectors such as health, digital infrastructure, transport, and energy.

In terms of increasing safety, we identified three pivotal layers addressed by XAI: enhancing model reliability, enabling better validation, and facilitating partial verification. Based on these insights, we propose a conceptual framework comprising reliability, validation and verification as steps towards safer ML applications. This framework provides actionable guidance for developers and underscores the potential of XAI not just to explain decisions but to fundamentally enhance safety in critical decision systems.

Summing up, this review highlights the potential of XAI in enhancing safety of ML applications in critical decision systems. The proposed framework represents a step forward in aligning technical development with practical needs, fostering safer and more trustworthy ML systems. However, the framework requires validation in real world scenarios to ensure its practical applicability.

7.2. Gaps in Research and Open Challenges

Building on the findings, several gaps in current research remain offering potential for future research.

7.2.1. Efficient Verification of Models for Certain Ranges

First of all, we would like to highlight that there is only a minor focus on how to define a permissible range for a model. Our findings suggest that verifying a model’s predictions for a certain range can help developers and companies to achieve this. This can be realized by designing efficient methods which consider a certain range of input parameters and systematically check a model’s behavior in this range.

7.2.2. Combination of Models

Another important direction that future research should consider is the use of multiple models which are sufficiently different. This would allow to compare the predictions of the models. However, even more certainty can be gained by comparing the explanations for the predictions. If these differ, the models’ decision should at least be questioned and checked manually. In this way, safety mechanisms could be created that inform people when uncertainties arise. This can enable a high degree of autonomy for models and increase safety at the same time.

7.2.3. Human Feedback

Human feedback is already an important source for model improvement [114]. However, the potential of XAI is currently neglected. Research should focus on how XAI can be applied to make such systems even more efficient. An approach would be conceivable in which a person cannot only provide feedback on incorrect predictions but can also simultaneously respond to the model’s explanation and pass on their explanation for a different decision to the model. In this way, the model could be provided with additional information and thus learn even more efficiently.

7.2.4. Providing Explanations to Different Users

Another topic that has to be addressed is compression of explanations and the question how to provide them to different users efficiently. It is of utmost importance that the end user can grasp a model’s explanation quickly. The time horizon of the explanation and the user’s level of knowledge can vary greatly from case to case. Therefore, static XAI methods are often not suitable. Consequently, it is relevant to deal with the question of how explanations can be adapted to the background of the targeted audience and the available time and computing resources without losing their meaningfulness.

7.2.5. Trade-Off Between Accuracy and Explainability

Another issue that should be addressed by future research is the trade-off between model performance and explainability. While explainability is desirable for several reasons and can contribute to safe application of models, it is usually accompanied by loss in performance [13]. This is problematic as it can lead to models making more wrong predictions. This in turn can lead to wrong decisions in critical decision systems even if they provide convincing explanations.

7.2.6. Scalability

Also scalability of XAI techniques is an important challenge. As different authors point out, many XAI methods are only hardly applicable in larger scales. Thus, checking every single prediction is not possible in many critical decision systems such as IDS or energy net analysis [66,75]. However, this is necessary in critical applications, as it is imperative to detect every potentially misleading prediction. Future research should therefore increasingly focus on how to make common XAI methods suitable for large scale applications.

7.2.7. Providing Explanations to Different Users

Another challenge is related to the presentation of explanations. In critical decision systems, quick decisions are often required [97]. This can only be realized when the user of an XAI system is able to immediately see and understand why a ML model made a prediction and can assess if the prediction is correct that way. This enables a user to identify if action is required. However, many XAI techniques are currently hard to understand for users not familiar with ML [86]. Therefore, research is required on how to efficiently convert the explanations provided by XAI techniques to explanations that can be understood by end users of a ML model.

7.2.8. Regulations and Certification Standards

Finally, we want to highlight the importance of regulatory measures when it comes to application of ML in critical decision systems. There will always remain uncertainty when applying ML models. In critical decision systems where high risks are at stake, the effects are potentially devastating. Authorities must therefore precisely define the responsibilities, rights and obligations of various stakeholders. This could not only lead to more certainty in case of failure but could also motivate researchers, developers and users to check their work more carefully. This in turn would lead to better models making less mistakes. When defining such laws, it should be regarded that they don’t suppress progress in the field of ML. Authorities therefore have to find a good balance between freedom and regulation. With the EU AI Act, the EU made a first step in this direction. However, there remain many open questions.

For example, regulation should also include harmonized certification standards. It is of utmost importance that ML systems are certifiable by a standardized process. This is the only way for efficiently developing, testing and introducing ML applications in big scale. Developers need clear guidelines and we even advocate for an independent instance responsible for certifying ML based systems. XAI should play a significant role for certification as it is valuable to provide insights in models that can impact the certification.

8. Conclusions

The literature review explored the applicability of XAI to improve safety in critical decision systems. Our review distinguishes itself from previous work by explicitly focusing on enhancing the safety of ML applications rather than just increasing transparency or user trust. This focus on creating and applying accountable ML models in critical decision systems sets our work apart from other reviews that primarily concentrate on enhancing interpretability or improving model performance without considering the full implications for safety. Through our review, we highlight practical ways in which XAI can improve the safety of ML applications in critical decision systems.

We rigorously collected, filtered, organized and summarized literature from IEEE Digital Library and ACM Digital Library. Analysis of the literature highlighted that healthcare, transportation, energy and digital infrastructure are frequently represented sectors in current research. Specific use cases of XAI include systems for medical diagnosis, autonomous driving, prediction of energy net stability and IDS. SHAP, LIME and GradCAM are predominant methods for explaining models’ decisions. In healthcare, XAI methods are mainly used to explain CNNs for supervised tasks. In contrast, unsupervised learning and LSTMs play a big role in digital infrastructure.

Also, our research revealed that XAI is used to enhance user experience (autonomous driving [60]), increase trust (ML based medical diagnosis [85,87]), enable making faster decisions (IDS and energy net stability [52,77,99]) and to improve models [31,55]. Although model improvement and faster decision-making in critical situations are related to the safety of ML applications, this is usually not the focus of researchers. Nevertheless, we identified use cases explicitly addressing the question of how to enhance safety of ML applications. We found three main layers that contribute to improved safety in critical decision systems: Reliability, Validation and Verification.

Particular solutions to enhance the safety of ML in critical decision systems include model improvement by revealing bias and improving robustness for enhanced reliability, validation of models in long-term tests under real operating conditions or with a human in the loop for validating a model, and formal methods for model verification.

Based on these three layers, we proposed the R4VR-framework. The framework provides developers and practitioners with actionable guidelines that facilitate the responsible deployment of ML models. Specifically, our framework aims to create models with clearly defined permissible usage ranges. It thereby aids end users in making safe, informed decisions in critical decision systems. These practical guidelines are particularly useful for high-stakes industries, where incorrect decisions can have severe consequences.

Finally, we pointed out research gaps and promising future directions of XAI research in critical decision systems. Gaps remain in scaling XAI techniques for real world deployment. Future research should also prioritize methods for efficiently validating and verifying ML models. Moreover, we see an opportunity for integrating XAI with advanced risk mitigation approaches, such as uncertainty quantification and human-in-the-loop systems to further ensure the correctness and trustworthiness of model decisions. Concerning the proposed R4VR-framework, there are still many open questions regarding its technical implementation and applicability in real world scenarios. The framework should be complemented by technical solutions for the single phases. Based on this, a toolkit can potentially be developed that provides developers and end users with easy to use methods for enhancing safety of ML applications in critical decision systems.

In summary, we contribute to research on XAI by providing a structured overview and organization of current XAI applications in critical decision systems, highlighting research gaps and proposing future research directions aimed at improving safety of ML models in critical decision systems. We encourage researchers to deal with the question of how to apply XAI to create more safe ML applications. By focusing on enhancing reliability, validation, and verification in critical decision systems, we can collectively work toward a future where ML models are not only accurate and efficient but also safe. This shift in research focus will be crucial to fully realizing the transformative potential of ML.

Author Contributions

C.R. and J.W. contributed to the design and implementation of the research, to the analysis of the results and to the writing of the manuscript. All authors have read and agreed to the published version of the manuscript.

Funding

This research received no external funding.

Conflicts of Interest

The authors declare no conflicts of interest.

Abbreviations

The following abbreviations are used in this manuscript:

AI	Artificial Imtelligence
CAM	Class Activation Mapping
CP-Nets	Coloured Petri nets
DNN	Deep neural network
DOAJ	Directory of open access journals
ELI5	Explain like I am 5
EU	European Union
GRU	Gated recurrent unit
HMI	Human machine interface
IG	Integrated Gradient
LD	Linear dichroism
LIME	Local interpretable model-agnostic explanations
LRP	Layer wise relevance propagation
LSTM	Long short term memory
MDPI	Multidisciplinary Digital Publishing Institute
ML	Machine learning
R4VR	Reliability, validation, verification, verification, validation, reliability
RQ	Research question
SHAP	Shapely additive explanations
SPAR-4-SLR	Systematic procedures and rationals for systematic literature reviews
TLA	Three letter acronym
XAI	Explainable artificial intelligence

Appendix A. Search String Used for IEEE Digital Library

(“Document Title”:Explanation OR “Document Title”: Explainable OR “Document Title”: Explainability OR “Document Title”: Explaining OR “Document Title”: interpretable OR “Document Title”: interpretability OR “Document Title”: explanations OR “Document Title”: XAI)

AND (“Document Title”:Machine Learning OR “Document Title”: ML OR “Document Title”: Deep Learning OR “Document Title”: AI OR “Document Title”: Artificial Intelligence OR “Document Title”: Neural Network OR “Document Title”: Supervised Learning OR “Document Title”: Unsupervised Learning OR “Document Title”: XAI)

AND (“Document Title”: Critical OR “Document Title”: Security OR “Document Title”: Safety OR “Document Title”: Legal OR “Document Title”: ethic OR “Document Title”: Civil protection OR “Document Title”: Nuclear OR “Document Title”: Defense OR “Document Title”: Public administration OR “Document Title”: Autonomous Driving OR “Document Title”: Autonomous Vehicle OR “Document Title”: Autonomous Vehicular OR “Document Title”: Self Driving OR “Document Title”: Medicine OR “Document Title”: Medical OR “Document Title”: Health OR “Document Title”: Disease OR “Document Title”: Healthcare OR “Document Title”: Hospital OR “Document Title”: 5G OR “Document Title”: 6G OR “Document Title”: IoT OR “Document Title”: Internet of Things OR “Document Title”: CPS OR “Document Title”: Cyber Physical System OR “Document Title”: Digital infrastructure OR “Document Title”: Control OR “Document Title”: Industry 4.0 OR “Document Title”: Agriculture OR “Document Title”: Food Supply OR “Document Title”: Energy OR “Document Title”: Electricity OR “Document Title”: Gas OR “Document Title”: Oil OR “Document Title”: Power OR “Document Title”: Aerospace OR “Document Title”: UAV OR “Document Title”: Wastewater OR “Document Title”: Drinking water OR “Document Title”: Transport OR “Document Title”: Transportation OR “Document Title”: Banking OR “Document Title”: Financial OR “Document Title”: Telecommunication OR “Document Title”: Space)

Appendix B. Search String Used for ACM Digital Library

[[Title: critical] OR [Title: security] OR [Title: safety] OR [Title: legal] OR [Title: ethic] OR [Title: civil protection] OR [Title: nuclear] OR [Title: defense] OR [Title: public administration] OR [Title: autonomous driving] OR [Title: autonomous vehicle] OR [Title: autonomous vehicular] OR [Title: self driving] OR [Title: medicine] OR [Title: medical] OR [Title: health] OR [Title: disease] OR [Title: healthcare] OR [Title: hospital] OR [Title: 5g] OR [Title: 6g] OR [Title: iot] OR [Title: internet of things] OR [Title: cps] OR [Title: cyber physical system] OR [Title: digital infrastructure] OR [Title: control] OR [Title: industry 4.0] OR [Title: agriculture] OR [Title: food supply] OR [Title: energy] OR [Title: electricity] OR [Title: gas] OR [Title: oil] OR [Title: power] OR [Title: aerospace] OR [Title: uav] OR [Title: wastewater] OR [Title: drinking water] OR [Title: transport] OR [Title: transportation] OR [Title: banking] OR [Title: financial] OR [Title: telecommunication] OR [Title: space]]

AND [[Title: machine learning] OR [Title: ml] OR [Title: deep learning] OR [Title: ai] OR [Title: artificial intelligence] OR [Title: neural network] OR [Title: supervised learning] OR [Title: unsupervised learning] OR [Title: xai]]

AND [[Title: explanation] OR [Title: explainable] OR [Title: explainability] OR [Title: explaining] OR [Title: interpretable] OR [Title: interpretability] OR [Title: explanations] OR [Title: xai]]

AND [E-Publication Date: (01/01/2020)]

References

Alimonda, N.; Guidotto, L.; Malandri, L.; Mercorio, F.; Mezzanzanica, M.; Tosi, G. A Survey on XAI for Cyber Physical Systems in Medicine. In Proceedings of the 2022 IEEE International Conference on Metrology for Extended Reality, Artificial Intelligence and Neural Engineering (MetroXRAINE), Rome, Italy, 26–28 October 2022; pp. 265–270. [Google Scholar] [CrossRef]
Khan, N.; Nauman, M.; Almadhor, A.S.; Akhtar, N.; Alghuried, A.; Alhudhaif, A. Guaranteeing Correctness in Black-Box Machine Learning: A Fusion of Explainable AI and Formal Methods for Healthcare Decision-Making. IEEE Access 2024, 12, 90299–90316. [Google Scholar] [CrossRef]
Renjith, V.; Judith, J. A Review on Explainable Artificial Intelligence for Gastrointestinal Cancer using Deep Learning. In Proceedings of the 2023 Annual International Conference on Emerging Research Areas: International Conference on Intelligent Systems (AICERA/ICIS), Kerala, India, 16–18 November 2023; pp. 1–6. [Google Scholar] [CrossRef]
European Parliament and Council of the European Union. Directive (EU) 2022/2557 of the European Parliament and of the Council of 14 December 2022 on the Resilience of Critical Entities and Repealing Council Directive 2008/114/EC (Text with EEA Relevance). Official Journal of the European Union, L 333, 27 December 2022. 2022, pp. 164–196. Available online: https://eur-lex.europa.eu/eli/dir/2022/2557/oj (accessed on 17 September 2024).
European Parliament. EU AI Act: First Regulation on Artificial Intelligence. 2023. Available online: https://www.europarl.europa.eu/topics/en/article/20230601STO93804/eu-ai-act-first-regulation-on-artificial-intelligence (accessed on 28 September 2024).
Adadi, A.; Berrada Khan, M. Peeking inside the black-box: A survey on explainable artificial intelligence (XAI). IEEE Access 2018, 6, 52138–52160. [Google Scholar] [CrossRef]
Kaur, D.; Uslu, S.; Rittichier, K.J.; Durresi, A. Trustworthy Artificial Intelligence: A Review. ACM Comput. Surv. (CSUR) 2022, 55, 1–38. [Google Scholar] [CrossRef]
Mahajan, P.; Aujla, G.S.; Krishna, C.R. Explainable Edge Computing in a Distributed AI—Powered Autonomous Vehicular Networks. In Proceedings of the 2024 IEEE International Conference on Communications Workshops (ICC Workshops), Denver, CO, USA, 9–13 March 2024; pp. 1195–1200. [Google Scholar] [CrossRef]
Paul, S.; Vijayshankar, S.; Macwan, R. Demystifying Cyberattacks: Potential for Securing Energy Systems With Explainable AI. In Proceedings of the 2024 International Conference on Computing, Networking and Communications (ICNC), Hawaii, HI, USA, 19–22 February 2024; pp. 430–434. [Google Scholar] [CrossRef]
Afzal-Houshmand, S.; Papamartzivanos, D.; Homayoun, S.; Veliou, E.; Jensen, C.D.; Voulodimos, A.; Giannetsos, T. Explainable Artificial Intelligence to Enhance Data Trustworthiness in Crowd-Sensing Systems. In Proceedings of the 2023 19th International Conference on Distributed Computing in Smart Systems and the Internet of Things (DCOSS-IoT), Pafos, Cyprus, 19–21 June 2023; pp. 568–576. [Google Scholar] [CrossRef]
Moghadasi, N.; Piran, M.; Valdez, R.S.; Baek, S.; Moghaddasi, N.; Polmateer, T.L.; Lambert, J.H. Process Quality Assurance of Artificial Intelligence in Medical Diagnosis. In Proceedings of the 2024 International Conference on Intelligent Systems and Computer Vision (ISCV), Fez, Morocco, 8–10 May 2024; pp. 1–8. [Google Scholar] [CrossRef]
Masud, M.T.; Keshk, M.; Moustafa, N.; Linkov, I.; Emge, D.K. Explainable Artificial Intelligence for Resilient Security Applications in the Internet of Things. IEEE Open J. Commun. Soc. 2024. [Google Scholar] [CrossRef]
Crook, B.; Schlüter, M.; Speith, T. Revisiting the Performance-Explainability Trade-Off in Explainable Artificial Intelligence (XAI). In Proceedings of the 2023 IEEE 31st International Requirements Engineering Conference Workshops (REW), Hannover, Germany, 4–8 September 2023; pp. 316–324. [Google Scholar] [CrossRef]
Molnar, C. Interpretable Machine Learning, 2nd ed.; Independently Published: Munich, Germany, 2022. [Google Scholar]
Gizzini, A.K.; Shukor, M.; Ghandour, A.J. Extending CAM-based XAI methods for Remote Sensing Imagery Segmentation. arXiv 2023, arXiv:2310.01837. [Google Scholar] [CrossRef]
Das, T.; Samandar, S.; Rouphail, N.; Williams, B.; Harris, D. Examining Factors Influencing the Acceleration Behavior of Autonomous Vehicles Through Explainable AI Analysis. In Proceedings of the 2024 Smart City Symposium Prague (SCSP), Prague, Czech Republic, 23–24 May 2024; pp. 1–6. [Google Scholar] [CrossRef]
Adams, J.; Hagras, H. A Type-2 Fuzzy Logic Approach to Explainable AI for regulatory compliance, fair customer outcomes and market stability in the Global Financial Sector. In Proceedings of the 2020 IEEE International Conference on Fuzzy Systems (FUZZ-IEEE), Glasgow, UK, 19–24 July 2020; pp. 1–8. [Google Scholar] [CrossRef]
Jia, Y.; McDermid, J.; Lawton, T.; Habli, I. The Role of Explainability in Assuring Safety of Machine Learning in Healthcare. IEEE Trans. Emerg. Top. Comput. 2022, 10, 1746–1760. [Google Scholar] [CrossRef]
European Commission. Critical Infrastructure Resilience at EU Level. 2024. Available online: https://home-affairs.ec.europa.eu/policies/internal-security/counter-terrorism-and-radicalisation/protection/critical-infrastructure-resilience-eu-level_en (accessed on 28 September 2024).
Tjoa, E.; Guan, C. A Survey on Explainable Artificial Intelligence (XAI): Toward Medical XAI. IEEE Trans. Neural Netw. Learn. Syst. 2021, 32, 4793–4813. [Google Scholar] [CrossRef]
Farkhadov, M.; Eliseev, A.; Petukhova, N. Explained Artificial Intelligence Helps to Integrate Artificial and Human Intelligence Into Medical Diagnostic Systems: Analytical Review of Publications. In Proceedings of the 2020 IEEE 14th International Conference on Application of Information and Communication Technologies (AICT), Tashkent, Uzbekistan, 7–9 October 2020; pp. 1–4. [Google Scholar] [CrossRef]
Jagatheesaperumal, S.K.; Pham, Q.V.; Ruby, R.; Yang, Z.; Xu, C.; Zhang, Z. Explainable AI Over the Internet of Things (IoT): Overview, State-of-the-Art and Future Directions. IEEE Open J. Commun. Soc. 2022, 3, 2106–2136. [Google Scholar] [CrossRef]
Zhang, Z.; Hamadi, H.A.; Damiani, E.; Yeun, C.Y.; Taher, F. Explainable Artificial Intelligence Applications in Cyber Security: State-of-the-Art in Research. IEEE Access 2022, 10, 93104–93139. [Google Scholar] [CrossRef]
Machlev, R.; Heistrene, L.; Perl, M.; Levy, K.; Belikov, J.; Mannor, S.; Levron, Y. Explainable Artificial Intelligence (XAI) techniques for energy and power systems: Review, challenges and opportunities. Energy AI 2022, 9, 100169. [Google Scholar] [CrossRef]
Kuznietsov, A.; Gyevnar, B.; Wang, C.; Peters, S.; Albrecht, S.V. Explainable AI for Safe and Trustworthy Autonomous Driving: A Systematic Review. IEEE Trans. Intell. Transp. Syst. 2024, 25, 19342–19364. [Google Scholar] [CrossRef]
Paul, J.; Lim, W.M.; O’Cass, A.; Hao, A.; Bresciani, S. Scientific Procedures and Rationales for Systematic Literature Reviews (SPAR-4-SLR). Int. J. Consum. Stud. 2021, 45, O1–O16. [Google Scholar] [CrossRef]
Kommission, E.; Generaldirektion Kommunikationsnetze, I.U.T. Ethik-Leitlinien für eine Vertrauenswürdige KI; Publications Office: Luxembourg, 2019. [Google Scholar] [CrossRef]
The IEEE Global Initiative on Ethics of Autonomous and Intelligent Systems. Ethically Aligned Design: A Vision for Prioritizing Human Well-Being with Autonomous and Intelligent Systems, Version 2. Technical Report, IEEE Standards Association. 2017. Available online: https://standards.ieee.org/wp-content/uploads/import/documents/other/ead_v2.pdf (accessed on 11 October 2024).
Amin, A.; Hasan, K.; Zein-Sabatto, S.; Chimba, D.; Ahmed, I.; Islam, T. An Explainable AI Framework for Artificial Intelligence of Medical Things. In Proceedings of the 2023 IEEE Globecom Workshops (GC Wkshps), Kuala Lumpur, Malaysia, 4–8 December 2023; pp. 2097–2102. [Google Scholar] [CrossRef]
Oseni, A.; Moustafa, N.; Creech, G.; Sohrabi, N.; Strelzoff, A.; Tari, Z.; Linkov, I. An Explainable Deep Learning Framework for Resilient Intrusion Detection in IoT-Enabled Transportation Networks. IEEE Trans. Intell. Transp. Syst. 2023, 24, 1000–1014. [Google Scholar] [CrossRef]
Shtayat, M.M.; Hasan, M.K.; Sulaiman, R.; Islam, S.; Khan, A.U.R. An Explainable Ensemble Deep Learning Approach for Intrusion Detection in Industrial Internet of Things. IEEE Access 2023, 11, 115047–115061. [Google Scholar] [CrossRef]
Mridha, K.; Uddin, M.M.; Shin, J.; Khadka, S.; Mridha, M.F. An Interpretable Skin Cancer Classification Using Optimized Convolutional Neural Network for a Smart Healthcare System. IEEE Access 2023, 11, 41003–41018. [Google Scholar] [CrossRef]
Jahan, S.; Alqahtani, S.; Gamble, R.F.; Bayesh, M. Automated Extraction of Security Profile Information from XAI Outcomes. In Proceedings of the 2023 IEEE International Conference on Autonomic Computing and Self-Organizing Systems Companion (ACSOS-C), Toronto, ON, Canada, 26–29 September 2023; pp. 110–115. [Google Scholar] [CrossRef]
Gu, R.; Wang, G.; Song, T.; Huang, R.; Aertsen, M.; Deprest, J.; Ourselin, S.; Vercauteren, T.; Zhang, S. CA-Net: Comprehensive Attention Convolutional Neural Networks for Explainable Medical Image Segmentation. IEEE Trans. Med. Imag. 2021, 40, 699–711. [Google Scholar] [CrossRef]
Shen, Z.; Jiang, X.; Huang, X. Deep Learning-based Interpretable Detection Method for Fundus Diseases: Diagnosis and Information Mining of Diseases based on Fundus Photography Images. In Proceedings of the 2023 3rd International Conference on Bioinformatics and Intelligent Computing, Sanya, China, 10–12 February 2023; pp. 305–309. [Google Scholar] [CrossRef]
Han, D.; Wang, Z.; Chen, W.; Zhong, Y.; Wang, S.; Zhang, H.; Yang, J.; Shi, X.; Yin, X. DeepAID: Interpreting and Improving Deep Learning-based Anomaly Detection in Security Applications. In Proceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security, New York, NY, USA, 15–19 November 2021; pp. 3197–3217. [Google Scholar] [CrossRef]
Apon, T.S.; Hasan, M.M.; Islam, A.; Alam, M.G.R. Demystifying Deep Learning Models for Retinal OCT Disease Classification using Explainable AI. In Proceedings of the 2021 IEEE Asia-Pacific Conference on Computer Science and Data Engineering (CSDE), Brisbane, Australia, 8–10 December 2021; pp. 1–6. [Google Scholar] [CrossRef]
Kapcia, M.; Eshkiki, H.; Duell, J.; Fan, X.; Zhou, S.; Mora, B. ExMed: An AI Tool for Experimenting Explainable AI Techniques on Medical Data Analytics. In Proceedings of the 2021 IEEE 33rd International Conference on Tools with Artificial Intelligence (ICTAI), Virtual, 1–3 November 2021; pp. 841–845. [Google Scholar] [CrossRef]
Gürbüz, E.; Turgut, Ö.; Kök, I. Explainable AI-Based Malicious Traffic Detection and Monitoring System in Next-Gen IoT Healthcare. In Proceedings of the 2023 International Conference on Smart Applications, Communications and Networking (SmartNets), Istanbul, Turkey, 25–27 July 2023; pp. 1–6. [Google Scholar] [CrossRef]
Nguyen, T.N.; Yang, H.J.; Kho, B.G.; Kang, S.R.; Kim, S.H. Explainable Deep Contrastive Federated Learning System for Early Prediction of Clinical Status in Intensive Care Unit. IEEE Access 2024, 12, 117176–117202. [Google Scholar] [CrossRef]
Drichel, A.; Meyer, U. False Sense of Security: Leveraging XAI to Analyze the Reasoning and True Performance of Context-less DGA Classifiers. In Proceedings of the 26th International Symposium on Research in Attacks, Intrusions and Defenses, Hong Kong, China, 16–18 October 2023; pp. 330–345. [Google Scholar] [CrossRef]
Friedrich, M.; Küls, J.; Findeisen, M.; Peinecke, N. HMI Design for Explainable Machine Learning Enhanced Risk Detection in Low-Altitude UAV Operations. In Proceedings of the 2023 IEEE/AIAA 42nd Digital Avionics Systems Conference (DASC), Barcelon, Spain, 1–5 October 2023; pp. 1–8. [Google Scholar] [CrossRef]
Li, J.; Chen, Y.; Wang, Y.; Ye, Y.; Sun, M.; Ren, H.; Cheng, W.; Zhang, H. Interpretable Pulmonary Disease Diagnosis with Graph Neural Network and Counterfactual Explanations. In Proceedings of the 2023 2nd International Conference on Sensing, Measurement, Communication and Internet of Things Technologies (SMC-IoT), Changsha, China, 29–31 December 2023; pp. 146–154. [Google Scholar] [CrossRef]
Gyawali, S.; Huang, J.; Jiang, Y. Leveraging Explainable AI for Actionable Insights in IoT Intrusion Detection. In Proceedings of the 2024 19th Annual System of Systems Engineering Conference (SoSE), Tacoma, WA, USA, 23–26 June 2024; pp. 92–97. [Google Scholar] [CrossRef]
Dutta, J.; Puthal, D.; Yeun, C.Y. Next Generation Healthcare with Explainable AI: IoMT-Edge-Cloud Based Advanced eHealth. In Proceedings of the GLOBECOM 2023—2023 IEEE Global Communications Conference, Kuala Lumpur, Malaysia, 4–8 December 2023; pp. 7327–7332. [Google Scholar] [CrossRef]
Haque, E.; Hasan, K.; Ahmed, I.; Alam, M.S.; Islam, T. Towards an Interpretable AI Framework for Advanced Classification of Unmanned Aerial Vehicles (UAVs). In Proceedings of the 2024 IEEE 21st Consumer Communications & Networking Conference (CCNC), Las Vegas, NV, USA, 6–9 January 2024; pp. 644–645. [Google Scholar] [CrossRef]
Astolfi, D.; De Caro, F.; Vaccaro, A. Wind Power Applications of eXplainable Artificial Intelligence Techniques. In Proceedings of the 2023 AEIT International Annual Conference (AEIT), Rome, Italy, 5–7 October 2023; pp. 1–6. [Google Scholar] [CrossRef]
Porambage, P.; Pinola, J.; Rumesh, Y.; Tao, C.; Huusko, J. XcARet: XAI based Green Security Architecture for Resilient Open Radio Access Networks in 6G. In Proceedings of the 2023 Joint European Conference on Networks and Communications & 6G Summit (EuCNC/6G Summit), Gothenburg, Sweden, 6–9 June 2023; pp. 699–704. [Google Scholar] [CrossRef]
Tahmassebi, A.; Martin, J.; Meyer-Baese, A.; Gandomi, A.H. An Interpretable Deep Learning Framework for Health Monitoring Systems: A Case Study of Eye State Detection using EEG Signals. In Proceedings of the 2020 IEEE Symposium Series on Computational Intelligence (SSCI), Canberra, Australia, 1–4 December 2020; pp. 211–218. [Google Scholar] [CrossRef]
Hamilton, D.; Kornegay, K.; Watkins, L. Autonomous Navigation Assurance with Explainable AI and Security Monitoring. In Proceedings of the 2020 IEEE Applied Imagery Pattern Recognition Workshop (AIPR), Washington, DC, USA, 13–15 October 2020; pp. 1–7. [Google Scholar] [CrossRef]
Kommineni, S.; Muddana, S.; Senapati, R. Explainable Artificial Intelligence based ML Models for Heart Disease Prediction. In Proceedings of the 2024 3rd International Conference on Computational Modelling, Simulation and Optimization (ICCMSO), Phuket, Thailand, 14–16 June 2024; pp. 160–164. [Google Scholar] [CrossRef]
Tan, B.; Zhao, J.; Su, T.; Huang, Q.; Zhang, Y.; Zhang, H. Explainable Bayesian Neural Network for Probabilistic Transient Stability Analysis Considering Wind Energy. In Proceedings of the 2022 IEEE Power & Energy Society General Meeting (PESGM), Austin, TX, USA, 17–21 July 2022; pp. 1–5. [Google Scholar] [CrossRef]
Nazat, S.; Li, L.; Abdallah, M. XAI-ADS: An Explainable Artificial Intelligence Framework for Enhancing Anomaly Detection in Autonomous Driving Systems. IEEE Access 2024, 12, 48583–48607. [Google Scholar] [CrossRef]
Sutthithatip, S.; Perinpanayagam, S.; Aslam, S. (Explainable) Artificial Intelligence in Aerospace Safety-Critical Systems. In Proceedings of the 2022 IEEE Aerospace Conference (AERO), Big Sky, MT, USA, 5–12 March 2022; pp. 1–12. [Google Scholar] [CrossRef]
Rožman, J.; Hagras, H.; Andreu-Perez, J.; Clarke, D.; Müeller, B.; Fitz, S. A Type-2 Fuzzy Logic Based Explainable AI Approach for the Easy Calibration of AI models in IoT Environments. In Proceedings of the 2021 IEEE International Conference on Fuzzy Systems (FUZZ-IEEE), Luxembourg, 11–14 July 2021; pp. 1–8. [Google Scholar] [CrossRef]
Zhang, X.; Han, L.; Zhu, W.; Sun, L.; Zhang, D. An Explainable 3D Residual Self-Attention Deep Neural Network for Joint Atrophy Localization and Alzheimer’s Disease Diagnosis Using Structural MRI. IEEE J. Biomed. Health Inform. 2022, 26, 5289–5297. [Google Scholar] [CrossRef]
Ren, C.; Xu, Y.; Zhang, R. An Interpretable Deep Learning Method for Power System Transient Stability Assessment via Tree Regularization. IEEE Trans. Power Syst. 2022, 37, 3359–3369. [Google Scholar] [CrossRef]
Jing, Y.; Liu, H.; Guo, R. An Interpretable Soft Sensor Model for Power Plant Process Based on Deep Learning. In Proceedings of the 2023 IEEE 7th Conference on Energy Internet and Energy System Integration (EI2), Hangzhou, China, 15–18 December 2023; pp. 2079–2085. [Google Scholar] [CrossRef]
Watson, M.; Al Moubayed, N. Attack-agnostic Adversarial Detection on Medical Data Using Explainable Machine Learning. In Proceedings of the 2020 25th International Conference on Pattern Recognition (ICPR), Milan, Italy, 10–15 January 2021; pp. 8180–8187. [Google Scholar] [CrossRef]
Liu, S.; Müller, S. Reliability of Deep Neural Networks for an End-to-End Imitation Learning-Based Lane Keeping. IEEE Trans. Intell. Transp. Syst. 2023, 24, 13768–13786. [Google Scholar] [CrossRef]
Manju, V.N.; Aparna, N.; Krishna Sowjanya, K. Decision Tree-Based Explainable AI for Diagnosis of Chronic Kidney Disease. In Proceedings of the 2023 5th International Conference on Inventive Research in Computing Applications (ICIRCA), Coimbatore, India, 3–5 August 2023; pp. 947–952. [Google Scholar] [CrossRef]
Rodríguez-Barroso, N.; Del Ser, J.; Luzón, M.V.; Herrera, F. Defense Strategy against Byzantine Attacks in Federated Machine Learning: Developments towards Explainability. In Proceedings of the 2024 IEEE International Conference on Fuzzy Systems (FUZZ-IEEE), Yokohama, Japan, 30 June–5 July 2024; pp. 1–8. [Google Scholar] [CrossRef]
Shukla, A.; Upadhyay, S.; Bachan, P.R.; Bera, U.N.; Kshirsagar, R.; Nathani, N. Dynamic Explainability in AI for Neurological Disorders: An Adaptive Model for Transparent Decision-Making in Alzheimer’s Disease Diagnosis. In Proceedings of the 2024 IEEE 13th International Conference on Communication Systems and Network Technologies (CSNT), Jabalpur, India, 6–7 April 2024; pp. 980–986. [Google Scholar] [CrossRef]
Haque, E.; Hasan, K.; Ahmed, I.; Alam, M.S.; Islam, T. Enhancing UAV Security Through Zero Trust Architecture: An Advanced Deep Learning and Explainable AI Analysis. In Proceedings of the 2024 International Conference on Computing, Networking and Communications (ICNC), Hawaii, HI, USA, 19–22 February 2024; pp. 463–467. [Google Scholar] [CrossRef]
Duamwan, L.M.; Bird, J.J. Explainable AI for Medical Image Processing: A Study on MRI in Alzheimer’s Disease. In Proceedings of the 16th International Conference on PErvasive Technologies Related to Assistive Environments, New York, NY, USA, 5–7 July 2023; pp. 480–484. [Google Scholar] [CrossRef]
Ray, I.; Sreedharan, S.; Podder, R.; Bashir, S.K.; Ray, I. Explainable AI for Prioritizing and Deploying Defenses for Cyber-Physical System Resiliency. In Proceedings of the 2023 5th IEEE International Conference on Trust, Privacy and Security in Intelligent Systems and Applications (TPS-ISA), Atlanta, GA, USA, 1–3 November 2023; pp. 184–192. [Google Scholar] [CrossRef]
Hellen, N.; Marvin, G. Explainable AI for Safe Water Evaluation for Public Health in Urban Settings. In Proceedings of the 2022 International Conference on Innovations in Science, Engineering and Technology (ICISET), Kumira, Bangladesh, 25–28 February 2022; pp. 1–6. [Google Scholar] [CrossRef]
Rjoub, G.; Bentahar, J.; Wahab, O.A. Explainable AI-based Federated Deep Reinforcement Learning for Trusted Autonomous Driving. In Proceedings of the 2022 International Wireless Communications and Mobile Computing (IWCMC), Dubrovnik, Croatia, 30 May–3 June 2022; pp. 318–323. [Google Scholar] [CrossRef]
Bi, C.; Luo, Y.; Lu, C. Explainable Artificial Intelligence for Power System Security Assessment: A Case Study on Short-Term Voltage Stability. In Proceedings of the 2023 IEEE Belgrade PowerTech, Belgrade, Serbia, 25–29 June 2023; pp. 1–6. [Google Scholar] [CrossRef]
Wang, K.; Yin, S.; Wang, Y.; Li, S. Explainable Deep Learning for Medical Image Segmentation with Learnable Class Activation Mapping. In Proceedings of the 2023 2nd Asia Conference on Algorithms, Computing and Machine Learning, Shanghai, China, 17–19 May 2023; pp. 210–215. [Google Scholar] [CrossRef]
Wickramasinghe, C.S.; Amarasinghe, K.; Marino, D.L.; Rieger, C.; Manic, M. Explainable Unsupervised Machine Learning for Cyber-Physical Systems. IEEE Access 2021, 9, 131824–131843. [Google Scholar] [CrossRef]
Yang, J.; Tang, D.; Yu, J.; Zhang, J.; Liu, H. Explaining Anomalous Events in Flight Data of UAV With Deep Attention-Based Multi-Instance Learning. IEEE Trans. Veh. Technol. 2024, 73, 107–119. [Google Scholar] [CrossRef]
Rezazadeh, F.; Chergui, H.; Mangues-Bafalluy, J. Explanation-Guided Deep Reinforcement Learning for Trustworthy 6G RAN Slicing. In Proceedings of the 2023 IEEE International Conference on Communications Workshops (ICC Workshops), Rome, Italy, 28 May–1 June 2023; pp. 1026–1031. [Google Scholar] [CrossRef]
Kalakoti, R.; Nõmm, S.; Bahsi, H. Improving Transparency and Explainability of Deep Learning Based IoT Botnet Detection Using Explainable Artificial Intelligence (XAI). In Proceedings of the 2023 International Conference on Machine Learning and Applications (ICMLA), Jacksonville, FL, USA, 15–17 December 2023; pp. 595–601. [Google Scholar] [CrossRef]
Ouhssini, M.; Afdel, K.; Akouhar, M.; Agherrabi, E.; Abarda, A. Interpretable Deep Learning for DDoS Defense: A SHAP-based Approach in Cloud Computing. In Proceedings of the 2024 International Conference on Circuit, Systems and Communication (ICCSC), Fez, Moroco, 28–29 June 2024; pp. 1–8. [Google Scholar] [CrossRef]
Reza, M.T.; Ahmed, F.; Sharar, S.; Rasel, A.A. Interpretable Retinal Disease Classification from OCT Images Using Deep Neural Network and Explainable AI. In Proceedings of the 2021 International Conference on Electronics, Communications and Information Technology (ICECIT), Khulna, Bangladesh, 14–16 September 2021; pp. 1–4. [Google Scholar] [CrossRef]
Rani, J.V.; Saeed Ali, H.A.; Jakka, A. IoT Network Intrusion Detection: An Explainable AI Approach in Cybersecurity. In Proceedings of the 2023 4th International Conference on Communication, Computing and Industry 6.0 (C216), Bangalore, India, 15–16 December 2023; pp. 1–6. [Google Scholar] [CrossRef]
Krishnaveni, S.; Sivamohan, S.; Chen, T.M.; Sathiyanarayanan, M. NexGuard: Industrial Cyber-Physical System Défense Using Ensemble Feature Selection and Explainable Deep Learning Techniques. In Proceedings of the 2023 2nd International Conference on Futuristic Technologies (INCOFT), Belagavi, India, 24–26 November 2023; pp. 1–10. [Google Scholar] [CrossRef]
Cavaliere, F.; Cioppa, A.D.; Marcelli, A.; Parziale, A.; Senatore, R. Parkinson’s Disease Diagnosis: Towards Grammar-based Explainable Artificial Intelligence. In Proceedings of the 2020 IEEE Symposium on Computers and Communications (ISCC), Rennes, France, 7–10 July 2020; pp. 1–6. [Google Scholar] [CrossRef]
Feifel, P.; Bonarens, F.; Köster, F. Reevaluating the Safety Impact of Inherent Interpretability on Deep Neural Networks for Pedestrian Detection. In Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Nashville, TN, USA, 20–25 June 2021; pp. 29–37. [Google Scholar] [CrossRef]
Kur, J.; Chen, J.; Huang, J. Scalable Industrial Control System Analysis via XAI-Based Gray-Box Fuzzing. In Proceedings of the 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE), Kirchberg, Luxembourg, 11–15 September 2023; pp. 1803–1807. [Google Scholar] [CrossRef]
Ahmad Khan, M.; Khan, M.; Dawood, H.; Dawood, H.; Daud, A. Secure Explainable-AI Approach for Brake Faults Prediction in Heavy Transport. IEEE Access 2024, 12, 114940–114950. [Google Scholar] [CrossRef]
Duell, J.; Fan, X.; Burnett, B.; Aarts, G.; Zhou, S.M. A Comparison of Explanations Given by Explainable Artificial Intelligence Methods on Analysing Electronic Health Records. In Proceedings of the 2021 IEEE EMBS International Conference on Biomedical and Health Informatics (BHI), Virtual, 27–30 July 2021; pp. 1–4. [Google Scholar] [CrossRef]
Wu, W.; Keller, J.M.; Skubic, M.; Popescu, M. Explainable AI for Early Detection of Health Changes Via Streaming Clustering. In Proceedings of the 2022 IEEE International Conference on Fuzzy Systems (FUZZ-IEEE), Padua, Italy, 18–23 July 2022; pp. 1–6. [Google Scholar] [CrossRef]
Yukta; Biswas, A.P.; Kashyap, S. Explainable AI for Healthcare Diagnosis in Renal Cancer. In Proceedings of the 2024 OPJU International Technology Conference (OTCON) on Smart Computing for Innovation and Advancement in Industry 4.0, Raigarh, India, 5–7 June 2024; pp. 1–6. [Google Scholar] [CrossRef]
Okolo, C.T. Navigating the Limits of AI Explainability: Designing for Novice Technology Users in Low-Resource Settings. In Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society, New York, NY, USA, 8–10 August 2023; pp. 959–961. [Google Scholar] [CrossRef]
Ketata, F.; Masry, Z.A.; Yacoub, S.; Zerhouni, N. A Methodology for Reliability Analysis of Explainable Machine Learning: Application to Endocrinology Diseases. IEEE Access 2024, 12, 101921–101935. [Google Scholar] [CrossRef]
Pawlicki, M.; Pawlicka, A.; Kozik, R.; Choraś, M. Explainability versus Security: The Unintended Consequences of xAI in Cybersecurity. In Proceedings of the 2nd ACM Workshop on Secure and Trustworthy Deep Learning Systems, New York, NY, USA, 2–20 July 2024; pp. 1–7. [Google Scholar] [CrossRef]
Vuppala, S.K.; Behera, M.; Jack, H.; Bussa, N. Explainable Deep Learning Methods for Medical Imaging Applications. In Proceedings of the 2020 IEEE 5th International Conference on Computing Communication and Automation (ICCCA), Greater Noida, India, 30–31 October 2020; pp. 334–339. [Google Scholar] [CrossRef]
Solano-Kamaiko, I.R.; Mishra, D.; Dell, N.; Vashistha, A. Explorable Explainable AI: Improving AI Understanding for Community Health Workers in India. In Proceedings of the CHI Conference on Human Factors in Computing Systems, New York, NY, USA, 11–16 May 2024. [Google Scholar] [CrossRef]
Hu, Q.; Liu, W.; Liu, Y.; Liu, Z. Interpretability Analysis of Pre-trained Convolutional Neural Networks for Medical Diagnosis. In Proceedings of the 2nd International Conference on Artificial Intelligence, Big Data and Algorithms (CAIBDA 2022), Nanjing, China, 17–19 June 2022; pp. 1–8. [Google Scholar]
Masood, U.; Farooq, H.; Imran, A.; Abu-Dayya, A. Interpretable AI-Based Large-Scale 3D Pathloss Prediction Model for Enabling Emerging Self-Driving Networks. IEEE Trans. Mob. Comput. 2023, 22, 3967–3984. [Google Scholar] [CrossRef]
Tabassum, S.; Parvin, N.; Hossain, N.; Tasnim, A.; Rahman, R.; Hossain, M.I. IoT Network Attack Detection Using XAI and Reliability Analysis. In Proceedings of the 2022 25th International Conference on Computer and Information Technology (ICCIT), Cox’s Bazar, Bangladesh, 17–19 December 2022; pp. 176–181. [Google Scholar] [CrossRef]
Oba, Y.; Tezuka, T.; Sanuki, M.; Wagatsuma, Y. Interpretable Prediction of Diabetes from Tabular Health Screening Records Using an Attentional Neural Network. In Proceedings of the 2021 IEEE 8th International Conference on Data Science and Advanced Analytics (DSAA), Porto, Portugal, 6–9 October 2021; pp. 1–11. [Google Scholar] [CrossRef]
Srivastava, D.; Pandey, H.; Agarwal, A.K.; Sharma, R. Opening the Black Box: Explainable Machine Learning for Heart Disease Patients. In Proceedings of the 2023 International Conference on Advanced Computing Technologies and Applications (ICACTA), Mumbai, India, 6–7 October 2023; pp. 1–5. [Google Scholar] [CrossRef]
Sherry, L.; Baldo, J.; Berlin, B. Design of Flight Guidance and Control Systems Using Explainable AI. In Proceedings of the 2021 Integrated Communications Navigation and Surveillance Conference (ICNS), Virtual Event, 20–22 April 2021; pp. 1–10. [Google Scholar] [CrossRef]
Sutthithatip, S.; Perinpanayagam, S.; Aslam, S.; Wileman, A. Explainable AI in Aerospace for Enhanced System Performance. In Proceedings of the 2021 IEEE/AIAA 40th Digital Avionics Systems Conference (DASC), San Antonio, TX, USA, 3–7 October 2021; pp. 1–7. [Google Scholar] [CrossRef]
Sun, S.C.; Guo, W. Approximate Symbolic Explanation for Neural Network Enabled Water-Filling Power Allocation. In Proceedings of the 2020 IEEE 91st Vehicular Technology Conference (VTC2020-Spring), Antwerpen, Belgium, 27–31 May 2020; pp. 1–4. [Google Scholar] [CrossRef]
Zhang, K.; Xu, P.; Zhang, J. Explainable AI in Deep Reinforcement Learning Models: A SHAP Method Applied in Power System Emergency Control. In Proceedings of the 2020 IEEE 4th Conference on Energy Internet and Energy System Integration (EI2), Wuhan, China, 30 October–1 November 2020; pp. 711–716. [Google Scholar] [CrossRef]
Lee, H.; Lim, H.; Lee, B. Explainable AI-based approaches for power quality prediction in distribution networks considering the uncertainty of renewable energy. In Proceedings of the 27th International Conference on Electricity Distribution (CIRED 2023), Rome, Italy, 12–15 June 2023; Volume 2023, pp. 584–588. [Google Scholar] [CrossRef]
Mahamud, A.H.; Dey, A.K.; Sajedul Alam, A.N.M.; Alam, M.G.R.; Zaman, S. Implementation of Explainable AI in Mental Health Informatics: Suicide Data of the United Kingdom. In Proceedings of the 2022 12th International Conference on Electrical and Computer Engineering (ICECE), Dhaka, Bangladesh, 21–23 December 2022; pp. 457–460. [Google Scholar] [CrossRef]
Brusini, L.; Cruciani, F.; Dall’Aglio, G.; Zajac, T.; Boscolo Galazzo, I.; Zucchelli, M.; Menegaz, G. XAI-Based Assessment of the AMURA Model for Detecting Amyloid-β and Tau Microstructural Signatures in Alzheimer’s Disease. IEEE J. Transl. Eng. Health Med. 2024, 12, 569–579. [Google Scholar] [CrossRef]
Price, J.; Yamazaki, T.; Fujihara, K.; Sone, H. XGBoost: Interpretable Machine Learning Approach in Medicine. In Proceedings of the 2022 5th World Symposium on Communication Engineering (WSCE), Nagoya, Japan, 16–18 September 2022; pp. 109–113. [Google Scholar] [CrossRef]
Zahoor, K.; Bawany, N.Z.; Ghani, U. Explainable AI for Healthcare: An Approach Towards Interpretable Healthcare Models. In Proceedings of the 2023 24th International Arab Conference on Information Technology (ACIT), Ajman, United Arab Emirates, 6–8 December 2023; pp. 1–7. [Google Scholar] [CrossRef]
Abella, J.; Perez, J.; Englund, C.; Zonooz, B.; Giordana, G.; Donzella, C.; Cazorla, F.J.; Mezzetti, E.; Serra, I.; Brando, A.; et al. SAFEXPLAIN: Safe and Explainable Critical Embedded Systems Based on AI. In Proceedings of the 2023 Design, Automation & Test in Europe Conference & Exhibition (DATE), Antwerpen, Belgium, 17–19 April 2023; pp. 1–6. [Google Scholar] [CrossRef]
Seetharaman, T.; Sharma, V.; Balamurugan, B.; Grover, V.; Agnihotri, A. An Efficient and Robust Explainable Artificial Intelligence for Securing Smart Healthcare System. In Proceedings of the 2023 Second International Conference on Smart Technologies for Smart Nation (SmartTechCon), Singapore, 18–19 August 2023; pp. 1066–1071. [Google Scholar] [CrossRef]
Li, B.; Qi, P.; Liu, B.; Di, S.; Liu, J.; Pei, J.; Yi, J.; Zhou, B. Trustworthy AI: From Principles to Practices. ACM Comput. Surv. 2023, 55, 1–46. [Google Scholar] [CrossRef]
Binder, A.; Montavon, G.; Lapuschkin, S.; Müller, K.R.; Samek, W. Layer-Wise Relevance Propagation for Neural Networks with Local Renormalization Layers. In Proceedings of the Artificial Neural Networks and Machine Learning—ICANN 2016, Barcelona, Spain, 6–9 September 2016; pp. 63–71. [Google Scholar] [CrossRef]
Korobov, M.; Lopuhin, K. ELI5 Documentation: Overview. 2024. Available online: https://eli5.readthedocs.io/en/latest/overview.html (accessed on 11 October 2024).
Onyeaka, H.; Anumudu, C.K.; Al-Sharify, Z.T.; Egele-Godswill, E.; Mbaegbu, P. COVID-19 pandemic: A review of the global lockdown and its far-reaching effects. Sci. Prog. 2021, 104, 00368504211019854. [Google Scholar] [CrossRef] [PubMed]
Mercaldo, F.; Brunese, L.; Cesarelli, M.; Martinelli, F.; Santone, A. Respiratory Disease Detection through Spectogram Analysis with Explainable Deep Learning. In Proceedings of the 2023 8th International Conference on Smart and Sustainable Technologies (SpliTech), Split, Croatia, 20–23 June 2023; pp. 1–6. [Google Scholar] [CrossRef]
Kalakoti, R.; Bahsi, H.; Nõmm, S. Improving IoT Security With Explainable AI: Quantitative Evaluation of Explainability for IoT Botnet Detection. IEEE Internet Things J. 2024, 11, 18237–18254. [Google Scholar] [CrossRef]
European Parliament and Council of the European Union. Regulation (EU) 2017/746 of the European Parliament and of the Council of 5 April 2017 on In Vitro Diagnostic Medical Devices and Repealing Directive 98/79/EC and Commission Decision 2010/227/EU. Official Journal of the European Union. 2017. Available online: https://eur-lex.europa.eu/legal-content/EN/TXT/PDF/?uri=CELEX:32017R0746 (accessed on 18 October 2024).
Patel, U.; Patel, V. A comprehensive review: Active learning for hyperspectral image classifications. Earth Sci. Inform. 2023, 16, 1975–1991. [Google Scholar] [CrossRef]

Figure 1. Overview of paper’s structure with results obtained in steps and connections between steps.

Figure 2. Development of dataset in different filtering steps: from originally 508 publications, 2 were dropped as duplicates and 422 were dropped because they don’t contribute to answering the RQ.

Figure 3. Distribution of publication years in final selection of papers: most titles were selected from IEEE Digital Library (blue) and only few from ACM Digital Library (orange); 2023 is year of most frequent publications.

Figure 4. Keyword cloud for keywords in selected publications; terms related to ML are dominant.

Figure 5. Histogram of 15 most common keywords in literature selection; deep learning, explainable ai and machine learning are the most common kewords (25–36 counts).

Figure 6. Classification of most common XAI methods in papers examined in taxonomy of XAI.

Figure 7. Grouping of purposes pursued by XAI in critical decision systems: support users and model improvement as two big groups of XAI purposes.

Figure 8. R4VR-framework for enhancing safety by XAI: Developer creates accountable ML model using XAI in the steps Reliability, ßtextitValidation and Verification; user applies model for safe decision in reverse steps Verification, Validation and Reliability.

Table 1. Inclusion- and exclusion- criteria for publications in the assembling stage of SPAR-4-SLR protocol.

Inclusion-Criteria	Exclusion-Criteria
Written in English	Not publicly available
Conference paper or journal	Book, magazine, inbook, patent etc.
	Published before 2020

Table 2. Key words used for SLR related to explainability, ML and critical decision systems.

Term for Explainability	Term for AI	Term for Critical
machine learning ml, deep learning, artificial intelligence, ai, neural network, supervised learning, xai	xai, explanation, explainable, explaining, interpretable, interpretability, explanations	critical, security, safety, legal, ethic, civil protection, nuclear, defense, public administration, autonomous driving, autonomous vehicle, autonomous vehicular, self driving, medicine, medical, health, disease, healthcare, hospital, 5g, 6g, iot, internet of things, cps, cyber physical system, digital infrastructure, control, industry 4.0, agriculture, food supply, energy, electricity, gas, oil, power, aerospace, uav, wastewater, drinking water, transport, transportation, banking, financial, telecommunication, space

Table 3. Number of papers found in each category when classifying papers for research type.

Type of Research	Number of Publications	Publications
Framework	31	[2,8,9,17,18,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54]
Method	28	[55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82]
Empirical study	15	[83,84,85,86,87,88,89,90,91,92,93,94,95,96,97]
Use case	8	[16,98,99,100,101,102,103,104]
Conceptual work	2	[20,105]

Table 4. Number of papers found in each category when classifying papers for sector of critical infrastructure.

Sector	Number of Publications	Publications
Health	38	[2,18,20,29,32,34,35,37,38,39,40,45,49,51,56,59,61,62,63,65,70,76,79,84,85,86,87,89,90,91,94,95,101,102,103,104,106,107]
Digital infrastructure	20	[30,31,33,36,41,44,48,55,60,66,71,73,74,75,77,78,81,88,92,93]
Transport	14	[8,16,42,46,50,53,54,64,68,72,80,82,96,97]
Energy	9	[9,47,52,57,58,69,98,99,100]
Drinking water	1	[67]
Financial market infrastructure	1	[17]
General	1	[105]

Table 5. Number of papers found in each category when classifying papers for XAI method.

XAI Method	Number of Publications	Publications
SHAP	32	[2,8,29,30,31,39,44,45,47,49,51,52,53,58,61,67,68,73,75,76,78,82,83,84,87,92,94,95,99,101,102,104]
LIME	18	[2,9,29,31,33,37,39,40,44,45,51,65,76,77,82,83,85,87,100,102,104]
Others	13	[17,18,39,41,42,52,55,60,62,71,79,80,81,91,107]
CAM	8	[29,32,35,42,56,70,85,89,104]
Custom	6	[34,36,66,69,72,106]
None	5	[48,74,90,97,105]
Decision Trees	4	[50,57,77,103]
ELI5	2	[39,76]
LRP	2	[41,42]

Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

© 2024 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).

Share and Cite

MDPI and ACS Style

Wiggerthale, J.; Reich, C. Explainable Machine Learning in Critical Decision Systems: Ensuring Safe Application and Correctness. AI 2024, 5, 2864-2896. https://doi.org/10.3390/ai5040138

AMA Style

Wiggerthale J, Reich C. Explainable Machine Learning in Critical Decision Systems: Ensuring Safe Application and Correctness. AI. 2024; 5(4):2864-2896. https://doi.org/10.3390/ai5040138

Chicago/Turabian Style

Wiggerthale, Julius, and Christoph Reich. 2024. "Explainable Machine Learning in Critical Decision Systems: Ensuring Safe Application and Correctness" AI 5, no. 4: 2864-2896. https://doi.org/10.3390/ai5040138

APA Style

Wiggerthale, J., & Reich, C. (2024). Explainable Machine Learning in Critical Decision Systems: Ensuring Safe Application and Correctness. AI, 5(4), 2864-2896. https://doi.org/10.3390/ai5040138

Article Menu