The why and how of trustworthy AI

1 Introduction

Artificial intelligence (AI) is penetrating numerous areas of our lives at a breathtaking speed. In the process, it is increasingly taking over central activities, on the one hand in safety and security-relevant domains, such as automated driving [60], but also in domains that affect human self-determination, such as in the granting of loans [20] or in hiring processes [2]. It is generally assumed that AI will only deploy its full potential if it is used in accordance with high quality standards and in line with our core values. In this context, the concept of trustworthiness has been coined, which has already been the subject of intensive research for several years [16], [17], [54]. For European practice, the Ethics Guidelines for Trustworthy AI of the High-Level Exert Group [23] published in 2019 are directional and it can be expected that their specifications will find their way into legal requirements through the recently published proposal for AI regulation of the European Commission [15] (for an analysis of the proposal, see [51]). It is noteworthy to mention that the High-Level Expert Group’s guidelines, similarly as numerous other AI ethics statements [9], leave open which entity has to comply with their requirements and how. Comparisons to related but simpler domains, such as IT security, make it seem plausible that trustworthiness can and must refer to the organizations that use or develop AI as well as to the AI systems themselves. Looking at the concrete example of “diversity, non-discrimination and fairness,” the HLEG recommends establishing a mechanism that allows others to flag issues related to bias and discrimination [23], which is a non-technical measure of the organization, while the recommendation to monitor fairness through metrics [23] can only be addressed by a technical examination of the specific AI application. For both interpretations, the operationalization of trustworthiness requirements is to a large extent still open [21], [43]. This paper takes both perspectives with a focus on AI applications which are based on Machine Learning (ML) technologies, presents approaches for operationalization, and discusses how they are related.

2 The why of trustworthy AI

The impressive progress of AI in recent years, for instance in medical diagnostics [59] and predictive maintenance [8], was largely driven by Machine Learning. The great potential of ML-based applications originates from the fact that their behavior is not predefined by programmed rules but inferred from data instead. More specifically, ML methods are designed to identify patterns in so-called training data and incorporate their statistical insights in a model. Deep neural networks in particular, which are a type of complex ML models generated from vast amounts of data, are capable of performing tasks that, up to date, could not be described effectively by human rules (e. g., image [12] and speech recognition [4]). However, the data-driven nature of ML gives rise to novel imponderables. For instance, ML-based applications bear the risk to incorporate and reproduce discriminatory patterns in real data [42], by which individuals as well as the reputation of the organization in charge might adversely be affected. This example illustrates why the recommendations and requirements in the numerous AI guidelines presented (see chapter 1) are not solely aimed at preventing bodily injury, property damage or financial loss, as it is the case with “classic” IT security and safety. Complementary, parts of them also consider the (intangible) impact of AI on individuals and society, given, for example, its increasing use to support sensitive decision-making processes (e. g., dismissal of workers [9] or sentencing delinquent persons [1]).

Given the over 700 AI policy initiatives from 60 countries and territories, as well as over 170 emerging AI-related regulation initiatives [47], it is apparent that there are different views around the world on how AI should be used. This partially originates from the diversity of national and territorial AI strategies driving official guidelines and AI regulation e. g., [14], [55], [46], and [52]. While requirements are globally different, one finds recurring elements in numerous AI guidelines, especially when looking at western initiatives. More precisely, there is a consensus among the variety of European as well as US American guidelines that privacy, fairness and transparency (see [34] and [64]) are also, in addition to the established objectives of reliability, safety, and security, essential quality dimensions of AI. Moreover, the EU level documents at which we focus in the following emphasize human autonomy and oversight (partially in contrast to other international perspectives) as a further key requirement for trustworthy AI. While reliability, safety, and security address the proper functioning and potential vulnerabilities of the system (see #2 [23] and Art. 15 [15]), especially the data-driven nature of ML, as mentioned, gives rise to novel aspects. Clearly, as soon as personal data is processed, the privacy of the individual needs to be protected. Here, the use of ML involves risks such as personal data being newly generated during operation (e. g., personal keystroke patterns [37]) or extracted from the ML model without authorization [48], [18]. This notion can be extended to other forms of data e. g., business sensitive data. Furthermore, fairness must be ensured in decision processes (see #5 [23] and Art. 10, 15 [15]), considering that ML algorithms might learn from a biased ground-truth or have a significantly lower performance with respect to certain groups of people [6] (e. g., due to inconsistent data quality). Following, as stated, data-driven AI thrives on concepts that are hard to understand or ill-defined for humans. It is therefore not surprising that often the presented solutions, i. e., ML models, are equally incomprehensible even for experts [7]. For cases were sound argumentations on quality or specific properties are needed, additional goals towards explainability can thus be desirable (see #4 [23] and Art. 13 [15]). As the terminology is not always consistent among the different documents, nor are there precise definitions in all cases, we consider (technical) explainability one aspect of transparency in the context of AI, while transparency is related to further aspects such as traceability (see #4, 7 [23] and Art. 12 [15]) and the provision of information (see #1, 2, 4, 7 [23] and Art. 13, 52 [15]). Finally, the High-level Expert Group and the European Commission have emphasized human autonomy and oversight among the most important requirements for the use of AI. While the former addresses risks stemming from the sheer fact that the automation provided by an AI application may (in some form) limit or manipulate the agency of humans (see “fundamentals rights impact analysis” [23] and Art. 5, 6 [15]), the latter aims at ensuring effective human supervision and, if necessary, intervention (see #1 [23] and Art. 14 [15]). As such, it appears that a broader conception of quality has been adopted in the context of AI, which is well captured by the term “trustworthiness”.

While a consensus is emerging on the key dimensions of trustworthy AI especially at European level, for a given AI application, practical test procedures are needed to evaluate whether the corresponding quality characteristics are actually present. Notably, the research regarding the measurement and implementation of AI trustworthiness is broad and ongoing [24], [22], [45]. However, while it is already apparent that effective measures to ensure properties such as reliability, fairness, or transparency often require access to the training data, the design process, and the representation of the output, there are still several challenges to be addressed in the concrete verification of requirements [62], [5]. For example, as individual parts of a complex ML model are typically challenging to interpret [63], quality aspects cannot be broken down (see [39] for a discussion on single component evaluations). Further challenges arise as AI applications often operate in complex environments which cannot be fully understood and captured by humans themselves (e. g., the indicators of diseases in medical data). Thus, often concise technical criteria are missing to evaluate the coverage of training and test data (see [19] for a combinatorial perspective). In open-world contexts in particular, it is typically not possible to quantify the application domain precisely. Looking further at the example of reliability, as opposed to other engineering disciplines such as construction where one can guarantee by computational proof (or reach a very high assurance level at least) that a building is statically safe – provided no huge earthquake occurs –, the verification of ML-based applications proves challenging. While promising approaches to attest the flawless functionality of an ML model are being developed (e. g., SMT-solver [35], linear relaxation [13], and branch-and-bound algorithms [61], see also [40] for a survey on verification algorithms), they are not yet sufficiently advanced to be of use for practical AI applications [5]. Additional complexity emerges in case the ML models are continuously trained during operation. Notably, even if the challenge of conducting an AI system audit was solved, still, giving a warranty appears difficult given the dynamics of ML. At the same time, however, AI systems are currently being applied to many areas of our lives whose trustworthiness must be ensured.

3 The how of trustworthy AI

As elaborated in the previous chapter, trustworthiness has been adopted as a broad notion of quality for AI applications, exceeding the classic concepts of IT security and safety. While requirements and recommendations that characterize “trustworthy AI” have been presented in numerous guidelines [34], they are, however, introduced at an abstract level and leave to a large extent open which measures or methods should best be taken to meet them [21], [43]. Overall, there is a lack of standards that would concretize requirements [11], and technical verification remains a challenge (see chapter 2). Parallelly, numerous organizations already use AI in practice and have high quality expectations for their systems. Thus, it is crucial that they take into account the imponderables caused by AI and find a systematic way of dealing with them. To this end, we observe that two fundamental perspectives must work together: i) a high technical quality of AI systems themselves is required, ii) the organization should make appropriate preparations (e. g., establish structures, processes and roles) for handling its AI applications and their development in a trustworthy manner. In the following, we refer to these two perspectives as the product and organizational perspectives.

The product perspective, detailed in Section 3.1, focuses on an individual “product”, meaning here an AI system, within its application context. The approach presented, serves to systematically evaluate the trustworthiness of an AI application with respect to the dimensions mentioned in the previous chapter. It is based on a structured analysis of risks and, moreover, involves their mitigation through mostly technical but, in parts, also non-technical measures along the lifecycle of the application. Notably, risks can extend well beyond the initial rollout, either because they were missed during conception of the system or did not even exist at the time. Such continuous or latent issues need to be addressed at the organizational level and the corresponding (organizational) superstructure is discussed in Section 3.2 (we refer to [53] and [41] for the importance of management support for achieving organizational objectives in the implementation of classic IT systems). This superstructure not only needs to reflect long-lasting risk situations, but in addition governs the general policies of the organization towards the dimensions of trustworthy AI and orchestrates general, often process-oriented, measures to maintain these policies in practice. We introduce AI management systems as potential candidates for such a superstructure and explore their role in operationalizing trustworthy AI from the organizational perspective.

3.1 Approach from the product perspective

Judging by the notion of trustworthiness endorsed by current EU-level documents, it is apparent that trustworthy AI can, in general, only be achieved by a broad analysis of both the application and its environment, accompanied by careful risk mitigation measures and checkups. However, existing test procedures for traditional IT systems cannot be straightforwardly used for AI (see chapter 2) and, moreover, do not address intangible risks related to discrimination and human autonomy, among others. For these reasons, novel technical foundations are needed for the assessment and control of AI risks. Below, we elaborate on the challenges in this regard and present an approach for the systematic evaluation of trustworthy AI, putting a particular focus on AI applications based on ML technologies.

Since the recommendations and requirements for trustworthy AI are abstract [21], [43], they must first be further operationalized in order to evaluate a specific AI application. Here, we observe three main challenges: First, various aspects of trustworthiness are not readily quantifiable by metrics or key performance indicators. For instance, it is unclear how to measure the user autonomy afforded by an AI application or the degree of its explainability. Second, typically, the concrete requirements and evaluation standards strongly depend on the application context as well as the specific AI method used. For example, while fairness is irrelevant for use cases in production plants, there are high risks in applications that support critical decisions such as algorithms for credit scoring or rating of job applicants. Following on this example, over 20 fairness metrics have been proposed [56], each of them quantifying a different aspect of discrimination. However, there is no statement by official bodies about which metric(s) should be applied for a given use case and, furthermore, which target interval to rate as acceptable. Third and lastly, disparate objectives associated with trustworthiness may conflict with one another leading to conflicting system-level requirements. One reason is that numerous requirements which reflect societal values are detached from performance-related quality requirements. For instance, a perfect predictor which violates the principle of fairness given a discrimination-inherent ground truth, creates a conflict between non-discrimination and optimizing its performance. Similarly, the prediction capabilities of an ML model may be compromised if personal information is removed from the training dataset for privacy reasons.

Interestingly, some of the aforementioned challenges can also be found, for example, in IT Security and Functional Safety, where the objectives of resistance to manipulation or malfunction often lead to very different quality criteria for different systems [58], [3]. Similar to the risk-based approaches established in these fields, we believe that the specification of trustworthiness criteria for an AI application must be based on a comprehensive risk analysis. Moreover, given the variety of methods for risk mitigation, the measures for meeting defined criteria must also be adapted to its respective AI application. Accordingly, for assuring and evaluating a trustworthy AI, we propose an approach consisting of two consecutive phases: i) top-down and ii) bottom-up. While the top-down phase serves to identify and break down risks into specific criteria for the AI system, the bottom-up phase encompasses the (likewise specific) measures taken that work towards meeting these criteria and argues for their fulfillment. In the following, we will briefly present the main steps of this approach which we have described in more detail in [49] and [50].

Figure 1

AI risk scheme. For a detailed elaboration on the dimensions, see [10], and on the risk areas, see [49].

Figure 2

Visualization of the product perspective approach. The l.h.s shows the top-down phase, the r.h.s the bottom-up phase.

As a basis for systematically analyzing and breaking down AI risks in the top-down phase, especially those stemming from the use of ML, we propose a scheme along the six dimensions presented in chapter 2 (see Figure 1). Since each dimension covers numerous aspects which must be weighted and evaluated differently depending on the application context, it is necessary to subdivide them respectively into more granular risk areas [49]. For instance, regarding privacy, different evaluation standards with respect to personal and business sensitive data may emerge. Similarly, when defining specific criteria with respect to transparency, a distinction should be made between explainability for technical experts and information of users and affected persons, for example. Finally, our proposed scheme includes dedicated risk areas “control of dynamics” to take into account risks which arise during operation such as model and data drift.

The top-down phase of our approach (see l. h. s. Figure 2) serves to specify trustworthiness criteria for measuring the control of the AI risks in the scheme presented. To increase efficiency, this phase begins with a protection requirement analysis at the dimension level, where an evaluation of the potential damage determines whether a dimension is relevant for the considered AI application. Exclusively for relevant dimensions, an in-depth analysis for each associated risk area follows. Based on the results, targets are formulated describing under which circumstances residual risks are acceptable. These targets are then translated into (preferably quantitative) criteria for the specific AI application. Let us illustrate on the example of ML-based credit scoring. Since this application has an impact on the financial capabilities of individuals, most likely fairness will be among the relevant dimensions. The in-depth risk analysis should be informed e. g., by the applicable anti-discrimination legislation, and help to identify input features which characterize potentially disadvantaged population groups. Lastly, use case-adequate fairness metrics (and target values, if applicable) are chosen, such as statistical parity and counterfactual fairness [57].

After defining the trustworthiness criteria for the AI application, the bottom-up phase of our approach assures and argues that they are fulfilled. To this end, measures (preferably technical) for risk mitigation and testing, which often need to be specifically adapted to the AI technology used, should be implemented, where applicable. We distinguish between four categories of measures, see Figure 2 (r. h. s.): i) data, ii) ML component, iii) embedding, and iv) operation. These categories, however, should not be considered separately as they represent different starting points for risk control along the life cycle. Since our approach focusses on AI applications whose functionality is typically derived directly from data, the selection and quality of data are of particular importance. Complementary, risks should be considered during the design and training procedure of the ML component included in the system (meaning the ML model and, if applicable, pre- and post-processing). Moreover, in numerous applications, the ML component operates in interaction with further, non-ML components (e. g., rule-based), which may serve for logging, monitoring or otherwise processing in- or outputs. Unlike for the ML component, conventional measures and tests can be used for this embedding. Lastly, ensuring the trustworthiness of an AI application does not end at the development stage since various levels of support and maintenance are needed during operation. This cannot, or rather should not, be addressed by technical solutions alone, but involve human oversight as well as allow for the option of human intervention in critical situations. Returning to the example of credit scoring, several measures can be employed to work towards fulfilling set target values with the fairness metrics chosen, for example bias-removing pre-processing of the training data or post-processing of the ML model outputs. Moreover, a monitoring of fairness indicators should be installed to check e. g., for potential deviations from the target values during operation.

The bottom-up phase (see r. h. s. Figure 2) encompasses the implementation of (technical as well as non-technical) measures with respect to the four abovementioned categories. Based on their documentation, a safeguarding argumentation is developed for each risk area under consideration in order to demonstrate, if possible, that the criteria from the top-down phase are met. Subsequently, a conclusion is drawn per dimension on whether the criteria are met or not. Finally, if residual risks still exist due to conflicting requirements between the dimensions for example, trade-offs should be discussed and carefully weighed prior to the final judgement about the application’s trustworthiness.

4 Conclusion

In this paper, we have outlined essential quality dimensions of trustworthy AI and argued that two different perspectives are involved in their operationalization, namely the product and organizational perspective. While the product perspective deals with the risks emerging from a specific AI application, the organizational perspective considers the handling of AI systems in view of organizational objectives and risks. For each of the perspectives, we have presented an approach for operationalizing and assuring trustworthy AI. The first approach serves to systematically evaluate an AI application from a product perspective. The second approach presents a management system specific to AI. Overall, we conclude that AI trustworthiness can only be achieved through a harmonized interplay between product and organizational perspectives since risks emerging from the product must be reflected in organizational policies as well as monitored, and in parts also mitigated by corresponding processes. In addition, as shown in both approaches presented, we believe that the implementation of concrete measures, from either perspective, should be based on a comprehensive risk analysis and specifically adapted to the respective results.

Our discussion of trustworthy AI focuses on ML-specific key dimensions for which there is a consensus in both academia and society. Furthermore, there are other important principles such as sustainability and environmental friendliness (“underrepresented compared to other ethical dimensions” [34]) which could be harmed in the use of ML, for example, due to the high energy consumption in training models. We believe that these principles are generally related to numerous non-ML-specific aspects and, thus, cannot be solely addressed from a product perspective, but other approaches are needed. For instance, environmental sustainability cannot be assessed without consideration of the resources from which electricity is generated.

Regarding the two perspectives introduced in this paper, we observe that the operationalization from the organizational perspective is further advanced since best practices are gradually emerging with respect to processes in AI development and operation (e. g., facilitating automation and tracking via MLflow, DVC, Jenkins) as well as AI management (as reflected in current standardization activities [25], [26]). In a next step, audit and certification schemes are needed, and it can be assumed that, once processes and procedures have been standardized, these schemes can be developed and applied in a similar fashion as for non-AI domains. From a product perspective, numerous testing tools are available for assessing specific trustworthiness-related characteristics in AI systems despite the challenges presented (see chapter 2). However, there is an apparent gap between the results of numerous tools and the formal requirements to deem a risk sufficiently mitigated or controlled. This gap between tools and abstract trustworthiness requirements should be addressed by future research. While testing methods need to be further developed, a promising approach for concretizing requirements is to define (analogously to the protection profiles of the Common Criteria [30]) adequate classes of AI use cases (e. g., regarding the application environment or sector) and specify criteria as well as evaluation standards in more detail for the respective classes.