Ensemble of explainable artificial intelligence predictions through discriminate regions: A model to identify COVID-19 from chest X-ray images

3.1 Dataset

Lung disease classification (COVID-19 vs Normal) is achieved by analyzing CXR images. In phase 1, we used 3,500 CXR images (class distribution is shown in Table 1) from the COVID-19 radiography database^[2] to train a base classifier, which was adapted from our previous study. Annotated images are generated using the cutting-edge XAI approach LIME in our earlier work [45], where we demonstrated how customized CNN recognized diseases in the region of interest. Furthermore, in this study, we generated annotations for further analysis using the GradCAM and Saliency approaches, and these were archived in the form of numpy arrays for further usage. The dataset will be shared as per the request and based on its use.

3.2 CNNs

Instead of computing statistical characteristics from images, a CNN is used to collect feature vectors from images. The first successful CNN [46] was constructed by Lecun for handwritten recognition; however, Fukushima introduced the concept as neocognitron in the 1980s [47]. Convolutional, max pooling, and dense layers make up deep CNNs. Convolution is a two-dimensional weight matrix calculated during training that retains spatial information. Since CNN computes more such weight matrices to extract feature maps from the image, max pooling is used to reduce dimensions and lower the cost of computing weights. Like an artificial neural network, the number of layers in the architecture, and the loss function to optimize the weights, number of epochs, and learning rate are the hyperparameters. A CNN architecture extracting the features from the image is a state-of-the-art technique that has changed the direction of research in image processing; later, it has moved to TL methods such as VGG16 [48], ResidualNet [49], InceptionNet [50], and Densely connected net [51], which are showing remarkable results.

3.3 XAI techniques

Gradient-weighted Class Activation Mapping produces a coarse localization map highlighting the essential regions in the image for predicting the target class using gradients of the target class flowing into the final convolutional layer [52]. The concept originated from the fact that fully connected layers lose spatial information, while convolutional layers retain it. When gradients go into the final convolutional layer, the information is used to assign importance to each neuron. To obtain the localization map, it first computes the gradient of the class score with respect to feature map activations of the convolutional layer and flows back with every layer of convolution. The same process is repeated at each other layer. Gradients are global averages pooled as an aggregation process as they flow back. These techniques are more trustworthy, and they were one of the successful techniques to describe visuals from CNN output.

Saliency map [53] is a ranking-based technique that ranks pixels based on their influence on the class score. Because the class score function is highly nonlinear, identifying the significance of a pixel is difficult. However, the importance of a pixel can be computed using a linear function using the first-order Taylor expansion. S c ( I ) = ‖ w ‖ T I + b , where w is the derivative of S with respect to the image I at the point (image) I 0 . w = d S c d I ∣ I 0

LIME [25] is another innovative explanation method that learns an interpretable model locally around the predictions in order to explain the predictions of any classifier in an understandable and accurate manner. Interpretable explanations must use a representation that is understandable to humans, regardless of the model’s real attributes. The output is a binary vector showing the presence or absence of a contiguous patch of pixels, via which the model can best predict the class.

There are other XAI methods such as Partial Dependency Plots (PDP), Accumulated Local Effects (ALE), Individual conditional expectation (ICE), SHAP, and ELI5. PDPs [54] illustrate the incremental impact of one or two features on the anticipated outcome of a machine-learning model. ALEs are more similar to PDP with a change in the way of computing the feature importance. It is based on differences in predictions rather than averages [55]. On the other hand, ICE plots [56] are used to assess the impact of a variable on a trained machine-learning model, assuming that all other variables remain constant. The objective of SHAP is to elucidate the prediction of an instance x by calculating the contribution of each feature to the prediction. The SHAP explanation technique [57] calculates the Shapley values using coalitional game theory. The feature values of a data instance function as participants in a coalition. Shapley values provide a method for equitably allocating the “payout” (i.e., the prediction) across the different attributes. It is inspired by a local surrogate model. ELI5 [58] is primarily designed for text data to explain the answers in a simple way.

Among all these methods, LIME, GradCAM, and Saliency mappings are visualization methods that explain images. Methods such as PDP, ALE, and ICE work on lower dimensions and structured data, for example, predicting a customer churn from customer demographical, potential to buy, and job-related features. SHAP has not been applied to any medical images so far. So we employ LIME, GradCAM, and Saliency maps to obtain the region of interest feature that is explained in the next section.

3.4 Proposed method

A collection of XAI models is created and trained to address the problem of binary classification that detects the presence or absence of an individual who has COVID-19. GradCAM, LIME, and Saliency maps are some of the XAI approaches used in this work. To produce ROIs, various XAI approaches are applied, and a CNN model is trained on each output before the final predictions are selected by a majority vote. Decisions made by a group of experts are almost always more trustworthy than decisions made by a single expert. As a result, the predictions are accurate.

The following is the summary of the process: (i) train a CNN model on given input images, (ii) share the stored model and images with XAI method to generate annotations, (iii) fusion of input images and XAI output (generated annotations) and training another custom DCNN on data generated from phase (ii), and (iv) get the predictions from the previous phase for all XAI techniques and finalize the prediction using majority voting. The entire flow is shown in Figure 1.

Figure 1

Flowchart that denotes the entire process flow.

The process is well explained in phases in Algorithm 1 to train the model, testing is in Algorithm 2, and the schematic is shown in Figure 2. Here, a DCNN with five CNN layers is designed to extract features from CXR images by optimizing the weights with an Adam optimizer and a learning rate of 0.0001. A drop-out layer is introduced to reduce overfitting at the same time.

Figure 2

Block diagram for proposed method: Ensemble XAI; DCNN: Deep convolutional neural net; F-DCNN: Fusion DCNN.

DCNN feature extractor is a binary classifier with a fully connected layer plus a sigmoid activation function that outputs the probability. The trained model, along with images given as input to the XAI method, produces the described region. These discriminated regions can be given more weight while combining image features with Fusion DCNN, which has a similar architecture. Since various models produce different probabilities, stacking is performed on the output with a mode of voting.

The time complexity for the proposed approach is estimated in a model-specific way [59] as per the architecture in a total of three parts. One is training the base model before applying XAI methods. Second is generating rois from model and XAI methods. Third is training on FDs. The first and third used the same architecture with the goal of capturing the features whatever is not captured before fusion. It can be defined [60] in the following way:

where “m” and “n” denote the size of features, “f” is the number of kernels, “s” is the size of the kernel, and “d” denotes the depth of the network. The second part again consists of three parts since the XAI methods are different. For simplification k 1 , k 2 , and k 3 denote the time taken by three XAI methods, respectively, to produce a region of interest. “k” is defined as:

and then, the summation is defined as follows:

In the aforementioned notation (2), the “Model” is one of the components, because the image and model will be passed to the XAI method. XAI method time depends on the architecture of the model as well. “x” is the number of XAI methods: in this case, it is three. The total time is the sum of (1) and (3). We have also measured the time for an epoch in each of these XAI methods. It excludes read/write intermediate results.

4.1 Ensemble of XAI methods

In the proposed method, COVID-19 and Normal CXR images of 224*224 size have been taken as input. Experiments were conducted on google collab^[3]. In all the experiments, DCNN architecture as shown in Figure 4 was used, where the input layer has a dimension of 224*224.

Figure 4

Layer-wise detail of the DCNN model.

Once the DCNN model is trained on COVID-19 and Normal classes, images and model were fed to the XAI technique, for example, GradCAM, through which annotated images were generated. The annotated images generated using GradCAM and Saliency maps are preprocessed further with Otsu’s thresholding method to get the segments. Otsu is a segmentation algorithm that generates lower and upper thresholds to segment the image based on the continuity of pixels.^[4] Similarly, LIME is another XAI model applied to this problem [45]. When we use LIME, the feature weights will vary between 0.0001 and 0.00000001. Fusion is a simple process of the addition of pixels, which intuitively gives more weight to the important regions though it does not show much interpretation when we plot the fusion image. Of course, it can be done through normalization but it does not convey anything to the human eye. Three XAI techniques (GradCAM, LIME, and Saliency maps) were used during experimentation to generate critical regions. Each generates local discriminate regions that are nothing but masked images and were used further to train a model along with original data. Error curves for these are shown in Figure 5.

Figure 5

Error curves: a–c are generated while training on the GradCAM, Saliency, and LIME FDSs.

The results reported in Table 2 for an ensemble of XAI are predicting all positive cases correctly without missing anything although the proposed method is on par with individual models in other metrics. The number of XAI methods in ensembles can be 3, 5, 7, etc. any odd number that avoids a tie between the models.

The experiments were conducted on Google Colab with default setting (12.7 GB RAM,107.7GB disk space), and it took an average of 12.56, 7.59, and 12.68 s for GradCAM, Saliency, and LIME per each epoch, respectively, whereas the base model took 3.86 s. The average time period here is for the number of times the experiment (10 runs) was conducted.

4.2 Correlation analysis with TL techniques

TL [61] has been applied to many real-world problems to address the following: (i) insufficient labeled data, (ii) incompatible computational power, and (iii) distribution mismatch. TL methods are DCNNs trained on a huge database for a classification problem. Since it is deep, the network can be used to extract features from any given image instead of images from the same distribution. This has hinged on the researchers training another network by taking the output feature vectors from DCNN, which we say it as TL method. Here, we are using the weights of the pre-trained network to generate the feature vectors. Various pre-trained networks such as VGG16 [48], ResNet [49], MobileNet [62], Xception [63], EfficientNet [64], DenseNet [51], Inception [50], and ConvNext [65] are available with different architectures and can be reused with imagenet pre-trained weights. These architectures had a difference in their design of architecture. For example, ResNet uses residual blocks, XceptionNet uses depth-wise separable convolutions, and DenseNet has dense blocks. A common concept among these is that they are built on top of convolutional layers but with different filter sizes. The recent ConvNext architecture is trained on Resnet based on the vision transformers concept. In this work, the authors have used some of the TL methods by restricting themselves to one version from every family of TL methods. The experiments were conducted on the FDs. A TL method is trained on FDs with binary cross-entropy loss, Adam optimizer for 50 epochs on every XAI-generated output.

TL methods have also been applied to the original dataset and recorded as base results to compare with the results in Table 3. The metrics are reported in Table 4. One can compare these results with the average metric computed across all the TL methods built on FDs, which is slightly higher than the individual models on the original data. It will be efficient if we compare it at the individual model level. However, the panel of the TL model’s decisions is much more reliable than the base one. This analysis can be done by comparing with the results from Table 2 and claim that the ensemble of customized model results is consistent.

Some of the assumptions were made while training all the networks for standardization purposes. Every network is trained for 50 epochs only with a learning rate of 0.001, an optimizer of Adam, and a patience of two after several trials. No other parameter tuning was done to improve the results or avoid overfitting. With the approach adopted, we can claim that some of the networks are weak classifiers that may tend to do well on a few data points when compared to others, which is actually the concept of ensembles. Some of the error curves are shown in Figure 6.

Figure 6

Error curves: a–c are generated while training on the GradCAM FD; d–f are generated while training on the Saliency FD. (a) GradCAM-VGG16, (b) GradCAM-Resnet, (c) GradCAM-MobileNet, (d) Saliency-VGG16, (e) Saliency-ResNet, and (f) Saliency-Mobilenet.

4.3 Handling CI

One of the major problems in machine learning is CI. CI is defined as there will be fewer samples ( < 10 % ) of the total when compared to the opposed class. There are a number of methods available to address CI on structured data such as under-sampling, over-sampling, and generation of additional data. However, there is no method to deal with images. A method that we have implemented on structure data [66] experimented on images is able to obtain significant results. The detailed architecture is shown in Figure 7.

Figure 7

A multi-stage approach to handle CI on images [66].

The images are sampled to create CI data with 100 images from COVID-19 and 900 from Normal class with the intention of creating a 90:10 ratio. Here, the minority class samples are from the COVID-19 class; in general, positive samples are rare for any disease. The experiments are conducted in the following way: (i) trained a customized model on the sample dataset with the parameters stated earlier, (ii) created mutual disjoint datasets (MDS) with repeated sampling from the majority class while combining with minority class samples, (iii) trained individual models on each MDS and saved the predictions, (iv) consolidated the predictions from all MDS with two-stage voting for minority class samples, and (v) comparison of the results before and after applying the method. The experiments have a choice of choosing models between customized models and TL methods. However, TL method is a good choice as there are fewer images. The results were recorded during the training original dataset, and MDS is given in Table 5.

This dataset has less than 10% of samples in the COVID-19 class. When creating MDS, it has been brought to 40% of the total with the undersampling method. The experiments were conducted on the original dataset before creating MDS and can be compared with the results generated from MDS. We can observe that all methods gave results on par with the based metric. Among TL methods, VGG16 has received low scores. However, it has performed well on test data using the proposed method.