GraphXCOVID: Explainable deep graph diffusion pseudo-Labelling for identifying COVID-19 on chest X-rays

2.1. Chest X-ray classification for COVID-19

The task of classifying CXRs has been widely investigated in the community. The go-to paradigm to address this problem is deep learning e.g. [27], [28], [29], [30]. The fast development of these techniques has been motivated by the release of several benchmarking datasets including ChestXray14 [27] and CheXpert [29], in which a large number of annotated samples from different pathologies (classes) are contained in each dataset. With new diseases such as COVID-19, where the annotated data is limited, one needs to rethink the whole design of the techniques. In this work, we motivate the power of semi-supervised learning for COVID-19. In what follows, we review related COVID-19 and SSL techniques.

The bulk of literature addressing the task of classifying CXRs for COVID-19 is largely based on deep supervised learning e.g.[9], [13], [14], [15], [16], [18], [19], [31], [32], [33], and several techniques arise every single day. Most of approaches apply pre-trained off-the-shelf networks; in which diverse generic architectures, including ResNet [34], DenseNet [35] and VGG [36]. Existing works thus leverage on fine tuned networks and networks trained from scratch on CXRs data. However, there are fundamental differences between natural and medical image classification including features and data size. Hence, as shown in [24], transfer learning might offer little benefit to performance due to the over-parameterisation of standard models. Moreover, the samples available for COVID-19 are scarse in comparison to other type of pneumonia, and one needs to deal with highly imbalanced datasets.

The key assumption of supervised techniques is a well-representative labelled dataset, and in new diseases, such as COVID-19, this core assumption is a strong one. Moreover, the available annotations might be far from being a definite expression of ground truth [37]. Therefore, by using deep supervised learning techniques are prone to labelling error and uncertainty that adversely affect the classification output. Although transfer learning [38] and Generative Adversarial Networks [39] mitigate, at some level, the lack of a large and representative dataset, they weakly account for the mismatch between expert annotation and ground truth annotation, which is generated by human bias and uncertainty, and the performance can be limited due to the differences among datasets as demonstrated in [24].

Motivated by the above-mentioned drawbacks in deep supervised learning and with the goal of generating a high sensitivity technique that largely decreases the need for large annotated set, we introduce a deep semi-supervised technique for the application to COVID-19 identification, which to our knowledge it is the first in its type. In the next subsection we discuss recent techniques in deep SSL for other domains and how they differ to ours.

2.2. Semi-Supervised classification for medical images

Semi-supervised learning has been applied in the medical domain since its early developments, in which model based techniques have been the main focus of attention e.g. [40], [41], [42], [43], [44]. These approaches have demonstrated the potential of SSL- but the generalisation of the feature space along with the computational requirements have raised limitations. Recently, with the advent of deep learning, deep semi-supervised learning has been investigated.

In the past few years, there has been a silent revolution in deep SSL techniques, e.g. [21], [22], [23], [45], that have sought to combine the theoretical underpinning of SSL [46] with the generalisation and feature extraction of deep neural networks.

The largest trend for new deep SSL methods involves consistency regularisation [20], [21], [22], [23]. The core idea of this perspective is to, for both labelled and unlabelled examples, induce perturbations $\delta$ and then add a regularisation term to the loss function such that the prediction of the model is invariant to the perturbation, i.e. $f(x+\delta )=f\left(x\right)$. The main variety between approaches stem from how to generate perturbations $\delta$. The definition of perturbations is indeed a complex problem, and no work has ever evaluated the performance of consistency based approaches outside the natural image domain.

To the best of our knowledge, there has been no deep SSL approach proposed for COVID-19 analysis. The potential performance of SSL has nevertheless been shown in our prior work [25] on CXRs analysis. We readily competed with SOTA supervised techniques for identification of several pathologies in CXRs, using a small fraction of the available labels. In this work, we extend this framework to build upon the concept of pseudo-labels, first introduced in [26]. However, compared to that initial work, there are now several key differences since a graph based approach is considered and the pseudo-labels are predicted from our diffusion model rather than the deep net.

3.1. Feature extraction & graph construction

The most common data representation is a Euclidean or grid-like structure. We rather define our data in a non-Euclidean domain with a graph. This framework offers different benefits including mathematical properties such as sparseness which allows for fast computation, and the ability to correct initially mislabelled data by smoothing the embeddings. We represent the dataset $X$ as a graph, where each node is an image, to produce pseudo-labels. Then, unlike pure model-based approaches or pure deep learning techniques, we introduce a hybrid model – that is, a combination between a model-based (energy model) and a deep learning framework.

A deep network $f_{\theta }$ is considered for updating the pseudo-labels generated by our optimisation model. It is initialised from the tiny labelled set $(x,y)\sim D_L$ by minimising:

where the loss function ${\ell }_H$ is cross entropy, which is the most common choice for classification tasks. This optimisation process only involves the provided small labelled set to construct the initial graph (i.e. it is run once as initialisation).

More precisely, a given set of data (or features) can be represented as an undirected weighted graph by $\mathcal{G}=(\mathcal{V},\mathcal{E},\mathcal{W})$ composed of $n$ nodes $\mathcal{V}=\{v_1,\dots ,v_n\}$, which are connected by edges $\mathcal{E}=\{\{v_i,v_j\}:v_i,v_j\in \mathcal{V}\}$ with weights $w_{ij}=S(i,j)\ge 0$ that correspond to some similarity measure $S$ between the features of nodes $i\in \mathcal{V}$ and $j\in \mathcal{V}$, and $w_{ij}=0$ if $(i,j)\notin \mathcal{E}$. In our setting, a node $v_i$ represents an image of the set $x_i\in X$. A fundamental question when using graph based approaches is how to effectively extract representative features to construct a robust graph. In order to avoid the sensitive task of hand-crafted feature selection, we rely on embeddings automatically produced by (2). Hence we reduce the potentially large generalisation error that appears when extracting hand-crafted features on a small training set $X_L$.

We consider as feature extractor the function ${\phi }_{\theta }:\mathcal{X}\to {\mathbb{R}}^P$, given by the bottleneck of the current network $f_{\theta }$, that maps the input to some feature space of dimension $P$. To construct our graph, we compute, for each sample, the set of descriptors by ${\mathbf{c}}_{\mathbf{i}}={\phi }_{\theta }\left(x_i\right)\in {\mathbb{R}}^P$ for $x_i\in X$, and connections are created through the k nearest neighbours (k-NN) approach in ${\mathbb{R}}^P$. An illustration of extracted features is given in Fig. 2 .

Notice that the model $f_{\theta }$ and the feature extractor ${\phi }_{\theta }$ are updated (see Section 3.3) through the exposition to pseudo-labels of unlabelled data. The graph thus evolves along our process. We now detail how pseudo-labels are obtained on the graph by following a transductive strategy.

3.2. Label diffusion as pseudo-Labelling

Pseudo-labels are estimated through a diffusion process on the whole graph containing both labeled and unlabelled examples. For each image, our method also assigns a score reflecting the uncertainty of the produced pseudo-label (see Section 3.3). To do this, we first generate the pseudo-labels through an optimisation model based on the normalised graph p-Dirichlet energy expressed as:

where the degree of the node $i$ is denoted by $d_i$, and the weights are computed from the descriptors described in previous section. The minimisation of this energy allows the diffusion of a labelling variable $u$. Whilst techniques in this line have been reported in the medical domain e.g.[41], [42], [47] and in the pure machine learning community e.g. [48], [49], they only seek to use eigenfunctions of a normalised Dirichlet energy based on the graph Laplacian for $p=2$ or only approximate $p\to 1$. However, later machine learning works [50] has demonstrated that using the non smooth $p=1$ Dirichlet energy (related to total variation) achieves better performance for label propagation.

With this motivation in mind, we introduce an optimisation model based on the normalised and non smooth $p=1$ Dirichlet energy. In the case $p=1$, model (3) is thus ${\Delta }_1\left(u\right)=\left|W{D}^{-1}u\right|$, where $D$ is the diagonal matrix containing the degrees $d_i$ and the $m\times n$ matrix $W$ encodes the $m$ edges in the graph. Each of these edges is represented on a different line $i$ of the matrix $W$, with the value $w_{ij}$ (resp. $-w_{ij}$) on the column $i$ (resp. $j$).

We now detail our multi-class model that will be applied to $L=3$ classes: “0: Healthy”, “1: Pneumonia” and “2: COVID-19”. For each class $k=1\cdots L$, we set a variable $u^k$ that contains the node values for class $k$ and denote $\mathbf{u}=[u^1,\cdots u^L]$. For unlabelled nodes $i>l$, we couple the $L$ variables with the constraint: ${\sum }_{k=1}^L{u}_i^k=0,\forall i>l$. We make the standard assumption that there exists a non empty set of labelled nodes ${\mathcal{I}}_k\subset \{1\cdots l\}$ for each class $k$. For these nodes, we set $u_i^k\ge ϵ$ if $i\in {\mathcal{I}}_k$ (positive response for the class), and $u_i^{k^{\prime }}\le -ϵ$ if $i\in {\mathcal{I}}_k$ and $k^{\prime }\ne k$ (negative output for the other classes).

Under such constraints, we seek to minimise the multi-class functional [51] that contains the sum of normalised ratios :

We consider an iterative scheme to optimise this problem:

where $t$ is a time index associated to the step $\Delta t>0$. This process diffuses information from labelled nodes to unlabelled ones. To avoid trivial solutions [52], [53], we apply shifting $u^{k,(t+1)}=u^{k,(t+1)}-\text{median}\left(u^{k,(t+1)}\right)$ and normalisation ${\mathbf{u}}^{(t+1)}={\mathbf{u}}^{(t+1)}/\parallel {\mathbf{u}}^{(t+1)}\parallel$ steps at the end of each iteration.

From (5), one can see that the solution $u^{k,(t+1)}$ satisfies:

so that we get a reduction of the normalised ratio ${\Delta }_1\left(u^{k,\left(t\right)}\right)/\left|u^{k,\left(t\right)}\right|$ along iterations $k$. When $L=1$, the scheme $u^{k,\left(t\right)}$ converges to a local minima of (4) which corresponds to a bivalued function that naturally segments the graph [53]. In the general case $L>1$, the convergence to a local minima of (4) can be ensured by using a modification of the scheme (5) as proposed in [54]. However, such adaptation comes at the cost of an important additional computational cost. Even of there is no theoretical guarantee for that, we observe a monotonous decrease of (4) with the scheme (5). As a consequence, we suggest to use the flow (5) that presents an acceptable trade-off between theoretical and practical aspects.

Once ${\mathbf{u}}^k$ has converged to some ${\mathbf{u}}^{*}=[u^{*,1},\cdots ,u^{*,L}]$, the label of each node is finally given by ${\widehat{y}}_i=argma{x}_j{u}_i^{*,j}$. In practice, our model (5) is solved using an accelerated primal dual algorithm [55]. Our generated pseudo-labels are denoted ${\widehat{Y}}_U={\left\{{\widehat{y}}_k\right\}}_{k=l+1}^n$ and used to update the classification network $f_{\theta }$ as explained in the next section.

We remark that there are several differences between our pseudo-label approach and that of [25], [26]. Firstly and unlike [25], our work generates and updates the embeddings from a deep net to construct, since the beginning, a stronger graph. Secondly, our optimisation model generates pseudo-labels, outside the deep net, and then they are iteratively updated from a deep net. Thirdly, our current model is designed to generate highly certain pseudo-labels since early stages by integrating an uncertainty measure, and also is equipped with a class balance term (see Section 3.3). With respect to [26], our model follows different principle when generating the pseudo-labels. Firstly, unlike [26], our technique is a graph based model. Secondly,  [26] generates the pseudo-labels directly from the deep net (i.e. inside the network) by taking maximum predicted probability from the network. On the other hand, we generate the pseudo-labels from our new optimisation model (drawn from the normalised graph p-Dirichlet energy) which is decoupled from the network (i.e. outside the network).

3.3. Deep graph pseudo-Labelling update

Although our graph diffusion model (5) generates relevant pseudo-labels [25], one can further decrease the uncertainty over time. This is one of the major problem in pseudo-labelling as naive pseudo-labels are limited by the confirmation bias. This can be addressed if we consider the two major bottlenecks in real-world problems. A first prevalent issue in the medical domain is to face highly imbalanced class samples. As illustrated in Fig. 3 , this is particularly true in the COVIDx dataset. The second one is to deal with inferred pseudo-labels with different levels of uncertainty. As it has been shown in several works e.g. [56], [57], [58], [59], these problems can be mitigated by weighting the importance over the inferred pseudo-labels and the classes, and update the pseudo-label certainty in an iterative fashion.

The problem of classifying with a highly imbalanced dataset has been widely studied in the literature, e.g. [56], [60]. We apply a common strategy for imbalanced class population [56], [61] and add a weighting factor inversely proportional to the effective number of samples for class $k:{\omega }_k\propto 1/E_n,n\in {\mathbb{Z}}_{>0}$ where $E_n$ is the total number of samples. For the second problem, we associate an uncertainty weighing factor, ${\upsilon }_i$, to each $u_{{\widehat{y}}_i}$ generated in the diffusion process. We use entropy as the measure for uncertainty [58], [62], [63], given by ${\upsilon }_i=1-(H\left(u_{{\widehat{y}}_i}\right)/\log \left(L\right))$, where $H$ refers to the entropy and $u_{{\widehat{y}}_i}$ is normalised beforehand with respect to the values in $u_i^{*}$.

We finally define the main driving optimisation (i.e. estimation of network parameters) as:

the loss in (7) is connected with model (1) but unlike the typical consistency loss, for ${\mathcal{L}}_{\mathcal{U}}$, we are using the philosophy of pseudo-labels which are generated by our diffusion model.

Let us summarise the overall process. We first optimise (2) for a set of epochs, this serves for extracting the embeddings from the deep net to construct a graph. We then perform (5) to diffuse the small labelled set to the unlabelled data. The output of this process is the generation of pseudo-labels for the unlabelled set. These pseudo-labels are then used to optimise (7), which in turn updates the model parameters. The whole process (feature extraction, graph update, pseudo label diffusion, network update) is then iterated.

4.1. Dataset description

We evaluate our approach on the COVIDx Dataset which is a multi-center dataset introduced in [9]. The dataset is composed of a total of 15,254 CXR images. The official partition only considers 13,975 CXR images across 13,870 patients, with a training set composed of 13,675 images and a test of 300 (1579 for the test set on the full dataset). This dataset is, up to our knowledge, the largest and most diverse one for COVID-19; as the samples are from different locations and acquired under different conditions and vendors. COVIDx indeed merges five highly diverse datasets repositories: COVID-19 image data collection [8], Actualmed COVID-19 Chest X-ray Dataset Initiative [64], COVID-19 Chest X-ray Dataset Initiative [65], RSNA Pneumonia Challenge dataset  [27], [66] and COVID-19 Radiography Database [67].

The COVIDx dataset contains threes classes: Healthy, Pneumonia and COVID-19. The class breakdown, for the full and official partition CXR images, is illustrated in Fig. 3. As it can be observed from these plots, it is a highly imbalanced dataset, in which COVID-19 samples are much smaller than for the other other two classes to further description in the dataset see [9].

Moreover, to further support the performance of our technique, we use an external dataset to evaluate the generalisation to out-of-distribution samples. To do this, we use the external dataset BIMCV-COVID19 [68], which is composed of images from 11 hospitals from the Valencian Region, Spain. We randomly selected a subset of 200 patient-level samples covering all hospitals, where 75% were COVID-19 confirmed cases as one is interested in show generalisation for the target disease.

4.2. Evaluation methodology & implementation details

We validate our proposed technique as follows. First we evaluate the performance of our approach compared to supervised techniques including the leading fully supervised paper in the field COVID-Net [9]. These comparisons include VGG-16 [36], ResNet-18 and ResNet-50 [34], InceptionV3 [69] and DenseNet-121 [35]. The selection of these architectures follows the same line of motivation as in [9]: they offer a clear advantage for dealing with the unique traits of COVID. Our experiments are then conducted using: i) the official partition, in which the test set is 300 samples split evenly across the classes; ii) the full COVIDx dataset, in which the main differences with respect i) is that the test set is composed of 1579 samples (see Fig. 3); and iii) an additional random partition. We ran all the experiments under the same conditions, and followed standard pre-processing protocol to normalise the images to have zero mean and unit variance. The images we resized to the resolution 480 $\times$ 480.

The evaluation is addressed from both qualitative and quantitative points of view. The former is based on visual outputs of our classification. The latter present the per class computation of sensitivity, positive predictive value and F1-scores. The overall performance is computed in terms of accuracy and error rate. Furthermore, for sake of completeness and guided by the field of estimation statistics, we report along with the error rate the confidence intervals (95%) of all techniques. Finally, we performed a data ablation study of our SSL method by using $10\%,20\%\text{and}30\%$ of labels.

We now give implementation details. For the COVID-Net [9] technique, we used the implementation and parameters provided by the authors. In particular, we considered the latest suggested model COVIDNet-CXR3-B. For the compared techniques, we used weight decay= 5e-4, momentum= 0.9 and learning rate 1e-2 (1e-3 for [26], and ResNet-18 for a fair comparison). For our technique, the k-NN neighborhood graph has been built with $k=50$. A ResNet-18 architecture has been used for the deep network $f_{\theta }$. In practice, we used a total number of epochs of 210 with and a weight decay of $2\times {10}^{-4}$ and learning rate was set to 5e-2 decreasing with cosine annealing. Furthermore, we follow standard protocol in semi-supervised learning to report our results, we randomly select the labelled samples over five repeated times, that is- one has five different splits. We then report the mean error over the splits. All techniques were implemented in PyTorch and using Stochastic Gradient Descent (SGD) as optimiser.

4.3. Results & discussion

We begin by evaluating the different methods on the official COVIDx partition. As a baseline comparison, we consider six supervised techniques: VGG-16 [36], ResNet-18 and ResNet-50 [34], InceptionV3 [69], DenseNet-121 [35] and COVID-Net [9]. To the best of our knowledge, there exists no semi-supervised technique dedicated to COVID-19 identification that we can compare. Hence, for the sake of fairness, we also adapted one semi-supervised technique, Pseudo-Labelling [26], that has a philosophy close to ours but builds on different principles including a minimum entropy criterion. The supervised methods use the full training set, whereas the SSL techniques only consider 30% of the labelled set.

We provide a detailed quantitative analysis to understand the performance of the different techniques. The per class metrics across the official data partition are thus reported in Table 1 . Concerning positive predictive values, we observe that our GraphXCOVID approach performs the best for healthy and pneumonia classes and it readily competes with COVID-Net in the COVID-19 class. Due to design limitations, VGG-16 performed the worst, whereas ResNet-50 presents performances closer to GraphXCOVID thanks to its residual architecture. We also observe that excepting for the COVID-19 class, InceptionV3 and DenseNet-121 both sightly outperform ResNet-50. In order to show the robustness of our technique, we also considered the full COVIDx dataset, a scenario closer to a real medical setting. We compared our method with the supervised approach of COVID-Net, the second better method on the official partition. As reported in Table 2 , GraphXCOVID here performs better for all classes and all considered metrics, while only using 30% of the labelled set.

Table 1.

Numerical comparison of our technique vs fully supervised approaches. The results report per class metrics, including sensitivity, positive predictive value and F1-scores along with the overall accuracy. Our technique readily competes with all supervised techniques whilst using far less labelled data. $†$ denotes the score reported in [9]. The references cited in this table are [9], [26], [34], [35], [36], [69].

Open in a new tab

Table 2.

Performance comparison of COVID-Net and our technique using the full dataset.

Open in a new tab

The second evaluation is done in terms of sensitivity. As shown in tables 1 and 2, GraphXCOVID reports the highest values for pneumonia and COVID-19. For COVID-19, the true positive proportion is significantly higher for our method (0.94) and COVID-Net (0.91) than for other ones ($\le$0.88). This observation is confirmed with the sensitivity results on the highly imbalanced full dataset (see Table 2).

To give a view of the relative performance of all techniques, the F1-scores are respectively reported in Tables 1 and 2 for the official and full partitions. It still underlines that VGG-16, ResNet-50, DenseNet-121 and InceptionV3 are not sufficiently competitive, whereas COVID-Net and our GraphXCOVID technique are readily performing in a similar level. However, looking deeper into the performance of these two last techniques, we observe that within a highly imbalanced scenario (see Table 2), our approach outperforms COVID-Net.

We also compare our technique to a SSL technique, with similar philosophy than ours but different in design, Pseudo-Labelling [26]. This technique generates the pseudo-labels directly from the network whilst our approach considers labels coming from the diffusion model. From Table 1, one can observe that GraphXCOVID offers a substantial improvement over Pseudo-Labelling for all metrics. Overall, we achieve better accuracy with an improvement of  8%, and reduce the error rate ($\pm 1.20$CI) by more than half as displayed in Fig 4 . The improvement comes from two parts. The first major benefit is related to pseudo-labels generation. The work of [26] provides naive pseudo-labels with the network itself, while our technique generates more certain ones that are iteratively updated using both our diffusion models and the network, along with a uncertainty weight. Secondly, our technique also accounts for imbalanced class distribution.

To further support the previous results and give a global performance view, we compute the error rate and the confidence intervals for each model. The results are reported in Fig. 4 for the official and a random partition of this set. For both experiments, VGG-16 reported the worst performance followed by ResNet-18. Our model performed the best among all the compared models, reporting an error of $5.4\pm 1.1$ at the 95% confidence level. As for the other criteria, COVID-Net ranked second, reporting an error of the model of $6.7\pm 1.23$ at the 95% confidence level. One can also observe from the top of Fig. 4 that supervised techniques are highly variable at the change of partition. Such approaches are indeed heavily reliant on the training set being well-representative and balance. In comparison, the variation is negligible with our SSL approach.

In order to analyse the robustness of our model, we run an ablation study for different label counts. In Fig. 5 (top), we present the error rates and confidence intervals obtained by GraphXCOVID with 10%, 20% and 30% of the available labels. Why do not increase the percentage of labels? First, we want to use the lowest possible number of labels. Secondly, we seek to keep the advantage of transductive inference. Indeed, as pointed out in early works e.g. [70], the benefit of a transductive model is decreased when a large number of labels is considered. This effect was observed in our experiments. We finally illustrate in Fig. 5 (bottom), the behaviour of our model along iterations. Around 200 epochs were required to reach a stable error rate when considering 30% of labels.

A visual illustration of the result is next provided in Fig. 6 , where the probability scores of our technique are reported and compared with the human prediction (GT). One can see that the obtained classifier $f_{\theta }$ easily differentiate between classes. However, the probability scores (from Fig. 6) are not enough to assist the radiologist in making the decision. To accommodate with this issue, we use a Gradient-weighted Class Activation Mapping [71] type solution to highlight abnormal and normal areas in the lungs, in which Pneumonia and COVID-19 are linked to abnormally regions. Samples outputs of the attention maps are displayed in Fig. 7 and compared with the human prediction (GT). The attention maps aims to accommodate with the mental model on how the radiologists work in a clinical scenario. Therefore, we project the attention in the lungs areas as the diseases affect this region by increasing the density. This tool is designed to help, in a friendly user-interface, the radiologist judging whether the diagnostic is correct or not, and in consequence to accelerate the decision. Additionally in Fig. 8 , we also display some misclassified samples. The intuition behind these cases is as follows. Firstly, the inherent complex appearance of the pathologies projected in the chest X-ray imposes a challenge in the predictive capability of the model. Secondly, the outcome is also effected by the difference in acquisition protocol and vendor machines, which introduce artifacts, blurry effects and noise in the chest x-ray images. This is translated to a tail distribution problem that affects the model’s capability to make predictions. However, we remark that the generalisation error is reduced with the proposed pseudo-labelling and uncertainty mechanism. In particular, we reported less missing cases of COVID than the compared techniques (see the per class result in Tables 1 and 2), and our model globally performs better than the compared techniques including when handling external data (see Fig. 12 along with the discussion).

Ablation Study. Finally and to further support the design of our technique, we performed an ablation study regarding the influence of the weighting factors, $\upsilon$ and $\omega$, in our model. We first report a performance comparison of our technique, in terms of error rate and using 30% of labelled data, when considering one, both and none of the two factors. The results are reported at the left side of Fig. 9 . We notice that removing both factors (red bar) increases more than twice the error. In contrast, we observe that a substantial decrease in the error rate is achieved when considering both factors (blue bar). In a closer look at the effect of each factor, we observed that while both factors (green and orange bars) indeed improve the performance, the factor with more influence is the uncertainty mechanism ($\upsilon$, see orange bar). We suggest that measuring the uncertainty of the pseudo-labels prevents from obtaining certain incorrect pseudo-labels in early training stages, that are propagated in the next epochs. This effect is illustrated in the right side of Fig. 9. In this illustration, we display $\upsilon$ for all the unlabelled samples, and plots reflect the comparison of the pseudo-label prediction with respect of the ground truth. The red and green areas denote the incorrect and correct pseudo-labels (wrt the ground truth). From this figure, we can observe that the certainty in the pseudo-labels is improved as the epochs evolve. This plot support the strength of $\upsilon$ in our model.

Additionally, we performed another ablation study to evaluate the performance of our technique with different amounts of labelled data data in the training set. This is illustrated in Fig. 10 . We observed that the results using 30% of the labelled set is a good trade-off between number of labels vs performance due to the transductivity behaviour. In line with earlier works in transduction e.g.[70], [72], a significant increase in performance is gained from an initial increase in the labelled percentage, from 5% to 30%. After $30\%$, the introduction of additional labelled data provides a tiny amount of extra accuracy. This highlights the success of our approach in its core aim, that is using semi-supervised learning to alleviate the need for large labelled datasets for COVID X-ray classification.

Moreover, we compared our technique (with and without our weighting factors) against two recent deep SSL approaches: MT  [45] and GCN [22]. We considered the same label count of 30% and used the parameters suggested in these works. The results are reported in Fig. 11 . We observe that our technique (with the weighing factors) performs the best. MT competes readily with the basic version of our model, i.e. without uncertainty and balance parameters. Finally, GCN was clearly outperformed by all compared models. The intuition behind our performance gain is three-fold. Firstly, our proposed optimisation model itself enforces the inherent relation between the unlabelled and labelled data. Secondly, the generation of accurate pseudo-labels with uncertainty estimation increases generalisation. Thirdly, our approach better deals with imbalance datasets where the number of COVID images makes up a tiny fraction of the data.

As a final set of experiments, we test the generalisation capability of our technique, and COVIDNet, to external datasets by including a performance comparison using the BIMCV-COVID19 [68] dataset (see Subsection 4.1. for details on the experiment). The results are displayed in Fig. 12 and show the error rate at the 95% confidence level. From the plots, we can observe that COVIDNet exhibits a substantial decrease in performance on the external dataset whilst our technique is more robust in this regard. Particularly, a strong degradation was observed in the COVID-19 class (see left side plot in Fig. 12). This is an expected behaviour of deep learning models, and is particularly noticed in the medical domain e.g. [73], as one assumes that the testing set should have similar distribution to the training set. Why is our model more robust to external data? Our technique has been carefully designed to mitigate, at some level, external distributions. This is achieved by discarding irrelevant samples with low confidence scores. More precisely, GraphXCOVID controls the predictive uncertainty on the generated pseudo-labels, which narrows the discrepancy between the external distribution samples. Contrary, COVIDNet is not equipped with any mechanism to accommodate with external distributions.

From the aforementioned findings, we emphasise a central message: the strong performance when using far less labelled data than the compared techniques is a core strength of our SSL technique. Moreover, we also underline the robustness of our technique when handling external data. We highlight the value of the vast available unlabelled data in medical domain, and in particular the potentials and benefits for diagnostic COVID-19 disease.