Handling class imbalance in COVID-19 chest X-ray images classification: Using SMOTE and weighted loss

1. Introduction

Nowadays, coronavirus is the most dreadful disease worldwide. This pandemic started in Wuhan, China, in December 2019. Since then, it has been rapidly spreading all over the globe. Recently, the World Health Organization (WHO) reported that there had been 119,960,700 confirmed coronavirus cases worldwide, including 2,656,822 deaths in less than a year [1]. It is shown in Fig. 1 that these numbers are exponentially growing every day, making the world’s first purpose is to reduce the number of contaminated individuals. Therefore, many strategies have been taken worldwide, including isolating infected patients, curfew, and lockdown. The International Committee for the Taxonomy of Viruses (ICTV) named the virus responsible for this disease SARS-CoV-2 [2]. COVID-19 is the name of the illness caused by SARS-CoV-2 [3]. This virus possesses powerful pathogenicity and transmissibility. It mainly attacks the human respiratory system and brings on what clinicians call a respiratory tract infection [4]. For the most part, coronavirus spreads through person-to-person contact. The contamination starts with droplets from an infected person (cough, sneeze, or breath) ejected into the air or a surface that a healthy individual could breathe or touch, then touches his mouth, nose, or eyes [5].

Thus, the virus can reach the respiratory system. The spectrum of COVID-19 ranges from asymptomatic to severe, often fatal infection. Severe infections occur more frequently in males and patients with chronic diseases such as diabetes, hypertension, heart disease, and immunocompromised states [6].

This pandemic and the resulting lockdown have had a detrimental impact on several vital domains such as education, entertainment, the economy, and especially the healthcare sector, which is at the epicenter of the COVID-19 pandemic [4]. Indeed, healthcare systems across the globe have undergone unprecedented crises. It is mainly due to unpredictable and massive health challenges, including hospitals’ overloading with COVID-19 patients. Therefore, the urgent mobilization of resources was required. Unfortunately, these systems were not designed to manage such a critical situation. The high medical expenses, the deficiency of protective equipment, and the shortages of ICU beds and ventilators highlighted the defects in the delivery of patient care. Furthermore, medical staff contamination risk is one of the significant vulnerabilities of healthcare systems worldwide. Given that most healthcare workers have to work on-site, many procedures have been employed to protect them, including the early deployment of viral testing for asymptomatic and/or frontline healthcare staff [7]. The early detection and isolation of infected people have become the ultimate solutions to the issues above.

The diagnosis of COVID-19 disease is often performed using a Reverse Transcription Polymerase chain reaction (RT-PCR) approach that consists of taking respiratory specimens for testing [8]. Despite the widespread use of the RT-PCR test for the clinical screening of COVID-19 patients, this technique is reported as complicated, time-consuming, and requires human intervention with only 63% accuracy [8]. Furthermore, the PCR machine must be installed in a specialized biosafety lab that may cost 15 thousand to 90 thousand USD and the RT-PCR kit costs between 120 and 130 USD [9]. Even high-income countries consider this tool too expensive [10]. Moreover, the shortage of these testing kits is causing a significant delay in detecting contaminated patients, making the situation more critical.

Medical imaging techniques such as Computed Tomography (CT) and Chest X-ray (CXR) are also used for COVID-19 diagnosis [11]. Indeed, CT has proven to be more sensitive, significantly faster, and cheaper than RTPCR in diagnosing COVID19 [12], [13]. However, this imaging modality is not recommended in regions of low disease spreading because of the significant rate of false positives [14]. Moreover, CT is often available only in large medical centers and is still in short supply in many countries. In addition, performing a CT scan for a patient with mild symptoms is unnecessary since patches can be detected even if the patient is asymptomatic [15]. In this case, CXR would be the optimal method for diagnosing COVID-19. Indeed, plain film chest x-rays are omnipresent globally, cheaper and with radiation doses 30 to 70 times lower than CT scans [16]. Thus, they are commonly used as the first screening modality for COVID-19. The American College of Radiology confirms that, for COVID-19, CXR is portable and easy to clean compared to CT [17]. Therefore, although chest CXR has lower sensitivity than chest CT [18], it is the typical imaging modality employed for the early screening of suspected cases [19]. Nevertheless, due to the similarity between the CXR imaging features of COVID-19 and other viral pneumonia, CXR represents a wide inter-observer variability. In the current circumstances, radiologists are under increasing work pressure, making their tasks more and more difficult.

Artificial Intelligence (AI) has recently shown the potential to improve medical imaging abilities, including accurate analysis, higher automation, and enhanced productivity [20]. Using AI, the research community has made a great effort to develop computer-aided diagnosis (CAD) systems to assist clinicians in their mission. Indeed, the second opinion of the CAD system can help increase the diagnosis’s accuracy, reduce inter and intra-observer variability, and avoid unnecessary procedures. Developing a CAD system can be challenging depending on the study stage [21]. Most research in this field has followed the traditional machine learning pipeline (ML). This pipeline requires a high level of application-specific expertise, especially for extracting and selecting the appropriate features from the images. Since 2012, Deep Learning (DL) based on Convolution Neural Networks (CNNs) has become an outstanding technique in several computer vision problems [22], [23]. The medical imaging community has adopted this technique for diagnosing or segmenting organs and forms in medical images [23], [24]. Many researchers investigating deep learning for medical diagnosis have achieved “near” human expert diagnosis performance [25].

In contrast with machine learning, CNNs do not need feature engineering. They automatically learn an appropriate image representation. A CNN has a unique architecture biologically inspired by the Artificial Neural Network (ANN). It has layers arranged in a hierarchical structure and is specially designed for learning visual features [26]. Many CNN-based systems are developed to automatically diagnose COVID-19 using both CT and X-ray imaging modalities [27]. Some are designed based on a pre-trained model with transfer learning [28], [29], and a few others are built using customized networks [30], [31], [32]. Most existing works suffer from a class imbalance, a common problem in deep learning-based classifiers, particularly in medical diagnostics [33], [34]. Class imbalance occurs when some classes have significantly more samples in the training dataset than others. It has been proven that class imbalance may have a negative impact on training a CNN model. Indeed, it can influence the convergence during the training step as well as the model generalization on the test set. It results in a model with poor predictive performance, specifically for the minority class [35]. There are two main categories of class imbalance solutions [36]. The first category is data-level solutions in which a change is made in the class distribution of the dataset itself, either by oversampling or undersampling the training dataset. The oversampling techniques are widely employed in deep learning and have been shown to be reliable. Random minority oversampling is the primary form of oversampling. It merely replicates randomly chosen samples from minority classes. It has been demonstrated to be efficient, although it can result in overfitting problems [35], [37]. SMOTE [37] is a more sophisticated oversampling technique that seeks to resolve this problem. It generates new synthetic samples by interpolating nearby data. The second category is classifier-level solutions. These solutions operate on the classification algorithm (model) while keeping the dataset unchanged. For example, some methods can alter the previous class probabilities [38]. Others add weights to the misclassification of samples from different classes [38]. Adding sample weights to the loss function is a straightforward technique to handle class imbalance. The idea is to weigh the loss calculated for different samples differently based on whether they are associated with the minority or the majority classes. A higher weight is assigned to the loss computed by the samples of minor classes [39]. Solutions from both categories can be combined to address the class imbalance problem [35]. Recently, Generative Adversarial Networks(GANs) [40], [41], [42] have been demonstrated to be a powerful technique to rebalance datasets [43] by generating artificial samples. However, GANs are fairly challenging to train and have high computational resources. The second solution involves altering the classification algorithm (model) while maintaining the dataset. Only a few studies on the detection of COVID-19 from X-ray images addressed the issue of class imbalance [40], [41], [42].

In this work, our main contributions are as follows:

• •

  Our objective is to develop an accurate deep learning model trained on a balanced dataset that assists radiologists in the early diagnosis of COVID-19 cases.

• •

  To this end, we trained six popular CNNs, including DenseNet201, CheXNet, MobileNetV2, ResNet152, VGG19, and Xception, via transfer learning (TL) on three different COVID-19 datasets. The models were developed to perform a multi-classification task that distinguishes between COVID-19, normal (no infection), and viral pneumonia cases.

• •

  To address the class imbalance issue, we first investigated the Weighted Categorical Loss (WCL) [40] and then the Synthetic Minority Oversampling Technique (SMOTE) [37] on each dataset separately.

• •

  After testing the developed models, we studied the outcomes of different evaluation criteria. This step aims to find the best model for the early detection of COVID-19 from CXR images.

• •

  After conducting an experimental comparison with other works, we noticed that the use of WCL and SMOTE to balance the datasets provided models with prominent results compared to the models that were trained with imbalanced data.

The rest of this paper is organized as follows. Section 2 reviews the state-of-the-art techniques employed for COVID-19 diagnosis using CXR images. Section 3 presents the methodology of this study. Section 4 introduces the experimental setup. An ablation study is provided in Section 5. Section 6 reports our results and discussion. Section 7 presents the limitations of our work. Finally, Section 8 summarizes our contribution and presents our future work. The source code is publicly released for research purposes.1

2. Related works

Several deep learning-based COVID-19 diagnosis systems have been developed based on data collected from CXR imaging samples. Some are built based on customized deep learning models. Others used trained deep learning models via transfer learning.

3. Methodology

4. Experimental setup

5. Ablation study

Before applying a balancing technique, the chosen models are trained on the three above-mentioned imbalanced datasets to compare the results and investigate which model has been improved or deteriorated. The models are trained using the categorical weighted loss function. Table 2 presents the classification outcomes obtained from the transfer learning of the six chosen deep learning models without applying a data balancing technique. By making an overview of this table, notice that all the pre-trained CNNs achieved high accuracy, sensitivity, specificity, precision, F1 score, and AUC. Especially VGG19 and DenseNet201, which generated the highest results on the three datasets. When training using dataset 1, DenseNet201 achieved the highest classification results with an accuracy of 97.51% and AUC of 97. 59%. Additionally, VGG19 provides slightly inferior results with an accuracy of 97.15% and an AUC of 97.73%. While MobileNetV2 generated the lowest results with only 94.10% of accuracy and 95.58% of AUC, The results are lower when the models are trained using datasets 2 and 3. Probably because some chest X-ray images collected in the above datasets contain artifacts including arrows, symbols, texts, and pixel-level noise. Nevertheless, DenseNet201 is still on top with accuracy and an AUC of 92.6% and 94.41%, respectively, using dataset 2. followed by VGG19, which generated an accuracy of 92.58%. When trained using dataset 3, DenseNet201 and VGG19 yielded good results. They provide an accuracy of 91.63% and 91.6%, respectively. Still, Xception and MobileNetV2 have the lowest outcomes.

6. Results and discussion

First, the weighted categorical cross-entropy loss function is employed for the training of the CNN models with the three chosen imbalanced datasets. The outcomes of this experiment are shown in Table 3. After several trials, it is observed that CheXNet provided the best classification results while trained on Dataset 1. CheXNet showed a PPV of 100% and almost a perfect AUC of 99.15 as well as high sensitivity, specificity, and F1_score of 98.86%, 99.43%, and 98.21%, respectively. The good results are probably due to the classification framework’s hierarchical structure, which considers correlations between different labels. This structure is detailed in [69]. CheXNet, which is originally a DenseNet121, pre-trained on 14 classes of Chest-ray 14 dataset. This architecture consists of one convolution layer with a (7 × 7) kernel, fifty-eight convolution layers with a (3 × 3) kernel, and sixty-five convolution layers of (1 × 1) kernel, four average pooling layers, and one fully connected layer. DenseNets simplify the connectivity pattern between layers [69]. In this work, we replace the final fully-connected layer with one that has three outputs, after which we apply a SoftMax. The weights of the network are initialized with weights from a model pre-trained on ImageNet [65]. The training parameters are detailed in Table 5. On the other hand, VGG19 succeeded in classifying the CXR images with an accuracy of 98.41% and an AUC of 98.81%. Using dataset 2, DenseNet201 provided an overall accuracy of 93.58% and an AUC of 95.38%, while ResNet152 provided the lowest accuracy of 87.55%. For dataset 3, DenseNet201 and VGG19 generated the highest outcomes with accuracy and an AUC over 94%. However, VGG19 has slightly outperformed DenseNet201 in this case. It is observed in Fig. 7 that the use of categorical weighted loss has improved the classification results of the DenseNet201 model. The improvement is highly remarkable with dataset 3. However, with dataset 1 and dataset 2, there is a weak improvement.

The same observation can be concluded from Fig. 8. The VGG19 successfully classified the x-ray images better using the weighted loss techniques on the three datasets. Later, SMOTE is used to balance the training set before it feeds to the CNN models. The outcomes of this experiment are shown in Table 4. The results of experiment 1 and experiment 2 are quite close in terms of top performing models and datasets. However, it is noticeable that DenseNet201 trained on dataset 1 with the application of SMOTE outperformed all the other scenarios in all the used evaluation metrics with an accuracy of 98.64%, a sensitivity of 98.36%, a specificity of 99.35%, a precision of 98.64%, an F1-Score of 98.63% and an AUC of approximately 99%. Followed by the VGG19, which provided very close outcomes. ResNet152’s results have improved slightly more than experiment 1’s results, while Xcpetion’s classification results have decreased somewhat. MobileNetV2 succeeded in classifying the image with high accuracy and AUC when trained with dataset 1 but generated poor results with datasets 2 and 3.

As depicted in Fig. 7, Fig. 8, SMOTE enhanced the AUC values of DenseNet201 and VGG19 compared to the AUC of the training with an imbalanced dataset. The proposed balancing techniques have remarkably increased the classification results of the chosen pre-trained deep learning models even though, in some cases, one of the techniques surpassed the other in terms of results. A summary of the proposed models that provided the best results is presented in Table 5, in addition to the hyper-parameters used for their training.

The best-obtained results are compared with the other recently proposed approaches to detect COVID-19 from X-ray images. Table 6 shows that our proposed model outperformed other works with the three datasets in terms of performance. For dataset 1, our model achieved an accuracy of almost 99%, while Chowdhury et al. [47] achieved only 97.9%. In addition, our model provided a perfect precision of 100%, compared with only 98% for Chowdhury et al. [47]. For dataset 2, our proposed WCL with the DenseNet201 model achieved a higher classification rate compared to the models proposed by Ozturk et al. [83] and Haghanifar et al. [79] in terms of accuracy and F1-score, as shown in Table 5. Our model that applied WCL with VGG19 on dataset 3 outperformed the COVID-NET model developed by Wang et al. [44] that, achieved an accuracy of 92.4%, a sensitivity of 88%, and a precision of 91%, while our approach achieved on the same dataset an accuracy of 94.87%, a sensitivity of 94.87% and a precision of almost 95%. Our approach also succeeded in classifying X-ray images for COVID-19 diagnosis better than Oh et al. [30] in terms of accuracy and F1-score.

7. Limitations

Due to a lack of computational resources and low memory, the training process of most DL networks takes many hours, an average of 3.5 h for each algorithm. Indeed, seven different deep convolution neural networks were trained with a multi-classification head in this work. Each CNN was trained with a batch size chosen randomly between 8, 16, or 32, with an input image of the size (244 × 244) while applying the WCL and the size of (128 × 128) while using the SMOTE. Overall, the training on the three datasets took 300 h. The computational complexity is proportional to the dataset size, the used balancing technique, and the depth of the trained CNN. The training of DensNet201 has the longest training time, 8h16m50s (for a 128 × 128 input image, batch size of 8 and 72 epochs). In contrast, the VGG19 had the lowest training time since it has fewer layers. The application of the WCL had a neglectable execution time compared to SMOTE. Indeed, the use of SMOTE as a balancing technique was very memory and time-consuming. The overall time of SMOTE running was between 15 and 90 min, depending on the size of the training set. One of the major limitations of this work was the high computational consumption. Moreover, the proposed model needs to be validated externally before undergoing any clinical use. This step is crucial as it properly assesses the model’s performance against real cases in different settings.

8. Conclusion

One of the major issues identified in most related works concerning the COVID-19 disease diagnosis using X-ray images is class imbalance. This paper presents an investigation of two classic data balancing techniques applied to three well-known datasets. Six deep learning models are trained on the selected datasets separately to identify which combination of the balancing technique and CNN architecture provides the best classification results. Each trained model was evaluated using benchmark performance metrics,e.g., accuracy, precision, area under curve, specificity, and F1 score under two different experiences concerned with imbalanced learning. With extensive trials, it was observed that models achieve different scores in different scenarios, among which DenseNet201 and VGG19 displayed better performance for the multi-classification of COVID-19 samples. This study was developed using limited computational resources, causing a long training time, especially when applying the SMOTE as a data balancing technique. As an extension to this work, we can investigate, using more powerful computational resources, other deep learning models and data balancing techniques such as data generation using generative adversarial networks (GAN). Furthermore, we can explore other datasets that contain more samples of COVID-19.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.