X-ray image based COVID-19 detection using evolutionary deep learning approach

3.1. Convolutional Neural Network (CNN)

Deep learning is a technique using in machine learning, in which a deep architecture designs several linear and non-linear processor modules to represent high-level functionality in a given dataset. A wide range of applications currently employ several such deep learning methods including restricted Boltzmann machines (RBMs), deep belief networks (DBNs), auto-encoders, and deep convolutional neural networks (CNNs). CNN approaches have been widely common in computer vision as well as in the field of medical image processing in the past decades (Anwar et al., 2018).

Bio-inspired versions of multi-layered perceptron are named as CNNs. They generally associate visual features from input images effectively. Such deep networks address small image pixel spots named temporal information through using multiple layer neurons and the weights shared within each convolutional level (Mahendran & Vedaldi, 2016). Three architectural concepts incorporate CNNs to verify some level of non-linearity in scale, transition, and disruption (Zheng et al., 2018). The local interconnections between the patterns of the neurons in adjoining CNN layers are chosen to take as a subcategory of units in the previous layer which have amenable spatially adjacent areas to leverage the spacial local correlation in the hidden units of the current layer. Furthermore, any filter in the CNN is repeated across the entire field of vision. Such filters exchange weight and bias vectors for establishing a feature map. The shared weight gradient is the same as the common shared parameter gradients. A feature map is generated when convolution operations are conducted on subsections of the entire image (Tian et al., 2020).

Let $X$ with $R^{A\times B}$ is used to determine the input in the convolutional layer and the input data size is defined as $A$ and $B$. The output for the convolutional layer is then stated as follows:

where $C_n$ represents the $m$th output of convolutional layer, and $n$ refers to the number of outputs equal to the filter size. The number of channels is represented by $N$; the convolutional operator is defined as *, and $X_c$ is the $c$th input data channel for the prior $(l-1$)th layer. For the present layer $l$, the width and height of the filter are indicated by $W$ and $H$, respectively. $W_m^l$ is the $m$th filter weight for the present $l$th layer, and $B_m^l$ is considered as $m$th bias. Generally, activation function in CNN structures is represented by $f$ which their most common functions are Sigmoid function, Rectified Linear Unit (ReLU), Leaky ReLU, and Tanh. The ReLU is the widely-used activation function which is defined in the following form:

The pooling technique allows in reducing the algorithm complexity with the feature maps, prevents proper over-fitting after the last fully convolutional layer, and significantly reduces training parameters to shorten the training time of the model. The most widely recognized method of pooling is max-pooling, which mainly involves traversing functional maps with flexible steps and evolving optimal values to the traversed areas. After performing the traversing operation, the updated feature maps will eventually be extracted. This is the entire process of the CNNs in general.

3.2. K-nearest neighbor

$K$-nearest neighbor (KNN) algorithm is a supervised-learning classification approach founded on samples of the closest training set throughout the feature domain which is considered as one of the simplest algorithms in machine learning applications (Kim et al., 2012, Ning et al., 2019). This algorithm is trained by the collection of vectors and labels of the training objects (here images). The unknown data points are simply allocated to their closest neighbors’ $K$ labels throughout the classification phase. The primary benefit of KNN algorithm is to work efficiently in binary classes since its decision is centered on a small neighborhood of related objects, and therefore can contribute to acceptable accuracy.

Instead of Softmax classifier as the last layer of the CNN model, we replace it with KNN classifier providing significant improvement in model accuracy and help to minimize the possibility of over-fitting in the classification performance. The objects are typically classified by majority voting on the labels of the closest $K$ neighbors. For $K$ $=$ 1, the entity is labeled basically as the nearest object class. If only two classes exist in the dataset, $K$ parameter has to be an odd integer number. Upon transforming each image to a vector of real numbers, we used Euclidean distance feature in KNN as the most widely accepted distance mechanism. The Euclidean distance of KNN algorithm is formulated as following:

where $x_i$ and $x_j$ are the $m-$dimensional vectors of the $i$th and $j$th samples, and the index $r$ indicates the $r$th real-valued feature of each sample.

3.3. Modified Competitive Swarm Optimizer (MCSO)

In order to solve large-scale optimization problems, Cheng and Jin (2014) introduced a robust and efficient particle swarm optimization (PSO) variant, known as competitive swarm optimizer (CSO). In CSO, the particles benefit from randomly chosen opponents and not from the strongest global or the individual position. The population of swarms is split randomly into two groups in each iteration and competitions between the particles in any group take place on a pair principle. The winner particle is directly transferred during every competition toward the next iteration, while the losing particle updates its position and velocity with a knowledge of the winning particle based on the following equations:

where $t$ represents the iteration counter, $G_1^t,G_2^t,G_3^t$ represent three $[0,1]$ range vectors that are generated randomly, while $x_w^t$ and $x_l^t$ are respectively the winner and loser particles. ${\overline{x}}^t$ represents the mean swarm position in iteration $t$, and the effect of ${\overline{x}}^t$ is handled by $\lambda$.

In order to strength more the search capabilities of CSO algorithm, we introduce a three-phase modification on this evolutionary algorithm to make an efficient balance between exploration and exploitation phases as well as the ability to escape from local minima. We name our novel evolutionary swarm intelligence algorithm as modified CSO (MCSO) in which the modifications we have applied over CSO are listed as follows:

-First modification:

Generally, during the beginning phases of the search procedure, the CSO algorithm converges increasingly. Nevertheless, once CSO hits global optima point, the convergence decreases significantly. We implement Cauchy Mutation (CM) evolutionary operator in CSO to set up a procedure that monitors and helps search agent improvements by escaping the local search space through spreading to proper positions. To this end, the following equation is used:

where $t$ shows the scale element. The probability distribution function of the Cauchy component is shown by:

The mutation operator disrupts the population of the search agents and enables them break of the local minima. The deployment of CM in CSO is defined as following:

where $W_j$ represents the matrix of weights, $x_{ij}$ is the $j$th position of the $i$th search agent and population size of the search agents is denoted by $P$. The Cauchy mutation operator is applied as follows:

where $A$ is a random vector generated by Cauchy.

-Second modification:

In CSO algorithm, per each iteration, the position of search agents is updated continuously. After finalizing the updating procedure, the search agents exceeding the search space boundaries are considered worthless in order to find the best solutions. Therefore, the search agents must be redirected to the search field, accomplished by the evolutionary boundary constraint handling (EBCH) technique. Once we incorporate this technique into CSP, its exploration and exploitation adaptability is enhanced effectively. Thus, the relevant attributes of the best search agent positions substitute the search agents that contravene the search space boundary conditions. The EBCH updating process is given by the following formula:

where $o_i$ represents the out-of-bound search agent, $\xi$ and $\varrho$ are random variables in [0,1] interval, ${\mathrm{ub}}_i$ and ${\mathrm{lb}}_i$ denote to the upper and lower bounds of the search space, respectively, and $x_i^b$ is considered as the $i$th best search agent.

-Third modification:

In order to improve CSO convergence speed, $\lambda$ parameter plays a key role. In the iterative cycle of CSO, the evolutionary chaotic maps are used to strengthen the ability of utilizing and keeping the core search step harmonized. This concept motivates us to fine tune the $\lambda$ parameter using a powerful chaotic map named tent map. The formula of tent map is given as following:

In Fig. 2, the detailed framework of the proposed MCSO algorithm is summarized as a flowchart.

3.4. Proposed MCSO-CNN

Within this section, we introduce our novel proposed deep neuroevolution approach named as MCSO-CNN. In order to achieve optimal CNN hyperparameters that increase the accuracy of X-ray image classification, the proposed method is optimized using the MSCO algorithm. Two essential challenges should be addressed by each evolutionary optimization technique, namely, solution representation and fitness function calculation. On the other hand, eleven key hyperparameters of CNN model are highlighted in the proposed classification model to be optimized by the MCSO including convolution filter size, number of filters, number of convolutional layers, activation function type, dropout rate, maxpooling size, learning rate, momentum rate, optimizer type, number of epochs, and batch size. Therefore, any solution in MCSO can be represented as an eleven dimensional vector that corresponds each to one of the eleven hyperparameters of CNN model. Learning rate, momentum rate, and dropout rate are the continuous value hyperparameters that MCSO can achieve their optimum values. In addition, number of convolutional layers, number of filters, kernel size, batch size, maxpooling size, and number of epochs are other discrete hyperparameters. Optimizer type and activation function type hyperparameters are the categorial ones that are utilized in the optimization procedure by MCSO. While MCSO actively searches the space for solutions, the values obtained optimally for the hyperparameters should be converted all into their corresponding discrete values. Thus, we design a cost-effective strategy to convert each real value into an integer value. In this regard, the continuous values in each hyperparameter are passed to a discrete search space as $D=[H_1,H_2,\dots ,H_n]$. To formulate the discretization model, the following equations are taken into consideration:

where $C$ represent a continues value in the $[0,1]$ interval for exploration purpose in the continuous search space area, $\zeta$ is an operator responsible to map from $C$ to $[1,n+1]$, and $\psi$ is another mapping operator from $\zeta$ to $[1,2,3,\dots ,n]$. Each integer value belonging to the solution’s continuous dimension can thereby be computed using the following formula:

Using Eqs. (4), (5), the proposed MCSO-CNN method starts randomly to initialize the population of $n$ solutions. $X_{ij}$, where $i=1..n$ and $j=1..11$ is a 11-dimensional vector standing for the $i$th solution encoding the eleven hyperparameters of CNN model. A continuous update of current solutions after generation of the first population can provide new solutions. We use CM strategy to help the CSO in order to escape from local optima by Eq. (9). For improving the convergence of CSO, we incorporate the EBCH technique into the CSO using Eq. (10). Then, we tune the parameter $\lambda$ with the tent chaotic map using Eq. (11) for providing a balance between exploration and exploitation steps. The whole process is repeated until the final threshold is met and the best solution is found as expected. The best CNN hyperparameter values can be adopted in this solution. We also require to specify a fitness function to find out the success of all the solutions that have been obtained by the MCSO-CNN model. To facilitate this, the input data from the COVID-19 dataset is divided into two distinct training and test sets. The training set optimizes CNN hyperparameters, while the test set evaluates the performance of the final COVID-19 detection model.

It should be noted that the CNN model is equipped with the hyperparameter values from any solution in MCSO. Therefore, the fitness function for the efficiency of the optimized CNN model for detection of COVID-19 can be interpreted. The images in the training set can be included in the training period as the inputs of the CNN model. The fitness value of solutions during optimization procedure is the accuracy of the CNN classifier in classification of input images. The accuracy factor denotes to the number of images classified correctly over all of the total input images in the dataset. Thus, the accuracy value can be formulated by using the following formula:

Precisely, a solution with higher accuracy has a higher fitness value and likewise. The objective of the proposed approach is thereby to achieve a solution that contains the optimum values of CNN hyperparameters of the highest accuracy (i.e. highest fitness value). The optimized CNN model is then used to classify images in the test set. The overall steps of the MCSO-CNN method are outlined by Algorithm 1.

4.1. Dataset description

The proposed DNE framework is evaluated and tested using X-ray images provided by Alqudah and Qazan (2020) which can be downloaded for free of use from Mendeley open source repository.1 The dataset consists of 1824 images including 912 images for normal cases and the rest for COVID-19 affected cases. Fig. 3 shows three samples of normal cases and also three samples of affected cases. Since CNN requires input images to be the same size, we scaled all the images to 224 × 224 pixels.

4.2. Parameter setting

The proposed DNE framework uses eleven parameters that should be learned automatically in a deep evolutionary scheme. With respect to parameterization of the MCSO evolutionary model, we set the maximum number of iterations to 20 and population size to 30. For the sake of fairness, all the algorithms used in the comparison phase, including the one proposed, are executed 10 times, then the average accuracy measurement is reported. Table 1 defines all hyperparameters and their categorical or integer coding values in the proposed DNE algorithm. In the experiments, we compare the proposed method with other models based on three different scenarios. In the first scenario, the proposed method is compared with other evolutionary algorithms used to optimize the hyperparameters of CNN model. To have a fair comparison, we set the maximum number of iterations and population size for all the compared algorithms to exactly the values used for the proposed method. Other specific parameters related to different evolutionary algorithms are set according to the optimal values obtained through a greedy search process or by their suggested values in their related literature (Al-Betar et al., 2021, Yue and Zhang, 2021, Zhou et al., 2021). In the second scenario, the proposed method is compared with other supervised learning classifiers in which the values of their parameters are set based on the best values reported in their corresponding papers, and also a greedy search process is performed to obtain the optimal values for these models. Finally, in the third scenario, the state-of-the-art image classification models are considered to compare with the proposed method. In this case, the values of parameters for the compared models are initialized based on the best values reported in their corresponding papers. For implementing the proposed method and other algorithms used in the experiments, we used TensorFlow which is a high-level platform developed in the Python programming language for machine learning techniques. Also, all algorithms are performed on a machine with one GeForce GTX 1080 Ti GPU and one 16 GB RAM.

4.3. Performance evaluation metrics

We employed standard performance metrics including accuracy, precision, recall, area under the curve (AUC), and F1-score to evaluate the prediction efficiency of the compared models. In these metrics, the number of positive samples correctly predicted is denoted by true positive (TP) while the number of positive samples wrongly predicted is represented by false negative (FN). On the other hand, the true negative (TN) is the accurately predicted number of negative samples, whereas a false positive (FP) is the wrong predicted number of negative samples. The equations related to the evaluation metrics are defined as follows:

• •
  Precision is expressed as the ratio of samples correctly classified in truly positive patients which is obtained by:

  $$\text{Precision}=\frac{\text{TP}}{\text{TP}+\text{FP}}$$

  (16)

• •
  Recall shows the rate of total positive samples correctly classified by the algorithm which can be calculated as follows:

  $$\text{Recall}=\frac{\text{TP}}{\text{TP}+\text{FN}}$$

  (17)

• •
  F1-score ($F1$) is the harmonic average of two other metrics precision (P) and recall (R) defined as follows:

  $$F1=\frac{2\times P\times R}{P+R}$$

  (18)

• •
  Accuracy is defined as the rate of the corrected classified cases:

  $$Accuracy=\frac{(TP+TN)}{(TP+TN+FP+FN)}$$

  (19)

• •
  AUC is described as the area under the ROC (receiver operating characteristic) curve showing the performance of classification/detection model. It is defined as follows:

  $$\text{AUC}={\int }_0^1\frac{\text{TP}}{P}\text{d}\frac{\text{FP}}{N}=\frac{1}{P.N}{\int }_0^1\text{TP}\text{d}\text{FP}$$

  (20)

4.4. Results and discussion

In this section, we perform an extensive performance comparison with different evolutionary algorithms, classifiers, and state-of-the-art deep learning architectural models to show the competency and efficiency of our proposed MCSO-CNN model.

4.5. Run-time analysis

In this section, we analyze the run-time of our proposed algorithm and other benchmarking methods for COVID-19 diagnosis. The time consuming results for all algorithms in terms of optimization, training, and testing times are tabulated in Table 6. As can be seen from this table, in terms of optimization time consumption, our proposed algorithm has the shortest time among other methods that have used the optimization method. Furthermore, among all methods, including those that have used optimization and non-optimization methods, our proposed method has the lowest consumption time for training and testing time indicators.

4.6. Remarks

In this section, we provide some discussions about the insight of the main results and also the reasons why the tested results are achieved. It is worth mentioning that the contributions of the proposed method can be investigated in three different perspectives. First, we proposed a novel evolutionary algorithm by incorporating three evolutionary operators: Cauchy mutation, evolutionary boundary constraint handling, and tent chaotic map into the search process of the original version of competitive swarm optimizer model. Second, the Softmax layer of CNN model is replaced with a KNN classifier to improve the classification accuracy. Third, the proposed evolutionary algorithm is applied to automatically achieve the optimal values of hyperparameters of CNN model. In order to show the efficiency of these three contributions, several experiments have been conducted in this paper. For the first perspective, the performance of the proposed evolutionary algorithm is compared with a number of state-of-the-art evolutionary methods including GA, DE, PSO, MFO, WOA, SSA, HHO, GOA, and CSO. The results of these experiments are reported in Table 2 where it is shown that the proposed method outperforms other compared methods. The main reason to obtain such results is to consider the three evolutionary operators in the proposed evolutionary algorithm which leads to a significant improvement in the accuracy of classification method. In other words, the proposed evolutionary algorithm makes a balance between the exploration and exploitation phases which results in increasing convergence speed and reducing the probability of falling into local optima. For the second perspective, the proposed method is compared with different supervised learning classifiers to investigate the performance of the KNN classifier in comparison to other models including Decision Tree, Random Forest, LightGBM, AdaBoost, Softmax, and Bagging. The results of these experiments are reported in Table 3 where we can see that the KNN classifier performs better than other models. This is due to the fact that the KNN classifier takes into account the agreement of the neighborhood labeling which results in achieving accurate outputs in the last layer of CNN model. Finally, for the third perspective, the performance of the proposed classification method is compared with different deep learning based classifiers to investigate that how the deep CNN model optimized by the proposed evolutionary algorithm can outperform other compared models. To this end, the proposed method is compared with DenseNet121, MobileNet, InceptionV3, XCeption, ResNet50, VGGNet19, and DeCoVNet models. The results of these experiments are shown in Table 4 where it can be seen that the proposed method is better than other models. Therefore, we can conclude that considering the KNN classifier in the last layer of CNN model and also using the proposed evolutionary algorithm to optimize the hyperparameters of CNN model significantly improve the classification accuracy.

In addition, to prove that the experimental results are statistically significant, we performed a ranking statistical test called the Friedman ranking test to theoretically evaluate and analyze the performance of the proposed model and all benchmark algorithms. The results of this statistical test reported in Table 5 prove that the experimental results are statistically significant. Also, the run time of the proposed method is compared with other methods and the results are shown in Table 6 based on optimization time, training time, and test time. These results demonstrate that the proposed classification method performs in a shorter time with respect to other compared models. Therefore, it can be concluded that the proposed method not only achieves more accurate classification results, it also performs faster than other compared methods.