Classification of lungs infected COVID-19 images based on inception-ResNet

1.1. COVID-19 and clinical manifestations

Caused by SARS-CoV-2, COVID-19 is an infectious disease and could cause a severe acute respiratory syndrome, typical inflammatory response, vascular injury, microangiopathy, angiogenesis and extensive thrombosis [5].

Early COVID-19 usually presents with respiratory symptoms. Symptomatic individuals may occasionally develop mild but self-limiting infections with side effects, including cough, breathing problems, fever, shortness of breath, fatigue and sore throat [6,7].

Few patients might develop more severe symptoms, often requiring hospitalization for pneumonia. In some cases, people may experience severe life-threatening complications, such as those leading to ARDS, sepsis, and septic strictures, leading to multiple systemic organ dysfunction failures [8].

1.2. Medical multimodality imaging

Generally, computed tomography (CT) is used to present images of pneumonia, tuberculosis, emphysema, or other pleura diseases (membrane covering the lungs). Classification of lung CT images is a necessary initial step in lung image analysis. The classification's significance of algorithms is particularly evident due to the strength of homogeneity, artifacts, and the proximity of different soft tissue gray levels. In this paper, a fully convolutional neural network is designed to classify chest infected with COVID-19 CT images end-to-end automatically. For sizes of diseased area distributions in digital lung photography and the diversity of shapes, we present multiscale image information into a fully convolutional dense network structure, which improves classification accuracy.

Medical imaging techniques are commonly used in clinical practice, including computed tomography (CT), magnetic resonance imaging (MRI) and ultrasound imaging (US), etc. [9,10]. The more accurate imaging techniques and precise pictures allow for a better screening of disorders, which can give crucial data on conditions like COVID-19. CT has become a necessary technical means for observing treatment response and prognosis evaluation with technology development. The potential COVID-19 performance characteristics can be extracted from the image through the newest medical imaging information processing, effectively solving the problem, which plays a significant role in routine clinical pathology sampling and analysis [11].

There are a series of parameters that can be modified in CNN, such as the number of processing units (neurons), the number of layers of the network model, the size of the convolution kernel, the size of the stride, the learning rate, and the activation function. After adjusting and changing various parameters, there are many typical networks based on CNN. The efficiency of CNN is significantly improved than the traditional image classification method,

2.1. Convolutional neural networks

At present, it is generally recognized in the industry that with good ability for feature extraction and recognition convolution neural network (CNN) is a deep artificial neural network and is the most widely used machine learning technology [14,15]. The sharing of weights is one of its most prominent features, which significantly reduces the number of weights and the complexity of the network model [16].

Convolutional neural network (CNN) minimizes the loss function by many convolution filters, nonlinear activation functions and pooling layers. Any component of the convolutional neural network is a simulation of biological neuronal activity, and all components have many inputs (denoted as x ₁,x ₂,x ₃,...,  x_n), that is, the so-called characteristic matrix. When the characteristic matrix (i. e., input) is fixed, the convolution layer will convolution the characteristic matrix and each filter{W_i}_k to obtain the pre-activated figure S. The specific Equations is as follows:

Note: Symbol · represents convolution operation, b_irepresents deviation parameter.

A typical CNN network structure usually includes convolutional layers and pooling layers (Pooling Layer), and the end is one or more fully-connected layers [17,18]. The network structure diagram is shown in Fig. 1 .

After the convolution operation is completed, a bias needs to be added to the result of the convolution, and then a nonlinear excitation function (in a neural network, the excitation function is usually relu) is used to obtain the final output of the layer. Result [19]. The Eq. 2 illustrates this process. $x_j^l$ represents the jth feature map of the lth layer, f represents the excitation function, M is the set of input feature maps, · represents the convolution operation, k represents the convolution kernel, and b represents the bias term [20].

2.2. Deep residual networks

ResNet is considered a continuation of deeper networks, in which optimal theory for training deeper networks is introduced [21]. The classic ResNet network has 50 layers, 101 layers and 152 layers. Among them, the 152-layer deep CNN won the 2015-ILSVRC championship. Furthermore, ResNet achieves a 28% progress on a prominent image recognition example dataset named cOco132 [22]. ResNet mainly exploits the idea of bypass channels in the "road network", using the following mathematical Eq. 3 and 4 [23]:

In Eq. 3, f is the transformed signal, and x is the original input. The original input is added to f(x) via a bypass path. In Eq. 4, g(x) is used to perform the residual operation. ResNet introduces shortcut channels within layers to realize connections between different layers, but these gates are data-independent and parameter-free compared to highway networks [24]. In a highway network, these layers represent non-residual functions when the shortcut path is closed. The submodule can be composed of two parts, a linear direct mapping x → x, and a nonlinear mapping F(x). If the direct mapping of x → x, is optimal, the learning algorithm can easily set all the weight parameters of the nonlinear mapping F(x) to 0. If there is no direct mapping, let the nonlinear mapping f(x) learn a linear x → x, mapping is difficult [25]. However, in ResNet, residual information is always passed, and the shortcut channel is never closed. Residual links (fast channel connections) accelerate the fusion of deep networks, allowing ResNet to avoid the gradient reduction problem [26]. A typical ResNet network structure is shown in Fig. 2 .

ReLU is an activation function. The purpose of using the activation function is to prevent gradient dispersion, deepen the propagation depth of the function, and reduce the gradient attenuation caused by deep convolution. The Equation of ReLU is as follows.

When x>0, R(x)=x, the derivative is 1; when x ≤ 0, R(x)=0, the derivative is 0.

3.1. Dataset

Firstly, we scanned the lung images of various pneumonia patients and healthy people obtained via X-ray and CT scanners at Jinyintan Hospital in Wuhan. We collected lung X-ray and CT images from 2045 normal subjects, 785 early patients, and 130 severe patients with COVID-19. These images were used with the consent of the patient and the hospital, and the data sets were collected and manually annotated by professional imaging physicians. The institutional review board grants ethics approval. A total of 2960 sample images were collected, and the sample images covered three main categories, marked as normal confirmed images(2045), images of early COVID-19(78) and images of severe COVID-19(130) [27,28]. All images were 299 × 299. The data distribution used in the experiment is shown in Table 1 .

We segment the lesion area and combine the characteristics of COVID-19 and clinical cognition to extract the quantitative CT image features of many characteristics of the COVID-19 infection image, and then use feature extraction to change the neural network to construct a prediction model for the diagnosis of COVID-19, and screen out 20 radiomic features for the diagnostic value of COVID-19. This provides a non-invasive detection method for COVID-19 prediction.

3.2. Inception-ResNet architecture and model training

We use the ImageNet dataset, which contains more than 1 million images, to train the CNN model Inception-ResNet-v2. The network contains 164 layers so that it could classify about 1000 object categories [29]. Therefore, the network model can learn rich attribute representations for various images. The initial ResNet block incorporates convolutional filters and residual connections of multiple sizes [30]. The motivation chooses its architecture based on experimental results and comparative analysis with other popular deep learning models. Inception-ResNet-v2 has an excellent trade-off between resource requirements and model performance (accuracy). Other extensive and bulky models cannot be selected because this model is expected to work in an edge environment. Fig. 5 depicts the architecture of the custom model [31].

The final Inception-ResNet-v2 network structure is shown in the Fig. 3 .

The structure of the stem network is a shallow feature extraction module of the combined network. Through a variety of different types of convolution parallel combinations, some of the structures split the large convolution kernel into series small convolution kernels, for example, 7 × 7 large convolution kernels are split into 1 × 7 and 7 × 1 small convolution kernels. This asymmetric structure can extract more multi-level structural features and increase diversity. The stem network structure in the combined network is shown in Fig. 4 .

Our process is image acquisition, image preparation and DL model building, and the final step is diagnosis and feedback. Data preparation is one of the most critical steps in many cases because it is pre-comparing the collected data efficiently that leads to accurate results [32]. It contains operations, including equalizing the number of images in each class, simple filtering and denoising, etc. Afterward, the used datasets are divided into two groups: the training set and the test set. We conduct many tuning experiments during the training process to obtain the optimized network parameters. The more the model learns, the more accurate the classification output will be. Then, after many experiments are aggregated and deployed, the model is tested on the remaining unseen images (the test set) and the DL network [33].

Fig. 5 shows more detailed information about the proposed system network.

3.3. Self-attention mechanism and 3D nature of CT images

In order to enhance the classification capability of Inception-ResNet further, we introduce a self-attention mechanism, in which we squeeze each channel of the output feature map, perform global average pooling, and obtain a feature vector [34]. Welearn the feature weight of each channel through this feature vector and convert the initial feature map into a weighted feature map. The performance of the network is further enhanced by this technique, which takes into account the relationships between various channels and adaptively modifies the relative feature intensities of the channels. The processing burden it introduces, according to experiments, is tolerable given the increased network performance [35].

In addition, CT images are natural 3D image sequences, and related studies have shown that better classification performance can be obtained if lung nodule CT images are input as 3D cubes. This is because 3D CT images contain dependencies between consecutive slices, and we can obtain image features of lung nodules at a deeper level to achieve better classification. This study will also use 3D CNN to improve the accuracy of lung nodule classification.

3.4. Multi-scale feature extraction for SENet lung infection classification model

The underlying network architecture of this subsection is SENet, and based on this, a SENet lung infection classification method with multi-scale feature extraction (FB_SENet) is proposed. The method first preprocesses the original dataset, and the preprocessing process is divided into region of interest extraction, transformation of pixel values, and image enhancement. After the data are pre-processed, they are fed into FB_SENet for classification output.

The overall flow of the experiment is shown in Fig. 6 .

FB_SENet is mainly composed of three 3 × 3 convolutions, one maximum pooling, four blocks, multi-branch feature extraction module and Softmax. In this paper, the extracted ROI is pre-processed and fed into FB_SENet. In the original SENet, the input layer is a 7 × 7 convolution, but in this paper, the 7 × 7 convolution is replaced by three 3 × 3 convolutions in order to capture more image features and obtain the same size of the perceptual field, which can reduce the number of network parameters and enhance the feature extraction capability of the network. Finally, feature extraction is performed by 4 Blocks and MFEM and the output is classified by Softmax.

3.5. Performance evaluation indicator

Given the labeling and classification results, we use accuracy (ACC), sensitivity (true positive rate, Recall, TPR) and specificity (SPE) as several commonly used main indicators to evaluate the performance of the proposed Inception-ResNet. Specifically, we can use TP (true positives), FN (false negatives), FP (false positives) and TN (true negatives) to calculate ACC, recall TPR and positive class prediction error PPV. Related equations are listed in the following Eqs. 6–12:

4.1. Experimental details

We crop images of 64 × 64 and 3D volume data of 48 × 48 × 9 from CT scans of lung nodules based on the annotation center and divide them into two parts, 75% of which are used for training validation and the rest as test set. To overcome the limited sample of nodules, we augment the nodules with translation, rotation, and flip operations [36]. This enhancement also helped extract features from lung nodules' CT scans that did not change with the three procedures. In addition to the traditional initial ResNet, we introduce a self-attention mechanism and 3D convolution, resulting in our new networks: 3D SE-IRNet and 3D SE-CDNet. We employ mini-batch gradient descent to minimize the CNN's loss function and learn weights in our experiments. We randomly initialize the weights using a Gaussian distribution and update the network parameters using standard backpropagation during training. The learning rate is initially set to 0.1 then and decreases by 5% every 1500 epochs. Batch size and momentum are initialized to 64 and 0.9. We maintain a similar data distribution between training and test sets to avoid over-or under-expression of features due to imbalanced distributions. The hardware environment for our experiments is based on NVIDIA Tesla P100. The software environment is Keras 2.1.0 and Tensorflow GPU 1.12.0 based on Ubuntu 16.04 operating system. The language used is Python 3.6.7.

4.2. Experimental metrics and results

Based on Inception-ResNet, the CNN model is trained to classify the collected CT images of different periods.

Each pre-trained model is trained on grayscale images. The model accuracy and loss plots of the Inception-ResNet CNN are shown in Fig. 7 a and b.

From the model in Fig. 7, only 1200 COVID-19 images out of 1200 images were identified, and only 15 non-COVID-19 images out of 1230 images.

Our experimental indicator results are shown in Table 2 , which presents the precision, recall, f1-score, and G-mean of Inception-ResNet for the dataset for detailed performance analysis [37].

The ROC curve of the experimental results is shown in Fig. 8 .

The results of normal images, early COVID-19 images and severe COVID-19 images are shown in Fig. 9 .

The selection of the number of feature groups s in the graded residual module. In order to achieve the best classification effect, therefore, the selection of s was verified in SENet, and the experimental results are shown in Table 3 .

When s=1, the network degenerates into an ordinary SENet, while when s=1, s=2, s=3, and s=4, all four evaluation indexes of the model improve to some extent, and when s=5, its accuracy decreases again. Therefore, it can be proved through experiments that when the number of feature groups s=4, the network can obtain the best performance at this time and its classification is the best.

Comparison with classical classification network models. Including: DenseNet, ResNet101, MnasNet, MobileNet2, ShuffleNetV2, SK_ResNet101, SENet. the classification results under the same data set are shown in Table 4 . From the table, it can be seen that the FB_SENet proposed in this paper is higher than the above classical classification models under each evaluation index, which further confirms the superiority of the method in this paper.