Enhancing brain tumor MRI classification with an ensemble of deep learning models and transformer integration

Improvement strategy 1

First, we discuss the basic concept of our approach, followed by a detailed description of the suggested hybrid architecture, as seen in Fig. 3. This model is inspired by the combination of CNNs for their powerful feature extraction with transformer encoder methods. CNNs have long dominated computer vision modeling. As widely recognized, deep learning CNN architectures possess the capability to examine the spatial interconnection existing among adjacent pixels within the region of interest delineated by the size of the convolutional filter, while disregarding the directional associations influenced by the separation between these pixels (Shamshad et al., 2023). To overcome this limitation, attention-based transformers have recently emerged, demonstrating enhanced effectiveness and robustness in managing both spatial pixel correlation and distance relations, thus enhancing the accuracy of visual recognition tasks.

Improvement strategy 2

Second, ViT demand extensive training datasets, a significant limitation in the field of medical image analysis (Liu, Lin & Cao, 2021). A hybrid approach is proposed that leverages the Swin Transformer architecture to mitigate this issue. The suggested hybrid block comprises the following modified layers: position embedding, patch partitioning, linear embedding, a Swin Transformer block, and patch merging. These layers are specifically adapted to process input features effectively.

Overall design of the method (with a block diagram and steps)

The suggested hybrid framework involves the following processing steps:

• Step one: Data pre-processing was conducted to prepare the data, including resizing, normalization, cropping, dividing the data into testing, training, and validation sets, and applying data augmentation techniques.

• Step two: Feature extraction was performed using advanced ensemble CNN architectures. To determine the optimal backbone network for the hybrid framework, an extensive experimental examination of six trained models was carried out.

• Step three: After determining the best hyperparameters using Bayesian Optimization, the combined features from the models were processed through the transformer encoder.

• Step four: The final results were produced by feeding the acquired features into the classification layer.

Algorithm implementation (with algorithm flowchart and pseudocode table)

The implementation algorithm of the HWST model is analyzed in this section. The pseudocode for the algorithm is provided in Table 1, and the corresponding flow chart is illustrated in Fig. 4.

Swin transformer block

The Swin Transformer is created by replacing the standard multi-head self-attention (MSA) module within a Transformer block with a module that utilizes shifted windows, as explained in ‘Self-Attention Using Shifted Windows’, while keeping other layers unchanged. A Swin Transformer block consists of a shifted window-based MSA module followed by a 2-layer MLP with GELU non-linearity in between. Layer normalization (LN) is applied before each MSA module and each MLP, and a residual connection is applied after each module.

Self-attention using shifted windows

The traditional Transformer architecture and its adaptation for image classification (Dosovitskiy et al., 2020) implement global self-attention, where each token’s relationship with all other tokens is computed. This global computation results in quadratic complexity in relative to the number of tokens, making it inefficient for vision tasks that require a vast number of tokens for dense predictions or high-resolution image representation.

Non-overlapping window-based self-attention

To enhance modelling efficiency, this approach calculates self-attention within specific local windows. These windows are organized to evenly divide the image without overlapping. Assuming each window consists of M × M patches, the computational complexity of both a global MSA module and a window-based one on an image with h × w patches is:

(1)${\displaystyle \Omega \left(\mathrm{MSA}\right)=4hw{C}^2+2{\left(hw\right)}^{2}C}$ (2)${\displaystyle \Omega \left(\mathrm{W}-\mathrm{MSA}\right)=4hw{C}^2+2M^{2}hwC.}$

The first equation shows that the complexity is quadratic in terms of the number of patches hw, making global self-attention computation impractical for large values of hw. In contrast, the latter equation demonstrates linear complexity when M is fixed (typically set to 7 by default). This design choice ensures that window-based self-attention remains scalable, providing a more feasible approach for handling large hw values.

MLP layer

The multilayer perceptron (MLP) is a neural network model known for its feed-forward design, which includes dense and dropout layers. In this study, the MLP features two non-linear layers that use Gaussian Error Linear Units (GELU). The MLP blocks are arranged in a consistent manner, with layers stacked in similar patterns. For example, let X ∈ ℝ^n×d denote token attributes, where n stands for the sequence length and d represents the dimension. Each block is mathematically described as:

(3)${\displaystyle Z=\sigma \left(XU\right),\tilde{Z}=s\left(Z\right),Y=\tilde{Z}V}$ (4)${\displaystyle z_0=\left[x_{\text{class}};x_p^{1}E;x_p^{2}E;\cdots ;x_p^{N}E\right]+E_{POS},E\in R^{\left(p^2\mathrm{\ast }C\right)\mathrm{\ast }D},E_{pos}\in R^{\left(N+1\right)\mathrm{\ast }D},}$ (5)${\displaystyle z_l^I=\mathrm{MSA}\left(LN\left(z_{l-1}\right)\right)+z_{l-1},l=1\dots ..L}$ (6)${\displaystyle z_l=\mathrm{MLP}\left(LN\left(z^{I}l\right)\right)+z_l^I,l=1\dots ..L,}$ (7)${\displaystyle y=LN\left(z_l^0\right).}$

Hyperparameters

The HWST model is trained in an end-to-end manner using specific hyperparameters to enhance its performance. The training parameters include a learning rate of 0.0001, which is reduced by a factor of 0.2, an epsilon value of 0.00001, and a patience of 7. Additionally, an early stopping strategy with a patience of 10 is implemented. This strategy can execute operations at multiple phases of learning, including different batch and epoch intervals.

The model is compiled with the Adam optimizer, a clip value of 0.2, and an epoch count of 100. During the selection of pre-trained models, an epoch of 100 is maintained while other parameters remain constant. The encoder transformer block is configured with a patch size of 2, a window size of 2, and global average pooling (GAP) for the shift size. Bayesian optimization, specifically using the Optuna framework, is employed to optimize additional parameters. These include dropout rates after each MLP layer, dropout rates after Swin-Attention, dropout at the end of each Swin-Attention block, drop-path regularization within skip connections, the number of attention heads, embedded dimensions, and nodes in the MLP layers.

This systematic optimization ensures an empirically refined configuration, enhancing the performance and adaptability of the encoder transformer block within the specified framework. The values of these parameters are detailed in Table 3. The output from the transformer encoder block is fed into a dense neural network consisting of two layers: the first with 256 nodes and a dropout rate of 0.5, and the second with 96 nodes and a dropout rate of 0.1. These specific values were determined through the hyperparameter optimization process.

This section provides an overview and explanation of the results obtained from the proposed models. Initially, we examined the sensitivity of the models’ parameters and reported the research findings accordingly. The classification performance of the models, both individually and in combination, is then presented and discussed. Table 4 summarizes the comparison of all models employed in this study, focusing on their final output shape and the number of trainable and non-trainable parameters.

Classification results

Before implementing the HWST model, a thorough analysis of parameter sensitivity was conducted to determine the optimal number of ensemble deep learning models. Various ensemble configurations, involving 1, 2, 3, and 4 deep learning models, were tested. The combinations of ensemble’s deep learning models were randomly selected from six options: DenseNet201, VGG16, GoogleNet, InceptionResNetV2, Xception, and EfficientNetB7.

The outcomes of the multiclass classification scenario, as shown in Table 5, present the classification evaluations based on overall identification accuracy (ACC), precision (PRE), and F1 score. The most favorable classification performance of the proposed model was observed when with three ensemble feature extractor models.

Subsequently, the transformer encoder was applied to the best-performing ensemble, incorporating hyperparameter optimization using the Bayesian optimizer. Table 3 details the specific parameters obtained from this optimization along with the corresponding search results.

This section presents the outcomes of the classification approaches implemented in this article. The model’s classification performance was evaluated across three scenarios: individual pre-trained models, ensemble models know as Ensemble A and Ensemble B, and the proposed hybrid model based on the ensemble transformer encoder mechanism. All experiments in this study utilized the Cheng dataset for the multiclass classification task using MRI images. This dataset was chosen for its reliability and its ability to demonstrate the models’ capacity to handle multiple classes concurrently.

(1) Individual pre-trained models classification results

Six state-of-the-art deep learning models were selected for the backbone ensemble network: DenseNet201, Xception, VGG16, GoogleNet, InceptionResNetV2, and EfficientNetB7. The individual classification performance metrics for each model were recorded and presented in Table 6. The models were trained under the same settings and system setup for consistency. Among these models, DenseNet201 demonstrated superior performance in terms of accuracy, sensitivity, precision, and F1-score compared to the other pre-trained models. Its high performance across multiple evaluation metrics indicates its effectiveness in classifying the dataset.

(2) Ensemble models classification results

Two ensemble learning setups were formulated and executed based on the performance of the previously evaluated individual deep learning models. These ensembles are identified as Ensemble A (consisting of DenseNet201, GoogleNet, and InceptionResNetV2) and Ensemble B (comprising DenseNet201, GoogleNet, and VGG16). The classification efficacy of each ensemble model is also presented in Table 7. Additionally, the evaluation of classification performance encompasses metrics such as accuracy, sensitivity, precision, and F1-score.

(3) The proposed hybrid model classification results

According to the findings of the prescribed ensemble learning inquiry, Ensemble A, comprising DenseNet201, GoogleNet, and InceptionResNetV2, demonstrated superior classification efficacy compared to Ensemble B. As delineated in Table 7, the proposed HWST model exhibited notably enhanced classification proficiency, achieving an overall accuracy of 99.34%. This indicates that the model based on Ensemble A, improves collective classification performance by approximately 0.659% in comparison to the individual Ensemble. The identification evaluation performance of HWST model is detailed in the below.

First, Fig. 7 shows the graph of both accuracy and loss during the training of the model. After the initial epochs, the model seems to reach a state of convergence, where both accuracy and loss do not show significant changes. This suggests that the model has reached an optimal or near-optimal state of learning. The high accuracy and low loss indicate a model that is well-fitted to the data, though the early spikes in validation loss suggest that it might have gone through moments of potential overfitting or instability before stabilizing.

The classification report of HWST model illustrates its performance in classifying brain tumor categories. Across all metrics (precision, recall, F1-score, and accuracy), the model demonstrates high precision values, indicating minimal false positives. The balanced F1-scores show the model’s ability to minimize both false positives and false negatives. For meningioma, the model achieves a precision, recall, and F1-score of 0.9930, indicating that 99.30% of instances classified as meningioma were indeed meningiomas. This high precision is matched by an equally high recall, meaning the model correctly identified 99.30% of actual meningioma cases. The F1-score, which balances precision and recall, is also 99.30%, affirming the model’s robust performance for this class. There were 143 instances of meningioma in the dataset.

Similarly, for glioma the model demonstrates strong precision, recall, and F1-score values of 0.9859. This indicates that approximately 98.59% of instances classified as gliomas were accurate, with the same percentage of actual glioma cases being correctly identified. The F1-score of 0.9859 further confirms the model’s effectiveness in capturing instances of glioma.

For pituitary tumors, the model achieves perfect precision, recall, and F1-score values of 1.0000. This indicates that all instances classified as pituitary tumors were indeed pituitary tumors, with the model correctly identifying all instances of this class. The perfect F1-score underscores the model’s flawless performance for pituitary tumors.

In addition, the ROC curve analysis for all classes yielded an AUC value of 1, as shown in Fig. 8. This perfect AUC score indicates that the model achieved optimal discrimination between positive and negative instances across all classes. Regarding computational efficiency, the total training time for the model was 7,695.43 s, and the total predicting time on unseen data was 17 s.

Finally, the confusion matrix provided in Fig. 9 illustrates the performance of a classification model in identifying three types of brain tumors. The model shows a high degree of accuracy as indicated by the concentration of higher numbers along the matrix’s diagonal, which represents correct predictions. Specifically, it correctly classified 142 meningiomas, 70 gliomas, and 93 pituitary tumors, with only one case each of meningioma and glioma being misclassified.

To further validate the model’s performance and interpretability, we employed Gradient-weighted Class Activation Mapping (Grad-CAM). Grad-CAM generates visual explanations by highlighting the regions of the input images that were most influential in the model’s decision-making process. This technique provides insight into which areas of the brain tumor images contributed to the classification, helping to ensure that the model’s predictions are based on meaningful features and supporting the reliability of its diagnoses. The Grad-CAM visualizations are presented in Fig. 10, illustrating the regions of interest for each tumor type, which corroborate the model’s accurate performance as indicated by the confusion matrix.

The efficacy of our HWST model for brain tumor classification was evaluated against existing methods and demonstrated significantly higher accuracy levels. Table 8 presents the comparative study of brain tumor classification. Our model achieved the highest accuracy among all compared methods when evaluated on the same dataset (Cheng, 2017) with an equivalent number of images (3,064), ensuring a fair comparison of predictive efficiency.

Application and economic benefits for society

This research introduces a methodological approach for early brain tumor detection using MRI image analysis and a self-attention mechanism. The proposed model shows promise in improving the accuracy of tumor identification and classification across various types. It has the potential to significantly impact neuro-oncology by enhancing diagnostic tools and aiding in precise tumor classification. Additionally, this model could advance brain tumor research, assist in image-guided surgeries, and support the development of targeted treatments and therapies.