Use of biplane quantitative angiographic imaging with ensemble neural networks to assess reperfusion status during mechanical thrombectomy

1. INTRODUCTION

Angiographic parametric imaging (API) is a quantitative angiographic tool that uses a digital subtraction angiograph (DSA) to semi-quantitatively analyze blood flow through vessels.[1–6] Time density curves (TDC) are obtained by tracking image intensity at a pixel across the sequence (Figure 1). Parametrization of a TDC results in the generation of several parameters such as mean transit time, time to peak, peak height (PH) and area under the TDC (AUC). Parametrization of TDCs at every pixel in a DSA enables the generation of multiple API maps where each map encodes one specific hemodynamic property (Figure 2) [7]. Moderate correlation between the imaging biomarkers and blood flow conditions makes the interpretation of the individual API maps a challenging task. This challenge may be solved by using a hybrid approach, where hemodynamic information encoded in API maps is combined with deep learning tools such as convolutional neural networks (CNN). Deep learning tools have already been used in the evaluation of neurovascular pathologies, such as stroke assessment with Alberta stroke program early computed tomography score (ASPECT) [8], identification of neurovascular pathologies [9] and using API to make predictions regarding aneurysm treatment success [3, 10]. A challenge to using traditional CNNs with API maps is that they cannot fully utilize the connectivity between the multiple views and maps to aid in decision making. However, intra-procedurally, clinicians use multiple views to make their decision. In addition, different API maps contain complementary information regarding hemodynamic properties. Thus, there is a need for a tool that enables the utilization of complementary information encoded in different API maps from multiple views.

Ensemble learning is a deep learning technique that allows for the utilization of information encoded in multiple sources to provide a decision. Each source is used to train a separate neural network. A weight is calculated for each network based on the performance of the network on the testing set. This weight is indicative of the contribution of the specific network to the final decision and is used to combine predictions from each individual network to give a final comprehensive prediction. An ensemble of networks performs better than single networks as it improves the generalization ability [11]. Such a tool allows for the use of information from multiple sources such as different modalities [12]. In this study, we aim to test the feasibility of using ensemble networks to combine hemodynamic information obtained from two API maps derived from bi-plane angiographic systems (two views), and compare the performance with single view individual API map CNNs. One space where such a tool may be implemented is the intra-procedural evaluation of reperfusion in acute ischemic stroke (AIS) patients.

Large vessel occlusions (LVO) cause AISs and account for up to 46% of all AIS cases [13–15]. LVO AISs are the second leading cause of death after coronary artery disease killing more than 130,000 Americans each year [16]. Treatment of such AISs involve recanalization of the blockage using treatments like mechanical thrombectomy (MT) or intravenous thrombolysis with recombinant tissue-type plasminogen activators [17]. Currently, intra-procedural assessment of each level of reperfusion during MT is achieved through subjective visual assessments of cerebral DSAs, commonly by providing a score on the modified thrombolysis in cerebral infarction (mTICI) scale [18, 19]. This leads to internal inconsistencies [20] as well as inclusion of bias by operators [21].

In previous studies, we have shown feasibility of using ensemble networks to assess level of reperfusion using either multiple view single API maps [22] or multiple API maps from a single view [23]. In this study, we aim to use ensemble networks to combine hemodynamic information obtained from both views of a bi-plane angiographic system as well as from multiple API maps to provide a robust tool that automatically assesses the level of reperfusion in AIS patients undergoing a MT.

2. MATERIALS AND METHODS

Patient data was collected and analyzed at a single center, and was within the scope of a research protocol approved by the institution’s Institutional Review Board. DSAs were retrospectively collected from 223 patients with anterior circulation LVOs undergoing MT to treat an AIS. From each patient, multiple DSAs were collected pre-, post- and intra-procedurally, resulting in a total of 525 DSAs. Presence of motion artifacts in DSAs, and occlusions in the posterior circulation resulted in 142 cases being excluded, thus 383 cases were used for analysis. For each case considered, anteroposterior (AP) and lateral view DSAs were collected.

Two experienced neuro-interventionalists analyzed the AP and lateral view DSAs for each case and assigned a label. Any disagreements in labels were resolved by consensus decision. Operators labeled the data independently of each other, were not involved in the procedure, and were blinded to the API maps and clinical outcomes. Each case was labeled on the mTICI scale which includes the following 6 categories: no perfusion (grade 0), partial perfusion beyond initial occlusion but not in distal arteries (grade 1), partial perfusion less than 50% (grade 2a), partial perfusion more than 50% but less than 100% (grade 2b), complete but delayed perfusion (grade 2c) and complete perfusion (grade 3) [24]. These measures were taken to prevent inclusion of bias in the labels used to train the network.

Assessment of reperfusion during MT is generally conducted based on the level of tissue perfusion during the capillary phase of the DSA [19, 25]. In order to limit the overlapping structures from arterial and venous phases of the DSA, each DSA was temporally cropped to only include the capillary phase and exclude arterial and venous phases. TDCs were created at each pixel and then the peak height and area under each TDC were calculated to obtain PH and AUC maps, respectively. Four datasets were created, AP-AUC, AP-PH, lateral-AUC, and lateral-PH maps. The overall workflow of the study is displayed in Figure 3.

Development of CNNs were done using Keras [26]. The architecture development was an iterative process based on optimizing the accuracy of classifying API maps as having insufficient reperfusion (mTICI 0,1,2a) or sufficient reperfusion (mTICI 2b,2c,3). The final architectures used are displayed in Figure 3. The network architecture was kept the same for each of the 4 datasets. Following the guidelines proposed by Radiology [27], and in order to prevent overfitting of the model, each dataset was split into a training set (70%, 268 cases), a validation set for hyperparameter tuning during training (10%, 39 cases), and a testing set (20%, 77 cases). In addition, a 20-fold Monte Carlo Cross Validation (MCCV) [28] was used to ensure network robustness and that the results obtained were not due to a specific training testing split.

The networks are ensembled by combining outputs from each contributing network into one final output, Figure 3. The combination can be done using different techniques, such as averaging, which assumes equal contribution to the task at hand from each source, thus the outputs from each network are averaged to provide a final output [29]. However, in most cases, each source does not have an equal contribution to the task at hand, and thus each network output needs to be weighted differently. The weights are decimal values between 0 and 1, and are treated like a percentage, such that the sum of the weights for each network equals one. The weights were calculated using two methods; grid search [29], and differential evolution directed optimization [30]. Grid search is a simple, but exhaustive method to finding the optimal weights where a course grid of values from 0 to 1 with a step size of 0.1 is generated, a cartesian product is used to generate all possible combinations of weights for each network. These weights are forced to sum up to one by calculating the sum of the absolute weight values (L1 norm) and dividing each weight by that value. Each combination of weights is tested, and the best performing weight combination is chosen as the final weights to be used for the classification task. The differential evolution directed optimization process is a stochastic method to find the global minimum of the loss function. The optimization was implemented using SciPy [31] and the algorithm is from Storn and Price [32]. A total of 10000 iterations were used with a tolerance of 1E-7. The loss function used in the optimization process was 1 - Matthews correlation coefficient (MCC). MCC is used in machine learning models to evaluate quality of binary classifications [33]. It has proven to be advantageous, as it takes into account class imbalance and uses every factor in the confusion matrix (true positives, false positives, true negatives and false negatives) [34]. Once the weights were calculated, the output from the final dense layer of each network was multiplied by the weight for the network, and the sum of the weighted outputs from each network was the final level of perfusion prediction.

The ensemble network was evaluated using accuracy, receiver operating characteristic (ROC) curves, sensitivity, specificity, and MCC. Each metric was averaged over the 20-fold MCCV. Sub group analysis was done using each of the four networks (AP-AUC, AP-PH, lateral-AUC, and lateral-PH) independently and each of the three ensembling techniques (averaging, grid search, and differential evolution directed optimization). Two-tailed McNemar’s p-test values were calculated in order to evaluate significance of any performance differences (p<0.05).

3. RESULTS

Each API map was created and normalized in 250 ms, each network took an average of 9 minutes to train, the ensembled weights were calculated in approximately 700 ms, and a single case can be classified in approximately 0.6 ms. Results including classification accuracy, area under the ROC curve (AUROC), MCC, sensitivity, and specificity from the four individual networks, as well as the ensembled network, are displayed in Table 1. The results indicate that best performance is achieved when using an ensembled network with weights obtained using either the grid search or differential evolution directed optimization methods.

Two-tailed McNemar’s test were conducted between each of the subsets in order to evaluate significance of any performance differences, these are displayed in Table 2. Grid weighted and differential evolution optimization weighted networks perform significantly better than the averaged network or any of the individual networks (AP-AUC, AP-PH, lateral-AUC, and lateral-PH maps networks). There is no significant difference in performance between the grid weighted and differential evolution optimization weighted networks.

Results of analysis on one specific case from the testing set are displayed in Figure 4. Using just the independent networks we see that the correct classification was obtained by the AP-AUC, AP-PH, and lateral-AUC maps, but not the lateral-PH map. This shows misclassifications can occur by a single network independently, however, when information from both views and multiple maps are combined using an ensemble network, the tool is able to correctly classify the case into the appropriate group. In each CNN classification table, the green highlight indicates the network classification.

4. DISCUSSION

In this study, we established two main findings. First, we demonstrated that the performance of assessing reperfusion in patients undergoing MT to treat an AIS using ensemble networks, to incorporate hemodynamic information from multiple API maps and imaging, is significantly better than using just a single API map from a single view. Second, we established that each single view map has a different contribution to the final decision, and using either grid searched or differential evolution optimization weighting, significantly outperforms simply averaging the outputs from each network.

In order to utilize information encoded in multiple API maps from different views, we trained each single view API map (AP-AUC, AP-PH, lateral-AUC, and lateral-PH) on a separate network and ensembled their outputs to provide a single final classification using different techniques (averaging, grid searched weights, and differential evolution optimization weights). Since each of the input maps were the same size and the vasculature dictating the level of reperfusion within each map were the same size, we used the same network architecture for each of the four maps. A 20-fold MCCV was used to ensure the robustness of results. In each fold of the MCCV, the data was split into a training set (70%, 268 cases), validation set for hyperparameter tuning during training (10%, 39 cases), and a testing set (20%, 77 cases) to ensure none of the networks were overfit. The results in assessing the level of reperfusion on the independent testing set for each of the four individual networks, as well as the ensembled networks are given in Table 1. Networks ensembled with grid searched or differential evolution optimization weights outperformed the other networks in every metric. MCC values were greater than 0.6, and this indicates strong positive relationship between the classifications and the ground truth labels [35, 36].

The two-tailed McNemar’s test was conducted to evaluate significance of difference in performance between the subgroups. The p-values are displayed in Table 2. It is observed that the ensemble networks with grid searched and differential evolution optimization weights significantly outperform all of the individual map networks as well as the average ensemble network (p-value < 0.05). There is no significant difference in performance between grid searched and differential evolution optimization networks (p-value > 0.05).

Previous studies have been conducted to automatically assess the level of reperfusion using API maps, however, these either only used two API maps from a single view [23] or used single API maps from two views [22]. These studies reported accuracies of 79.6% and 81.0%, AUROCs of 0.85 and 0.86, and MCCs of 0.59 and 0.62, respectively. With an average accuracy of 83.0%, AUROC of 0.86, and MCC of 0.66, the method used in this study performed better than those in the previous studies. This method also incorporates additional information and thus is potentially more robust in assessing level of reperfusion in patients undergoing a MT to treat an AIS.

There were limitations to this study. First, we are limited to only demonstrating a technical feasibility of our method due to the size of our monocentric dataset. Second, we are only classifying cases as having sufficient (mTICI 2b,2c,3) or insufficient (mTICI 0,1,2a) reperfusion, instead of classifying each case into the specific mTICI class. This was done due to the low number of cases per mTICI grade (mTICI 0 – 82, mTICI 1 – 5, mTICI 2a – 82, mTICI 2b – 140, mTICI 2c – 48, mTICI 3 – 26) which leads to a drastic decrease in performance. These two limitations may be addressed by collecting additional cases from multiple centers. Lastly, we are currently only using PH and AUC maps, other maps such as time to peak may be used as they contain additional information that may aid in providing a finer classification instead of just two classes.

Such a tool that uses multiple sources of information to assess level of reperfusion can be a valuable resource for neuro-interventionalists. Instead of having to rely on the potentially subjective visual assessment of angiograms, this tool can intra-operatively assess the level of reperfusion and aid the clinician. In addition, this study proves that quantitative angiographic information can be used with such deep learning techniques to make decisions and aid clinicians. This opens the door to making decisions on other treatment outcome scales and for other endovascular interventions such as aneurysm treatment outcome predictions. This tool can also be used as a training tool for medical students instructing them how to score angiograms on the mTICI scale.

5. CONCLUSION

In this technical feasibility study, we developed an ensemble neural network that uses different quantitative angiographic information from multiple views to automatically assess the level of reperfusion in patients undergoing a MT to treat an AIS. Best performance was achieved when using the grid search technique to calculate weights of each contributing map, as it achieved an accuracy of 83.0%, AUROC of 0.86, MCC of 0.66, sensitivity of 0.90, and specificity of 0.74. This study has demonstrated the ability to combine hemodynamic information incorporated in multiple quantitative angiographic maps and views to assess the level of reperfusion during MT procedures. The improvement in performance was significant, and opens the door to using such tools for other endovascular interventions.

Supplementary Material