Detecting myocardial scar using electrocardiogram data and deep neural networks

Data pipeline

Data for both training and application of the trained models is supplied through a data pipeline consisting of several preprocessing steps arranged in two separate branches for ECG recordings and for clinical parameters, respectively (Figure 1).

Figure 1:

Overview of the data pipeline for preprocessing.

First, ECG recordings stored in XML HL7v3 format (Health Level Seven, Inc. 2020) are loaded in which then, artefacts, e.g. zero lines at the beginning and end of the recordings, are removed by cropping. Then the values are scaled to locate them in a narrower range, which makes them more accessible for deep learning algorithms. In the next step, data augmentation procedures (subsampling) are applied to compensate for the relatively small number of training records before data is saved in the last step of the data pipeline.

Parallel to preprocessing the ECG recordings, clinical parameters run through a second branch of the pipeline. After initial processing, i.e. loading and categorisation, attributes are transformed into one-hot encoded vectors which are then concatenated into a single vector to be used as input for the neural network.

The pipeline provides data for two experimental setups in which models for an ECG-based model and for a combined model based on ECG data and clinical parameters are examined. Both strands with preprocessed ECG samples and encoded clinical parameters can be combined using an anonymised distinct identifier (record-ID). This enables to link clinical parameters to the corresponding ECG recording so that these two types of data can either be used together or separately.

ECG model

CNNs have originally been designed for pattern identification in images (LeCun et al. 1989) and have already been used for ECG analysis, yielding considerable performance for specific tasks (e.g. Strodthoff and Strodthoff (2019)). Therefore, CNN are a promising architectural choice for the task of detecting MS in ECG recordings.

As starting point of our DL model, we used the CNN architecture as proposed by Strodthoff and Strodthoff (2019), which was developed for identification of acute MI. The following characteristics of the architecture were optimised for our specific medical purposes by grid search optimisation:

1. Number of convolutional layers

2. Number of filters in convolutional layers

3. Kernel/filter size of convolutional layers

4. Pooling (yes/no) after convolutional layers

5. Additional dropout between pooling and convolutional layers with varying dropout rates

Furthermore, we extended the initial starting configuration with additional fully-connected dense layers, experimentally adjusting their number, width, and the respective dropout rates.

The final optimised CNN (Appendix, Figure 3) consists of an input layer (width = 2000) followed by a successive series of convolutional and pooling layers reducing the input to a width of 64. In between these layers, dropout with a rate of 0.25 proofed to successfully prevent the model from overfitting. Four fully-connected layers (width = 64), each followed by a dropout rate of 0.1, aggregate the information into a final layer. This layer yields a probability distribution over two classes (presence of MS versus absence of MS) based on a softmax activation.

Combined model

Besides the experiments with ECG recordings alone, we investigated whether using ECG data together with clinical parameters of the individual patient in a combined DL model could improve diagnostic performance. Therefore, the CNN model described before was combined with a fully-connected feedforward network (FFN) which was applied to the records with clinical parameters (Appendix, Figure 4). The characteristics of this network were optimised analogously to that of the CNN. The final FFN structure has an input layer of 55 nodes followed by four fully-connected layers (width = 256). All layers but the first are connected with dropout rates of 0.5.

The outputs of the CNN and FNN form the 4-node input for a final FFN with three hidden layers (width = 4), each having a dropout rate of 0.25. As for the ECG-only model, the final output layer of the combined model contains two nodes for the overall prediction (presence of MS, absence of MS) and uses a softmax activation. The combined model was trained in a separate end-to-end training process and does not rely on pre-trained sub-models.

Experimental results

We now present performance measures for both models. Table 1 contains areas under the curve (AUC) of receiver operating characteristic analyses, sensitivities, specificities, and accuracies of the ECG-only (listed in the columns on the left hand side) and the combined model (listed on the right hand side). The measures were determined on subsample-level and on patient-level applying 6-fold cross-validation with 50 repetitions for each split, see paragraph Performance metrics of section Materials and methods.

In summary, the ECG model achieved a mean AUC, sensitivity, specificity, and accuracy on patient-level of 0.81 ± 0.06, 70.0 ± 2.4, 73.0 ± 20.7, and 70.2 ± 7.9%, respectively. The mean AUC, sensitivity, specificity, and accuracy on patient-level of the combined model are 0.89 ± 0.06, 70.0 ± 20.4, 84.3 ± 13.9, and 78.0 ± 7.0%, respectively.

With an outlook to a potential clinical application, the positive predictive value (PPV) and negative predictive value (NPV) of the ECG model would be 73.0 ± 17.5 and 72.3 ± 11.3%, respectively. The PPV and NPV of the combined model would be 84.2 ± 12.2 and 78.2 ± 14.0%, respectively.

The results of both models are discussed comparatively and related to alternative approaches in the following section. Graphs of the receiver operating characteristic (ROC) analyses can be found as Figure 5 in the Appendix.

Model performance

Our results presented in Table 1 show that combining ECG data and clinical parameters produced the best average performance over the six cross-validation runs with respect to each of the measures (AUC = 0.89, Sensitivity = 70%, Specificity = 84.3%, and Accuracy = 78%). The same holds for the maximal values and for the results of the single runs. Apparently, supplementary information in the form of clinical parameters can improve model performance.

We observed high measures for AUC, sensitivity and specificity but not necessarily in the same models. The model with the highest AUC can be seen as the model with the best compromise between specificity and sensitivity and depending on the application, models selected based on maximum sensitivity or maximum specificity can be more appropriate. Our approach computes models with good performance for all three perspectives.

The area under the receiver operating characteristic curve was used as performance measure as this statistical method is well established in medical studies evaluating diagnostic tests. With specific clinical requirements, such an unweighted compromise might not be ideal. For example, within a diagnostic rule-out approach optimal sensitivity is aimed, whereas in case of a diagnostic rule-in algorithm, a high specificity is of greater importance.

Due to the relatively small case number of n = 114, we augmented data by subsampling and applied 6-fold cross-validation. Therefore, our validation datasets contained data from 19 patients in each cross-validation run so that validation was performed with n = 1900 on subsample-level and n = 19 on patient-level. Predictions on subsample-level were condensed into patient-level predictions by computing the mean of all the subsample predictions of the corresponding patient. Surprisingly, standard deviation of model performance on subsample-level is close to that on patient-level despite the different size of the validation datasets. Furthermore, there is no significant difference between the model performance on subsample-level and on patient-level, indicating that subsampling together with accumulating predictions to patient-level is viable.

More elaborate statistical evidence, e.g. confidence intervals or test measures, cannot be reported as the influence of fluctuations is too strong for the sample size given (see discussion in Isaksson et al. (2008) and Bengio and Grandvalet (2004)). We therefore applied cross-validation with repeated training and validation (50 times for each of the six split-variants), reporting mean and variance values for sensitivity, specificity, accuracy, and AUC.

Using accuracy must be handled with care as it is considered to fail as performance metric for imbalanced class distributions. Table 3 shows that patients with LE and those without are almost equally distributed so that accuracy is applicable in our case.

Comparison to existing approaches

Table 2 relates the performance of our models to that of applying the Selvester score (Selvester et al. 1971; Selvester et al. 1985), human-based ECG evaluation, and two SVM-based approaches. The measures of both our models deliver comparably high performance measures and reach (ECG-based model) or exceed human performance (combined model) as published by Asch et al. (2006), Carpenter et al. (2015), and Markendorf et al. (2019).

Comparing our results with that of the other methods listed in Table 2, it must be taken into account that the underlying cohorts differ in size, structure and time of study. Considering the larger training sets used by Goovaerts et al. (2019) and by Dima et al. (2013) and the limited number of cases used by Chaudhry et al. (2017) and Bignoto et al. (2018), our results are similar. For a more substantiated comparison, the methods would have to be applied to the identical data, see Bengio and Grandvalet (2004). It is noteworthy that only Goovaerts et al. (2019) uses AUC as performance measure which is of high relevance in the medical context and is the only measure that does not depend on a single threshold.

In summary, our proposed model achieves reasonable performance on a relatively small case number used for training, which is a promising result, encouraging further research and validation.

Limitations

Over recent years, DL has frequently been used in the medical context. A recent meta-analysis has shown that existing DL models detecting diseases by analysis of medical images achieve diagnostic performance equivalent to that of health-care professionals (Liu et al. 2019). However, only 25 studies out of 31587 under investigation could be included in this meta-analysis. The rest had to be left out for reasons such as missing external validation or lack of transparency. Limited amount of appropriate data is often one of the main obstacles for a comprehensive use of DL models in general and particularly in healthcare.

In case of the approach presented in this article, the limited number of 114 patients and the heterogeneity within the data also constituted a key limitation for model development. Due to this limitation, we exclusively performed an internal validation (in-sample validation) using 6-fold cross-validation, see Altman and Royston (2000) for a description of different types of validation sets. An external validation (out-of sample validation) with data derived from new, previously unknown patients is currently missing and will be critical to ensure the high performance measures achieved in this study.

It should further be mentioned that the data partly includes redundant information, e.g. different leads of the ECG may contain similar patterns, or that the clinical parameters body-mass-index (BMI) and body-surface-area (BSA) are not completely independent.

Furthermore, all data used for training and validation was derived from patients scheduled for an MRI examination. This means that ECG recordings, MRI diagnosis, and clinical parameters used in our approach were obtained in patients with a relevant risk for cardiovascular diseases. It is uncertain whether the performance achieved for the cohort used in this study can be maintained for a putative healthy cohort.

Another limitation is the black box nature of the DL model proposed in this study. We currently do not elucidate which patterns of the ECG or features of the clinical parameters contribute to the final decision of the DL model and how this decision is reached.

Future work

Currently, enrolling patients and therefore collecting further medical data is an ongoing process of our project and will allow us to use a more comprehensive dataset for model training and hyperparameter optimisation in the future. A larger set of medical data will also enable to hold-out data for external validation. Therefore, our main focus will be to test the generalisation performance of our DL model by reproducing the promising performance measures achieved in the presented study for this out-of-sample data. Furthermore, an interesting aspect for validation will be the comparison of the diagnostic performance of our DL model with established diagnostic approaches used in daily clinical routine.

Another important issue for future work is to elucidate the specific ECG patterns and relevant clinical parameters leading to the diagnostic prediction of the DL model. Potential methods from the emerging field of explainable artificial intelligence (XAI) such as layer-wise relevance propagation (LRP) (Bach et al. 2015), local interpretable model-agnostic explanations (LIME) (Ribeiro et al. 2016) and deep learning important features (DeepLIFT) (Shrikumar et al. 2017) may provide transparency and traceability and thereby dissolve the black-box nature of the DL model. Such methods can further facilitate the use of DL methods in clinical routine due to an increase of trust (Andras et al. 2018).

Medical data

Data on clinical parameters, ECG and MRI are used from 114 patients enrolled in the Kerckhoff Biomarker Registry for the training and evaluation of the deep neural networks.

Algorithmic procedure

In the following subsection, we describe in detail components of the data pipeline presented within section Results. For an overview of the data pipeline, see Figure 1. Furthermore, we present the performance metrics used for evaluation of the DL models and provide details about the training procedure.

Augmentation

In order to prevent the neural network from overfitting on the comparably small dataset of 114 records, we increased the amount of unique ECG samples by data augmentation via subsampling. This is medically justified as the presence of MS and the thereby associated impaired conductivity of the myocardium should lead to permanent changes in the shapes of the ECG waveforms present in all subsamples. Therefore, shorter samples than a standard of 10 s contain features relevant for distinguishing pathological ECG with patterns caused by MS from those without such patterns.

In order to capture at least two heartbeats in each sample, a subsample size of 4 s was used. This approach has already been used previously (Strodthoff and Strodthoff 2019). Figure 2 illustrates the subsampling process used in this work. Subsampling generates new, distinct, non-identical samples from an existing sample using equidistant strides of an extraction window to extract overlapping subsamples. The proportion of overlap varies according to the window size w, the sample size s and the subsampling factor f. based on these parameters, the stride distance d is defined as

d = ⌊ s − w f ⌋   ,

where, f ∈ ℕ > 0 .

Figure 2:

Subsample extraction from 12-lead ECG. An extraction window of size w is slided over the ECG sample of size s in equidistant strides of size d.

The clinical parameters for each subsample were copied from the original dataset sample without further adaption. We used a window-size of 2000 timesteps and a subsampling factor of 100. The stride distance varied between 33 and 150, depending on the size of the cropped samples, which were in the range of 2664–5000 time steps.