. Author manuscript; available in PMC: 2022 Mar 14.

Published in final edited form as: Proc SPIE Int Soc Opt Eng. 2020 Mar 16;11315:1131513. doi: 10.1117/12.2549066

Automatic A-line coronary plaque classification using combined deep learning and textural features in intravascular OCT images

Juhwan Lee ^a, Chaitanya Kolluru ^a, Yazan Gharaibeh ^a, David Prabhu ^a, Vladislav N Zimin ^b, Hiram Bezerra ^b, David Wilson ^a,^c

PMCID: PMC8920332 NIHMSID: NIHMS1778604 PMID: 35291576

Abstract

We developed a fully automated method for classifying A-line coronary plaques in intravascular optical coherence tomography images using combined deep learning and textural features. The proposed method was trained on 4,292 images from 48 pullbacks giving 80 manually labeled, volumes of interest. Preprocessing steps including guidewire/shadow removal, lumen boundary detection, pixel shifting, and noise reduction were employed. We built a convolutional neural network to extract the deep learning features from the preprocessed image. Traditional textural features were also extracted and combined with deep learning features. Feature selection was performed using the minimum redundancy maximum relevance method. Combined features were utilized as inputs for a random forest classifier. After classification, conditional random field (CRF) method was used for classification noise cleaning. We determined a sub-feature set with the most predictive power. With CRF noise cleaning, sensitivities/specificities were 82.2%/90.8% and 82.4%/89.2% for fibrolipidic and fibrocalcific classes, respectively, with good Dice coefficients. The classification noise cleaning step improved performance metrics by nearly 10–15%. The predicted en face classification maps of entire pullbacks agreed favorably to the manually labeled counterparts. Both assessments suggested that our automated measurements gave clinically relevant results. The proposed method is very promising with regards to both clinical treatment planning and research applications.

Keywords: Intravascular optical coherence tomography, plaque classification, combined features, convolutional neural network, textural features, random forest

1. INTRODUCTION

Atherosclerosis is a chronic disease which results in plaque deposition in the arterial wall and remodeling and narrowing of the arteries. Coronary plaques are the important markers for different kinds of cardiovascular outcomes and still remain a serious challenge during intervention. The most widely used interventional approaches are percutaneous coronary intervention and coronary artery bypass graft surgery.¹ The choice of appropriate treatment typically relies on the morphology of the vessel wall and plaque distribution in the artery.²

Intravascular ultrasound (IVUS) is an intracoronary imaging modality that provides real-time tomographic views³ and has been widely utilized to analyze plaque morphology to quantify tissue components. Particularly, virtual histology (VH)-IVUS is able to identify four tissue types, such as fibrous tissue, fibro-fatty tissue, necrotic core, and dense calcium, using the reflected ultrasound radio frequency signals.⁴ Despite these advantages, IVUS has a limited axial resolution ranging from 150 to 200 μm and a lateral resolution ranging from 150–300 μm. Another well-known limitation of VH-IVUS lies in its electrocardiogram-gated acquisition. Due to this limitation, only one VH frame can be acquired in each cardiac cycle while recording 30 gray scale images. Intravascular optical coherence tomography (IVOCT) is a catheter-based imaging modality providing cross-sectional images of coronary arteries with high contrast and high resolution using a near-infrared light.⁵ Compared to IVUS, IVOCT has improved penetration depth for calcifications and better resolutions of 12–18 μm and 20–90 μm for axial and lateral directions respectively.⁵ IVOCT is a promising modality for analyzing the inner parts of the vessels and has been proven to be successful for identifying vulnerable thin cap fibroatheroma.⁶

Our group has proposed multiple plaque characterization approaches on IVOCT image volumes, including machine and deep learnings. Kolluru et al.⁷ classified each A-line of raw IVOCT images in polar domain into either lipid, calcium, or other classes using a simple convolutional neural network (CNN). Prabhu et al.⁸ suggested a machine learning method to identify coronary plaques using a comprehensive set of hand-crafted features. Particularly, they extracted novel lumen morphology, optical, digital edge, and texture features, and obtained high sensitivities of 81.4% and 94.5% for fibrolipidic and fibrocalcific tissues respectively. Gharaibeh et al.⁹ utilized the advanced CNN model (SegNet) for identifying calcified tissues from IVOCT images. They measured calcification score based on various clinical attributes to determine the impact of calcifications on stent deployment. Lee et al.¹⁰ proposed a fully automated plaque characterization method using a deep learning model to identify lipidous and calcified tissues. Most recently, Lee et al.¹¹ developed an automated method for classifying coronary plaques based on hybrid processing.

In this study, we propose the fully automated A-line-based classification method for predicting coronary plaques using deep learning and textural features. In order to prevent possible data loss, all image processing procedures are performed on the raw IVOCT images in the (r,θ) domain rather than the (x,y) domain. After image processing including pre-processing, feature extraction, classification, and post-processing, results are converted to (x,y) domain for better visualization.

2. METHODS

The proposed method is composed of five main parts including pre-processing, feature extraction, feature selection, classification, and post-processing. First, a series of pre-processing steps including guidewire and shadow removal, pixel-shifting, and speckle noise reduction were applied to raw IVOCT images in the (r,θ) domain. The CNN and textural features were subsequently extracted from the resulting image, and the optimum feature set was determined. After feature selection, each A-line was classified as a relevant tissue type and the classification errors were compensated using the conditional random field (CRF) method. Classification results were quantitatively validated using 5-fold cross validation. Figure 1 shows the overall classification procedure of the proposed method.

Figure 1. — Overall workflow of the proposed method for A-line-based plaque classification. The proposed method has two slightly different pipelines for training and testing sets. Pre-processing is applied to both the training and testing sets, but feature selection and optimization are only performed during training process. For testing set, only the selected features are extracted and used as the input for trained classifier.

2.1. Pre-processing

From an accumulated intensity map generated by adding all the pixel values of each A-line, the shadow regions have two easily distinguishable boundaries (upper and lower).¹² In order to detect these boundaries, we applied dynamic programming¹³ for all image frames. The guidewire and corresponding shadow were then removed from all IVOCT image frames. The lumen boundaries were detected using dynamic programming¹³, and each A-line was pixel-shifted to the leftmost so that all rows have the same starting pixel of lumen boundary. Typically, IVOCT signal is propagated approximately 1–2 mm from the vessel wall. Therefore, we only considered up to 200 pixels (~ 1 mm) to investigate reliable information. Through this process, the IVOCT image (968 × 496 pixels) was cropped to 200 × 496 pixels without any data loss. Lastly, in order to effectively reduce speckle noises and highlight the tissue boundaries, non-local mean filtering was implemented on the resulting images. The sizes of the search window and comparison window were 21 and 5 respectively. The degree of smoothing was set to the standard deviation of noise estimated from the image.

2.2. Feature extraction

2.2.1. Extraction of deep learning features

The CNN model was composed of three pairs of convolutional and ReLU layers, two maximum pooling layers, and three fully connected layers. For each convolutional layer, the filters with varying kernel sizes of 11, 9, and 7 pixels created corresponding feature maps when it went through the input image with a stride of two pixels. In our model, each convolutional layer included 32, 64, and 96 filters respectively. ReLU activation was implemented to keep non-negative values and to replace negative values with zero to accelerate the training process.¹⁴ The maximum pooling layer with a pool size of two pixels was then used for dimensionality reduction by finding maximum value of each kernel. The output of the final convolutional layer was connected to the three fully connected layers. The first two fully connected layers had 100 output units, while the last layer only included three units corresponding to the three classes of interest. To extract the deep learning features, we removed the last classification layer after training the CNN model and utilized the activations of the previous fully-connected layer which had 100 CNN features as the training input for a new classifier.

2.2.2. Textural features

a). First order statistics

The first order statistics (FOS) are one of the most widely used traditional families of textural features. Despite the simplicity, these features are powerful indicator for detecting spatial association of local patterns. In this study, the mean, skewness, kurtosis, variance, and standard deviation were extracted from a 200×10 window mask. The mean value was also measured from a single A-line having the size of 200 × 1 pixels. In addition to these traditional FOS features, we estimated global intensity threshold minimizing the intra-class variance using Otsu’s method.¹⁵

b). Fractal dimension

The concept of the fractal dimension (FD) is based on the complexity of the fractal patterns which can be expressed as the ratio of the change in detail to the change in scale.¹⁶ Coronary plaques often present some level of self-similarity within the local lesion. Therefore, the FD would be a good indicator for analyzing the irregularity of IVOCT images. FD was computed using the box counting method.

c). Neighborhood gray-tone difference matrix

The neighborhood gray-tone difference matrix (NGTDM) provides the difference between the intensity level of a current pixel and the average of surrounding neighbors. For the image intensity value I(x,y) at a certain position (x,y), the average intensity over the square neighborhood centered at (x,y) is described by:

f (x, y) = \frac{1}{(W - 1)} \sum_{m = - K}^{K} \sum_{n = - K}^{K} I (x + m, y + n)

(1)

where (m,n)≠(0,0), K indicates the window size and W = (2K+1)². The i^th element of the NGTDM for all pixels is given by:

s (i) = \sum_{x = 0}^{M - 1} \sum_{y = 0}^{N - 1} | i - f_{i} |

(2)

For a given window size, the distance of the neighborhood was set to 1 pixel to consider relatively small plaque regions. In this study, five features were estimated from the NGTDM: coarseness, contrast, busyness, complexity, and texture strength.¹⁷ The details of these features are referred to in ^18–20.

d). Gray-level run length matrix

The gray-level run length matrix (GLRLM) features are effective descriptors of textural images, since the coarse textures present relatively long gray-level runs, while the fine textures have short runs due to frequent gray-level variations.²¹ These features strive to capture the relation between gray-level values appearing along pixel sequences. From the GLRLM, the following 7 features were extracted from a given window mask: short run emphasis (SRE), long run emphasis (LRE), gray-level non-uniformity (GLN), run-length non-uniformity (RLN), run percentage (RP), low gray-level run emphasis (LGRE), and high gray-level run emphasis (HGRE). All features were calculated in terms of four different angular directions (0°, 45°, 90°, and 135°) and averaged. The quantization level was set to 16.

e). Local binary pattern

The local binary pattern (LBP) is a structural measure to represent uniform texture patterns by comparing the intensity of the central pixel with the intensities of a circular neighborhood.²² These features are based on a circular symmetric neighborhood of P members of a circle with radius R. Due to the circular symmetry of the neighborhood, LBP features are rotational invariant. In our experiment, we used the uniform rotation invariant (URI)²², which is given as follows:

U R I_{P, R} = {\frac{\sum_{P = 0}^{P - 1} s (G_{P} - G_{C}), i f U (L_{P, R}) \leq 2}{P + 1, O t h e r w i s e}

(3)

where G_c and G_p are the intensities of the central pixel and the p^th neighboring pixel, respectively. U indicates the number of transitions between 0 and 1 and vice versa, and L_P,R denotes the binary pattern. The function s(x) was 1 when the value x is equal to or greater than 0, otherwise s(x) = 0. The member of P was set to 8 and radius R was set to 1. Consequently, we extracted a total of ten LBP features (LBP₀ - LBP₉) according to the above procedures.

f). Gray-level difference matrix

The gray-level difference matrix (GLDM) represents the first order statistics of local property values describing the size and prominence of textural elements in an image. These features are based on the probability density function (PDF) of gray levels which is calculated from a subtracted image. From the obtained PDF, four forms are typically available: (0, d), (d, 0), (−d, d), and (−d, −d). Considering relatively small plaque regions in the pre-processed image, the sample distance d was set to 1. In our experiment, the five GLDM features were extracted, which are mean, contrast, entropy, angular second moment, and inverse difference moment.²³

g). Gray-level co-occurrence matrix

The gray-level co-occurrence matrix (GLCM) features are the second-order statistical descriptors which are computed from angular nearest-neighbor spatial-dependence matrices.²⁴ These features describe the spatial interrelationships of the gray levels along a specific orientation θ and distance d. The distance d was set to d=1, since the regions of interest were relatively small. The orientation angles were set to 0°, 45°, 90°, and 135°, and these were averaged to ensure rotational invariance. From the co-occurrence matrix, 19 features were extracted: autocorrelation, contrast, correlation, cluster prominence, cluster shade, dissimilarity, energy, entropy, homogeneity, maximum probability, sum of squares, sum average, sum variance, sum entropy, difference variance, difference entropy, information measure of correlation, maximal correlation coefficient, and inverse difference normalized.

h). Law’s texture feature

Law’s texture (LT) feature was used to characterize various types of texture features based on one-dimensional vectors comprising L5 (Level) = [1,4,6,4,1], E5 (Edge) = [−1,−2,0,2,1], S5 (Spot) = [−1,0,2,0,−1], and R5 (Ripple) = [1−4,6,4,−1].²⁵ These vectors were multiplied to form a set of two-dimensional image kernels used as filters: L5E5/E5L5, L5R5/R5L5, E5S5/S5E5, S5S5, R5R5, L5S5/S5L5, E5E5, E5R5/R5E5, and S5R5/R5S5. Input images were convolved with these kernels so that a set of filtered images were created to remove certain symmetric pairs. Each kernel was formed from orthogonal matrices and was subsequently averaged to provide rotational invariance. In many previous studies^26–28, the sum of absolute value/the number of pixels (SAV) and the sum of squared values/number of pixels (SSV) have produced the outstanding discrimination performance. SAV and SSV were calculated from each of the nine convolution kernels, thereby obtained 18 LT features, which are SAV (L5E5/E5L5), SAV (L5R5/R5L5), SAV (E5S5/S5E5), SAV (S5S5), SAV (R5R5), SAV (L5S5/S5L5), SAV (E5E5), SAV (E5R5/R5E5), SAV (S5R5/R5S5), SSV (L5E5/E5L5), SSV (L5R5/R5L5), SSV (E5S5/S5E5), SSV (S5S5), SSV (R5R5), SSV (L5S5/S5L5), SSV (E5E5), SSV (E5R5/R5E5), and SSV (S5R5/R5S5). Therefore, we extracted a total of 72 textural features in IVOCT images.

2.2.3. Textural feature selection using principal component analysis

Typically, the textural features present different outcomes according to the types of images, target lesions, and so forth. In order to reduce the dimensionalities and select the best features from all textural feature sets, principal component analysis (PCA) was performed followed by a varimax rotation. This method is able to effectively reduce the dimensionality without a significant performance loss, since the selected features are linearly correlated with the original feature sets. The first principal component describes the direction of the maximum variance of the features, and the second component represents the next largest amount of variance. Therefore, PCA can condense most of the statistically important information into the first few components, while other components only include noises. In this study, we used the primary principal component to select the best texture features from IVOCT images. The selected textural features were then combined with CNN features.

2.3. Feature optimization using minimum-redundancy maximum relevancy

Not all of the combined features are informative for plaque identification, and some may negatively influence the classification performance. To select a subset of the most relevant features, minimum redundancy maximum relevance (mRMR) criterion was used on the combined feature sets. The mRMR approach ranks the importance of the features according to the relevance to the target and the redundancy between themselves.²⁹ The ranked feature with a smaller index means a better trade-off between the maximum relevance and minimum redundancy. We determined the statistical dependency using mutual information to find a maximally relevant and minimally redundant set of features. This procedure allowed the ranked features having the maximum dependency. Subsequently, we divided all the combined features into 10 subsets using the mRMR method and quantitatively evaluated the classification performance to select optimum feature sets.

2.4. Classification model

Random forest (RF) is a combination of classification trees in which each tree is grown on a bootstrap sample of the training data and casts a vote for the prediction of given data set. RF not only works efficiently on large data sets with a low risk of overfitting, but also provides robust performance for noisy data.³⁰ This classifier is also able to estimate the importance of features in the classification. The basic unit of RF tree is typically grown using the CART³¹ methodology. More specifically, RF trees are grown non-deterministically based on two stages of randomization procedure. Grown trees are not pruned.^32,33 The main purpose of these procedures is to separate each tree to ensure low variance during training. At the beginning, each tree is introduced by a bootstrapped version of the original data, and the second stage of randomness is added when growing the tree. RF uses only a small subset of predictors selected at each node as a candidate to determine the best split.³⁴ The final classification result is determined by voting over all trees in the forest.

2.5. Post-processing using fully connected conditional random field

Local minima in training and noise in the input images can cause false predictions with small isolated regions or holes in the results because single A-line does not reflect the spatial similarities. To compensate for this limitation, a fully connected CRF³⁵ that models the conditional probability of a set of latent variables was implemented to each classification. CRF assigns a new label, which has more similar spatial characteristics to surrounding pixels, based on the probability estimates. The CRF method is very useful for modeling large pixel-neighborhoods and ideal for plaque classification in IVOCT images.

3. EXPERIMENTAL METHODS

3.1. Image acquisition

IVOCT images were collected on a frequency-domain ILUMIEN OCT system (St. Jude Medical Inc., St. Paul, Minnesota, USA). The system was equipped with a tunable laser light source sweeping from 1,250 to 1,360 nm at a frame rate of 180 fps. Imaging pullback was performed with a pullback speed of 36 mm/s, and an axial resolution of 20 um. A total of 4,292 image frames were acquired from 48 pullbacks with 80 lesion volumes of interest (VOIs) having calcification (32 VOIs), lipidous regions (36 VOIs), and both (12 VOIs). All IVOCT images were manually labeled by two skilled cardiologists from the Cardiovascular Imaging Core Laboratory, Harrington Heart and Vascular Institute, University Hospitals Cleveland Medical Center, Cleveland, OH, USA, hereafter called Core Lab. According to our previous paper⁷, A-line labels of fibrolipidic, fibrocalcific, and other classes were determined by counting the number of pixels in A-line in each class. Consequently, we obtained the ground truth consisting of nearly 390,000 fibrolipidic, 550,000 fibrocalcific, and 1,190,000 other classes.

3.2. Network training for feature extraction and plaque classification

The CNN model used for feature extraction was optimized using adaptive moment estimation (ADAM) optimizer³⁶ with an initial learning rate of 0.001. For ADAM optimizer, the drop factor and drop period were set to 0.2 and 5 respectively. Since the training data set was unbalanced, we computed the class weights for each class as the inversed median frequency of class proportions and utilized them to weight the loss function. The energy function was computed using a cross-entropy loss function over the softmax outputs. To prevent overfitting of the network, the training was set to stop when the validation loss did not improve over 5 epochs or when the training was performed for 100 epochs. The size of the receptive field was 77×1 pixels. Since the proposed method only considers the first 200 pixels from the lumen boundary along r axis, we can easily avoid the edge effect which may cause potential misclassification by using parameterized paddings (5×1 pixels) on the left and right edges. For each pullback, the left padding was set to zero, and the five pixels after region of interest in r axis were used for the right padding.

The number of trees and the number of variables are the two main parameters of the RF classifier. Using bootstrap sampling, the bootstrap samples are drawn from the two-third of the training data set, while the rest, called the out-of-bag (OOB) data, is omitted when building each tree so as to evaluate the performance of the classifier and to build measures of importance. The optimum number of trees was determined using the OOB error rate. Additionally, it is very important to select the optimum number of variables to provide sufficiently low correlation with adequate predictive power. Breiman et al.³² reported that the highest classification performance is typically obtained when using the number of variables equal to the square root of the number of trees. We adopted this approach in our training process. For example, when the number of trees was set to 100, the number of variables was set to 10. All image processing procedures were performed using a MATLAB software package (R2018b, MathWorks Inc.) on a NVIDIA GeForce GTX 1080 Ti GPU with 64GB of RAM installed in a Dell Precision T7600.

3.3. Performance validation

Five-fold cross validation was carried out to validate the classification performance of the proposed method. Classification performance was quantitatively evaluated in terms of sensitivity, specificity, and Dice coefficient. Sensitivity characterizes the relevant tissues and specificity determines the ability to detect non-relevant tissues. Dice coefficient measures how similar the predicted results and ground truth data are in terms of plaque distribution.³⁷

4. RESULTS

After PCA feature selection, a total of 39 textural features including FOS, intensity, NGTDM, GLRLM, LBP, and LT were selected from the original feature set (Table 1). Selected textural features were combined with pre-determined CNN features. Therefore, we composed 139 combined features including 39 textural and 100 CNN deep features, and these were divided into 10 sub-feature sets based on the mRMR optimization approach. Figure 2 shows mean metrics of the fibrocalcific plaque for different sub-feature sets after CRF noise cleaning. Interestingly, classification results were not significantly improved as the number of features increased. Although the standard deviations of Dice were very low (0.005 for fibrolipidic, 0.002 for fibrocalcific, and 0.001 for other classes), the sub-feature set 1 having 9 CNN features and 5 textural features showed the best classification results among all sub-feature sets. Additionally, we divided the sub-feature set 1 into 14 subsets and quantitatively compared results to find the most indicative features. Subsets 1 and 2 were not able to detect fibrolipidic tissue, while subset 3 showed a remarkable improvement (Fig. 3). Although the number of features increased, classification results were not significantly improved after subset 3. This result indicates that the combined features used in this study are very sensitive to each tissue. Consequently, we selected the six features of CNN 71, GLRLM (RP), CNN65, SAV (S5S5), CNN 91, and SSV (S5R5/R5L5) as the best subset and performed further image analysis.

Table 1.

Selected textural features. A total of 72 original textural features were reduced to 39 features using PCA method. FD, GLDM, and GLCM features were all excluded.

No.	Feature Sets	Features	Description
1	FOS	Mean 1	TF₁
2		Mean 2	TF₂
3		Variance	TF₃
4		Standard Deviation	TF₄
5	Global Intensity Threshold		TF₅
6	NGTDM	Complexity	TF₆
7	NGTDM	Texture Strength	TF₇
8	GLRLM	SRE	TF₈
9		LRE	TF₉
10		GLN	TF₁₀
11		RLN	TF₁₁
12		RP	TF₁₂
13		LRGE	TF₁₃
14		HRGE	TF₁₄
15	LBP	LBP₀	TF₁₅
16		LBP₁	TF₁₆
17		LBP₂	TF₁₇
18		LBP₄	TF₁₈
19		LBP₆	TF₁₉
20		LBP₈	TF₂₀
21		LBP₉	TF₂₁
22	LT	(SAV) L5E5/E5L5	TF₂₂
23		(SAV) L5R5/R5L5	TF₂₃
24		(SAV) E5S5/S5E5	TF₂₄
25		(SAV) S5S5	TF₂₅
26		(SAV) R5R5	TF₂₆
27		(SAV) L5S5/S5L5	TF₂₇
28		(SAV) E5E5	TF₂₈
29		(SAV) E5R5/R5E5	TF₂₉
30		(SAV) S5R5/R5S5	TF₃₀
31		(SSV) L5E5/E5L5	TF₃₁
32		(SSV) L5R5/R5L5	TF₃₂
33		(SSV) E5S5/S5E5	TF₃₃
34		(SSV) S5S5	TF₃₄
35		(SSV) R5R5	TF₃₅
36		(SSV) L5S5/S5L5	TF₃₆
37		(SSV) E5E5	TF₃₇
38		(SSV) E5R5/R5E5	TF₃₈
39		(SSV) S5R5/R5S5	TF₃₉

Open in a new tab

Figure 2. — Classification results of fibrocalcific plaque according to accumulative sub-feature sets obtained using mRMR method after CRF noise cleaning. A total of 139 combined features were divided into 10 sub-feature sets (x-axis). For example, the feature set 1 includes 14 features, feature set 2 includes 28 features, and feature set 10 includes 139 features. Sensitivity and specificity are shown in the left figure, and Dice coefficients are shown in the right figure. Fibrolipidic and other classes have very similar results.

Figure 3. — Classification results for different subsets of feature set 1 after CRF noise cleaning. Panels (a)-(f) show results obtained using subsets 1–6, respectively. Subset 1 has only one feature, and subset 6 includes six features. For better understanding, the results for only 6 of 14 subsets were provided. For each figure, the inner ring is ground truth label, and outer ring is predicted result. Red, green, and blue indicate fibrocalcific, fibrolipidic, and other plaques, respectively, and white is the guidewire.

Figure 4 depicts plaque classification results in the anatomical domain before and after CRF noise cleaning. For better understanding, the ground truth and predicted results were overlapped as the inner and outer rings respectively. Before noise cleaning, numerous spotty errors were distributed all over the regions and these were clearly removed using the CRF method. The proposed method still had minor misclassifications especially in fibrolipidic tissue, but it was very small causing no changes in clinical decision making. The proposed method showed significant classification results in other cases without any identified plaques. Table 2 shows mean metrics before and after CRF noise cleaning. Considering that plaques are largely distributed in the coronary artery, we also evaluated classification results by creating the en-face (θ, z) image where each pixel has the vector of class probabilities for the corresponding A-line. As shown in Fig. 5, the proposed method showed numerous spotty classification errors over the main tissues. However, these misclassifications were clearly removed using CRF method.

Table 2.

Mean metrics including sensitivity, specificity, and Dice coefficient measured from five-fold cross validation before and after CRF noise cleaning. The overall classification performance was drastically improved after noise cleaning, and fibrocalcific class had better results compared to fibrolipidic class.

Classes		Sensitivity (%)	Specificity (%)	Dice
Fibrolipidic	before CRF	61.5±16.0	89.4±4.8	0.529±0.050
Fibrolipidic	after CRF	82.2±15.3	90.8±3.7	0.626±0.061
Fibrocalcific	before CRF	67.2±15.9	89.2±6.9	0.598±0.067
Fibrocalcific	after CRF	82.4±4.0	92.5±5.2	0.753±0.030
Other	before CRF	81.3±7.3	73.6±9.7	0.859±0.040
Other	after CRF	83.5±5.6	86.8±3.3	0.889±0.037

Open in a new tab

Figure 5. — A-line plaque classification results in en-face (*θ,z*) view. Panels show: (left) ground truth, (center) result prior to CRF noise cleaning, and (right) result after CRF noise cleaning. Red, green, and blue indicate fibrocalcific, fibrolipidic, and other classes, respectively. The images were created using one test fold data including 8 VOIs (700 image frames).

5. DISCUSSION

The optimization of the textural features based on the PCA method was useful for better classification results. This is because that textural features often show varying characteristics according to the image type and target lesion. It was not necessary to perform additional feature selection on CNN features, since these were already optimized through max-pooling processes. The feature selection is essential because not all the features are informative for plaque classification. In this study, the six features of CNN 71, GLRLM (RP), CNN65, SAV (S5S5), CNN 91, and SSV (S5R5/R5L5) were finally selected as the subset with the most predictive power from 172 features. This result has significant implications for current plaque characterization research. We observed that only a few features having high priority could provide outstanding classification results. Additionally, the proposed method has high clinical applicability in terms of computational time as the feature dimensionality is significantly decreased. One interesting result from the feature selection is that FOS and global intensity threshold, which were known to be very useful for plaque characterization in many previous studies,^38–41 were not selected as the final subset, despite relatively high priorities. FD was also expected to be useful due to the self-similarity over all the scale in IVOCT images, but it was excluded by the PCA method. Since the IVOCT signal usually produce some level of self-similarity within some range, the FD may be used as a distinctive feature in pixel-wise classification.

CRF noise cleaning effectively minimize classification errors and showed a remarkable improvement for all classes. In our previous study for discriminating each A-line using relatively simple CNN architecture⁷, the overall sensitivity increased by 10–15% after CRF noise cleaning. We also performed pixel-wise plaque characterization based on SegNet deep learning classifier, and interestingly the classification results were not significantly improved by the CRF method¹⁰. This inferior performance in the pixel-wise approach is due to the network having already sufficiently considered spatial similarity between each pixel during training. Therefore, there was only little margin for improvement. On the other hand, the A-line-based approach is vulnerable to spatial properties in IVOCT images and the CRF method well-compensated it. These results indicate that the CRF noise cleaning method is more appropriate for the A-line-based approach than the pixel-wise approach.

Hyperparameter optimization found that the selected six features have outstanding discriminability for each plaque in IVOCT images. In case of using the weakly correlated features, it is known that the classification performance can be improved as the training condition is deepened. However, the training conditions did not have significant influence because the best features with the highest priorities and correlations were selected using PCA and mRMR methods before the training process. Therefore, if the best feature sets are obtained, it is possible to produce comparable results even in the limited training condition indicating that feature selection is more important than the training condition for determining classification performance. This advantage would be very helpful for clinical application in terms of reducing computational time.

One limitation of the A-line-based classification approach is the inability to provide the exact plaque location. Nevertheless, the reason why we performed the A-line-based approach is due to the indeterminable labeling of lipidous tissue. Typically, the calcium can be easily identified because of inhomogeneous resulting in highly backscattering core with well-distinguishable boundaries, whereas the lipid shows a quick drop-off in signal making it difficult to identify the outer boundary. The A-line-based approach would be a good alternative as it only considers the existence of a certain plaque in each A-line. Additionally, the pixel-wise manual annotation may have high inter- and intra-observer variability as it completely depends on the reader’s experience; however, the proposed approach can solve this problem and provide promising results.

In summary, we developed an automated A-line-based plaque classification method in IVOCT images using combined deep learning and textural features. The proposed method showed outstanding classification results for both the fibrolipidic and fibrocalcific tissues with substantial improvement using CRF noise cleaning. We also found that the selected features were able to accurately classify the coronary plaques even in the limited training conditions. These results indicate the potential usefulness of the proposed method for IVOCT-based coronary artery disease diagnosis.

ACKNOWLEDGEMENTS

This project was supported by the National Heart, Lung, and Blood Institute through grants NIH R21HL108263, NIH R01HL114406, and NIH R01HL143484. This research was conducted in space renovated using funds from an NIH construction grant (C06 RR12463) awarded to Case Western Reserve University. The content of this report is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health. The grants were obtained via collaboration between Case Western Reserve University and University Hospitals of Cleveland. This work made use of the High-Performance Computing Resource in the Core Facility for Advanced Research Computing at Case Western Reserve University. The veracity guarantor, Yazan Gharaibeh, affirms to the best of his knowledge that all aspects of this paper are accurate.

REFERENCES

[1].Serruys PW, Morice M-C, Kappetein AP, Colombo A, Holmes DR, Mack MJ, Ståhle E, Feldman TE, van den Brand M, Bass EJ, Van Dyck N, Leadley K, Dawkins KD and Mohr FW, “Percutaneous Coronary Intervention versus Coronary-Artery Bypass Grafting for Severe Coronary Artery Disease,” New England Journal of Medicine 360(10), 961–972 (2009). [DOI] [PubMed] [Google Scholar]
[2].Mehanna E, Bezerra HG, Prabhu D, Brandt E, Chamié D, Yamamoto H, Attizzani GF, Tahara S, Van Ditzhuijzen N, Fujino Y, Kanaya T, Stefano G, Wang W, Gargesha M, Wilson D and Costa MA, “Volumetric Characterization of Human Coronary Calcification by Frequency Domain Optical Coherence Tomography,” Circ J 77(9), 2334–2340 (2013). [DOI] [PMC free article] [PubMed] [Google Scholar]
[3].Athanasiou LS, Karvelis PS, Tsakanikas VD, Naka KK, Michalis LK, Bourantas CV and Fotiadis DI, “A Novel Semiautomated Atherosclerotic Plaque Characterization Method Using Grayscale Intravascular Ultrasound Images: Comparison With Virtual Histology,” IEEE Transactions on Information Technology in Biomedicine 16(3), 391–400 (2012). [DOI] [PubMed] [Google Scholar]
[4].Nair A, Margolis MP, Kuban BD and Vince DG, “Automated coronary plaque characterisation with intravascular ultrasound backscatter: ex vivo validation,” EuroIntervention 3(1), 113–120 (2007). [PubMed] [Google Scholar]
[5].Bezerra HG, Costa MA, Guagliumi G, Rollins AM and Simon DI, “Intracoronary optical coherence tomography: a comprehensive review clinical and research applications,” JACC Cardiovasc Interv 2(11), 1035–1046 (2009). [DOI] [PMC free article] [PubMed] [Google Scholar]
[6].Tearney GJ, Regar E, Akasaka T, Adriaenssens T, Barlis P, Bezerra HG, Bouma B, Bruining N, Cho J, Chowdhary S, Costa MA, de Silva R, Dijkstra J, Di Mario C, Dudek D, Dudeck D, Falk E, Falk E, Feldman MD, et al. , “Consensus standards for acquisition, measurement, and reporting of intravascular optical coherence tomography studies: a report from the International Working Group for Intravascular Optical Coherence Tomography Standardization and Validation,” J. Am. Coll. Cardiol. 59(12), 1058–1072 (2012). [DOI] [PubMed] [Google Scholar]
[7].Kolluru C, Prabhu D, Gharaibeh Y, Bezerra H, Guagliumi G and Wilson D, “Deep neural networks for A-line-based plaque classification in coronary intravascular optical coherence tomography images,” J Med Imaging (Bellingham) 5(4), 044504 (2018). [DOI] [PMC free article] [PubMed] [Google Scholar]
[8].Prabhu D, Bezerra H, Kolluru C, Gharaibeh Y, Mehanna E, Wu H and Wilson D, “Automated A-line coronary plaque classification of intravascular optical coherence tomography images using handcrafted features and large datasets,” J Biomed Opt 24(10), 1–15 (2019). [DOI] [PMC free article] [PubMed] [Google Scholar]
[9].Gharaibeh Y, Prabhu DS, Kolluru C, Lee J, Zimin V, Bezerra HG and Wilson DL, “Coronary calcification segmentation in intravascular OCT images using deep learning: application to calcification scoring,” JMI 6(4), 045002 (2019). [DOI] [PMC free article] [PubMed] [Google Scholar]
[10].Lee J, Prabhu D, Kolluru C, Gharaibeh Y, Zimin VN, Bezerra HG, Wilson DL and Wilson DL, “Automated plaque characterization using deep learning on coronary intravascular optical coherence tomographic images,” Biomed. Opt. Express, BOE 10(12), 6497–6515 (2019). [DOI] [PMC free article] [PubMed] [Google Scholar]
[11].Lee J, Prabhu D, Kolluru C, Gharaibeh Y, Zimin VN, Dallan LAP, Bezerra HG and Wilson DL, “Fully automated plaque characterization in intravascular OCT images using hybrid convolutional and lumen mor-phology features,” Scientific Reports Accepted (2020). [DOI] [PMC free article] [PubMed] [Google Scholar]
[12].Wang Z, Kyono H, Bezerra G, Wilson H, Costa D,A, Rollins MA, “Automatic segmentation of intravascular optical coherence tomography images for facilitating quantitative diagnosis of atherosclerosis,” Proceedings of SPIE - The International Society for Optical Engineering (2011). [Google Scholar]
[13].Amini AA, Weymouth TE and Jain RC, “Using dynamic programming for solving variational problems in vision,” IEEE Transactions on Pattern Analysis and Machine Intelligence 12(9), 855–867 (1990). [Google Scholar]
[14].Krizhevsky A, Sutskever I and Hinton GE, “ImageNet Classification with Deep Convolutional Neural Networks,” [Advances in Neural Information Processing Systems 25], F. Pereira, C. J. C. Burges, L. Bottou, and K. Q. Weinberger, Eds., Curran Associates, Inc., 1097–1105 (2012). [Google Scholar]
[15].Otsu N, “A Threshold Selection Method from Gray-Level Histograms,” IEEE Transactions on Systems, Man, and Cybernetics 9(1), 62–66 (1979). [Google Scholar]
[16].Mandelbrot BB and Wheeler JA, “The Fractal Geometry of Nature,” American Journal of Physics 51(3), 286–287 (1983). [Google Scholar]
[17].Amadasun M and King R, “Textural features corresponding to textural properties,” IEEE Transactions on systems, man, and Cybernetics 19(5), 1264–1274 (1989). [Google Scholar]
[18].Vince DG, Dixon KJ, Cothren RM and Cornhill JF, “Comparison of texture analysis methods for the characterization of coronary plaques in intravascular ultrasound images,” Computerized medical imaging and graphics 24(4), 221–229 (2000). [DOI] [PubMed] [Google Scholar]
[19].Stoitsis J, Golemati S and Nikita KS, “A modular software system to assist interpretation of medical images—Application to vascular ultrasound images,” IEEE Transactions on Instrumentation and Measurement 55(6), 1944–1952 (2006). [Google Scholar]
[20].Hadjerci O, Hafiane A, Conte D, Makris P, Vieyres P and Delbos A, “Computer-aided detection system for nerve identification using ultrasound images: A comparative study,” Informatics in Medicine Unlocked 3, 29–43 (2016). [Google Scholar]
[21].“Texture information in run-length matrices.”, IEEE Transactions on Image Processing 7(11), 1602–1609 (1998). [DOI] [PubMed] [Google Scholar]
[22].Ojala T, Pietikainen M and Maenpaa T, “Multiresolution gray-scale and rotation invariant texture classification with local binary patterns,” IEEE Transactions on pattern analysis and machine intelligence 24(7), 971–987 (2002). [Google Scholar]
[23].Conners RW and Harlow CA, “A Theoretical Comparison of Texture Algorithms,” IEEE Transactions on Pattern Analysis and Machine Intelligence PAMI-2(3), 204–222 (1980). [DOI] [PubMed] [Google Scholar]
[24].Haralick RM, Shanmugam K and Dinstein I, “Textural features for image classification,” IEEE Transactions on systems, man, and cybernetics 3(6), 610–621 (1973). [Google Scholar]
[25].Laws KI, “Rapid Texture Identification,” Image Processing for Missile Guidance 0238, 376–382, International Society for Optics and Photonics (1980). [Google Scholar]
[26].Pietikäinen M, Rosenfeld A and Davis LS, “Experiments with texture classification using averages of local pattern matches,” IEEE Transactions on Systems, Man, and Cybernetics SMC-13(3), 421–426 (1983). [Google Scholar]
[27].Mougiakakou SGR, Golemati S, Gousias I, Nicolaides AN and Nikita KS, “Computer-aided diagnosis of carotid atherosclerosis based on ultrasound image statistics, laws’ texture and neural networks,” Ultrasound Med Biol 33(1), 26–36 (2007). [DOI] [PubMed] [Google Scholar]
[28].Hwang YN, Lee JH, Kim GY, Shin ES and Kim SM, “Characterization of coronary plaque regions in intravascular ultrasound images using a hybrid ensemble classifier,” Computer methods and programs in biomedicine 153, 83–92 (2018). [DOI] [PubMed] [Google Scholar]
[29].Peng Hanchuan, Long Fuhui and Ding C, “Feature selection based on mutual information criteria of max-dependency, max-relevance, and min-redundancy,” IEEE Transactions on Pattern Analysis and Machine Intelligence 27(8), 1226–1238 (2005). [DOI] [PubMed] [Google Scholar]
[30].Abdolmanafi A, Duong L, Dahdah N and Cheriet F, “Deep feature learning for automatic tissue classification of coronary artery using optical coherence tomography,” Biomed Opt Express 8(2), 1203–1220 (2017). [DOI] [PMC free article] [PubMed] [Google Scholar]
[31].Breiman L, Friedman J, Stone CJ and Olshen RA, [Classification and Regression Trees], Taylor & Francis; (1984). [Google Scholar]
[32].Breiman L, “Random Forests,” Machine Learning 45(1), 5–32 (2001). [Google Scholar]
[33].Archer KJ and Kimes RV, “Empirical characterization of random forest variable importance measures,” Computational Statistics & Data Analysis 52(4), 2249–2260 (2008). [Google Scholar]
[34].Ok AO, Akar O and Gungor O, “Evaluation of random forest method for agricultural crop classification,” European Journal of Remote Sensing 45(1), 421–432 (2012). [Google Scholar]
[35].Krähenbühl P and Koltun V, “Efficient Inference in Fully Connected CRFs with Gaussian Edge Potentials,” [Advances in Neural Information Processing Systems 24], J. Shawe-Taylor, R. S. Zemel, P. L. Bartlett, F. Pereira, and K. Q. Weinberger, Eds., Curran Associates, Inc., 109–117 (2011). [Google Scholar]
[36].Kingma DP and Ba J, “Adam: A Method for Stochastic Optimization,” arXiv:1412.6980 [cs] (2014). [Google Scholar]
[37].Dice LR, “Measures of the Amount of Ecologic Association Between Species,” Ecology 26(3), 297–302 (1945). [Google Scholar]
[38].Ughi GJ, Adriaenssens T, Sinnaeve P, Desmet W and D’hooge J, “Automated tissue characterization of in vivo atherosclerotic plaques by intravascular optical coherence tomography images,” Biomed Opt Express 4(7), 1014–1030 (2013). [DOI] [PMC free article] [PubMed] [Google Scholar]
[39].Athanasiou LS, Exarchos TP, Naka KK, Michalis LK, Prati F and Fotiadis DI, “Atherosclerotic plaque characterization in Optical Coherence Tomography images,” 2011 Annual International Conference of the IEEE Engineering in Medicine and Biology Society, 4485–4488 (2011). [DOI] [PubMed] [Google Scholar]
[40].Rico-Jimenez JJ, Campos-Delgado DU, Villiger M, Otsuka K, Bouma BE and Jo JA, “Automatic classification of atherosclerotic plaques imaged with intravascular OCT,” Biomed Opt Express 7(10), 4069–4085 (2016). [DOI] [PMC free article] [PubMed] [Google Scholar]
[41].Kolluru C, Prabhu D, Gharaibeh Y, Wu H and Wilson DL, “Voxel-based plaque classification in coronary intravascular optical coherence tomography images using decision trees,” Proc SPIE Int Soc Opt Eng 10575 (2018). [DOI] [PMC free article] [PubMed] [Google Scholar]

PERMALINK

Automatic A-line coronary plaque classification using combined deep learning and textural features in intravascular OCT images

Juhwan Lee

Chaitanya Kolluru

Yazan Gharaibeh

David Prabhu

Vladislav N Zimin

Hiram Bezerra

David Wilson