Abstract
Microchannel formation is known to be a significant marker of plaque vulnerability, plaque rupture, and intraplaque hemorrhage, which are responsible for plaque progression. We developed a fully-automated method for detecting microchannels in intravascular optical coherence tomography (IVOCT) images using deep learning. A total of 3,075 IVOCT image frames across 41 patients having 62 microchannel segments were analyzed. Microchannel was manually annotated by expert cardiologists, according to previously established criteria. In order to improve segmentation performance, pre-processing including guidewire detection/removal, lumen segmentation, pixel-shifting, and noise filtering was applied to the raw (r,θ) IVOCT image. We used the DeepLab-v3 plus deep learning model with the Xception backbone network for identifying microchannel candidates. After microchannel candidate detection, each candidate was classified as either microchannel or no-microchannel using a convolutional neural network (CNN) classification model. Our method provided excellent segmentation of microchannel with a Dice coefficient of 0.811, sensitivity of 92.4%, and specificity of 99.9%. We found that pre-processing and data augmentation were very important to improve results. In addition, a CNN classification step was also helpful to rule out false positives. Furthermore, automated analysis missed only 3% of frames having microchannels and showed no false positives. Our method has great potential to enable highly automated, objective, repeatable, and comprehensive evaluations of vulnerable plaques and treatments. We believe that this method is promising for both research and clinical applications.
Keywords: Intravascular optical coherence tomography, microchannel, deep learning, segmentation, classification
1. INTRODUCTION
Microchannel formation is known to be a significant marker of plaque vulnerability, plaque rupture, and intraplaque hemorrhage, which are responsible for plaque progression1,2. A microchannel can be defined as a no-signal tubuloluminal structure without a connection to the vessel lumen recognized on ≥3 consecutive cross-sectional intravascular optical coherence tomography (IVOCT) images2. The high-resolution of IVOCT (axial: 12–18 μm) allows a unique assessment of microchannel, suggesting an unprecedented opportunity to assess the plaque vulnerability.
Despite these advantages, IVOCT has some challenges in identifying microchannels that need to be addressed. First, rapid interpretation of IVOCT remains a challenge for many interventional cardiologists, since a single IVOCT pullback includes more than 500 image frames. Second, manual analysis of microchannels can be subject to high inter- and intra-observer variability. This creates a confound for widespread quantitative and visual evaluation, especially when one considers cardiologists having variable experience. Real-time, automated IVOCT determination of microchannels would support interventional cardiologists and streamline clinical workflow. Further, quantitative assessment of microchannel will support clinical research studies to elucidate the underlying mechanisms and factors behind plaque progression.
In this paper, we build on our previous studies3–11 and present a new method for automated detection of microchannel in IVOCT images. Our method identifies microchannel candidates using the semantic segmentation deep learning model. Subsequently, each candidate is classified as either microchannel or no-microchannel using a convolutional neural network (CNN) classification model. Results are quantitatively assessed from the perspective of detection and manual editing.
2. IMAGE ANALYSIS METHODS
The proposed method is composed of three main steps including pre-processing, identification of microchannel candidates, and classification of microchannel candidates. First, to improve segmentation performance, pre-processing including guidewire and shadow removal, lumen segmentation, pixel-shifting, and noise reduction was implemented to raw polar (r,θ) IVOCT images. The DeepLab-v3 plus deep learning model was used to identify microchannel candidates across the pullback, and each candidate was classified as either microchannel or no-microchannel using a simple CNN model. Segmentation performance was quantitatively assessed using conventional metrics such as Dice coefficient and sensitivity. Figure 1 shows the overall workflow of the proposed method.
Figure 1.
Overall workflow of the proposed method for microchannel detection in IVOCT images.
2.1. Data augmentation
To enrich datasets for training, we first applied data augmentation proposed by our group5. Briefly, we concatenated all the raw polar (r,θ) IVOCT images to form one large 2D array, where the r and θ indicate tissue depth and catheter rotation. We then resampled new images by changing an offset angular shift. In this paper, we shifted the starting A-line six times by increments of 80 A-lines. The algorithm details are described elsewhere5.
2.2. Pre-processing
A modified version of previously proposed pre-processing3,6 was applied to raw IVOCT data in the polar (r,θ) domain. (1) The guidewire and corresponding shadow region were detected/removed using dynamic programming12. (2) We detected the lumen boundary using the deep learning approach previously developed by our group5. (3) Each A-line of raw IVOCT image was pixel-shifted to the left so that all rows have the same starting pixel along the radial (θ) direction. A pixel-shifting enables the creation of a smaller region of interest, which simplifies processing, and aligns tissues so that different lesions look more similar to the network. (4) We used the first 1.5 mm (300 pixels) from the lumen boundary in the r direction since IVOCT signal has limited penetration depth. Therefore, the size of IVOCT image was cropped from (968×496) to (200×496) without any data loss. (5) A Gaussian filter with a standard deviation of 1 pixel and a kernel size of (7,7) was used to reduce speckle noise. The resulting images were used as an input to the semantic segmentation deep learning model for identifying microchannel candidates.
2.3. Automated identification of microchannel candidates
We implemented the DeepLab-v3 plus semantic segmentation model13 to determine microchannel candidates (Fig. 2). This model takes advantage of two modules (encoder and decoder). An encoder module captures the contextual information of microchannel at different scales, while a decoder module effectively recovers microchannel boundaries. We briefly describe the three key concepts as follows. (1) Atrous convolution controls the resolution of features and adjusts a field-of-view of filters. It allows the network to learn multi-scale contextual information and generalizes standard convolution operations. (2) Depth-wise separable convolution performs an independent spatial convolution for each input channel, and the point-wise convolution is followed to combine the output from the depth-wise convolution. This process greatly reduces computational complexity without degradation of segmentation performance. (3) Encoder and decoder. The encoder extracts object features at different resolutions, whereas the decoder generates an output label having the same size as the input image (Fig. 2). In this study, we used the Xception14 as the backbone network for feature extraction. For each encoder layer, the batch normalization and rectified linear unit (ReLU) were followed after convolution.
Figure 2.
DeepLab-v3 plus architecture. The left and right figures indicate the pre-processed IVOCT image and predicted label, respectively. The input image is shown following log transformation for improved visualization.
2.4. Classification of microchannel candidates
After segmentation, each microchannel candidate was classified as either microchannel or no-microchannel using a CNN model. For this purpose, we created a bounding box of each candidate by taking account of maximum horizontal and vertical lengths (Fig. 1). A CNN model consisted of 3 convolutional, 2 maximum pooling, and 1 fully-connected layers. The input size of the network was (30×30×1). Each convolutional layer included convolutional, batch normalization, and ReLU layers. Convolutional layers had the same number of filters (3) and different kernel sizes (8, 16, and 32) with a stride of 2 pixels. The batch normalization and ReLU accelerate the training process and reduce the sensitivity to network initialization15. The maximum pooling layer with a pool size of 2 pixels was subsequently implemented to reduce dimensionality. The fully connected layer was followed by the final convolutional layer. The fully connected layer included 2 output units along with the Softmax activation corresponding to the two classes of interest. To improve classification performance, we applied data augmentation by rotating each bounding box from 30° to 180° with a 30° interval, giving us a six times greater number of positive cases.
3. EXPERIMENTAL METHODS
3.1. Image acquisition
This study was a sub-study of the TRiple Assessment of Neointima Stent FOrmation to Reabsorbable polyMer with Optical Coherence Tomography (TRANSFORM-OCT) trial16. A total of 575 image frames across 48 pullbacks were included in the final analysis. IVOCT images were acquired with a frequency-domain OCT system (ILUMIEN OPTIS; St. Jude Medical Inc.), which uses a tunable light source sweeping from 1,250 to 1,360 nm. Imaging pullback was performed with a frame rate of 180 fps, pullback speed of 36 mm/s, and axial resolution of approximately 20 μm. This study was approved by the Institutional Review Board of University Hospitals Cleveland Medical Center (Cleveland, OH, USA).
In order to manually annotate the microchannel, the raw IVOCT data (r,θ) was transformed to the Anatomical (x,y) domain. Two experts from the Cardiovascular Imaging Core Laboratory, University Hospitals Cleveland Medical Center, manually analyzed each image. The first expert’s annotations were used as the ground truth, and the second expert’s annotations to compute inter-observer agreement. Each expert was blind to other experts’ annotation. According to the definition in2, microchannel was identified using a dedicated software developed by our group (OCTOPUS). Specifically, no-signal tubuloluminal structure without a connection to the vessel lumen presenting more than 3 consecutive IVOCT images was considered as “microchannel”. Additional class “other” was used to include residual regions that did not meet the criteria of microchannel.
3.2. Network training
Segmentation and classification networks were optimized using the adaptive moment estimation (ADAM) optimizer17. The initial learning rate, drop factor, and drop period were empirically set to 0.001, 0.2, and 5, respectively, for both networks. To gradually reduce the learning rate, we multiplied the initial learning rate by a drop factor every drop period. The weights were fine-tuned starting from the last layer by changing the learning rates of the previous layers. Because our dataset was imbalanced, we computed the inversed median frequency of class proportions and used them as the class weight. To avoid over-fitting during the training, we stopped the training when the validation loss did not improve over 5 consecutive epochs or when a maximum number of epochs (50) was reached, whichever occurred first. The crossentropy loss function over the softmax was used as the output extensively. All image processing and network training were done using MATLAB (R2018b, MathWorks, Inc.) on an NVIDIA Geforce TITAN RTX GPU (64 GM RAM).
3.3. Performance validation
We divided a total of 48 IVOCT pullbacks (575 images × 6 times = 3,450 images) into the training (70%), validation (15%), and test (15%) sets. Segmentation performance was quantitatively evaluated in terms of conventional metrics such as positive predictive value (PPV), negative predictive value (NPV), sensitivity, specificity, and Dice coefficient.
4. RESULTS
We examined the importance of pre-processing and data augmentation for detecting microchannel in IVOCT images. Figure 3 shows microchannel detection results on the raw polar IVOCT images with and without data augmentation. Without data augmentation, network showed numerous false positives (Fig. 3B). Most of them were removed with data augmentation, but microchannels were not detected properly (Fig. 3C). Sensitivity was less than 40% and the Dice coefficient was less than 0.3 (Table 1), indicating that the raw IVOCT is not acceptable for microchannel detection. Figure 4 shows microchannel detection results on the pre-processed images with and without data augmentation. Microchannel was well detected without data augmentation, but there were some false positives across the entire test sets (Fig 4B). These misdetections were resolved with data augmentation (Fig 4C). With pre-processing, our method showed significant improvements with sensitivity of 83.6% and Dice coefficient of 0.741 although the size of predicted microchannel candidate was slightly overestimated (Table 1). Data augmentation also significantly improved results (sensitivity: 93.3% and Dice coefficient: 0.805). Quantitative metrics are shown in Table 1.
Figure 3.
Microchannel detection results on the raw IVOCT image. Panels are: (A) original image, (B) prediction without augmentation, and (C) prediction with data augmentation. Network was not able to detect microchannels on the raw IVOCT images, and data augmentation did not improve results. The yellow is lumen, and the blue is microchannel.
Table 1.
Quantitative performance metrics including PPV, NPV, Dice coefficient, sensitivity, and specificity.
| PPV (%) | NPV (%) | Dice | Sensitivity (%) | Specificity (%) | |
|---|---|---|---|---|---|
| Original image wo augmentation | 19.7 | 99.6 | 0.260 | 38.5 | 99.9 |
| Original image w augmentation | 25.7 | 99.9 | 0.273 | 99.9 | 99.9 |
| Pre-processing wo augmentation | 72.2 | 100.0 | 0.741 | 83.6 | 99.9 |
| Pre-processing w augmentation | 75.7 | 100.0 | 0.805 | 93.3 | 99.9 |
| After classification | 77.0 | 100.0 | 0.811 | 92.4 | 99.9 |
Figure 4.
Microchannel detection results on the pre-processed IVOCT image. Panels are: (A) original image, (B) prediction without augmentation, and (C) prediction with data augmentation. Data augmentation greatly improved segmentation results. Yellow and blue indicate lumen and microchannel, respectively.
Candidate classification step was very helpful to improve detection results. Figure 5 shows microchannel detection results before and after candidate classification. Although pre-processing and data augmentation were useful, some images had challenging false positives which look like small calcifications (Fig 5B). With classification step, residual false positives were effectively ruled out while preserving microchannel (Fig 5C). PPV and Dice coefficient were also improved from 75.7% and 0.805 to 77.0% and 0.811, respectively (Table 1). In addition, our method missed only 1 microchannel image out of 68 test sets, indicating excellent detection performance.
Figure 5.
Microchannel detection results with and without candidate classification. Panels are (A) original image, (B) detection without candidate classification, and (C) prediction with candidate classification. Classification model was very helpful to rule out false positives. Yellow and blue indicate lumen and microchannel, respectively.
5. DISCUSSION
The formation of microchannel is associated with plaque vulnerability, plaque rupture, and intraplaque hemorrhage1,2. Although manual analysis can be labor intensive and subject to high inter- and intra-observer variability, no studies have attempted to develop an automated method for detecting microchannel in IVOCT images. To the best of our knowledge, this was the first study to automatically detect microchannels in IVOCT images. We implemented several important algorithmic steps such as pre-processing, data augmentation, identification of microchannel candidates, and candidate classification. The main findings of this study can be summarized as follows: (1) pre-processing including guidewire/shadow removal, lumen segmentation, pixel shifting, and noise filtering significantly improved detection results; (2) Deeplab-v3 plus network properly segmented microchannel candidates although the size was slightly overestimated; (3) Data augmentation enriched datasets for training and was very helpful to improve detection performance; (4) Classification network effectively ruled out residual false positives.
We found that all algorithmic steps of the proposed method played a very important role in improving detection performance. First, data augmentation was used to provide more examples and to change locations of microchannels to improve spatial invariance of methods. Since we augmented data by shifting the starting A-line on one large 2D array, our augmentation method is free from any data loss or distortion, allowing us to obtain much more realistic images. Therefore, we used more than 3,000 IVOCT images for training deep learning segmentation network, and it was very helpful to get rid of numerous false positives (Fig 4). Second, given that IVOCT signal has limited penetration depth, we reduced the size of input image from (968×496) to (200×496) without any data loss using pre-processing previously developed by our group3,6. This process made a deep learning network solely focusing on the specific region of interest and significantly improved segmentation results in terms of sensitivity (29.2% to 93.3%) and Dice coefficient (0.273 to 0.805) (Table 1). Third, classification step effectively ruled out false positives arising from segmentation step. Due to the speckle noise in IVOCT images, segmentation network does not always work well and occasionally shows inaccurate predictions. Classification network was trained on more than 5,500 images with additional data augmentation (angle rotation), and trained network showed improved classification results with a sensitivity of 97%.
Our method could enable a comprehensive assessment of high-risk plaque in a coronary artery. Plaque neo-vascularization and microchannel are known to be markers of plaque vulnerability and rupture1,2. Kitabata et al. reported that patients with microchannels had more unstable angina (87% vs. 17%), a thinner fibrous cap (60 μm vs. 100 μm), and a higher occurrence of plaque rupture (50% vs 28%)2. In addition, a study of patients with cardiac allograft vasculopathy found those with high-grade rejection had thicker intima (0.34 mm vs. 0.15 mm), a higher prevalence of macrophages (44% vs. 15%), and a higher prevalence of intimal microchannels (46% vs. 11%)18. Therefore, combining quantitative assessments of microchannel with other representative plaques, such as lipid, fibrous cap, and calcification, could provide a more comprehensive assessment of coronary plaques in IVOCT images.
This study has several limitations. First, we used a limited number of IVOCT data. Although we compensated with our special data augmentation, it will be possible to improve the detection performance with more data. Second, we only used IVOCT images with microchannels for training network, and the model was not evaluated on images without microchannels. Third, we used a conventional deep learning semantic segmentation for microchannel detection. Results may improve with the use of more advanced deep learning models.
We developed an automated method for microchannel detection using deep learning in IVOCT images. Our method has great potential to enable highly automated, objective, repeatable, and comprehensive evaluations of high-risk plaques and treatments. We believe that this method is promising for both research and clinical applications.
ACKNOWLEDGEMENTS
This project was supported by the National Heart, Lung, and Blood Institute through grants NIH R21HL108263, NIH R01HL114406, and NIH R01HL143484. This work was also supported by American Heart Association Grant #20POST35210974/Juhwan Lee/2020. This research was conducted in space renovated using funds from an NIH construction grant (C06 RR12463) awarded to Case Western Reserve University. The content of this report is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health. The grants were obtained via collaboration between Case Western Reserve University and University Hospitals of Cleveland. This work made use of the High-Performance Computing Resource in the Core Facility for Advanced Research Computing at Case Western Reserve University. The veracity guarantor, Justin N. Kim, affirms to the best of his knowledge that all aspects of this paper are accurate.
REFERENCES
- [1].Uemura S, Ishigami K, Soeda T, Okayama S, Sung JH, Nakagawa H, Somekawa S, Takeda Y, Kawata H, Horii M and Saito Y, “Thin-cap fibroatheroma and microchannel findings in optical coherence tomography correlate with subsequent progression of coronary atheromatous plaques,” Eur Heart J 33(1), 78–85 (2012). [DOI] [PubMed] [Google Scholar]
- [2].Kitabata H, Tanaka A, Kubo T, Takarada S, Kashiwagi M, Tsujioka H, Ikejima H, Kuroi A, Kataiwa H, Ishibashi K, Komukai K, Tanimoto T, Ino Y, Hirata K, Nakamura N, Mizukoshi M, Imanishi T and Akasaka T, “Relation of Microchannel Structure Identified by Optical Coherence Tomography to Plaque Vulnerability in Patients With Coronary Artery Disease,” The American Journal of Cardiology 105(12), 1673–1678 (2010). [DOI] [PubMed] [Google Scholar]
- [3].Kolluru C, Prabhu D, Gharaibeh Y, Bezerra H, Guagliumi G and Wilson D, “Deep neural networks for A-line-based plaque classification in coronary intravascular optical coherence tomography images,” J Med Imaging (Bellingham) 5(4), 044504 (2018). [DOI] [PMC free article] [PubMed] [Google Scholar]
- [4].Prabhu D, Bezerra H, Kolluru C, Gharaibeh Y, Mehanna E, Wu H and Wilson D, “Automated A-line coronary plaque classification of intravascular optical coherence tomography images using handcrafted features and large datasets,” J Biomed Opt 24(10), 1–15 (2019). [DOI] [PMC free article] [PubMed] [Google Scholar]
- [5].Gharaibeh Y, Prabhu DS, Kolluru C, Lee J, Zimin V, Bezerra HG and Wilson DL, “Coronary calcification segmentation in intravascular OCT images using deep learning: application to calcification scoring,” JMI 6(4), 045002 (2019). [DOI] [PMC free article] [PubMed] [Google Scholar]
- [6].Lee J, Prabhu D, Kolluru C, Gharaibeh Y, Zimin VN, Bezerra HG, Wilson DL and Wilson DL, “Automated plaque characterization using deep learning on coronary intravascular optical coherence tomographic images,” Biomed. Opt. Express, BOE 10(12), 6497–6515 (2019). [DOI] [PMC free article] [PubMed] [Google Scholar]
- [7].Lu H, Lee J, Ray S, Tanaka K, Bezerra HG, Rollins AM and Wilson DL, “Automated stent coverage analysis in intravascular OCT (IVOCT) image volumes using a support vector machine and mesh growing,” Biomed. Opt. Express, BOE 10(6), 2809–2828 (2019). [DOI] [PMC free article] [PubMed] [Google Scholar]
- [8].Lee J, Prabhu D, Kolluru C, Gharaibeh Y, Zimin VN, Dallan LAP, Bezerra HG and Wilson DL, “Fully automated plaque characterization in intravascular OCT images using hybrid convolutional and lumen morphology features,” 1, Scientific Reports 10(1), 2596 (2020). [DOI] [PMC free article] [PubMed] [Google Scholar]
- [9].Lu H, Lee J, Jakl M, Wang Z, Cervinka P, Bezerra HG and Wilson DL, “Application and Evaluation of Highly Automated Software for Comprehensive Stent Analysis in Intravascular Optical Coherence Tomography,” Sci Rep 10(1), 2150 (2020). [DOI] [PMC free article] [PubMed] [Google Scholar]
- [10].Lee J, Gharaibeh Y, Kolluru C, Zimin VN, Dallan LAP, Kim JN, Bezerra HG and Wilson DL, “Segmentation of Coronary Calcified Plaque in Intravascular OCT Images Using a Two-Step Deep Learning Approach,” IEEE Access 8, 225581–225593 (2020). [DOI] [PMC free article] [PubMed] [Google Scholar]
- [11].Kolluru C, Lee J, Gharaibeh Y, Bezerra HG and Wilson DL, “Learning With Fewer Images via Image Clustering: Application to Intravascular OCT Image Segmentation,” IEEE Access 9, 37273–37280 (2021). [DOI] [PMC free article] [PubMed] [Google Scholar]
- [12].Wang Z, Kyono H, M.d, B. HG, Wilson DL, M.d, C. MA and Rollins AM, “Automatic segmentation of intravascular optical coherence tomography images for facilitating quantitative diagnosis of atherosclerosis,” Optical Coherence Tomography and Coherence Domain Optical Methods in Biomedicine XV 7889, 100–106, SPIE; (2011). [Google Scholar]
- [13].Chen L-C, Zhu Y, Papandreou G, Schroff F and Adam H, “Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation,” Computer Vision – ECCV 2018, Ferrari V, Hebert M, Sminchisescu C, and Weiss Y, Eds., 833–851, Springer International Publishing; (2018). [Google Scholar]
- [14].Chollet F, “Xception: Deep Learning with Depthwise Separable Convolutions,” 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 1800–1807 (2017). [Google Scholar]
- [15].Krizhevsky A, Sutskever I and Hinton GE, “ImageNet Classification with Deep Convolutional Neural Networks,” Proceedings of the 25th International Conference on Neural Information Processing Systems - Volume 1, 1097–1105, Curran Associates Inc., USA: (2012). [Google Scholar]
- [16].Guagliumi G, Shimamura K, Sirbu V, Garbo R, Boccuzzi G, Vassileva A, Valsecchi O, Fiocca L, Canova P, Colombo F, Tensol Rodriguez Pereira G, Nakamura D, Attizzani GF, Cereda A, Satogami K, De Luca L, Saia F and Capodanno D, “Temporal course of vascular healing and neoatherosclerosis after implantation of durable- or biodegradable-polymer drug-eluting stents,” European Heart Journal 39(26), 2448–2456 (2018). [DOI] [PubMed] [Google Scholar]
- [17].Kingma D and Ba J, “Adam: A Method for Stochastic Optimization,” International Conference on Learning Representations (2014). [Google Scholar]
- [18].Dong L, Maehara A, Nazif TM, Pollack AT, Saito S, Rabbani LE, Apfelbaum MA, Dalton K, Moses JW, Jorde UP, Xu K, Mintz GS, Mancini DM and Weisz G, “Optical Coherence Tomographic Evaluation of Transplant Coronary Artery Vasculopathy With Correlation to Cellular Rejection,” Circ Cardiovasc Interv 7(2), 199–206 (2014). [DOI] [PubMed] [Google Scholar]