Image Enhanced Mask R-CNN: A Deep Learning Pipeline with New Evaluation Measures for Wind Turbine Blade Defect Detection and Classification
"> Figure 1
<p>Bounding Box Accuracy—yellow box shows the bounding box of the manually labelled defect, and the black box is an example predicted bounding box that may be generated by the detection model. BBA computes the difference between two overlapping bounding boxes by calculating the area between the overlapping boxes. If there is no overlap between the bounding boxes, the BBA value will be 0.</p> "> Figure 2
<p>Example outputs of DL algorithms. The figure shows three outputs of YOLOv3, YOLOv4 and Mask R-CNN. All algorithms recognised the defects, however, YOLOv3 incorrectly classified a crack defect as a void defect; and the prediction boxes did not comprehensively cover the large-sized defect area, such as erosion defect, in YOLOv4.</p> "> Figure 3
<p>Performance Evaluation Diagrams for YOLOv3, YOLOv4 and Mask R-CNN. (<b>a</b>) Traditional Performance Evaluation. (<b>b</b>) Detection Speed Evaluation. (<b>c</b>) mean Weighted Average Performance Evaluation. (<b>d</b>) RR and FLR Evaluation.</p> "> Figure 4
<p>Performance comparison between Mask R-CNN(D2) and IE Mask R-CNN(D2). (<b>a</b>) Overall performance comparisons. (<b>b</b>) mWA evaluation and FLR comparisons. Left axis is used for mWA, and the right axis is used for FLR.</p> "> Figure 5
<p>IE Mask R-CNN: The proposed Image Enhanced Mask R-CNN pipeline.</p> "> Figure 6
<p>Mask R-CNN architecture with ResNetX-101 backbone and Fully Connected layers (FC layers). This architecture was embedded in the proposed IE Mask R-CNN pipeline.</p> ">
Abstract
:1. Introduction
2. Related Methods
3. Materials and Methods
3.1. Dataset and Image Augmentation
3.2. Traditional Performance Evaluation Measures for Defect Detection
- True Positive (TP) predictions—a defect area that is correctly detected and classified by the model.
- False Positive (FP) predictions—an area that has been incorrectly identified as a defect. There are two types of FPs. (1) The predicted area does not overlap with a labelled area; and (2) the predicted area is overlapping with a labelled area, but the defect’s type is misclassified.
- False Negative (FN) predictions—a labelled area that has not been detected by the model.
3.3. Proposed Performance Evaluation Measures for Defect Detection
3.3.1. Prediction Box Accuracy
3.3.2. Recognition Rate
3.3.3. False Label Rate
3.4. Experimental Setup
4. Results
4.1. Performance Evaluation of YOLOv3, YOLOv4, and Mask R-CNN
4.2. Comparison of YOLOv3, YOLOv4 and Mask R-CNN
4.3. An Investigation into Whether Image Enhancement Can Further Improve the Results of the Mask R-CNN Model
4.4. IE Mask R-CNN: Proposed Deep Learning Defect Detection Pipeline
5. Discussion and Conclusions
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Acknowledgments
Conflicts of Interest
Abbreviations
WTB | Wind Turbine Blade |
NDT | Non-Destructive Testing |
DL | Deep Learning |
CNN | Convolutional Neural Network |
YOLO | You Only Look Once |
YOLOv2 | YOLO version 2 |
YOLOv3 | YOLO version 3 |
YOLOv4 | YOLO version 4 |
Mask R-CNN | Mask Region-based Convolutional Neural Network |
mAP | Mean Average Precision |
PBA | Prediction Box Accuracy |
RR | Recognition Rate |
FLR | False Label Rate |
IE Mask R-CNN | Image Enhanced Mask R-CNN |
ML | Machine Learning |
SVM | Support Vector Machine |
LR | Linear Regression |
RF | Random Forest |
AUC | Area Under the Curve |
HOG | Histogram of Oriented Gradient feature |
ILSVRC | ImageNet Large Scale Visual Recognition Challenge |
k-NN | k-Nearest Neighbour |
DT | Decision Tree |
D0 | Dataset D0 |
D1 | Dataset D1 |
D2 | Dataset D2 |
D3 | Dataset D3 |
TP | True Positive |
FP | False Positive |
FN | False Negative |
IoU | Intersection over Union |
BBA | Bounding Box Accuracy |
WidthA | Width Accuracy |
HeightA | Height Accuracy |
VIA | VGG Image Annotator |
WA | Weighted Average |
mWA | mean Weighted Average |
std | Standard Deviation |
WB | White Balance |
AHE | Adaptive Histogram Equalisation |
References
- Toft, H.S.; Branner, K.; Berring, P.; Sørensen, J.D. Defect distribution and reliability assessment of wind turbine blades. Eng. Struct. 2011, 33, 171–180. [Google Scholar] [CrossRef]
- Chatzakos, P.; Avdelidis, N.; Hrissagis, K.; Gan, T. Autonomous Infrared (IR) Thermography based inspection of glass reinforced plastic (GRP) wind turbine blades (WTBs). In Proceedings of the 2010 IEEE Conference on Robotics, Automation and Mechatronics, Singapore, 28–30 June 2010; pp. 557–562. [Google Scholar]
- Maierhofer, C.; Myrach, P.; Krankenhagen, R.; Röllig, M.; Steinfurth, H. Detection and Characterization of Defects in Isotropic and Anisotropic Structures Using Lockin Thermography. J. Imaging 2015, 1, 220–248. [Google Scholar] [CrossRef] [Green Version]
- Kim, D.Y.; Kim, H.; Jung, W.S.; Lim, S.; Hwang, J.; Park, C. Visual testing system for the damaged area detection of wind power plant blade. In Proceedings of the IEEE ISR 2013, Seoul, Korea, 24–26 October 2013; pp. 1–5. [Google Scholar]
- Hongwu, Q.; Haixin, S.; Wei, C.; Mengcong, D. Structural Health Monitoring WTB Using the Effectiveness of Graphical Programming Packages Analysis on Acoustic Emission Data. In Proceedings of the 2015 IEEE Fifth International Conference on Big Data and Cloud Computing, Dalian, China, 26–28 August 2015; pp. 207–212. [Google Scholar]
- Li, Z.; Soutis, C.; Haigh, A.; Sloan, R.; Gibson, A.; Karimian, N. Microwave imaging for delamination detection in T-joints of wind turbine composite blades. In Proceedings of the 2016 46th European Microwave Conference (EuMC), London, UK, 4–6 October 2016; pp. 1235–1238. [Google Scholar]
- Aust, J.; Shankland, S.; Pons, D.; Mukundan, R.; Mitrovic, A. Automated Defect Detection and Decision-Support in Gas Turbine Blade Inspection. Aerospace 2021, 8, 30. [Google Scholar] [CrossRef]
- Eugene Chian, Y.T.; Tian, J. Surface Defect Inspection in Images Using Statistical Patches Fusion and Deeply Learned Features. AI 2021, 2, 17–31. [Google Scholar] [CrossRef]
- Reddy, A.; Indragandhi, V.; Ravi, L.; Subramaniyaswamy, V. Detection of Cracks and damage in wind turbine blades using artificial intelligence-based image analytics. Measurement 2019, 147, 106823. [Google Scholar] [CrossRef]
- Yang, P.; Dong, C.; Zhao, X.; Chen, X. The Surface Damage Identifications of Wind Turbine Blades Based on ResNet50 Algorithm. In Proceedings of the 2020 39th Chinese Control Conference (CCC), Shenyang, China, 27–29 July 2020; pp. 6340–6344. [Google Scholar]
- Deng, J.; Lu, Y.; Lee, V.C.S. Imaging-based crack detection on concrete surfaces using You Only Look Once network. Struct. Health Monit. 2020. [Google Scholar] [CrossRef]
- Shijie, J.; Ping, W.; Peiyi, J.; Siping, H. Research on data augmentation for image classification based on convolution neural networks. In Proceedings of the 2017 Chinese Automation Congress (CAC), Jinan, China, 20–22 October 2017; pp. 4165–4170. [Google Scholar]
- Perez, L.; Wang, J. The Effectiveness of Data Augmentation in Image Classification using Deep Learning. arXiv 2017, arXiv:1712.04621. [Google Scholar]
- Qiao, Y.; Truman, M.; Sukkarieh, S. Cattle segmentation and contour extraction based on Mask R-CNN for precision livestock farming. Comput. Electron. Agric. 2019, 165, 104958. [Google Scholar] [CrossRef]
- Kanan, C.; Cottrell, G.W. Color-to-Grayscale: Does the Method Matter in Image Recognition? PLoS ONE 2012, 7, e29740. [Google Scholar] [CrossRef] [Green Version]
- Redmon, J.; Farhadi, A. YOLOv3: An Incremental Improvement. arXiv 2018, arXiv:1804.02767. [Google Scholar]
- Bochkovskiy, A.; Wang, C.Y.; Liao, H.Y.M. YOLOv4: Optimal Speed and Accuracy of Object Detection. arXiv 2020, arXiv:2004.10934. [Google Scholar]
- He, K.; Gkioxari, G.; Dollár, P.; Girshick, R. Mask R-CNN. In Proceedings of the 2017 IEEE International Conference on Computer Vision (ICCV), Venice, Italy, 22–29 October 2017; pp. 2980–2988. [Google Scholar]
- Li, C.; Heinemann, P.; Sherry, R. Neural network and Bayesian network fusion models to fuse electronic nose and surface acoustic wave sensor data for apple defect detection. Sens. Actuators B Chem. 2007, 125, 301–310. [Google Scholar] [CrossRef]
- Karayiannis, Y.A.; Stojanovic, R.; Mitropoulos, P.; Koulamas, C.; Stouraitis, T.; Koubias, S.; Papadopoulos, G. Defect detection and classification on web textile fabric using multiresolution decomposition and neural networks. In Proceedings of the 6th IEEE International Conference on Electronics, Circuits and Systems (Cat. No.99EX357) (ICECS’99), Paphos, Cyprus, 5–8 September 1999; Volume 2, pp. 765–768. [Google Scholar]
- Tilocca, A.; Borzone, P.; Carosio, S.; Durante, A. Detecting Fabric Defects with a Neural Network Using Two Kinds of Optical Patterns. Text. Res. J. 2002, 72, 545–550. [Google Scholar] [CrossRef]
- Graham, D.; Maas, P.; Donaldson, G.; Carr, C. Impact damage detection in carbon fibre composites using HTS SQUIDs and neural networks. NDT E Int. 2004, 37, 565–570. [Google Scholar] [CrossRef]
- Soukup, D.; Huber-Mörk, R. Convolutional Neural Networks for Steel Surface Defect Detection from Photometric Stereo Images. In Advances in Visual Computing; Bebis, G., Boyle, R., Parvin, B., Koracin, D., McMahan, R., Jerald, J., Zhang, H., Drucker, S.M., Kambhamettu, C., El Choubassi, M., et al., Eds.; Springer International Publishing: Cham, Switzerland, 2014; pp. 668–677. [Google Scholar]
- Wang, Z.; Hu, M.; Zhai, G. Application of Deep Learning Architectures for Accurate and Rapid Detection of Internal Mechanical Damage of Blueberry Using Hyperspectral Transmittance Data. Sensors 2018, 18, 1126. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Narayanan, B.N.; Beigh, K.; Loughnane, G.; Powar, N. Support vector machine and convolutional neural network based approaches for defect detection in fused filament fabrication. In Applications of Machine Learning; Zelinski, M.E., Taha, T.M., Howe, J., Awwal, A.A.S., Iftekharuddin, K.M., Eds.; International Society for Optics and Photonics, SPIE: San Diego, CA, USA, 2019; Volume 11139, pp. 283–291. [Google Scholar]
- Wang, J.; Fu, P.; Gao, R.X. Machine vision intelligence for product defect inspection based on deep learning and Hough transform. J. Manuf. Syst. 2019, 51, 52–60. [Google Scholar] [CrossRef]
- Kawiecki, G. Application of Neural Networks to Defect Detection in Cantilever Beams with Linearized Damage Behavior. J. Intell. Mater. Syst. Struct. 1999, 10, 797–801. [Google Scholar] [CrossRef]
- Jasinien, E.; Raiutis, R.; Literis, R.; Voleiis, A.; Vladiauskas, A.; Mitchard, D.; Amos, M. NDT of wind turbine blades using adapted ultrasonic and radiographic techniques. Insight Non-Destr. Test. Cond. Monit. 2009, 51, 477–483. [Google Scholar] [CrossRef] [Green Version]
- Protopapadakis, E.; Doulamis, N. Image Based Approaches for Tunnels’ Defects Recognition via Robotic Inspectors. In Advances in Visual Computing; Bebis, G., Boyle, R., Parvin, B., Koracin, D., Pavlidis, I., Feris, R., McGraw, T., Elendt, M., Kopper, R., Ragan, E., et al., Eds.; Springer International Publishing: Cham, Switzerland, 2015; pp. 706–716. [Google Scholar]
- Yu, Y.; Cao, H.; Liu, S.; Yang, S.; Bai, R. Image-based damage recognition of wind turbine blades. In Proceedings of the 2017 2nd International Conference on Advanced Robotics and Mechatronics (ICARM), Hefei and Tai’an, China, 27–31 August 2017; pp. 161–166. [Google Scholar]
- Russakovsky, O.; Deng, J.; Su, H.; Krause, J.; Satheesh, S.; Ma, S.; Huang, Z.; Karpathy, A.; Khosla, A.; Bernstein, M.; et al. ImageNet Large Scale Visual Recognition Challenge. Int. J. Comput. Vis. (IJCV) 2015, 115, 211–252. [Google Scholar] [CrossRef] [Green Version]
- Qiu, Z.; Wang, S.; Zeng, Z.; Yu, D. Automatic visual defects inspection of wind turbine blades via YOLO-based small object detection approach. J. Electron. Imaging 2019, 28, 1–11. [Google Scholar] [CrossRef]
- Shihavuddin, A.; Chen, X.; Fedorov, V.; Nymark Christensen, A.; Andre Brogaard Riis, N.; Branner, K.; Bjorholm Dahl, A.; Reinhold Paulsen, R. Wind Turbine Surface Damage Detection by Deep Learning Aided Drone Inspection Analysis. Energies 2019, 12, 676. [Google Scholar] [CrossRef] [Green Version]
- Padilla, R.; Passos, W.L.; Dias, T.L.B.; Netto, S.L.; da Silva, E.A.B. A Comparative Analysis of Object Detection Metrics with a Companion Open-Source Toolkit. Electronics 2021, 10, 279. [Google Scholar] [CrossRef]
- Dutta, A.; Gupta, A.; Zissermann, A. VGG Image Annotator (VIA). Version: 2.0.10. 2016. Available online: http://www.robots.ox.ac.uk/~vgg/software/via/ (accessed on 18 December 2020).
- Kubota, Y.; Shiono, T. White Balance Control System. US Patent No. 3,627,911, 14 December 1971. [Google Scholar]
- Umbaugh, S.E. Computer Vision and Image Processing: A Practical Approach Using Cviptools with Cdrom, 1st ed.; Prentice Hall PTR: Upper Saddle River, NJ, USA, 1997. [Google Scholar]
- Pizer, S.M.; Amburn, E.P.; Austin, J.D.; Cromartie, R.; Geselowitz, A.; Greer, T.; ter Haar Romeny, B.; Zimmerman, J.B.; Zuiderveld, K. Adaptive histogram equalization and its variations. Comput. Vis. Graph. Image Process. 1987, 39, 355–368. [Google Scholar] [CrossRef]
- Afifi, M.; Brown, M.S. What Else Can Fool Deep Learning? Addressing Color Constancy Errors on Deep Neural Network Performance. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), Seoul, Korea, 27–28 October 2019. [Google Scholar]
- He, K.; Zhang, X.; Ren, S.; Sun, J. Deep Residual Learning for Image Recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, 27–30 June 2016. [Google Scholar]
- Hoxha, E.; Vidal, Y.; Pozo, F. Damage Diagnosis for Offshore Wind Turbine Foundations Based on the Fractal Dimension. Appl. Sci. 2020, 10, 6972. [Google Scholar] [CrossRef]
- Zhang, X.; Zhao, Z.; Wang, Z.; Wang, X. Fault Detection and Identification Method for Quadcopter Based on Airframe Vibration Signals. Sensors 2021, 21, 581. [Google Scholar] [CrossRef] [PubMed]
- Li, H.; Zhang, X.; Xu, F. Experimental Investigation on Centrifugal Compressor Blade Crack Classification Using the Squared Envelope Spectrum. Sensors 2013, 13, 12548–12563. [Google Scholar] [CrossRef] [Green Version]
- Sanati, H.; Wood, D.; Sun, Q. Condition Monitoring of Wind Turbine Blades Using Active and Passive Thermography. Appl. Sci. 2018, 8, 2004. [Google Scholar] [CrossRef] [Green Version]
- Wang, J.; Huo, L.; Liu, C.; Peng, Y.; Song, G. Feasibility Study of Real-Time Monitoring of Pin Connection Wear Using Acoustic Emission. Appl. Sci. 2018, 8, 1775. [Google Scholar] [CrossRef] [Green Version]
Author | Year | Method | Result | Limitation |
---|---|---|---|---|
Kawiecki [27] | 1999 | Neural Network | <15% test error | Data collection requires professional NDT techniques. CNN architecture is outdated. |
Jasinien et al. [28] | 2009 | Ultrasonic & radiographic | N/A | Requires professional NDT. Paper lacks a thorough evaluation and only provides example outputs. |
Protopapadakis & Doulamis [29] | 2015 | CNN, SVM, k-NN, DT | CNN: 88.6%, SVM: 71.6%, k-NN: 74.6%, DT: 67.3% | N/A. |
Yu et al. [30] | 2017 | CNN+SVM | 100% Accuracy | Methods can only classify the defects but cannot provide location information of the defect in the images. |
Qiu et al. [32] | 2019 | YSODA (CNN) | 91.3% Accuracy | Detection speed is slower than YOLOv3 |
Shihavuddin et al. [33] | 2019 | Faster R-CNN | 81.10% mAP@IoU(0.5) | Slow detection speed. |
Reddy et al. [9] | 2019 | CNN | 94.94% Accuracy | Method only achieved high accuracy in binary classification mode (fault vs. non-fault). |
Yang et al. [10] | 2020 | CNN | 95.58% Accuracy | Long training time. |
Deng et al. [11] | 2020 | YOLOv2 (CNN) | 77% mAP@IoU(0.5) | Outdated YOLO version. Slow detection speed. |
Type | Number of Images |
---|---|
Crack | 55 |
Erosion | 62 |
Void | 52 |
Other | 22 |
Total | 191 |
Dataset | Image Augmentation Settings |
---|---|
D0 | Original |
D1 | Original |
+Vertical Flip + Horizontal Flip | |
+90 Rotation + 180 Rotation + 270 Rotation | |
+Greyscale Original | |
D2 | Original |
+Vertical Flip + Horizontal Flip | |
+90 Rotation + 180 Rotation + 270 Rotation | |
D3 | Greyscale Original |
+Greyscale Vertical Flip + Greyscale Horizontal Flip | |
+Greyscale 90 Rotation + Greyscale 180 Rotation + Greyscale 270 Rotation |
Dataset | Number of Training Images | Number of Testing Images | Total Images |
---|---|---|---|
D0 | 147 | 44 | 191 |
D1 | 1069 | 268 | 1337 |
D2 | 923 | 223 | 1146 |
D3 | 923 | 223 | 1146 |
Prediction Box Accuracy (PBA) | |||||
Defect type | Dataset D0 | Dataset D1 | Dataset D2 | Dataset D3 | |
Crack | 87.88% ± 0.11 | 69.20% ± 0.10 | 84.83% ± 0.076 | 64.29% ± 0.23 | |
Erosion | 66.06% ± 0.24 | 75.35% ± 0.083 | 84.79% ± 0.080 | 79.20% ± 0.091 | |
Void | 79.10% ± 0.17 | 80.10% ± 0.097 | 99.32% ± 0.052 | 99.22% ± 0.051 | |
Other | 71.49% ± 0.10 | 92.76% ± 0.098 | 89.03% ± 0.074 | 70.04% ± 0.094 | |
std(PBA) | ±0.09 | ±0.10 | ±0.069 | ±0.15 | |
WA(PBA) | 77.59% ± 0.19 | 81.62% ± 0.10 | 91.99% ± 0.07 | 83.98% ± 0.13 | |
Recognition Rate (RR) and (False Label Rate (FLR)) | |||||
Defect type | Dataset D0 | Dataset D1 | Dataset D2 | Dataset D3 | |
Cracks | 45.00% (15.0%) | 45.21% (14.4%) | 36.63% (7.0%) | 51.45% (21.4%) | |
Erosion | 62.50% (0.0%) | 76.36% (8.2%) | 65.93% (5.5%) | 84.95% (12.9%) | |
Void | 47.92% (16.7%) | 51.14% (8.2%) | 39.69% (4.1%) | 63.24% (3.2%) | |
Other | 33.33% (0.0%) | 53.16% (3.1%) | 40.74% (7.4%) | 62.96% (3.7%) | |
std(RR) | ±0.12 | ±0.13 | ±0.14 | ±0.14 | |
WA(RR) | 48.89% (12.2%) | 53.94% (8.8%) | 43.60% (5.6%) | 63.18% (11.7%) | |
Detection -score | |||||
Defect type | Dataset D0 | Dataset D1 | Dataset D2 | Dataset D3 | |
Crack | 41.38% | 42.49% | 43.40% | 39.69% | |
Erosion | 76.92% | 77.32% | 72.85% | 77.91% | |
Void | 42.25% | 57.31% | 50.92% | 73.51% | |
Other | 50.00% | 68.00% | 47.37% | 72.73% | |
std() | ±0.17 | ±0.15 | ±0.13 | ±0.17 | |
WA() | 49.25% | 58.67% | 52.95% | 63.08% | |
Average Performance | |||||
Defect type | Dataset D0 | Dataset D1 | Dataset D2 | Dataset D3 | |
Crack | 58.09% | 52.30% | 60.73% | 51.81% | |
Erosion | 68.49% | 76.34% | 74.52% | 80.69% | |
Void | 56.42% | 62.85% | 63.31% | 78.66% | |
Other | 51.61% | 71.31% | 56.58% | 67.34% | |
mAP@IoU(0.5) | 37.10% | 55.69% | 49.66% | 53.28% | |
mStd | ±0.13 | ±0.13 | ±0.11 | ±0.15 | |
mWA | 58.58% | 64.74% | 62.58% | 70.08% | |
mFLR | 12.2% | 8.8% | 5.6% | 11.7% |
Prediction Box Accuracy | |||||
Defect type | Dataset D0 | Dataset D1 | Dataset D2 | Dataset D3 | |
Crack | 70.99% ± 0.25 | 79.50% ± 0.082 | 80.08% ± 0.11 | 77.47% ± 0.16 | |
Erosion | 88.45% ± 0.37 | 65.95% ± 0.20 | 73.27% ± 0.18 | 69.51% ± 0.15 | |
Void | 89.55% ± 0.45 | 89.71% ± 0.14 | 91.33% ± 0.092 | 88.76% ± 0.097 | |
Other | 46.48% ± 0.30 | 50.60% ± 0.24 | 46.65% ± 0.34 | 59.64% ± 0.17 | |
std(PBA) | ±0.20 | ±0.17 | ±0.19 | ±0.12 | |
WA(PBA) | 82.08% ± 0.23 | 81.71% ± 0.16 | 84.08% ± 0.14 | 80.83% ± 0.14 | |
Recognition Rate (RR) and (False Label Rate (FLR)) | |||||
Defect type | Dataset D0 | Dataset D1 | Dataset D2 | Dataset D3 | |
Crack | 35.00% (20.0%) | 64.36% (18.6%) | 59.30% (9.9%) | 62.79% (23.3%) | |
Erosion | 75.00% (6.3%) | 93.75% (25.9%) | 81.72% (12.9%) | 94.62% (23.7%) | |
Void | 50.00% (6.3%) | 84.09% (0.0%) | 85.95% (4.3%) | 85.41% (3.8%) | |
Other | 83.33% (33.3%) | 50.00% (25.0%) | 44.44% (22.2%) | 40.74% (11.1%) | |
std(RR) | ±0.22 | ±0.20 | ±0.20 | ±0.24 | |
WA(RR) | 53.33% (11.1%) | 79.11% (12.9%) | 73.17% (9.0%) | 76.52% (15.1%) | |
Detection F1-score | |||||
Defect type | Dataset D0 | Dataset D1 | Dataset D2 | Dataset D3 | |
Crack | 22.22% | 55.66% | 62.04% | 48.57% | |
Erosion | 78.57% | 70.05% | 75.74% | 72.93% | |
Void | 58.33% | 90.90% | 87.79% | 88.05% | |
Other | 54.55% | 33.33% | 30.77% | 42.11% | |
std() | ±0.23 | ±0.24 | ±0.25 | ±0.21 | |
WA() | 55.07% | 73.98% | 74.09% | 69.60% | |
Average Performance (type classification) | |||||
Defect type | Dataset D0 | Dataset D1 | Dataset D2 | Dataset D3 | |
Crack | 42.74% | 66.51% | 67.14% | 62.94% | |
Erosion | 80.67% | 76.58% | 76.91% | 79.02% | |
Void | 65.96% | 88.23% | 88.36% | 87.41% | |
Other | 61.45% | 44.64% | 40.62% | 47.50% | |
mAP@IoU(0.5) | 39.55% | 55.58% | 56.53% | 53.67% | |
mStd | ±0.22 | ±0.20 | ±0.22 | ±0.19 | |
mWA | 63.49% | 78.28% | 77.11% | 75.65% | |
mFLR | 11.1% | 12.9% | 9.0% | 15.1% |
Prediction Box Accuracy | |||||
Defect type | Dataset D0 | Dataset D1 | Dataset D2 | Dataset D3 | |
Crack | 89.64% ± 0.37 | 89.05% ± 0.15 | 88.49% ± 0.15 | 85.81% ± 0.17 | |
Erosion | 86.17% ± 0.069 | 89.39% ± 0.18 | 86.50% ± 0.21 | 87.02% ± 0.21 | |
Void | 76.99% ± 0.23 | 88.66% ± 0.11 | 87.38% ± 0.087 | 86.77% ± 0.083 | |
Other | 81.64% ± 0.12 | 89.94% ± 0.045 | 90.84% ± 0.049 | 89.70% ± 0.087 | |
std(PBA) | ±0.055 | ±0.054 | ±0.019 | ±0.017 | |
WA(PBA) | 83.56% ± 0.15 | 89.05% ± 0.14 | 87.80% ± 0.14 | 86.68% ± 0.15 | |
Recognition Rate (RR) and (False Label Rate (FLR)) | |||||
Defect type | Dataset D0 | Dataset D1 | Dataset D2 | Dataset D3 | |
Crack | 75.00% (0.0%) | 78.68% (2.9%) | 90.16% (4.1%) | 87.70% (0.8%) | |
Erosion | 75.00% (6.3%) | 88.00% (8.0%) | 93.75% (8.8%) | 87.50% (7.5%) | |
Void | 40.54% (5.4%) | 72.02% (3.0%) | 74.82% (2.2%) | 72.66% (2.2%) | |
Other | 50.00% (0.0%) | 66.67% (0.0%) | 88.00% (0.0%) | 80.00% (4.0%) | |
std(RR) | ±0.18 | ±0.092 | ±0.083 | ±0.072 | |
WA(RR) | 56.00% (4.0%) | 77.42% (3.9%) | 84.97% (4.1%) | 81.42% (3.0%) | |
Detection F1-score | |||||
Defect type | Dataset D0 | Dataset D1 | Dataset D2 | Dataset D3 | |
Crack | 85.71% | 84.77% | 90.52% | 92.58% | |
Erosion | 78.57% | 85.11% | 87.74% | 85.33% | |
Void | 50.00% | 80.28% | 83.13% | 81.67% | |
Other | 66.67% | 80.00% | 93.62% | 84.44% | |
std() | ±0.16 | ±0.028 | ±0.045 | ±0.047 | |
WA() | 66.67% | 82.86% | 87.44% | 86.45% | |
Average Performance | |||||
Defect type | Dataset D0 | Dataset D1 | Dataset D2 | Dataset D3 | |
Crack | 83.45% | 84.17% | 89.72% | 88.70% | |
Erosion | 79.91% | 87.50% | 89.33% | 86.62% | |
Void | 55.84% | 80.32% | 81.78% | 82.03% | |
Other | 66.10% | 78.87% | 90.82% | 84.72% | |
mAP@IoU(0.5) | 57.47% | 77.53% | 82.57% | 76.80% | |
mStd | ±0.13 | ±0.06 | ±0.05 | ±0.05 | |
mWA | 68.74% | 83.11% | 86.74% | 84.85% | |
mFLR | 4.0% | 3.9% | 4.1% | 3.0% |
Prediction Box Accuracy (PBA) | ||||
Defect type | Mask R-CNN vs. YOLOv3 | Mask R-CNN vs. YOLOv4 | YOLOv4 vs. YOLOv3 | |
Crack | +24.20% | +8.99% | +15.21% | |
Erosion | +7.29% | +20.54% | −13.25% | |
Void | −11.84% | −2.33% | −9.51% | |
Other | +20.81% | +40.24% | −19.44% | |
std(PBA) | −0.131 | −0.151 | +0.02 | |
WA(PBA) | +3.82% | +6.09% | −2.27% | |
Recognition Rate (RR) and (False Label Rate (FLR)) | ||||
Defect type | Mask R-CNN vs. YOLOv3 | Mask R-CNN vs. YOLOv4 | YOLOv4 vs. YOLOv3 | |
Crack | +38.72% (−17.30%) | +25.80% (−14.50%) | +12.92% (−2.80%) | |
Erosion | +8.80% (−4.10%) | +0.00% (−17.10%) | +8.80% (+13.00%) | |
Void | +11.58% (−1.00%) | −9.27% (+2.20%) | +20.85% (−3.20%) | |
Other | +25.04% (−3.70%) | +38.00% (−25.00%) | −12.96% (+21.30%) | |
std(RR) | −0.057 | −0.117 | 0.06 | |
WA(RR) | +21.79% (−7.60%) | +5.87% (−8.80%) | +15.93% (+1.20%) | |
Detection -score | ||||
Defect type | Mask R-CNN vs. YOLOv3 | Mask R-CNN vs. YOLOv4 | YOLOv4 vs. YOLOv3 | |
Crack | +50.82% | +34.85% | +15.97% | |
Erosion | +9.83% | +17.70% | −7.86% | |
Void | +9.62% | −7.78% | +17.40% | |
Other | +20.89% | +60.28% | −39.39% | |
std() | −0.125 | −0.195 | +0.07 | |
WA() | +24.37% | +13.47% | +10.90% | |
Average Performance | ||||
Defect type | Mask R-CNN vs. YOLOv3 | Mask R-CNN vs. YOLOv4 | YOLOv4 vs. YOLOv3 | |
Crack | +37.91% | +23.22% | +14.70% | |
Erosion | +8.64% | +12.75% | −4.10% | |
Void | +3.12% | −6.46% | +9.58% | |
Other | +22.25% | +46.18% | −23.93% | |
mAP@IoU(0.5) | 29.29% | 26.99% | 2.30% | |
mStd | −0.104 | −0.154 | +0.05 | |
mAW | +16.66% | +8.47% | +8.19% | |
mFLR | −7.60% | −8.80% | +1.20% |
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |
© 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).
Share and Cite
Zhang, J.; Cosma, G.; Watkins, J. Image Enhanced Mask R-CNN: A Deep Learning Pipeline with New Evaluation Measures for Wind Turbine Blade Defect Detection and Classification. J. Imaging 2021, 7, 46. https://doi.org/10.3390/jimaging7030046
Zhang J, Cosma G, Watkins J. Image Enhanced Mask R-CNN: A Deep Learning Pipeline with New Evaluation Measures for Wind Turbine Blade Defect Detection and Classification. Journal of Imaging. 2021; 7(3):46. https://doi.org/10.3390/jimaging7030046
Chicago/Turabian StyleZhang, Jiajun, Georgina Cosma, and Jason Watkins. 2021. "Image Enhanced Mask R-CNN: A Deep Learning Pipeline with New Evaluation Measures for Wind Turbine Blade Defect Detection and Classification" Journal of Imaging 7, no. 3: 46. https://doi.org/10.3390/jimaging7030046
APA StyleZhang, J., Cosma, G., & Watkins, J. (2021). Image Enhanced Mask R-CNN: A Deep Learning Pipeline with New Evaluation Measures for Wind Turbine Blade Defect Detection and Classification. Journal of Imaging, 7(3), 46. https://doi.org/10.3390/jimaging7030046