A Study of Defect Detection Techniques for Metallographic Images
<p>Zeiss Axiovert 200 Mat optical microscope.</p> "> Figure 2
<p>(<b>a</b>) Basic block of residual network; (<b>b</b>) bottleneck residual network; (<b>c</b>) dilated residual network.</p> "> Figure 3
<p>Framework of ResNet and feature pyramid network (FPN).</p> "> Figure 4
<p>A 3D example of dilated convolution. Convolution layer with kernel size 3 × 3, (<b>a</b>) normal dilated convolution with rate = 1; (<b>b</b>) dilated convolution with rate = 2; (<b>c</b>) dilated convolution with rate = 3.</p> "> Figure 5
<p>M-ResNet retraining strategy.</p> "> Figure 6
<p>Five different classes of metallographic images differentiated form left to right. The top rows are normal images, and bottom rows are defective images.</p> "> Figure 7
<p>Examples of data augmentations with 9 operations that can be used to train the models for each image. (<b>a</b>) original, (<b>b</b>–<b>d</b>) flipping, (<b>e</b>–<b>g</b>) zooming, (<b>h</b>–<b>j</b>) rotation.</p> "> Figure 8
<p>The window-based program used to implement the proposed M-ResNet for metallographic analysis.</p> "> Figure 9
<p>Selected examples of defect detection results on the Metal Industries Research and Development Centre (MIRDC) dataset for various methods. Left column is original images, and middle and right columns are results of ResNet-50 and M-ResNet-50 models.</p> ">
Abstract
:1. Introduction
2. Related Works
3. Multi-Scale ResNet for Defect Detection
3.1. Deep Residual Network
3.2. Architecture of Multi-Scale Resnets
3.2.1. Upsampling and Concatenation
3.2.2. Bottleneck Residual Blocks
3.2.3. Feature Pyramid Network
3.2.4. Dilated Convolution
3.3. Retraining Strategy
4. Experimental Results and Analysis
4.1. Metallographic Dataset
4.2. Experimental Results and Analysis
4.3. Comparison with Other Object Detectors
5. Conclusions
Author Contributions
Funding
Conflicts of Interest
References
- Girshick, R.; Donahue, J.; Darrell, T.; Malik, J. Rich feature hierarchies for accurate object detection and semantic segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition CVPR, Columbus, OH, USA, 23–28 June 2014; pp. 580–587. [Google Scholar]
- Liu, W.; Anguelov, D.; Erhan, D.; Szegedy, C.; Reed, S.; Fu, C.-Y.; Berg, A.C. SSD: Single shot multibox detector. In Proceedings of the European Conference on Computer Vision (ECCV), Amsterdam, The Netherlands, 11–14 October 2016. [Google Scholar]
- Redmon, J.; Divvala, S.; Girshick, R.; Farhadi, A. You only look once: Unified, real-time object detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition CVPR, Las Vegas, NV, USA, 26 June–1 July 2016. [Google Scholar]
- He, K.; Zhang, X.; Ren, S.; Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition CVPR, Las Vegas, NV, USA, 26 June–1 July 2016. [Google Scholar]
- Bochkovskiy, A.; Wang, C.Y.; Mark Liao, H.Y. YOLOv4: Optimal speed and accuracy of object detection. In Proceedings of the Conference on Computer Vision and Pattern Recognition CVPR, Nanjing, China, 14–19 June 2020. [Google Scholar]
- Latala, Z.; Wojnar, L. Computer-aided versus manual grain size assessment in a single phase material. Mater. Charact. 2001, 46, 227–233. [Google Scholar] [CrossRef]
- Petersen, K.; Aldrich, C.; DeVenter, J. Analysis of ore particles based on textural pattern recognition. Miner. Eng. 1998, 11, 959–977. [Google Scholar] [CrossRef]
- Codaro, E.; Nakazato, R.; Horovistiz, A.; Riberio, L.; Hein, L. An image processing method for morphology characterization and pitting corrosion evaluation. Mater. Sci. Eng. A 2002, 334, 298–306. [Google Scholar] [CrossRef]
- Lee, S.G.; Mao, Y.; Gokhale, A.; Harris, J.; Horstemeyer, M. Application of digital image processing for automatic detection of characterization of cracked constituent particles/inclusions in wrought aluminum alloys. Mater. Charact. 2009, 60, 964–970. [Google Scholar] [CrossRef]
- Han, F.; Zou, J.; Ai, Y.; Xu, C.; Liu, S.; Liu, S. Image classification and analysis during the additive manufacturing process based on deep convolutional neural networks. In Proceedings of the 20th International Conference on Electronic Packaging Technology(ICEPT), Hong Kong, China, 11–15 August 2019. [Google Scholar]
- Boyadjian, Q.; Vanderesse, N.; Toews, M.; Bocher, P. Detecting defects in materials using deep convolutional neural networks. In Lecture Notes in Computer Science (LNCS); Springer: Berlin/Heidelberg, Germany, 2020; Volume 12131, pp. 293–306. [Google Scholar]
- Lin, J.; Ma, L.; Yao, Y. Segmentation of casting defect regions for the extraction of microstructural properties. Eng. Appl. Artif. Intell. 2019, 85, 150–163. [Google Scholar] [CrossRef]
- Voulodimos, A.; Nikolaos, D.; Anastasios, D.; Eftychios, P. Deep learning for computer vision: A brief review. Comput. Intell. Neurosci. 2018, 2018, 7068349. [Google Scholar] [CrossRef] [PubMed]
- Guo, Y.; Liu, Y.; Oerlemans, A.; Lao, S.; Wu, S.; Lew, M. Deep learning for visual understanding: A review. Neurocomputing 2016, 187, 27–48. [Google Scholar] [CrossRef]
- Uijlings, J.R.; van de Sande, K.E.; Gevers, T.; Smeulders, A.W. Selective search for object recognition. Int. J. Comput. Vis. 2013, 104, 154–171. [Google Scholar] [CrossRef] [Green Version]
- Girshick, R. Fast R-CNN. arXiv 2015, arXiv:1504.08083. [Google Scholar]
- Ren, S.; He, K.; Girshick, R.; Sun, J. Faster r-cnn: Towards real-time object detection with region proposal networks. In Advances in Neural Information Processing Systems; Neural Information Processing Systems Foundation Inc.: San Diego, CA, USA, 2015; pp. 91–99. [Google Scholar]
- He, K.; Gkioxari, G.; Dollár, P.; Girshick, R. Mask R-CNN. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), Venice, Italy, 22–29 October 2017. [Google Scholar]
- Redmon, J.; Farhadi, A. YOLO9000: Better, faster, stronger. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA, 21–26 July 2017. [Google Scholar]
- Krizhevsky, A.; Sutskever, I.; Hinton, G.E. Imagenet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems; Neural Information Processing Systems Foundation Inc.: San Diego, CA, USA, 2012; pp. 1097–1105. [Google Scholar]
- ASM Micrograph Database. Available online: https://www.asminternational.org/materials-resources/online-databases/-/journal_content/56/10192/15468789/DATABASE (accessed on 25 September 2020).
- Lin, T.Y.; Dollár, P.; Girshick, R.B.; He, K.; Hariharan, B.; Belongie, S.J. Feature pyramid networks for object detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA, 21–26 July 2017; pp. 21–26. [Google Scholar]
- Bulo, S.R.; Porzi, L.; Kontschieder, P. In-place activated batchnorm for memory-optimized training of DNNs. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, USA, 18–23 June 2018. [Google Scholar]
- Zhao, H.; Zhang, Y.; Liu, S.; Shi, J.; Loy, C.C.; Lin, D.; Jia, J. PSANet: Point-wise spatial attention network for scene parsing. In Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany, 8–14 September 2018. [Google Scholar]
- Yu, F.; Koltun, V. Multi-scale context aggregation by dilated convolutions. In Proceedings of the International Conference on Learning Representations (ICLR), San Juan, PR, USA, 2–4 May 2016. [Google Scholar]
- Everingham, M.; Van~Gool, L.; Williams, C.K.I.; Winn, J.; Zisserman, A. The pascal visual object classes (voc) challenge. Int. J. Comput. Vis. 2010, 88, 303–338. [Google Scholar] [CrossRef] [Green Version]
- Chen, S.; Abhinav, S.; Saurabh, S.; Abhinav, G. Revisting unreasonable effectiveness of data in deep learning era. In Proceedings of the International Conference on Computer Vision (ICCV), Venice, Italy, 22–29 October 2017; pp. 843–852. [Google Scholar]
- Long, X.; Deng, K.; Wang, G.; Zhang, Y.; Dang, Q.; Gao, Y.; Shen, H.; Ren, J. PP-YOLO: An effective and efficient implementation of object detector. arXiv 2020, arXiv:2007.12099. [Google Scholar]
- Jiao, L.; Zhang, F.; Liu, F.; Yang, S.; Li, L.; Feng, Z.; Qu, R. A survey of deep learning-based object detection. IEEE Access 2019, 7, 128837–128868. [Google Scholar] [CrossRef]
- Wei, S.; Su, H.; Ming, J.; Wang, C.; Yan, M.; Kumar, D.; Shi, J.; Zhang, X. Precise and robust ship detection for high-resolution SAR imagery based on HR-SDNet. Remote Sens. 2020, 12, 167. [Google Scholar] [CrossRef] [Green Version]
- Shorten, C.; Khoshgoftaar, T.M. A survey on image data augmentation for deep learning. J. Big Data 2019, 6, 1–48. [Google Scholar] [CrossRef]
- Antoniou, A.; Storkey, A.; Edwards, H. Data augmentation generative adversarial networks. arXiv 2017, arXiv:1711.04340. [Google Scholar]
Layer | Operation Type | Input (Pixel) | Filter | Stride | Dilation | Output (Pixel) | |
---|---|---|---|---|---|---|---|
0 | Convolution | 64 | 2 | Conv1 | |||
1 | Max pooling | 2 | |||||
2–5 | Bottleneck residual network Network | 256 | 1 | Conv2 | |||
6–9 | Bottleneck residual network | 1 | |||||
10–13 | Bottleneck residual network | 1 | |||||
14–17 | Bottleneck residual network | 512 | 2 | Conv3 | |||
18–21 | Bottleneck residual network | 1 | |||||
22–25 | Bottleneck residual network | 1 | |||||
26–29 | Bottleneck residual network | 1 | |||||
30–33 | Dilated residual network | 1024 | 2 | 3 | Conv4 | ||
34–37 | Dilated residual network | 1 | 3 | ||||
38–41 | Dilated residual network | 1 | 3 | ||||
42–45 | Dilated residual network | 1 | 3 | ||||
46–49 | Dilated residual network | 1 | 3 | ||||
50–53 | Dilated residual network | 1 | 3 | ||||
54–57 | Dilated residual network | 2048 | 2 | 3 | Conv5 | ||
58–61 | Dilated residual network | 1 | 3 | ||||
62–65 | Dilated residual network | 1 | 3 | ||||
66 | Convolution | 2048 | 1 | ||||
67 | Convolution | 18 | 1 | ||||
68 | Large-object detection | ||||||
69 | Route | 29 | |||||
70–73 | Dilated residual network | 1024 | 2 | 2 | Conv4 | ||
74–77 | Dilated residual network | 1 | 2 | ||||
78–81 | Dilated residual network | 1 | 2 | ||||
82–85 | Dilated residual network | 1 | 2 | ||||
86–89 | Dilated residual network | 1 | 2 | ||||
90–93 | Dilated residual network | 1 | 2 | ||||
94–97 | Dilated residual network | 2048 | 2 | 2 | Conv5 | ||
98–101 | Dilated residual network | 1 | 2 | ||||
102–105 | Dilated residual network | 1 | 2 | ||||
106 | Convolution | 1024 | 1 | ||||
107 | 2 Upsampling | 1024 | |||||
108 | Concatenation | 107, 33 | |||||
109 | Convolution | 1024 | 1 | ||||
110 | Convolution | 18 | 1 | ||||
111 | Medium-object detection | ||||||
112 | Route | 29 | |||||
113–116 | Dilated residual network | 1024 | 2 | 1 | Conv4 | ||
117–120 | Dilated residual network | 1 | 1 | ||||
121–124 | Dilated residual network | 1 | 1 | ||||
125–128 | Dilated residual network | 1 | 1 | ||||
129–132 | Dilated residual network | 1 | 1 | ||||
133–136 | Dilated residual network | 1 | 1 | ||||
137–140 | Dilated residual network | 2048 | 2 | 1 | Conv5 | ||
141–144 | Dilated residual network | 1 | 1 | ||||
145–148 | Dilated residual network | 1 | 1 | ||||
149 | Convolution | 512 | 1 | ||||
150 | 4 Upsampling | 512 | |||||
151 | Concatenation | 150, 17 | |||||
152 | Convolution | 512 | 1 | ||||
153 | Convolution | 18 | 1 | ||||
154 | Small-object detection |
Dataset | Method | APXS (%) | APS (%) | APM (%) | APL (%) | mAP (%) | Speed (FPS) |
---|---|---|---|---|---|---|---|
MIRDC dataset | M-ResNet-50 | 67.5 | 78.7 | 83.1 | 84.7 | 78.5 | 45 |
M-ResNet-101 | 73.4 | 80.1 | 85.4 | 86.7 | 81.4 | 32 | |
M-ResNet-152 | 78.9 | 85.4 | 88.3 | 90.2 | 85.7 | 27 | |
ResNet-50 | 55.8 | 60.7 | 68.1 | 75.7 | 65.1 | 87 | |
ResNet-101 | 62.5 | 67.1 | 72.4 | 78.4 | 70.1 | 58 | |
ResNet-152 | 69.7 | 73.4 | 76.3 | 82.2 | 75.4 | 44 |
Method | Backbone Network | mAP (%) | Speed (FPS) |
---|---|---|---|
Faster R-CNN | ResNet-101 | 77.8 | - |
SSD300 | VGG-16 | 74.7 | 48 |
YOLOv4 | darknet53 | 78.1 | 47 |
ResNet-50 | ResNet-50 | 65.1 | 87 |
M-ResNet-50 | ResNet-50 | 78.5 | 45 |
ResNet-101 | ResNet-101 | 70.1 | 58 |
M-ResNet-101 | ResNet-101 | 81.4 | 32 |
ResNet-152 | ResNet-152 | 75.4 | 44 |
M-ResNet-152 | ResNet-152 | 85.7 | 27 |
© 2020 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).
Share and Cite
Wu, W.-H.; Lee, J.-C.; Wang, Y.-M. A Study of Defect Detection Techniques for Metallographic Images. Sensors 2020, 20, 5593. https://doi.org/10.3390/s20195593
Wu W-H, Lee J-C, Wang Y-M. A Study of Defect Detection Techniques for Metallographic Images. Sensors. 2020; 20(19):5593. https://doi.org/10.3390/s20195593
Chicago/Turabian StyleWu, Wei-Hung, Jen-Chun Lee, and Yi-Ming Wang. 2020. "A Study of Defect Detection Techniques for Metallographic Images" Sensors 20, no. 19: 5593. https://doi.org/10.3390/s20195593
APA StyleWu, W. -H., Lee, J. -C., & Wang, Y. -M. (2020). A Study of Defect Detection Techniques for Metallographic Images. Sensors, 20(19), 5593. https://doi.org/10.3390/s20195593