Dual-Domain Prior-Driven Deep Network for Infrared Small-Target Detection
<p>The workflow of the proposed dual-domain prior-driven deep network (DPDNet). The targets in <math display="inline"><semantics><mover accent="true"><mi>x</mi><mo>¯</mo></mover></semantics></math> become more salient after being processed by the sparse-characteristic-driven module. The high-frequency-characteristic-driven module is embedded into the main detection network to extract and fuse dual-domain features of the targets.</p> "> Figure 2
<p>The comparison results between the original images and the corresponding <math display="inline"><semantics><mover accent="true"><mi>x</mi><mo>¯</mo></mover></semantics></math> generated by the sparse-characteristic-driven module. The small targets are more discernible to the naked eye in <math display="inline"><semantics><mover accent="true"><mi>x</mi><mo>¯</mo></mover></semantics></math>. The targets of each image are highlight by the red dotted squares.</p> "> Figure 3
<p>Examples showing the outputs of the high-frequency characteristic extraction module. The first line shows the original images. The second line shows the output after applying Equation (15).</p> "> Figure 4
<p>An illustration of the detection module: The input image is downsampled to different resolutions by the encoder. There are densely distributed nodes in the skip connection module, where the high-frequency-characteristic-driven module is inserted and used to repeatedly fuse the multilayer features. Finally, the feature resolution is restored by the decoder, and the targets are predicted.</p> "> Figure 5
<p>An illustration of connections among nodes in the network: It mainly consists of two U-Nets with different numbers of layers, and each internal node fuses the features of the layer from the same level as well as the upper and lower layers. In the same layer, nonadjacent nodes conduct skip connections, as shown by the dotted line in the figure.</p> "> Figure 6
<p>An illustration of the high-frequency-characteristic-driven module in the network; it consists of two parts: The input feature map is propagated through the spatial-domain pathway and the frequency pathway. Then, the features of the two paths are integrated.</p> "> Figure 7
<p>Visualization results of feature maps propagated among sub-nodes after the high-frequency-characteristic-driven module.</p> "> Figure 8
<p>(<b>a</b>–<b>x</b>) are some representative infrared images in the SIRST dataset with different backgrounds. For a better display, the demarcated area is enlarged by red square [<a href="#B33-remotesensing-15-03827" class="html-bibr">33</a>].</p> "> Figure 9
<p>The comparison results between the proposed DPDNet and previous CNN-based methods. The DPDNet achieves a higher true positive rate at the same false positive rate.</p> "> Figure 10
<p>The definition of the feature layers.</p> "> Figure 11
<p>The 3D visualization results of the original images and <math display="inline"><semantics><mover accent="true"><mi>x</mi><mo>¯</mo></mover></semantics></math>. The second and fourth lines show the pixel values of the original images and <math display="inline"><semantics><mover accent="true"><mi>x</mi><mo>¯</mo></mover></semantics></math>, respectively. The target pixels are more distinguishable from the background pixels in <math display="inline"><semantics><mover accent="true"><mi>x</mi><mo>¯</mo></mover></semantics></math> than those in the original images.</p> "> Figure 12
<p>The feature maps and their corresponding activation maps. The target regions are well activated and utilized during the feature propagation.</p> ">
Abstract
:1. Introduction
- Purely model-driven methods face challenges in achieving precise modeling because of their heavy dependence on scholars’ expertise and experience. These methods are typically simplified forms of real data, which limits their ability to address complex real-world scenarios. Consequently, they encounter difficulties in terms of their detection performance and robustness.
- While current data-driven methods have demonstrated favorable results, achieving precise detection of infrared small targets is still a difficult task. The main challenges arise from the limited proportion of pixels occupied by targets, the serious imbalance between the foreground and background, and the weak semantic connections between the targets and the environment [12]. Since data-driven approaches fail to leverage domain-specific knowledge effectively, simply deepening the neural network has shown minimal impact on improving detection performance.
- Most deep networks for infrared small-target detection rely on supervised learning, which utilizes labeled data. However, the simulation of infrared data is not perfect, and measured data lack samples in areas of infrared target detection and recognition [13]. Even if a batch of high-quality labeled samples is amassed, the training models are still sensitive to the variations of backgrounds, potentially leading to poor generalization performance on new datasets [14].
- We propose a novel dual-domain prior-driven deep network for infrared small-target detection, which integrates the data-driven methods with the model-driven approaches.
- We guide supervised data-driven models by proposing prior-driven modules that embed domain knowledge at both the input and the inner levels of the network.
- We analyze the effectiveness and reliability of the prior-driven modules in guiding learning and enhancing the expression capability of target features.
2. Background
2.1. Informed Machine Learning
2.2. Sparse Characteristic of Infrared Small Targets
2.3. High-Frequency Characteristics of Infrared Small Targets
3. Methods
3.1. Sparse-Characteristic-Driven Module
3.2. High-Frequency Characteristic Extraction Module
3.3. Detection Module
4. Results
4.1. Dataset
4.2. Implementation Details
4.3. Evaluation Metrics
4.4. Effect Verification
4.5. Ablation Study
5. Discussion
5.1. Physics Explanation of
5.2. Physical Interpretability Discussion
6. Conclusions
Author Contributions
Funding
Data Availability Statement
Conflicts of Interest
References
- Song, Q.; Wang, Y.; Dai, K.; Bai, K. Single frame infrared image small target detection via patch similarity propagation based background estimation. Infrared Phys. Technol. 2020, 106, 103197. [Google Scholar] [CrossRef]
- Fan, J.; Wei, J.; Huang, H.; Zhang, D.; Chen, C. IRSDT: A Framework for Infrared Small Target Tracking with Enhanced Detection. Sensors 2023, 23, 4240. [Google Scholar] [CrossRef] [PubMed]
- Deshpande, S.D.; Er, M.H.; Venkateswarlu, R.; Chan, P. Max-mean and max-median filters for detection of small targets. In Proceedings of the SPIE’s International Symposium on Optical Science, Engineering, and Instrumentation, Denver, CO, USA, 4 October 1999; pp. 74–83. [Google Scholar] [CrossRef]
- Rivest, J.F.; Fortin, R. Detection of dim targets in digital infrared imagery by morphological image processing. Opt. Eng. 1996, 35, 1886–1893. [Google Scholar] [CrossRef]
- Wei, Y.; You, X.; Li, H. Multiscale patch-based contrast measure for small infrared target detection. Pattern Recognit. 2016, 58, 216–226. [Google Scholar] [CrossRef]
- Chen, C.L.P.; Li, H.; Wei, Y.; Xia, T.; Tang, Y.Y. A Local Contrast Method for Small Infrared Target Detection. IEEE Trans. Geosci. Remote Sens. 2014, 52, 574–581. [Google Scholar] [CrossRef]
- Dai, Y.; Wu, Y. Reweighted Infrared Patch-Tensor Model With Both Nonlocal and Local Priors for Single-Frame Small Target Detection. IEEE J. Sel. Top. Appl. Earth Observ. Remote Sens. 2017, 10, 3752–3767. [Google Scholar] [CrossRef]
- Gao, C.; Meng, D.; Yang, Y.; Wang, Y.; Zhou, X.; Hauptmann, A.G. Infrared Patch-Image Model for Small Target Detection in a Single Image. IEEE Trans. Image Process. 2013, 22, 4996–5009. [Google Scholar] [CrossRef] [PubMed]
- Liu, M.; Du, H.; Zhao, Y.; Dong, L.; Hui, M. Image Small Target Detection based on Deep Learning with SNR Controlled Sample Generation. In Current Trends in Computer Science and Mechanical Automation; Wang, S.X., Ed.; De Gruyter Open Ltd.: Warsaw, Poland, 2017; Volume 1, pp. 211–220. Available online: https://www.webofscience.com/wos/alldb/full-record/WOS:000594997600023 (accessed on 6 May 2023).
- Hou, Q.; Wang, Z.; Tan, F.; Zhao, Y.; Zheng, H.; Zhang, W. RISTDnet: Robust Infrared Small Target Detection Network. IEEE Geosci. Remote Sens. Lett. 2022, 19, 7000805. [Google Scholar] [CrossRef]
- Dai, Y.; Wu, Y.; Zhou, F.; Barnard, K. Attentional Local Contrast Networks for Infrared Small Target Detection. IEEE Trans. Geosci. Remote Sens. 2021, 59, 9813–9824. [Google Scholar] [CrossRef]
- Lu, Z.; Zhi-Yong, Z.; Shan-Zhu, X.; Huan-Zhang, L.U. Detection of Dim Infrared Targets by Multi-Direction Prediction of Background. J. Signal Process. 2010, 26, 1646–1651. [Google Scholar]
- Bingwei, H.; Zhiyong, S.; Hongqi, F.; Ping, Z.; Weidong, H.; Xiaofeng, Z.; Jianguo, L.; Hongyan, S.; Wei, J.; Yongjie, Z.; et al. A Dataset for Infrared Image Dim-Small Aircraft Target Detection and Tracking Under Ground/Air Background; Science Data Bank: Beijing, China, 2019; p. 912518825. [Google Scholar] [CrossRef]
- Huang, Z.; Yao, X.; Liu, Y.; Dumitru, C.O.; Datcu, M.; Han, J. Physically Explainable CNN for SAR Image Classification. ISPRS J. Photogramm. Remote Sens. 2022, 190, 25–37. [Google Scholar] [CrossRef]
- Xia, C.; Li, X.; Zhao, L.; Shu, R. Infrared Small Target Detection Based on Multiscale Local Contrast Measure Using Local Energy Factor. IEEE Geosci. Remote Sens. Lett. 2020, 17, 157–161. [Google Scholar] [CrossRef]
- Chen, Z.; Chen, S.; Zhai, Z.; Zhao, M.; Jie, F.; Li, W. Infrared small-target detection via tensor construction and decomposition. Remote Sens. Lett. 2021, 12, 900–909. [Google Scholar] [CrossRef]
- Zhang, X.; Ding, Q.; Luo, H.; Hui, B.; Chang, Z. Zhang. Infrared small target detection based on an image-patch tensor model. Infrared Phys. Technol. 2019, 99, 55–63. [Google Scholar] [CrossRef]
- Beckh, K.; Müller, S.; Jakobs, M.; Toborek, V.; Tan, H.; Fischer, R.; Welke, P.; Houben, S.; von Rueden, L. Explainable Machine Learning with Prior Knowledge: An Overview. arXiv 2021, arXiv:2105.10172. [Google Scholar]
- Karniadakis, G.E.; Kevrekidis, I.G.; Lu, L.; Perdikaris, P.; Wang, S.; Yang, L. Physics-informed machine learning. Nat. Rev. Phys. 2021, 3, 6. [Google Scholar] [CrossRef]
- Wu, L.; Fang, S.; Ma, Y.; Fan, F.; Huang, J. Infrared small target detection based on gray intensity descent and local gradient watershed. Infrared Phys. Technol. 2022, 123, 104171. [Google Scholar] [CrossRef]
- Li, F.; Liu, S.; Qin, H. Dim Infrared Targets Detection Based on Adaptive Bilateral Filtering. Guangzi Xuebao/Acta Photonica Sin. 2010, 39, 1129–1131. [Google Scholar] [CrossRef]
- Wang, F.; Li, C.; Wu, B.; Yu, K.; Jin, C. Infrared small target detection method based on multi-scale feature fusion. J. Phys. Conf. Ser. 2021, 2024, 012012. [Google Scholar] [CrossRef]
- Zhang, X.; Ru, J.; Wu, C. Infrared Small Target Detection Based on Gradient Correlation Filtering and Contrast Measurement. IEEE Trans. Geosci. Remote Sens. 2023, 61, 5603012. [Google Scholar] [CrossRef]
- Zhang, Y.; Zhang, J.; Wang, D.; Chen, C. Infrared small target detection based on morphology and wavelet transform. In Proceedings of the 2011 2nd International Conference on Artificial Intelligence, Management Science and Electronic Commerce (AIMSEC), Zhengzhou, China, 8–10 August 2011; pp. 4033–4036. [Google Scholar] [CrossRef]
- Fan, Z.; Bi, D.; Xiong, L.; Ma, S.; He, L.; Ding, W. Dim infrared image enhancement based on convolutional neural network. Neurocomputing 2018, 272, 396–404. [Google Scholar] [CrossRef]
- IR Image Small Target Detection Based on Multi-Scale Feature Fusion. Available online: https://www.webofscience.com/wos/alldb/full-record/CSCD:2104565 (accessed on 6 May 2023).
- Zhang, Y.; Zhang, Y.; Shi, Z.; Zhang, J.; Wei, M. Design and Training of Deep CNN-Based Fast Detector in Infrared SUAV Surveillance System. IEEE Access 2019, 7, 137365–137377. [Google Scholar] [CrossRef]
- Li, B.; Xiao, C.; Wang, L.; Wang, Y.; Lin, Z.; Li, M.; An, W.; Guo, Y. Dense Nested Attention Network for Infrared Small Target Detection. arXiv 2022, arXiv:2106.00487. [Google Scholar] [CrossRef]
- Ell, T.A. Quaternion-Fourier transforms for analysis of two-dimensional linear time-invariant partial differential systems. In Proceedings of the 32nd IEEE Conference on Decision and Control, San Antonio, TX, USA, 15–17 December 1993; Volume 2, pp. 1830–1841. [Google Scholar] [CrossRef]
- Zhang, R.; Mu, C.; Xu, M.; Xu, L.; Shi, Q.; Wang, J. Synthetic IR Image Refinement Using Adversarial Learning With Bidirectional Mappings. IEEE Access 2019, 7, 153734–153750. [Google Scholar] [CrossRef]
- Ronneberger, O.; Fischer, P.; Brox, T. U-Net: Convolutional Networks for Biomedical Image Segmentation. arXiv 2015, arXiv:1505.04597. [Google Scholar] [CrossRef]
- Woo, S.; Park, J.; Lee, J.-Y.; Kweon, I.S. CBAM: Convolutional Block Attention Module. arXiv 2018, arXiv:1807.06521. [Google Scholar] [CrossRef]
- Dai, Y.; Wu, Y.; Zhou, F.; Barnard, K. Asymmetric Contextual Modulation for Infrared Small Target Detection. arXiv 2020, arXiv:2009.14530. [Google Scholar] [CrossRef]
- He, K.; Zhang, X.; Ren, S.; Sun, J. Deep Residual Learning for Image Recognition. arXiv 2015, arXiv:1512.03385. [Google Scholar] [CrossRef]
- Duchi, J.; Hazan, E.; Singer, Y. Adaptive subgradient methods for online learning and stochastic optimization. J. Mach. Learn. Res. 2011, 12, 2121–2159. [Google Scholar]
- Glorot, X.; Bengio, Y. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the International Conference on Artificial Intelligence and Statistics, Sardinia, Italy, 13–15 May 2010; Available online: https://www.semanticscholar.org/paper/Understanding-the-difficulty-of-training-deep-Glorot-Bengio/b71ac1e9fb49420d13e084ac67254a0bbd40f83f (accessed on 6 May 2023).
- Han, J.; Moradi, S.; Faramarzi, I.; Zhang, H.; Zhao, Q.; Zhang, X.; Li, N. Small Target Detection Based on the Weighted Strengthened Local Contrast Measure. IEEE Geosci. Remote Sens. Lett. 2021, 18, 1670–1674. [Google Scholar] [CrossRef]
- Miss Detection vs. False Alarm: Adversarial Learning for Small Object Segmentation in Infrared Images. IEEE Conference Publication. IEEE Xplore. Available online: https://ieeexplore.ieee.org/document/9009584 (accessed on 6 May 2023).
- Kwan, C.; Budavari, B. Enhancing Small Moving Target Detection Performance in Low-Quality and Long-Range Infrared Videos Using Optical Flow Techniques. Remote Sens. 2020, 12, 24. [Google Scholar] [CrossRef]
Method | |||
---|---|---|---|
(Tr = 50%) | |||
Filtering-Based: Top-Hat [4] | 7.143 | 79.84 | 1012 |
Local-Contrast-Based: WSLCM [8] | 1.158 | 77.95 | 5446 |
Local-Rank-Based: IPI [3] | 25.67 | 85.55 | 11.47 |
CNN-Based: MDvsFA-cGAN [38] | 60.3 | 89.35 | 56.35 |
CNN-Based: ACM [33] | 70.33 | 93.91 | 3.728 |
CNN-Based: ALCNet [11] | 73.33 | 96.57 | 30.47 |
CNN-Based: DNANet [28] | 76.24 | 97.71 | 12.8 |
DPDNet (Ours) | 78.64 | 95.56 | 2.15 |
Method | #Params(M) | |||
---|---|---|---|---|
(Tr = 50%) | ||||
DNANet-ResNet10 [28] | 76.24 | 97.71 | 12.8 | 2.61 |
DNANet-ResNet18 [28] | 77.47 | 98.48 | 2.35 | 4.7 |
DNANet-ResNet34 [28] | 77.54 | 98.1 | 2.51 | 8.79 |
DPDNet-ResNet10 (Ours) | 78.64 | 95.56 | 2.15 | 1.81 |
Shared Module | Sparse-Characteristic-Driven Module | High-Frequency-Characteristic-Driven Module | |||
---|---|---|---|---|---|
(Tr = 50%) | |||||
ResNet10 and Dense-Net | 96.24 | 97.71 | 12.8 | ||
✓ | 78.18 | 94.44 | 5.09 | ||
✓ | 78.53 | 95.19 | 6.23 | ||
✓ | ✓ | 78.64 | 95.56 | 2.15 |
Method Description | #Params(M) | |||
---|---|---|---|---|
(Tr = 50%) | ||||
Layer 0 | 77.08 | 94.81 | 7.02 | 2.6 |
Layer 0, 1 | 78.04 | 94.81 | 4.8 | 2.5 |
Layer 0, 1, 2 | 78.2 | 96.3 | 0.72 | 2.27 |
Layer 0, 1, 2, 3 | 78.64 | 95.56 | 2.15 | 1.84 |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2023 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Hao, Y.; Liu, Y.; Zhao, J.; Yu, C. Dual-Domain Prior-Driven Deep Network for Infrared Small-Target Detection. Remote Sens. 2023, 15, 3827. https://doi.org/10.3390/rs15153827
Hao Y, Liu Y, Zhao J, Yu C. Dual-Domain Prior-Driven Deep Network for Infrared Small-Target Detection. Remote Sensing. 2023; 15(15):3827. https://doi.org/10.3390/rs15153827
Chicago/Turabian StyleHao, Yutong, Yunpeng Liu, Jinmiao Zhao, and Chuang Yu. 2023. "Dual-Domain Prior-Driven Deep Network for Infrared Small-Target Detection" Remote Sensing 15, no. 15: 3827. https://doi.org/10.3390/rs15153827
APA StyleHao, Y., Liu, Y., Zhao, J., & Yu, C. (2023). Dual-Domain Prior-Driven Deep Network for Infrared Small-Target Detection. Remote Sensing, 15(15), 3827. https://doi.org/10.3390/rs15153827