CN111079683A - Remote sensing image cloud and snow detection method based on convolutional neural network - Google Patents
Remote sensing image cloud and snow detection method based on convolutional neural network Download PDFInfo
- Publication number
- CN111079683A CN111079683A CN201911350284.6A CN201911350284A CN111079683A CN 111079683 A CN111079683 A CN 111079683A CN 201911350284 A CN201911350284 A CN 201911350284A CN 111079683 A CN111079683 A CN 111079683A
- Authority
- CN
- China
- Prior art keywords
- feature
- convolution
- network
- feature map
- image
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000001514 detection method Methods 0.000 title claims abstract description 47
- 238000013527 convolutional neural network Methods 0.000 title claims abstract description 22
- 230000004927 fusion Effects 0.000 claims abstract description 37
- 238000000034 method Methods 0.000 claims abstract description 20
- 238000012545 processing Methods 0.000 claims abstract description 9
- 208000037170 Delayed Emergence from Anesthesia Diseases 0.000 claims description 59
- 238000010586 diagram Methods 0.000 claims description 24
- 230000006870 function Effects 0.000 claims description 10
- 238000012549 training Methods 0.000 claims description 10
- 101100400452 Caenorhabditis elegans map-2 gene Proteins 0.000 claims description 8
- 238000011176 pooling Methods 0.000 claims description 8
- 101150064138 MAP1 gene Proteins 0.000 claims description 5
- 238000010606 normalization Methods 0.000 claims description 4
- 238000007781 pre-processing Methods 0.000 claims description 4
- 210000002569 neuron Anatomy 0.000 claims description 2
- 238000013461 design Methods 0.000 abstract description 3
- 238000012360 testing method Methods 0.000 description 5
- 238000013528 artificial neural network Methods 0.000 description 4
- 230000010339 dilation Effects 0.000 description 3
- 230000000694 effects Effects 0.000 description 2
- 238000000605 extraction Methods 0.000 description 2
- 238000011160 research Methods 0.000 description 2
- 238000005070 sampling Methods 0.000 description 2
- 238000012706 support-vector machine Methods 0.000 description 2
- 230000002411 adverse Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000012937 correction Methods 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 238000002474 experimental method Methods 0.000 description 1
- 230000008140 language development Effects 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000007500 overflow downdraw method Methods 0.000 description 1
- 238000003672 processing method Methods 0.000 description 1
- 238000001228 spectrum Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/10—Terrestrial scenes
- G06V20/13—Satellite images
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/25—Fusion techniques
- G06F18/253—Fusion techniques of extracted features
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02A—TECHNOLOGIES FOR ADAPTATION TO CLIMATE CHANGE
- Y02A90/00—Technologies having an indirect contribution to adaptation to climate change
- Y02A90/10—Information and communication technologies [ICT] supporting adaptation to climate change, e.g. for weather forecasting or climate simulation
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Evolutionary Biology (AREA)
- Evolutionary Computation (AREA)
- General Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Remote Sensing (AREA)
- Astronomy & Astrophysics (AREA)
- Image Analysis (AREA)
Abstract
The invention belongs to the field of image processing, and aims to realize a remote sensing image cloud and snow detection method for carrying out multi-scale and multi-path feature fusion on a convolutional neural network-based multi-convolutional layer, which is reasonable in design and high in identification accuracy. And carrying out pixel-level accurate classification on the image. Therefore, the technical scheme adopted by the invention is that the remote sensing image cloud and snow detection method based on the convolutional neural network comprises the following steps: the coding part of the network is as follows: performing feature coding on input image information; the decoding part of the network is: and extracting the resolution information of the original image by the multi-scale fusion module of the basic depth features of the coding structure codes to generate a cloud and snow object detection result consistent with the resolution of the original image. The method is mainly applied to weather image processing occasions.
Description
Technical Field
The invention belongs to the field of computer vision. In particular to a remote sensing image cloud and snow detection method for extracting and fusing multi-scale and multi-path semantic information based on a multi-feature layer of a convolutional neural network.
Background
The remote sensing image is generally used for land monitoring, target detection, geographical mapping and the like, and the distribution of clouds and snow in the image has a large influence on the spectrum of the remote sensing image. Improving the detection accuracy of the cloud and snow in the remote sensing image has become a target for many remote sensing image applications. The cloud and snow present in the remote sensing image can have adverse effects on remote sensing applications, such as atmospheric correction, target recognition, target detection, and the like.
Therefore, the method has very important significance for improving the cloud and snow detection precision of the optical satellite image remote sensing image. Cloud and snow detection of remote sensing images in the remote sensing images is a multi-classification problem, and machine learning methods such as Artificial Neural Networks (ANNs) and Support Vector Machines (SVM) are applied to remote sensing image classification research. These methods use manually designed functions and binary classifiers without utilizing advanced functions. Today, Convolutional Neural Networks (CNNs) have become the focus of research. Convolutional Neural Networks (CNN) have been applied to classification detection, target recognition and target detection, which is a typical deep learning algorithm, and the wide application of CNN framework in the fields of speech recognition and image semantic detection has become very important. Since it is capable of accurately extracting semantic information from a large number of input images. Many deep Convolutional Neural Networks (CNNs) for cloud and snow detection are pixel-based predictions.
Disclosure of Invention
In order to overcome the defects of the prior art, the invention aims to realize a remote sensing image cloud and snow detection method for carrying out multi-scale and multi-path feature fusion on a multi-convolution layer based on a convolutional neural network, which has reasonable design and high identification accuracy. And carrying out pixel-level accurate classification on the image. Therefore, the technical scheme adopted by the invention is that the remote sensing image cloud and snow detection method based on the convolutional neural network comprises the following steps:
the coding part of the network is as follows: performing feature coding on input image information;
the decoding part of the network is: and extracting the resolution information of the original image by the multi-scale fusion module of the basic depth features of the coding structure codes to generate a cloud and snow object detection result consistent with the resolution of the original image.
And (3) taking the comprehensive cross entropy loss and the mean square error loss as loss functions, training the proposed network by using a comprehensive loss function target, and evaluating the network performance by using an accuracy rate and a mean Intersection over Union.
The base depth features include global information and local information and are obtained by fusing output features of different convolutional layers.
The encoding part of the network, namely the step of performing characteristic encoding on the input image information, specifically comprises the following steps:
(1) processing an input image to a uniform size of 256 × 256, taking a residual network ResNet-50 structure with 50 layers of neurons as a coding structure part of the proposed network, and decomposing the residual network ResNet-50 into a preprocessing unit and 4 residual blocks according to the residual blocks according to 5 stage processing structures of ResNet-50;
(2) inputting the images with uniform size into a ResNet-50 network structure, wherein after the images are subjected to a series of convolution, batch normalization, pooling and ReLU operations, each residual block outputs a feature, and the resolution of the output feature of each residual block is as follows: the residual block 1 is 32 × 32, the residual block 2 is 16 × 16, the residual block 3 is 16 × 16, and the residual block 4 is 16 × 16, for a total of 4 local residual block output characteristics.
The encoding part of the network, namely the process of fusing the features generated in the step 1 by using the multi-scale fusion module, specifically comprises the following steps:
(1) performing 3 × 3 convolution on the features output by the residual block 1 with the output feature resolution of 32 × 32, wherein the convolution step size of the convolution is 1, and obtaining a 32 × 32 feature map which is marked as a feature map 1;
(2) performing expansion convolution on the features output by the residual block 2 with the output feature resolution of 16 × 16 by 3 × 3, wherein the convolution step of the expansion convolution is 1, and the expansion rate is 6, so as to obtain a 16 × 16 feature map which is marked as a feature map 2;
(3) performing expansion convolution on the features output by the residual block 3 with the output feature resolution of 16 × 16 by 3 × 3, wherein the convolution step of the expansion convolution is 1, and the expansion rate is 12, so as to obtain a 16 × 16 feature map which is marked as a feature map 3;
(4) performing expansion convolution on the features output by the residual block 4 with the output feature resolution of 16 × 16 by 3 × 3, wherein the convolution step of the expansion convolution is 1, and the expansion rate is 18, so as to obtain a 16 × 16 feature map which is marked as a feature map 4;
(5) the features output by the residual block 4 with the output feature resolution of 16 × 16 are subjected to convolution by 3 × 3 through a global average pooling layer, the convolution step length of the convolution is 1, and a 16 × 16 feature map is obtained and recorded as a feature map 5;
(6) respectively performing 2 times of upsampling on the feature map 2, the feature map 3, the feature map 4 and the feature map 5 to generate a feature map 2a, a feature map 3a, a feature map 4a and a feature map 5 a;
(7) cascading the feature diagram 1 with a feature diagram 2a, a feature diagram 3a, a feature diagram 4a and a feature diagram 5a to obtain a feature cascading diagram A;
(8) performing convolution on the feature cascade graph A by 1 multiplied by 1, wherein the convolution step of the convolution is 1, obtaining a feature graph of 32 multiplied by 32, and marking as a feature fusion graph B;
(9) performing double upsampling on the feature fusion image B to generate a feature fusion image C with the resolution of 64 multiplied by 64;
(10) the resolution ratio output by the second residual error unit in the residual error block 1 is 64 multiplied by 64 characteristic graph, the convolution step of the convolution is 1, so as to obtain 64 multiplied by 64 characteristic graph, and the characteristic graph is cascaded with the characteristic fusion graph C with the resolution ratio of 64 multiplied by 64, so as to obtain a characteristic fusion graph D;
(11) and (3) performing 3 × 3 convolution on the feature fusion image D, wherein the convolution step of the convolution is 1, obtaining a 64 × 64 feature image, and performing 4 times of upsampling on the feature image to generate a 256 × 256 detection image.
The training step for the proposed network specifically comprises the following steps:
(1) calculating the cross entropy loss and mean square error loss of the prediction detection graph and the labeled detection graph, and updating the weight by using a back propagation algorithm;
(2) after the network training is completed, the predicted performance is measured by using the pixel accuracy and mIOU (Mean Intersection over Union).
The invention has the characteristics and beneficial effects that:
the method is reasonable in design, local information and global information are fully considered, the calculation detection accuracy of the image is improved by adopting a multi-scale multi-convolution layer feature fusion method, the provided network is trained by taking the combination of cross entropy loss and mean square error loss as a minimum target loss function, and the cloud and snow detection accuracy of the image remote sensing image is effectively improved.
Description of the drawings:
fig. 1 is an overall network structure proposed by the present invention.
Fig. 2 is an input image of the inventive test.
Fig. 3 is a cloud and snow label of an input image tested by the present invention.
FIG. 4 shows the results of the detection of the present invention.
Detailed Description
In order to reduce the influence of external factors such as uncertainty, ambiguity and the like, fully utilize information in an image and obtain better feature representation, the invention provides a remote sensing image cloud and snow detection method for extracting and fusing multi-path semantic information on multiple convolution layers and multiple scales based on a convolution neural network. According to the method, firstly, the semantic features of different levels in the ResNet-50 model are extracted by adopting the expansion convolution with different expansion rates to increase the receptive field, and the multi-scale multi-path feature fusion is carried out on the multi-convolution layer, so that the cloud and snow detection performance of the image remote sensing image is improved.
The technical scheme of the invention is realized by the following steps:
1. performing feature coding on input image information;
2. extracting basic depth features, fusing the basic depth features through a multi-scale fusion module, recovering original image resolution information, and generating a cloud and snow object detection result which is consistent with the original image resolution;
3. and (3) taking the comprehensive cross entropy loss and the mean square error loss as loss functions, training the proposed network by using the comprehensive loss function target, and evaluating the network performance by using the accuracy and the mIOU.
The relationship among the above three steps is that step 2 processes the encoded information obtained in step 1 and generates a result, the encoded information obtained in step 1 is the basis of step 2, step 1 and step 2 are the whole network structure of the present invention, and step 3 is the training method of the network of the present invention.
The specific implementation method of the step 1 comprises the following steps:
(3) the input image is processed to a uniform size of 256 x 256, a ResNet-50 network structure is used as a coding structure part of the proposed network, and the ResNet-50 network is decomposed into a pre-processing unit plus 4 residual blocks by residual blocks according to a 5-stage processing structure of ResNet-50.
(4) Inputting the images with uniform size into a ResNet-50 network structure, wherein after the images are subjected to a series of convolution, batch normalization, pooling and ReLU operations, each residual block outputs a feature, and the resolution of the output feature of each residual block is as follows: the residual block 1 is 32 × 32, the residual block 2 is 16 × 16, the residual block 3 is 16 × 16, and the residual block 4 is 16 × 16, for a total of 4 local residual block output characteristics.
The specific implementation method of the step 2 comprises the following steps:
(12) the features output by the residual block 1 with the output feature resolution of 32 × 32 are convolved by 3 × 3, the convolution step of the convolution is 1, and a feature map of 32 × 32 is obtained and recorded as the feature map 1.
(13) The features output by the residual block 2 with the output feature resolution of 16 × 16 are subjected to expansion convolution with 3 × 3, the convolution step of the expansion convolution is 1, the expansion rate is 6, and a 16 × 16 feature map is obtained and recorded as the feature map 2.
(14) The feature output by the residual block 3 with the output feature resolution of 16 × 16 is subjected to 3 × 3 dilation convolution, the convolution step of the dilation convolution is 1, the dilation rate is 12, and a 16 × 16 feature map is obtained and recorded as the feature map 3.
(15) The features output by the residual block 4 with the output feature resolution of 16 × 16 are subjected to expansion convolution of 3 × 3, the convolution step of the expansion convolution is 1, the expansion rate is 18, and a 16 × 16 feature map is obtained and recorded as the feature map 4.
(16) And (3) performing convolution on the features output by the residual block 4 with the output feature resolution of 16 × 16 by 3 × 3 through a global average pooling layer, wherein the convolution step size of the convolution is 1, and obtaining a 16 × 16 feature map which is marked as a feature map 5.
(17) The feature map 2, the feature map 3, the feature map 4, and the feature map 5 are up-sampled by 2 times to generate a feature map 2a, a feature map 3a, a feature map 4a, and a feature map 5a, respectively.
(18) And cascading the feature map 1 with the feature map 2a, the feature map 3a, the feature map 4a and the feature map 5a to obtain a feature cascade map A.
(19) And (4) performing convolution on the feature cascade diagram A by 1 × 1, wherein the convolution step of the convolution is 1, and obtaining a 32 × 32 feature diagram which is marked as a feature fusion diagram B.
(20) The feature fusion map B is up-sampled twice to generate a feature fusion map C with a resolution of 64 × 64.
(21) And (3) performing 1 × 1 convolution on the feature map with the resolution of 64 × 64 output by the second residual unit in the residual block 1, wherein the convolution step of the convolution is 1, so as to obtain a 64 × 64 feature map, and cascading the feature map and the feature fusion map C with the resolution of 64 × 64 to obtain a feature fusion map D.
(22) And (3) performing 3 × 3 convolution on the feature fusion image D, wherein the convolution step of the convolution is 1, obtaining a 64 × 64 feature image, and performing 4 times of upsampling on the feature image to generate a 256 × 256 detection image.
The specific implementation method of the step 3 comprises the following steps:
(3) and calculating the cross entropy loss and the mean square error loss of the prediction detection graph and the labeled detection graph, and updating the weight by using a back propagation algorithm.
(4) After the network training is completed, the predicted performance is measured by using the pixel accuracy and mIOU (Mean Intersection over Union).
The embodiments of the present invention will be described in detail with reference to the accompanying drawings.
The invention provides a method for detecting cloud and snow of a remote sensing image by using a multi-path characteristic multi-scale fusion module, aiming at the problem of how to fully utilize global information and local information in the cloud and snow detection of the remote sensing image. As shown in FIG. 1, in the network structure of the present invention, the output characteristics of each residual block in the ResNet-50 network structure are subjected to multi-scale expansion convolution to extract characteristics, the characteristics are subjected to size unification by a sampling layer and are cascaded, the characteristics are cascaded with low-level characteristics again after convolution fusion, and then the cascaded characteristics are subjected to convolution operation and upsampling to restore the original resolution of an image, so that the classification result is more reliable. The method is equivalent to that feature extraction of different scales is carried out on different feature layers at a feature extraction end in a network, and the receptive fields of convolution kernels of different scales are different, so that scale information contained in features obtained by each path is different, and finally a series of features from local to global are obtained. Such fusion results take into account both local information and global information. The output of the network is a detection image with the resolution consistent with the original image, the detection accuracy is calculated by using the existing label of the image, and finally the proposed network is trained by taking the minimum cross entropy loss and the mean square error loss as the targets.
In this embodiment, a remote sensing image cloud and snow detection method based on a convolutional neural network includes the following steps:
step S1, performing feature encoding on the input image information, the specific processing method of this step is as follows:
step S1.1, processing the input image to a uniform size of 256 multiplied by 256, taking a ResNet-50 network structure as a pre-trained basic convolutional neural network, and decomposing the input image into a preprocessing unit and 4 residual blocks according to the residual block according to 5 stage processing structures of ResNet-50.
Step S1.2, inputting the images with uniform sizes into a ResNet-50 network structure, wherein after the images are subjected to a series of convolution, batch normalization, pooling and ReLU operations, each residual block outputs a feature, and the resolution of the output feature of each residual block is as follows: the residual block 1 is 32 × 32, the residual block 2 is 16 × 16, the residual block 3 is 16 × 16, and the residual block 4 is 16 × 16, for a total of 4 local residual block output characteristics.
And S2, recovering the resolution information of the original image by passing the coding information obtained in the step S1 through a multi-scale fusion module, and generating a cloud and snow object detection result consistent with the resolution of the original image. The specific treatment method in the step is as follows:
s2.1, performing 3 × 3 convolution on the features output by the residual block 1 with the output feature resolution of 32 × 32, wherein the convolution step size of the convolution is 1, and obtaining a 32 × 32 feature map which is recorded as a feature map 1.
And S2.2, performing 3 × 3 expansion convolution on the features output by the residual block 2 with the output feature resolution of 16 × 16, wherein the convolution step of the expansion convolution is 1, and the expansion rate is 6, and obtaining a 16 × 16 feature map which is recorded as the feature map 2.
And S2.3, performing 3 × 3 expansion convolution on the features output by the residual block 3 with the output feature resolution of 16 × 16, wherein the convolution step of the expansion convolution is 1, the expansion rate is 12, and a 16 × 16 feature map is obtained and is marked as a feature map 3.
And S2.4, performing 3 × 3 expansion convolution on the features output by the residual block 4 with the output feature resolution of 16 × 16, wherein the convolution step of the expansion convolution is 1, the expansion rate is 18, and a 16 × 16 feature map is obtained and is marked as a feature map 4.
S2.5, the features output by the residual block 4 with the output feature resolution of 16 × 16 are subjected to convolution of 3 × 3 through a global average pooling layer, the convolution step of the convolution is 1, and a 16 × 16 feature map is obtained and recorded as a feature map 5.
And S2.7, respectively performing 2-time upsampling on the feature map 2, the feature map 3, the feature map 4 and the feature map 5 to generate a feature map 2a, a feature map 3a, a feature map 4a and a feature map 5 a.
And S2.8, cascading the feature diagram 1 with the feature diagram 2a, the feature diagram 3a, the feature diagram 4a and the feature diagram 5a to obtain a feature cascade diagram A.
S2.9, performing convolution on the feature cascade diagram A by 1 × 1, wherein the convolution step of the convolution is 1, obtaining a feature diagram of 32 × 32, and marking as a feature fusion diagram B.
S2.10, the feature fusion image B is subjected to double upsampling to generate a feature fusion image C with the resolution of 64 x 64.
S2.11, performing 1 × 1 convolution on the feature map with the resolution of 64 × 64 output by the second residual unit in the residual block 1, wherein the convolution step size of the convolution is 1, so as to obtain a 64 × 64 feature map, and cascading the feature map with the feature fusion map C with the resolution of 64 × 64 to obtain a feature fusion map D.
S2.12, performing 3 × 3 convolution on the feature fusion image D, wherein the convolution step of the convolution is 1, obtaining a 64 × 64 feature image, and performing up-sampling on the feature image by 4 times to generate a 256 × 256 detection image.
And S3, training the network provided by the steps S1 and S2 by taking the comprehensive cross entropy loss and mean square error loss function as targets, and evaluating the network performance by using the accuracy and the mIoU. The specific treatment method in the step is as follows:
and S3.1, calculating the cross entropy loss and the mean square error loss of the prediction detection map and the marked detection map, and updating the weight by using a back propagation algorithm.
And S3.2, after the network training is finished, measuring the prediction performance of the network by using the accuracy and mIOU (Mean Intersection ratio).
The following experiment was conducted according to the method of the present invention to explain the recognition effect of the present invention.
And (3) testing environment: python2.7(python language development environment); tensorflow (neural network algorithm library); ubuntu16.04 (operating system) system; NVIDIA GTX 1080Ti GPU (graphics processor developed by NVIDIA corporation)
And (3) testing sequence: the selected data set is an image data set for detecting cloud and snow images made of ZY-3 remote sensing satellite cloud and snow images, and comprises 21290 satellite remote sensing images
Testing indexes are as follows: the invention uses Pixel Accuracy (Pixel Accuracy) and mIOU as performance evaluation indexes. Pixel accuracy refers to pixel classification accuracy. mlou refers to the ratio of intersection to union of erroneous pixels that are predicted correctly on average. The index data are calculated by different algorithms which are popular at present, and then result comparison is carried out, so that the method provided by the invention obtains a better result in the field of cloud and snow detection of the image remote sensing image.
The test results were as follows:
as can be seen from the comparison data, the pixel accuracy and mIOU of the invention are obviously improved compared with the prior algorithm.
It should be emphasized that the embodiments described herein are illustrative rather than restrictive, and thus the present invention is not limited to the embodiments described in the detailed description, but also includes other embodiments that can be derived from the technical solutions of the present invention by those skilled in the art.
Claims (6)
1. A remote sensing image cloud and snow detection method based on a convolutional neural network is characterized by comprising the following steps:
the coding part of the network is as follows: performing feature coding on input image information;
the decoding part of the network is: and extracting the resolution information of the original image by the multi-scale fusion module of the basic depth features of the coding structure codes to generate a cloud and snow object detection result consistent with the resolution of the original image.
2. The remote sensing image cloud and snow detection method based on the convolutional neural network as claimed in claim 1, wherein cross entropy loss and mean square error loss are synthesized as loss functions, the proposed network is trained with a comprehensive loss function target, and network performance is evaluated by using an accuracy and a mean cross over ratio mIOU (mean cross over Unit).
3. The convolutional neural network-based remote sensing image cloud and snow detection method as claimed in claim 1, wherein the base depth features comprise global information and local information and are obtained by fusing output features of different convolutional layers.
4. The method for detecting the cloud and snow of the remote sensing image based on the convolutional neural network as claimed in claim 1, wherein a coding part of the network, namely a step of performing characteristic coding on input image information, specifically comprises the following steps:
(1) processing an input image to a uniform size of 256 × 256, taking a residual network ResNet-50 structure with 50 layers of neurons as a coding structure part of the proposed network, and decomposing the residual network ResNet-50 into a preprocessing unit and 4 residual blocks according to the residual blocks according to 5 stage processing structures of ResNet-50;
(2) inputting the images with uniform size into a ResNet-50 network structure, wherein after the images are subjected to a series of convolution, batch normalization, pooling and ReLU operations, each residual block outputs a feature, and the resolution of the output feature of each residual block is as follows: the residual block 1 is 32 × 32, the residual block 2 is 16 × 16, the residual block 3 is 16 × 16, and the residual block 4 is 16 × 16, for a total of 4 local residual block output characteristics.
5. The method for detecting the cloud and snow of the remote sensing image based on the convolutional neural network as claimed in claim 1, wherein a coding part of the network, namely a process of fusing the features generated in the step 1 by using a multi-scale fusion module, specifically comprises the following steps:
(1) performing 3 × 3 convolution on the features output by the residual block 1 with the output feature resolution of 32 × 32, wherein the convolution step size of the convolution is 1, and obtaining a 32 × 32 feature map which is marked as a feature map 1;
(2) performing expansion convolution on the features output by the residual block 2 with the output feature resolution of 16 × 16 by 3 × 3, wherein the convolution step of the expansion convolution is 1, and the expansion rate is 6, so as to obtain a 16 × 16 feature map which is marked as a feature map 2;
(3) performing expansion convolution on the features output by the residual block 3 with the output feature resolution of 16 × 16 by 3 × 3, wherein the convolution step of the expansion convolution is 1, and the expansion rate is 12, so as to obtain a 16 × 16 feature map which is marked as a feature map 3;
(4) performing expansion convolution on the features output by the residual block 4 with the output feature resolution of 16 × 16 by 3 × 3, wherein the convolution step of the expansion convolution is 1, and the expansion rate is 18, so as to obtain a 16 × 16 feature map which is marked as a feature map 4;
(5) the features output by the residual block 4 with the output feature resolution of 16 × 16 are subjected to convolution by 3 × 3 through a global average pooling layer, the convolution step length of the convolution is 1, and a 16 × 16 feature map is obtained and recorded as a feature map 5;
(6) respectively performing 2 times of upsampling on the feature map 2, the feature map 3, the feature map 4 and the feature map 5 to generate a feature map 2a, a feature map 3a, a feature map 4a and a feature map 5 a;
(7) cascading the feature diagram 1 with a feature diagram 2a, a feature diagram 3a, a feature diagram 4a and a feature diagram 5a to obtain a feature cascading diagram A;
(8) performing convolution on the feature cascade graph A by 1 multiplied by 1, wherein the convolution step of the convolution is 1, obtaining a feature graph of 32 multiplied by 32, and marking as a feature fusion graph B;
(9) performing double upsampling on the feature fusion image B to generate a feature fusion image C with the resolution of 64 multiplied by 64;
(10) the resolution ratio output by the second residual error unit in the residual error block 1 is 64 multiplied by 64 characteristic graph, the convolution step of the convolution is 1, so as to obtain 64 multiplied by 64 characteristic graph, and the characteristic graph is cascaded with the characteristic fusion graph C with the resolution ratio of 64 multiplied by 64, so as to obtain a characteristic fusion graph D;
(11) and (3) performing 3 × 3 convolution on the feature fusion image D, wherein the convolution step of the convolution is 1, obtaining a 64 × 64 feature image, and performing 4 times of upsampling on the feature image to generate a 256 × 256 detection image.
6. The remote sensing image cloud and snow detection method based on the convolutional neural network as claimed in claim 1, wherein the training step of the proposed network specifically comprises the following steps:
(1) calculating the cross entropy loss and mean square error loss of the prediction detection graph and the labeled detection graph, and updating the weight by using a back propagation algorithm;
(2) after the network training is finished, the pixel accuracy and average cross-over ratio mIOU (mean Intersection overlap Union) are used for measuring the prediction performance of the network.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911350284.6A CN111079683B (en) | 2019-12-24 | 2019-12-24 | Remote sensing image cloud and snow detection method based on convolutional neural network |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911350284.6A CN111079683B (en) | 2019-12-24 | 2019-12-24 | Remote sensing image cloud and snow detection method based on convolutional neural network |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111079683A true CN111079683A (en) | 2020-04-28 |
| CN111079683B CN111079683B (en) | 2023-12-12 |
Family
ID=70317348
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911350284.6A Active CN111079683B (en) | 2019-12-24 | 2019-12-24 | Remote sensing image cloud and snow detection method based on convolutional neural network |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111079683B (en) |
Cited By (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111553289A (en) * | 2020-04-29 | 2020-08-18 | 中国科学院空天信息创新研究院 | Remote sensing image cloud detection method and system |
| CN111611932A (en) * | 2020-05-22 | 2020-09-01 | 哈尔滨工业大学（深圳） | Remote sensing image cloud detection method based on full convolution network, terminal and storage medium |
| CN111627012A (en) * | 2020-05-28 | 2020-09-04 | 华北电力大学（保定） | Deep neural network surface defect detection method based on feature fusion |
| CN111883181A (en) * | 2020-06-30 | 2020-11-03 | 海尔优家智能科技（北京）有限公司 | Audio detection method and device, storage medium and electronic device |
| CN112418165A (en) * | 2020-12-07 | 2021-02-26 | 武汉工程大学 | Small-size target detection method and device based on improved cascade neural network |
| CN112597882A (en) * | 2020-12-22 | 2021-04-02 | 自然资源部国土卫星遥感应用中心 | Remote sensing image snow detection method based on deep convolutional neural network |
| CN112734642A (en) * | 2021-01-12 | 2021-04-30 | 武汉工程大学 | Remote sensing satellite super-resolution method and device of multi-scale texture transfer residual error network |
| CN113240589A (en) * | 2021-04-01 | 2021-08-10 | 重庆兆光科技股份有限公司 | Image defogging method and system based on multi-scale feature fusion |
| CN113936204A (en) * | 2021-11-22 | 2022-01-14 | 安徽师范大学 | High-resolution remote sensing image cloud and snow identification method and device fusing terrain data and deep neural network |
| CN114821121A (en) * | 2022-05-09 | 2022-07-29 | 盐城工学院 | Image classification method based on RGB three-component grouping attention weighted fusion |
| CN115546076A (en) * | 2022-12-05 | 2022-12-30 | 耕宇牧星(北京)空间科技有限公司 | Remote sensing image thin cloud removing method based on convolutional network |
| CN116229272A (en) * | 2023-03-14 | 2023-06-06 | 中国人民解放军陆军军事交通学院镇江校区 | High-precision remote sensing image detection method and system based on representative point representation |
| CN118212543A (en) * | 2023-12-11 | 2024-06-18 | 自然资源部国土卫星遥感应用中心 | Bilateral fusion and lightweight network improved radiation abnormal target detection method |
Citations (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2018045269A1 (en) * | 2016-09-02 | 2018-03-08 | Ohio State Innovation Foundation | System and method of otoscopy image analysis to diagnose ear pathology |
| CN108596184A (en) * | 2018-04-25 | 2018-09-28 | 清华大学深圳研究生院 | Training method, readable storage medium storing program for executing and the electronic equipment of image, semantic parted pattern |
| CN108710830A (en) * | 2018-04-20 | 2018-10-26 | 浙江工商大学 | A kind of intensive human body 3D posture estimation methods for connecting attention pyramid residual error network and equidistantly limiting of combination |
| CN108846446A (en) * | 2018-07-04 | 2018-11-20 | 国家新闻出版广电总局广播科学研究院 | The object detection method of full convolutional network is merged based on multipath dense feature |
| CN109190752A (en) * | 2018-07-27 | 2019-01-11 | 国家新闻出版广电总局广播科学研究院 | The image, semantic dividing method of global characteristics and local feature based on deep learning |
| CN109685842A (en) * | 2018-12-14 | 2019-04-26 | 电子科技大学 | A kind of thick densification method of sparse depth based on multiple dimensioned network |
| CN110110719A (en) * | 2019-03-27 | 2019-08-09 | 浙江工业大学 | A kind of object detection method based on attention layer region convolutional neural networks |
| CN110119728A (en) * | 2019-05-23 | 2019-08-13 | 哈尔滨工业大学 | Remote sensing images cloud detection method of optic based on Multiscale Fusion semantic segmentation network |
| CN110163836A (en) * | 2018-11-14 | 2019-08-23 | 宁波大学 | Based on deep learning for the excavator detection method under the inspection of high-altitude |
| CN110188720A (en) * | 2019-06-05 | 2019-08-30 | 上海云绅智能科技有限公司 | A kind of object detection method and system based on convolutional neural networks |
| CN110276316A (en) * | 2019-06-26 | 2019-09-24 | 电子科技大学 | A kind of human body critical point detection method based on deep learning |
| US20190325203A1 (en) * | 2017-01-20 | 2019-10-24 | Intel Corporation | Dynamic emotion recognition in unconstrained scenarios |
| CN110543890A (en) * | 2019-07-22 | 2019-12-06 | 杭州电子科技大学 | Deep neural network image matching method based on characteristic pyramid |
-
2019
- 2019-12-24 CN CN201911350284.6A patent/CN111079683B/en active Active
Patent Citations (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2018045269A1 (en) * | 2016-09-02 | 2018-03-08 | Ohio State Innovation Foundation | System and method of otoscopy image analysis to diagnose ear pathology |
| US20190325203A1 (en) * | 2017-01-20 | 2019-10-24 | Intel Corporation | Dynamic emotion recognition in unconstrained scenarios |
| CN108710830A (en) * | 2018-04-20 | 2018-10-26 | 浙江工商大学 | A kind of intensive human body 3D posture estimation methods for connecting attention pyramid residual error network and equidistantly limiting of combination |
| CN108596184A (en) * | 2018-04-25 | 2018-09-28 | 清华大学深圳研究生院 | Training method, readable storage medium storing program for executing and the electronic equipment of image, semantic parted pattern |
| CN108846446A (en) * | 2018-07-04 | 2018-11-20 | 国家新闻出版广电总局广播科学研究院 | The object detection method of full convolutional network is merged based on multipath dense feature |
| CN109190752A (en) * | 2018-07-27 | 2019-01-11 | 国家新闻出版广电总局广播科学研究院 | The image, semantic dividing method of global characteristics and local feature based on deep learning |
| CN110163836A (en) * | 2018-11-14 | 2019-08-23 | 宁波大学 | Based on deep learning for the excavator detection method under the inspection of high-altitude |
| CN109685842A (en) * | 2018-12-14 | 2019-04-26 | 电子科技大学 | A kind of thick densification method of sparse depth based on multiple dimensioned network |
| CN110110719A (en) * | 2019-03-27 | 2019-08-09 | 浙江工业大学 | A kind of object detection method based on attention layer region convolutional neural networks |
| CN110119728A (en) * | 2019-05-23 | 2019-08-13 | 哈尔滨工业大学 | Remote sensing images cloud detection method of optic based on Multiscale Fusion semantic segmentation network |
| CN110188720A (en) * | 2019-06-05 | 2019-08-30 | 上海云绅智能科技有限公司 | A kind of object detection method and system based on convolutional neural networks |
| CN110276316A (en) * | 2019-06-26 | 2019-09-24 | 电子科技大学 | A kind of human body critical point detection method based on deep learning |
| CN110543890A (en) * | 2019-07-22 | 2019-12-06 | 杭州电子科技大学 | Deep neural network image matching method based on characteristic pyramid |
Non-Patent Citations (1)
| Title |
|---|
| 谭明明，范影乐，武薇，佘青山，甘海涛: "多路径卷积神经网络的轮廓感知", vol. 24, no. 10, pages 1750 * |
Cited By (17)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111553289A (en) * | 2020-04-29 | 2020-08-18 | 中国科学院空天信息创新研究院 | Remote sensing image cloud detection method and system |
| CN111611932A (en) * | 2020-05-22 | 2020-09-01 | 哈尔滨工业大学（深圳） | Remote sensing image cloud detection method based on full convolution network, terminal and storage medium |
| CN111627012B (en) * | 2020-05-28 | 2021-12-21 | 华北电力大学（保定） | Deep neural network surface defect detection method based on feature fusion |
| CN111627012A (en) * | 2020-05-28 | 2020-09-04 | 华北电力大学（保定） | Deep neural network surface defect detection method based on feature fusion |
| CN111883181A (en) * | 2020-06-30 | 2020-11-03 | 海尔优家智能科技（北京）有限公司 | Audio detection method and device, storage medium and electronic device |
| CN112418165A (en) * | 2020-12-07 | 2021-02-26 | 武汉工程大学 | Small-size target detection method and device based on improved cascade neural network |
| CN112418165B (en) * | 2020-12-07 | 2023-04-07 | 武汉工程大学 | Small-size target detection method and device based on improved cascade neural network |
| CN112597882A (en) * | 2020-12-22 | 2021-04-02 | 自然资源部国土卫星遥感应用中心 | Remote sensing image snow detection method based on deep convolutional neural network |
| CN112734642A (en) * | 2021-01-12 | 2021-04-30 | 武汉工程大学 | Remote sensing satellite super-resolution method and device of multi-scale texture transfer residual error network |
| CN113240589A (en) * | 2021-04-01 | 2021-08-10 | 重庆兆光科技股份有限公司 | Image defogging method and system based on multi-scale feature fusion |
| CN113936204A (en) * | 2021-11-22 | 2022-01-14 | 安徽师范大学 | High-resolution remote sensing image cloud and snow identification method and device fusing terrain data and deep neural network |
| CN114821121A (en) * | 2022-05-09 | 2022-07-29 | 盐城工学院 | Image classification method based on RGB three-component grouping attention weighted fusion |
| CN114821121B (en) * | 2022-05-09 | 2023-02-03 | 盐城工学院 | Image classification method based on RGB three-component grouping attention weighted fusion |
| CN115546076A (en) * | 2022-12-05 | 2022-12-30 | 耕宇牧星(北京)空间科技有限公司 | Remote sensing image thin cloud removing method based on convolutional network |
| CN116229272A (en) * | 2023-03-14 | 2023-06-06 | 中国人民解放军陆军军事交通学院镇江校区 | High-precision remote sensing image detection method and system based on representative point representation |
| CN116229272B (en) * | 2023-03-14 | 2023-10-31 | 中国人民解放军陆军军事交通学院镇江校区 | High-precision remote sensing image detection method and system based on representative point representation |
| CN118212543A (en) * | 2023-12-11 | 2024-06-18 | 自然资源部国土卫星遥感应用中心 | Bilateral fusion and lightweight network improved radiation abnormal target detection method |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111079683B (en) | 2023-12-12 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111079683B (en) | Remote sensing image cloud and snow detection method based on convolutional neural network | |
| CN110188765B (en) | Image semantic segmentation model generation method, device, equipment and storage medium | |
| CN111738124B (en) | Remote sensing image cloud detection method based on Gabor transformation and attention | |
| CN110598600A (en) | Remote sensing image cloud detection method based on UNET neural network | |
| US20240257423A1 (en) | Image processing method and apparatus, and computer readable storage medium | |
| CN114998566B (en) | Interpretable multi-scale infrared dim small target detection network design method | |
| CN109871749B (en) | Pedestrian re-identification method and device based on deep hash and computer system | |
| CN113139618B (en) | Robustness-enhanced classification method and device based on integrated defense | |
| CN114140831A (en) | Human body posture estimation method and device, electronic equipment and storage medium | |
| CN115984714B (en) | Cloud detection method based on dual-branch network model | |
| CN115527072A (en) | Chip surface defect detection method based on sparse space perception and meta-learning | |
| Patel et al. | A novel approach for semantic segmentation of automatic road network extractions from remote sensing images by modified UNet | |
| CN117523645B (en) | Face key point detection method and device, electronic equipment and storage medium | |
| CN116129280B (en) | Method for detecting snow in remote sensing image | |
| Elagamy et al. | HACR-MDL: handwritten Arabic character recognition model using deep learning | |
| CN117523333A (en) | Attention mechanism-based earth surface coverage classification method | |
| CN108154107B (en) | Method for determining scene category to which remote sensing image belongs | |
| CN112597882A (en) | Remote sensing image snow detection method based on deep convolutional neural network | |
| CN113705489B (en) | Remote sensing image fine-granularity airplane identification method based on priori regional knowledge guidance | |
| CN116630610A (en) | ROI region extraction method based on semantic segmentation model and conditional random field | |
| CN117274971B (en) | Image processing method applied to water meter data extraction and electronic equipment | |
| CN116030347B (en) | High-resolution remote sensing image building extraction method based on attention network | |
| CN118155093B (en) | Noctilucent remote sensing image cloud detection method and device based on day and night data fusion | |
| CN118212543B (en) | Bilateral fusion and lightweight network improved radiation abnormal target detection method | |
| CN113705322B (en) | Handwritten Chinese character recognition method and device based on threshold graph neural network |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |