CN112633123B - Heterogeneous remote sensing image change detection method and device based on deep learning - Google Patents
Heterogeneous remote sensing image change detection method and device based on deep learning Download PDFInfo
- Publication number
- CN112633123B CN112633123B CN202011503728.8A CN202011503728A CN112633123B CN 112633123 B CN112633123 B CN 112633123B CN 202011503728 A CN202011503728 A CN 202011503728A CN 112633123 B CN112633123 B CN 112633123B
- Authority
- CN
- China
- Prior art keywords
- remote sensing
- source
- sensing image
- image
- change detection
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 230000008859 change Effects 0.000 title claims abstract description 137
- 238000001514 detection method Methods 0.000 title claims abstract description 95
- 238000013135 deep learning Methods 0.000 title claims abstract description 23
- 238000000034 method Methods 0.000 claims abstract description 60
- 238000006243 chemical reaction Methods 0.000 claims abstract description 24
- 238000012549 training Methods 0.000 claims description 66
- 230000006870 function Effects 0.000 claims description 26
- 230000008569 process Effects 0.000 claims description 19
- 238000004364 calculation method Methods 0.000 claims description 11
- 238000012937 correction Methods 0.000 claims description 10
- 125000004122 cyclic group Chemical group 0.000 claims description 8
- 238000007781 pre-processing Methods 0.000 claims description 8
- 238000011176 pooling Methods 0.000 claims description 5
- 238000012360 testing method Methods 0.000 claims description 5
- 238000005070 sampling Methods 0.000 claims description 4
- 238000012545 processing Methods 0.000 abstract description 4
- 230000000007 visual effect Effects 0.000 abstract description 2
- 230000003287 optical effect Effects 0.000 description 7
- 230000004927 fusion Effects 0.000 description 4
- 238000011160 research Methods 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 3
- 230000006872 improvement Effects 0.000 description 3
- 238000011835 investigation Methods 0.000 description 3
- 238000004458 analytical method Methods 0.000 description 2
- 230000007547 defect Effects 0.000 description 2
- 230000001419 dependent effect Effects 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000003384 imaging method Methods 0.000 description 2
- 238000012544 monitoring process Methods 0.000 description 2
- 230000011218 segmentation Effects 0.000 description 2
- 230000009466 transformation Effects 0.000 description 2
- 238000006424 Flood reaction Methods 0.000 description 1
- 238000012952 Resampling Methods 0.000 description 1
- 230000004913 activation Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 239000011159 matrix material Substances 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000010606 normalization Methods 0.000 description 1
- 230000005855 radiation Effects 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 238000001228 spectrum Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/10—Terrestrial scenes
- G06V20/13—Satellite images
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/048—Activation functions
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/20—Analysis of motion
- G06T7/246—Analysis of motion using feature-based methods, e.g. the tracking of corners or segments
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10032—Satellite or aerial image; Remote sensing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Life Sciences & Earth Sciences (AREA)
- General Engineering & Computer Science (AREA)
- Evolutionary Computation (AREA)
- Artificial Intelligence (AREA)
- Biophysics (AREA)
- Computing Systems (AREA)
- Biomedical Technology (AREA)
- Software Systems (AREA)
- Computational Linguistics (AREA)
- Health & Medical Sciences (AREA)
- Mathematical Physics (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Multimedia (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Astronomy & Astrophysics (AREA)
- Remote Sensing (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Evolutionary Biology (AREA)
- Image Analysis (AREA)
- Image Processing (AREA)
Abstract
The invention discloses a heterogeneous remote sensing image change detection method based on deep learning, which relates to the technical field of image processing and is used for solving the problems of inaccurate change detection and single image source, and the method comprises the following steps: receiving a multi-time heterogeneous remote sensing image; inputting the heterologous remote sensing image into a preset GAN network to perform image source conversion to obtain a single-source remote sensing image; and inputting the single-source remote sensing image into a preset change detection network to obtain a binary change graph. The invention also discloses a heterogeneous remote sensing image change detection device, which is used for obtaining a binary change map of the remote sensing image through converting the heterogeneous remote sensing image and a change detection network. The invention can realize the domain conversion of two remote sensing images at the same time; the method can effectively solve the problem of data difference between the heterogeneous remote sensing images, and the change areas of different time-phase images are extracted through the deep learning change detection network, so that the visual effect is improved to a certain extent.
Description
Technical Field
The invention relates to the technical field of image processing, in particular to a method and a device for detecting the change of a heterologous remote sensing image based on deep learning.
Background
When processing remote sensing images, change detection is one of the very important subjects, and the change detection plays an important role in practical applications such as disaster relief, agricultural investigation, urban planning, military monitoring and the like. Along with the progress of remote sensing image processing technology, the diversity of remote sensing data is increased by various resolutions and the appearance of various sensors, and sufficient data guarantee is provided for developing change detection. When the change is detected, the change trend is usually required to be detected, and the detection is often dependent on only one data form, so that a certain limitation exists in the aspect of response time to an emergency or time resolution.
Compared with the optical remote sensing image which only depends on one data form, the optical remote sensing image with various spatial and time resolutions can not only provide the spectrum information of the ground object, but also reflect the texture, geometric shape and other information of the ground object, thereby ensuring the possibility and accuracy of change detection. For example, two major sources of optical and SAR (synthetic aperture radar) images are often used in the investigation of remote sensing images. When the situation of extreme weather and natural disasters is faced, for example, when natural disasters such as floods and earthquakes are encountered, optical remote sensing images before and after an event are difficult to use in many cases due to severe weather conditions, and SAR has the characteristics of all weather and all weather, so that the SAR can acquire high-quality images of a target area, and the acquired images can timely and accurately reflect the conditions before and after the event of the target area through the heterogeneous remote sensing image change detection of a multisource sensor.
At present, at home and abroad, the technology research based on remote sensing image change detection mostly only adopts a single data source such as an optical remote sensing image or an SAR image, and is limited to fusion between optical images or fusion between multi-band multi-polarization SAR images when the remote sensing technology analysis research is carried out by using fusion images, so that researches related to change detection by using fusion data of different imaging modes in heterologous images are rarely available. At present, research on the aspect of heterogeneous image change detection can be mainly divided into three types:
1. Traditional methods based on differential image information, such as local joint distribution and manifold learning, multi-modal variation detection Markov model (M3 CD), supervised Homogeneous Pixel Transformation (HPT), and image regression based on an attractive matrix (AM-IR), etc. The methods utilize the similarity of similar features between heterogeneous images, but when the feature difference between the images is large or the detection area range is large, the information of the differential image is rough, so that the extracted image has low precision.
2. Methods based on traditional classification, such as post-class comparison (PC-CC), multi-temporal phase segmentation and hybrid classification (MS-CC), and the like. Although such methods can extract the change region after classification, the accuracy of the method is often dependent on the accuracy of the classifier, and when the method is faced with a multi-class classification problem, the model is often complex.
3. Deep learning-based methods, such as Symmetric Convolutional Coupled Networks (SCCN), logarithmic Transformation Feature Learning (LTFL) based methods, and condition-based countermeasure networks (cGAN), among others. These methods demonstrate the effectiveness of the deep learning method in the detection of heterogeneous remote sensing image changes. Although these methods can reduce the difference between images by unifying the heterogeneous remote sensing images into the same feature space through post-encoding conversion. However, most of these methods are based on an unsupervised idea, and finally, the threshold segmentation method is adopted to process the result, which is easy to cause erroneous judgment of the change region, so the precision is often not high.
In summary, the existing heterogeneous image change detection methods have certain defects, and an image change detection method with high degree of completion and accuracy is needed.
Disclosure of Invention
In order to overcome the defects in the prior art, one of the purposes of the present invention is to provide a method for detecting the change of a heterologous remote sensing image, which converts the heterologous remote sensing image into a single-source remote sensing image through a GAN network, and further obtains a binary change map through a change detection network.
One of the purposes of the invention is realized by adopting the following technical scheme:
A heterogeneous remote sensing image change detection method comprises the following steps:
receiving a multi-time heterogeneous remote sensing image;
Inputting the heterologous remote sensing image into a preset GAN network to perform image source conversion to obtain a single-source remote sensing image;
and inputting the single-source remote sensing image into a preset change detection network to obtain a binary change graph, wherein the preset change detection network is a deep learning network.
Further, the method for receiving the multi-time heterogeneous remote sensing image further comprises the following steps:
preprocessing the multi-time heterogeneous remote sensing image, wherein the preprocessing comprises geometric correction, atmospheric correction and geographic registration.
Further, the multi-time heterogeneous remote sensing image comprises two heterogeneous remote sensing images, and the source domain of the single-source remote sensing image is the source domain of any one of the heterogeneous remote sensing images.
Further, the single-source remote sensing image is input into a preset change detection network to obtain a binary change graph, and the method comprises the following steps:
inputting the single-source remote sensing image into a preset change detection network for prediction to obtain an output result z;
substituting the result z into Sigmond function sigma to calculate a two-class probability map of the pixel, wherein the calculation formula of Sigmond function sigma is as follows:
And taking one category with the highest probability in the two-category probability map as a final prediction result, and generating a binary change map corresponding to the final prediction result.
Further, the preset GAN network includes an encoder, two generators G, F, and two discriminators D X、DY.
Further, the preset GAN network is a GAN network that is pre-trained, and the training process of the preset GAN network includes the following steps:
Receiving two remote sensing images of an X source and a Y source;
respectively constructing data sets of the X-source remote sensing image and the Y-source remote sensing image, and dividing the data sets into a training set and a testing set;
Inputting the training set into a GAN network for training, wherein the training iteration times are preset values;
and after training is completed, taking the GAN network as the preset GAN network.
Further, the training set is input into a GAN network for training, which comprises the following steps:
learning the characteristics of the two remote sensing images of the X source and the Y source through the encoder;
inputting the two remote sensing images of the X source and the Y source into a generator G and a generator F:
The X source remote sensing image or the Y source remote sensing image and the characteristic are input into a generator G, the X source remote sensing image is converted into a Y source image domain, or the Y source remote sensing image is converted into an X source image domain, and the converted X source remote sensing image and Y source remote sensing image are called G (X) and G (Y);
Inputting the X-source remote sensing image or the Y-source remote sensing image and the characteristics into a generator F to obtain a reconstructed image F (X) or F (Y), wherein the reconstruction loss L recon meets the following conditions:
Wherein a represents an X source image domain or a Y source image domain; p data (a) represents the probability of data distribution in the a-source image domain,/> Representing the expected value of the sample set in a;
Inputting the converted image G (X) or G (Y) into the generator F, and converting the image G (X) or G (Y) into an X-source image domain or a Y-source image domain, which is called a cyclic image F (G (X)) or F (G (Y)), wherein a loss value L cyc in the conversion process satisfies:
wherein a represents an X source image domain or a Y source image domain;
And inputting the cyclic image F (G (x)) or F (G (y)) into a generator G and a generator F for iterative training until the iterative times reach the preset value.
Further, the preset change detection network is a change detection network which is trained in advance, the preset change detection network comprises four coding layers, each coding layer comprises two convolution layers and a pooling layer, and each of four decoding layers corresponding to the coding layer comprises two convolution layers and an up-sampling layer; the training process of the preset change detection network comprises the following steps of:
Receiving a plurality of double-time-phase remote sensing images, namely b 1、b2, wherein b represents an image source domain;
Inputting the double-phase remote sensing image b 1、b2 into a change detection network for training;
and when the change detection network is circularly trained until no change occurs, stopping training, and taking the change detection network as the preset change detection network.
Further, the loss function of the change detection network is a focus loss function L fl, and the calculation of the focus loss function satisfies: Wherein b' is the probability of changing the pixel, alpha is the weight coefficient, and r is the modulation coefficient.
The second objective of the present invention is to provide a device for detecting a change of a heterologous remote sensing image, which converts the heterologous remote sensing image into a single-source remote sensing image through a GAN network, and further obtains a binary change map through a change detection network.
The second purpose of the invention is realized by adopting the following technical scheme:
A heterologous remote sensing image change detection comprising:
the receiving module is used for receiving the multi-time heterogeneous remote sensing image;
the conversion module is used for inputting the heterogeneous remote sensing image into a preset GAN network to perform image source conversion to obtain a single-source remote sensing image;
And the output module is used for inputting the single-source remote sensing image into a preset change detection network to obtain a binary change graph, and the preset change detection network is a deep learning network.
Compared with the prior art, the invention has the beneficial effects that:
The invention provides a heterogeneous remote sensing image change detection method which can be used for completing the extraction of change areas of different types of remote sensing images. According to the invention, the distribution of the remote sensing heterogeneous images is learned by utilizing the generated countermeasure GAN network, so that the domain conversion of two remote sensing images can be realized simultaneously; the method can effectively solve the problem of data difference between the heterogeneous remote sensing images, and the change areas of different time-phase images are extracted through the deep learning change detection network, so that the visual effect is improved to a certain extent, and the method has important significance in the fields of urban planning, disaster monitoring, loss evaluation, natural resource investigation and the like.
Drawings
FIG. 1 is a flowchart of a method for detecting a change in a remote sensing image according to a first embodiment of the present invention;
FIG. 2 is a flow chart of a method of predicting a change detection network in accordance with a first embodiment of the present invention;
fig. 3 is a flowchart of a GAN network training procedure according to a second embodiment of the invention;
fig. 4 is a flowchart of a GAN network iterative training procedure according to a second embodiment of the invention;
FIG. 5 is a flow chart of a change detection network training process according to a second embodiment of the present invention;
Fig. 6 is a block diagram of a heterogeneous remote sensing image change detection device according to a third embodiment of the present invention.
Detailed Description
The invention will now be described in more detail with reference to the accompanying drawings, to which it should be noted that the description is given below by way of illustration only and not by way of limitation. Various embodiments may be combined with one another to form further embodiments not shown in the following description.
Example 1
An embodiment provides a method for detecting the change of a heterologous remote sensing image, which aims to detect the change of the heterologous remote sensing image in a mode of combining GAN with a change detection network, has high precision and has universality in the field of remote sensing detection.
Referring to fig. 1, a method for detecting a change of a heterologous remote sensing image includes the following steps:
s110, receiving a multi-time heterogeneous remote sensing image;
S110, the multi-time heterogeneous remote sensing image received in the step S is subjected to preprocessing, including geometric correction, atmospheric correction and geographic registration, in order to increase the accuracy of the subsequent model identification and analysis; the geometric correction is mainly used for preventing deformation generated when the expression requirements of the image and the reference system are inconsistent, the atmospheric correction is mainly used for eliminating radiation errors caused by atmospheric influences, because the S110 is used for receiving the heterogeneous remote sensing image, the remote sensing image received by the heterogeneous remote sensing image is usually transmitted by different remote sensors, and the remote sensing images of different sensors have different possible image ranges, so that geographical registration is needed to be carried out to the areas with the same size range; in addition, other preprocessing processes can be added according to actual requirements, for example, resampling the images to the same resolution, so as to prevent the problem of different resolutions caused by sensors with different characteristics.
The specific source domain in S110 is not specifically limited in this embodiment, and examples include SAR images, optical images Lidar images, and the like. Of course, according to the different receiving source domains, the data of the corresponding source domain is also required to be selected for training.
S120, inputting the heterologous remote sensing image into a preset GAN network to perform image source conversion to obtain a single-source remote sensing image;
In this embodiment, the heterologous remote sensing images refer to two heterologous remote sensing images, and when in conversion, one heterologous remote sensing image is converted into the source domain of the other heterologous remote sensing image, that is, the source domain of the single-source remote sensing image is the source domain of any one of the heterologous remote sensing images. Of course, when more than two types of heterogeneous remote sensing are needed to be detected, the change detection can be realized by adding corresponding source domain conversion training during GAN network training, and the specific training method is referred to in the second embodiment.
The preset GAN network in S120 refers to a trained generation type countermeasure network, and the GAN network includes a generator and a discriminator, which enable the generator to generate high quality pictures by introducing countermeasure training, the generator is used for generating pictures, the discriminator is used for scoring the generated pictures, and the capability of the generator and the discriminator is continuously improved during training, so that the capability of the whole model is improved.
In this embodiment, two heterologous remote sensing images are provided, and correspondingly, the preset GAN network in this embodiment includes an encoder, two generators G, F and two discriminators D X、DY.
The GAN network of the embodiment can realize domain conversion of remote sensing images of different sensors and different imaging modes.
S130, inputting the single-source remote sensing image into a preset change detection network to obtain a binary change graph, wherein the preset change detection network is a deep learning network.
It should be noted that, the single-source remote sensing image includes both the converted single-source remote sensing image and the remote sensing image which is not required to be converted in the heterogeneous remote sensing image, for example, for the heterogeneous remote sensing image AB, after the remote sensing image of the a source domain is converted into the B source domain remote sensing image through the preset GAN network, the remote sensing image input into the preset change detection network includes the converted a source domain remote sensing image and the B source domain remote sensing image which is not required to be converted.
The deep learning network in S130 may be an existing deep learning change detection network, and Unet networks are used in this embodiment.
Referring to fig. 2, S130 specifically includes the following steps:
S1301, inputting the single-source remote sensing image into a preset change detection network for prediction to obtain an output result z;
S1302, substituting the result z into Sigmond functions sigma to calculate a second class probability map of the pixel, wherein a calculation formula of the Sigmond functions sigma is as follows:
s1303, taking one category with highest probability in the two-category probability map as a final prediction result, and generating a binary change map corresponding to the final prediction result.
In actual operation, the remote sensing image change detection is performed by the heterogeneous remote sensing image change detection method described in the embodiment, the accuracy reaches more than 0.96, and the robustness is good.
Example two
The second embodiment mainly explains and describes the training process of the preset GAN network and the preset change detection network in the first embodiment.
Referring to the illustration of figure 3 of the drawings,
The training process of the preset GAN network comprises the following steps:
S210, receiving two remote sensing images of an X source and a Y source;
s220, respectively constructing data sets of the X-source remote sensing image and the Y-source remote sensing image, and dividing the data sets into a training set and a testing set;
when the data set is divided in S220, the remote sensing images can be cut at 50% overlapping degree, each image is cut into m images with the size of n multiplied by n, and n is more than or equal to 256; in this embodiment, the remote sensing image is required to be a 3-channel image or a gray scale image of color RGB.
The ratio of the training set to the test set in S220 is not particularly limited in this embodiment, and may be set according to the actual data set size or the like.
S230, inputting the training set into a GAN network for training, wherein the training iteration times are preset values;
The preset value can be set according to actual requirements, for example, the iteration number is set to 200000.
Referring to fig. 4, the training process of S230 includes the following steps:
s2301, learning the characteristics of the two remote sensing images of the X source and the Y source through the encoder;
s2302, inputting the two remote sensing images of the X source and the Y source into a generator G and a generator F:
The X source remote sensing image or the Y source remote sensing image and the characteristic are input into a generator G, the X source remote sensing image is converted into a Y source image domain, or the Y source remote sensing image is converted into an X source image domain, and the converted X source remote sensing image and Y source remote sensing image are called G (X) and G (Y);
Inputting the X-source remote sensing image or the Y-source remote sensing image and the characteristics into a generator F to obtain a reconstructed image F (X) or F (Y), wherein the reconstruction loss L recon meets the following conditions:
Wherein a represents an X source image domain or a Y source image domain; p data (a) represents the probability of data distribution in the a-source image domain,/> Representing the expected value of the sample set in a;
Inputting the converted image G (X) or G (Y) into the generator F, and converting the image G (X) or G (Y) into an X-source image domain or a Y-source image domain, which is called a cyclic image F (G (X)) or F (G (Y)), wherein a loss value L cyc in the conversion process satisfies:
wherein a represents an X source image domain or a Y source image domain;
The purpose of introducing reconstruction loss L recon in S2302 is to make F (X) or F (Y) as close as possible to the original X-source or Y-source remote sensing image; while the purpose of introducing the loss value L cyc is to achieve by reducing the loop consistency loss L cyc: f (G (X)) or F (G (Y)) is brought close to the original X-source remote sensing image or Y-source remote sensing image.
S2303, inputting the cyclic image F (G (x)) or F (G (y)) into the generator G and the generator F for iterative training until the iterative times reach the preset value.
The loss values of L recon and L cyc of the GAN network model can be gradually reduced in the process of multiple cycles, the conversion effect is improved through continuous learning and iterative training, and finally a model with good effect and stable parameters for conversion is obtained and is used as a preset GAN network model.
It should be noted that, during training, only one type of remote sensing image in the source domain needs to be selected for conversion, for example, only the training is needed to convert the X source remote sensing image into the Y source remote sensing image, and the training is not needed to be performed on the mutual conversion of the two middle source domain remote sensing images.
And S240, after training is completed, taking the GAN network as the preset GAN network.
Referring to fig. 5, the training process of the preset change detection network includes the following steps:
S310, receiving a plurality of double-phase remote sensing images, namely b 1、b2, wherein b represents an image source domain;
the remote sensing image of S310 is a remote sensing image output after GAN network training is completed, for example, G (x) and Y source remote sensing images.
It should be noted that the change detection network in this embodiment is an improvement on the deep learning network Unet network. The principle is that a series of features are extracted by performing operations such as convolution, normalization, activation and the like on b 1、b2 after dimension stitching, and the features are downsampled by using a maximum pooling operation so as to extract the features on different resolutions and different layers. And when up-sampling is carried out through deconvolution and the original resolution is restored, the image is encoded by connecting the low-layer and high-layer features in a jump mode, so that the global feeling is enhanced.
S320, inputting the double-phase remote sensing image b 1、b2 into a change detection network for training;
S330, when the change detection network is circularly trained until no improvement is achieved, training is stopped, and the change detection network is used as the preset change detection network.
The loss function during training is a focus loss function L fl, and the calculation of the focus loss function satisfies the following conditions:
Wherein b' is the probability of changing the pixel, alpha is the weight coefficient, and r is the modulation coefficient.
Since the change area in the change detection generally occupies a relatively small area, but most of the change area is often a non-change area, and the problem of unbalanced category exists, in the design of the loss function in this embodiment, the focal point loss function is used, and the influence caused by unbalanced category is eliminated by improving the weight coefficient and the modulation coefficient of the change category which occupies a relatively small area.
The preset change detection network in this embodiment includes four coding layers, each coding layer includes two convolution layers and a pooling layer, and each of four decoding layers corresponding to the coding layer includes two convolution layers and an upsampling layer.
The last three layers of the decoding layer can output binary change prediction graphs with different scales, and the loss function during downsampling in this embodiment is realized through three different focal point loss functions L fl1、Lfl2、Lfl3. Finally, the total Loss is calculated using the weighted Loss assigning weights w1, w2 and w 3.
The formula is as follows:
Loss=w1×l fl1+w2*Lfl2+w3*Lfl3, where w1=0.2, w2=0.3, w3=0.5;
of course, the calculation method of the Loss is not limited to the calculation method described in the present embodiment.
Preferably, when training the change detection network, a dynamic learning rate strategy can be adopted, the basic learning rate is set to be 0.01, and when the model is not improved for more than 5 times in the circulation process, the learning rate can be reduced in an exponential manner, so that the learning range of the model is enlarged, and the model is searched towards different directions. When the model is not lifted for more than 100 cycles, or when the learning rate is less than 1e-7, the cycle will stop, i.e. the training is stopped in S330.
Example III
An embodiment III discloses a device corresponding to the method for detecting a change in a heterologous remote sensing image in the above embodiment, which is a virtual device structure in the above embodiment, as shown in FIG. 6, and includes:
A receiving module 410, configured to receive a multi-temporal heterogeneous remote sensing image;
the conversion module 420 is configured to input the heterologous remote sensing image into a preset GAN network to perform image source conversion, so as to obtain a single-source remote sensing image;
The output module 430 is configured to input the single-source remote sensing image into a preset change detection network to obtain a binary change map.
Preferably, the method for receiving the multi-time heterogeneous remote sensing image comprises the following steps:
preprocessing the multi-time heterogeneous remote sensing image, wherein the preprocessing comprises geometric correction, atmospheric correction and geographic registration.
The multi-time heterogeneous remote sensing image comprises two heterogeneous remote sensing images, and the source domain of the single-source remote sensing image is the source domain of any one of the heterogeneous remote sensing images.
Preferably, the single-source remote sensing image is input into a preset change detection network to obtain a binary change graph, which comprises the following steps:
inputting the single-source remote sensing image into a preset change detection network for prediction to obtain an output result z;
substituting the result z into Sigmond function sigma to calculate a two-class probability map of the pixel, wherein the calculation formula of Sigmond function sigma is as follows:
And taking one category with the highest probability in the two-category probability map as a final prediction result, and generating a binary change map corresponding to the final prediction result.
Preferably, the preset GAN network includes an encoder, two generators G, F, and two discriminants D X、DY.
Preferably, the preset GAN network is a GAN network which is trained in advance, and the training process of the preset GAN network includes the following steps:
Receiving two remote sensing images of an X source and a Y source;
respectively constructing data sets of the X-source remote sensing image and the Y-source remote sensing image, and dividing the data sets into a training set and a testing set;
Inputting the training set into a GAN network for training, wherein the training iteration times are preset values;
and after training is completed, taking the GAN network as the preset GAN network.
Inputting the training set into a GAN network for training, comprising the following steps:
learning the characteristics of the two remote sensing images of the X source and the Y source through the encoder;
inputting the two remote sensing images of the X source and the Y source into a generator G and a generator F:
The X source remote sensing image or the Y source remote sensing image and the characteristic are input into a generator G, the X source remote sensing image is converted into a Y source image domain, or the Y source remote sensing image is converted into an X source image domain, and the converted X source remote sensing image and Y source remote sensing image are called G (X) and G (Y);
Inputting the X-source remote sensing image or the Y-source remote sensing image and the characteristics into a generator F to obtain a reconstructed image F (X) or F (Y), wherein the reconstruction loss L recon meets the following conditions:
Wherein a represents an X source image domain or a Y source image domain; p data (a) represents the probability of data distribution in the a-source image domain,/> Representing the expected value of the sample set in a;
Inputting the converted image G (X) or G (Y) into the generator F, and converting the image G (X) or G (Y) into an X-source image domain or a Y-source image domain, which is called a cyclic image F (G (X)) or F (G (Y)), wherein a loss value L cyc in the conversion process satisfies:
wherein a represents an X source image domain or a Y source image domain;
And inputting the cyclic image F (G (x)) or F (G (y)) into a generator G and a generator F for iterative training until the iterative times reach the preset value.
Preferably, the preset change detection network is a pre-trained change detection network, the preset change detection network comprises four coding layers, each coding layer comprises two convolution layers and a pooling layer, and each of four decoding layers corresponding to the coding layer comprises two convolution layers and an up-sampling layer; the training process of the preset change detection network comprises the following steps of:
Receiving a plurality of double-time-phase remote sensing images, namely b 1、b2, wherein b represents an image source domain;
Inputting the double-phase remote sensing image b 1、b2 into a change detection network for training;
and when the change detection network is circularly trained until no improvement is performed, stopping training, and taking the change detection network as the preset change detection network.
The loss function of the change detection network is a focus loss function L fl, and the calculation of the focus loss function satisfies the following conditions: Wherein b' is the probability of changing the pixel, alpha is the weight coefficient, and r is the modulation coefficient.
It should be noted that, in the embodiment of the foregoing apparatus for detecting a change based on a heterologous remote sensing image, each unit and module included are only divided according to functional logic, but not limited to the above division, so long as the corresponding function can be implemented; in addition, the specific names of the functional units are also only for distinguishing from each other, and are not used to limit the protection scope of the present invention.
It will be apparent to those skilled in the art from this disclosure that various other changes and modifications can be made which are within the scope of the invention as defined in the appended claims.
Claims (9)
1. The heterogeneous remote sensing image change detection method based on deep learning is characterized by comprising the following steps of:
receiving a multi-time heterogeneous remote sensing image;
Inputting the heterologous remote sensing image into a preset GAN network to perform image source conversion to obtain a single-source remote sensing image;
Inputting the single-source remote sensing image into a preset change detection network to obtain a binary change graph, wherein the preset change detection network is a deep learning network; the method comprises the following steps of:
inputting the single-source remote sensing image into a preset change detection network for prediction to obtain an output result z;
Substituting the result z into a Sigmoid function Wherein the Sigmoid function/>, is a two-class probability map of pixelsThe calculation formula of (2) is as follows: /(I)；
And taking one category with the highest probability in the two-category probability map as a final prediction result, and generating a binary change map corresponding to the final prediction result.
2. The method for detecting the change of the heterogeneous remote sensing image based on the deep learning according to claim 1, wherein the method for receiving the multi-time heterogeneous remote sensing image further comprises the following steps:
preprocessing the multi-time heterogeneous remote sensing image, wherein the preprocessing comprises geometric correction, atmospheric correction and geographic registration.
3. The method for detecting the change of the heterogeneous remote sensing image based on the deep learning according to claim 1, wherein the multi-temporal heterogeneous remote sensing image comprises two heterogeneous remote sensing images, and the source domain of the single-source remote sensing image is the source domain of any one of the heterogeneous remote sensing images.
4. The method for detecting a change in a heterogeneous remote sensing image based on deep learning as claimed in claim 1 or 3, wherein the predetermined GAN network comprises an encoder, two generators G, F and two discriminators D X、DY.
5. The method for detecting a change in a heterogeneous remote sensing image based on deep learning as claimed in claim 4, wherein the preset GAN network is a GAN network which is trained in advance, and the training process of the preset GAN network comprises the following steps:
Receiving two remote sensing images of an X source and a Y source;
respectively constructing data sets of the X-source remote sensing image and the Y-source remote sensing image, and dividing the data sets into a training set and a testing set;
Inputting the training set into a GAN network for training, wherein the training iteration times are preset values;
and after training is completed, taking the GAN network as the preset GAN network.
6. The method for detecting the change of the heterogeneous remote sensing image based on the deep learning according to claim 5, wherein the training set is input into a GAN network for training, comprising the following steps:
learning the characteristics of the two remote sensing images of the X source and the Y source through the encoder;
inputting the two remote sensing images of the X source and the Y source into a generator G and a generator F:
The X source remote sensing image or the Y source remote sensing image and the characteristic are input into a generator G, the X source remote sensing image is converted into a Y source image domain, or the Y source remote sensing image is converted into an X source image domain, and the converted X source remote sensing image and Y source remote sensing image are called G (X) and G (Y);
Inputting the X-source remote sensing image or the Y-source remote sensing image and the characteristics into a generator F to obtain a reconstructed image F (X) or F (Y), wherein the reconstruction loss The method meets the following conditions:
wherein a represents an X source image domain or a Y source image domain,/> Representing the probability of data distribution in the a-source image domain,/>Representing the expected value of the sample set in a;
inputting the converted image G (X) or G (Y) into the generator F, and converting the G (X) or G (Y) into an X source image domain or a Y source image domain, called a cyclic image F (X) or F (G (Y)), wherein the loss value in the conversion process The method meets the following conditions:
wherein a represents an X source image domain or a Y source image domain;
and inputting the cyclic image F (G (x)) or F (G (y)) into a generator G and a generator F for iterative training until the iterative times reach the preset value.
7. The depth learning-based heterogeneous remote sensing image change detection method according to claim 3, wherein the preset change detection network is a pre-trained change detection network, the preset change detection network comprises four coding layers, each coding layer comprises two convolution layers and a pooling layer, and each of four decoding layers corresponding to the coding layer comprises two convolution layers and an up-sampling layer; the training process of the preset change detection network comprises the following steps of:
Receiving a plurality of double-time-phase remote sensing images, namely b 1、b2, wherein b represents an image source domain;
Inputting the double-phase remote sensing image b 1、b2 into a change detection network for training;
and when the change detection network is circularly trained until no change occurs, stopping training, and taking the change detection network as the preset change detection network.
8. The method for detecting the change of the heterogeneous remote sensing image based on the deep learning according to claim 7, wherein the loss function of the change detection network is a focal point loss functionThe calculation of the focus loss function satisfies the following conditions: Wherein/> To change the probability of the pixel,/>As the weight coefficient of the light-emitting diode,Is the modulation factor.
9. Heterogeneous remote sensing image change detection device based on deep learning, its characterized in that includes:
the receiving module is used for receiving the multi-time heterogeneous remote sensing image;
the conversion module is used for inputting the heterogeneous remote sensing image into a preset GAN network to perform image source conversion to obtain a single-source remote sensing image;
the output module is used for inputting the single-source remote sensing image into a preset change detection network to obtain a binary change graph; the method comprises the following steps of:
inputting the single-source remote sensing image into a preset change detection network for prediction to obtain an output result z;
Substituting the result z into a Sigmoid function Wherein the Sigmoid function/>, is a two-class probability map of pixelsThe calculation formula of (2) is as follows: /(I)；
And taking one category with the highest probability in the two-category probability map as a final prediction result, and generating a binary change map corresponding to the final prediction result.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011503728.8A CN112633123B (en) | 2020-12-18 | 2020-12-18 | Heterogeneous remote sensing image change detection method and device based on deep learning |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011503728.8A CN112633123B (en) | 2020-12-18 | 2020-12-18 | Heterogeneous remote sensing image change detection method and device based on deep learning |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112633123A CN112633123A (en) | 2021-04-09 |
| CN112633123B true CN112633123B (en) | 2024-06-04 |
Family
ID=75317008
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011503728.8A Active CN112633123B (en) | 2020-12-18 | 2020-12-18 | Heterogeneous remote sensing image change detection method and device based on deep learning |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112633123B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113689352A (en) * | 2021-08-25 | 2021-11-23 | 闫殿武 | Remote sensing image product and manufacturing method thereof |
| CN114170145B (en) * | 2021-11-12 | 2023-04-07 | 西安理工大学 | Heterogeneous remote sensing image change detection method based on multi-scale self-coding |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103020939A (en) * | 2012-12-18 | 2013-04-03 | 武汉大学 | Method for removing large-area thick clouds for optical remote sensing images through multi-temporal data |
| CN109903299A (en) * | 2019-04-02 | 2019-06-18 | 中国矿业大学 | A kind of conditional generates the heterologous remote sensing image registration method and device of confrontation network |
| CN110599401A (en) * | 2019-08-19 | 2019-12-20 | 中国科学院电子学研究所 | Remote sensing image super-resolution reconstruction method, processing device and readable storage medium |
| WO2020150264A1 (en) * | 2019-01-15 | 2020-07-23 | Portland State University | Feature pyramid warping for video frame interpolation |
| CN111898507A (en) * | 2020-07-22 | 2020-11-06 | 武汉大学 | Deep learning method for predicting earth surface coverage category of label-free remote sensing image |
| CN111899172A (en) * | 2020-07-16 | 2020-11-06 | 武汉大学 | Vehicle target detection method oriented to remote sensing application scene |
| CN112016436A (en) * | 2020-08-28 | 2020-12-01 | 北京国遥新天地信息技术有限公司 | Remote sensing image change detection method based on deep learning |
-
2020
- 2020-12-18 CN CN202011503728.8A patent/CN112633123B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103020939A (en) * | 2012-12-18 | 2013-04-03 | 武汉大学 | Method for removing large-area thick clouds for optical remote sensing images through multi-temporal data |
| WO2020150264A1 (en) * | 2019-01-15 | 2020-07-23 | Portland State University | Feature pyramid warping for video frame interpolation |
| CN109903299A (en) * | 2019-04-02 | 2019-06-18 | 中国矿业大学 | A kind of conditional generates the heterologous remote sensing image registration method and device of confrontation network |
| CN110599401A (en) * | 2019-08-19 | 2019-12-20 | 中国科学院电子学研究所 | Remote sensing image super-resolution reconstruction method, processing device and readable storage medium |
| CN111899172A (en) * | 2020-07-16 | 2020-11-06 | 武汉大学 | Vehicle target detection method oriented to remote sensing application scene |
| CN111898507A (en) * | 2020-07-22 | 2020-11-06 | 武汉大学 | Deep learning method for predicting earth surface coverage category of label-free remote sensing image |
| CN112016436A (en) * | 2020-08-28 | 2020-12-01 | 北京国遥新天地信息技术有限公司 | Remote sensing image change detection method based on deep learning |
Non-Patent Citations (3)
| Title |
|---|
| A deep translation(GAN) based on change detection network for optical and SAR remote sensing images;Li xinghua等;《ISPRS Journal of Photo-grammetry and Remote Sensing》;179:14-34 * |
| Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks;Jun-Yan Zhu等;《IEEE International Conference on Computer Vision》;1-20 * |
| 基于生成对抗网络的遥感图像变化检测;牛旭东;《中国优秀硕士学位论文全文数据库》工程科技II辑;摘要、第1.2节、第1.3节、第2.1节、第3节、第4.4.2节 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112633123A (en) | 2021-04-09 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111524135B (en) | Method and system for detecting defects of tiny hardware fittings of power transmission line based on image enhancement | |
| CN113780296B (en) | Remote sensing image semantic segmentation method and system based on multi-scale information fusion | |
| CN110728658A (en) | High-resolution remote sensing image weak target detection method based on deep learning | |
| CN110555841B (en) | SAR image change detection method based on self-attention image fusion and DEC | |
| CN113901900A (en) | Unsupervised change detection method and system for homologous or heterologous remote sensing image | |
| CN114972989B (en) | Single remote sensing image height information estimation method based on deep learning algorithm | |
| CN114742799B (en) | Industrial scene unknown type defect segmentation method based on self-supervision heterogeneous network | |
| CN114283120B (en) | Domain-adaptive-based end-to-end multisource heterogeneous remote sensing image change detection method | |
| CN112633123B (en) | Heterogeneous remote sensing image change detection method and device based on deep learning | |
| CN112950780A (en) | Intelligent network map generation method and system based on remote sensing image | |
| CN117058367A (en) | Semantic segmentation method and device for high-resolution remote sensing image building | |
| CN117173412A (en) | Medical image segmentation method based on CNN and Transformer fusion network | |
| CN113255607A (en) | Remote sensing house detection method and device based on multi-model fusion | |
| CN115376019A (en) | Object level change detection method for heterogeneous remote sensing image | |
| CN115984714B (en) | Cloud detection method based on dual-branch network model | |
| CN117292117A (en) | Small target detection method based on attention mechanism | |
| CN114724023A (en) | Twin network-based water body change detection method | |
| CN114663749A (en) | Training method and device for landslide mass recognition model, electronic equipment and storage medium | |
| CN114612315A (en) | High-resolution image missing region reconstruction method based on multi-task learning | |
| Drees et al. | Multi-modal deep learning with sentinel-3 observations for the detection of oceanic internal waves | |
| CN117576567B (en) | Remote sensing image change detection method using multi-level difference characteristic self-adaptive fusion | |
| CN118365879A (en) | Heterogeneous remote sensing image segmentation method based on scene perception attention | |
| CN116051984B (en) | Weak and small target detection method based on Transformer | |
| CN117292261A (en) | Heterogeneous remote sensing image change detection method | |
| CN114764880B (en) | Multi-component GAN reconstructed remote sensing image scene classification method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |