CN110728706B - SAR image fine registration method based on deep learning - Google Patents

Abstract

The invention discloses a deep learning-based precise registration method for SAR images, which mainly solves the problems that traditional methods cannot correct local distortion and are time-consuming. The implementation scheme is: 1) acquiring a training data set; 2) constructing a neural network for fine registration of SAR images; 3) constructing a loss function of a neural network model for fine registration of SAR images; 4) using training data 5) Input the SAR image to be registered and the SAR image used as a reference into the trained network model to obtain a well-registered SAR image. SAR image. The invention can correct the overall deformation and local distortion between SAR images, improve the registration performance, speed up the registration speed, and can be used for SAR image fusion and change detection.

Description

SAR image fine registration method based on deep learning

technical field

The invention belongs to the technical field of image processing, in particular to a method for fine registration of SAR images, which can be used for SAR image fusion and change detection.

Background technique

Synthetic Aperture Radar SAR is an active microwave imaging system that has the ability to observe all-weather ground and sea surfaces under different climate and light conditions. It has played an important role in many applications such as geological resource exploration, ocean monitoring and urban planning. . With the continuous development of SAR imaging technology, the SAR imaging system has acquired a large amount of valuable earth observation data. In SAR image processing, it is often necessary to analyze and process two or more SAR images, such as SAR image fusion and SAR image change detection, and SAR image registration technology is the premise of these image processing tasks.

At present, the registration methods of SAR images can be divided into two categories. The first category is the region-based method, which establishes the matching relationship between images based on similarity measures such as cross-correlation or mutual information to realize the registration of SAR images. However, such methods are very time-consuming and the registration accuracy is not high. . The second type of method is the feature-based method, which establishes a matching relationship to achieve SAR image registration by comparing the distances between local invariant feature descriptors and setting a threshold for screening. Although this kind of method is faster and has higher registration accuracy than the first kind of method, it still has the problem of time-consuming.

Dellinger proposed a typical feature-based SAR image in his paper "SAR-SIFT: A SIFT-like Algorithm for SARImages" "IEEE Transaction on Geoscience&Remote Sensing", 2015, 53(1):453-466) Registration method, this method first constructs the SAR-Harris scale space, then searches for extreme points in the scale space, and filters the extreme points to find stable feature points, and then generates a scale-invariant synthetic aperture radar based on the feature points. The feature transforms the SAR-SIFT feature descriptor, and finally uses the nearest neighbor method to delete the wrong matching point pairs. The disadvantage of this method is that in the process of establishing the scale space and obtaining the feature descriptor, the amount of calculation is large, and the time-consuming is relatively serious.

The traditional SAR image registration is to perform consistent correction on the whole image by estimating the geometric transformation parameters between the images. However, in addition to the overall deformation between the actual two SAR images, there are also local distortions due to the difference in the observation angle. The existence of these local distortions is not conducive to the subsequent SAR image processing, but the traditional two SAR image registration cannot Correcting these local distortions affects the registration performance and the improvement of registration speed.

SUMMARY OF THE INVENTION

The purpose of the present invention is to propose a deep learning-based fine registration method for SAR images to correct the overall deformation and local distortion between SAR images, improve the registration performance, and speed up the registration speed.

To achieve the above object, the implementation steps of the technical solution of the present invention include the following:

(1) Acquire multiple SAR images with overlapping areas obtained by observing the same scene from different perspectives, and obtain a data set Φ;

(2) Construct a neural network for fine registration of SAR images:

(2a) Constructing a sub-convolutional neural network for correcting the overall deformation between images. The sub-convolutional neural network consists of eight convolutional layers and a global average pooling layer in turn, in which the activation functions of the first seven convolutional layers adopt The ReLU function, the output of the eighth convolutional layer has three feature maps, the activation function is a linear function, and the global average pooling layer averages all the values in each feature map to obtain the horizontal translation corresponding to the geometric deformation between images, respectively. The three output values of vertical translation and rotation angle are globally corrected by quadratic linear interpolation method according to these three output values;

(2b) Constructing a sub-residual neural network for correcting local deformation between SAR images. The sub-residual neural network consists of eight convolutional layers and four deconvolutional layers whose activation function is the ReLU function in turn. The first three The activation function after the first deconvolution layer is the same as the ReLU function, and the activation function after the fourth deconvolution layer is the Tanh function; according to the sub-residual neural network, a feature map and the offset of the corresponding pixel value of the reference image are output The quantity u is used to correct the local distortion of the image corrected by the sub-convolutional neural network by using the quadratic linear interpolation method;

(2c) connecting the two sub-neural networks constructed in steps (2a) and (2b) in turn to form a fine registration neural network for SAR images;

(2d) Construct the loss function Loss of the neural network for fine registration of SAR images:

(2d1) Construct the loss function of the sub-convolutional neural network: loss _glo =||xy|| ₁ +(1-SSIM(x,y))

(2d2) Construct the loss function of the sub-residual neural network:

(2d3) Add the loss functions constructed by (2d1) and (2d2) to obtain the loss function of the neural network for fine registration of SAR images: Loss=loss _glo +loss _loc ,

和

分别表示沿水平和垂直方向的局部微分；In the formula, x is the corrected image obtained by the sub-convolutional neural network, y is the reference image, x ₁ is the corrected image obtained by the residual neural network, ||·|| ₁ represents the 1-norm, and SSIM(·) is the structure similarity function,

and

represent the local differential along the horizontal and vertical directions, respectively;

(3) Use the dataset Φ obtained in (1) to train the neural network constructed in (2) to obtain a registered image:

(3a) Set the learning rate parameter to 0.0001;

(3b) Randomly extract two SAR images from the dataset Φ, one as the image to be registered and one as the reference image, and input the two images into the neural network for fine registration of SAR images;

(3c) Using the back-propagation algorithm to update the weights of the neural network for fine registration of the SAR image constructed in (2) to reduce the value of the Loss function;

(3d) Repeat steps 3b) and 3c) until the loss function Loss of the neural network for fine registration of SAR images converges, and the trained neural network model for fine registration of SAR images is obtained;

(3e) Input the reference image and the image to be registered into the trained neural network model together to obtain the registered SAR image.

Compared with the prior art, the present invention has the following advantages:

1. The present invention adopts a deep learning method to construct a network model and a loss function, intelligently realizes the fine registration of SAR images, can correct the overall deformation and local distortion between SAR images, and improves the registration performance.

2. Since the present invention designs an end-to-end neural network model suitable for SAR image registration, the SAR image to be registered and the SAR image used as a reference are input into the neural network model, and the output can be well-registered. The SAR image can avoid the traditional complex feature descriptor extraction process, simplify the operation process and speed up the registration speed.

Description of drawings

Fig. 1 is the realization flow chart of the present invention;

Fig. 2 is a simulation result diagram of SAR image registration using the present invention.

Detailed ways

The embodiments and effects of the present invention will be further described below with reference to the accompanying drawings.

1, the implementation steps of the present invention are as follows:

Step 1: Obtain the data sets for network training and testing.

其中

表示获取的第i张大小为m×n的图片，N表示图片总数。Select a specific scene, transmit signal pulses to the scene, and the signal will enter the radar receiver after reflection. According to the SAR imaging technology, multiple SAR images of the same scene observed from different perspectives are obtained, and these images form a dataset:

in

Indicates the acquired i-th image of size m×n, where N represents the total number of images.

The second step is to construct a neural network model for fine registration of SAR images.

The network consists of a sub-convolutional neural network for correcting the overall deformation between SAR images and a sub-residual neural network for correcting the local deformation between SAR images.

2a) Construct a sub-convolutional neural network that corrects the overall deformation between SAR images. The network consists of eight convolutional layers and a global average pooling layer in turn. The number of output feature maps of the first seven convolutional layers is 32, 64, 128, 256, 256, 256, 256, the convolution kernel sizes are 7×7, 5×5, 3×3, 3×3, 3×3, 3×3, 3×3, and the steps are 2×2, the activation function is the ReLU function, the number of output feature maps of the 8th convolutional layer is 3, the size of the convolution kernel is 3×3, the step size is 2×2, and the activation function is a linear function; global The average pooling layer averages all the values in each feature map, and obtains three output values, which correspond to the horizontal translation, vertical translation and rotation angle of the geometric deformation between the images. Quadratic linear interpolation is used according to these three output values. method for global correction of images to be registered;

2b) Construct a sub-residual neural network for correcting local deformation between SAR images. The network consists of eight convolutional layers and four deconvolutional layers in turn. The number of output feature maps of the first four convolutional layers is 32 and 64, respectively. , 128, 256, the size of the convolution kernel is 7×7, 5×5, 3×3, 3×3, the step size is 2×2, the activation function is the ReLU function, and the next four layers of convolution layer output The number of feature maps is 256, the size of the convolution kernel is 3 × 3, the stride is 1 × 1, the activation function is the ReLU function, and the number of output feature maps of the next three deconvolution layers is 256, 128, 64, the convolution kernel size is 3 × 3, 3 × 3, 5 × 5, the stride is 2 × 2, the activation function is the ReLU function, and the number of output feature maps of the last deconvolution layer is 1. The size of the convolution kernel is 7×7, the stride is 2×2, and the activation function is the Tanh function. The output result of the network is abbreviated as u, which represents the offset of the corresponding pixel values between the two images. The image after network correction is subjected to local distortion correction.

Step 3, construct the loss function Loss of the neural network used for fine registration of SAR images;

3a) Construct a sub-convolutional neural network loss function loss _glo for correcting the overall deformation between SAR images:

(3a1) Input the image to be registered and the reference image y into the sub-convolutional neural network to obtain the first corrected image x, and use the 1-norm as the pixel similarity according to the pixel similarity between the first corrected image x and the reference image y The sex measurement standard constructs the ||xy|| ₁ function, the larger the function value, the worse the pixel similarity;

(3a2) According to the structural similarity between the first corrected image x and the reference image y, select the structural similarity function SSIM( ) as the structural similarity measure to construct a (1-SSIM(x,y)) function, the larger the function value is , the structural similarity is worse;

(3a3) Add the functions constructed in (3a1) and (3a2) to get the loss function:

loss _glo =||xy|| ₁ +(1-SSIM(x,y))

where ||·|| ₁ represents the 1-norm, and SSIM(·) is the structural similarity function;

3b) Construct the loss function loss _loc of the sub-residual neural network for correcting the local deformation between SAR images:

(3b1) Input the first corrected image x into the sub-residual neural network to obtain the second corrected image x ₁ , and use the 1-norm as the pixel similarity according to the pixel similarity between the second corrected image x ₁ and the reference image y Metric construction ||x ₁ -y|| ₁ , the larger the function value, the worse the pixel similarity;

(3b2) According to the structural similarity between the second corrected image x ₁ and the reference image y, select the structural similarity function SSIM( ) as the structural similarity measure to construct a (1-SSIM(x ₁ , y)) function, the function value The larger the value, the worse the structural similarity;

作为空间平滑性的衡量标准，函数值越大，则空间平滑度越差；(3b3) According to the offset of the residual neural network output, it needs to have spatial smoothness, and the spatial smoothness is reflected in the horizontal gradient and the vertical gradient, so the constructor function

As a measure of spatial smoothness, the larger the function value, the worse the spatial smoothness;

(3b4) Add the functions constructed in (3b1), (3b2) and (3b3) to get the loss function:

和

分别表示沿水平和沿垂直方向的偏导数。in,

and

are the partial derivatives along the horizontal and vertical directions, respectively.

3c) Combine the sub-convolutional neural network loss function loss _glo in 3a) for correcting the overall deformation between SAR images with the loss function loss of the sub-residual neural network in 3b) for correcting the overall deformation between SAR images _loc is added to obtain the loss function Loss of the neural network used for fine registration of SAR images:

Loss=loss _glo + loss _loc .

Step 4: Using the data set Φ in (1), train the neural network constructed in (2) to obtain a trained network model.

4a) Randomly extract two SAR images from the dataset Φ, one as the image to be registered and one as the reference image, and input the two images into the neural network for fine registration of SAR images;

4b) Using the back propagation algorithm to update the weights of the neural network for fine registration of SAR images;

4c) Set the learning rate to 0.0001, and repeat steps 4a) and 4b) until the loss function Loss of the neural network for fine registration of SAR images converges, and the trained neural network model for fine registration of SAR images is obtained;

Step 5: Input the SAR image to be registered and the SAR image used as a reference into the trained network model to obtain a registered SAR image.

The effect of the present invention can be further illustrated by the following simulation:

1. Simulation conditions:

From the dataset Φ, randomly extract two images such as 2(a) and 2(b), and use 2(a) as the reference SAR image and 2(b) as the SAR image to be registered;

2. Simulation process:

In the first step, 2(a) and 2(b) are registered using the traditional SAR-SIFT registration method, and the registration result is shown in Fig. 2(c).

In the second step, the neural network of the present invention is used for image registration, and the registration result is shown in Figure 2(d).

In the third step, the corresponding pixel values in Figure 2(a) and Figure 2(c) are subtracted and the absolute value is obtained, and the result is shown in Figure 2(e).

The fourth step is to subtract the corresponding pixel values of Fig. 2(a) and Fig. 2(d) and take the absolute value, and the result is shown in Fig. 2(f).

It can be seen from the two images in Figure 2(e) and Figure 2(f) that the deep learning-based fine registration method for SAR images of the present invention can correct the overall deformation and local distortion between SAR images and improve registration performance.

The above description is only a specific example of the present invention, and does not constitute any limitation to the present invention. Obviously, for those skilled in the art, after understanding the content and principles of the present invention, they may not deviate from the principles and structures of the present invention. Under certain circumstances, various modifications and changes in form and details are made, but these modifications and changes based on the idea of the present invention are still within the protection scope of the claims of the present invention.

Claims (5)

1. A SAR image fine registration method based on deep learning is characterized by comprising the following steps:

(1) acquiring a plurality of SAR images with overlapped regions obtained by observing the same scene from different visual angles to obtain a data set phi;

(2) constructing a neural network for SAR image fine registration:

(2a) constructing a sub-convolution neural network for correcting overall deformation among images, wherein the sub-convolution neural network sequentially comprises eight convolution layers and a global average pooling layer, the first seven convolution layer activation functions adopt ReLU functions, the eighth layer convolution layer outputs 3 characteristic graphs, the activation functions are linear functions, the global average pooling layer averages all values in each characteristic graph to obtain three output values of horizontal translation, vertical translation and rotation angles respectively corresponding to the geometric deformation among the images, and global correction is performed on the images to be registered by adopting a quadratic linear interpolation method according to the three output values;

(2b) constructing a sub-residual neural network for correcting local deformation among SAR images, wherein the sub-residual neural network sequentially consists of eight convolution layers and four deconvolution layers, the activation functions of the first three deconvolution layers are ReLU functions, and the activation functions of the fourth deconvolution layer are Tanh functions; outputting 1 characteristic image and offset u of corresponding pixel values of a reference image according to the sub residual error neural network, and performing local distortion correction on the image corrected by the sub convolution neural network by adopting a quadratic linear interpolation method;

(2c) sequentially connecting the two sub-neural networks constructed in the steps (2a) and (2b) to form an SAR image fine registration neural network;

(2d) constructing a Loss function Loss of a neural network for SAR image fine registration:

(2d1) constructing a loss function for the sub-convolutional neural network: loss_glo＝||x-y||₁+(1-SSIM(x,y))

(2d2) Constructing a loss function of the sub-residual neural network:

(2d3) adding the loss functions constructed by the (2d1) and the (2d2) to obtain the loss function of the neural network for fine registration of the SAR image: loss is less_glo+loss_loc，

Wherein x is a corrected image obtained by the sub-convolution neural network, y is a reference image, and x₁Is a corrected image obtained by a sub-residual neural network, | | · | | non-woven phosphor₁Representing a 1-norm, SSIM (. cndot.) is a structural similarity function,

and

representing local differentials in the horizontal and vertical directions, respectively;

(3) training the neural network constructed in the step (2) by using the data set phi obtained in the step (1) to obtain a well-registered image:

(3a) setting the learning rate parameter as 0.0001;

(3b) randomly extracting two SAR images from a data set phi, taking one SAR image as an image to be registered and taking the other SAR image as a reference image, and inputting the two images into a neural network for fine registration of the SAR images;

(3c) updating the weight of the neural network for fine registration of the SAR image constructed in the step (2) by adopting a back propagation algorithm so as to reduce the Loss function value;

(3d) repeating the steps 3b) and 3c) until the Loss function Loss of the neural network of the SAR image fine registration is converged to obtain a trained SAR image fine registration neural network model;

(3e) and inputting the reference image and the image to be registered into the trained neural network model together to obtain the registered SAR image.

2. The method of claim 1, wherein the number of the output feature maps of the first seven convolutional layers in the sub-convolutional neural network (2a) consisting of eight convolutional layers and one global average pooled layer is respectively 32, 64, 128, 256, and the convolutional kernel sizes are respectively 7 × 7, 5 × 5, 3 × 3, and the step sizes are all 2 × 2; the number of output characteristic maps of the 8 th convolutional layer is 3, the size of the convolutional kernel is 3 x 3, and the step size is 2 x 2.

3. The method according to claim 1, wherein the number of feature maps output by the first four convolutional layers in the sub-residual neural network (2b) consisting of eight convolutional layers and four deconvolution layers is 32, 64, 128, 256, respectively, the sizes of convolution kernels are 7 × 7, 5 × 5, 3 × 3 in sequence, the step sizes are 2 × 2, the number of feature maps output by the last four convolutional layers is 256, the sizes of convolution kernels are 3 × 3, and the step sizes are 1 × 1; the number of output feature maps of the first three deconvolution layers is 256, 128 and 64 respectively, the sizes of convolution kernels are sequentially 3 × 3, 3 × 3 and 5 × 5, the step sizes are all 2 × 2, the number of output feature maps of the last deconvolution layer is 1, the size of the convolution kernel is 7 × 7, and the step size is 2 × 2.

4. The method of claim 1, wherein (2d1) is implemented as follows:

(2d11) inputting the image to be registered and the reference image y into a sub-convolution neural network to obtain a first corrected image x, and constructing | | x-y | | | Y (Y) by taking 1-norm as a pixel similarity measurement standard according to the pixel similarity of the first corrected image x and the reference image y₁The function is characterized in that the larger the function value is, the poorer the pixel similarity is;

(2d12) according to the structural similarity of the first correction image x and the reference image y, selecting a structural similarity function SSIM (·) as a structural similarity measurement standard structure (1-SSIM (x, y)) function, wherein the larger the function value is, the worse the structural similarity is;

(2d13) adding (2d11) to the function constructed in (2d12) to obtain the loss function of (2d 1):

loss_glo＝||x-y||₁+(1-SSIM(x,y))。

5. the method of claim 1, wherein (2d2) is implemented as follows:

(2d21) inputting the first correction image x into a sub-residual neural network to obtain a second correction image x₁From the second correction image x₁And constructing the pixel similarity of the reference image y by taking 1-norm as a pixel similarity measurement standard₁-y||₁The larger the function value is, the poorer the pixel similarity is;

(2d22) from the second correction image x₁Structural similarity with a reference image y, and selecting a structural similarity function SSIM (-) as a structural similarity measurement standard structure (1-SSIM (x)₁Y)) function, the larger the function value, the worse the structural similarity;

(2d23) according to the offset of the residual error neural network output, the spatial smoothness is required, and the spatial smoothness is expressed on a horizontal gradient and a vertical gradient, so that a function is constructed

As a measurement standard of the spatial smoothness, the larger the function value is, the worse the spatial smoothness is;

(2d24) adding the functions constructed in (2d21), (2d22) and (2d23) to obtain the loss function of (2d 2):

Images