CN115439361B - Underwater image enhancement method based on self-adversarial generative adversarial network - Google Patents

Abstract

The invention discloses an underwater image enhancement method based on a self-countermeasure generation countermeasure network, which is used for realizing the improvement of the underwater image quality based on a self-countermeasure mode and a double-input type discriminator. A new constraint is added to the enhancement process by the self-countermeasure mode, i.e., the constraint generator causes the second generated image to be superior to the first generated image. The guiding function of the discriminator on the generator is enhanced through the dual-input discriminator, and the quality of the enhanced image is further improved.

Description

Underwater image enhancement method based on self-adversarial generative adversarial network

Technical field

The invention belongs to the field of image processing technology, and specifically relates to an underwater image enhancement method based on a self-confrontational generative adversarial network.

Background technique

Since underwater images do not have reference images, the paired underwater images in the underwater image enhancement dataset are often obtained by generating degraded underwater images or enhancing underwater images. When using these images to train the model, the enhanced image The quality can only be close to the quality of the original underwater image or the quality of the underwater image enhanced by other methods. The method of training using underwater images and natural images instead of paired data improves the upper limit of image quality after enhancement. However, due to the use of images from two different domains, the enhancement results will be unnatural and the image quality will be degraded. .

Contents of the invention

The present invention proposes an underwater image enhancement method based on a self-confrontational generative adversarial network, which does not require underwater reference images, but trains on a commonly used natural image quality database to transfer quality improvements to low-quality underwater images. Achieve underwater image enhancement.

In order to achieve the above objects, the technical solutions adopted by the present invention are as follows:

The underwater image enhancement method based on the self-confrontation generative adversarial network adopts the self-confrontation mode. Its overall structure is shown in Figure 1. The self-confrontation mode does not have a corresponding reference image as the final target of training. In the self-confrontation mode, the input image is Get the output image after generator G , output image/> Input it again into the generator G to get the output image G(G(x)). The two output images are simultaneously input into the discriminator D for discrimination. The purpose is to constrain the generator so that the second generated image is better than the first one. generated image.

The overall optimization goals of the self-confrontation mode are:

(4-1)

The self-adversarial mode first obtains an output image from the generator G : Then change/> Input into the generator to get G(G(x)). At this time, the fixed generator G trains the discriminator D:

(4-2)

In Formula 4-2, the second output image G(G(x)) from the generator is expected to be judged as a positive sample by D, while the first output image It is hoped that the sample will be judged as a negative sample. At this time, the generator G is trained:

(4-3)

In formula 4-3, you want the first output image It is better than the second output image G(G(x)), thus forming a confrontation between the generator and the discriminator. However, unlike adversarial training, in this confrontation process, there is no real set of positive samples to It forms a confrontation with negative samples, but the negative samples are in a state of confrontation with themselves.

Generative adversarial underwater image enhancement method based on self-adversarial mode:

The method proposed in this article consists of a generator and two discriminators. The overall process of the method is shown in Figure 2. In the figure, x is a low-quality image, y is a high-quality image, G is the generator, and D is the discriminator of adversarial training. device, It is a self-confrontation mode discriminator. The structure of the two discriminators is the proposed dual-input image quality comparison model structure, instead of the traditional single-input two-classifier structure to determine whether the image is true or false, so it can obtain better discriminant results. . G(x) is the image after the first enhancement, and G(G(x)) is the image after the second enhancement. Among them, the discriminator/> Together with the generator G, it forms a self-confrontation mode. After being enhanced by the generator G, the original image Discriminate the two enhanced images through the discriminator to constrain the quality improvement during the generation process, and obtain better quality images through repeated iterative enhancement. In addition, the discriminator D is the Patch-GAN structure and discriminator/> It is a two-class discriminator structure. The combination of the two can make discrimination from two perspectives: global and detailed.

Compared with existing underwater image processing technology, the above technical solution can achieve the following beneficial effects:

This method is based on the self-confrontational mode and the dual-input discriminator to improve the quality of underwater images. The self-confrontational mode adds a new constraint to the enhancement process, that is, the constraint generator makes the second generated image better than the first generated image. The dual-input discriminator strengthens the guiding role of the discriminator on the generator and further improves the quality of the enhanced image.

At the same time, unpaired natural images are also used for training, and then the quality improvements are transferred to underwater images, which not only solves the problem of insufficient number of paired underwater images to a certain extent, but also solves the problem of using existing underwater images. Problems in training pairs of underwater images generated by artificial methods in image augmentation datasets. Experiments have proven that this method can effectively improve the quality of underwater images, and the generated images are more visually beautiful.

Description of the drawings

Figure 1 is a structural diagram of the self-confrontation mode.

Figure 2 is a method flow chart.

Figure 3 is the generator structure.

Figure 4 is the structure diagram of discriminator D.

Figure 5 is the discriminator Structure diagram.

Figure 6 is a partial image of the KADID-10k data set.

Figure 7 is some high-quality images in the KonIQ-10k data set.

Figure 8 is a diagram showing the increased results of the method of the present invention and the enhanced results of 8 methods in the U45 data set.

Figure 9 is a comparison chart between the method of the present invention and five other more advanced underwater image enhancement methods on the U45 data set.

Detailed ways

The present invention will be further described below in conjunction with the accompanying drawings:

An underwater image enhancement method based on a self-confrontational generative adversarial network, which is characterized in that: the method adopts a self-confrontational mode, and the structure of the self-confrontational mode is that the input image x passes through the generator G to obtain the output image , output image/> Input it again into the generator G to get the output image/> , the two output images are input to the discriminator D at the same time for discrimination, and the generator D is constrained so that the output image/> Better than output image/> ;

The overall optimization goals of the self-confrontation mode are:

(4-1)

The self-adversarial mode first obtains an output image from the generator G : Then change/> Input into the generator to get G(G(x)). At this time, the fixed generator G trains the discriminator D:

(4-2)

In Formula 4-2, the second output image G(G(x)) from the generator is expected to be judged as a positive sample by D, while the first output image It is hoped that the sample will be judged as a negative sample. At this time, the generator G is trained:

(4-3)

In formula 4-3, you want the first output image It is better than the second output image G(G(x)), thus forming a confrontation between the generator and the discriminator. However, unlike adversarial training, in this confrontation process, there is no real set of positive samples to It forms a confrontation with negative samples, but the negative samples are in a state of confrontation with themselves.

The specific application of self-confrontation mode to underwater image enhancement methods is as follows:

S1: After the original input image x is enhanced by the generator G, the output image is obtained And input it into the discriminator D to discriminate against the high-quality image y, update and fix the discriminator D, and update the generator G;

S2: Output image Input it again into the generator G for enhancement to obtain the output image/> , the output image G(G(x)) and the high-quality image y are input to the discriminator D for discrimination, and the discriminator D is updated;

S3: Self-adversarial discriminator Output image after enhancement/> and output image/> Discriminate through the discriminator/> To constrain the quality improvement during the generation process, iterative enhancement is used to obtain better quality images.

The generator G is an encoding and decoding network, and the overall structure is shown in Figure 3. The encoding part consists of a convolutional connection with a convolution kernel size of 3×3, and a residual block is added after each convolution to enhance network depth and feature extraction capabilities. Reflective padding is used before each convolution to ensure that the feature map size of the image is half of the size before each downsampling. After each convolutional layer, a Batch-norm layer and LeakyReLU activation function are used to increase the robustness and nonlinearity of the network. The decoding part consists of multiple upsampling cascades. After each upsampling, a residual block is added to enhance the image reconstruction capability of the decoding part. Skip connections are used between encoding and decoding to supplement the image information lost during downsampling. Finally, the Tanh activation function is used to avoid the vanishing gradient problem.

The discriminator specifically uses a dual discriminator to implement self-confrontation mode and supervised learning. The dual discriminator structure is shown in Figures 4 and 5. The discriminator D is a Patch-GAN structure, the discriminator It is a traditional binary classification network structure. The structure of the discriminator D is shown in Figure 4. Discriminator/> The structure is shown in Figure 5. Both discriminators have two input images/> and/> , the input image size is 512×512. The two inputs enter two feature extraction modules with the same structure. The feature extraction module consists of three convolutional layers with a convolution kernel size of 3×3 and a stride of 2. After the feature extraction module, the extracted feature maps are concatenated, and the concatenated feature maps are input into the Inception module, which contains 3 convolutional layers of different sizes and 1 pooling layer. The sizes of the convolution kernels are 5×5, 3×3, and 1×1 respectively, and the size of the pooling layer is 3×3. Afterwards, the feature maps of convolutional layers and pooling layers of different sizes obtained through the Inception module are concatenated and input into the Reduction module for downsampling. The Reduction module contains two branches. One branch is a single convolution layer with a convolution kernel size of 3×3 and a step size of 2. The other branch is a superimposed three-layer convolution layer with a convolution kernel of the convolution layer. The sizes are 1×1, 3×3, and 3×3 from top to bottom, and the step sizes of the convolutional layers are 1, 1, and 2 from top to bottom. Then, the feature maps obtained after downsampling the two different branches of Reduction are connected, so that the obtained feature map contains networks of different depths and obtains richer feature information. Finally, after two convolutional layers with a convolution kernel size of 3×3 and a step size of 2, a feature map of size 7×7 is obtained, and the discriminator/> Directly output a 7×7 feature map for discrimination. After the discriminator D passes through the adaptive average pooling layer, it is input to two fully connected layers.

The specific training methods are as follows:

Each parameter update consists of two steps. In the first step, the low-quality image x is input into the generator G to obtain the enhanced image G(x), and then the obtained enhanced image G(x) and high-quality image y are input into the discriminator D. Discriminate and update the parameters of the discriminator D at the same time. After that, fix the parameters of the discriminator D and update the parameters of the generator G. In the second step, the enhanced image G(x) obtained in the first step is input into the generator G to obtain the enhanced image G(G(x)) again, and then the second enhanced image G( G(x)) and high-quality image y are input to the discriminator D for discrimination, and the parameters of the discriminator D are updated. At the same time, the second enhanced image G(G(x)) and the first enhanced image are G(x) is input to the self-adversarial discriminator Discriminate in and update the self-adversarial discriminator/> parameters, and then update the parameters of generator G again.

The method model was trained using the Adam optimizer, the learning rate was set to 0.001, the batch size was set to 2, and a total of 50 iterative trainings were performed. Training was performed on a computer equipped with an i7 processor, Nvidia Titan XP GPU and 64G memory, using the Pytorch framework.

The loss function is used in the above training method:

The entire training process of the method includes 3 adversarial losses, 2 perceptual losses and 2 norm loss. In the first step, this paper uses adversarial loss to constrain the optimization process of the generator and discriminator, and uses perceptual loss and The weighted sum of norm losses is used as content loss to ensure that the image retains content information during the enhancement process. The total loss function is shown in formula (4-4).

(4-4)

in and/> They are/> The weight of norm loss and perceptual loss, this value is determined through experiments and is set to 0.1 in this article. /> ,/> and/> As shown in formulas (4-5) to (4-7).

(4-5)

(4-6)

(4-7)

Among them, D is the discriminator, G is the generator, x is the input image, and G(x) is the enhanced image obtained in the first step. is the jth convolutional layer of the VGG-19 network pre-trained on the ImageNet dataset.

In the second step, this paper uses 2 different adversarial losses, one of which is similar to the adversarial loss in the first step, used to guide the generator to generate an image quality closer to the reference image, and the other adversarial loss The loss is used in the self-confrontational mode to further improve the enhancement results by discriminating the two enhanced images obtained in the two steps. At the same time, in order to preserve the image content, perceptual loss and The weighted sum of norm losses serves as constraints. The total loss function is shown in formula (4-8).

(4-8)

and/> They are/> The weights of norm loss and perceptual loss, the values here are consistent with the first step, ,/> ,/> and/> As shown in formulas (4-9) to (4-12).

(4-9)

(4-10)

(4-11)

(4-12)

in is the discriminator of self-adversarial mode, and G(G(x)) is the enhanced image obtained in the second step.

Data set selection:

According to the degradation problems such as low contrast, color shift, low illumination, and blur existing in underwater images, distorted images and high-quality images with similar underwater degradation types in the natural image data set are selected to train the proposed underwater image enhancement method. .

KADID-10k is a natural image quality evaluation data set, which contains 81 original images and 25 types of distortion images corresponding to these 81 images. Each distortion contains 5 levels, totaling 10125 distortion images. From all distortion images with a distortion level of 5 in the KADID-10k data set, this paper selected distortion images with distortion types in underwater images including color distortion, noise, low contrast, etc., a total of 1620 distorted images, some of which are shown in the figure 6 shown.

KonIQ-10k is a large IQA dataset that contains 10,073 non-artificial natural distortion images. These 10,073 natural images receive quality scores through voting. Since there are only 81 original reference images in the KADID-10k dataset, these 81 images are It is far from enough to train the model, so this article selected 1539 high-quality images from the KonIQ-10k data set to add to the high-quality image collection. Some images are shown in 7.

45 low-quality underwater images on the U45 data set were used for testing, and the enhancement results of the method of the present invention were compared with 8 underwater image enhancement methods included in the U45 data set. In addition, this article also compared with some Other state-of-the-art underwater image enhancement methods are compared:

Comparison of experimental results on the U45 data set:

The U45 real underwater image data set is a data set that contains 45 underwater images with different color differences, low contrast underwater images and foggy underwater images, as well as the results of 8 enhancement methods to enhance these underwater images. . The U45 data set was used as a test set, and the enhancement results of the method of the present invention were compared with all the enhancement results contained in it, as shown in Figure 8.

In Figure 8, the enhancement results of the method of the present invention and the enhancement results of 8 methods in the U45 data set are shown. The leftmost (a) is the original image, and the remaining images from left to right are from (b) CycleGAN, (c) FE, ( d) DewaterNet, (e) RB, (f) RED, (g) UDCP, (h) UIBLA, (i) WSCT, (j) method of the present invention.

Through image comparison analysis:

(b) CycleGAN corrects the color shift, but the enhancement effect is not good enough. The enhancement result has an obvious checkerboard texture, as shown in Figure 8(b).

(c) The enhancement result of FE has an obvious red color cast, as shown in Figure 8(c).

(d) The overall color of the enhancement result of DewaterNet is darker, as shown in Figure 8(d).

(e) Some darker images also appear in the RB enhancement results, as shown in Figure 8(e).

(f) RED has insufficient color correction, as shown in Figure 8(f).

(g) UDCP, (h) UIBLA, (i) WSCT cannot completely correct the chromatic aberration, and the chromatic aberration of some enhancement results is more serious, as shown in Figure 8(g-i).

The method of the present invention can well correct various chromatic aberration problems, and the visual effect of the enhanced image is better, as shown in Figure 8(j).

In order to further prove the performance of the method of the present invention, this paper also compares the method of the present invention with five other more advanced underwater image enhancement methods on the U45 data set, as shown in Figure 9.

The method proposed by Yang et al. cannot completely correct the color of greenish images, as shown in Figure 9(b).

DeepSESR does not fully implement color correction and introduces a red color cast in some images, as shown in Figure 9(c).

UWCNN introduces a more severe red color cast on light-colored images, as shown in Figure 9(d).

FUnIEGAN's color correction of greenish underwater images results in yellowish images, as shown in Figure 9(e).

The image enhanced by HybridDectionGAN deviates from the original color of the image, and an obvious checkerboard texture appears in the image, as shown in Figure 9(f).

The method of the present invention realizes color correction, while also retaining the original color of the image, and the visual effect is more beautiful, as shown in Figure 9(g).

Two common underwater image quality evaluation indicators, UCIQE and UIQM, are used to objectively evaluate the method proposed in the present invention. The larger the value of UCIQE and UIQM, the better the underwater image quality. The evaluation results are shown in the table above, where the top three for each indicator are marked in bold. It can be seen from the table that the result of the method of the present invention on UCIQE can reach the third place, but the result on UIQM is not ideal because the method of the present invention uses interpolation in the upsampling process to ensure that the size of the output image is consistent with the input image. The size is consistent, so the method of the present invention does not score high in the underwater image definition measurement component, resulting in an overall evaluation score that is not high enough.

This method is based on self-confrontational mode and dual-input discriminator to improve underwater image quality. The self-adversarial mode adds a new constraint to the enhancement process, which constrains the generator to make the second generated image better than the first generated image. The dual-input discriminator strengthens the guidance role of the discriminator on the generator and further improves the quality of the enhanced image. At the same time, unpaired natural images are also used for training, and then the quality improvements are transferred to underwater images, which not only solves the problem of insufficient number of paired underwater images to a certain extent, but also solves the problem of using existing underwater images. Problems in training pairs of underwater images generated by artificial methods in image augmentation datasets. Experiments have proven that the method proposed in this article can effectively improve the quality of underwater images, and the generated images are more visually beautiful.

The above description of the disclosed embodiments enables those skilled in the art to make or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be practiced in other embodiments without departing from the spirit or scope of the invention. Therefore, the present invention is not to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims (7)

1. An underwater image enhancement method based on a self-confrontational generative adversarial network, characterized in that: the method adopts a self-confrontational mode. The structure of the self-confrontational mode is that the input image x passes through the generator G to obtain the output image G(x), and the output The image G(x) is input into the generator G again to obtain the output image G(G(x)). The two output images are simultaneously input into the discriminator D for discrimination. The generator D is constrained so that the output image G(G(x) )) is better than the output image G(x). The specific application of self-confrontation mode to underwater image enhancement methods is as follows: S1: After the original input image x is enhanced by the generator G, the output image G(x) is obtained and input into the discriminator D for discrimination with the high-quality image y. The discriminator D is updated and fixed, and the generator G is updated; S2: The output image G(x) is input again into the generator G for enhancement to obtain the output image G(G(x)). The output image G(G(x)) and the high-quality image y are input into the discriminator D for discrimination. , and update the discriminator D; S3: The self-adversarial discriminator D _s distinguishes the enhanced output image G(x) and the output image G(G(x)). The discriminator D _s is used to constrain the quality improvement in the generation process. Iterative enhancement is repeated to obtain quality improvement. good images; The overall optimization goals of the self-confrontation mode are: The self-adversarial mode first obtains an output image G(x) from the generator G: then inputs G(x) into the generator G to obtain G(G(x)). At this time, the generator G is fixed to train the discriminator D. : In Formula 4-2, the second output image G(G(x)) from the generator G hopes to be judged as a positive sample by the discriminator D, while the first output image G(x) hopes to be judged as a negative sample. Sample, now train the generator G: In Formula 4-3, it is hoped that the first output image G(x) is better than the second output image G(G(x)), thus forming a confrontation between the generator and the discriminator. 2. The underwater image enhancement method based on self-adversarial generative adversarial network according to claim 1, characterized in that: the generator G is a codec network, and the coding part is connected by a convolution kernel with a size of 3×3. Composition, adding a residual block after each convolution to enhance network depth and feature extraction capabilities; using reflection padding before each convolution to ensure that the feature map size of the image is half of the size before each downsampling; in each convolution After the accumulation layer, the Batch-norm layer and LeakyReLU activation function are used to increase the robustness and nonlinearity of the network; the decoding part consists of multiple upsampling cascades; after each upsampling, a residual block is added to enhance The image reconstruction ability of the decoding part is improved; skip connections are used between encoding and decoding, and Tanh activation functions are used to avoid the vanishing gradient problem. 3. The underwater image enhancement method based on self-antagonistic generative adversarial network according to claim 1, characterized in that: the discriminator D is a Patch-GAN structure, and the discriminator D _s is a binary classification network structure. 4. The underwater image enhancement method based on self-adversarial generative adversarial network according to claim 1, characterized in that: in said S1, the output image G(x) is obtained and input into the discriminator D together with the high-quality image y When making the discrimination, the parameters of the discriminator D need to be updated at the same time. After that, the parameters of the discriminator D are fixed and the parameters of the generator G are updated. 5. The underwater image enhancement method based on self-adversarial generative adversarial network according to claim 1, characterized in that: using adversarial loss to constrain the optimization process of the generator and the discriminator, and using perceptual loss and _L1 norm loss The weighted sum of is used as the content loss to ensure that the image retains content information during the enhancement process. 6. The underwater image enhancement method based on self-adversarial generative adversarial network according to claim 1 or 4, characterized in that: in S2 and S3, the second enhanced image G (G (x)) needs to be combined with high The high-quality image y is input to the discriminator D for discrimination, and the parameters of the discriminator D are updated. At the same time, the second enhanced image G(G(x)) and the first enhanced image G(x) are input to the self- Discrimination is performed in the adversarial discriminator D _s , and the second enhanced image G(G(x)) and the first enhanced image G(x) are input into the self-adversarial discriminator D _s for discrimination, and the self-adversarial discriminant is updated. parameters of generator D _s , and then update the parameters of generator G again. 7. The underwater image enhancement method based on self-confrontation generative adversarial network according to claim 6, characterized in that: using adversarial loss in the self-confrontation mode, the enhancement result is achieved by discriminating the two enhanced images obtained. For further improvement, at the same time, in order to preserve the image content, a weighted sum of perceptual loss and _L1 norm loss is used as a constraint.