[go: up one dir, main page]
More Web Proxy on the site http://driver.im/

CN114782247A - Image super-resolution reconstruction method - Google Patents

Image super-resolution reconstruction method Download PDF

Info

Publication number
CN114782247A
CN114782247A CN202210363002.1A CN202210363002A CN114782247A CN 114782247 A CN114782247 A CN 114782247A CN 202210363002 A CN202210363002 A CN 202210363002A CN 114782247 A CN114782247 A CN 114782247A
Authority
CN
China
Prior art keywords
loss
image
charbon
reconstruction method
network
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202210363002.1A
Other languages
Chinese (zh)
Inventor
许淑华
齐鸣鸣
张谦
孙亚新
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Wenzhou University of Technology
Original Assignee
Wenzhou University of Technology
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Wenzhou University of Technology filed Critical Wenzhou University of Technology
Priority to CN202210363002.1A priority Critical patent/CN114782247A/en
Publication of CN114782247A publication Critical patent/CN114782247A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
    • G06T3/4053—Scaling of whole images or parts thereof, e.g. expanding or contracting based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • Molecular Biology (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • Artificial Intelligence (AREA)
  • Evolutionary Computation (AREA)
  • General Health & Medical Sciences (AREA)
  • Biomedical Technology (AREA)
  • Computing Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Health & Medical Sciences (AREA)
  • Image Analysis (AREA)

Abstract

The invention discloses a super-resolution model (SRPUGAN-Charbon) based on PUGAN-Charbon. The model includes a generator network of synthetic super-resolution (SR) images and a discriminator network trained to distinguish SR images from true high-resolution (HR) images. The image super-resolution reconstruction method provided by the invention uses Charbonnier loss to process the abnormal value of the SR image, retains the low-frequency characteristic of the SR image, and uses positive unlabeled classification (PU) in generation of a countermeasure network (GAN), so that a discriminator is trained properly, and the stability of training is further improved. Extensive experiments on 3 reference datasets including Set5, Set14 and BSDS500 showed that the proposed SRPUGAN-Charbon method outperformed the most advanced methods in terms of PSNR, SSIM and visual effect.

Description

Image super-resolution reconstruction method
Technical Field
The invention relates to a super-resolution reconstruction method, in particular to an image super-resolution reconstruction method.
Background
Recently, the generation of countermeasure networks (GAN) has rapidly evolved from using GAN in super-resolution reconstruction (SR) due to the wide use to synthesize true high frequency details of images. However, the GAN training process has instability, and the main reason is that the discriminator in GAN keeps the positive and negative (true and false) discrimination criteria of the generated sample in the whole learning process, and does not consider the situation that the quality of the generated sample is gradually improved and is more vivid than a real sample.
Disclosure of Invention
Aiming at the defects in the prior art, the invention aims to provide an image super-resolution reconstruction method with a more stable training process.
In order to realize the purpose, the invention provides the following technical scheme: an image super-resolution reconstruction method is characterized by comprising the following steps: the method comprises the following steps:
inputting an LR image x into a generator network, storing a good, clear and distinguishable local structure of the generated image by using a-Charbon regularization to obtain a corresponding reconstructed image G (x), and then calculating content loss of a real HR image y and the reconstructed image G (x) by using a Charbon penalty function;
step two, importing the real HR image y and the reconstructed image G (x) into VGG, extracting respective high-level features phi (y) and phi (G (x)), and calculating the content loss of the high-level features phi (y) and phi (G (x)) by using a Charbon penalty function;
inputting the extracted high-level features phi (y) and phi (G (x)) into a discriminator network, obtaining the confrontation loss based on PU classification regularization, and determining a final target loss function as the weighted sum of the content loss and the confrontation loss;
step four, realizing network back propagation by using a self-adaptive a-Charbon method and PU classification regularization, calculating gradient of each layer, and updating a parameter theta of a discriminator network and a generator network according to a training strategydAnd thetaGTo iteratively optimize the network;
and step five, repeating the step one to the step four until the loss function value is minimum, and finishing.
As a further improvement of the invention, the PU framework in the third step is as follows:
Figure BDA0003581628270000021
wherein, PdataRepresenting the distribution of real samples, Z being the prior distribution PzRandom noise of medium sampling, D (x) is the probability of x being true data predicted by the discriminator, f1(. is a loss of classifying the input as a true sample, f2(. is) the loss of classifying the input as a generated sample, and π is the prior knowledge, i.e., the proportion of high quality samples in the generated sample.
As a further improvement of the invention, the adaptive evolution equation of a-Charbon in the fourth step is as follows:
Figure BDA0003581628270000022
where y is the estimated high resolution image, bkTransformation matrix, x, representing warping, blurring and decimation operationskRepresenting a sequence of low-resolution images, gamma1Is a regularization parameter, γ2(y-y0) Is a data fidelity item in which, among other things,
Figure BDA0003581628270000023
is the a-Charbon regularization term.
As a further improvement of the present invention, the objective loss function in the third step is as follows:
l＝lcharbon+0.008lVGG-charbon+2×10-6lα-charbon+10-3lPU-Gen，
wherein lcharbonFor content loss,/VGG-charbonLoss to improve VGG loss,/α-charbonIs a-Charb loss, lPU-GenIndicating a loss of antagonism.
As a further improvement of the invention, the content loss is defined as follows:
Figure BDA0003581628270000031
where r is the upsampling factor, W and H are the width and height, respectively, of the HR image,
Figure BDA0003581628270000032
ε＝10-3ρ (m) is the Charbonier penalty function, X is the LR image and y is the original HR image.
As a further improvement of the present invention, the loss for improving the VGG loss is defined as follows:
Figure BDA0003581628270000033
wherein phi5，4Represents the characteristic diagram, W, obtained after the 4 th convolution Relu before the 5 th maximum pooling level in VGG networks5，4And H5，4Respectively representing the width and height of the corresponding feature map.
As a further improvement of the invention, the a-Charb loss is defined as follows:
Figure BDA0003581628270000034
wherein, thetaGRepresents the parameters of the generator, and alpha is more than or equal to 0 and less than or equal to 2.
As a further improvement of the invention, the resistance loss is defined as follows:
Figure BDA0003581628270000035
wherein ε is 10-2To prevent the logarithmic term from being 0, thetadAnd the parameters of the discriminator are represented, pi represents class prior knowledge, namely, positive data proportion in unmarked data, n represents the number of training samples, and lambda is a regularization parameter. The invention has the beneficial effects that:
the discriminators of the models are designed based on the Charbonnier penalty function as the loss function, and the training stability of the discriminators is improved.
The generated SR image samples are processed as unlabeled samples, focusing the generator on improving the generated low quality SR image samples to improve the performance of the generator.
A new perception loss set is proposed, and the real texture and the background outline details of a reconstructed image are enhanced as much as possible through the weighted sum of content loss, feature loss, texture loss and Charbonier relative resistance loss.
Drawings
Fig. 1 is a schematic diagram of a network architecture of the image super-resolution reconstruction method of the present invention.
Detailed Description
The invention will be further described in detail with reference to the following examples, which are given in the accompanying drawings.
Referring to fig. 1, in the image super-resolution reconstruction method of this embodiment, a positive unmarked GAN of a Charbonnier loss function is used to perform single-image super-resolution reconstruction. The method aims to generate a super-resolution image which is reasonable in vision and has perceptual texture details, and comprises the following specific contents:
the whole training process can be divided into five steps:
(1) inputting the LR image x into a generator network, storing a good, clear and distinguishable local structure of the generated image by using a-Charbon regularization to obtain a corresponding reconstructed image G (x), and then calculating content loss of the real HR image y and the reconstructed image G (x) by using a Charbon penalty function.
(2) And (2) importing the real HR image y and the reconstructed image G (x) into VGG, extracting respective high-level features phi (y) and phi (G (x)), and similarly, calculating the content loss of the high-level features phi (y) and phi (G (x)) by using a Charbon penalty function.
(3) The extracted high-level features phi (y) and phi (G (x)) are input into a discriminator network, and regularization is carried out based on PU classification to obtain the penalty loss, and the final target loss function is a weighted sum of the content loss and the penalty loss.
(4) Realizing network back propagation by using self-adaptive a-Charbon method and PU classification regularization, and calculating each layerGradient, according to a training strategy, by updating the parameters θ of the discriminator network and the generator networkdAnd thetaGTo iteratively optimize the network.
(5) And repeating the steps until the loss function value is minimum, and finishing the network training work.
The above PU model obtaining process is as follows:
allowing the arbiter D to treat high quality generated samples as real data and focus on the generated low quality samples. D needs to learn how to distinguish high quality samples from other low quality samples. Under the guidance of real samples, the high-quality samples are identified from the generated samples to be very similar to the problem of positive unlabeled classification, so that the discriminator is properly trained and trained to be in a correct state, the conditions are not too good or too bad, and the stability of the discriminator is further improved. The general framework of PUGAN is as follows:
Figure BDA0003581628270000051
wherein, PdataRepresenting the distribution of real samples, z being the distribution P from the prior experimentz(i.e., Gaussian distribution) of sampled random noise, D (x) is the probability that x is predicted by the discriminator to be true data, f1(. is a loss classifying the input as a true sample, f2(. cndot.) is the loss of classifying the input as a generating sample, and π is the a priori knowledge, i.e., the proportion of high quality samples in the generating sample.
The basic feature of the a-Charbon adaptive SR method is that the regularization part is automatically switched according to the image structure. Furthermore, the parameter a selected by the control model is automatically determined by the program. The proposed a-Charbon adaptive SR evolution equation is shown in the following formula (2):
Figure BDA0003581628270000052
from equation (2), the high resolution image y, b can be estimatedkTransformation matrix, x, representing warping, blurring and decimation operationskRepresenting a sequence of low-resolution images, gamma1Is a regularization parameter, γ2(y-y0) Is a data fidelity item in which, among other things,
Figure BDA0003581628270000061
is the a-Charbon regularization term.
The objective loss function also includes content loss and antagonistic loss, as shown in the following equation (3):
l＝lcharbon+0.008lVGG-charbon+2×10-6lα-charbon+10-3lPU-Gen， (3)
wherein the content loss comprises Charbonier loss lcharbonLoss l of improvement of VGG lossVGG-charbona-Charb loss lα-charbon。lPU-GenIndicating a loss of antagonism.
Content loss
Although many of the widely used Mean Square Errors (MSE) losses in single image super-resolution reconstruction can improve PSNR, they are less robust in dealing with outliers, so a robust loss function/is used hereincharbonTo deal with outliers,/charbonThe definition is shown in the following formula (4):
Figure BDA0003581628270000062
where r is the upsampling factor, W and H are the width and height, respectively, of the HR image,
Figure BDA0003581628270000063
ρ (m) is the Charbonier penalty function [32 ]]X is the LR image and y is the original HR image.
VGG losses are introduced into the SR, which is used not only for pixel level losses but also for perceptual losses. In this text,/VGG-charbonThe loss is defined as shown in the following formula (5):
Figure BDA0003581628270000064
wherein phi is5，4Represents the characteristic diagram, W, obtained after the 4 th convolution Relu before the 5 th maximum pooling level in VGG networks5，4And H5，4Respectively representing the width and height of the corresponding feature map.
We further use the a-Charbon regularization technique to preserve good, clearly distinguishable local structures of the generated image, such as edges, lα-charbonThe regularization term is defined as shown in equation (6) below:
Figure BDA0003581628270000071
wherein, thetaGRepresents the parameters of the generator, and alpha is more than or equal to 0 and less than or equal to 2.
To combat the loss
The generation loss is added to the perception loss, the purpose is to encourage the network to support the solution residing on the natural image manifold by trying to cheat D, an unknown prior penalty term is added to the generation loss, the generation sample is processed into an unmarked sample, the generator focuses on the generation sample with low quality, the generation network is effectively optimized, and the performance of the generation network is further improved.
The penalty function is defined as:
Figure BDA0003581628270000072
wherein ε is 10-2To prevent the logarithmic term from being 0, thetadThe parameters of the discriminator are represented, pi represents class prior knowledge, namely the proportion of positive data (data with high quality) in the unlabeled data, n represents the number of training samples, and lambda is a regularization parameter.
The above description is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments, and all technical solutions belonging to the idea of the present invention belong to the protection scope of the present invention. It should be noted that modifications and adaptations to those skilled in the art without departing from the principles of the present invention should also be considered as within the scope of the present invention.

Claims (8)

1. An image super-resolution reconstruction method is characterized by comprising the following steps: the method comprises the following steps:
inputting an LR image x into a generator network, storing a good, clear and distinguishable local structure of the generated image by using a-Charbon regularization to obtain a corresponding reconstructed image G (x), and then calculating content loss of a real HR image y and the reconstructed image G (x) by using a Charbon penalty function;
step two, importing the real HR image y and the reconstructed image G (x) into VGG, extracting respective high-level features phi (y) and phi (G (x)), and calculating the content loss of the high-level features phi (y) and phi (G (x)) by using a Charbon penalty function;
inputting the extracted high-level features phi (y) and phi (G (x)) into a discriminator network, obtaining the countermeasure loss based on PU classification regularization, and determining a final target loss function as the weighted sum of the content loss and the countermeasure loss;
step four, realizing network back propagation by using a self-adaptive a-Charbon method and PU classification regularization, calculating gradient of each layer, and updating a parameter theta of a discriminator network and a generator network according to a training strategydAnd thetaGTo iteratively optimize the network;
and step five, repeating the step one to the step four until the loss function value is minimum, and finishing.
2. The image super-resolution reconstruction method according to claim 1, characterized in that: the PU framework in the third step is as follows:
Figure FDA0003581628260000011
wherein, PdataRepresenting the distribution of real samples, Z being the distribution P from the prior experimentzRandom noise of middle sampling, D (x) is x predicted as real number by discriminatorAccording to the probability, f1(. is a loss classifying the input as a true sample, f2(. cndot.) is the loss of classifying the input as a generating sample, and π is the a priori knowledge, i.e., the proportion of high quality samples in the generating sample.
3. The image super-resolution reconstruction method according to claim 1 or 2, characterized in that: the adaptive evolution equation of a-Charbon in the fourth step is as follows:
Figure FDA0003581628260000021
where y is the estimated high resolution image, bkTransformation matrix, X, representing warping, blurring and decimation operationskRepresenting a sequence of low-resolution images, gamma1Is a regularization parameter, γ2(y-y0) Is a data fidelity item in which, among other things,
Figure FDA0003581628260000022
is the a-Charbon regularization term.
4. The image super-resolution reconstruction method according to claim 1 or 2, characterized in that: the objective loss function in step three is as follows:
l＝lcharbon+0.008lVGG-charbon+2×10-6lα-charbon+10-3lPU-Gen
wherein lcharbonFor content loss,/VGG-charbonLoss to improve VGG loss,/α-charbonIs a-Charb loss, lPU-GenIndicating a loss of resistance.
5. The image super-resolution reconstruction method according to claim 4, characterized in that: the content loss is defined as follows:
Figure FDA0003581628260000023
where r is the upsampling factor, W and H are the width and height, respectively, of the HR image,
Figure FDA0003581628260000024
ε＝10-3ρ (m) is the Charbonier penalty function, X is the LR image and y is the original HR image.
6. The image super-resolution reconstruction method according to claim 5, characterized in that: the loss of improvement in VGG loss is defined as follows:
Figure FDA0003581628260000031
wherein phi5，4Represents the characteristic graph, W, obtained after 4 th convolution Relu before the 5 th maximum pooling layer in VGG network5，4And H5，4Respectively representing the width and height of the corresponding feature map.
7. The image super-resolution reconstruction method according to claim 6, wherein: the a-Charb loss is defined as follows:
Figure FDA0003581628260000032
wherein, thetaGRepresents the parameters of the generator, and alpha is more than or equal to 0 and less than or equal to 2.
8. The image super-resolution reconstruction method according to claim 4, characterized in that: the resistance loss is defined as follows:
Figure FDA0003581628260000033
wherein，ε＝10-2For preventing the logarithmic term from being 0, thetadAnd the parameters of the discriminator are represented, pi represents class prior knowledge, namely positive data proportion in unlabeled data, n represents the number of training samples, and lambda is a regularization parameter.
CN202210363002.1A 2022-04-06 2022-04-06 Image super-resolution reconstruction method Pending CN114782247A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202210363002.1A CN114782247A (en) 2022-04-06 2022-04-06 Image super-resolution reconstruction method

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202210363002.1A CN114782247A (en) 2022-04-06 2022-04-06 Image super-resolution reconstruction method

Publications (1)

Publication Number Publication Date
CN114782247A true CN114782247A (en) 2022-07-22

Family

ID=82426557

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202210363002.1A Pending CN114782247A (en) 2022-04-06 2022-04-06 Image super-resolution reconstruction method

Country Status (1)

Country Link
CN (1) CN114782247A (en)

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20180075581A1 (en) * 2016-09-15 2018-03-15 Twitter, Inc. Super resolution using a generative adversarial network
CN109978762A (en) * 2019-02-27 2019-07-05 南京信息工程大学 A kind of super resolution ratio reconstruction method generating confrontation network based on condition
CN110189253A (en) * 2019-04-16 2019-08-30 浙江工业大学 A kind of image super-resolution rebuilding method generating confrontation network based on improvement
CN111507898A (en) * 2020-03-16 2020-08-07 徐州工程学院 Image super-resolution reconstruction method based on self-adaptive adjustment

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20180075581A1 (en) * 2016-09-15 2018-03-15 Twitter, Inc. Super resolution using a generative adversarial network
CN109978762A (en) * 2019-02-27 2019-07-05 南京信息工程大学 A kind of super resolution ratio reconstruction method generating confrontation network based on condition
CN110189253A (en) * 2019-04-16 2019-08-30 浙江工业大学 A kind of image super-resolution rebuilding method generating confrontation network based on improvement
CN111507898A (en) * 2020-03-16 2020-08-07 徐州工程学院 Image super-resolution reconstruction method based on self-adaptive adjustment

Non-Patent Citations (3)

* Cited by examiner, † Cited by third party
Title
MAISELI ET AL.: "Adaptive Charbonnier superresolution method with robust edge preservation capabilities", 《JOURNAL OF ELECTRONIC IMAGING》, vol. 22, no. 4, 16 December 2013 (2013-12-16), pages 1 - 12 *
SHUHUA XU ET AL.: "A Positive-Unlabeled Generative Adversarial Network for Super-Resolution Image Reconstruction Using a Charbonnier Loss", 《TRAITEMENT DU SIGNAL》, vol. 39, no. 3, 30 June 2022 (2022-06-30), pages 1061 - 1069 *
TIANYU GUO ET AL.: "On positive-unlabeled classification in GAN", 《2020 IEEE/CVF CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION (CVPR)》, 5 August 2020 (2020-08-05), pages 8382 - 8390 *

Similar Documents

Publication Publication Date Title
Song et al. Efficient residual dense block search for image super-resolution
CN110136063B (en) Single image super-resolution reconstruction method based on condition generation countermeasure network
CN108122197B (en) Image super-resolution reconstruction method based on deep learning
CN109583474B (en) Training sample generation method for industrial big data processing
CN109671022B (en) Image texture super-resolution enhancement method based on depth feature translation network
CN109993702B (en) Full-text image super-resolution reconstruction method based on generation countermeasure network
CN112801904B (en) Hybrid degraded image enhancement method based on convolutional neural network
CN113096020B (en) Calligraphy font creation method for generating confrontation network based on average mode
CN111383173A (en) Baseline-based image super-resolution reconstruction method and system
CN115661062A (en) Industrial defect sample generation method and system based on countermeasure generation network
CN115867933A (en) Computer-implemented method, computer program product and system for processing images
CN115063318A (en) Adaptive frequency-resolved low-illumination image enhancement method and related equipment
CN114820401A (en) Method for enhancing marine backlight infrared image by combining histogram transformation and edge information
CN109272539A (en) The decomposition method of image texture and structure based on guidance figure Total Variation
CN116091823A (en) Single-feature anchor-frame-free target detection method based on fast grouping residual error module
Chen et al. Joint denoising and super-resolution via generative adversarial training
CN114782247A (en) Image super-resolution reconstruction method
CN113935899A (en) Ship plate image super-resolution method based on semantic information and gradient supervision
CN110717960B (en) Method for generating building rubbish remote sensing image sample
CN111986079A (en) Pavement crack image super-resolution reconstruction method and device based on generation countermeasure network
CN115908379A (en) Electric cooker liner image data enhancement method based on mask generation countermeasure network
CN114565806A (en) Feature domain optimization small sample image conversion method based on characterization enhancement
CN114419178A (en) Mural corresponding line draft generation method and equipment based on deep learning
CN113012079A (en) Low-brightness vehicle bottom image enhancement method and device and storage medium
CN115035315B (en) Ceramic tile color difference grading detection method and system based on attention mechanism

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination