[go: up one dir, main page]
More Web Proxy on the site http://driver.im/

CN116777766A - Image enhancement method, electronic device and storage medium - Google Patents

Image enhancement method, electronic device and storage medium Download PDF

Info

Publication number
CN116777766A
CN116777766A CN202310609286.2A CN202310609286A CN116777766A CN 116777766 A CN116777766 A CN 116777766A CN 202310609286 A CN202310609286 A CN 202310609286A CN 116777766 A CN116777766 A CN 116777766A
Authority
CN
China
Prior art keywords
image
generator
loss function
sample
network
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202310609286.2A
Other languages
Chinese (zh)
Inventor
洪铁鑫
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Chengdu Ck Technology Co ltd
Original Assignee
Chengdu Ck Technology Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Chengdu Ck Technology Co ltd filed Critical Chengdu Ck Technology Co ltd
Priority to CN202310609286.2A priority Critical patent/CN116777766A/en
Publication of CN116777766A publication Critical patent/CN116777766A/en
Pending legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/20—Image preprocessing
    • G06V10/26—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G06N3/0455—Auto-encoder networks; Encoder-decoder networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/0464—Convolutional networks [CNN, ConvNet]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/0475—Generative networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/084—Backpropagation, e.g. using gradient descent
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/094—Adversarial learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Evolutionary Computation (AREA)
  • General Health & Medical Sciences (AREA)
  • Health & Medical Sciences (AREA)
  • Artificial Intelligence (AREA)
  • Software Systems (AREA)
  • Computing Systems (AREA)
  • Biomedical Technology (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • Data Mining & Analysis (AREA)
  • Molecular Biology (AREA)
  • General Engineering & Computer Science (AREA)
  • Mathematical Physics (AREA)
  • Multimedia (AREA)
  • Databases & Information Systems (AREA)
  • Medical Informatics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Image Analysis (AREA)

Abstract

The embodiment of the application discloses an image enhancement method, electronic equipment and a storage medium, wherein the method comprises the following steps: acquiring a first image, wherein the first image is a dim light image to be enhanced; inputting the first image into a target generator for processing to obtain a second image, wherein the second image is an image with enhanced dim light; the target generator is trained based on a generator in a generative countermeasure network, the generative countermeasure network comprises the generator and a discriminator, the generator comprises a semantic segmentation network, and the semantic segmentation network is used for assisting in extracting dim light features of different layers in the first image.

Description

Image enhancement method, electronic device and storage medium
Technical Field
The present application relates to the field of image processing technologies, and in particular, to an image enhancement method, an electronic device, and a storage medium.
Background
With the development and popularization of electronic devices such as smart phones and network cameras, more and more users select to use the electronic devices to collect images, and meanwhile, the requirements of users on image collection of the electronic devices are higher and higher. At night or under other poor illumination conditions, the image acquired by the electronic device often has problems of image quality degradation, scene image information loss and the like caused by over-darkness of the image, so that the use experience of a user is affected, and therefore, the enhancement of the image quality of a darkness image becomes necessary.
In the related art, the enhancement of the dim light image is mainly performed based on the convolutional neural network, however, because the convolutional neural network has a single structure, a large number of data sets are needed in the training process, and the local optimal solution is easy to fall into, a good result cannot be obtained, and the enhancement effect of the dim light image is poor.
Disclosure of Invention
The embodiment of the application provides an image enhancement method, electronic equipment and a storage medium, which are used for solving the technical problem of poor enhancement effect of a dim light image in the related technology.
According to a first aspect of the present application, there is disclosed an image enhancement method, the method comprising:
acquiring a first image, wherein the first image is a dim light image to be enhanced;
inputting the first image into a target generator for processing to obtain a second image, wherein the second image is an image with enhanced dim light;
the target generator is trained based on a generator in a generative countermeasure network, the generative countermeasure network comprises the generator and a discriminator, the generator comprises a semantic segmentation network, and the semantic segmentation network is used for assisting in extracting dim light features of different layers in the first image.
According to a second aspect of the present application, there is disclosed an image enhancement apparatus, the apparatus comprising:
the acquisition module is used for acquiring a first image, wherein the first image is a dim light image to be enhanced;
the enhancement module is used for inputting the first image into a target generator for processing to obtain a second image, wherein the second image is an image with enhanced dim light;
the target generator is trained based on a generator in a generative countermeasure network, the generative countermeasure network comprises the generator and a discriminator, the generator comprises a semantic segmentation network, and the semantic segmentation network is used for assisting in extracting dim light features of different layers in the first image.
According to a third aspect of the present application, an electronic device is disclosed comprising a memory, a processor and a computer program stored on the memory, the processor executing the computer program to implement the image enhancement method as in the first aspect.
According to a fourth aspect of the present application, there is disclosed a computer readable storage medium having stored thereon a computer program/instruction which, when executed by a processor, implements the image enhancement method as in the first aspect.
According to a fifth aspect of the present application, a computer program product is disclosed, comprising a computer program/instructions which, when executed by a processor, implement the image enhancement method as in the first aspect.
In the embodiment of the application, a first image is acquired, wherein the first image is a dim light image to be enhanced; inputting the first image into a target generator for processing to obtain a second image, wherein the second image is an image with enhanced dim light; the target generator is trained based on a generator in a generated countermeasure network, the generated countermeasure network comprises a generator and a discriminator, the generator comprises a semantic segmentation network, and the semantic segmentation network is used for assisting in extracting dim light features of different layers in the first image.
Therefore, in the embodiment of the application, the dim light image can be enhanced based on the target generator obtained by the generation type countermeasure network training, because the generation type countermeasure network has two different networks instead of a single network, and the training mode adopts the countermeasure training mode, the continuous game of the two networks of the generator and the discriminator can fully utilize the training data; in addition, the generator comprises the characteristics that the semantic segmentation network can fully mine the image, so that the target generator obtained through training can learn the relatively accurate priori dim light characteristics, and the enhancement effect is relatively good when the dim light image is enhanced based on the target generator.
Drawings
FIG. 1 is a flowchart of an image enhancement method according to an embodiment of the present application;
FIG. 2 is a diagram illustrating an exemplary network configuration of a generator in a generative countermeasure network provided by an embodiment of the present application;
FIG. 3 is a flow chart of a model training method provided by an embodiment of the present application;
fig. 4 is a schematic structural diagram of an image enhancement device according to an embodiment of the present application;
fig. 5 is a block diagram of an electronic device according to an embodiment of the present application.
Detailed Description
In order that the above-recited objects, features and advantages of the present application will become more readily apparent, a more particular description of the application will be rendered by reference to the appended drawings and appended detailed description.
It should be noted that, for simplicity of description, the method embodiments are shown as a series of acts, but it should be understood by those skilled in the art that the embodiments are not limited by the order of acts, as some steps may occur in other orders or concurrently in accordance with the embodiments. Further, those skilled in the art will appreciate that the embodiments described in the specification are presently preferred embodiments, and that the acts are not necessarily required by the embodiments of the application.
In recent years, technology research such as computer vision, deep learning, machine learning, image processing, image recognition and the like based on artificial intelligence has been advanced significantly. Artificial intelligence (Artificial Intelligence, AI) is an emerging scientific technology for studying and developing theories, methods, techniques and application systems for simulating and extending human intelligence. The artificial intelligence discipline is a comprehensive discipline and relates to various technical categories such as chips, big data, cloud computing, internet of things, distributed storage, deep learning, machine learning, neural networks and the like. Computer vision is an important branch of artificial intelligence, and particularly, machine recognition is a world, and computer vision technologies generally include technologies such as face recognition, living body detection, fingerprint recognition and anti-counterfeit verification, biometric feature recognition, face detection, pedestrian detection, object detection, pedestrian recognition, image processing, image recognition, image semantic understanding, image retrieval, word recognition, video processing, video content recognition, behavior recognition, three-dimensional reconstruction, virtual reality, augmented reality, synchronous positioning and map building (SLAM), computational photography, robot navigation and positioning, and the like. With research and progress of artificial intelligence technology, the technology expands application in various fields, such as security protection, city management, traffic management, building management, park management, face passing, face attendance, logistics management, warehouse management, robots, intelligent marketing, computed photography, mobile phone images, cloud services, intelligent home, wearing equipment, unmanned driving, automatic driving, intelligent medical treatment, face payment, face unlocking, fingerprint unlocking, personnel verification, intelligent screen, intelligent television, camera, mobile internet, network living broadcast, beauty, make-up, medical beauty, intelligent temperature measurement and the like.
Taking the image processing field of dim light enhancement (also called dim light enhancement) as an example, the dim light enhancement has wide application in different fields, including video acquisition, automatic driving, computational photography, and the like.
The current dim light enhancement method is divided into a traditional method and a deep learning mode. Conventional methods of dim light enhancement include histogram matching based approaches, retinex based approaches, and fusion based approaches. The dim light enhancement method based on deep learning is mainly based on a convolutional neural network at present, the convolutional neural network has a single structure, a large number of data sets are needed in the training process, and a local optimal solution is easy to fall into, so that a good result cannot be obtained. In addition, dark areas of a dark-light image often hide much noise, which is amplified after these areas are emphasized, resulting in degradation of the image quality. The conventional method and convolutional neural network cannot solve well in the face of this situation.
In order to solve the above problems, embodiments of the present application provide an image enhancement method, an electronic device, and a storage medium.
For ease of understanding, some concepts involved in embodiments of the application are first described below.
A generated antagonism network (Generative Adversarial Network, GAN), which is a combination of two networks: the Generator (also called a generation network, generator, or G) is responsible for generating analog data, and the arbiter (also called a discrimination network, a Discriminator, or D) is responsible for determining whether the input data is real or generated. The generator is required to continuously optimize the data generated by the generator to ensure that the data cannot be judged by the discriminator, and the discriminator is required to optimize the data to ensure that the data can be judged by the discriminator more accurately, so that the relationship between the data and the data forms the countermeasure, and the countermeasure network is called.
The semantic segmentation network is used for carrying out semantic segmentation on the image, wherein the semantic segmentation of the image is to allocate a semantic category to each pixel in the input image so as to obtain pixelated dense categories.
Compared with the prior art, the embodiment of the application provides a method for enhancing the dim light image based on a generated type countermeasure network, wherein the generated type countermeasure network has two different networks instead of a single network, the training mode adopts a countermeasure training mode, and the two networks of a generator and a discriminator are continuously game to fully utilize a limited data set to achieve an optimal result. Compared with other methods, the generated countermeasure network can generate clearer and more real samples, can well remove noise in a dark area in a dark image to obtain a high-quality image, and has relatively less required data volume, and the generated countermeasure network model only uses back propagation in the training process without a complex Markov chain, so that the time complexity can be obviously reduced.
An image enhancement method provided by an embodiment of the present application is described next.
Fig. 1 is a flowchart of an image enhancement method according to an embodiment of the present application, as shown in fig. 1, the method may include the following steps: step 101 and step 102;
in step 101, a first image is acquired, wherein the first image is a darklight image to be enhanced.
In the embodiment of the application, the first image can be an image shot by mobile equipment such as a smart phone, an unmanned aerial vehicle and the like, or can be an image shot by a network camera.
In step 102, inputting the first image into a target generator for processing to obtain a second image, wherein the second image is an image with enhanced dim light; the target generator is trained based on a generator in a generated countermeasure network, the generated countermeasure network comprises a generator and a discriminator, the generator comprises a semantic segmentation network, and the semantic segmentation network is used for assisting in extracting dim light features of different layers in the first image.
In the embodiment of the application, a multi-frame fusion mode can be used for obtaining a high-quality ev-0 real image (GT for short) and a dim light data sample of ev-2, wherein the data samples ev-0 and ev-2 need to be paired; the data is then fed into a generative countermeasure network whose generator continuously generates high quality image samples from the dim light image of ev-2, and whose arbiter then continuously scores the image samples generated by the generator by GT data. The image generated by the generator needs to be deceptively judged to be a high-quality image and a true sample by the discriminator, and the discriminator is used for judging that the image generated by the generator is a false sample as far as possible, the two networks continuously fight against training, and the data content is fully utilized to train an optimal model, namely the target generator.
In the embodiment of the application, the semantic segmentation network is introduced into the generator of the generated type countermeasure network, and the network structure can play a role in enhancing the characteristics and fully mining the data.
In some embodiments, the semantic segmentation network in the generator network may be a Unet network, where the Unet network may simply stitch the feature map of the encoder to the upsampled feature map of each stage decoder, forming a ladder; by the architecture of skip splicing, the decoder is allowed to learn the relevant features lost in the encoder pooling at each stage, with accurate and faster inferences.
For example, as shown in fig. 2, a network structure of a generator in a generative countermeasure network is shown, the generator adds a uiet network on the basis of an original res net network, wherein the uiet network is connected with the res net network and is located behind the res net network; the network structure of the discriminators in the generated countermeasure network may employ a network structure in the related art.
In the embodiment of the application, when the target generator processes the first image, the basic feature map of the first image is extracted through the residual network, and the basic feature map is processed through the Unet network to obtain the second image.
As can be seen from the above embodiment, in this embodiment, a first image is acquired, where the first image is a dark image to be enhanced; inputting the first image into a target generator for processing to obtain a second image, wherein the second image is an image with enhanced dim light; the target generator is trained based on a generator in a generated countermeasure network, the generated countermeasure network comprises a generator and a discriminator, the generator comprises a semantic segmentation network, and the semantic segmentation network is used for assisting in extracting dim light features of different layers in the first image.
Therefore, in the embodiment of the application, the dim light image can be enhanced based on the target generator obtained by the generation type countermeasure network training, because the generation type countermeasure network has two different networks instead of a single network, and the training mode adopts the countermeasure training mode, the continuous game of the two networks of the generator and the discriminator can fully utilize the training data; in addition, the generator comprises the characteristics that the semantic segmentation network can fully mine the image, so that the target generator obtained through training can learn the relatively accurate priori dim light characteristics, and the enhancement effect is relatively good when the dim light image is enhanced based on the target generator.
Fig. 3 is a flowchart of a model training method according to an embodiment of the present application, and as shown in fig. 3, the method may include the following steps: step 301, step 302 and step 303;
in step 301, a training set is obtained, wherein the training set comprises: a plurality of sample image pairs, each sample image pair comprising: a sample normal light image and a corresponding sample dim light image.
In the embodiment of the application, the sample images in the training set may be automatically generated images, for example, a multi-frame fusion mode may be adopted to generate the sample images in each sample image pair.
In step 302, a generative countermeasure network, a first loss function corresponding to a generator in the generative countermeasure network, and a second loss function corresponding to a arbiter are constructed.
In some embodiments, to improve the quality of the output image of the generator, the first loss function may be composed of at least one of the following loss functions: a third loss function, a fourth loss function, and a fifth loss function; the third loss function is used for balancing the enhancement degree between a dim light area and other areas in the sample dim light image; the fourth loss function is used for measuring the structural similarity between the output image of the generator and the corresponding normal light image of the sample; the fifth loss function is used to measure the style gap between the output image of the generator and the corresponding sample normal light image. The second loss function may employ a conventional counterloss function.
For example, the first Loss function is Loss1, the second Loss function is Loss2, the third Loss function is Loss3, the fourth Loss function is Loss4, and the fifth Loss function is Loss5. Los1=los3×a+los4×b+los5×c, where a, b, c are weight coefficients, and can be set according to actual requirements; loss2 may employ a conventional counterloss function V (G, D).
Alternatively, the third loss function may be determined by: acquiring brightness weight items of all pixel points in a sample dim light image; according to the brightness weight items of the pixel points, calculating a weight image of a sample dim light image, wherein the dim light weight value of a dim light area in the weight image is larger than the dim light weight values of other areas; and determining a third loss function according to the weight image, the output image of the generator and the corresponding sample normal light image.
In the embodiment of the application, the brightness weight item I of the pixel point in the image can be obtained by calculating in the following manner: with the general idea, it is considered that the pixel brightness with good exposure tends to approach 0.5 with a large probability, and the formula of i is as follows:wherein Y is the pixel value of the pixel point on the Y channel, and mu and delta 1 Is a parameter set according to actual requirements.
For example, a third loss functionWherein, G theta (lin) is the output image of the generator, lgt is the sample normal light image, m is the number of pixel points in the image, and w 1 For the weight of the dim light,it can be seen that the closer the brightness weight term of the pixel is to 0.2, the larger the value of the dark light weight of the pixel is, delta 2 Is a parameter set according to actual requirements.
Alternatively, the fourth loss function may be determined by: a fourth loss function is determined based on the structural similarity (Structural Similarity, SSIM) of the output image of the generator to the corresponding sample normal light image.
For example, the fourth Loss function loss4=ssim (lgt, gθ (lin)).
Alternatively, the fifth loss function may be determined by: extracting a first feature of an output image of the generator through a style migration network; extracting second characteristics of the corresponding normal light images of the sample through a style migration network; a fifth loss function is determined based on the first feature and the second feature. The style migration network may be a VGG19 network.
For example, a fifth loss functionVGG (lgt) is a second feature of the sample normal light image, VGG (G theta (lin)) is a first feature of the output image of the generator.
In the embodiment of the application, when the generated countermeasure network is trained, in addition to the usual countermeasure loss, in order to obtain the high-level information of the image to improve the visual quality of the image, a fifth loss function is introduced, the false image generated by the generator and the GT are sent into the style migration network to perform feature extraction, and the extracted features are subjected to root mean square error as constraint conditions (namely the fifth loss function). Further, for the task of darkened image enhancement, the darkened areas in the image should get more attention, thus introducing a third loss function to balance the degree of enhancement between the darkened areas and other areas.
In step 303, performing iterative training on the generated countermeasure network for multiple times based on the training set, during each iterative training process, firstly fixing the discriminator, inputting the sample darkness image into the generator for processing, calculating a first loss value according to the sample normal light image corresponding to the sample darkness image, the output image of the generator and a first loss function, performing back propagation update on parameters in the generator according to the first loss value, then fixing the generator, inputting the output image of the generator and the corresponding sample normal light image into the discriminator for processing, calculating a second loss value according to the output result of the discriminator and a second loss function, and performing back propagation update on parameters in the discriminator according to the second loss value; and performing iterative training for a plurality of times until the generated countermeasure network converges to obtain a target generator.
In some embodiments, to enhance the training effect, the sample dim light image may be enhanced, and the enhanced sample data and GT data are simultaneously sent into the generating type countermeasure network, where the generator network and the discriminator network begin the countermeasure training. Accordingly, the step 303 may include the steps of: step 3031 and step 3032;
in step 3031, preprocessing is performed on the sample dim light image in the training set; wherein the pretreatment comprises at least one of the following treatments: adding random noise, image rotation and image clipping.
In step 3032, the generated countermeasure network is trained for a plurality of iterations by the sample normal light image and the corresponding preprocessed dim light image in the training set.
In one example, firstly, a training generator of the arbiter is fixed, and underexposed sample dim light images are taken as input of the generator in the generation type countermeasure network, in order to acquire content information of the underexposed sample dim light images as much as possible through data, a residual structure and a Unet network structure are introduced into the generator, and the generator continuously generates false samples, so that the false samples can be hoped to deceive the arbiter to obtain high scores. Wherein the loss function of the generator relates to the following loss function: since the study content is darkness image enhancement, darkness areas in the image should be given greater weight, so that an area loss function (namely a third loss function) is introduced to be unconstrained; the false sample and GT generated by the generator calculate an SSIM loss function (namely a fourth loss function) to enhance the detail information of the network output image; in order to obtain high-level information of the image to improve the visual quality of the image, the false image and the GT generated by the generator are input into a VGG19 network to perform feature extraction, and root mean square error is performed on the extracted features to serve as constraint conditions (namely, a fifth loss function).
Then, the fixed generator trains a discriminator, and takes the false samples generated by the generator and the true samples with good exposure as inputs of the discriminator, the discriminator needs to judge the false samples as far as possible to lower the false samples, wherein the loss function of the discriminator adopts the conventional countermeasure loss function of the generated countermeasure network.
Compared with the prior art, the embodiment of the application provides a method for enhancing the dim light image based on a generated type countermeasure network, wherein the generated type countermeasure network has two different networks instead of a single network, the training mode adopts a countermeasure training mode, and the two networks of a generator and a discriminator are continuously game to fully utilize a limited data set to achieve an optimal result. Compared with other methods, the generated countermeasure network can generate clearer and more real samples, can well remove noise in a dark area in a dark image to obtain a high-quality image, and has relatively less required data volume, and the generated countermeasure model only uses counter propagation in the training process without a complex Markov chain, so that the time complexity can be obviously reduced.
Fig. 4 is a schematic structural diagram of an image enhancement device according to an embodiment of the present application, and as shown in fig. 4, an image enhancement device 400 may include: an acquisition module 401 and an enhancement module 402;
an obtaining module 401, configured to obtain a first image, where the first image is a dark light image to be enhanced;
the enhancement module 402 is configured to input the first image into a target generator for processing, so as to obtain a second image, where the second image is a dark light enhanced image;
the target generator is trained based on a generator in a generative countermeasure network, the generative countermeasure network comprises the generator and a discriminator, the generator comprises a semantic segmentation network, and the semantic segmentation network is used for assisting in extracting dim light features of different layers in the first image.
As can be seen from the above embodiment, in this embodiment, a first image is acquired, where the first image is a dark image to be enhanced; inputting the first image into a target generator for processing to obtain a second image, wherein the second image is an image with enhanced dim light; the target generator is trained based on a generator in a generated countermeasure network, the generated countermeasure network comprises a generator and a discriminator, the generator comprises a semantic segmentation network, and the semantic segmentation network is used for assisting in extracting dim light features of different layers in the first image.
Therefore, in the embodiment of the application, the dim light image can be enhanced based on the target generator obtained by the generation type countermeasure network training, because the generation type countermeasure network has two different networks instead of a single network, and the training mode adopts the countermeasure training mode, the continuous game of the two networks of the generator and the discriminator can fully utilize the training data; in addition, the generator comprises the characteristics that the semantic segmentation network can fully mine the image, so that the target generator obtained through training can learn the relatively accurate priori dim light characteristics, and the enhancement effect is relatively good when the dim light image is enhanced based on the target generator.
Alternatively, as an embodiment, the semantic segmentation network may comprise a Unet network.
Alternatively, as an embodiment, the target generator may be trained by:
obtaining a training set, the training set comprising: a plurality of sample image pairs, each sample image pair comprising: a sample normal light image and a corresponding sample dim light image;
constructing the generated countermeasure network, a first loss function corresponding to a generator in the generated countermeasure network and a second loss function corresponding to a discriminator;
performing iterative training on the generated type countermeasure network for a plurality of times based on the training set, firstly fixing the discriminator in each iterative training process, inputting the sample dim light image into the generator for processing, calculating a first loss value according to the sample normal light image corresponding to the sample dim light image, the output image of the generator and the first loss function, performing back propagation update on parameters in the generator according to the first loss value, then fixing the generator, inputting the output image of the generator and the corresponding sample normal light image into the discriminator for processing, calculating a second loss value according to the output result of the discriminator and the second loss function, and performing back propagation update on the parameters in the discriminator according to the second loss value; and performing iterative training for a plurality of times until the generated type countermeasure network converges to obtain the target generator.
Alternatively, as an embodiment, the first loss function may be composed of at least one of the following loss functions: a third loss function, a fourth loss function, and a fifth loss function;
the third loss function is used for balancing the enhancement degree between a dim light area and other areas in the sample dim light image;
the fourth loss function is used for measuring the structural similarity between the output image of the generator and the corresponding normal light image of the sample;
the fifth loss function is used for measuring the style gap between the output image of the generator and the corresponding sample normal light image.
Alternatively, as an embodiment, the third loss function may be determined by:
acquiring brightness weight items of all pixel points in the sample dim light image;
calculating a weight image of the sample dim light image according to the brightness weight item of each pixel point, wherein the dim light weight value of a dim light area in the weight image is larger than that of other areas;
and determining the third loss function according to the weight image, the output image of the generator and the corresponding sample normal light image.
Alternatively, as an embodiment, the fifth loss function may be determined by:
extracting a first feature of the output image of the generator through a style migration network;
extracting second characteristics of the corresponding normal light images of the sample through the style migration network;
the fifth loss function is determined from the first feature and the second feature.
Optionally, as an embodiment, the performing multiple iterative training on the generated countermeasure network based on the training set may include:
preprocessing the sample dim light image in the training set;
performing repeated iterative training on the generated countermeasure network through the sample normal light image in the training set and the corresponding preprocessed dim light image;
wherein the pretreatment comprises at least one of the following treatments: adding random noise, image rotation and image clipping.
Any one step and specific operation in any one step in the embodiment of the image enhancement method provided by the application can be completed by corresponding modules in the image enhancement device. The procedure of the respective operations performed by the respective modules in the image enhancement apparatus refers to the procedure of the respective operations described in the embodiment of the image enhancement method.
For the device embodiments, since they are substantially similar to the method embodiments, the description is relatively simple, and reference is made to the description of the method embodiments for relevant points.
Fig. 5 is a block diagram of an electronic device including a processing component 522, which further includes one or more processors, and memory resources represented by memory 532, for storing instructions, such as applications, executable by the processing component 522, provided in an embodiment of the present application. The application programs stored in the memory 532 may include one or more modules each corresponding to a set of instructions. Further, the processing component 522 is configured to execute instructions to perform the methods described above.
The electronic device may also include a power component 526 configured to perform power management of the electronic device, a wired or wireless network interface 550 configured to connect the electronic device to a network, and an input output (I/O) interface 558. The electronic device may operate based on an operating system stored in memory 532, such as Windows Server, macOS XTM, unixTM, linuxTM, freeBSDTM, or the like.
According to yet another embodiment of the present application, there is also provided a computer-readable storage medium having stored thereon a computer program/instruction which, when executed by a processor, implements the steps of the image enhancement method according to any of the embodiments described above.
According to a further embodiment of the present application, there is also provided a computer program product comprising a computer program/instruction which, when executed by a processor, implements the steps of the image enhancement method according to any of the embodiments described above.
In this specification, each embodiment is described in a progressive manner, and each embodiment is mainly described by differences from other embodiments, and identical and similar parts between the embodiments are all enough to be referred to each other.
It will be apparent to those skilled in the art that embodiments of the present application may be provided as a method, apparatus, or computer program product. Accordingly, embodiments of the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, embodiments of the application may take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.
Embodiments of the present application are described with reference to flowchart illustrations and/or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
While preferred embodiments of the present application have been described, additional variations and modifications in those embodiments may occur to those skilled in the art once they learn of the basic inventive concepts. It is therefore intended that the following claims be interpreted as including the preferred embodiment and all such alterations and modifications as fall within the scope of the embodiments of the application.
Finally, it is further noted that relational terms such as first and second, and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or terminal that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or terminal. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article or terminal device comprising the element.
The foregoing has described in detail the image enhancement method, the electronic device and the storage medium provided by the present application, and specific examples have been applied to illustrate the principles and the embodiments of the present application, and the above examples are only used to help understand the method and the core idea of the present application; meanwhile, as those skilled in the art will have variations in the specific embodiments and application scope in accordance with the ideas of the present application, the present description should not be construed as limiting the present application in view of the above.

Claims (10)

1. A method of image enhancement, the method comprising:
acquiring a first image, wherein the first image is a dim light image to be enhanced;
inputting the first image into a target generator for processing to obtain a second image, wherein the second image is an image with enhanced dim light;
the target generator is trained based on a generator in a generative countermeasure network, the generative countermeasure network comprises the generator and a discriminator, the generator comprises a semantic segmentation network, and the semantic segmentation network is used for assisting in extracting dim light features of different layers in the first image.
2. The method of claim 1, wherein the semantic segmentation network comprises a Unet network.
3. The method according to claim 1 or 2, characterized in that the target generator is trained by:
obtaining a training set, the training set comprising: a plurality of sample image pairs, each sample image pair comprising: a sample normal light image and a corresponding sample dim light image;
constructing the generated countermeasure network, a first loss function corresponding to a generator in the generated countermeasure network and a second loss function corresponding to a discriminator;
performing iterative training on the generated type countermeasure network for a plurality of times based on the training set, firstly fixing the discriminator in each iterative training process, inputting the sample dim light image into the generator for processing, calculating a first loss value according to the sample normal light image corresponding to the sample dim light image, the output image of the generator and the first loss function, performing back propagation update on parameters in the generator according to the first loss value, then fixing the generator, inputting the output image of the generator and the corresponding sample normal light image into the discriminator for processing, calculating a second loss value according to the output result of the discriminator and the second loss function, and performing back propagation update on the parameters in the discriminator according to the second loss value; and performing iterative training for a plurality of times until the generated type countermeasure network converges to obtain the target generator.
4. A method according to claim 3, wherein the first loss function consists of at least one of the following loss functions: a third loss function, a fourth loss function, and a fifth loss function;
the third loss function is used for balancing the enhancement degree between a dim light area and other areas in the sample dim light image;
the fourth loss function is used for measuring the structural similarity between the output image of the generator and the corresponding normal light image of the sample;
the fifth loss function is used for measuring the style gap between the output image of the generator and the corresponding sample normal light image.
5. The method of claim 4, wherein the third loss function is determined by:
acquiring brightness weight items of all pixel points in the sample dim light image;
calculating a weight image of the sample dim light image according to the brightness weight item of each pixel point, wherein the dim light weight value of a dim light area in the weight image is larger than that of other areas;
and determining the third loss function according to the weight image, the output image of the generator and the corresponding sample normal light image.
6. The method of claim 4, wherein the fifth loss function is determined by:
extracting a first feature of the output image of the generator through a style migration network;
extracting second characteristics of the corresponding normal light images of the sample through the style migration network;
the fifth loss function is determined from the first feature and the second feature.
7. A method according to claim 3, wherein said training the generated countermeasure network for a plurality of iterations based on the training set comprises:
preprocessing the sample dim light image in the training set;
performing repeated iterative training on the generated countermeasure network through the sample normal light image in the training set and the corresponding preprocessed dim light image;
wherein the pretreatment comprises at least one of the following treatments: adding random noise, image rotation and image clipping.
8. An electronic device comprising a memory, a processor and a computer program stored on the memory, characterized in that the processor executes the computer program to implement the method of any one of claims 1-7.
9. A computer readable storage medium having stored thereon a computer program/instruction, which when executed by a processor, implements the method of any of claims 1-7.
10. A computer program product comprising computer programs/instructions which, when executed by a processor, implement the method of any of claims 1-7.
CN202310609286.2A 2023-05-26 2023-05-26 Image enhancement method, electronic device and storage medium Pending CN116777766A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202310609286.2A CN116777766A (en) 2023-05-26 2023-05-26 Image enhancement method, electronic device and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202310609286.2A CN116777766A (en) 2023-05-26 2023-05-26 Image enhancement method, electronic device and storage medium

Publications (1)

Publication Number Publication Date
CN116777766A true CN116777766A (en) 2023-09-19

Family

ID=88009062

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202310609286.2A Pending CN116777766A (en) 2023-05-26 2023-05-26 Image enhancement method, electronic device and storage medium

Country Status (1)

Country Link
CN (1) CN116777766A (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN118506008A (en) * 2024-07-15 2024-08-16 安徽高哲信息技术有限公司 Method, device, electronic equipment and medium for detecting thermally damaged corn particles

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN118506008A (en) * 2024-07-15 2024-08-16 安徽高哲信息技术有限公司 Method, device, electronic equipment and medium for detecting thermally damaged corn particles

Similar Documents

Publication Publication Date Title
CN111178183B (en) Face detection method and related device
CN110807757B (en) Image quality evaluation method and device based on artificial intelligence and computer equipment
CN116797504A (en) Image fusion method, electronic device and storage medium
CN114936377A (en) Model training and identity anonymization method, device, equipment and storage medium
CN112101456A (en) Attention feature map acquisition method and device and target detection method and device
CN116994021A (en) Image detection method, device, computer readable medium and electronic equipment
CN116777766A (en) Image enhancement method, electronic device and storage medium
CN117036770A (en) Detection model training and target detection method and system based on cascade attention
CN113673308B (en) Object identification method, device and electronic system
CN115620054A (en) Defect classification method and device, electronic equipment and storage medium
CN111626212B (en) Method and device for identifying object in picture, storage medium and electronic device
CN116797505A (en) Image fusion method, electronic device and storage medium
CN112906679B (en) Pedestrian re-identification method, system and related equipment based on human shape semantic segmentation
CN116777767A (en) Image processing method, electronic device and storage medium
CN115841605A (en) Target detection network training and target detection method, electronic device and storage medium
CN112464916B (en) Face recognition method and model training method thereof
Chen et al. An efficient and real-time emergency exit detection technology for the visually impaired people based on YOLOv5
CN114882079B (en) Image registration detection method, electronic device and storage medium
CN112329736A (en) Face recognition method and financial system
CN114332955B (en) Pedestrian re-identification method and device and computer readable storage medium
CN116957961A (en) Method for enhancing image contrast, electronic device and storage medium
CN114155157B (en) Image processing method, device, electronic device and storage medium
CN118470466B (en) Model processing method, device, equipment, medium and product
CN118379586B (en) Training method, device, equipment, medium and product of key point prediction model
CN116542292B (en) Training method, device, equipment and storage medium of image generation model

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination