[go: up one dir, main page]
More Web Proxy on the site http://driver.im/

CN109509156B - Image defogging processing method based on generation countermeasure model - Google Patents

Image defogging processing method based on generation countermeasure model Download PDF

Info

Publication number
CN109509156B
CN109509156B CN201811289748.2A CN201811289748A CN109509156B CN 109509156 B CN109509156 B CN 109509156B CN 201811289748 A CN201811289748 A CN 201811289748A CN 109509156 B CN109509156 B CN 109509156B
Authority
CN
China
Prior art keywords
image
fog
network
layer
free
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN201811289748.2A
Other languages
Chinese (zh)
Other versions
CN109509156A (en
Inventor
郑军
李俊
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Matrixtime Robotics Shanghai Co ltd
Original Assignee
Matrixtime Robotics Shanghai Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Matrixtime Robotics Shanghai Co ltd filed Critical Matrixtime Robotics Shanghai Co ltd
Priority to CN201811289748.2A priority Critical patent/CN109509156B/en
Publication of CN109509156A publication Critical patent/CN109509156A/en
Application granted granted Critical
Publication of CN109509156B publication Critical patent/CN109509156B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00Image enhancement or restoration
    • G06T5/73Deblurring; Sharpening
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06NCOMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00Computing arrangements based on biological models
    • G06N3/02Neural networks
    • G06N3/04Architecture, e.g. interconnection topology
    • G06N3/045Combinations of networks
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00Indexing scheme for image analysis or image enhancement
    • G06T2207/20Special algorithmic details
    • G06T2207/20081Training; Learning
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00Indexing scheme for image analysis or image enhancement
    • G06T2207/20Special algorithmic details
    • G06T2207/20084Artificial neural networks [ANN]

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • Molecular Biology (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • Artificial Intelligence (AREA)
  • Evolutionary Computation (AREA)
  • General Health & Medical Sciences (AREA)
  • Biomedical Technology (AREA)
  • Computing Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Health & Medical Sciences (AREA)
  • Image Analysis (AREA)

Abstract

The invention relates to an image defogging processing method based on a generation countermeasure model, which directly converts a foggy image into a fogless image through a trained generation countermeasure model, wherein the generation countermeasure model comprises a generation network and a discrimination network, the generation network is used for generating a fogless recovery image and comprises the following steps: the first-layer subnet is used for extracting a characteristic map; the second-layer sub-network comprises 5 convolutional layers which are sequentially connected, and the input end of the second convolutional layer is connected with the output end of the previous-layer convolutional layer and the output end of the first-layer sub-network and is used for fusing the global feature map extracted by the first-layer sub-network to obtain a fog-free recovery image; the judging network is used for judging whether the fog-free recovery image output by the generating network is a real fog-free image. Compared with the prior art, the invention has the advantages of good defogging effect, simple process and the like.

Description

Image defogging processing method based on generation countermeasure model
Technical Field
The invention relates to an image processing method, in particular to an image defogging processing method based on a generation countermeasure model.
Background
Fog and haze are common atmospheric phenomena in real life. Because a lot of atmospheric tiny particles with certain sizes exist in foggy weather, the atmospheric tiny particles absorb the reflected light of a target object/scene, and the reflected light of the atmospheric tiny particles and the reflected light of the target object are mixed together to enter a camera for imaging, so that the imaging definition is interfered by different programs. Due to the blurring and noise of imaging, inconvenience is brought to outdoor photographing application in foggy weather, and great difficulty is caused to various algorithms based on computer vision, such as target recognition/tracking, scene segmentation, automatic driving, and the like.
With the advancement of image processing technology, image defogging work has been greatly advanced in recent years. The existing image defogging algorithm can be mainly divided into two types: the first type is based on artificially defined image characteristics, clustering and information statistics are carried out, the transmissivity and the atmospheric light intensity in an atmospheric scattering model are estimated, and then a defogged image is obtained through the reverse solution of the scattering model; the method has a general restoring effect on the foggy image, and has the main limitation that the estimation accuracy and robustness of the transmissivity and the atmospheric light intensity are poor through artificially defined characteristics and a traditional image processing method, and image color distortion and large image noise often occur. The second type is that a deep convolution network model is adopted, training is carried out based on a large sample size, the network learns how to estimate the transmittance and the atmospheric light intensity, and then a fog-free image is solved through a model formula. The deep learning model has the advantages that the extraction method of the characteristic is not required to be artificially defined, the network can learn how to extract the required characteristic, and the defogging effect is higher in recovery accuracy and universality than that of the traditional image processing method. However, the current deep learning defogging method has the defect that the end-to-end image defogging cannot be well realized.
Disclosure of Invention
The invention aims to overcome the defects of the prior art and provide an image defogging processing method based on a generation countermeasure model.
The purpose of the invention can be realized by the following technical scheme:
an image defogging processing method based on a generation countermeasure model, which directly converts a foggy image into a fogless image through a trained generation countermeasure model, wherein the generation countermeasure model comprises a generation network and a discrimination network, the generation network is used for generating a fogless recovery image, and the method comprises the following steps:
the first-layer subnet is used for extracting a characteristic map;
the second-layer sub-network comprises 5 convolutional layers which are sequentially connected, and the input end of the second convolutional layer is connected with the output end of the previous-layer convolutional layer and the output end of the first-layer sub-network and is used for fusing the global feature map extracted by the first-layer sub-network to obtain a fog-free recovery image;
the judging network is used for judging whether the fog-free recovery image output by the generating network is a real fog-free image.
Further, the first-layer subnet includes 4 convolution modules, 4 pooling layers and two full-connection layers, each pooling layer is correspondingly connected behind one convolution module, and the two full-connection layers are connected behind the pooling layer.
Further, the convolution module of the first-layer subnet includes two connected convolution layers.
Furthermore, in the generative confrontation model, a ReLU nonlinear activation layer is arranged behind each convolution layer.
Furthermore, the discriminating network comprises 4 convolutional layers, 1 full-connection layer and 1 Sigmoid active layer
Further, the sample database adopted in the generation of the confrontation model training is generated by the following method:
the method comprises the steps of obtaining a fog-free image set, carrying out fog-adding processing on the fog-free image to generate fog images under different illumination intensities and fog concentrations, obtaining the fog image set, and forming a sample database by the fog-free image set and the corresponding fog image set.
Further, the fogging processing function adopted by the fogging processing is as follows:
G(I)=T(I)+a*(1-T(I))
where I is the original fog-free RGB image t (I) { t (p) | p is any pixel of the image I }, t (p) is the transmittance of the pixel p, a is the atmospheric light intensity, and g (I) is the generated fog image.
Further, the network parameters are updated by using an image recovery cost function during the generation countermeasure model training, where the image recovery cost function L is represented as:
L=E_GT+E_D
E_GT=|I’-I|
E_D=minmax(log(D(I)+log(1-D(I’))
wherein, E _ GT represents the difference cost obtained by comparing the fog-free restored image with the real fog-free image, I is the real fog-free image, I' is the fog-free restored image, E _ D represents the discrimination function for discriminating whether the fog-free restored image is the fog image or the fog-free image, and D is the discrimination network conversion.
Compared with the prior art, the invention has the following beneficial effects:
1) the invention adopts the generation countermeasure model to realize the end-to-end image defogging treatment, has the advantages that the judgment of the judgment network is carried out on the recovery image of the generation network, the estimation of intermediate parameters is not needed, and simultaneously, the good defogging effect can be obtained.
2) In the generated countermeasure model, a ReLU nonlinear activation layer is arranged behind each convolution layer, so that the whole network has the capability of simulating a high-order nonlinear function.
3) The method adopts the defined recovery cost function to train the countermeasure model until the final convergence is reached, so that the generated countermeasure model has higher precision.
Drawings
FIG. 1 is a schematic flow diagram of the present invention;
fig. 2 is a schematic diagram of a network structure for generating a countermeasure model according to the present invention.
Detailed Description
The invention is described in detail below with reference to the figures and specific embodiments. The present embodiment is implemented on the premise of the technical solution of the present invention, and a detailed implementation manner and a specific operation process are given, but the scope of the present invention is not limited to the following embodiments.
The invention realizes an image defogging processing method based on a generation countermeasure model, converts a foggy image into a fogless image through a trained generation countermeasure model, does not need to estimate intermediate parameters, and can obtain a good defogging effect.
As shown in fig. 1, the method comprises the following specific steps:
step S101, a sample database is obtained.
Firstly, a fog-free image set is obtained, fog adding processing is carried out on the fog-free image set at different concentrations based on an atmospheric scattering model, data enhancement is carried out, a corresponding fog data set is generated, and the fog-free image set and the fog image set form a sample database.
The fogging processing function adopted by the fogging processing is as follows:
G(I)=T(I)+a*(1-T(I))
wherein, I is an original RGB image without fog, f (I) represents a random processing function for the image, t (I) { t (p) | p is any pixel of the image I } is a propagation coefficient corresponding to the image, t (p) is a propagation coefficient of the pixel p, a is an atmospheric light intensity, and g (I) is a generated image with fog. t (p) is estimated according to the pixel depth value d (p), t (p) e ^ (beta ^ d (p)). beta and a are randomly taken from (0.8-1.2) and (0.6, 1).
In this embodiment, the NYU Depth V2 Dataset is obtained, and includes a fog-free clear RGB image and a Depth image registered therewith, which are used as a fog-free image set in a training sample. And carrying out fogging treatment on the fog-free image in the NYU Depth V2 Dataset to generate foggy images under different illumination intensities and fogging concentrations, and using the foggy images as a foggy image set in the training sample. In this embodiment, the fog-fog free image pair is cropped and scaled to a 128 x 128 image size.
Step S102, constructing and generating a confrontation model.
As shown in fig. 2, the network structure for generating the countermeasure model of the present invention includes two parts, one is a generation network and the other is a discrimination network, wherein the generation network is responsible for generating the fog-free image corresponding to the content of the input fog-free image, and the discrimination network is responsible for discriminating whether the fog-free image output by the generation network is a real fog-free image.
The generation network is divided into two layers, a first layer of sub-network comprises 4 convolution modules, 4 pooling layers and two full-connection layers, and the 4 pooling layers are respectively connected into the 4 convolution modules and then carry out maximum value down-sampling on the feature map to realize feature pooling; the second tier subnet contains 5 convolutional layers. The input of the first layer is 128-128 foggy images, the output is 32-32 feature maps, the feature dimension is 8, the feature maps are connected with the feature maps output by the first convolution layer of the second layer and input into the second convolution layer of the second layer together; the second layer input is a 128 x 128 foggy image and maintains this resolution all the time, outputting a restored 128 x 128 fogless image.
Each convolution module of the first layer of sub-network comprises 2 layers of convolution layers, the sizes of convolution kernels of the 2 layers of convolution layers are 3 x 3, and the number of convolution kernels contained in each layer of 5 modules is 128,256,256,512 respectively. Rearranging and converting one-dimensional vectors output by the fully connected layer into a feature map with the dimension of 32 x 8 and connecting the feature map to a second layer of subnets; the convolution kernels of the second tier subnets are each 5 x 64 in size.
In this embodiment, a ReLU nonlinear activation layer may also be added after each convolutional layer of the network is generated, so that the entire network has the capability of simulating a high-order nonlinear function.
The discrimination network comprises 4 convolution layers, 1 full-connection layer and 1 Sigmoid active layer, and a Leaky RuLU nonlinear active layer is arranged behind each convolution layer in the discrimination network, so that gradient return is more stable.
Step S103, training and generating a confrontation model.
And setting the learning rate and momentum parameters of the network, and training the generated confrontation model by utilizing the matchvnet until the network converges.
During training, calculating gradient by adopting an image recovery cost function, returning the gradient to update network parameters, wherein the image recovery cost function L comprises two items, one item is a difference cost E _ GT which is obtained by comparing with a real fog-free image, I is the real fog-free image, and I' is a network recovery image; a discrimination function E _ D ═ minmax (log (D (I)) + log (1-D (I'))) for the fog image and the fog-free image is used to discriminate whether the restored image is a fog image or a fog-free image, and D is the discrimination network transform.
And step S104, inputting a fog image, and directly obtaining a fog-free image by using the trained generation confrontation model.
The foregoing detailed description of the preferred embodiments of the invention has been presented. It should be understood that numerous modifications and variations could be devised by those skilled in the art in light of the present teachings without departing from the inventive concepts. Therefore, the technical solutions available to those skilled in the art through logic analysis, reasoning and limited experiments based on the prior art according to the concept of the present invention should be within the scope of protection defined by the claims.

Claims (5)

1. An image defogging processing method based on a generation countermeasure model is characterized in that the method directly converts a foggy image into a fogless image through a trained generation countermeasure model, wherein the generation countermeasure model comprises a generation network and a discrimination network, and the generation network is used for generating a fogless recovery image and comprises the following steps:
the first-layer subnet is used for extracting a characteristic map;
the second-layer sub-network comprises 5 convolutional layers which are sequentially connected, and the input end of the second convolutional layer is connected with the output end of the previous-layer convolutional layer and the output end of the first-layer sub-network and is used for fusing the global feature map extracted by the first-layer sub-network to obtain a fog-free recovery image;
the judging network is used for judging whether the fog-free recovery image output by the generating network is a real fog-free image or not;
the first layer of the subnet comprises 4 convolution modules, 4 pooling layers and two full-connection layers, wherein each pooling layer is correspondingly connected behind one convolution module, and the two full-connection layers are connected behind the pooling layers;
the discrimination network comprises 4 convolution layers, 1 full-connection layer and 1 Sigmoid activation layer;
when the generated confrontation model is trained, updating network parameters by adopting an image recovery cost function, wherein the image recovery cost function L is expressed as:
L=E_GT+E_D
E_GT=|I’-I|
E_D=minmax(log(D(I)+log(1-D(I’))
wherein, E _ GT represents the difference cost obtained by comparing the fog-free restored image with the real fog-free image, I is the real fog-free image, I' is the fog-free restored image, E _ D represents the discrimination function for discriminating whether the fog-free restored image is the fog image or the fog-free image, and D is the discrimination network conversion.
2. The generative confrontation model-based image defogging method according to claim 1, wherein the convolution module of the first layer subnet comprises two connected layers of convolution.
3. The image defogging method according to claim 1, wherein a ReLU nonlinear activation layer is arranged after each convolution layer in the generative countermeasure model.
4. The image defogging processing method according to claim 1, wherein the sample database adopted in the training of the generative confrontation model is generated by:
the method comprises the steps of obtaining a fog-free image set, carrying out fog-adding processing on the fog-free image to generate fog images under different illumination intensities and fog concentrations, obtaining the fog image set, and forming a sample database by the fog-free image set and the corresponding fog image set.
5. The image defogging method according to claim 4, wherein the fog processing adopts a fog processing function as follows:
G(I)=T(I)+a*(1-T(I))
where I is the original fog-free RGB image t (I) { t (p) | p is any pixel of the image I }, t (p) is the transmittance of the pixel p, a is the atmospheric light intensity, and g (I) is the generated fog image.
CN201811289748.2A 2018-10-31 2018-10-31 Image defogging processing method based on generation countermeasure model Active CN109509156B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201811289748.2A CN109509156B (en) 2018-10-31 2018-10-31 Image defogging processing method based on generation countermeasure model

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201811289748.2A CN109509156B (en) 2018-10-31 2018-10-31 Image defogging processing method based on generation countermeasure model

Publications (2)

Publication Number Publication Date
CN109509156A CN109509156A (en) 2019-03-22
CN109509156B true CN109509156B (en) 2021-02-05

Family

ID=65747290

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201811289748.2A Active CN109509156B (en) 2018-10-31 2018-10-31 Image defogging processing method based on generation countermeasure model

Country Status (1)

Country Link
CN (1) CN109509156B (en)

Families Citing this family (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110136075B (en) * 2019-04-18 2021-01-05 中国地质大学(武汉) Remote sensing image defogging method for generating countermeasure network based on edge sharpening cycle
CN110378848B (en) * 2019-07-08 2021-04-20 中南大学 Image defogging method based on derivative map fusion strategy
CN111639542A (en) * 2020-05-06 2020-09-08 中移雄安信息通信科技有限公司 License plate recognition method, device, equipment and medium
CN113706395A (en) * 2020-05-21 2021-11-26 无锡科美达医疗科技有限公司 Image defogging method based on antagonistic neural network
CN111882495B (en) * 2020-07-05 2022-08-02 东北林业大学 Image highlight processing method based on user-defined fuzzy logic and GAN
CN112150379A (en) * 2020-09-22 2020-12-29 武汉工程大学 Image defogging method and device for enhancing generation of countermeasure network based on perception discrimination
CN113610787B (en) * 2021-07-27 2024-11-08 广东省科技基础条件平台中心 Training method and device for image defect detection model and computer equipment
CN117575946A (en) * 2023-10-19 2024-02-20 南京诺源医疗器械有限公司 Image processing method, apparatus, electronic device, and computer-readable storage medium

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106127702A (en) * 2016-06-17 2016-11-16 兰州理工大学 A kind of image mist elimination algorithm based on degree of depth study
CN106910175A (en) * 2017-02-28 2017-06-30 武汉大学 A kind of single image defogging algorithm based on deep learning
CN107798669A (en) * 2017-12-08 2018-03-13 北京小米移动软件有限公司 Image defogging method, device and computer-readable recording medium
CN108615226A (en) * 2018-04-18 2018-10-02 南京信息工程大学 A kind of image defogging method fighting network based on production
CN108665432A (en) * 2018-05-18 2018-10-16 百年金海科技有限公司 A kind of single image to the fog method based on generation confrontation network

Family Cites Families (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11080591B2 (en) * 2016-09-06 2021-08-03 Deepmind Technologies Limited Processing sequences using convolutional neural networks
US10510146B2 (en) * 2016-10-06 2019-12-17 Qualcomm Incorporated Neural network for image processing
CN107909621A (en) * 2017-11-16 2018-04-13 深圳市唯特视科技有限公司 It is a kind of based on it is twin into confrontation network medical image synthetic method
CN108171762B (en) * 2017-12-27 2021-10-12 河海大学常州校区 Deep learning compressed sensing same-class image rapid reconstruction system and method
CN108596026B (en) * 2018-03-16 2020-06-30 中国科学院自动化研究所 Cross-view gait recognition device and training method based on double-flow generation countermeasure network

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106127702A (en) * 2016-06-17 2016-11-16 兰州理工大学 A kind of image mist elimination algorithm based on degree of depth study
CN106910175A (en) * 2017-02-28 2017-06-30 武汉大学 A kind of single image defogging algorithm based on deep learning
CN107798669A (en) * 2017-12-08 2018-03-13 北京小米移动软件有限公司 Image defogging method, device and computer-readable recording medium
CN108615226A (en) * 2018-04-18 2018-10-02 南京信息工程大学 A kind of image defogging method fighting network based on production
CN108665432A (en) * 2018-05-18 2018-10-16 百年金海科技有限公司 A kind of single image to the fog method based on generation confrontation network

Also Published As

Publication number Publication date
CN109509156A (en) 2019-03-22

Similar Documents

Publication Publication Date Title
CN109509156B (en) Image defogging processing method based on generation countermeasure model
CN109584248B (en) Infrared target instance segmentation method based on feature fusion and dense connection network
CN109410144B (en) End-to-end image defogging processing method based on deep learning
CN106910175B (en) Single image defogging algorithm based on deep learning
US20180231871A1 (en) Depth estimation method for monocular image based on multi-scale CNN and continuous CRF
CN112102182B (en) Single image reflection removing method based on deep learning
CN110379020B (en) Laser point cloud coloring method and device based on generation countermeasure network
CN116229461A (en) Indoor scene image real-time semantic segmentation method based on multi-scale refinement
WO2023212997A1 (en) Knowledge distillation based neural network training method, device, and storage medium
CN109919073B (en) Pedestrian re-identification method with illumination robustness
CN115861380B (en) Method and device for tracking visual target of end-to-end unmanned aerial vehicle under foggy low-illumination scene
US11361534B2 (en) Method for glass detection in real scenes
CN111768415A (en) Image instance segmentation method without quantization pooling
CN108447060B (en) Foreground and background separation method based on RGB-D image and foreground and background separation device thereof
CN111582074A (en) Monitoring video leaf occlusion detection method based on scene depth information perception
CN116310098A (en) Multi-view three-dimensional reconstruction method based on attention mechanism and variable convolution depth network
CN109657538B (en) Scene segmentation method and system based on context information guidance
CN112164010A (en) Multi-scale fusion convolution neural network image defogging method
CN111008979A (en) Robust night image semantic segmentation method
Yang et al. [Retracted] A Method of Image Semantic Segmentation Based on PSPNet
CN116310095A (en) Multi-view three-dimensional reconstruction method based on deep learning
CN109859222A (en) Edge extracting method and system based on cascade neural network
CN110751271B (en) Image traceability feature characterization method based on deep neural network
Babu et al. An efficient image dahazing using Googlenet based convolution neural networks
CN110738624B (en) Area-adaptive image defogging system and method

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant