CN112215788A - Multi-focus image fusion algorithm based on improved generation countermeasure network - Google Patents
Multi-focus image fusion algorithm based on improved generation countermeasure network Download PDFInfo
- Publication number
- CN112215788A CN112215788A CN202010966366.XA CN202010966366A CN112215788A CN 112215788 A CN112215788 A CN 112215788A CN 202010966366 A CN202010966366 A CN 202010966366A CN 112215788 A CN112215788 A CN 112215788A
- Authority
- CN
- China
- Prior art keywords
- image
- network
- generator
- discriminator
- focus
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 230000004927 fusion Effects 0.000 title claims abstract description 29
- 238000011176 pooling Methods 0.000 claims abstract description 6
- 238000012549 training Methods 0.000 claims description 19
- 238000000034 method Methods 0.000 claims description 8
- 238000013527 convolutional neural network Methods 0.000 claims description 6
- ORILYTVJVMAKLC-UHFFFAOYSA-N Adamantane Natural products C1C(C2)CC3CC1CC2C3 ORILYTVJVMAKLC-UHFFFAOYSA-N 0.000 claims description 3
- 238000002474 experimental method Methods 0.000 claims description 3
- 238000005457 optimization Methods 0.000 claims description 3
- 230000005540 biological transmission Effects 0.000 abstract description 3
- 238000010586 diagram Methods 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 238000013461 design Methods 0.000 description 2
- 230000003044 adaptive effect Effects 0.000 description 1
- 230000003042 antagnostic effect Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000003745 diagnosis Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20212—Image combination
- G06T2207/20221—Image fusion; Image merging
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- General Engineering & Computer Science (AREA)
- General Health & Medical Sciences (AREA)
- Software Systems (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- Biophysics (AREA)
- Biomedical Technology (AREA)
- Mathematical Physics (AREA)
- Computational Linguistics (AREA)
- Health & Medical Sciences (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Evolutionary Biology (AREA)
- Image Analysis (AREA)
Abstract
The invention discloses a multi-focus image fusion algorithm based on an improved generation countermeasure network, which is applied to target images extracted by focusing photographing at different positions in the same scene. Firstly, designing a generator network and a discriminator network, cutting down a pooling layer in a network structure in order to avoid information loss caused by an image in a network model transmission process, and extracting image characteristics through convolution stacking; secondly, constructing a loss function for generating a countermeasure network, and optimizing network parameters to obtain an optimal network model; finally, inputting the acquired target image into a trained model to obtain a fused image; when the multi-focus image fusion algorithm is carried out, a generator in the generation countermeasure network generates a fusion image, the generated image and a source image are input into a discriminator, and if the discriminator cannot discriminate, the generated image is the best fusion image.
Description
Technical Field
The invention relates to the technical field of image processing, in particular to a multi-focus image fusion algorithm based on an improved generation countermeasure network.
Background
With the rapid development of technologies such as computers, sensors and the like, the device brings convenience to the life of people to a great extent. Among them, digital images, which are products of these new technologies, have also slowly permeated into the lives of people, and have also played an important role in communication between people. As the amount of image information obtained by people increases, it is important to process the image information. Because the focal length of the optical lens is set in a certain range, only objects within the depth of field can be clearly displayed in a picture, other objects may present a blurry state, and a common technique for acquiring a full-focus image is to fuse a plurality of images of the same scene taken under different focal length settings, i.e., a multi-focus image fusion technique. The multi-focus image fusion technology can fuse focus images under different focal lengths, and the fused images can retain the detail characteristics of source images to the maximum extent, so that richer information is provided for practical application fields such as military detection, medical diagnosis and target recognition.
Disclosure of Invention
The present invention is directed to solving, at least to some extent, one of the technical problems in the related art. Therefore, an object of the present invention is to provide a multi-focus image algorithm based on an improved generation countermeasure network, which processes target images extracted from different positions of focused shots in the same scene to obtain a fused image containing rich detail information.
1. According to the embodiment of the invention, the multi-focus image fusion algorithm based on the improved generation countermeasure network is applied to target images extracted by focusing photographing at different positions in the same scene, and comprises the following steps:
s1: designing a network structure for generating a generator and a discriminator in a countermeasure network, cutting down a pooling layer in the network structure, and extracting image features by using convolution stacking;
s2: constructing an objective function of a network model by generating a network structure of a countermeasure network;
s3: training through a training set to obtain an optimal generative confrontation network model;
s4: applying the generator model of the countermeasure network generated in step S3, inputting the source image into the generator to obtain the generated image, and performing target update for the discrimination of the generated image and the source image by the discriminator. Preferably, the generator in step S1 is a 5-layer convolutional neural network, the first and second layers use convolution kernels of 5x5, the third and fourth layers use convolution kernels of 3x3, and the last layer uses convolution kernels of 1x1, the step size of each convolution kernel is set to 1, and the input of the generator is formed by connecting two multi-focus images, that is, the input channel is 2
The discriminator in step S1 is an 8-layer convolutional neural network, each layer uses convolution kernels of 3 × 3, the convolution kernel step size of the second, third, and fourth layers is set to 2, and the convolution kernel step size of the remaining layers is 1.
Preferably, the objective function of the created generation confrontation network model in step S2 includes an objective function of the generator network and an objective function of the discriminator network
LGAN＝{min(LG),min(LD)}；
The loss function of the generator comprises two parts, wherein one part is the loss resistance of the generator and the discriminator and is represented by V, and the other part is the content loss of the image detail information in the generation process and is represented by LcontentRepresents:
then L isGCan be expressed as
LG＝V+αLcontent
L for loss function of discriminatorDRepresents:
preferably, in step S3, an optimal generative confrontation network model is obtained through training of a training set, 50 pairs of multi-focus images are used as the training set of the experiment, each pair of multi-focus images is divided into sub-blocks by a sliding window with a step size of 14 and a size of 64x64, the sub-blocks are expanded to a size of 76x76 in a filling manner and are used as the input of the generator, the size of the fused image output by the generator is still 64x64, and the generated fused image is used as the input of the discriminator and the Adam optimization algorithm is used until the maximum training times are reached.
Preferably, in step S4, the fused image is obtained by the generator, the fused image is updated by the discriminator, and the two input source images I1、I2By means of a generator G, a fusion image I is obtainedfThe discriminator D is used for fusing the images IfSource image I1、I2The extracted image characteristics are judged, and a fused image I is judgedfWhether or not to include the source image I1、I2If the discriminator can discriminate, the fused image I is continuously updatedf(ii) a If the discriminator cannot discriminate, it indicates that the image generated by the generator is the best fused image.
The invention provides a multi-focus image fusion algorithm based on an improved generation countermeasure network, which realizes the extraction of image information at different focus positions by utilizing the generation countermeasure network and generates a fusion image containing rich detail information. Firstly, designing a generator network and a discriminator network, cutting down a pooling layer in a network structure in order to avoid information loss caused by an image in a network model transmission process, and extracting image characteristics through convolution stacking. Secondly, constructing a loss function for generating the countermeasure network, and optimizing network parameters to obtain an optimal network model. And finally, inputting the acquired target image into the trained model to obtain a fused image.
Drawings
The accompanying drawings, which are included to provide a further understanding of the invention and are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and together with the description serve to explain the principles of the invention and not to limit the invention. In the drawings:
FIG. 1 is a block diagram of a method of a multi-focus image fusion algorithm based on an improved generation countermeasure network according to the present invention;
FIG. 2 is a diagram of a generator network structure in a multi-focus image fusion algorithm based on an improved generation countermeasure network according to the present invention;
FIG. 3 is a diagram of a network structure of a discriminator in a multi-focus image fusion algorithm based on an improved generation countermeasure network according to the present invention;
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments.
Examples of which are illustrated in the accompanying drawings, wherein like reference numerals refer to the same or similar elements or elements having the same or similar function throughout. The embodiments described below with reference to the drawings are illustrative and intended to be illustrative of the invention and are not to be construed as limiting the invention.
Example 1:
as shown in fig. 1-3, an image fusion algorithm based on an improved generation of a countermeasure network. And designing a generator network and a discriminator network, cutting down a pooling layer in a network structure in order to avoid information loss caused by the image in the network model transmission process, and extracting image characteristics through convolution stacking. Secondly, constructing a loss function for generating the countermeasure network, and optimizing network parameters to obtain an optimal network model. And finally, inputting the acquired target image into the trained model to obtain a fused image.
The image fusion algorithm based on the improved generation countermeasure network specifically comprises the following steps:
s1: designing a network structure for generating a generator and a discriminator in a countermeasure network, cutting down a pooling layer in the network structure, and extracting image features by using convolution stacking;
s2: constructing an objective function of a network model by generating a network structure of a countermeasure network;
s3: training through a training set to obtain an optimal generative confrontation network model;
s4: applying the generator model of the countermeasure network generated in step S3, inputting the source image into the generator to obtain the generated image, and performing target update for the discrimination of the generated image and the source image by the discriminator.
Example 2:
as shown in fig. 1 to 3, according to the steps of embodiment 1, a network structure for generating generators and discriminators in a countermeasure network is designed in S1. The generator aims to extract more detail information in the source image and generate a fused image with abundant details. The generator is a 5-layer convolutional neural network, with the first and second layers using convolution kernels of 5x5, the third and fourth layers using convolution kernels of 3x3, and the last layer using convolution kernels of 1x 1. The step size of each layer of convolution kernel is set to be 1, and the input of the generator is formed by connecting two multi-focus images, namely the input channel is 2. The purpose of the discriminator is to discriminate whether the target image is the image generated by the generator or the real image, and to classify the target image by extracting the features of the target image. The discriminator is a convolutional neural network with 8 layers, each layer uses convolution kernel of 3x3, the convolution kernel step of the second, third and fourth layers is set to be 2, and the convolution kernel step of the other layers is 1.
Example 3:
as shown in FIGS. 1 to 3, according to the step of embodiment 1, the objective function for generating the countermeasure network model built in S2 includes the objective function of the generator network and the objective function of the discriminator network
LGAN＝{min(LG),min(LD)}；
The generator loss function includes two parts, one part is that the antagonistic loss of the generator and the arbiter is represented by V. The other part is L for content loss in the process of generating image detail informationcontentRepresents:
then L isGCan be expressed as
LG＝V+αLcontent
In order to generate a better fused image, a discriminator is introduced. L for loss function of discriminatorDRepresents:
example 4:
as shown in fig. 1 to 3, according to the step of embodiment 1, the optimal generative confrontation network model is obtained by training through the training set in S3. 50 pairs of multi-focus images were used as a training set for the experiment. To enable a better training model, each pair of multi-focused images was divided into sub-blocks with a sliding window size of 64x64 with a step size of 14, and these sub-blocks were expanded in a padding fashion to a size of 76x76, which was used as input to the generator. The fused image output by the generator is still 64x64 in size. The resulting fused image is used as input to a discriminator and the Adam optimization algorithm is used until the maximum number of training passes is reached.
Example 5:
as shown in fig. 1 to 3, according to the step of embodiment 1, the fused image is obtained by the generator in S4, and the fused image is updated by the discriminator. Inputting two source images I1、I2By means of a generator G, a fusion image I is obtainedfThe discriminator D is used for fusing the images IfSource image I1、I2The extracted image characteristics are judged, and a fused image I is judgedfWhether or not to include the source image I1、I2If the discriminator can discriminate, the fused image I is continuously updatedf(ii) a If the discriminator cannot discriminate, it indicates that the image generated by the generator is the best fused image.
In summary, the multi-focus image fusion algorithm based on the improved generation countermeasure network realizes end-to-end adaptive fusion and avoids the complicated fusion rule of design. And (3) extracting image features by adopting convolution stacking through the design of the generator network and the discriminator network. Secondly, constructing a loss function for generating the countermeasure network, and optimizing network parameters to obtain an optimal network model. And finally, inputting the acquired target image into the trained model to obtain a fused image. The algorithm can better extract the detail information and the edge characteristics of the two source images, and achieves better fusion effect.
In the description herein, references to the description of the term "one embodiment," "some embodiments," "an example," "a specific example," or "some examples," etc., mean that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the invention. In this specification, the schematic representations of the terms used above do not necessarily refer to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.
The above description is only for the preferred embodiment of the present invention, but the scope of the present invention is not limited thereto, and any person skilled in the art should be considered to be within the technical scope of the present invention, and the technical solutions and the inventive concepts thereof according to the present invention should be equivalent or changed within the scope of the present invention.
Claims (5)
1. A multi-focus image fusion algorithm based on an improved generation countermeasure network is applied to target images extracted by focusing photographing at different positions in the same scene, and is characterized by comprising the following steps:
s1: designing a network structure for generating a generator and a discriminator in a countermeasure network, cutting down a pooling layer in the network structure, and extracting image features by using convolution stacking;
s2: constructing an objective function of a network model by generating a network structure of a countermeasure network;
s3: training through a training set to obtain an optimal generative confrontation network model;
s4: applying the generator model of the countermeasure network generated in step S3, inputting the source image into the generator to obtain the generated image, and performing target update for the discrimination of the generated image and the source image by the discriminator.
2. The multi-focus image fusion algorithm based on the improved generation countermeasure network of claim 1, wherein: the generator in step S1 is a 5-layer convolutional neural network, the first layer and the second layer both use convolution kernels of 5x5, the third layer and the fourth layer use convolution kernels of 3x3 and the last layer uses convolution kernels of 1x1, the step size of each layer of convolution kernels is set to 1, and the input of the generator is formed by connecting two multi-focus images, that is, the input channel is 2;
the discriminator in step S1 is an 8-layer convolutional neural network, each layer uses convolution kernels of 3 × 3, the convolution kernel step size of the second, third, and fourth layers is set to 2, and the convolution kernel step size of the remaining layers is 1.
3. The multi-focus image fusion algorithm based on the improved generation countermeasure network of claim 1, wherein: the objective function of the created generation countermeasure network model in step S2 includes the objective function of the generator network and the objective function of the discriminator network
LGAN＝{min(LG),min(LD)}；
The loss function of the generator comprises two parts, wherein one part is the loss resistance of the generator and the discriminator and is represented by V, and the other part is the content loss of the image detail information in the generation process and is represented by LcontentRepresents:
then L isGCan be expressed as
LG＝V+αLcontent
L for loss function of discriminatorDRepresents:
4. the multi-focus image fusion algorithm based on the improved generation countermeasure network of claim 1, wherein: in step S3, an optimal generative confrontation network model is obtained through training of a training set, 50 pairs of multi-focus images are used as the training set of an experiment, each pair of multi-focus images is divided into sub-blocks by a sliding window with a step size of 14 and a size of 64x64, the sub-blocks are expanded to 76x76 in a filling manner and are used as the input of a generator, the size of a fused image output by the generator is still 64x64, the generated fused image is used as the input of a discriminator, and an Adam optimization algorithm is used until the maximum training times are reached.
5. The multi-focus image fusion algorithm based on the improved generation countermeasure network of claim 1, wherein: in step S4, the fused image is obtained through the generator, the fused image is updated through the discriminator, and the two input source images I1、I2By means of a generator G, a fusion image I is obtainedfThe discriminator D is used for fusing the images IfSource image I1、I2The extracted image characteristics are judged, and a fused image I is judgedfWhether or not to include the source image I1、I2If the discriminator can discriminate, the fused image I is continuously updatedf(ii) a If the discriminator cannot discriminate, it indicates that the image generated by the generator is the best fused image.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010966366.XA CN112215788A (en) | 2020-09-15 | 2020-09-15 | Multi-focus image fusion algorithm based on improved generation countermeasure network |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010966366.XA CN112215788A (en) | 2020-09-15 | 2020-09-15 | Multi-focus image fusion algorithm based on improved generation countermeasure network |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112215788A true CN112215788A (en) | 2021-01-12 |
Family
ID=74049550
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010966366.XA Pending CN112215788A (en) | 2020-09-15 | 2020-09-15 | Multi-focus image fusion algorithm based on improved generation countermeasure network |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112215788A (en) |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113112439A (en) * | 2021-04-14 | 2021-07-13 | 展讯半导体(南京)有限公司 | Image fusion method, training method, device and equipment of image fusion model |
| CN113610732A (en) * | 2021-08-10 | 2021-11-05 | 大连理工大学 | Full-focus image generation method based on interactive counterstudy |
| CN113723470A (en) * | 2021-08-09 | 2021-11-30 | 北京工业大学 | Pollen image synthesis method and device fusing multilayer information and electronic equipment |
| CN116597268A (en) * | 2023-07-17 | 2023-08-15 | 中国海洋大学 | Efficient multi-focus image fusion method and model building method thereof |
| CN114782297B (en) * | 2022-04-15 | 2023-12-26 | 电子科技大学 | Image fusion method based on motion-friendly multi-focus fusion network |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109325931A (en) * | 2018-08-22 | 2019-02-12 | 中北大学 | Based on the multi-modality images fusion method for generating confrontation network and super-resolution network |
-
2020
- 2020-09-15 CN CN202010966366.XA patent/CN112215788A/en active Pending
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109325931A (en) * | 2018-08-22 | 2019-02-12 | 中北大学 | Based on the multi-modality images fusion method for generating confrontation network and super-resolution network |
Non-Patent Citations (1)
| Title |
|---|
| 王娟 等: "基于改进生成对抗网络的多聚焦图像融合", 《科学技术与工程》 * |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113112439A (en) * | 2021-04-14 | 2021-07-13 | 展讯半导体(南京)有限公司 | Image fusion method, training method, device and equipment of image fusion model |
| CN113723470A (en) * | 2021-08-09 | 2021-11-30 | 北京工业大学 | Pollen image synthesis method and device fusing multilayer information and electronic equipment |
| CN113723470B (en) * | 2021-08-09 | 2024-08-27 | 北京工业大学 | Pollen image synthesis method and device integrating multilayer information and electronic equipment |
| CN113610732A (en) * | 2021-08-10 | 2021-11-05 | 大连理工大学 | Full-focus image generation method based on interactive counterstudy |
| CN113610732B (en) * | 2021-08-10 | 2024-02-09 | 大连理工大学 | Full-focus image generation method based on interactive countermeasure learning |
| CN114782297B (en) * | 2022-04-15 | 2023-12-26 | 电子科技大学 | Image fusion method based on motion-friendly multi-focus fusion network |
| CN116597268A (en) * | 2023-07-17 | 2023-08-15 | 中国海洋大学 | Efficient multi-focus image fusion method and model building method thereof |
| CN116597268B (en) * | 2023-07-17 | 2023-09-22 | 中国海洋大学 | Efficient multi-focus image fusion method and model building method thereof |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112215788A (en) | Multi-focus image fusion algorithm based on improved generation countermeasure network | |
| Wang et al. | Selective light field refocusing for camera arrays using bokeh rendering and superresolution | |
| US10645368B1 (en) | Method and apparatus for estimating depth of field information | |
| CN110555434B (en) | Method for detecting visual saliency of three-dimensional image through local contrast and global guidance | |
| EP3236391B1 (en) | Object detection and recognition under out of focus conditions | |
| CN107122796B (en) | A kind of remote sensing image classification method based on multiple-limb network integration model | |
| Ma et al. | Defocus image deblurring network with defocus map estimation as auxiliary task | |
| JP6011862B2 (en) | 3D image capturing apparatus and 3D image capturing method | |
| Jung et al. | Active confocal imaging for visual prostheses | |
| Li et al. | End-to-end learning of deep convolutional neural network for 3D human action recognition | |
| WO2019019086A1 (en) | Image contrast enhancement method and device, and storage medium | |
| CN108805889A (en) | The fining conspicuousness method for segmenting objects of margin guide and system, equipment | |
| CN111612711A (en) | Improved picture deblurring method based on generation countermeasure network | |
| Anwar et al. | Deblur and deep depth from single defocus image | |
| CN111047543A (en) | Image enhancement method, device and storage medium | |
| CN114187221A (en) | Infrared and visible light image fusion method based on adaptive weight learning | |
| Vadathya et al. | Learning light field reconstruction from a single coded image | |
| CN112241940B (en) | Fusion method and device for multiple multi-focus images | |
| CN113112439B (en) | Image fusion method, training method, device and equipment of image fusion model | |
| CN115965844B (en) | Multi-focus image fusion method based on visual saliency priori knowledge | |
| Li et al. | No-reference stereoscopic image quality assessment based on local to global feature regression | |
| Cherian et al. | Image Augmentation Using Hybrid RANSAC Algorithm | |
| Czúni et al. | Depth-based indexing and retrieval of photographic images | |
| Frechette et al. | Gradient based multifocus video image fusion | |
| Xu et al. | Multi-focus image fusing based on non-negative matrix factorization |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20210112 |
|
| WD01 | Invention patent application deemed withdrawn after publication |