JP7455574B2 - Image processing method, program, learned model manufacturing method, image processing device, and image processing system - Google Patents

Description

The present invention relates to an image processing method capable of controlling fluctuations in noise level of an image accompanying super-resolution image processing in machine learning DL (Deep Learning).

Non-Patent Document 1 discloses a method of performing super resolution image processing using machine learning DL to obtain a high resolution image from a low resolution captured image.

Xintao Wang, Ke Yu, Shixiang Wu, Jinjin Gu, Yihao Liu, Chao Dong, Chen Change Loy, Yu Qiao, Xiaoou Tang, “ESRGAN: Enhanced Super-R eoslution Generative Adversarial Networks”, arXiv:1809.00219, 2018

However, in the method disclosed in Non-Patent Document 1, the noise level of the image fluctuates (increases or decreases) as the resolution increases. This is due to the high-resolution training images and low-resolution training images that are training images used for learning in the machine learning DL. That is, when a high-resolution training image is downsampled to generate a low-resolution training image, noise is averaged accordingly, and the noise level of the low-resolution training image is reduced. Here, noise is noise included in high-resolution training images.

By the way, conventionally, after downsampling a high-resolution training image to generate a low-resolution training image, noise is added to the obtained low-resolution training image. However, if there is no correlation between the noise included in the high-resolution training image and the noise added to the low-resolution training image, the noise level of the high-resolution image obtained with DL super-resolution image processing will fluctuate.

SUMMARY OF THE INVENTION Therefore, an object of the present invention is to provide an image processing method and the like that can control fluctuations in image noise accompanying image processing.

An image processing method according to an aspect of the present invention is an image processing method executed using a computer, and the method includes processing a second image having a different noise level from the first image based on a first image. a first step of generating, and a second step of generating a third image regarding noise contained in the first image or the second image, based on the first image and the second image. and a third step of generating a fourth image by downsampling at least one of the first image and the second image, and weighting the third image and the fourth image. a fourth step of generating a fifth image by adding; and a fifth step of performing neural network learning using the fifth image and the first image or the second image. and has.

Other objects and features of the invention are illustrated in the following examples.

According to the present invention, it is possible to provide an image processing method and the like that can control fluctuations in image noise accompanying image processing.

3 is a diagram showing the flow of learning of a neural network in Example 1. FIG. 1 is a block diagram of an image processing system in Example 1. FIG. 1 is an external view of an image processing system in Example 1. FIG. 7 is a flowchart regarding weight learning in Example 1. FIG. 5 is a flowchart regarding generation of an output image in Example 1. FIG. 3 is a block diagram of an image processing system in Example 2. FIG. 3 is an external view of an image processing system in Example 2. FIG. 3 is a block diagram of an image processing system in Example 3. FIG. 7 is a flowchart regarding generation of an output image in Example 3. It is an explanatory diagram of the outline of each example.

The following describes in detail an embodiment of the present invention with reference to the drawings. In each drawing, the same components are given the same reference symbols, and duplicate descriptions are omitted.

First, before giving a specific explanation, the gist of each embodiment will be explained. Each embodiment suppresses fluctuations in the noise level of an image due to DL super-resolution image processing (increasing the resolution of an image). Note that machine learning DL uses a multilayer neural network for image processing. Furthermore, in DL super-resolution image processing, a high-resolution training image and a corresponding low-resolution training image (patches that are partial images extracted therefrom) are used in learning weights used in a multilayer neural network. Here, the weights include a filter used for convolution, a bias for addition, and the like.

Next, an overview of each embodiment will be described with reference to FIG. In each example, first, noise is removed (reduced) from a first image, which is a high-resolution training image, to generate a second image having a different noise level from the first image. Note that the method for generating a second image with a different noise level from the first image is not limited to removing (reducing) noise. Other methods will be described later. Next, at least one of the first image and the second image is downsampled to generate a fourth image. Here, the fourth image is a low-resolution image corresponding to the high-resolution training image, and the noise level has decreased due to downsampling.

Next, the first image and the second image are downsampled and then subtracted to generate a third image. More specifically, the first image and the second image are downsampled pixel by pixel and then subtracted, or the first image and the second image are respectively downsampled and then subtracted pixel by pixel. By doing so, a third image is generated. Here, the third image is a low-resolution noise image obtained by downsampling the noise included in the first image, which is a high-resolution training image. That is, the third image is an image related to noise contained in the first image or the second image. Note that the method for generating the third image is not limited to this. Other methods will be described later.

Next, a third image that is a low-resolution noise image and a fourth image that is a low-resolution image are weighted and added to generate a fifth image that is a low-resolution training image. The reason why the low-resolution noise images are weighted and added is to improve the noise level of the low-resolution images whose noise level has decreased due to downsampling. Note that details regarding the method for determining the weights for weighted addition will be described later. Next, a fifth image, which is a low-resolution training image, is input into the multilayer neural network.

Next, the weights of the multilayer neural network are optimized (learned) so that the error between the output and the first image (or second image), which is a high-resolution training image, is reduced. That is, the neural network is trained using the fifth image and the first image or the second image. More specifically, the neural network is trained to convert the fifth image to one of the first image or the second image, which has a higher noise level. The learned neural network is a network that performs super-resolution image processing (up-sampling) on input images.

According to each embodiment, the high-resolution training image and the low-resolution training image have mutually correlated noise, and the noise level can also be made equal. This allows the multilayer neural network to learn weights for performing DL super-resolution image processing while suppressing noise level fluctuations. That is, it is possible to generate a trained model that can perform DL super-resolution image processing while suppressing fluctuations in noise level. Note that the noise correlated with the high-resolution training image may be obtained by downsampling the noise included in the high-resolution training image. That is, the case where the high-resolution training image and the low-resolution training image have mutually correlated noise includes the case where the low-resolution training image has a component obtained by downsampling the noise contained in the high-resolution training image. Further, in DL super-resolution image processing, it is possible to control (suppress) fluctuations in the noise level by adjusting the weights when weighted and added low-resolution noise images are added. That is, this is possible by making the noise level of the high-resolution training image and the low-resolution training image equal.

Note that the above-described image processing method is an example, and each embodiment is not limited to this. Details of other image processing methods will be explained in each embodiment below.

First, an image processing system according to a first embodiment of the present invention will be described. In this embodiment, a multilayer neural network learns and executes super-resolution image processing.

FIG. 2 is a block diagram of the image processing system 100 in this embodiment. FIG. 3 is an external view of the image processing system 100. The image processing system 100 includes a learning device (image processing device) 101, an imaging device 102, an image estimation device (image processing device) 103, a display device 104, a recording medium 105, an output device 106, and a network 107.

The learning device 101 includes a storage section (storage means) 101a, a noise level section (noise level means) 101b, a noise image generation section (noise image generation means) 101c, and a downsampling section 101d (downsampling means). The learning device 101 also includes a noise addition section (noise addition means) 101e and a learning section (learning means) 101f.

The imaging device 102 includes an optical system 102a and an image sensor 102b. The optical system 102a collects light that has entered the imaging device 102 from the subject space. The image sensor 102b receives (photoelectrically converts) an optical image (subject image) formed through the optical system 102a, and acquires a captured image. The image sensor 102b is, for example, a CCD (Charge Coupled Device) sensor, a CMOS (Complementary Metal-Oxide Semiconductor) sensor, or the like. The captured image acquired by the imaging device 102 includes blur caused by aberrations and diffraction of the optical system 102a, and noise caused by the imaging element 102b.

The image estimation device 103 includes a storage section 103a, an acquisition section 103b, and a super-resolution image processing section (estimation means) 103c. The image estimation device 103 acquires a captured image, performs DL super-resolution image processing that suppresses noise level fluctuations, and generates an estimated image (output image). A multilayer neural network is used for DL super-resolution image processing, and weight information is read from the storage unit 103a. The weights (weight information) are learned by the learning device 101, and the image estimation device 103 reads out the weight information from the storage unit 101a via the network 107 in advance and stores it in the storage unit 103a. The weight information to be saved may be the weight value itself or may be in an encoded format. Note that details regarding weight learning and DL super-resolution image processing using weights will be described later.

The output image is output to at least one of the display device 104, the recording medium 105, and the output device 106. The display device 104 is, for example, a liquid crystal display or a projector. The user can perform editing work while checking the image being processed through the display device 104. The recording medium 105 is, for example, a semiconductor memory, a hard disk, a server on a network, or the like. The output device 106 is a printer or the like. The image estimation device 103 has a function of performing development processing and other image processing as necessary.

Next, a method for learning weights (weight information) (method for manufacturing a learned model) executed by the learning device 101 in this embodiment will be described with reference to FIGS. 1 and 4. FIG. 1 is a diagram showing the flow of learning weights in a multilayer neural network. FIG. 4 is a flowchart regarding weight learning. Each step in FIG. 4 is mainly executed by the noise level section 101b, the noise image generation section 101c, the downsampling section 101d, the noise addition section 101e, and the learning section 101f.

First, in step S101, the noise level unit 101b generates an intermediate patch (second image) based on the high resolution patch (first image). In this embodiment, the high-resolution patch is a high-resolution image with less blur due to aberrations and diffraction of the optical system 102b. The high-resolution patch includes noise components caused by the image sensor 102b. The intermediate patch is an image that includes the same subject as the high-resolution patch, but has a different noise level (more or less noise) than the high-resolution patch. Note that a patch refers to an image having a predetermined number of pixels (for example, 64×64 pixels, etc.). Furthermore, the number of pixels of the patches does not necessarily have to match. For example, the number of pixels in a high resolution patch is greater than the number of pixels in a corresponding low resolution patch. Further, the number of pixels of the high-resolution patch and the intermediate patch are equal.

In this embodiment, an intermediate patch having a different noise level (less noise) than the high-resolution patch is generated by removing noise from the high-resolution patch using the noise removal method BM3D. In this embodiment, a plurality of original images stored in the storage unit 101a are high-resolution captured images. The original image may be a captured image acquired by the imaging device 102, or may be a captured image generated by an imaging simulation. Images collected from the Internet may also be used. Further, the original image is an image containing noise caused by the image sensor. In this embodiment, the image sensor 102b is used as the image sensor, but the present invention is not limited to this. For example, it may be noise caused by another image sensor, or it may be noise based on a random number sequence generated using normal random numbers. Then, by extracting partial regions of a prescribed pixel size from the plurality of original images (high-resolution captured images), a plurality of high-resolution patches are obtained. In this embodiment, the original image is a PNG image, but the present invention is not limited to this. For example, other image formats such as BMP and JPG may be used, or undeveloped RAW images may be used.

Preferably, the original image is a high-resolution image with less blurring (less influence) due to aberrations and diffraction. This is because if the original image (high-resolution image) does not contain high-frequency components, there will be no high-frequency components estimated from the low-resolution image in DL super-resolution image processing.

In this example, an intermediate patch having a different noise level (less noise) than the high-resolution patch is generated by removing noise from the high-resolution patch using the noise removal method BM3D, but the present invention is not limited to this. do not have. For example, other noise removal methods such as NLM (non-local means) and DL may be used. Alternatively, by adding noise to a high-resolution patch, intermediate patches with different noise levels (noisy) may be generated. Note that the noise to be added may be noise based on a random number sequence generated using normal random numbers. Alternatively, the noise may be noise based on a random number sequence generated using a pseudo random number generator using uniform random numbers or irrational numbers.

Subsequently, in step S102, the noise image generation unit 101c generates a low resolution noise patch (first image or second image) based on the high resolution patch (first image) and intermediate patch (second image). A third image related to the noise contained in the image is generated. In this embodiment, the number of pixels of the high-resolution patch and the intermediate patch is 256×256 pixels, and the number of pixels of the low-resolution noise patch is 64×64 pixels, but the number of pixels is not limited to this.

Note that in this embodiment, a low-resolution noise patch is generated from a high-resolution patch and an intermediate patch by the following method, but the method is not limited to this. First, a high-resolution patch and an intermediate patch are subtracted to generate a high-resolution noise patch. In this embodiment, the intermediate patch is subtracted from the high resolution patch, but conversely, the high resolution patch may be subtracted from the intermediate patch. Next, the high-resolution noise patch is downsampled using bicubic interpolation to generate a low-resolution noise patch. Note that the downsampling method is not limited to this, and may be bilinear interpolation or nearest neighbor interpolation. Alternatively, as shown in FIG. 10, a low-resolution noise patch may be generated by downsampling the high-resolution patch and the intermediate patch, respectively, and then subtracting them.

Further, in step S101, when adding noise to a high resolution patch to generate an intermediate patch having a different noise level (noisy), a low resolution noise patch may be generated by the following method. That is, the high-resolution noise patch obtained by subtracting the high-resolution patch and the intermediate patch is the noise added to the high-resolution patch. Therefore, in step S101, a low resolution noise patch may be generated by storing the noise added to the high resolution patch and downsampling it.

Subsequently, in step S103, the downsampling unit 101d generates a downsampling patch (fourth image) based on at least one of the high resolution patch (first image) and the intermediate patch (second image). Note that the generated low-resolution downsampling patch has the same number of pixels as the low-resolution noise patch. Note that in this embodiment, at least one of the high-resolution patch and the intermediate patch is downsampled by bicubic interpolation to generate a downsampled patch, but the invention is not limited to this. For example, bilinear interpolation or nearest neighbor interpolation may be used as the downsampling method. Moreover, the order of step S102 and step S103 may be reversed. Further, when the high-resolution patch and the intermediate patch are down-sampled in step S102, the results may be stored and used as the down-sampling patch. That is, it is not necessarily necessary to perform downsampling again.

Subsequently, in step S104, the noise addition unit 101e generates a low resolution patch (fifth image) based on the low resolution noise patch (third image) and the downsampling patch (fourth image). In this embodiment, a low-resolution noise patch and a downsampling patch are weighted and added to generate a low-resolution patch. Note that the weights used for weighted addition are determined based on the noise level of the high-resolution patch. In this embodiment, the weights are determined based on the ISO sensitivity of the image sensor 102b of the imaging device 102 that acquires the original image, but the weight is not limited thereto. For example, the determination may be based on at least one of the average value, standard deviation, or variance when noise included in a high-resolution patch is approximated by Gaussian noise. Further, in step S101, when noise is added to the high-resolution patch to generate intermediate patches with different noise levels (noisy), the weights used for weighted addition may be determined by the following method. That is, it may be determined based on the standard deviation of a random number sequence generated by normal random numbers, which is the source of the noise to be added. In this embodiment, only one type of weight is used for weighted addition.

Subsequently, in step S105, the learning unit 101f inputs the low resolution patch (fifth image) 201 to a multilayer neural network (neural network) to generate an estimated patch (estimated image) 202. The estimated patch 202 is a low-resolution patch 201 that has been subjected to DL super-resolution image processing (high resolution) while suppressing fluctuations in noise level. Ideally, the estimated patch 202 matches the high resolution patch (first image) 200. This is because the noise included in the low resolution patch 201 and the noise included in the high resolution patch 200 are correlated.

On the other hand, in step S101, when noise is added to the high-resolution patch to generate an intermediate patch with a different noise level (noisy), the estimated patch 202 matches the intermediate patch (second image). In this case, the high-resolution patch that has noise that is correlated with the noise that the low-resolution patch 201 has is an intermediate patch.

In this embodiment, the configuration of the multilayer neural network shown in FIG. 1 is used, but the configuration is not limited to this. CN in FIG. 1 represents a convolutional layer. In the CN, the sum of the convolution of the input, the filter, and the bias is calculated, and the result is nonlinearly transformed by an activation function. The initial values of each component and bias of the filter are arbitrary, and are determined by random numbers in this embodiment. As the activation function, for example, ReLU (Rectified Linear Unit) or a sigmoid function can be used. The output of each layer except the final layer is called a feature map. Skip connections 211, 212, and 213 combine feature maps output from non-consecutive layers. The feature maps may be combined by summing each element or by concatenation in the channel direction. In this embodiment, the sum of each element is used. Further, a skip connection 210 may also be included. This is a path in which the estimated patch 202 is generated by enlarging the low resolution patch 201 by interpolation (increasing the resolution) and combining it with the residual obtained from the multilayer neural network. An estimated patch 202 is generated for each of the plurality of low resolution patches 201. PS in FIG. 1 represents the Pixel Shuffle layer. PS sorts low-resolution feature maps and generates high-resolution feature maps. For example, a low resolution feature map of 64 x 64 x 64 pixels is rearranged to generate a high resolution feature map of 128 x 128 x 16 pixels. The size of the feature map is expressed as length x width x depth (channel). The feature map obtained by PS may be convolved and the number of channels of the feature map may be adjusted. In this embodiment, Pixel Shuffle is used to increase the resolution of the feature map, but the present invention is not limited to this. For example, deconvolution or interpolation may be used.

Subsequently, in step S106, the learning unit 101f updates the weights (weight information) of the multilayer neural network based on the error between the estimated patch 202 and the high resolution patch (first image) 200. Here, the weights include filter components and biases of each layer. Although the error backpropagation method is used to update the weights, the present invention is not limited thereto. For mini-batch learning, errors between a plurality of high-resolution patches 200 and their corresponding estimated patches 202 are determined, and weights are updated. For example, the L2 norm or the L1 norm may be used as the error function (Loss function). The weight updating method (learning method) is not limited to mini-batch learning, but may be batch learning or online learning.

Further, in step S101, when noise is added to the high-resolution patch to generate an intermediate patch with a different noise level (noisy), the estimated patch 202 matches the intermediate patch (second image). Therefore, the learning unit 101f updates the weights of the multilayer neural network based on the error between the estimated patch 202 and the intermediate patch. The relationship between the estimated patch 202 and the intermediate patch is as described in step S104.

Subsequently, in step S107, the learning unit 101f determines whether learning of weights is completed. Completion can be determined based on whether the number of repetitions of learning (updating weights) has reached a specified value, or whether the amount of change in weights at the time of updating is smaller than a specified value. If it is determined that the process is incomplete, the process returns to step S101 and a plurality of new high-resolution patches and new low-resolution patches are acquired. On the other hand, if it is determined that the learning has been completed, the learning device 101 (learning section 101f) ends the learning and stores the weight information in the storage section 101a.

Next, with reference to FIG. 5, generation of an output image executed by the image estimation device 103 in this embodiment will be described. FIG. 5 is a flowchart regarding generation of an output image. Each step in FIG. 5 is mainly executed by the acquisition unit 103b and the super-resolution image processing unit 103c of the image estimation device 103.

First, in step S201, the acquisition unit 103b acquires a captured image and weight information. The captured image is a PNG image, as in the case of learning, and is transmitted from the imaging device 102 in this embodiment. The weight information is transmitted from the learning device 101 and stored in the storage unit 103a.

Subsequently, in step S202, the super-resolution image processing unit 103c inputs the captured image to a multilayer neural network to which the acquired weights are applied, and generates an estimated image (output image). The estimated image is an image obtained by suppressing fluctuations in noise level from the captured image and increasing the resolution of the captured image. A multilayer neural network similar to the configuration shown in FIG. 1 is used to generate the estimated image. Note that when inputting a captured image to a multilayer neural network, it is not necessary to cut it out to the same size as the training patch used during learning.

Here, a case where there is no correlation between the noise of high-resolution patches and the noise of low-resolution patches will be described. In this case, since the noise level differs depending on the location of the high-resolution patch and the low-resolution patch, the system learns to remove noise in some locations and emphasize noise in other locations. That is, as the resolution increases, the noise level of the image fluctuates (increases or decreases), and the quality of the super-resolution image deteriorates.

Next, a case will be described where the high-resolution patch has no noise and only the low-resolution patch has noise. In this case, the system learns to increase the resolution of low-resolution patches and at the same time remove noise from the low-resolution patches. However, real images always contain noise caused by the image sensor, and removing the noise results in a rather unnatural high-resolution image. Furthermore, when a large high-resolution image is obtained by increasing the resolution of a low-resolution image (enlarging the image), the unnaturalness of the image from which noise has been removed stands out.

On the other hand, according to this embodiment, the noise level of the image does not change as the resolution increases. That is, it is possible to increase the resolution of a captured image while keeping the level of noise, which is a factor that gives naturalness to the image, to be the same as the noise level of the captured image.

In this embodiment, the learning device 101 and the image estimation device 103 are separate entities, but the present invention is not limited to this. The learning device 101 and the image estimation device 103 may be configured integrally. That is, learning (the process shown in FIG. 4) and estimation (the process shown in FIG. 5) may be performed within a single device.

With the above configuration, according to this embodiment, it is possible to provide an image processing system that suppresses fluctuations in the noise level of images accompanying DL super-resolution image processing.

Next, an image processing system according to a second embodiment of the present invention will be described. In this embodiment, similarly to the first embodiment, a multilayer neural network learns and executes super-resolution image processing. The image processing system of the present embodiment differs from the first embodiment in that an imaging device acquires a captured image and generates a super-resolution image in which fluctuations in noise level accompanying DL super-resolution image processing are suppressed.

FIG. 6 is a block diagram of the image processing system 300 in this embodiment. FIG. 7 is an external view of the image processing system 300. The image processing system 300 includes a learning device (image processing device, first device) 301 and an imaging device (second device) 302 that are connected via a network 303. Note that in this embodiment, the learning device 301 and the imaging device 302 do not need to be always connected via the network 303.

The learning device 301 includes a storage section (storage means) 311, a noise level section (noise level means) 312, a noise image generation section (noise image generation means) 313, and a downsampling section (downsampling means) 314. The learning device 301 also includes a noise addition section (noise addition means) 315 and a learning section (learning means) 316. These are used to learn weights (weight information) for performing DL super-resolution image processing that suppresses fluctuations in noise level using a multilayer neural network.

The imaging device 302 images the subject space to obtain a captured image, and uses the read weight information to generate a super-resolution image from the captured image. Details regarding weight learning performed by the learning device 301 and DL super-resolution image processing performed by the imaging device 302 will be described later. The imaging device 302 includes an optical system 321 and an image sensor 322. The image estimation unit 323 includes an acquisition unit 323a and a super-resolution image processing unit (estimation means) 323b, and uses weight information stored in the storage unit 324 to extract a super-resolution image with higher resolution than the image sensor 322 from the captured image. Execute generation of resolved image.

Note that the learning of the weights of the multilayer neural network executed by the learning device 301 is the same as in the first embodiment, and therefore the description thereof will be omitted. Only details regarding the DL super-resolution image processing executed by the imaging device 302 will be described later.

The weight information is learned in advance by the learning device 301 and stored in the storage unit 311. The imaging device 302 reads weight information from the storage unit 311 via the network 303 and stores it in the storage unit 324. The captured image (output image) subjected to the DL super-resolution image processing is stored in the recording medium 325. When a user issues an instruction regarding display of an output image, the saved output image is read out and displayed on the display unit 326. Note that a captured image already stored in the recording medium 325 may be read out and the image estimation unit 323 may perform DL super-resolution image processing. The above series of controls are performed by the system controller 327.

Next, generation of an output image executed by the image estimation unit 323 in this embodiment will be explained. Since the procedure of DL super-resolution image processing is substantially the same as that in FIG. 5 of the first embodiment, the flowchart is omitted. Each step of the DL super-resolution image processing is mainly executed by the acquisition section 323a and the super-resolution image processing section (super-resolution image processing means) 323b of the image estimation section 323.

First, in step S201, the acquisition unit 323a acquires a captured image and weight information. The captured image is a PNG image, as in the case of learning, and is acquired by the imaging device 302 and stored in the storage unit 324 in this embodiment. The weight information is transmitted from the learning device 301 and stored in the storage unit 324.

Subsequently, in step S202, the super-resolution image processing unit 323b inputs the captured image to a multilayer neural network to which the acquired weights are applied, and generates an estimated image (output image). The estimated image is an image obtained by suppressing fluctuations in noise level from the captured image and increasing the resolution of the captured image. A multilayer neural network similar to the configuration shown in FIG. 1 is used to generate the estimated image. Note that when inputting a captured image to a multilayer neural network, it is not necessary to cut it out to the same size as the training patch used during learning.
With the above configuration, according to this embodiment, it is possible to provide an image processing system that suppresses fluctuations in the noise level of images accompanying DL super-resolution image processing.

Next, an image processing system according to a third embodiment of the present invention will be described. The image processing system of this embodiment is different from the embodiment in that it includes a processing device (computer) that transmits a captured image that is a target of image processing to an image estimation device and receives a processed output image from the image estimation device. 1 and 2 are different.

FIG. 8 is a block diagram of the image processing system 400 in this embodiment. The image processing system 400 includes a learning device (first device) 401, an imaging device 402, an image estimation device (second device) 403, and a processing device (computer, third device) 404. The learning device 401 and the image estimation device 403 are, for example, servers. Computer 404 is, for example, a user terminal (personal computer or smartphone). Computer 404 is connected to image estimation device 403 via network 405. Image estimation device 403 is connected to learning device 401 via network 406. That is, the computer 404 and the image estimation device 403 are configured to be able to communicate, and the image estimation device 403 and the learning device 401 are configured to be able to communicate.

The learning device 401 includes a storage section (storage means) 401a, a noise level section (noise level means) 401b, a noise image generation section (noise image generation means) 401c, and a downsampling section 401d (downsampling means). The learning device 401 also includes a noise addition section (noise addition means) 401e and a learning section (learning means) 401f. The imaging device 402 includes an optical system 402a and an imaging element 402b. Note that the configurations of the learning device 401 and the imaging device 402 are the same as those of the learning device 101 and the imaging device 102 of Example 1, respectively, so their descriptions will be omitted.

The image estimation device 403 includes a storage section 403a, an acquisition section 403b, a super-resolution image processing section (estimation means) 403c, and a communication section (reception means) 403d. The storage unit 403a, the acquisition unit 403b, and the super-resolution image processing unit 403c are the same as the storage unit 103a, the acquisition unit 103b, and the super-resolution image processing unit 103c of the image estimation device 103 of the first embodiment, respectively. . The communication unit 403d has a function of receiving a request transmitted from the computer 404 and a function of transmitting an output image generated by the image estimation device 403 to the computer 404.

The computer 404 includes a communication section (transmission means) 404a, a display section 404b, an image processing section 404c, and a recording section 404d. The communication unit 404a has a function of transmitting a request to the image estimation device 403 for causing the image estimation device 403 to perform processing on a captured image, and a function of receiving an output image processed by the image estimation device 403. The display unit 404b has a function of displaying various information. The information displayed by the display unit 404b includes, for example, the captured image transmitted to the image estimation device 403 and the output image received from the image estimation device 403. The image processing unit 404c has a function of further performing image processing on the output image received from the image estimation device 403. The recording unit 404d records the captured image acquired from the imaging device 402, the output image received from the image estimation device 403, and the like.

Next, the image processing in this embodiment will be described with reference to FIG. 9. FIG. 9 is a flowchart relating to the generation of an output image in this embodiment. Note that the image processing in this embodiment is equivalent to the DL super-resolution image processing (FIG. 5) described in the first embodiment. The image processing shown in FIG. 9 is started when an instruction to start image processing is given by the user via the computer 404.

First, the operation of the computer 404 will be explained. In step S401, the computer 404 transmits a request for processing the captured image to the image estimation device 403. Note that the method of transmitting the captured image to be processed to the image estimation device 403 does not matter. For example, the captured image may be uploaded to the image estimation device 403 at the same time as step S401, or may be uploaded to the image estimation device 403 before step S401. Further, the captured image may be an image stored on a server different from the image estimation device 403. Note that in step S401, the computer 404 may transmit ID information for authenticating the user together with a request for processing the captured image.

Subsequently, in step S402, the computer 404 receives the output image generated within the image estimation device 403. The output image is an image obtained by subjecting the captured image to DL super-resolution image processing (higher resolution) while suppressing fluctuations in noise level as in the first embodiment.

Next, the operation of the image estimation device 403 will be explained. First, in step S501, the image estimation device 403 receives a request to process a captured image transmitted from the computer 404. The image estimation device 403 determines that processing on the captured image (DL super-resolution image processing) has been instructed, and executes the processing from step S502 onwards.

Subsequently, in step S502, the image estimation device 403 acquires weight information. The weight information is information (learned model) learned using the same method as in Example 1 (FIG. 4). The image estimation device 403 may acquire weight information from the learning device 401, or may acquire weight information that has been acquired from the learning device 401 in advance and stored in the storage unit 403a. The following step S503 is similar to step S202 of the first embodiment. Subsequently, in step S504, the image estimation device 403 transmits the output image to the computer 404.

As described above, as in this embodiment, the image estimation device 403 may be configured to be controlled using the computer 404 that is communicably connected to the image estimation device 403.

(Other examples)
The present invention provides a system or device with a program that implements one or more of the functions of the above-described embodiments via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. This can also be achieved by processing. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

According to each embodiment, it is possible to provide an image processing method, an image processing apparatus, a program, and a storage medium that can suppress fluctuations in image noise caused by image processing.

Although preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes can be made within the scope of the invention.

101 Learning device (image processing device)
101b Noise level section (noise level means)
101c Noise image generation unit (noise image generation means)
101d Downsampling section (downsampling means)
101e Noise addition section (noise addition means)
101f Learning Department (learning means)

Claims (18)

An image processing method performed using a computer, the method comprising:
a first step of generating a second image having a different noise level from the first image based on the first image;
a second step of generating a third image regarding noise contained in the first image or the second image based on the first image and the second image;
a third step of generating a fourth image by downsampling at least one of the first image and the second image;
a fourth step of generating a fifth image by weighted addition of the third image and the fourth image;
An image processing method comprising: a fifth step of performing neural network learning using the fifth image and the first image or the second image. In the fifth step, the neural network is trained to convert the fifth image to an image having a higher noise level among the first image or the second image. The image processing method according to claim 1. 3. The image processing method according to claim 1, wherein in the first step, the second image is generated by reducing noise in the first image. Claims 1 to 3, wherein in the second step, the third image is generated by subtracting the first image and the second image pixel by pixel and then downsampling them. The image processing method according to any one of . Claims 1 to 3, wherein in the second step, the third image is generated by downsampling the first image and the second image, and then subtracting each pixel. 3. The image processing method according to any one of 3. 3. The image processing method according to claim 1, wherein in the first step, the second image is generated by adding noise to the first image. 7. The image processing method according to claim 6, wherein in the second step, the third image is generated based on the added noise. 8. The image processing method according to claim 1, wherein the weight in the weighted addition is determined based on noise in the first image or the second image. 9. The image processing method according to claim 8, wherein the weight is determined based on at least one of an average value, standard deviation, or variance when the noise is approximated by Gaussian noise. 9. The image processing method according to claim 8, wherein the weight is determined based on ISO sensitivity when the first image is captured. The image processing method according to any one of claims 1 to 10, wherein the neural network performs super-resolution image processing on the input image. 12. The image processing method according to claim 1, further comprising a sixth step of generating an estimated image by inputting an input image to the neural network. A program that causes a computer to execute the image processing method according to any one of claims 1 to 12. A method for manufacturing a trained model executed using a computer, the method comprising:
a first step of generating a second image having a different noise level from the first image based on the first image;
a second step of generating a third image regarding noise contained in the first image or the second image based on the first image and the second image;
a third step of generating a fourth image by downsampling at least one of the first image and the second image;
a fourth step of generating a fifth image by weighted addition of the third image and the fourth image;
A method for manufacturing a trained model, comprising: a fifth step of performing neural network learning using the fifth image and the first image or the second image. noise level means for generating a second image having a different noise level from the first image based on the first image;
Noise image generation means for generating a third image related to noise contained in the first image or the second image based on the first image and the second image;
downsampling means for generating a fourth image by downsampling at least one of the first image and the second image;
noise addition means for generating a fifth image by weighted addition of the third image and the fourth image;
An image processing apparatus comprising: learning means for performing neural network learning using the fifth image and the first image or the second image. An image processing system including a first device and a second device capable of communicating with the first device,
The first device includes:
noise level means for generating a second image having a different noise level from the first image based on the first image;
Noise image generation means for generating a third image related to noise contained in the first image or the second image based on the first image and the second image;
downsampling means for generating a fourth image by downsampling at least one of the first image and the second image;
noise addition means for generating a fifth image by weighted addition of the third image and the fourth image;
a learning means for learning a neural network using the fifth image and the first image or the second image,
The image processing system is characterized in that the second device includes an estimator that generates an estimated image from an input image using the neural network. An image processing system including a first device, a second device capable of communicating with the first device, and a third device capable of communicating with the second device,
The first device includes:
noise level means for generating a second image having a different noise level from the first image based on the first image;
Noise image generation means for generating a third image related to noise contained in the first image or the second image based on the first image and the second image;
downsampling means for generating a fourth image by downsampling at least one of the first image and the second image;
noise addition means for generating a fifth image by weighted addition of the third image and the fourth image;
a learning means for learning a neural network using the fifth image and the first image or the second image,
The third device includes a transmitting means for transmitting a request for causing the second device to perform processing on the captured image,
The second device includes:
receiving means for receiving the request transmitted by the transmitting means;
An image processing system comprising: estimating means for generating an estimated image from an input image using the neural network. An image processing device comprising: an estimation unit that generates an estimated image from an input image using the neural network trained using the image processing method according to any one of claims 1 to 11.