JP2014086957A - Image processing device and image processing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To solve the problem in which, when color image data is generated by interpolating pixels while noise is included in RAW image data, the noise may be emphasized, resulting in deterioration in image quality.SOLUTION: Noise reduction processing is performed on RAW image data, and pixel interpolation processing is performed on image data subjected to the noise reduction processing, thereby generating color image data. Then, the pixel interpolation processing is performed using edge information generated in the noise reduction processing.

Description

  The present invention relates to an image processing apparatus and an image processing method for reducing noise included in image data.

  Digital imaging devices such as digital still cameras and digital video cameras are widely spread and are generally used. These digital imaging devices generate digital image data by converting light received by a photoelectric charge conversion device (imaging device) such as a CCD or CMOS sensor into a digital signal. Here, in order to obtain image data indicating a color image, color filters having different transmittances according to the wavelength of light are regularly arranged in front of the image sensor. The digital imaging device generates a color signal from the difference in the amount of light transmitted through the color filter. Therefore, data corresponding to the arrangement of the color filters is recorded in image data (hereinafter referred to as RAW image data) recorded by the digital imaging apparatus. Thereafter, a series of image processing such as white balance correction and pixel interpolation is performed on the RAW image data, and image data indicating a general color image such as an RGB image is generated. Thus, digital image processing for generating a general color image from image data recorded by an image sensor included in a digital imaging device is generally called development processing.

  In the process of generating digital image data, dark current noise, thermal noise, shot noise, and the like are generated due to the characteristics of the imaging device and circuit, and the noise is mixed into the digital image data. With recent miniaturization of image sensors and higher pixel counts, the pixel pitch has been minimized, making noise more conspicuous, especially when shooting sensitivity is increased, resulting in noticeable noise and significant deterioration in image quality. It is a factor. Therefore, in order to reduce such noise and obtain a high-quality image, it is necessary to reduce noise mixed in the RAW image data.

  Conventionally, a technique for reducing noise by applying a low-pass filter that passes a signal component below a noise frequency is known. However, it is difficult to obtain a high-quality image because edges are blurred in addition to noise. Therefore, many methods have been proposed for determining noise and edge information by some method and adaptively reducing noise.

  In general, in adaptive noise reduction, in order to reduce noise of a target pixel, a plurality of pixels near the target pixel are selected as reference pixels, and the target pixel is replaced with an appropriate weighted average value of the reference pixels. To reduce noise.

  As one of adaptive noise reduction methods, a region including a target pixel (target region) is determined, the similarity between the target pixel and the reference pixel is obtained for each region, and a weighted average value corresponding to the similarity is calculated. Thus, noise reduction is realized (Patent Document 1, Patent Document 2).

  In addition, adaptive interpolation processing that reduces edge blur has also been proposed in pixel interpolation (Patent Document 3).

Special table 2007-536662 gazette JP 2011-39675 A JP-A-8-298669

  In a conventional method in development processing, pixel interpolation processing is performed to generate RGB image data, and then noise reduction processing is performed on the RGB image data. In this case, a process of detecting an edge in an image overlappingly in the preceding pixel interpolation process and the subsequent noise reduction process has been applied. Further, when color image data is generated by performing pixel interpolation while including noise in RAW image data, there is a problem that noise may be emphasized, resulting in deterioration of image quality.

  An image processing apparatus according to the present invention includes a noise reduction unit that performs noise reduction processing on RAW image data, and pixel interpolation that generates color image data by performing pixel interpolation processing on the image data subjected to noise reduction processing by the noise reduction unit. Means.

  According to the present invention, in the development processing for generating color image data from RAW image data, it is possible to reduce image quality deterioration by applying pixel interpolation processing to image data with reduced noise in the RAW image data. Also, pixel interpolation processing can be performed with reference to image edge information obtained by noise reduction processing. Therefore, there is an effect of improving the image quality while reducing the load related to image edge detection in the development processing.

It is a block diagram which shows the example of the hardware constitutions of the image processing apparatus which concerns on the Example of this invention. It is a block diagram which shows the example of the logic structure of the image processing apparatus which concerns on the Example of this invention. It is a flowchart figure which shows an example of the flow of the image processing which concerns on the Example of this invention. It is a schematic diagram explaining the Bayer arrangement which concerns on the Example of this invention. It is a flowchart figure which shows an example of the flow of the noise reduction process which concerns on the Example of this invention. It is a flowchart figure which shows an example of the flow of the pixel interpolation process which concerns on the Example of this invention. It is a schematic diagram explaining the pixel interpolation process which concerns on the Example of this invention. It is a schematic diagram explaining the noise reduction method which concerns on the Example of this invention. It is a schematic diagram explaining the function which calculates the weight of the reference pixel which concerns on the Example of this invention.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  The configuration of the image processing apparatus in this embodiment will be described with reference to FIG.

  In FIG. 1, the image processing apparatus includes a CPU 101, a RAM 102, an HDD 103, a general-purpose interface (I / F) 104, a monitor 108, and a main bus 109. The general-purpose I / F 104 connects an imaging device 105 such as a camera, an input device 106 such as a mouse and a keyboard, and an external memory 107 such as a memory card to a main bus 109.

  In the following, various processes realized by the CPU 101 operating various software (computer programs) stored in the HDD 103 will be described.

  First, the CPU 101 activates an image processing application stored in the HDD 103, expands it in the RAM 102, and displays a user interface (UI) on the monitor 108. Subsequently, various data stored in the HDD 103 and the external memory 107, RAW image data captured by the imaging device 105, instructions from the input device 106, and the like are transferred to the RAM 102. Further, various calculations are performed on the data stored in the RAM 102 based on instructions from the CPU 101 in accordance with the processing in the image processing application. The calculation result is displayed on the monitor 108 or stored in the HDD 103 or the external memory 107. Note that RAW image data stored in the HDD 103 or the external memory 107 may be transferred to the RAM 102. Further, RAW image data transmitted from a server via a network (not shown) may be transferred to the RAM 102.

  In the above configuration, details of processing for generating and outputting image data with reduced noise by inputting image data to the image processing application based on a command from the CPU 101 will be described.

(NonLocalMeans)
First, the noise reduction process described in the present embodiment will be described. In this embodiment, an example using a technique called NonLocalMeans (NLM) will be described. This method multiplies the pixel values of a plurality of reference pixels that exist around the pixel of interest, including the pixel of interest that is subject to noise reduction, adaptively weights the pixel values of the pixel of interest by adding all the results. Noise is reduced by replacement. When the number of reference pixels is N _S , the pixel value of the reference pixel is I _j (j = 1 to N _S ), and the weight of the reference pixel is w _j (j = 1 to N _S ), attention after noise reduction processing The pixel value I _new of the pixel is given by

  Next, a method for calculating the weight of the reference pixel will be described with reference to FIGS.

In FIG. 8A, reference numeral 801 denotes an example of image data, and the pixel value of each pixel is represented as I (x, y) with the upper left pixel as the origin. Here, reference numeral 802 denotes a pixel of interest, and its pixel value is I (4, 4). Reference numeral 803 denotes a region of interest, which is a 3 × 3 pixel rectangular region centered on the pixel of interest 802 that is a noise reduction target. Reference numeral 804 denotes a reference pixel, which is a pixel in a 5 × 5 pixel (N _S = 25) rectangular area including the target pixel 802. Reference numeral 805 denotes a reference area of the reference pixel I (2, 2), which is a 3 × 3 rectangular area having the same size as the area of interest centered on the reference pixel I (2, 2). A reference area exists for each reference pixel, but only the reference area of the reference pixel I (2, 2) is shown here.

In order to obtain the weight of the reference pixel I (2, 2), first, the similarity is calculated by comparing the region of interest 803 with the reference region 805 of the reference pixel I (2, 2). Note that the similarity may be obtained by a desired method. For example, as shown in FIG. 8B, the pixel of the attention area 803 is b _s (p, q), and the pixel of the reference area is b _j (p, q) (j = 1 to N _S ). Then, assuming that the difference between spatially corresponding pixels in the region of interest 803 and the reference region 805 is the similarity, the similarity C _j is expressed by the following equation.

The smaller the value of the similarity C _{j, the} higher the similarity between the focus area and the reference area. Therefore, the weight is determined according to the degree of similarity. The weight may be determined so that the weight is larger as the similarity C _j is smaller and the weight is smaller as the similarity C _j is larger as in the function shown in FIG.

  Here, h is a variable that controls the magnitude of the weight. Increasing h increases the noise reduction effect but blurs the edge.

  Similarly, the weight of each reference pixel can be obtained by sequentially comparing the attention area 803 and the reference area of each reference pixel.

  In addition, although the noise reduction process in a present Example demonstrated the process which determines the weight of a reference pixel based on the similarity degree of an attention area and a reference area, it determines based on the similarity degree of an attention pixel and a reference pixel. It may be a process. For example, the weight of the reference pixel may be obtained using an ε-filter, a bilateral filter, or the like.

(Logical configuration of image processing device)
Hereinafter, image processing in the present embodiment will be described with reference to FIGS.

  FIG. 2 is a schematic diagram illustrating a logical configuration of the image processing apparatus according to the present exemplary embodiment, which includes an input unit 201, a storage unit 202, a processing unit 203, and an output unit 204. The processing unit 203 includes a white balance processing unit 205, a noise reduction processing unit 206, a pixel interpolation processing unit 207, and a correction processing unit 208. In FIG. 2, an input unit 201 inputs image data and image processing parameters to the image processing apparatus. The image processing parameter is a parameter used in white balance processing described later. Data to be input is input from the imaging device 105, the HDD 103, or the external memory 107 based on an instruction from the CPU 101. Of course, image data and image processing parameters obtained by photographing with the imaging device 105 may be temporarily stored in a storage device such as the HDD 103 and then input.

  The storage unit 202 holds image data input from the input unit 201, image processing parameters, image data obtained as a result of image processing, and the like. The storage unit 202 stores data in the RAM 102, the HDD 103, the external memory 107, or the like based on an instruction from the CPU 101. Of course, it is also possible to read the stored data based on an instruction from the CPU 101.

  A processing unit 203 reads image data and image processing parameters and performs image processing. The processing unit 203 first reads image data and image processing parameters from the input unit 201 or the storage unit 202 based on an instruction from the CPU 101. Then, image processing is applied based on the read data, and the processing result is output to the storage unit 202 and the output unit 204. Details of the image processing in the processing unit 203 will be described later.

  The output unit 204 outputs image data processed by the processing unit 203, image data stored in the storage unit 202, and the like. The output unit 204 outputs to the monitor 108, the HDD 103, etc. based on an instruction from the CPU 101. The output destination is not limited to this. For example, the output destination may be output to the external memory 107 connected to the general-purpose I / F 104 or an external server (not shown), or may be output by connecting a printer or the like. Absent.

(Main processing flow)
In the following, details of each process for generating color image data from RAW image data captured by the imaging apparatus 105 in the logical configuration of the image processing apparatus described in FIG. 2 will be described with reference to the flowchart of FIG. 3.

  In step S <b> 301, the input unit 201 inputs RAW image data and stores it in the storage unit 202. In this embodiment, it is assumed that the input RAW image data is image data taken by an imaging device having a Bayer array color filter. As shown in FIG. 4, the Bayer array has rows in which green (G) pixels and red (R) pixels are alternately repeated, and blue (B) pixels and green (G) pixels are alternately repeated. Are alternately arranged. However, the application range of the present embodiment is not limited to image data captured by an imaging apparatus having a Bayer array color filter. Any image data that can be subjected to noise reduction processing and pixel interpolation when generating color image data from RAW image data can be applied to image data captured by an imaging device having an arbitrary color filter.

  In step S302, the white balance processing unit 205 reads out the RAW image data input in step S301 from the storage unit 202 and applies white balance processing. Here, the image processing parameters applied to the white balance processing may be read from the storage unit 202 as additional information of the RAW image data, or input from the input unit 201 by separately specifying appropriate image processing parameters from a UI or the like. May be. In general, image processing parameters for white balance processing are given as gain values for RGB corresponding to the light sources at the time of shooting, and are corrected by multiplying image data by gain values corresponding to each color.

  In step S303, the noise reduction processing unit 206 applies noise reduction processing to the image data to which the white balance processing is applied in step S302. Details of the noise reduction processing will be described later.

  In step S304, the pixel interpolation processing unit 207 applies pixel interpolation processing (demosaic processing) to the image data to which the noise reduction processing is applied in step S303. Details of the pixel interpolation processing will be described later.

  In step S305, the correction processing unit 208 applies the correction process to the image data to which the noise reduction process is applied in step S304. Here, the correction processing includes general processing for correcting the output image so as to be seen suitably. For example, edge enhancement for increasing sharpness, γ correction for correcting brightness, color correction for increasing vividness, and the like. The details of the correction process are not the main point of the present embodiment, and thus the description thereof is omitted. The processing unit 203 stores the image data to which the correction process is applied in the storage unit 202.

  In step S306, the output unit 204 reads out and outputs the image data to which the correction process is applied in step S305 from the storage unit 202.

  In addition, the process which produces | generates color image data from RAW image data is not restricted to this. For example, processing for correcting image quality degradation caused by the characteristics of a lens or sensor may be included.

(Noise reduction processing)
Hereinafter, details of the noise reduction processing in step S303 described in FIG. 3 will be described with reference to the flowchart of FIG.

  In step S501, the noise reduction processing unit 206 acquires image data to which the white balance process is applied in step S302.

  In step S502, the noise reduction processing unit 206 acquires parameters for noise reduction processing. Here, when the above-mentioned NonLocalMeans method is applied as noise reduction processing, the parameters are a pixel used as a region of interest, a reference pixel, a control parameter h, and the like. The noise reduction parameter may be input from the input unit 201, or a parameter stored in advance in the storage unit 202 may be read.

  In step S503, the noise reduction processing unit 206 selects a processing pixel to which the noise reduction processing is applied, from the image data acquired in step S501.

  In step S504, the noise reduction processing unit 206 performs noise reduction processing on the processing pixel selected in step S503 based on the parameters acquired in step S502. Specifically, equations (1) to (3) may be applied sequentially.

In step S505, the noise reduction processing unit 206 performs edge information generation. The edge information E _i is obtained by the following equation using the ratio of the sum of the weights w _j calculated by the equation (3) and the sum of squares.

The larger the value of the edge information, the higher the possibility that the target pixel is an edge. For example, if the noise cannot be reduced, that is, if the weights of reference pixels other than the target pixel are 0, E _i = 1. On the other hand, when there is no noise, that is, when the weights of the reference pixels are all equal, E _i = 1 / Ns. As a result, the edge information has a large value in an area including a reference pixel having a weight of 0, such as an edge portion, and the edge information is not included in an area including many pixels of the same weight, such as a flat portion. Small value. Note that the edge information may be a value that can determine whether or not the target pixel is an edge, and is not limited thereto. For example, the number of reference pixels whose weight is equal to or less than a threshold can be used as edge information. Specifically, the reference pixel has a corresponding weight when the change in the pixel value is small, such as in a flat portion, but the number of reference pixels having a weight of 0 or extremely small increases in a region where the change in the pixel value is large, such as an edge portion. To do. Therefore, it can be used as edge information by counting the number of reference pixels whose weight is 0 or very small. Furthermore, information other than the weight, such as the similarity C _{j and} the difference between pixel values before and after noise reduction correction, can also be used as edge information. For example, the smaller the value of the similarity Cj is, the more the reference region is similar to the target region and the reference region. Therefore, the sum of the similarities Cj can be used as edge information that is smaller as the flat portion is larger and larger as the edge portion is larger. Further, the difference between the pixel values before and after the noise correction becomes larger as the flat portion changes the pixel value due to noise reduction. Therefore, it can be used as edge information that is larger in the flat portion and smaller in the edge portion.

  In step S506, the noise reduction processing unit 206 determines whether the noise reduction processing for all pixels has been completed. If completed, the process proceeds to step S507, and if not completed, the process proceeds to step S503 to continue the process.

  In step S507, the noise reduction processing unit 206 outputs the image data whose noise is reduced in step S504 and the edge information calculated in step S505.

(Pixel interpolation processing)
Hereinafter, details of the pixel interpolation processing in step S304 described in FIG. 3 will be described with reference to the flowchart in FIG.

  In step S601, the pixel interpolation processing unit 207 acquires the image data and edge information after noise reduction obtained in step S303.

  In step S602, the pixel interpolation processing unit 207 selects a processing pixel to which the pixel interpolation process is applied from the image data acquired in step S601.

  In step S603, the pixel interpolation processing unit 207 acquires edge information corresponding to the processing pixel selected in step S602 from the edge information acquired in step S601.

  In step S604, the pixel interpolation processing unit 207 determines whether the processing pixel is an edge portion. The determination may be made based on the size of the edge information. For example, the threshold value Eth is determined, and if the edge information Ei is smaller than the threshold value Eth, the process proceeds to step S605 as a flat portion. On the other hand, if the edge information Ei is larger than the threshold Eth, it is determined that the edge portion is an edge portion, and the process proceeds to step S606.

  In step S605, the pixel interpolation processing unit 207 applies linear interpolation to the processing pixel. Here, FIG. 7 shows pixels in a Bayer array, where RXX is a red pixel, GXX is a green pixel, and BXX is a blue pixel (XX is an arbitrary integer). A method for calculating the green pixels G33 and B33 corresponding to the pixel position of the red pixel R33 will be described. Since the processing pixel is a pixel determined not to be an edge in step S604, it is calculated by the following equation as an average value of neighboring G pixels or B pixels.

  Note that the value of the red pixel when the processing pixel is a blue pixel can be calculated in the same manner as Equation (6).

  In step S606, the pixel interpolation processing unit 207 applies adaptive interpolation to the processing pixel. Similarly, a method for calculating G33 and B33 corresponding to the pixel position of R33 will be described. Since the processing pixel is a pixel determined to be an edge in step S604, first, the vertical and horizontal pixel difference values DV and DH are calculated in order to obtain the edge direction.

  Next, based on the magnitude relationship between DV and DH, the pixel value of G33 is calculated by the following equation.

  In this way, by changing the pixel to be averaged according to the edge direction, pixel interpolation that preserves the edge of the image becomes possible. Similarly, in the case of blue pixel interpolation, the pixel difference values DN and NP are calculated by the following equation.

  Next, based on the magnitude relationship between DN and DP, the pixel value of B33 is calculated by the following equation.

  In this embodiment, simple linear interpolation and adaptive interpolation methods have been described. However, if two or more different interpolation methods are applied based on the edge information acquired from the noise reduction processing in step S303, a desired interpolation method can be obtained. You can use it. For example, the method described in Patent Document 1 can also be used.

  In step S <b> 607, the pixel interpolation processing unit 207 determines whether the pixel interpolation processing for all pixels has been completed. If completed, the process proceeds to step S608, and if not completed, the process proceeds to step S602 to continue the process.

  In step S608, the pixel interpolation processing unit 207 outputs the image data after the pixel interpolation processing.

  With the above processing, pixel interpolation processing can be applied to the image data after noise reduction. Therefore, an image with reduced image quality deterioration can be provided. Also, pixel interpolation processing can be performed with reference to image edge information obtained by noise reduction processing. As a result, it is possible to improve image quality by using the edge information obtained by the noise reduction process for the pixel interpolation process while reducing the load related to the edge detection of the image in the development process.

<Other examples>
The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, or the like) of the system or apparatus reads the program. It is a process to be executed.

Claims (14)

Noise reduction means for performing noise reduction processing on RAW image data;
An image processing apparatus comprising: pixel interpolating means for generating color image data by subjecting image data subjected to noise reduction processing by the noise reducing means to pixel interpolation processing.   The said noise reduction means determines the weight of the reference pixel of the vicinity of the attention pixel in the said RAW image data, and correct | amends this attention pixel using the weight of the determined reference pixel. Image processing device.   The image processing according to claim 2, wherein the noise reduction unit determines a weight of the reference pixel based on a similarity between the target pixel and a reference pixel or a similarity between the target region and a reference region. apparatus.   The image processing apparatus according to claim 3, wherein the noise reduction unit generates edge information indicating whether or not the pixel of interest is an edge based on information on the determined weight.   The image processing apparatus according to claim 4, wherein the edge information is generated based on the similarity.   The image processing apparatus according to claim 4, wherein the edge information is generated based on a difference between the pixel of interest and a pixel corrected by the noise reduction unit.   The image processing apparatus according to claim 4, wherein the pixel interpolation unit performs the pixel interpolation process using edge information generated by the noise reduction unit.   The image according to any one of claims 4 to 6, wherein the pixel interpolation unit determines an interpolation method of a pixel to be subjected to pixel interpolation processing based on the edge information generated by the noise reduction unit. Processing equipment.   The image processing apparatus according to claim 1, wherein the RAW image data is image data captured by an imaging apparatus having a Bayer array color filter. Input means for inputting RAW image data;
Noise reduction means for outputting image data obtained by reducing noise of the input RAW image data, and edge information of each pixel of the RAW image data;
Determining means for determining a pixel interpolation method for each pixel of the image data with reduced noise output by the noise reducing means based on edge information of each pixel output by the noise reducing means;
An image processing apparatus comprising: a pixel interpolation unit that interpolates each pixel of the image data with reduced noise using the pixel interpolation method determined by the determination unit. Input means for inputting RAW image data;
An image processing apparatus comprising output means for outputting color image data from the input RAW image data,
The color image data output by the output means is more than color image data obtained by performing pixel interpolation processing on the input RAW image data and performing noise reduction processing on the color image data after pixel interpolation processing. Color image data in which noise is reduced and edges of edge portions are enhanced.
An image processing apparatus.   An imaging apparatus including the image processing apparatus according to claim 1. A noise reduction step of performing noise reduction processing on the RAW image data;
An image processing method comprising: a pixel interpolation step of generating color image data by subjecting the image data subjected to noise reduction processing by the noise reduction step to pixel interpolation processing.   A program for causing a computer to function as the image processing apparatus according to any one of claims 1 to 11.

Images