JP2018195189A - Image processing device, image processing method, and program - Google Patents

Abstract

To provide a technology capable of increasing suppression efficiency of noises of an edge periphery in a noise suppression process.SOLUTION: An image processing device includes: reduction means for reducing an inputted image and generating a reduced image; first detection means for generating a first edge signal by performing an edge detection for the inputted image; second detection means for performing the edge detection for the reduced image and generating a second edge signal; and calculation means for calculating a synthesis ratio between the inputted image and the reduced image based on a third edge signal acquired by correcting the first edge signal based on the second edge signal. When the second edge signal exceeds a predetermined value, the calculation means corrects the first edge signal so that a degree of the influence of the second edge signal becomes small as compared with the case when not exceeding the predetermine value.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an image processing apparatus, an image processing method, and a program.

  In recent years, there has been a trend toward miniaturization of pixels of an image sensor, and noise in image signals obtained from the image sensor tends to increase due to the miniaturization of pixels. Therefore, as a method for suppressing this noise by signal processing, a method using multi-rate signal processing is known. For example, there is a method in which an image signal is divided into a plurality of frequency components, and noise of each divided image is suppressed and then recombined (see Patent Document 1). In Patent Document 1, an image signal is divided into a plurality of frequency component images, and noise suppression (NR) processing is performed on each of the divided images. There are various frequency component dividing methods. For example, a low-frequency image can be generated by pre-filtering the original image and then reducing the size to half. The high-frequency image can be generated by enlarging the low-frequency image to the same size as the original image and obtaining a difference between the original image and the enlarged image. By dividing the image into a plurality of frequency components in this way, it becomes possible to apply noise suppression processing suitable for the frequency component to each divided image. Further, by reducing the image size, noise can be reduced with a small filter length.

  When noise suppression is performed, the high-frequency noise is visually inconspicuous due to the fine granularity of the noise, while the low-frequency noise is visually noticeable due to the large granularity, and is lower than the high-frequency noise. It is desirable to remove the noise strongly. In addition, the composition processing of the divided image can be executed by a simple operation such as enlarging the low-frequency image and adding it to the high-frequency image. However, if the divided image is blurred by the noise suppression processing, The blur of the corresponding frequency band also remains as it is. Therefore, in the method of suppressing noise by dividing into frequency components, it is necessary to suppress the noise suppression effect to such an extent that blur can be allowed in each frequency band while performing different noise suppression for each frequency component.

  As another noise suppression method, first, a plurality of reduced images are generated from an image signal, noise suppression processing is performed on the plurality of reduced images, and edge signals are extracted to determine a composition ratio of each image based on the edge signals. There is a method of calculating and synthesizing a plurality of image signals based on the synthesis ratio (see Patent Document 2). In this method, a reduced image in which the horizontal and vertical sizes are each halved and a reduced image in which the size is ¼ are generated. Then, noise suppression processing is performed on each image, and an edge signal is extracted from the reduced image. In image synthesis, the edge signal is enlarged to the original size, the composition ratio is determined based on the enlarged edge signal, a larger size image is used for the strong part of the edge signal, and the weak part of the edge signal is reduced. The image is enlarged to the original size.

  As a result, the original size image is often used for the edge portion of the combined image, and conversely, the reduced image is not often used. For this reason, even if the noise suppression process is applied to the reduced image and the edge portion is blurred, the final image is not affected. Therefore, it is possible to apply strong noise suppression processing to the reduced image, and it is possible to obtain an image with an enhanced noise suppression effect of low-frequency components without blurring the edge as a combined image.

JP 2008-15741 A JP 2009-199104

  However, in the synthesis ratio calculation method described in Patent Document 2, since the edge signal detected from the reduced image is enlarged, the edge signal becomes thicker than the actual edge. As a result, a range wider than the actual edge range in the original size image is detected as an edge. Since the synthesis processing is performed with the fat edge signal, an upper layer image is used around the edge, and noise may remain around the edge.

  Therefore, the present invention provides a technique that makes it possible to enhance the effect of suppressing noise around edges in noise suppression processing.

In order to achieve the above object, the present invention provides an image processing apparatus comprising:
Reduction means for reducing the input image and generating a reduced image;
First detection means for performing edge detection on the input image to generate a first edge signal;
Second detection means for performing edge detection on the reduced image to generate a second edge signal;
Calculation means for calculating a synthesis ratio of the input image and the reduced image based on a third edge signal obtained by correcting the first edge signal based on the second edge signal; When the second edge signal exceeds a predetermined value, the calculating means is configured to reduce the degree of influence of the second edge signal as compared with a case where the second edge signal does not exceed the predetermined value. Correct the edge signal.

  ADVANTAGE OF THE INVENTION According to this invention, the technique for improving the noise suppression effect of the edge periphery in noise suppression processing can be provided.

1 is a block diagram showing a configuration example of an image processing apparatus corresponding to an embodiment of the invention. The block diagram which shows the structural example of the noise suppression process part classified by band corresponding to embodiment of invention. 5 is a flowchart showing an example of processing in the image processing apparatus corresponding to the embodiment of the invention. The figure for demonstrating the noise suppression process corresponding to embodiment of invention, the figure for demonstrating the correction process of a 2nd edge signal, and the figure for demonstrating the calculation process of a synthetic | combination ratio.

  DESCRIPTION OF EMBODIMENTS Hereinafter, embodiments for carrying out the present invention will be described in detail with reference to the accompanying drawings.

  FIG. 1 is a block diagram illustrating a configuration example of an image processing apparatus 100 corresponding to an embodiment of the invention. In this embodiment, a digital camera will be described as an example of the image processing apparatus 100. However, the image processing apparatus 100 is not limited to a digital camera as long as it can perform noise suppression processing on an image signal. It can also be realized as an arbitrary information processing apparatus such as a telephone, a smartphone, a PDA, a tablet terminal, a portable media player, or an imaging apparatus. FIG. 1 shows a configuration including the A / D converter 104 from the optical system 101 in consideration of the case of functioning as a digital camera or the like. However, the image processing apparatus 100 may be realized in a form that does not have a configuration from the optical system 101 to the A / D converter 104. In the image processing apparatus 100 of FIG. 1, each block may be configured by hardware using a dedicated logic circuit or memory except for a specific physical device such as an image sensor or a display element. Alternatively, the processing program stored in the memory may be configured by software by a computer such as a CPU executing the processing program.

  In FIG. 1, an optical system 101 includes a lens group including a zoom lens and a focus lens, an aperture device, and a shutter device. The optical system 101 adjusts the magnification, focus position, or light amount of the subject image that reaches the image sensor 102. The imaging element 102 is a photoelectric conversion element such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor) image sensor, and converts an object image into an electrical signal to generate an image signal. The pre-processing circuit 103 includes a CDS (Correllated Double Sampling) circuit and an amplifier circuit. The CDS circuit suppresses dark current contained in the image signal generated by the image sensor 102, and the amplifier circuit amplifies the image signal output from the CDS circuit. The A / D converter 104 converts the image signal output from the preprocessing circuit 103 into a digital image signal and outputs the digital image signal to the image processing unit 105.

  The image processing unit 105 performs predetermined image processing including white balance processing, noise reduction (NR) processing, tone conversion processing, edge enhancement correction processing, and the like on the input image signal, The obtained image signal is output as a luminance signal Y and color difference signals U and V. The image processing unit 105 can also calculate a brightness value of the subject and a focus value indicating the focus state of the subject from the image signal. Note that the image processing unit 105 can perform similar image processing not only on the image signal output from the A / D converter 104 but also on the image signal read from the recording medium 109.

  The control unit 106 controls the operation in each block constituting the digital camera 100 corresponding to this embodiment, and controls the operation of the digital camera 100. Based on the luminance value obtained from the image signal processed by the image processing unit 105 and the instruction input from the operation member 110, drive control of the optical system 101 and the image sensor 102 is also performed. The display memory 107 is a memory that temporarily stores an image signal that is a source of an image displayed on the display device 108. The display device 108 is configured by a liquid crystal display or an organic EL (Electro Luminescence) display, and displays an image using an image signal generated by the image sensor 102 or an image signal read from the recording medium 109. The display device 108 can also function as an electronic viewfinder by updating and displaying continuous image signals read from the image sensor 102 as needed. The display device 108 also displays character information such as the status display of the digital camera 100, the shutter speed, aperture value, sensitivity information determined on the user side or the digital camera 100 side, and the luminance distribution measured by the image processing unit 105. A graph or the like can also be displayed.

  The recording medium 109 may be configured to be detachable from the digital camera 100 or may be built in the digital camera 100. The operation member 110 is a member that a user operates to input a predetermined instruction to the digital camera 100, and can be configured by various buttons, switches, dials, touch panels, and the like. The bus 111 is used to exchange image signals among the image processing unit 105, the control unit 106, the display memory 107, and the recording medium 109.

  Next, an example of an operation at the time of shooting of the digital camera 100 in the present embodiment will be described. When the operation member 110 is operated by the user and an instruction to start shooting preparation is input, the control unit 106 starts controlling the operation of each block.

  First, the image sensor 102 photoelectrically converts a subject image transmitted through the optical system 101 to generate an analog image signal, and the analog image signal processed by the A / D converter 104 by the preprocessing circuit 103 is converted into a digital image. Convert to signal. The image processing unit 105 performs image processing such as white balance processing, noise suppression processing, gradation conversion processing, and contour correction processing on the image signal output from the A / D converter 104. The image signal processed by the image processing unit 105 is temporarily held in the display memory 107, and the image signal read from the display memory 107 is displayed on the display device 108 as an image. As described above, the display device 108 functions as an electronic viewfinder by updating the subject image in real time and displaying it on the display device 108 based on the image signals continuously generated by the image sensor 102.

  These processes are repeated until the user operates the shutter button included in the operation member 110. When the user operates the shutter button, the control unit 106 again adjusts the operation of the optical system 101 based on the luminance value and the focus value obtained by the image processing unit 105 to shoot a still image. The image processing unit 105 performs various image processing including noise suppression processing on the captured image signal of the still image, and a still image file based on the image signal output from the image processing unit 105 is stored on the recording medium 109. To be recorded.

  Next, the noise suppression processing in the image processing unit 105 will be described in detail. FIG. 2 is a block diagram illustrating a detailed configuration example of the band-specific noise suppression processing unit 200 included in the image processing unit 105. The noise suppression processing unit for each band 200 according to this embodiment includes a reduction processing unit 201, a noise suppression processing unit 202, an edge signal detection processing unit 203, a noise suppression processing unit 204, an edge signal detection processing unit 205, and an edge signal correction processing unit 206. An enlargement processing unit 207, an enlargement processing unit 208, a composition ratio calculation processing unit 209, and an image composition processing unit 210.

  In FIG. 2, the input image input to the band-specific noise suppression processing unit 200 is input in parallel to the reduction processing unit 201, the noise suppression processing unit 202, and the edge signal detection processing unit 203, and is processed. The band-specific noise suppression processing unit 200 includes two noise suppression processing units, a noise suppression processing unit 202 and a noise suppression processing unit 204, but the noise suppression processing unit 202 has an input of an original size that has not been reduced. The processing unit performs noise suppression on the image, and the noise suppression processing unit 204 is a processing unit that performs noise suppression on the image reduced by the reduction processing unit 201. Similarly, the band-specific noise suppression processing unit 200 includes two edge detection processing units, an edge signal detection processing unit 203 and an edge signal detection processing unit 205, but the edge signal detection processing unit 203 is reduced. The edge signal detection processing unit 205 is a processing unit that performs edge detection on the image reduced by the reduction processing unit 201. The image resulting from the noise suppression process and the edge signal generated by edge detection are combined through correction processing, enlargement processing, and combination ratio calculation processing corresponding to the present embodiment, and an output image is obtained.

  Hereinafter, the processing in the band-by-band noise suppression processing unit 200 corresponding to the present embodiment will be described in more detail with reference to the flowchart of FIG. 3. FIG. 3 is a flowchart illustrating an example of a processing procedure performed by the band-specific noise suppression processing unit 200. The processing corresponding to the flowchart can be realized, for example, by executing a program (stored in a ROM or the like) corresponding to one or more processors functioning as the image processing unit 105 and the noise suppression processing unit 200 for each band. Upon receiving the image signal output from the A / D converter 104, the image processing unit 105 performs pixel color interpolation processing, and then performs noise-by-band noise suppression processing unit 200 for each color (for example, R (red (red)). ), G (green), and B (blue) color components). Note that white balance processing, color conversion processing, gradation conversion processing, and the like are omitted.

  First, in step S <b> 301, the reduction processing unit 201 generates a reduced image in which pixels are thinned out to ½ size in the horizontal and vertical directions after pre-filtering the input image. As the prefilter, the coefficient of [1 2 1] is applied to both horizontal and vertical. The coefficient is a coefficient for averaging (smoothing) the weighting for the target pixel and the weighting for each pixel adjacent to both sides of the target pixel at a ratio of 2: 1. Thereby, the reduced image is generated as an image composed of low frequency components in the input image.

  Next, in S302, the noise suppression processing unit 202 performs noise suppression processing on the same input image as the input image processed by the reduction processing unit 201 in S301. Hereinafter, the noise-suppressed image generated at this time is referred to as a first NR image. In the noise suppression process, for example, a 7 × 7 pixel reference area is set around the target pixel as shown in FIG. 4A, and a difference from the target pixel is obtained for each pixel in the reference area. Then, the average value is calculated only with pixels whose difference is within the threshold, and the average value is output as the value of the target pixel. By performing the noise suppression process for each pixel, it is possible to obtain an output image in which noise is suppressed for the entire input image to be processed. The threshold value may be changed for each image, and the threshold value may be changed according to the signal level in consideration of light shot noise of the sensor.

  In step S <b> 303, the edge signal detection processing unit 203 detects an edge signal for the input image. The edge signal detected at this time is hereinafter referred to as a first edge signal. The detection of the first edge signal is performed by applying a Laplacian filter shown in Formula 1 to each pixel of the input image. The filter output result for each pixel becomes the edge signal of the pixel, and the filter output results of all the pixels can be combined into the first edge signal of the input image.

  In step S <b> 304, the noise suppression processing unit 204 performs noise suppression processing on the input image (reduced image) reduced by the reduction processing unit 201. The noise suppression processing method is the same as the processing content of the noise suppression processing unit 202. The reduced image that has been subjected to the noise suppression process generated at this time is hereinafter referred to as a second NR image. In step S <b> 305, the edge signal detection processing unit 205 detects an edge signal for the reduced image. The edge signal detection method is the same as the processing content of the edge signal detection processing unit 203. The edge signal detected at this time is hereinafter referred to as a second edge signal.

  Although the processing from S301 to S305 has been described in order, the execution order of each step need not follow the description order of the flowchart of FIG. For example, S301 to S303 may be executed in parallel, and S304 and S305 may be executed in parallel after the reduced image is generated in S301.

  In step S <b> 306, the edge signal correction processing unit 206 performs correction processing corresponding to the present embodiment on the second edge signal detected by the edge signal detection processing unit 205. The edge signal correction processing unit 206 outputs the value of the input signal as it is in a range where the magnitude of the input second edge signal is smaller than the predetermined value E according to the relationship shown in FIG. On the other hand, when the magnitude of the second edge signal exceeds a predetermined value, the value of the input signal is corrected to a value not exceeding the predetermined value E. At this time, the value can be corrected so as to decrease as the degree of the value of the second edge signal exceeding the predetermined value E increases. Alternatively, in a range smaller than the predetermined value E, the output edge signal is corrected so as to increase as the input edge signal increases, and when the predetermined value E is exceeded, the output edge signal becomes smaller than the predetermined value E. You may correct | amend as follows. In this way, the second edge signal obtained from the reduced image is corrected so that the signal value does not exceed the predetermined value E. In the present embodiment, the description is made with one broken line, but the correction method is not limited to this. For example, it suffices if the value of the output signal is reduced when the value of the input signal exceeds a predetermined value E. As an example, the output signal may not be linearly reduced as shown in FIG. 4B, but may be a non-linear form that decreases stepwise. Or you may correct | amend uniformly so that the value exceeding the predetermined value E may be made into the fixed value (it may also contain 0) below this predetermined value.

  In step S <b> 307, the enlargement processing unit 207 performs enlargement processing by bilinear interpolation so that the second NR image generated by the noise suppression processing in the noise suppression processing unit 204 has the same size as the input image. In step S <b> 308, the enlargement processing unit 208 performs enlargement processing by bilinear interpolation so that the second edge signal of the reduced image corrected by the edge signal correction processing unit 206 has the same size as the input image. The processing content of the enlargement processing unit 208 is the same as the processing content of the enlargement processing unit 207. The enlarged edge signal obtained at this time is hereinafter referred to as a third edge signal.

  Next, in S309, the composition ratio calculation processing unit 209 uses the first edge signal from the input image generated in S303 and the third edge signal generated in S308 to use the image used in the image composition processing unit 210. A composition ratio calculation process for composition is performed. The synthesis ratio calculation processing unit 209 performs addition processing of the first edge signal and the third edge signal to generate a fourth edge signal used for calculation of the synthesis ratio, and FIG. The composition ratio is calculated according to the relationship shown in FIG. In the present embodiment, the value corresponding to each pixel of the fourth edge signal is compared with the first threshold value TH1 and the second threshold value TH2. If the edge amount of the pixel indicated by the fourth edge signal is equal to or smaller than the first threshold, the composition ratio of the first NR image is set to 0% of the two images to be synthesized, and the second NR image is synthesized. The ratio is 100%. On the other hand, if it is larger than the second threshold, it is set to 100% of the first NR image, and the composition ratio of the second NR image is set to 0%. In the range where the value of the fourth edge signal is larger than the first threshold and equal to or smaller than the second threshold, the composition ratio is calculated linearly according to the edge amount.

  As described above, in this embodiment, the second edge signal is corrected so as to reduce the high contrast edge signal that is larger than the predetermined value E, that is, included in the first edge signal. Therefore, it is possible to effectively prevent the edge signal around the high contrast edge from being enlarged by the enlargement process of S308. Therefore, it is possible to make the composition ratio calculated around the edge of the high contrast thin, and to make the composition ratio of the reduced image from the low range dominant. In this example, the second edge signal subjected to the correction process is enlarged to generate a third edge signal, and the third edge signal is combined with the first edge signal to generate a fourth edge signal. Although the explanation is given, it is not limited to this. The second edge signal before the correction process is enlarged and combined with the first edge signal, and for the edge signal, the pixel whose value of the second edge signal exceeds the predetermined value E is You may make it correct | amend so that the level of the applicable edge signal may be lowered | hung. Alternatively, a gain based on the second edge signal before the correction process is applied to the first edge signal to amplify the first edge signal, and the value of the second edge signal is a predetermined value E. For pixels that exceed this value, the gain may be set small. That is, when the value of the second edge signal exceeds the predetermined value E, the first edge signal may be corrected so that the degree of influence of the second edge signal is smaller than when the value does not exceed the predetermined value E.

  Next, in S310, the image composition processing unit 210 performs composition processing on the first NR image from the noise suppression processing unit 202 and the enlarged second NR image from the enlargement processing unit 207 according to the composition ratio calculated in S309. The output signal of the noise suppression processing unit 200 for each band is generated. In the synthesis process, the first NR image and the enlarged second NR image are weighted and added for each pixel according to the synthesis ratio. By using this composition ratio, the use ratio of the first NR image is increased when the edge amount is large, so that the sense of resolution is maintained, and the use ratio of the second NR image is increased when the edge amount is small. Increases effectiveness. Note that the first NR image is used for the portion of the first edge signal in the vicinity of the high contrast edge, while the second NR image having a high noise suppression effect is used in the periphery of the edge to reduce the noise step around the edge. Can do.

  According to the above embodiment, when calculating the synthesis ratio using the edge signal detected from the reduced image, the noise level around the edge with high contrast is not noticeable, but the edge resolution and flatness are obtained. Noise reduction processing that achieves both noise suppression effects can be realized. In the above-described embodiment, an example in which one reduced image is generated for an input image and the two images are combined has been described. However, the present invention is not limited to this. The number of images to be generated may be two or more as long as a plurality of images having a difference in frequency components included in each image are generated from one image and the plurality of images are combined. Absent. In addition, as in the above-described embodiment, since the noise in the high-frequency component is suppressed in the reduced image compared to the input image, at least in the pixel where the reduced image is synthesized, the noise in the high-frequency component compared to the input image. Will decrease. Therefore, although the degree of the effect as noise reduction is reduced, the noise suppression processing unit can be omitted.

(Other examples)
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

DESCRIPTION OF SYMBOLS 100: Digital camera, 101: Optical system, 102: Image pick-up element, 103: Pre-processing circuit, 104: A / D converter, 105: Image processing part, 106: Control part, 107: Display memory, 108: Display apparatus 109: Recording medium, 110: Operation member, 111: Bus

Claims (11)

Reduction means for reducing the input image and generating a reduced image;
First detection means for performing edge detection on the input image to generate a first edge signal;
Second detection means for performing edge detection on the reduced image to generate a second edge signal;
Calculation means for calculating a synthesis ratio of the input image and the reduced image based on a third edge signal obtained by correcting the first edge signal based on the second edge signal;
When the second edge signal exceeds a predetermined value, the calculating means is configured to reduce the degree of influence of the second edge signal as compared with a case where the second edge signal does not exceed the predetermined value. An image processing apparatus that corrects an edge signal of the image. The calculating means includes
Correction means for correcting a value larger than the predetermined value to a value not exceeding the predetermined value for the second edge signal;
Enlarging means for enlarging the corrected second edge signal to a size corresponding to the input image to generate an enlarged edge signal;
The image processing apparatus according to claim 1, wherein the third edge signal is generated by adding the first edge signal and the enlarged edge signal.   3. The image according to claim 2, wherein the correction unit corrects the value of the second edge signal exceeding the predetermined value so that the value decreases as the degree of exceeding the predetermined value increases. Processing equipment.   The image processing apparatus according to claim 2, wherein the correction unit corrects the value of the second edge signal exceeding the predetermined value to a constant value equal to or less than the predetermined value. First noise suppression means for performing noise suppression processing on the input image to generate a first NR image;
A second noise suppression means for performing a noise suppression process on the reduced image to generate a second NR image; the first NR image; and the second NR expanded to the size of the input image The image processing apparatus according to claim 1, further comprising a combining unit that combines an image with the combining ratio.   The calculation means is configured such that the smaller the value of the third edge signal, the higher the ratio of the enlarged second NR image in the composition ratio, and the greater the value of the third edge signal, the greater the value in the composition ratio. The image processing apparatus according to claim 5, wherein the synthesis ratio is calculated so that a ratio of the first NR image is high. The calculating means includes
When the value of the third edge signal is equal to or lower than the first threshold, the ratio of the enlarged second NR image in the synthesis ratio is set to 100%,
When the value of the third edge signal is larger than a second threshold value that is larger than the first threshold value, the ratio of the first NR image in the synthesis ratio is set to 100%,
When the value of the third edge signal is greater than the first threshold and less than or equal to the second threshold, the first NR image at the synthesis ratio according to an increase in the value of the third edge signal The image processing apparatus according to claim 6, wherein the composition ratio is calculated so that the ratio of the image data increases.   The image processing apparatus according to claim 1, further comprising an imaging unit that captures a subject and generates the input image. It further comprises input means for inputting the input image,
The image processing apparatus according to claim 1, wherein the input unit reads and inputs the input image from a recording medium on which the input image is recorded. A reduction process for reducing the input image and generating a reduced image;
A first detection step of performing edge detection on the input image to generate a first edge signal;
A second detection step of performing edge detection on the reduced image to generate a second edge signal;
Calculating a composite ratio of the input image and the reduced image based on a third edge signal obtained by correcting the first edge signal based on the second edge signal;
In the calculating step, when the second edge signal exceeds a predetermined value, the first edge signal is less affected than when the second edge signal does not exceed the predetermined value. An image processing method characterized by correcting an edge signal.   The program for functioning a computer as each means of the image processing apparatus of any one of Claim 1 to 9.

Images