JP6644877B2 - Image processing device - Google Patents

Description

  The present invention relates to an image processing apparatus and an image processing method, and more particularly to an image processing apparatus that interpolates RGB color information from a RAW image obtained from a single-plate sensor or the like.

  In recent years, the resolution of images generally used in the field of video surveillance has shifted from the conventional analog resolution (720 pix × 480 pix) to the Full HD resolution (1920 pix × 1080 pix), and the image quality has been dramatically improved. In the future, there is a trend to further increase the resolution. In addition, in a social situation where safety and security are required, in the field of surveillance, in addition to faithfully reproducing images captured by cameras, images captured by cameras can be used to detect people, animals, objects (for example, automobiles and motorcycles). Image processing techniques for detecting various types of information have also been improved. Among such image processing, a technique for improving the visibility and fidelity of an image has recently been regarded as important.

  In general, a CCD sensor or a CMOS sensor includes a single-plate system in which each color of RGB is captured by one sensor, and a multi-plate system in which RGB is optically separated. A color image is generated using RGB color information (R pixel value, G pixel value, B pixel value) for each pixel. However, in a single-plate sensor, a value is obtained through a primary color filter. Only one color information is output at the pixel position. An image that is read from this sensor and lacks pixel information is called a RAW image. A Bayer pattern is often used as a primary color filter in a sensor that generates this RAW image.

  As shown in FIG. 1, two color information out of the three color information is missing when the RAW image is left as it is. Therefore, in order to compensate for the two missing pieces of color information, a process of interpolating RGB color information called demosaicing is performed. As the demosaicing process, the simplest method is linear interpolation, in which missing color information to be interpolated is interpolated from surrounding color information of the same color. That is, when the color information of R is missing, interpolation is performed using the average value of the surrounding R pixels. In a single region having a small frequency component of an image, good interpolation can be performed using linear interpolation. However, in an area where an edge portion of an image having a high frequency component exists, color information called false color, which the subject does not originally have, is interpolated.

  The false color will be described with reference to FIG. Assume that color information at pixel position 2 is missing and is interpolated by linear interpolation. Interpolation is performed using the color information P1 at the pixel position 1 and the color information P3 at the pixel position 3 by linear interpolation. As a result, the interpolated color information P2a becomes an average value of the two pieces of color information P1 and P3. At this time, as shown in the drawing, for example, if the color information P2b originally possessed by the subject is a value larger than the interpolated color information P2a, the interpolated color information P2a becomes color information originally possessed by the subject and a false color Has occurred. Thus, false colors are likely to occur near the edges.

  In order to prevent false colors, there is a method of changing an interpolation processing method according to an edge direction of an image. For example, an ACPI method (adaptive color brain interpolation method) is known. In the ACPI method, an edge direction is evaluated from a RAW image, and interpolation is performed using pixels arranged in a direction in which color information changes little. Although false colors can be suppressed by this, one of only a limited number of interpolation directions that is as orthogonal as possible to the edge gradient direction is selected, and false colors cannot be completely prevented. .

  Furthermore, in the single-plate sensor, a color moiré phenomenon occurs in addition to the false color. Color moire is a phenomenon that occurs in a subject image having high frequency components. In the case of a single-plate sensor, the demosaicing process is performed with four R, G, G and B pixels as one set. At this time, the number of R / B pixels is only 1/4 the resolution of the number of pixels. For this reason, aliasing occurs with respect to the high-frequency component, and the color information cannot be correctly reproduced, or a periodic shading pattern occurs.

  Various techniques have been proposed for the demosaicing process, and techniques for reducing noise and the like generated thereby and improving image quality are known (for example, see Patent Documents 1 and 2).

JP 2005-354610 A Japanese Patent No. 4520886

Jim S Jimmy Li and Sharmil Randhawa, "Color filter array demosaicking using high-order interpolation techniques with a weighted median filter for sharp color edge preservation", IEEE Transactions on Image Processing, September 2009, vol.18, No.9, pp.1946-1957 Xin Li, "Demosaicing by Successive Approximation", IEEE Transactions on Image Processing, March 2005, vol.14, No.3, pp.370-379

  In the conventional demosaicing process, it is not possible to reliably suppress false colors to such an extent that they are not visually noticeable. In terms of reliable suppression, there is a method of applying a common spatial LPF (low-pass filter) to all the color channels, but the resolution is reduced. Recently, techniques using super-resolution or deep learning have been reported, but there is a problem that a processing load is too large as a measure against a problem occurring in a very small part of an image area.

  The present invention has been made in view of such a situation, and has as its object to sufficiently suppress visually conspicuous false colors with a relatively small processing amount.

The present invention provides a demosaicing processor that acquires a RAW image and performs a demosaicing process, and a median filter using a predetermined pixel area for an R pixel value and a B pixel value with respect to a processing result of the demosaicing processor. And a filtering unit for performing the filtering a plurality of times.
Further, the RAW image has an RGB array in which certain unit patterns are periodically arranged (having translational symmetry in two dimensions), and the filtering unit performs processing before the median filter is applied. For the R pixel value, the G pixel value, and the B pixel value obtained by the demosaicing process as the color information of each pixel, (R pixel value−G pixel value) and (B pixel value− G pixel value), the median filter may be applied to the calculated values, and the G pixel value used for the difference may be added to the filtered value.
In addition, the demosaicing processor may perform the demosaicing process using an adaptive color brain interpolation method.
Further, an image quality adjustment processor that executes one or more types of image quality adjustment processing may be provided between the demosaicing processor and the filtering unit. Here, what is called image quality adjustment processing includes not only image processing generally used in cameras such as γ correction, but also processing that greatly emphasizes image features such as dark area correction of night monitoring video. set to target.

  ADVANTAGE OF THE INVENTION According to this invention, the color moire etc. which arise by the demosaicing process of a RAW image can be reduced reliably.

The figure explaining the interpolation process of the single board type sensor output pixel concerning background art. The figure explaining the false color by linear interpolation concerning background art. FIG. 1 is a block diagram of an image processing system 1 according to an embodiment. 6 is a flowchart illustrating processing of the false color remover 14 according to the embodiment. 7 is a hardware implementation example of median filter processing by the false color remover 14. 7 shows another hardware implementation example of the median filter processing by the false color remover 14.

  Next, embodiments for implementing the present invention (hereinafter, simply referred to as “embodiments”) will be specifically described with reference to the drawings. In the present embodiment, color moiré is reduced by a non-linear iterative method using a median filter at a stage subsequent to the demosaicing process. The median filter uses pixels in an arbitrary m × n area.

  FIG. 3 is a block diagram illustrating a schematic configuration of the image processing system 1 according to the present embodiment. Here, the description focuses mainly on processing characteristic of the present embodiment.

  The image processing system 1 includes a camera 20 and an image processing device 10. The camera 20 has a single-plate sensor, and the sensor surface is covered with the above-described Bayer pattern on-chip color filter. The camera 20 has a RAW image output function and can output a RAW image to the image processing device 10. That is, the RAW image has the pixel data of the RGB array of the Bayer pattern.

  The image processing device 10 includes a RAW image input unit 11, a demosaicing processor 12, an image quality adjustment processor 13, a false color remover 14, and an image output unit 15.

  The RAW image input unit 11 acquires a RAW image from the camera 20. It is assumed that the RAW image has been subjected to black level correction, fixed pattern noise correction, and the like before being passed to the demosaicing processor 12.

  The demosaicing processor 12 performs a demosaicing process on the RAW image from the RAW image input unit 11 and outputs a color-separated RGB image. In this example, the ACPI method is used as the demosaicing processing, but another interpolation method such as a VNG (Variable Number of Gradients) method can be arbitrarily applied.

  The image quality adjustment processor 13 performs operations on luminance areas such as gamma correction, dark area (black) correction, highlight (knee) correction, tone mapping, and operation on hue and saturation areas such as independent color masking and haze removal. And various kinds of image processing useful for improving image quality and visibility, including an operation in a space area and an operation in a time area. Since the processing of the image quality adjustment processor 13 may enhance or restore a false color, it is considered that applying the false color removal processing by the median filter after the image quality adjustment processing is more effective.

  The false color remover 14 (iterative filtering unit) calculates a difference value between a G pixel and an R pixel and a B pixel in the RGB color information of the image data after the demosaicing process as an iterative process. A median filter of m × n pixels is repeatedly applied to the subsequent difference pixel data by the number of times designated by the user. As a result, color moire caused by the demosaicing process is reduced.

  The image output unit 15 outputs the processed image data in the false color remover 14 to a predetermined storage device or display device. In the case of RGB output (4: 4: 4 format), the image output unit 15 performs output without performing any particular conversion process, and when performing Y / C output, performs an RGB → YC conversion process. Output. The arithmetic expression used for the conversion process uses an arithmetic expression standardized such as BT.709.

  FIG. 4 shows the details of the iterative processing by the false color remover 14. The false color remover 14 acquires the user's instruction and sets the number of times of repetition (S10). If it has already been set, this setting process is skipped.

Next, the false color remover 14 calculates a difference value between the R pixel value and the B pixel value and the G pixel value at each pixel position of the RGB image after the demosaicing process (S11). That is, for example, when the pixel value X (R pixel value, G pixel value, B pixel value) of a certain pixel is X _i (R _i , G _i , B _i ), RG difference pixel data ΔX _i (R _i − G _i , B _i -G _i ) are calculated.

Next, the false color remover 14 performs a median filter process of m × n pixels on the image data represented by the calculated RG difference pixel data ΔX _i (R _i -G _i , B _i -G _i ). Is performed (S12). The median filter process is a process of calculating a median of data of m × n pixels (window) around a pixel to be processed, and employing the calculated median as a processing result. At this time, there is a case where m × n peripheral pixels do not exist at the image end. In this case, non-existing pixels are excluded from the median value calculation and the processing is performed. Alternatively, as a padding process, a process is performed by copying the pixel value at the end of the image so as to be m × n. The result of the median calculation does not change depending on whether or not padding is performed.

Thereafter, the false color remover 14 adds the G pixel value (Gi) subtracted in S11 to each pixel data ΔXf (Rf, Bf) after the median filter processing (S13), The pixel value is held as a processing result (S14). That is, (R _i , G _i , B _i ) is overwritten with (R _f + G _i , G _i , B _f + G _i ). The G pixel value is not processed and the input pixel value is held as it is.

  If the number of repetitions has not reached the set number (N of S15), the process returns to S11. At this time, in the processing after S11, the R pixel value, the G pixel value, and the B pixel value obtained in the previous processing are used. When the number of repetitions has reached the set number (Y in S15), the iterative method processing ends. Note that the processing of each step from S11 to S14 is performed for all pixels in the image. Pixel values that have existed since the time of the RAW image can be excluded from processing, but there is no significant difference in the effect.

  FIG. 5 shows a hardware implementation example of the median filter processing by the false color remover 14. The configuration in FIG. 5 corresponds to the operation for the R pixel values from S11 to S13 in the repetitive processing shown in FIG. 4, and the same applies to the B pixel values. The window size of the median filter is 3 × 3. Further, in order to perform the repetitive processing a plurality of times, the configuration in FIG. 5 may be simply cascaded.

First, R pixel values in the j-th iteration (R _i ^j), G pixel value (G _i ^j) is input to the subtracter 40, the subtracter 40 calculates the difference between them, the flip-flop (FF) 31 Is output to the 1H delay line 41. Each of the flip-flops (FF) 31 to 39 is a register that holds one pixel value, and holds a value of 3 × 3 pixels from which a median is calculated. The 1H delay lines 41 and 42 are digital delay lines that hold pixel values for one horizontal line, and can be configured by a first-in first-out (FIFO) memory or the like. Since the difference from the subtractor 40 can be negative, the bit depth is increased by one compared to before the subtraction.

The median filter 43 extracts the median value of the nine input pixel values and outputs the median value as a filtered pixel value (R _f ^j ). The median filter 43 is, for example, a hard-coded algorithm of a bubble sort, and can be realized by providing nine stages of parallel registers that can compare values between adjacent units and performing a pipeline operation.

2H + 2 tapped delay line 44, from the pixel values (R _i ^j) is input to a subtracter 40, a delay until the corresponding filtered pixel value (R _f ^j) is output from the median filter 43 The same delay is applied to the G pixel value (G _i ^j ). The adder 45 adds the filtered pixel value (R _f ^j) and delayed pixel values (R _i ^j), and outputs.

  When the configuration of FIG. 5 is implemented by an FPGA, the 1H delay lines 41 and 42 and the 2H + 2 tap delay line 44 can be configured using an embedded block RAM (EBR) inside the FPGA. One EBR has a capacity of about 18 Kbit or 36 Kbit, and can use a very high transfer bandwidth as compared with an external RAM. Other median filters 43 and the like can be configured by combining Programmable Function Units (PFU) or slices, which are basic units of the FPGA. As described above, since the false color remover 14 of the present example does not require a multiplier (DSP slice), it may be possible to easily incorporate the false color remover 14 using extra resources inside the FPGA. In an image having a high aspect ratio, that is, a horizontally long image, lateral chromatic aberration becomes conspicuous near both sides thereof. When trying to correct even the false color due to such aberration using a single type of window, it may be better to make the window horizontally long in accordance with the aspect ratio.

ここでは、客観的評価値を導入している。評価結果には、CIELAB表色系の色差式を用いて評価しており、差分値（ΔＥab）が小さいほど本来の被写体に近い画素値が得られていることを意味する。反復処理を複数回実施することで効果があることが客観的評価からも確認できる。具体的には、デモザイキング処理（ＡＣＰＩ）のみでは、差分値（ΔＥab）は1.94である。デモザイキング処理を行った結果に対して反復処理を１回のみ実施した結果、差分値（ΔＥab）は1.3095となり、色モアレの低減効果が確認された。そしてさらに反復処理を３回実施した結果、差分値（ΔＥab）は1.2663となり、複数回実施することで、さらに色モアレ低減の効果を確認することができた。Table 1 shows the measured values of the color moiré reduction effect by the false color remover 14 described with reference to FIGS.

Here, an objective evaluation value is introduced. The evaluation result is evaluated using a color difference formula of the CIELAB color system, and a smaller difference value (ΔEab) means that a pixel value closer to the original subject is obtained. It can be confirmed from an objective evaluation that the effect is obtained by performing the repetitive processing a plurality of times. Specifically, only in the demosaicing process (ACPI), the difference value (ΔEab) is 1.94. As a result of performing the iterative process only once on the result of the demosaicing process, the difference value (ΔEab) was 1.3095, and the effect of reducing color moiré was confirmed. Further, as a result of performing the repetition processing three more times, the difference value (ΔEab) was 1.2663. By performing the repetition processing a plurality of times, the effect of reducing the color moiré could be further confirmed.

  FIG. 6 shows another example of hardware implementation of the median filter processing by the false color remover 14. This example differs from FIG. 5 in that a switching median filter 48 is provided instead of the median filter 43. The same components as those in FIG. 5 are denoted by the same reference numerals, and description thereof will be omitted. In this example, the output of the FF 35 that outputs the value of the pixel corresponding to the center of the filter window is extracted and provided to the comparator 46 and the subtractor 49.

  Comparator 46 compares the center pixel value with a predetermined threshold and outputs the resulting Boolean logic value. The pixel address generator 47 controls the operation timing of the entire hardware in FIG. 6 and can output an address indicating where the center pixel value output from the FF 35 is located in the image.

  The switching median filter (SMF) 48 of the present example is an order statistical filter or a stack filter whose operation can be switched according to conditions given from the comparator 46 and the pixel address generator 47 and the like. As an example, if the central pixel value is less than the threshold, SMF48 operates as an α-trimmed average filter, where a sorted N (N = m × n, in this example, 9) column of pixel values Are discarded, and the arithmetic average of (1-α) N pixel values near the center is output. When the center pixel is a pixel existing in the Bayer array (that is, not a pixel generated by interpolation), the center pixel value is output without a median filter. In other cases, it operates as a normal median filter. When the trimmed average filter is used, in the R and B channels having relatively low sensitivity, the S / N in the low luminance area can be improved, and the color gradation can be precisely reproduced. Also, by maintaining the original signal from the Bayer array, the edges of the image can be preserved.

  In this example, the original signal from the Bayer array is output as it is. However, if the original signal is larger than the output of the filter, the original signal may be discarded and the output of the filter may be used. Note that the conditional filter operation is not limited to the above-described combinations, and as a condition, an evaluation result of noise or a high-frequency component may be used, and a Double Window Modified Trimmed Mean filter, a stack filter, an ε filter, etc. Good.

  The subtracter 49 subtracts the center pixel value from the FF 35 from the filtered pixel value from the SMF 48, and outputs the result to the multiplier 50 as a correction amount. The multiplier 50 multiplies the correction amount by a predetermined fixed constant and outputs the result to the subtractor 51 as a G correction amount. The subtractor 51 subtracts the G correction amount from the multiplier 50 from the G pixel value from the 2H + 2 tap delay line 44 and outputs the result. These configurations are provided to compensate for a change in luminance due to the correction of the R pixel value, and are not essential. The strength of the compensation is determined by appropriately choosing a predetermined fixed constant. The multiplier 50 that multiplies the fixed constant can be configured by a PFU or a slice.

  In this example, by using a filter with averaging, a pixel value having a decimal value is obtained, and the pixel value having the fixed-point value can be used during the iterative calculation inside the false color remover 14. And the output of the false color remover 14. For example, during color space conversion, a fixed-point pixel value can be multiplied by a BT.709 conversion coefficient.

  As described above, according to the image processing system 1 of the present embodiment, a color moiré reduction effect can be obtained by repeatedly processing a non-linear filter such as a median filter. The repetitive processing is not limited to performing the same processing continuously, and a part of the plurality of repetitive processing may be performed before the processing of the image quality adjustment processor 13 or the output from the image output unit 14 may be performed. After that, the processing may be performed by software processing using a personal computer or the like. In addition, the on-chip color filter is not limited to the Bayer filter, even when a complementary color filter having four colors as one set, or when one of the G pixels of the Bayer filter is replaced with a total transmission or infrared transmission filter, This is applicable if G pixel values (or pseudo luminance values) are generated at all pixel positions by interpolation.

1 Image processing system, 10 Image processing device, 11 RAW image input unit, 12 Demosaicing processor, 13 Image quality adjustment processor, 14 False color remover (iterative filtering unit), 15 Image output unit, 20 camera

  The present invention can be used by being incorporated into devices such as digital cameras and video cameras, as well as image editing software used in RAW development or post-production, and cloud services for storing and editing images. it can.

Claims (5)

A demosaicing processor for performing a demosaicing process on a RAW image obtained from an image sensor provided with a color filter having translational symmetry in two dimensions;
A false color remover that performs a median filter a plurality of times using a predetermined window for each R pixel value and B pixel value with respect to the processing result by the demosaicing processor;
The false color eliminator performs, as a process prior to applying the median filter, an R pixel value, a G pixel value, and a B pixel value obtained in the demosaicing process as color information of each pixel. (R pixel value−G pixel value) and (B pixel value−G pixel value) were calculated, the calculated value was subjected to the median filter, and the value after filtering was used as a difference. G pixel values are added,
Further, the median filter performed by the false color remover a plurality of times is a switching median filter, and operates as an α-trimmed average filter when a value of a center pixel of the window is smaller than a predetermined threshold value, or When the center pixel of the window is a pixel existing in the Bayer array, the operation of outputting the value of the center pixel as it is is performed,
The image processing apparatus according to claim 1, wherein the bit depth of a pixel value is increased as compared to before being input to the false color remover, while the median filter is performed a plurality of times. The RAW image has an RGB array of a Bayer pattern,
The demosaicing processor performs processing using an adaptive color brain interpolation method as the demosaicing processing,
Before passing the processing result by the demosaicing processing unit to the false color elimination unit , the image processing unit includes an image quality adjustment processing unit that performs image processing of at least a luminance region or a color region to improve image quality and visibility. The image processing apparatus according to claim 1.   The image processing apparatus according to claim 1, wherein the window is set in a horizontally long rectangular shape corresponding to an aspect ratio of the RAW image. Demosaicing processing means for performing demosaicing processing on a RAW image obtained from an image sensor provided with a color filter having translational symmetry in two dimensions;
A false color removing unit that performs a median filter a plurality of times using a predetermined window for each of the R pixel value and the B pixel value with respect to a processing result obtained by the demosaicing processing unit;
The false color removing unit performs, as a process prior to applying the median filter, an R pixel value, a G pixel value, and a B pixel value obtained by demosaicing as color information of each pixel. (R pixel value−G pixel value) and (B pixel value−G pixel value) were calculated, the calculated value was subjected to the median filter, and the value after filtering was used as a difference. G pixel values are added,
Further, the median filter executed multiple times by the false color removing means operates as an α-trimmed average filter when the value of the center pixel of the window is smaller than a predetermined threshold, or the center pixel of the window is , When the pixel exists in the Bayer array, the operation of outputting the value of the central pixel as it is is performed,
An image processing method according to an image processing apparatus, wherein the bit depth of a pixel value increases before the median filter is executed a plurality of times as compared to before being input to the false color removing unit. The image processing method according to claim 4, wherein the window is set in a horizontally long rectangular shape corresponding to an aspect ratio of the RAW image.

Images