JP4688753B2 - Interpolation processing apparatus and interpolation processing method - Google Patents

Description

  The present invention relates to an interpolation processing apparatus and an interpolation processing method.

  In the conventional interpolation processing apparatus and interpolation processing method, in an imaging apparatus having an image sensor in which red (R), green (G), and blue (B) color filters are arranged according to a Bayer array, G, The color component values of B or B, R, R, and G are insufficient. For example, as shown in Patent Document 1 below, based on the local pixel signal distribution for each color in order to improve the resolution. Thus, the pixel signal of each pixel is replaced with an average value, and an interpolation method based on the linear similarity ratio between the known color geometric figure and the insufficient color geometric figure is assumed.

JP 2001-197512 A

  This conventional method assumes that there is a positive correlation between each color component value (for example, an R component value, a G component value, and a B component value in a Bayer array) in a region near the interpolation target pixel. For this reason, there is a problem that interpolation cannot be appropriately performed in a region where there is no positive correlation between the color component values (for example, a boundary between one color and another color), and an interpolation error increases.

  The present invention provides an interpolation processing apparatus and an interpolation processing method that can always perform interpolation using an optimal interpolation method regardless of how color component values change in a region near the interpolation target pixel.

The present invention provides the first to Nth (N is a natural number of 2 or more) N different color components,
For a pixel having a Jth color component value (J is any one of 1 to N) (hereinafter referred to as an interpolation target pixel),
In an interpolation processing apparatus that obtains a color image by calculating a Kth color component value (K is any one of 1 to N excluding J),
An insufficient color low frequency component calculating means for outputting a low frequency component value of the Kth color component in the interpolation target pixel;
An insufficient color increment calculating means for outputting an increment value of the Kth color component in the interpolation target pixel;
Correlation determining means for determining whether a positive correlation, a negative correlation, or no correlation exists between the Jth color component and the Kth color component;
Based on the result of discrimination in the correlation discriminating means, the increment value of the Kth color component calculated by the insufficient color increment calculating means and the Kth color calculated by the insufficient color low frequency component calculating means. Computing means for computing the low-frequency component value of the component ;
From at least two or more directions around the interpolation target pixel, it has a edge detecting means for detecting the direction in which the presence of an edge,
An interpolation processing apparatus comprising: a calculation performed by the insufficient color low-frequency component calculation unit, the insufficient color increment calculation unit, and the correlation determination unit according to a detection result of the edge detection unit. provide.

  According to the present invention, it is possible to always perform interpolation by an optimal interpolation method regardless of how color component values change in a region near the interpolation target pixel.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  FIG. 1 is a block diagram showing a configuration of an image pickup apparatus provided with an interpolation processing apparatus according to an embodiment of the present invention.

  The light incident from the lens 1 forms an image on the imaging surface of the image sensor 2 constituted by, for example, a solid-state imaging device. The imaging signal output from the image sensor 2 is converted into a digital value by the A / D converter 3, stored in the frame memory 4, and further subjected to color interpolation processing in the interpolation processing circuit 5 and output as a video signal.

  Here, the imaging surface of the image sensor 2 is composed of a plurality of photoelectric conversion elements arranged two-dimensionally as shown in FIG. That is, each photoelectric conversion element is covered with one of color filters corresponding to three types of color components of red (R), green (G), and blue (B) according to the Bayer array. Each photoelectric conversion element corresponds to each pixel of an image captured by the imaging apparatus illustrated in FIG. 1, and the position occupied by each photoelectric conversion element on the imaging surface corresponds to the position of each pixel. In FIG. 2, the horizontal and vertical directions respectively correspond to the horizontal direction (H) and vertical direction (V) of the image, and each pixel is two-dimensionally arranged on the imaging surface. (Or HV plane) can be represented by coordinate values.

  That is, horizontal coordinate values (i−2, i−1, i, i + 1, i + 2, i + 3,...) Representing horizontal positions and vertical coordinate values (j−2, j−1, j) representing vertical positions. , J + 1, j + 2, j + 3,...)) Can be used to represent each pixel. For example, a pixel having a horizontal coordinate value i and a vertical coordinate value j can be represented by (i, j).

  Hereinafter, the operation will be described in detail with reference to FIG. In the following description, pixels corresponding to photoelectric conversion elements covered with R color filters are R pixels, pixels corresponding to photoelectric conversion elements covered with G color filters are G pixels, and B color filters. A pixel corresponding to the covered photoelectric conversion element is referred to as a B pixel.

  Each photoelectric conversion element of the image sensor 2 outputs an analog signal corresponding to any of R, G, and B color components. In other words, an analog signal corresponding to R is output from the photoelectric conversion element covered with the R color filter, and an analog signal corresponding to G is output from the photoelectric conversion element covered with the G color filter. An analog signal corresponding to B is output from the photoelectric conversion element covered with the filter. The analog signal output from each photoelectric conversion element is converted into a digital value by the A / D converter 3. Since each photoelectric conversion element corresponds to each pixel, the digital value output from the A / D converter 3 corresponds to the color component value of each pixel.

  The digital value output from the A / D converter 3 is written in the frame memory 4 as the color component value of each pixel. At this time, each color component value is written in association with the position of each pixel on the imaging surface, that is, the position on the HV coordinate plane.

  Here, since each of the photoelectric conversion elements is covered with a color filter corresponding to any one of the three color components R, G, and B, only one color component value is provided for each pixel. I can't get it. That is, only the R component value is obtained for the R pixel, only the G component value is obtained for the G pixel, and only the B component value is obtained for the B pixel. In order to obtain a color image, three types of R component value, G component value, and B component value are required for each pixel. Therefore, while the R component value is known for the R pixel, the G component value and B component value are known. The value is unknown, the G component value is known for the G pixel, while the B component value and the R component value are unknown, and the B component value is known for the B pixel, the R component Value and G component value are unknown. The interpolation processing circuit 5 obtains a color image by obtaining an unknown color component value in each pixel by interpolation.

  In the following description, a color for which a color component value is obtained in each pixel may be referred to as a known color component, and a color for which the color component value is unknown may be referred to as an insufficient color component. Further, the color component value of the known color component may be referred to as a known color component value, and the color component value of the insufficient color component may be referred to as an insufficient color component value.

  The color component values written in the frame memory 4 are distributed and written to the two-dimensional memories 6r, 6g, and 6b by the demultiplexer 5 for each of the R component value, the G component value, and the B component value. That is, the R component value is written into the two-dimensional memory 6r, the G component value is written into the two-dimensional memory 6g, and the B component value is written into the two-dimensional memory 6b. Here, in each of the two-dimensional memories 6r, 6g, and 6b, the color component value is written in association with the position of each pixel on the imaging surface, that is, the position on the HV coordinate plane.

  That is, the state in which the R component value is written in the two-dimensional memory 6r is shown in FIG. 3A, and the state in which the G component value is written in the two-dimensional memory 6g is shown in FIG. The state in which the B component value is written in the memory 6b can be expressed as shown in FIG. 3A to 3C, which show the state of the frame memory, the R component value obtained by the image sensor 2 is indicated by the capital letter G in the pixel indicated by the capital letter R. This indicates that the G component value obtained by the image sensor 2 is written in the pixel, and the B component value obtained by the image sensor 2 is written in the pixel indicated by the capital letter B.

  The frame memory 4 is necessary when the image sensor 2 is a so-called interlaced readout solid-state imaging device that reads out the color component values of each pixel shown in FIG. When the image sensor 2 is a so-called progressive readout type solid-state imaging device in which the color component values of each pixel shown in FIG. 2 are sequentially read out row by row, the signal read out from the image sensor 2 is directly decoded. The multiplexer 6 only needs to be distributed, and the frame memory 4 becomes unnecessary.

  When the insufficient color component value for each pixel is interpolated in the interpolation processing circuit 5, the color component value is also written in the two-dimensional memory 6r, 6g, 6b in association with the position of each pixel. When the interpolation of all insufficient color component values is completed, each color component value written in the two-dimensional memory 6r, 6g, 6b is output to the outside of the interpolation processing circuit 5 as a signal Sout.

  Hereinafter, the internal configuration of the interpolation processing circuit 5 will be described. The interpolation processing circuit 5 includes a demultiplexer 6, two-dimensional memories 6r, 6g, and 6b, difference calculation circuits 7h, 7v, 7p, and 7m, a comparison circuit 7c, low-pass filter circuits (Low Pass Filter, LPF) 9a and 9b, and a high-pass filter. Circuits (High Pass Filter, HPF) 8 a and 8 b, a coefficient determination circuit 11, change amount calculation circuits 10 a and 10 b, a divider 10 c, a subtractor 12, multipliers 13 and 14, and an adder 15.

Here, the operations of the demultiplexer 6 and the two-dimensional memories 6r, 6g, and 6b are as described above. The difference calculation circuits 7h, 7v, 7p, and 7m and the comparison circuit 7c constitute an edge detection circuit 7, and the high-pass filter circuits 8a and 8b and the coefficient determination circuit 11 constitute a correlation determination circuit 8, and a low-pass filter circuit 9a. The subtractor 12 constitutes a known color increment calculation circuit 20, the change amount calculation circuits 10a and 10b and the divider 10c constitute a ratio calculation circuit 10, and the ratio calculation circuit 10 and the known color increment calculation circuit 20 increase an insufficient color. A calculation circuit 30 is configured.
The detailed operation of each component will be described later together with the description of the operation of the interpolation processing circuit 5, that is, the calculation performed to interpolate the insufficient color component value for each pixel in the interpolation processing circuit 5. .

  The operation of the interpolation processing circuit 5, that is, the calculation performed to interpolate the insufficient color component value for each pixel in the interpolation processing circuit 5 will be described below. In the following description, the R component value for the pixel existing at the position of the coordinate (i, j) on the HV coordinate plane is R (i, j), the G component value is G (i, j), and B The component value is represented by B (i, j).

The interpolation processing circuit 5 interpolates the insufficient color component value for each pixel in the following order.
(1) Interpolation of G component value for R pixel and G component value for B pixel (2) Interpolation of R component value and B component value for G pixel (3) B component value for R pixel and R component value for B pixel interpolation

  First, a method of interpolating the G component value for the R pixel and the G component value for the B pixel will be described. The description will be made from a method of interpolating G component values for R pixels.

  FIGS. 4A and 4B are diagrams showing the state in which R component values and G component values are written in the two-dimensional memories 6r and 6g, with the R pixel as the center. A method of interpolating the G component value G (i, j) for the R pixel existing at the position of the coordinate (i, j) on the HV plane will be described below.

  In the following description, a pixel that interpolates an insufficient color component value may be referred to as an interpolation target pixel.

First, the difference calculation circuit 7h generates a value dh that represents the difference in the horizontal direction of the image, the difference calculation circuit 7v displays a value dv that represents the difference in the vertical direction of the image, the difference calculation circuit 7p, and the difference calculation circuit 7p increases the image to the right. The difference calculation circuit 7m calculates a value dm indicating the difference in the diagonally upward direction of the image. dh, dv, dp, dm are calculated by the following equations.
dh = | G (i−1, j) −G (i + 1, j) |
dv = | G (i, j-1) -G (i, j + 1) |
dp = | G (i−1, j) −G (i, j−1) |
+ | G (i, j + 1) -G (i + 1, j) |
dm = | G (i-1, j) -G (i, j + 1) |
+ | G (i, j-1) -G (i + 1, j) |

  Next, the comparison circuit 7c determines in which direction the edge exists with the interpolation target pixel as the center. This process is performed according to the flow shown in FIG. Here, an edge represents a place where a color changes from one color to another color in a color image. 6A to 6E show examples of various edges. 6A to 6E, the white portion and the hatched portion are different in color. 6A shows a case where no edge exists, FIG. 6B shows a case where an edge exists in the horizontal direction, FIG. 6C shows a case where an edge exists in the vertical direction, and FIG. (D) represents a case where an edge exists in a diagonally upward direction, and FIG. 6 (E) represents a case where an edge exists in a diagonally upward direction. Details of the coefficient determining circuit for determining the direction of the edge will be described below.

  First, in step ST01, dh and dv are compared with a threshold value γ1. When both dh and dv are smaller than the threshold γ1, the process proceeds to step ST02, and it is determined that no edge exists. In other cases, the process proceeds to step ST03 to determine in which direction the edge exists. First, in step ST03, the absolute value of the difference between dh and dv is compared with a threshold σ1. If the absolute value of the difference between dh and dv is smaller than the threshold σ1, the process proceeds to step ST04. Otherwise, the process proceeds to step ST07. In step ST04, dp and dm are compared to determine whether there is an edge in the diagonally upward direction or an edge in the diagonally upward direction. That is, if dp is smaller than dm, the process proceeds to step ST05, where it is determined that an edge exists in the diagonally upward direction. Otherwise, the process proceeds to step ST06, where it is determined that an edge exists in the diagonally upward direction. On the other hand, in step ST07, dh and dv are compared to determine whether an edge exists in the horizontal direction or an edge exists in the vertical direction. That is, if dh is smaller than dv, the process proceeds to step ST08, where it is determined that an edge exists in the horizontal direction, and otherwise, the process proceeds to step ST09, where it is determined that an edge exists in the vertical direction.

  The determination result of the comparison circuit 7c is output to the low-pass filter circuits 9a and 9b, the change amount calculation circuits 10a and 10b, and the coefficient determination circuit 11.

  Next, the operation of the high pass filter circuit 8a will be described. The high-pass filter circuit 8a calculates the high frequency component value of the known color component R. In addition, four types of values Ruhpf, Rbhpf, Rlhpf, and Rrhpf represented by the following formulas are calculated as the high frequency component values.

Ruhpf
= {-R (i-2, j-2) + 2R (i, j-2) -R (i + 2, j-2)} / 2
Rbhpf
= {-R (i-2, j + 2) + 2R (i, j + 2) -R (i + 2, j + 2)} / 2
Rlhpf
= {-R (i-2, j-2) + 2R (i-2, j) -R (i-2, j + 2)} / 2
Rrhpf
= {-R (i + 2, j-2) + 2R (i + 2, j) -R (i + 2, j + 2)} / 2

Next, the operation of the high pass filter circuit 8b will be described. The high pass filter circuit 8b calculates the high frequency component value of the insufficient color component G. Note that four types of values Guhpf, Gbhpf, Glhpf, and Grhpf represented by the following formulas are calculated as high-frequency component values.
Guhpf
= {-G (i-2, j-1) + 2G (i, j-1) -G (i + 2, j-1)} / 2
Gbhpf
= {-G (i-2, j + 1) + 2G (i, j + 1) -G (i + 2, j + 1)} / 2
Glhpf
= {-G (i-1, j-2) + 2G (i-1, j) -G (i-1, j + 2)} / 2
Grhpf
= {-G (i + 1, j-2) + 2G (i + 1, j) -G (i + 1, j + 2)} / 2

Next, the operation of the low-pass filter circuit 9a will be described. The low pass filter circuit 9a calculates a low frequency component value Rlpf of the known color component R. Here, the calculation formula of Rlpf is switched depending on in which direction the edge is present in the comparison circuit 7c (that is, according to the signal ED indicating the detection result from the comparison circuit 7c). That is, when it is determined that there is no edge, when it is determined that an edge exists in the diagonally upward direction, and when it is determined that an edge exists in the diagonally upward direction, Rlpf is calculated by the following equation.
Rlpf
= {R (i, j-2) + R (i-2, j) + 4R (i, j)
+ R (i + 2, j) + R (i, j + 2)} / 8
When it is determined that an edge exists in the horizontal direction, Rlpf is calculated by the following equation.
Rlpf
= {R (i-2, j) + 2R (i, j) + R (i + 2, j)} / 4
When it is determined that an edge exists in the vertical direction, Rlpf is calculated by the following equation.
Rlpf
= {R (i, j-2) + 2R (i, j) + R (i, j + 2)} / 4

  As described above, when the comparison circuit 7c determines that there is no edge, when it is determined that an edge exists in the diagonally upward direction, and when it is determined that an edge exists in the diagonally upward direction, interpolation is performed. (Same color component) of pixels that are aligned in the horizontal and vertical directions with respect to the target pixel and the interpolation target pixel and that are located in the vicinity of the interpolation target pixel (in the illustrated example, adjacent to each other with one other pixel in between) The low-frequency component value Rlpf of the known color component is calculated by performing LPF processing (calculation that takes an average or weighted average) using the pixel values.

  When the comparison circuit 7c determines that an edge exists in the horizontal direction, the pixel is interpolated in the horizontal direction with respect to the interpolation target pixel and the interpolation target pixel, and is located in the vicinity of the interpolation target pixel (in the example shown in the drawing, Low-frequency component values of known color components by performing LPF processing (calculation that takes an average or weighted average) using pixel values (of the same color component) of adjacent pixels with one other pixel in between Rlpf is calculated.

  When the comparison circuit 7c determines that an edge is present in the vertical direction, the pixel is interpolated in the vertical direction with respect to the interpolation target pixel and the interpolation target pixel, and is located in the vicinity of the interpolation target pixel (in the illustrated example, Low-frequency component values of known color components by performing LPF processing (calculation that takes an average or weighted average) using pixel values (of the same color component) of adjacent pixels with one other pixel in between Rlpf is calculated.

Next, the operation of the low-pass filter circuit 9b will be described. The low pass filter circuit 9b calculates the low frequency component value Glpf of the insufficient color component G. Here, the calculation formula of Glpf is switched according to which direction the edge is present in the comparison circuit 7c (that is, according to the signal ED indicating the detection result from the comparison circuit 7c).
Specifically, when it is determined that there is no edge, when it is determined that there is an edge in the diagonally upward direction, and when it is determined that an edge exists in the diagonally upward direction, Glpf is calculated by the following equation.
= {G (i, j-1) + G (i-1, j) + G (i + 1, j) + G (i, j + 1)} / 4
When it is determined that an edge exists in the horizontal direction, Glpf is calculated by the following equation.
Glpf
= {G (i-1, j) + G (i + 1, j)} / 2
When it is determined that an edge exists in the vertical direction, Glpf is calculated by the following equation.
Glpf
= {G (i, j-1) + G (i, j-1)} / 2

  As described above, when the comparison circuit 7c determines that there is no edge, when it is determined that an edge exists in the diagonally upward direction, and when it is determined that an edge exists in the diagonally upward direction, interpolation is performed. It is better to perform LPF processing (calculation that takes an average or weighted average) using signals (of the same color component) of adjacent pixels that are arranged in the target pixel and its upper, lower, left, and right sides and that do not place other pixels in between. A low frequency component value Glpf of G is calculated.

  When the comparison circuit 7c determines that an edge is present in the horizontal direction, the pixel is arranged in the horizontal direction with respect to the pixel to be interpolated and adjacent pixels (with the same color component) without any other pixels in between. ) The low frequency component value Glpf of the insufficient color component G is calculated by performing LPF processing (calculation that takes an average or weighted average) using the signal.

  If the comparison circuit 7c determines that an edge is present in the vertical direction, the pixel is arranged in the vertical direction with respect to the pixel to be interpolated, and adjacent pixels (with the same color component) without any other pixels in between. ) The low frequency component value Glpf of the insufficient color component G is calculated by performing LPF processing (calculation that takes an average or weighted average) using the signal.

Next, the operation of the change amount calculation circuit 10a will be described. The change amount calculation circuit 10a calculates the change amount Rdisp of the known color component R. Here, the calculation formula of Rdisp is switched according to which direction the edge is present in the comparison circuit 7c (that is, according to the signal ED indicating the detection result from the comparison circuit 7c). That is, when it is determined that no edge exists, Rdisp is calculated by the following equation.
Rdisp
= 1 + {| R (i, j-2) + R (i, j + 2) | + | R (i-2, j) -R (i + 2, j) |} / 2
If it is determined that there is an edge in the diagonally upward direction, Rdisp is calculated according to the following equation.
Rdisp
= 1 + {| (R (i, j-2) + R (i-2, j))-(R (i, j + 2) + R (i + 2, j)) |} / 2
If it is determined that an edge exists in the diagonally upward direction, Rdisp is calculated according to the following equation.
Rdisp
= 1 + {| (R (i, j-2) + R (i + 2, j))-(R (i, j + 2) + R (i-2, j)) |} / 2
If it is determined that an edge exists in the horizontal direction, Rdisp is calculated according to the following equation.
Rdisp = 1 + {| R (i-2, j) -R (i + 2, j) |} / 2
If it is determined that an edge exists in the vertical direction, Rdisp is calculated by the following equation.
Rdisp = 1 + {| R (i, j-2) -R (i, j + 2) |} / 2

  As described above, when the comparison circuit 7c determines that there is no edge, the pixels are arranged in the horizontal direction with respect to the interpolation target pixel, located on the opposite side with respect to the interpolation target pixel, and in the vicinity of the interpolation target pixel. Are located in the vertical direction with respect to the interpolation target pixel and located on the opposite side with the interpolation target pixel at the center. In addition, the similarity ratio “1/2” (described later) is added to the sum of the absolute values of the pixel value differences of pixels located in the vicinity of the interpolation target pixel (adjacent to the interpolation target pixel in the illustrated example). A product obtained by adding a predetermined constant, for example, 1 to the product is obtained as the change amount Rdisp of the known color component R.

  When the comparison circuit 7c determines that an edge is present in the diagonally upward direction, the line is parallel to the diagonally upward line passing through the interpolation target pixel and the diagonally upward line passing through the interpolation target pixel. Also, the sum of the pixel values of the pixels located on the upper left line and in the vicinity of the pixel to be interpolated is parallel to the diagonal line rising to the right passing through the pixel to be interpolated, and rising to the right through the pixel to be interpolated. The absolute value of the difference from the sum of the pixel values of pixels located on the lower right side of the diagonal line and in the vicinity of the pixel to be interpolated is multiplied by a similarity ratio of “1/2”. A value obtained by adding a predetermined constant, for example, 1 is obtained as the change amount Rdisp of the known color component R.

  When the comparison circuit 7c determines that there is an edge in the diagonally upward left direction, the line is parallel to the diagonally upward left line passing through the interpolation target pixel and from the diagonally upward left line passing through the interpolation target pixel. Also, the sum of the pixel values of the pixels located on the upper right line and in the vicinity of the pixel to be interpolated is parallel to the line in the diagonally upward direction passing through the pixel to be interpolated, and is left upward through the pixel to be interpolated. The absolute value of the difference between the sum of the pixel values of the pixels located on the lower left of the diagonal line and in the vicinity of the interpolation target pixel is multiplied by the similarity ratio of “1/2” Then, a predetermined constant, for example, 1 is added as the change amount Rdisp of the known color component R.

  When the comparison circuit 7c determines that there is an edge in the horizontal direction, the comparison circuit 7c is arranged in the horizontal direction with respect to the interpolation target pixel and is located in the vicinity of the interpolation target pixel (in the example shown, there is one in the middle). A value obtained by multiplying the absolute value of the pixel value difference (of the same color component) of a pixel adjacent to another pixel by the similarity ratio “1/2” and adding a predetermined constant, for example, 1 Is obtained as the change amount Rdisp of the known color component R.

  When it is determined that an edge exists in the vertical direction, the edges are aligned in the vertical direction with respect to the interpolation target pixel and are located in the vicinity of the interpolation target pixel (in the example shown in the figure, one other pixel is placed between them). A value obtained by multiplying the absolute value of the difference between the pixel values of the adjacent pixels (of the same color component) by the similarity ratio “½” and adding a predetermined constant, for example, 1 is the known color component R The amount of change Rdisp is obtained.

  Here, the “similarity ratio” is located in the vicinity of the interpolation target pixel from the interpolation target pixel, and is used for the change amount calculation in the change amount calculation circuit 10a (in the illustrated example, one other pixel is interposed between the interpolation target pixels). An insufficient color pixel (in the illustrated example, interpolation) that is located in the vicinity of the interpolation target pixel from the interpolation target pixel and is used for the change amount calculation by the change amount calculation circuit 10b described later with respect to the distance to the pixel adjacent to the pixel. This is the ratio of the distance to the pixel adjacent to the target pixel) and is also called the linear similarity ratio of the insufficient color geometric figure to the known color geometric figure.

Next, the operation of the change amount calculation circuit 10b will be described. The change amount calculation circuit 10b calculates a change amount Gdisp of the insufficient color component G. Here, the Gdisp calculation formula is switched depending on in which direction the edge is present in the comparison circuit 7c (that is, according to the signal ED indicating the detection result from the comparison circuit 7c). That is, when it is determined that no edge exists, Gdisp is calculated by the following equation.
Gdisp
= 1 + | G (i, j-1) -G (i, j + 1) |
+ | G (i-1, j) -G (i + 1, j) |
If it is determined that an edge exists in the upward and diagonal direction, Gdisp is calculated by the following equation.
Gdisp
= 1 + | (G (i, j-1) + G (i-1, j))
− (G (i, j + 1) + G (i + 1, j)) |
If it is determined that an edge exists in the diagonally upward direction, Gdisp is calculated according to the following equation.
Gdisp
= 1 + | (G (i, j-1) + G (i + 1, j))
− (G (i, j + 1) + G (i−1, j)) |
When it is determined that an edge exists in the horizontal direction, Gdisp is calculated by the following equation.
Gdisp = 1 + | G (i-1, j) -G (i + 1, j) |
If it is determined that an edge exists in the vertical direction, Gdisp is calculated by the following equation.
Gdisp = 1 + | G (i, j-1) -G (i, j + 1) |

  As described above, when the comparison circuit 7c determines that there is no edge, the pixels are arranged in the horizontal direction with respect to the interpolation target pixel, located on the opposite side with respect to the interpolation target pixel, and in the vicinity of the interpolation target pixel. The absolute value of the difference between the pixel values (of the same color component) of the pixels located in (in the example shown, adjacent to the interpolation target pixel) and the pixels that are arranged in the vertical direction with respect to the interpolation target pixel and centered on the interpolation target pixel As the sum of the absolute values of the pixel value differences (of the same color component) of the pixels located in the opposite side and in the vicinity of the interpolation target pixel (adjacent to the interpolation target pixel in the illustrated example) Is obtained as the change amount Gdisp of the insufficient color component G.

  When the comparison circuit 7c determines that an edge is present in the diagonally upward direction, the line is parallel to the diagonally upward line passing through the interpolation target pixel and the diagonally upward line passing through the interpolation target pixel. Also, the sum of the pixel values of the pixels located on the upper left line and in the vicinity of the pixel to be interpolated is parallel to the diagonal line rising to the right passing through the pixel to be interpolated, and rising to the right through the pixel to be interpolated. A value obtained by adding a predetermined constant, for example, 1 to the absolute value of the difference from the sum of pixel values of pixels located on the lower right side of the diagonal line and in the vicinity of the interpolation target pixel The change amount Gdisp of the insufficient color component G is obtained.

  When the comparison circuit 7c determines that there is an edge in the diagonally upward left direction, the line is parallel to the diagonally upward left line passing through the interpolation target pixel and from the diagonally upward left line passing through the interpolation target pixel. Also, the sum of the pixel values of the pixels located on the upper right line and in the vicinity of the pixel to be interpolated is parallel to the line in the diagonally upward direction passing through the pixel to be interpolated, and is left upward through the pixel to be interpolated. A value obtained by adding a predetermined constant, for example, 1 to the absolute value of the difference from the sum of the pixel values of pixels located on the lower left side of the diagonal line and in the vicinity of the interpolation target pixel, The change amount Gdisp of the insufficient color component G is obtained.

  When the comparison circuit 7c determines that an edge is present in the horizontal direction, the comparison circuit 7c is arranged in the horizontal direction with respect to the interpolation target pixel and is located in the vicinity of the interpolation target pixel (in the example illustrated, the interpolation target pixel A value obtained by adding a predetermined constant, for example, 1 to the absolute value of the difference between the pixel values (of the same color component) of the pixels adjacent to each other is obtained as the change amount Gdisp of the insufficient color component G.

  When the comparison circuit 7c determines that there is an edge in the vertical direction, the comparison circuit 7c is arranged in the vertical direction with respect to the interpolation target pixel and is located in the vicinity of the interpolation target pixel (in the illustrated example, the interpolation target pixel A value obtained by adding a predetermined constant, for example, 1 to the absolute value of the difference between the pixel values (of the same color component) of the pixels adjacent to each other is obtained as the change amount Gdisp of the insufficient color component G.

Next, the operation of the divider 10c will be described. The divider 10c outputs a value obtained by dividing Gdisp by Rdisp. That is, the value t output from the divider 10c can be expressed by the following calculation formula.
t = Gdisp / Rdisp

Next, the operation of the coefficient determination circuit 11 will be described. The coefficient determination circuit 11 outputs a coefficient s having a value of “1”, “0”, or “−1” based on the high-frequency component values calculated by the high-pass filter circuits 8a and 8b. It should be noted that the value of the coefficient s depends on in which direction the edge is present in the comparison circuit 7c (that is, according to the signal ED indicating the detection result from the comparison circuit 7c). It is decided according to the flow.
When the comparison circuit 7c determines that there is no edge, when it is determined that there is an edge in the horizontal direction, and when it is determined that an edge exists in the vertical direction, processing according to the flow of FIG. If it is determined in 7c that an edge exists in the diagonally upward direction, and if it is determined that an edge exists in the diagonally upward direction, the process of the flow of FIG. 8 is performed.

Here, “condition 1” in step ST11 in FIG. 7 and step ST21 in FIG. 8 and “condition 2” in step ST23 in FIG. 8 are specifically expressed as follows.
(Condition 1): The high frequency component value of the insufficient color component is smaller than a predetermined threshold, for example, 0.
(Condition 2): Signs of high-frequency component values of known color components and high-frequency component values of insufficient color components are different.

  That is, in the case of following the flow of FIG. 7, first, in step ST11, it is determined whether or not the condition 1 is satisfied, that is, whether or not the high frequency component value of the insufficient color component is smaller than a predetermined threshold, for example, 0. If it is determined that the high frequency component value of the insufficient color component is 0, the process proceeds to step ST12, and the value of s is set to 0. Otherwise, the process proceeds to step ST13, and the value of s is set to 1.

  In the case of following the flow of FIG. 8, first in step ST21, it is determined whether or not the condition 1 is satisfied, that is, whether or not the high frequency component value of the insufficient color component is smaller than a predetermined threshold, for example, 0. If it is determined that the high frequency component value of the insufficient color component is 0, the process proceeds to step ST22, and the value of s is set to 0. Otherwise, the process proceeds to step ST23, and it is determined whether or not the condition 2 is satisfied, that is, whether the high-frequency component value of the known color component and the high-frequency component value of the insufficient color component are different. If it is determined that the signs are different, the process proceeds to step ST24, and the value of s is set to -1. Otherwise, the process proceeds to step ST25, and the value of s is set to 1.

  Hereinafter, the operation of the coefficient determination circuit 11 will be described in more detail for each determination result of the comparison circuit 7c. In the following equations, β represents a threshold value that takes a predetermined positive value.

First, a case where the comparison circuit 7c determines that there is no edge will be described. In this case, the coefficient determination circuit 11 determines the value of the coefficient s according to the flow shown in FIG. Here, in step ST11, when the high-frequency component values Guhpf, Gbhpf, Glhpf, and Grhpf of the insufficient color component calculated by the high-frequency component calculation circuit 8b satisfy all of the following expressions (1A) to (1D), It is determined that the component value is 0 and condition 1 is satisfied.
| Guhpf | <β (1A)
| Gbhpf | <β (1B)
| Glhpf | <β (1C)
| Grhpf | <β (1D)

  That is, when all of the above equations (1A) to (1D) are satisfied, the process proceeds to step ST12 and s = 0 is set. Otherwise, the process proceeds to step ST13, where s = 1.

  Next, a case will be described in which the comparison circuit 7c determines that an edge exists in a diagonally upward direction. In this case, the coefficient determination circuit 11 determines the value of the coefficient s according to the flow shown in FIG. Here, in step ST21, when the high-frequency component values Guhpf, Gbhpf, Glhpf, and Grhpf of the insufficient color component calculated by the high-frequency component calculation circuit 8b satisfy all the above equations (1A) to (1D), It is determined that the component value is 0.

That is, when all of the above equations (1A) to (1D) are satisfied, the process proceeds to step ST22, and s = 0 is set. Otherwise, the process proceeds to step ST23. In step ST23, the high-frequency component values Ruhpf, Rbhpf, Rlhpf, Rrhpf of the known color component calculated by the high-frequency component calculation circuit 8a and the high-frequency component values Guhpf, Gbhpf, Glhpf of the insufficient color component calculated by the high-frequency component calculation circuit 8b are calculated. When Grhpf satisfies any of the following conditions (2A) to (2D), the sign of the high-frequency component value of the known color component is different from the sign of the high-frequency component value of the insufficient color component, and it is determined that the condition 2 is satisfied.
(2A) Guhpf> β, Glhpf> β, Ruhpf <−β, and Rlhpf <−β
(2B) Guhpf <−β, Glhpf <−β, and Ruhpf> β, and Rlhpf> β
(2C) Gbhpf> β, Glhpf> β, Rbhpf <−β, and Rrhpf <−β
(2D) Gbhpf <−β, Grhpf <−β, Rbhpf> β, and Rrhpf> β

  That is, when any of the conditions (2A) to (2D) is satisfied, the process proceeds to step ST24, where s = −1, and otherwise s = 1.

  Next, a case will be described in which the comparison circuit 7c determines that an edge is present in the diagonally upward direction. In this case, the coefficient determination circuit 11 determines the value of the coefficient s according to the flow shown in FIG. Here, in step ST21, when the high-frequency component values Guhpf, Gbhpf, Glhpf, and Grhpf of the insufficient color component calculated by the high-frequency component calculation circuit 8b satisfy all the above equations (1A) to (1D), It is determined that the component value is 0.

That is, when the above equations (1A) to (1D) are satisfied, the process proceeds to step ST22 and s = 0 is set. Otherwise, the process proceeds to step ST23. In step ST23, the high-frequency component values Ruhpf, Rbhpf, Rlhpf, Rrhpf of the known color component calculated by the high-frequency component calculation circuit 8a and the high-frequency component values Guhpf, Gbhpf, Glhpf of the insufficient color component calculated by the high-frequency component calculation circuit 8b are calculated. When Grhpf satisfies any of the following conditions (3A) to (3D), the sign of the high-frequency component value of the known color component is different from the sign of the high-frequency component value of the insufficient color component, and it is determined that the condition 2 is satisfied.
(3A) Guhpf> β, Grhpf> β, and Ruhpf <−β, and Rrhpf <−β
(3B) Guhpf <−β, Grhpf <−β, and Ruhpf> β, and Rrhpf> β
(3C) Gbhpf> β, Glhpf> β, Rbhpf <−β, and Rlhpf <−β
(3D) Gbhpf <−β, Glhpf <−β, and Rbhpf> β, and Rlhpf> β

  That is, when any of the conditions (3A) to (3D) is satisfied, the process proceeds to step ST24 where s = −1, and otherwise s = 1.

  In the above example, when it is determined by the comparison circuit 7c that an edge exists in the diagonally upward direction, and when it is determined that an edge exists in the diagonally upward direction (as in the case where it is determined that no edge exists) In addition, when the high-frequency component values Guhpf, Gbhpf, Glhpf, and Grhpf of the insufficient color component satisfy all the expressions (1A) to (1D), it is determined that the high-frequency component value of the insufficient color component is 0. When it is determined in the circuit 7c that an edge exists in the diagonally upward direction, when both the expressions (1A) and (1C) are satisfied, or when both the expressions (1B) and (1D) are satisfied, the insufficient color When it is determined that the high frequency component value of the component is 0, and the comparison circuit 7c determines that an edge exists in the diagonally upward left direction, both of the expressions (1A) and (1D) are satisfied Or equation (1B) and the formula (1C) when satisfying both high-frequency component value of the missing color components may be determined to be 0.

Next, a case where the comparison circuit 7c determines that an edge exists in the horizontal direction will be described. In this case, the coefficient determination circuit 11 determines the value of the coefficient s according to the flow shown in FIG. Here, in step ST11, when the high frequency component values Guhpf and Gbhpf of the insufficient color component calculated by the high frequency component calculation circuit 8b satisfy the following condition (4), the high frequency component value of the insufficient color component is 0, and the condition 1 Is determined to be satisfied.
(4) | Guhpf | <β and | Gbhpf | <β

  That is, when the above condition (4) is satisfied, the process proceeds to step ST12 and s = 0. Otherwise, the process proceeds to step ST13, where s = 1.

Next, a case where the comparison circuit 7c determines that an edge exists in the vertical direction will be described. In this case, the coefficient determination circuit 11 determines the value of the coefficient s according to the flow shown in FIG. Here, in step ST11, when the high-frequency component values Glhpf and Grhpf of the insufficient color component calculated by the high-frequency component calculation circuit 8b satisfy the following condition (5), the high-frequency component value of the insufficient color component is 0, and the condition 1 Is determined to be satisfied.
(5) | Glhpf | <β and | Grhpf | <β

  That is, when the above condition (5) is satisfied, the process proceeds to step ST12 and s = 0. Otherwise, the process proceeds to step ST13, where s = 1.

  As described above, when the coefficient determination circuit 11 determines that there is no edge in the comparison circuit 7c, when it is determined that an edge exists in the horizontal direction, and when it is determined that an edge exists in the vertical direction, FIG. When the condition 1 is satisfied (that is, when the high-frequency component value of the insufficient color component is smaller than a predetermined threshold, for example, 0), the coefficient s is set to 0, and the condition 1 is satisfied. When not satisfied, the value of the coefficient s is set to 1.

  On the other hand, when the comparison circuit 7c determines that an edge exists in the diagonally upward right direction and when it is determined that an edge exists in the diagonally upward left direction, processing is performed according to the flow of FIG. When the value of the coefficient s is 1, the condition 1 is not satisfied, and the condition 2 is satisfied (when the high frequency component value of the known color component and the high frequency component value of the insufficient color component are different), the coefficient s When the value is -1 and neither condition 1 nor condition 2 is satisfied, the value of the coefficient s is set to 1.

  As will be described later, if the coefficient s is 1, the insufficient color increment ΔK is added to the insufficient color low frequency component Klpf, and if the coefficient s is −1, the insufficient color increment ΔK is subtracted from the insufficient color low frequency component Klpf. If the coefficient s is 0, the insufficient color low-frequency component Klpf is not added to or subtracted from the insufficient color low-frequency component Klpf, and the insufficient color low-frequency component Klpf is output as the interpolation processing result.

  The above is the operation of the coefficient determination circuit 11.

Next, the operation of the subtractor 12 will be described. The subtractor 12 obtains a value ΔR calculated by the following equation.
ΔR = R (i, j) −Rlpf

  Next, operations of the multiplier 13, the multiplier 14, and the adder 15 will be described. The multiplier 13 outputs a value obtained by multiplying the output value ΔR of the subtractor 12 by the output value t of the divider 10c. The multiplier 14 outputs a value obtained by multiplying the output value of the multiplier 13 by the output value s of the coefficient determination circuit 11. The adder 15 outputs a value obtained by adding the output value of the multiplier 14 to the output value Glpf of the low-pass filter circuit 9b.

  The value output from the adder 15 is written in the two-dimensional memory 6g as the G component value G (i, j) of the interpolation target pixel.

In other words, in the interpolation processing circuit 5, the insufficient color component value G (i, j) is interpolated for the R pixel according to the following equation.
G (i, j) = Glpf + s · t · {(R (i, j) −Rlpf}

  In the interpolation processing circuit 5, the G color component values are interpolated for all the R pixels according to the interpolation method described above.

  Next, a method for interpolating G component values for B pixels will be described.

  FIGS. 9A and 9B are diagrams showing the B component value and the G component value written in the two-dimensional memories 6b and 6g, with the B pixel as the center. If the R component value shown in FIGS. 4A and 4B is replaced with the B component value, it is apparent that FIGS. 9A and 9B are obtained. Therefore, the method for interpolating the G component value for the B pixel can be handled in the same manner as the method for interpolating the G component value for the R pixel.

  In other words, the method for interpolating the G component for the B pixel can be described in the same sentence as the method for interpolating the G component for the R pixel, and a detailed description thereof will be omitted.

  Through the above processing, interpolation of the G component value for the R pixel and the G component value for the B pixel is completed. FIGS. 10A to 10C schematically show the state of writing to the two-dimensional memories 6r, 6g, and 6b at this time. In addition, in the figures showing the state of writing to the frame memory, including FIGS. 10A to 10C, the G component value obtained by interpolation is written in the pixel indicated by the lowercase letter g. Represents. In the subsequent interpolation processing, the G component value g obtained by interpolation can also be handled as a known color component.

  In other words, the G component value in the R pixel and the G component value in the B pixel can also be handled as known color component values.

  Next, a method for interpolating R component values and B component values for G pixels will be described.

  In the pixel array shown in FIG. 2, a line in which the R pixel and the G pixel are arranged in the horizontal direction is defined as an even line, and a line in which the G pixel and the B pixel are arranged in the horizontal direction is defined as an odd line. When the pixel array of FIG. 2 is viewed centering on the G pixels in the even rows and the pixel array of FIG. 2 is viewed centering on the G pixels in the odd rows, the arrangement of the R pixels and the B pixels is different. Therefore, the interpolation of the R component value and the B component value for the G pixels in the even rows is different from the interpolation of the R component value and the B component value for the G pixels in the odd rows. Here, first, interpolation of R component values and B component values for even-numbered G pixels will be described first, and then interpolation of R component values and B component values for odd-numbered G pixels will be described.

  Of the interpolation of the R component value and the B component value for the G pixels in the even rows, the interpolation of the R component value for the G pixels in the even rows will be described.

  FIGS. 11A to 11C show a state in which each color component value is written in the two-dimensional memories 6r, 6g, and 6b with the G pixels in even rows as the center. 11A shows the state of the two-dimensional memory 6g, FIG. 11B shows the state of the two-dimensional memory 6r, and FIG. 11C shows the state of the two-dimensional memory 6b.

  Hereinafter, a method of interpolating the R component value R (i, j) for the G pixel existing at the position of the coordinate (i, j) on the HV coordinate plane shown in FIGS. State.

  First, the operations of the difference calculation circuits 7h, 7v, 7p, 7m and the comparison circuit 7c will be described. These circuits do not perform any particular operation.

  Next, operations of the high-pass filter circuits 8a and 8b and the coefficient determination circuit 11 will be described. The high-pass filter circuits 8a and 8b do not perform any particular operation, and the coefficient determination circuit 11 always outputs 1 as the value of the coefficient s.

Next, the operation of the low-pass filter circuit 9a will be described. The low pass filter circuit 9a calculates the low frequency component value Glpf of the known color component G according to the following equation.
Glpf = {G (i-1, j) + 2G (i, j) + G (i + 1, j)} / 4

Next, the operation of the low-pass filter circuit 9b will be described. The low-pass filter circuit 9b calculates the low frequency component value Rlpf of the insufficient color component R according to the following equation.
Rlpf = {R (i-1, j) + R (i + 1, j)} / 2

  Next, the operation of the variation calculation circuits 10a and 10b will be described. The change amount calculation circuit 10a always outputs 1 as the change amount Gdisp of the known color component G. Further, the change amount calculation circuit 10b always outputs 1 as the change amount Rdisp of the insufficient color component R.

Next, the operation of the divider 10c will be described. The divider 10c outputs a value obtained by dividing Rdisp by Gdisp. That is, the output value t of the divider 10c can be expressed by the following calculation formula.
t = Rdisp / Gdisp

  Here, since the values of Gdisp and Rdisp are always 1, the value of t is always 1.

Next, the operation of the subtractor 12 will be described. The subtractor 12 calculates a value ΔG calculated by the following equation.
ΔG = G (i, j) −Glpf

  Next, operations of the multiplier 13, the multiplier 14, and the adder 15 will be described. The multiplier 13 outputs a value obtained by multiplying the output value ΔG of the subtractor 12 by the output value t of the divider 10 c. The multiplier 14 outputs a value obtained by multiplying the output value of the multiplier 13 by the output value s of the coefficient determination circuit 11. The adder 15 outputs a value obtained by adding the output value of the multiplier 14 to the output value Rlpf of the low-pass filter circuit 9b.

  The value output from the adder 15 is written in the two-dimensional memory 6r as the R component value R (i, j) of the interpolation target pixel.

In other words, the deficient color component value R (i, j) is interpolated for the G pixel in the interpolation processing circuit 5 according to the following equation. In the following formula, the values of the coefficient s and the coefficient t are always 1.
R (i, j) = Rlpf + s · t · (G (i, j) −Glpf)

  In the interpolation processing circuit 5, the insufficient color component value R (i, j) is interpolated for the interpolation target pixel in accordance with the interpolation method described above.

  Next, interpolation of B component values for G pixels in even rows, that is, B components for G pixels existing at the position of coordinates (i, j) on the HV coordinate plane shown in FIGS. A method for interpolating the value B (i, j) will be described.

  First, the operations of the difference calculation circuits 7h, 7v, 7p, 7m and the comparison circuit 7c will be described. These circuits do not perform any particular operation.

  Next, operations of the high-pass filter circuits 8a and 8b and the coefficient determination circuit 11 will be described. The high-pass filter circuits 8a and 8b do not perform any particular operation, and the coefficient determination circuit 11 always outputs 1 as the value of the coefficient s.

Next, the operation of the low-pass filter circuit 9a will be described. The low pass filter circuit 9a calculates the low frequency component value Glpf of the known color component G according to the following equation.
Glpf = {G (i, j-1) + 2G (i, j) + G (i, j + 1)} / 4

Next, the operation of the low-pass filter circuit 9b will be described. The low-pass filter circuit 9b calculates the low frequency component value Blpf of the insufficient color component B according to the following equation.
Blpf = {B (i, j-1) + B (i, j + 1)} / 2

  Next, the operation of the variation calculation circuits 10a and 10b will be described. The change amount calculation circuit 10a always outputs 1 as the change amount Gdisp of the known color component G. The change amount calculation circuit 10b always outputs 1 as the change amount Bdisp of the insufficient color component B.

Next, the operation of the divider 10c will be described. The divider 10c outputs a value obtained by dividing Bdisp by Gdisp. That is, the output value t of the divider 10c can be expressed by the following calculation formula.
t = Bdisp / Gdisp

  Here, since the values of Gdisp and Bdisp are always 1, the value of t is always 1.

Next, the operation of the subtractor 12 will be described. The subtractor 12 calculates a value ΔG calculated by the following equation.
ΔG = G (i, j) −Glpf

  Next, operations of the multiplier 13, the multiplier 14, and the adder 15 will be described. The multiplier 13 outputs a value obtained by multiplying the output value ΔG of the subtractor 12 by the output value t of the divider 10 c. The multiplier 14 outputs a value obtained by multiplying the output value of the multiplier 13 by the output value s of the coefficient determination circuit 11. The adder 15 outputs a value obtained by adding the output value of the multiplier 14 to the output value Blpf of the low-pass filter circuit 9b.

  The value output from the adder 15 is written in the two-dimensional memory 6b as the B component value B (i, j) of the interpolation target pixel.

In other words, in the interpolation processing circuit 5, the insufficient color component value B (i, j) is interpolated for the G pixel according to the following equation. In the following formula, the values of the coefficient s and the coefficient t are always 1.
B (i, j) = Blpf + s · t · (G (i, j) −Glpf)

  In the interpolation processing circuit 5, the insufficient color component value B (i, j) is interpolated for the G pixel in accordance with the interpolation method described above.

  Next, a method of interpolating R component values and B component values for G pixels in odd rows will be described.

  12A to 12C show a state in which each color component value is written in the two-dimensional memories 6r, 6g, and 6b, centering on the G pixels in odd rows. 12A shows the state of the two-dimensional memory 6g, FIG. 12B shows the state of the two-dimensional memory 6b, and FIG. 12C shows the state of the two-dimensional memory 6r. When the R component value shown in FIGS. 11A to 11C is replaced with the B component value and the B component value is replaced with the R component value, it is apparent that FIGS. 12A to 12C are obtained. . Therefore, the method of interpolating the R component values for the G pixels in the odd rows can be handled in the same manner as the method for interpolating the B component values for the G pixels in the even rows, and a method of interpolating the B component values for the G pixels in the odd rows. Can be handled in the same manner as the method of interpolating R component values for G pixels in even rows.

  In other words, the method of interpolating the R component values for the odd-numbered G pixels can be described in the same sentence as the method for interpolating the B-component values for the even-numbered G pixels, and a detailed description thereof will be omitted. Further, the method of interpolating the B component values for the odd-numbered G pixels can also be described in the same sentence as the method for interpolating the R-component values for the even-numbered G pixels, and detailed description thereof will be omitted.

  Through the above processing, interpolation of the R component value and the B component value for the G pixel is completed. FIGS. 13A to 13C schematically show the state of writing to the two-dimensional memories 6r, 6g, and 6b at this time. In addition, in the figures showing the state of the frame memory including FIGS. 13A to 13C, the R component value and the B component value obtained by interpolation are respectively written in the pixels indicated by the small letters r and b. Represents that In the subsequent processing, R component values and B component values obtained by interpolation can also be handled as known color components.

  In other words, the R component value and the B component value in the G pixel can also be handled as known color component values.

  Finally, a method for interpolating the R component value for the B pixel and the B component value for the R pixel will be described. First, the description will be made from the method of interpolating the R component value for the B pixel.

  FIGS. 14A to 14C are diagrams showing the state in which the R component value, the G component value, and the B component value are written in the two-dimensional memories 6r, 6g, and 6b, with the B pixel as the center. FIG. 14A shows the two-dimensional memory 6b, FIG. 14B shows the two-dimensional memory 6g, and FIG. 14C shows the two-dimensional memory 6r. A method of interpolating the R component value R (i, j) for the B pixel existing at the position of the coordinate (i, j) on the HV coordinate plane shown in FIGS. 14 (A) to (C) will be described below. .

First, the difference calculation circuit 7h represents the difference in the horizontal direction of the image, dh, the difference calculation circuit 7v represents the difference in the vertical direction of the image, the difference calculation circuit 7p represents the difference in the upward-right oblique direction of the image, and the difference calculation circuit 7m. Calculates dm representing the difference in the diagonal direction that rises to the left of the image. Note that dh, dv, dp, and dm are calculated by the following equations.
dh = | G (i−1, j) −G (i + 1, j) |
dv = | G (i, j-1) -G (i, j + 1) |
dp
= {| G (i, j-1) -G (i-1, j) |
+ | G (i, j + 1) -G (i + 1, j) |} / 2
dm
= {| G (i, j-1) -G (i + 1, j) |
+ | G (i, j + 1) -G (i-1, j) |} / 2

  Next, the comparison circuit 7c determines in which direction the edge exists with the interpolation target pixel as the center. Specifically, the direction in which the edge exists is determined according to the flow of FIG. 15 based on the values of dh, dv, dp, and dm. The procedure is shown below.

  First, in step ST31, dh and dv are compared with a threshold value γ2. When both dh and dv are smaller than the threshold γ2, the process proceeds to step ST32, and it is determined that no edge exists. Otherwise, the process proceeds to step ST33, and the absolute value of the difference between dh and dv is compared with the threshold σ2. If the absolute value of the difference between dh and dv is smaller than the threshold σ2, the process proceeds to step ST34. Otherwise, the process proceeds to step ST37. In step ST34, dp and dm are compared to determine whether there is an edge in the diagonally upward direction or an edge in the diagonally upward direction. That is, if dp is smaller than dm, the process proceeds to step ST35, where it is determined that an edge exists in the diagonally upward direction. Otherwise, the process proceeds to step ST36, where it is determined that an edge exists in the diagonally upward direction. On the other hand, in step ST37, dh and dv are compared to determine whether there is an edge in the horizontal direction or an edge in the vertical direction. That is, if dh is smaller than dv, the process proceeds to step ST38, where it is determined that an edge exists in the horizontal direction, and otherwise, the process proceeds to step ST39, where it is determined that an edge exists in the vertical direction.

Next, the operation of the high pass filter circuit 8a will be described. The high-pass filter circuit 8a calculates the high frequency component value of the known color component G. Note that four types of values Guhpf, Gbhpf, Glhpf, and Grhpf represented by the following formulas are calculated as high-frequency component values.
Guhpf
= {-G (i-1, j-1) + 2G (i, j-1) -G (i + 1, j-1)} / 2
Gbhpf
= {-G (i-1, j + 1) + 2G (i, j + 1) -G (i + 1, j + 1)} / 2
Glhpf
= {-G (i-1, j-1) + 2G (i-1, j) -G (i-1, j + 1)} / 2
Grhpf
= {-G (i + 1, j-1) + 2G (i + 1, j) -G (i + 1, j + 1)} / 2

Next, the operation of the high pass filter circuit 8b will be described. The high pass filter circuit 8b calculates the high frequency component value of the insufficient color component R. In addition, four types of values Ruhpf, Rbhpf, Rlhpf, and Rrhpf represented by the following formulas are calculated as the high frequency component values.
Ruhpf
= {-R (i-1, j-1) + 2R (i, j-1) -R (i + 1, j-1)} / 2
Rbhpf
= {-R (i-1, j + 1) + 2R (i, j + 1) -R (i + 1, j + 1)} / 2
Rlhpf
= {-R (i-1, j-1) + 2R (i-1, j) -R (i-1, j + 1)} / 2
Rrhpf
= {-R (i + 1, j-1) + 2R (i + 1, j) -R (i + 1, j + 1)} / 2

Next, the operation of the low-pass filter circuit 9a will be described. The low pass filter circuit 9a calculates the low frequency component value Glpf of the known color component G. Here, the calculation formula of Glpf is switched depending on in which direction the edge is present in the comparison circuit 7c. That is, when it is determined that there is no edge, when it is determined that an edge exists in the diagonally upward direction, and when it is determined that an edge exists in the diagonally upward direction, Glpf is calculated by the following equation.
Glpf
= {G (i, j-1) + G (i-1, j) + 4G (i, j)
+ G (i + 1, j) + G (i, j + 1)} / 8
When it is determined that an edge exists in the horizontal direction, Glpf is calculated by the following equation.
Glpf = {G (i-1, j) + 2G (i, j) + G (i + 1, j)} / 4
When it is determined that an edge exists in the vertical direction, Rlpf is calculated by the following equation.
Glpf = {G (i, j-1) + 2G (i, j) + G (i, j + 1)} / 4

Next, the operation of the low-pass filter circuit 9b will be described. The low pass filter circuit 9b calculates the low frequency component value Rlpf of the insufficient color component R. Here, the calculation formula of Rlpf is switched depending on in which direction the edge is determined by the comparison circuit 7c. That is, when it is determined that there is no edge, when it is determined that an edge exists in the diagonally upward direction, and when it is determined that an edge exists in the diagonally upward direction, Rlpf is calculated by the following equation.
Rlpf
= {R (j-1, i) + R (i-1, j) + R (i + 1, j) + R (j + 1, i)} / 4
When it is determined that an edge exists in the horizontal direction, Rlpf is calculated by the following equation.
Rlpf = {R (i-1, j) + R (i + 1, j)} / 2
When it is determined that an edge exists in the vertical direction, Rlpf is calculated by the following equation.
Rlpf = {R (i, j-1) + R (i, j + 1)} / 2

Next, the operation of the change amount calculation circuit 10a will be described. The change amount calculation circuit 10a calculates the change amount Gdisp of the known color component G. Here, the Gdisp calculation formula is switched depending on in which direction the edge is present in the comparison circuit 7c. That is, when it is determined that no edge exists, Gdisp is calculated by the following equation.
Gdisp
= 1 + | G (i, j-1) -G (i, j + 1) |
+ | G (i-1, j) -G (i + 1, j) |
If it is determined that an edge exists in the upward and diagonal direction, Gdisp is calculated by the following equation.
Gdisp
= 1 + | (G (i, j-1) + G (i-1, j))
− (G (i, j + 1) + G (i + 1, j)) |
If it is determined that an edge exists in the diagonally upward direction, Gdisp is calculated according to the following equation.
Gdisp
= 1 + | (G (i, j-1) + G (i + 1, j))
− (G (i, j + 1) + G (i−1, j)) |
When it is determined that an edge exists in the horizontal direction, Gdisp is calculated by the following equation.
Gdisp = 1 + | G (i-1, j) -G (i + 1, j) |
If it is determined that an edge exists in the vertical direction, Gdisp is calculated by the following equation.
Gdisp = 1 + | G (i, j-1) -G (i, j + 1) |

Next, the operation of the change amount calculation circuit 10b will be described. The change amount calculation circuit 10b calculates the change amount Rdisp of the insufficient color component R. Here, the calculation formula of Rdisp is switched depending on in which direction the edge is determined by the comparison circuit 7c. That is, when it is determined that no edge exists, Rdisp is calculated by the following equation.
Rdisp
= 1 + | R (i, j-1) -R (i, j + 1) |
+ | R (i-1, j) -R (i + 1, j) |
If it is determined that there is an edge in the diagonally upward direction, Rdisp is calculated according to the following equation.
Rdisp
= 1 + | (R (i, j-1) + R (i-1, j))
− (R (i, j + 1) + R (i + 1, j)) |
If it is determined that an edge exists in the diagonally upward direction, Rdisp is calculated according to the following equation.
Rdisp
= 1 + | (R (i, j-1) + R (i + 1, j))
− (R (i, j + 1) + R (i−1, j)) |
If it is determined that an edge exists in the horizontal direction, Rdisp is calculated according to the following equation.
Rdisp = 1 + | R (i-1, j) -R (i + 1, j) |
If it is determined that an edge exists in the vertical direction, Rdisp is calculated by the following equation.
Rdisp = 1 + | R (i, j-1) -R (i, j + 1) |

Next, the operation of the divider 10c will be described. The divider 10c outputs a value obtained by dividing Rdisp by Gdisp. That is, the value output from the divider 10c can be expressed by the following calculation formula.
t = Rdisp / Gdisp

  Next, the operation of the coefficient determination circuit 11 will be described. The coefficient determination circuit 11 outputs a coefficient s having a value of “1”, “0”, or “−1” based on the high-frequency component values calculated by the high-pass filter circuits 8a and 8b. Note that the value of the coefficient s is determined according to the flow of either FIG. 7 or FIG. 8 depending on in which direction the edge is present in the comparison circuit 7c.

First, a case where the comparison circuit 7c determines that there is no edge will be described. In this case, the coefficient determination circuit 11 determines the value of the coefficient s according to the flow shown in FIG. Here, in step ST11, when the high-frequency component values Ruhpf, Rbhpf, Rlhpf, and Rrhpf of the insufficient color component calculated by the high-frequency component calculation circuit 8b satisfy all of the following expressions (6A) to (6D), It is determined that the component value is 0 and condition 1 is satisfied.
| Ruhpf | <β (6A)
| Rbhpf | <β (6B)
| Rlhpf | <β (6C)
| Rrhpf | <β (6D)

  That is, when all the above equations (6A) to (6D) are satisfied, the process proceeds to step ST12, and s = 0 is set. Otherwise, the process proceeds to step ST13, and s = 1 is set.

  Next, a case will be described in which the comparison circuit 7c determines that an edge exists in a diagonally upward direction. In this case, the coefficient determination circuit 11 determines the value of the coefficient s according to the flow shown in FIG. Here, in step ST21, when the high-frequency component values Ruhpf, Rbhpf, Rlhpf, and Rrhpf of the insufficient color component calculated by the high-frequency component calculation circuit 8b satisfy all the above equations (6A) to (6D), It is determined that the component value is 0 and condition 1 is satisfied.

That is, when the above condition is satisfied, the process proceeds to step ST22 and s = 0 is set. In other cases, the process proceeds to step ST23. In step ST23, the high-frequency component values Guhpf, Gbhpf, Glhpf, Grhpf of the known color component calculated by the high-frequency component calculation circuit 8a and the high-frequency component values Ruhpf, Rbhpf, Rlhpf of the insufficient color component calculated by the high-frequency component calculation circuit 8b are calculated. When Rrhpf satisfies any of the following conditions (7A) to (7D), the sign of the high-frequency component value of the known color component is different from the sign of the high-frequency component value of the insufficient color component, and it is determined that the condition 2 is satisfied.
(7A) Ruhpf> β and Rlhpf> β and Guhpf <−β and Glhpf <−β
(7B) Ruhpf <−β and Rlhpf <−β and Guhpf> β and Glhpf> β
(7C) Rbhpf> β and Rrhpf> β and Gbhpf <−β and Grhpf <−β
(7D) Rbhpf <−β and Rrhpf <−β and Gbhpf> β and Grhpf> β

  That is, when any one of the conditions (7A) to (7D) is satisfied, the process proceeds to step ST24 and s = −1. Otherwise, the process proceeds to step ST25 and s = 1.

  Next, a case will be described in which the comparison circuit 7c determines that an edge is present in the diagonally upward direction. In this case, the coefficient determination circuit 11 determines the value of the coefficient s according to the flow shown in FIG. 8. In step ST21, the high-frequency component values Ruhpf, Rbhpf, Rlhpf of the insufficient color component calculated by the high-frequency component calculation circuit 8b are calculated. When Rrhpf satisfies all of the above equations (6A) to (6D), it is determined that the high frequency component value of the insufficient color component is 0 and the condition 1 is satisfied.

That is, when the above condition is satisfied, the process proceeds to step ST22 and s = 0 is set. In other cases, the process proceeds to step ST23. In step ST23, the high-frequency component values Guhpf, Gbhpf, Glhpf, Grhpf of the known color component calculated by the high-frequency component calculation circuit 8a and the high-frequency component values Ruhpf, Rbhpf, Rlhpf of the insufficient color component calculated by the high-frequency component calculation circuit 8b are calculated. When Rrhpf satisfies any of the following conditions (8A) to (8D), it is determined that the high-frequency component value of the known color component and the high-frequency component value of the insufficient color component are different and that the condition 2 is satisfied.
(8A) Ruhpf> β and Rrhpf> β and Guhpf <−β and Grhpf <−β
(8B) Ruhpf <−β and Rrhpf <−β and Guhpf> β and Grhpf> β
(8C) Rbhpf> β and Rlhpf> β and Gbhpf <−β and Glhpf <−β
(8D) Rbhpf <−β and Rlhpf <−β and Gbhpf> β and Glhpf> β

  That is, when any one of the conditions (6A) to (6D) is satisfied, the process proceeds to step ST24 and s = −1. Otherwise, the process proceeds to step ST25 and s = 1.

  In the above example, when it is determined by the comparison circuit 7c that an edge exists in the diagonally upward direction, and when it is determined that an edge exists in the diagonally upward direction (as in the case where it is determined that no edge exists) In addition, when the high-frequency component values Ruhpf, Rbhpf, Rlhpf, and Rrhpf of the insufficient color component satisfy all the expressions (6A) to (6D), it is determined that the high-frequency component value of the insufficient color component is 0. If the circuit 7c determines that an edge is present in the diagonally upward direction, the insufficient color is obtained when both the expressions (6A) and (6C) are satisfied, or when both the expressions (6B) and (6D) are satisfied. When it is determined that the high-frequency component value of the component is 0, and the comparison circuit 7c determines that an edge exists in the diagonally upward left direction, both the expressions (6A) and (6D) are satisfied Or equation (6B) and the formula (6C) when satisfying both high-frequency component value of the missing color components may be determined to be 0.

Next, a case where the comparison circuit 7c determines that an edge exists in the horizontal direction will be described. In this case, the coefficient determination circuit 11 determines the value of the coefficient s according to the flow shown in FIG. Here, in step ST11, when the high frequency component values Ruhpf and Rbhpf of the insufficient color component calculated by the high frequency component calculation circuit 8b satisfy the following condition (9), the high frequency component value of the insufficient color component is 0, and the condition 1 Is determined to be satisfied.
(9) | Ruhpf | <β and | Rbhpf | <β

  That is, when the above condition (9) is satisfied, the process proceeds to step ST12, and s = 0 is set. Otherwise, the process proceeds to step ST13, and s = 1 is set.

Next, a case where the comparison circuit 7c determines that an edge exists in the vertical direction will be described. In this case, the coefficient determination circuit 11 determines the value of the coefficient s according to the flow shown in FIG. Here, in step ST11, when the high-frequency component values Rlhpf and Rrhpf of the insufficient color component calculated by the high-frequency component calculation circuit 8b satisfy the following condition (10), the high-frequency component value of the insufficient color component is 0, and the condition 1 Is determined to be satisfied.
(10) | Rlhpf | <β and | Rrhpf | <β

  That is, when the above equation (10) is satisfied, the process proceeds to step ST12, and s = 0 is set. Otherwise, the process proceeds to step ST13, and s = 1 is set.

  The above is the operation of the coefficient determination circuit 11.

Next, the operation of the subtractor 12 will be described. The subtractor 12 obtains a value ΔG calculated by the following equation.
ΔG = G (i, j) −Glpf

  Next, operations of the multiplier 13, the multiplier 14, and the adder 15 will be described. The multiplier 13 outputs a value obtained by multiplying the output value ΔG of the subtractor 12 by the output value t of the divider 10 c. The multiplier 14 outputs a value obtained by multiplying the output value of the multiplier 13 by the output value s of the coefficient determination circuit 11. The adder 15 outputs a value obtained by adding the output value of the multiplier 14 to the output value Rlpf of the low-pass filter circuit 9b.

  The value output from the adder 15 is written in the two-dimensional memory 6r as the R component value R (i, j) of the interpolation target pixel.

In other words, in the interpolation processing circuit 5, the insufficient color component value R (i, j) is interpolated for the B pixel according to the following equation.
R (i, j) = Rlpf + s · t · (G (i, j) −Glpf)

  In the interpolation processing circuit 5, the R color component values for all the B pixels are interpolated according to the interpolation method described above.

  Next, a method for interpolating B component values for R pixels will be described. FIGS. 16A to 16C are diagrams showing the R component value, the G component value, and the B component value written in the two-dimensional memories 6r, 6g, and 6b with the R pixel as the center. FIG. 16A shows the two-dimensional memory 6r, FIG. 16B shows the two-dimensional memory 6g, and FIG. 16C shows the two-dimensional memory 6b.

  If the R component value shown in FIGS. 14A to 14C is replaced with the B component value and the B component value is replaced with the R component value, it is apparent that FIGS. 16A to 16C are obtained. Therefore, the method of interpolating the B component value for the R pixel can be handled in the same manner as the method of interpolating the R component value for the B pixel.

  In other words, since the method of interpolating the B component value for the R pixel can be described in the same sentence as the method of interpolating the R component for the B pixel, detailed description thereof is omitted.

  Through the above processing, the interpolation of the R component value for the B pixel and the B component value for the R pixel is completed. FIGS. 17A to 17C schematically show how the R, G, and B component values are written in the two-dimensional memories 6r, 6g, and 6b at this time.

  In FIGS. 17A to 17C, the R component value, the G component value, and the B component value are given to all the pixels. That is, the interpolation of all the insufficient color component values is completed.

When J is a known color component and K is an insufficient color component for a pixel existing at coordinates (i, j) on the HV plane, the interpolation processing performed by the interpolation processing device according to the embodiment of the present invention is as follows. It can be expressed by the following formula.
K (i, j) = Klpf + s · t (J (i, j) −Jlpf)

  Here, Klpf is the low-frequency component value of the known color component calculated by the low-pass filter circuit 9b, and the coefficient s is “1”, “0”, or “−1” depending on the correlation determination result in the correlation determination circuit 8. The coefficient t is the output value of the ratio calculation circuit 10, J (i, j) -Jlpf is the output value of the known color increment calculation circuit 20, and t (J (i, j ) −Jlpf) is an output value of the insufficient color increment calculation circuit 30.

  The effects of the interpolation processing apparatus according to the embodiment of the present invention will be described below. In the following, a color filter arranged in a one-dimensional direction is used for the sake of simplicity, but it will be apparent that the same effect can be obtained when the arrangement of color filters is arranged on a two-dimensional plane.

  18A to 18C are diagrams illustrating a color filter composed of a color component represented by J and a color component represented by K, and a color component value obtained from the color filter. That is, the horizontal axis of FIGS. 18A to 18C represents the pixel position, the upper “J” and “K” represent the color of the color filter at the position of each pixel, and the vertical axis represents the color component value. The symbol “◯” represents the color component value of each pixel. Each pixel is represented by coordinates (i-1, i, i + 1, etc.) on the horizontal axis.

  Here, FIG. 18A shows a case where the increase and decrease of the J component and the K component coincide with each other, FIG. 18B shows a case where the J component changes but the K component hardly changes, and FIG. Represents a case where the increase and decrease of the J component and the K component are opposite.

  Hereinafter, as shown in FIG. 18A, when the increase / decrease between different color components coincides, it is said that there is “positive correlation”, and one color as shown in FIG. When the component increases or decreases but the other color component hardly changes, it is said that there is no correlation, and the increase / decrease between different color components as shown in FIG. 18C is reversed , "There is a negative correlation". Hereinafter, a case where the K component is interpolated at the position of the J pixel (the pixel corresponding to the photoelectric conversion element covered with the J color filter) will be described as an example.

  In the following description, the J component value for the pixel at the coordinate i is represented by J (i), and the K component value is represented by K (i).

  19A to 19C, K component values given to the J pixels at the positions of the coordinates i-2 and i are represented by “●” marks by interpolation according to the conventional technique. In the interpolation according to the prior art, interpolation is performed on the assumption that there is a similar relationship between signal changes of the J component and the K component. For example, when interpolating the K component value K (i) for the J pixel at the position of the coordinate i (hereinafter referred to as the interpolation target pixel), the J pixel in the vicinity of the interpolation target pixel (at the position of the coordinate i-2 and the coordinate i + 2). A triangle T1 formed by a point obtained by replacing the J component value of J pixel) with the average value Jave of the J component, and a point corresponding to the J component value J (i) in the interpolation target pixel, and K pixels in the vicinity of the interpolation target pixel A triangle T2 formed by replacing the K component value of (K pixel at position i−1, position i + 1) with the K component average value Kave and the K component value K (i) given to the interpolation target pixel is Interpolation processing is performed so that they are similar.

  In other words, the similarity between the triangle T1 and the triangle T2 (the distance from the coordinate i-1 to the coordinate i + 1 with respect to the distance from the coordinate i-2 to the coordinate i + 2) is added to the increment of the J component calculated by J (i) -Jave. Ratio) to determine the increment of the K component. Then, the K component value for the J pixel is obtained by adding the obtained increment of the K component to the average value of the K component. As shown in FIG. 19A, when there is a positive correlation between the J component and the K component, the signal change between the J component and the K component has a similar relationship. The components can be interpolated appropriately. On the other hand, there is no correlation between the J component and the K component as shown in FIG. 19B, or a negative correlation between the J component and the K component as shown in FIG. 19C. When there is, there is no similar relationship between the signal changes of the J component and the K component, so that interpolation error occurs in the interpolation according to the conventional technique.

  Interpolation errors caused by conventional interpolation can be eliminated as follows. That is, when there is a negative correlation between the J component and the K component, the increase / decrease in the J component and the K component is opposite, so the value obtained by subtracting the increment of the K component from the average value of the K component is the K component value for the J pixel. As long as you ask. Further, when there is no correlation between the J component and the K component, the K component value hardly changes. Therefore, the average value of the K component itself may be obtained as the K component value for the J pixel. The interpolation processing apparatus according to the embodiment of the present invention enables such interpolation. Details are described below.

The interpolation processing by the interpolation processing device according to the embodiment of the present invention can be expressed by the following equation for a one-dimensional color filter.
K (i) = Klpf + s · t (J (i) −Jlpf)

  In the above equation, t (J (i) −Jlpf) corresponds to the increment of the K component, and Klpf corresponds to the average value of the K component. When s = 1, the value obtained by adding the K component increment to the average value of the K component is obtained as the K component value. When s = 0, the average value of the K component itself is obtained as the K component value. , S = −1, a value obtained by subtracting the increment of the K component from the average value of the K component is obtained as the K component value. In the interpolation processing apparatus according to the present invention, the correlation discriminating circuit 8 discriminates the correlation between the J component and the K component, and when there is a positive correlation between the J component and the K component, s = 1 is set, and the J component and the K component When there is no correlation between the components, s = 0, and when there is a negative correlation between the J component and the K component, s = -1, so that it is appropriate depending on the correlation between the J component and the K component. Can be interpolated.

  In the following description, t (J (i) -Jlpf) corresponds to the increment calculated from the J component, and a method for determining the value of s by determining the correlation between the J component and the K component. I do.

  First, it will be described that t (J (i) −Jlpf) corresponds to the increment of the K component.

t is a ratio of Jdisp and Kdisp calculated by the following formula, that is, a value calculated by Kdisp / Jdisp.
Jdisp = 1 + {| J (i−2) −J (i + 2) |} / 2
Kdisp = 1 + | K (i−1) −K (i + 1) |

  Here, the meaning of Jdisp and Kdisp will be described. The second term of Jdisp and Kdisp is a value representing the magnitude of the change amount of the J component and the K component around the interpolation target pixel, respectively. The second term of Jdisp is divided by 2. This is because the distance between pixels used in the calculation of Jdisp is twice the distance between pixels used in the calculation of Kdisp.

  Jdisp and Kdisp are values obtained by adding 1 to the values indicating the magnitudes of changes in the J and K components around the interpolation target pixel, respectively. Therefore, Jdisp and Kdisp are also the values of the changes in the J and K components. It can be said that the value represents the size.

  Accordingly, the ratio of Jdisp to Kdisp, t calculated by Kdisp / Jdisp represents the ratio of the change amount of the J component and the change amount of the K component. Since J (i) -Jlpf represents the increment of the J component, t (J (i) -Jlpf) is obtained by multiplying the increment of the J component by the ratio of the variation amount of the J component and the variation amount of the K component. The increment is being calculated.

  This will be described in more detail with reference to FIGS. 20A shows a case where the K component increment K (i) −Klpf is approximately equal to the J component increment J (i) −Jlpf, and FIG. 20B shows the K component increment K (i) −. This represents a case where Klpf is approximately half of the J component increment J (i) −Jlpf. As is apparent from the calculation formulas for Jdisp and Kdisp, in the case of FIG. 20A, Jdisp and Kdisp have substantially the same value, and t = 1. On the other hand, in FIG. 20B, Kdisp is approximately half that of FIG. 20A, and t = 0.5. In either case, the calculation result of the expression t (J (i) −Jlpf) is almost the same value as the increment of the K component. That is, it can be seen that the increment of the K component is calculated by the equation t (J (i) −Jlpf).

  The meaning of adding 1 in the first term of Jdisp and Kdisp is as follows. The coefficient t is calculated as a value obtained by dividing Kdisp by Jdisp. Here, when K (i−1) = K (i + 1), if the first term of Kdisp does not exist, the value of Kdisp becomes 0. In general, since division by 0 is impossible, the value of t cannot be calculated at this time. In order to avoid such an incomputable state, 1 is added to the first term.

  Moreover, overcorrection can be prevented by providing an upper limit for the value of t. t is calculated by dividing Kdisp by Jdisp. For example, if the value of Kdisp becomes extremely large or the value of Jdisp becomes extremely small due to the influence of noise included in the image, the value of t is also more than necessary. May become large. When the value of t becomes larger than necessary, the value representing the increment of the K component, t (J (i) −Jlpf), becomes larger than necessary, and as a result, the required insufficient color component value becomes larger than necessary. It may be smaller than necessary. That is, an insufficient color component value is obtained in a state where overcorrection is applied. Therefore, an upper limit is set to the value of t. For example, if the value of t is 1 or more, if t = 1, it is possible to avoid overcorrection when interpolating the insufficient color component value. it can.

  In the above description using FIGS. 20A and 20B, the case where there is a positive correlation is taken as an example, but it is obvious that the same effect can be obtained when there is a negative correlation.

  Next, a method for determining the correlation between the J component and the K component will be described.

  The correlation between the J component and the K component can be determined using the high frequency components of the J component and the K component. 21A to 21I show changes in the J component value, the K component value, the high frequency component Jhpf of the J component value, and the high frequency component Khpf of the K component value in accordance with the correlation between the J component and the K component. Represents. When there is a positive correlation between the J component and the K component, the signs of Jhpf and Khpf match as shown in FIGS. When there is no correlation between the J component and the K component, the value of Khpf is 0 as shown in FIGS. When there is a negative correlation between the J component and the K component, the signs of Jhpf and Khpf are reversed as shown in FIGS. That is, when Khpf = 0, it is considered that there is no correlation between the J component and the K component, and when s = 0 and the signs of Jhpf and Khpf are opposite, there is a negative correlation between the J component and the K component. Assuming that s = −1, and otherwise s = 1, interpolation can be appropriately performed according to the correlation between the J component and the K component.

In FIGS. 21D to 21F, an ideal state is considered, and Khpf = 0 when there is no correlation. However, the correlation is actually caused by noise and other effects included in the image. When there is no value, the value of Khpf becomes a small value. Therefore, in the interpolation processing apparatus according to the embodiment of the present invention, when determining whether the value of Khpf is 0, the threshold β is used to cope with the calculation error of Khpf. That is, the value of Khpf is considered to be 0 when the following condition (11) is satisfied.
(11) | Khpf | <β

Similarly, when determining whether the signs of Jhpf and Khpf are opposite, the threshold β is used to deal with the calculation error of Jhpf and Khpf. That is, when any of the following conditions (12A) and (12B) is satisfied, it is determined that the signs of Jhpf and Khpf are opposite.
(12A) Khpf> β and Jhpf <−β
(12B) Khpf <−β and Jhpf> β

  In the interpolation processing apparatus according to the embodiment of the present invention, the edge detection circuit 7 determines in which direction the edge exists around the interpolation target pixel, and the correlation determination circuit 8 determines according to the result. The type of is changing. In other words, if it is determined that there is an edge in the diagonally upward right direction or the diagonally upward left direction, it is determined whether there is a positive correlation, a negative correlation, or no correlation. Whether or not there is a correlation is determined, and negative correlation is not considered.

  This can be explained as follows. Negative correlations are likely to occur when looking at changes in color components across an edge. For example, when the change of the color component is observed so as to cross the edge that changes from a red color to a green color, the R component decreases while the G component increases, and a negative correlation is found between the R component and the G component. Will occur. On the contrary, when the edge is not crossed, it is unlikely that a negative correlation occurs between different color components, and it is considered that there is no need to consider the negative correlation. Therefore, when it is determined that no edge exists, it is not necessary to consider that a negative correlation occurs between the known color component and the insufficient color component. Further, when there is a horizontal edge, if the change in the color component of the pixel located in the horizontal direction around the pixel to be interpolated is observed, the edge is not crossed and negative correlation need not be considered. In the interpolation processing device according to the embodiment of the present invention, when there is a horizontal edge, the interpolation processing is performed using only the color component value of the pixel located in the horizontal direction centering on the interpolation target pixel. Negative correlation is not considered. The same applies to the case where an edge exists in the vertical direction. On the other hand, when there is an edge in the diagonally upward right direction or the diagonally upward left direction, the interpolation processing apparatus according to the embodiment of the present invention uses the color component values of pixels located in the horizontal and vertical directions centering on the interpolation target pixel. Interpolate using. In this case, since the change of the color component crosses the edge both in the horizontal direction and in the vertical direction, it is necessary to take into account the negative correlation.

  In short, correlation determination suitable for the direction of the edge can be performed.

  Similarly, the low-pass filter circuit 9b and the unknown color increment calculation circuit 30 can perform operations suitable for the direction of the edge by changing the calculation performed internally according to the edge detection result of the edge detection circuit 7.

  As mentioned above, although embodiment of this invention was described using the interpolation processing apparatus by embodiment of this invention, embodiment of this invention is not restricted to this. For example, in the embodiment of the present invention, the interpolation processing is realized by hardware processing by the interpolation processing circuit 5, but the same processing may be realized by software calculation.

  Further, the configuration of the interpolation processing circuit 5 is not limited to that described in the embodiment of the present invention. For example, a line memory may be used instead of the two-dimensional memories 6r, 6g, 6b. Further, operations performed by the circuits inside the interpolation processing circuit 5 such as the difference circuits 7h, 7v, 7p, and 7m and the low-pass filter circuits 9a and 9b are not limited to those described in the embodiment of the present invention.

  In the embodiment of the present invention, when the R component value and the B component value for the G pixel are interpolated, the values of the coefficient t and the coefficient s are always set to 1. However, the R component value for the G pixel, B The interpolation of component values is not limited to this. For example, as with the interpolation of other color component values, high-frequency components are calculated, and whether a positive correlation exists between the G component, the R component, and the B component, or a negative correlation. It may be determined whether or not there is a correlation.

In the above embodiment, a Bayer array type color filter is used, but the color filter is not limited to this.
In the above-described embodiment, the two-dimensional image pickup device is used in which color filters of three colors R, G, and B are arranged in a Bayer shape, but the number of types of color filters is three. The present invention is not limited to this. Generally speaking, the first to Nth (N is a natural number equal to or greater than 2) N different color components, the Jth (J is any one of 1 to N). Interpolation for obtaining a color image by calculating the Kth (K is any one of 1 to N except for J) color component values of pixels having the following color component values (hereinafter referred to as interpolation target pixels). It is applicable to a processing device.

  Further, in the embodiment of the present invention, the direction in which the edge exists is determined using the difference circuits 7h, 7v, 7p, and 7m and the comparison circuit 7c, but these circuits are not provided, and the high-frequency component value is always used. There may be an example in which interpolation is performed by determining whether a positive correlation exists, a negative correlation exists, or no correlation exists.

  In short, various modifications can be considered based on the embodiment of the present invention.

  As an application example of the present invention, it can be applied to an imaging apparatus.

It is a block diagram which shows the imaging device provided with the interpolation processing apparatus of embodiment of this invention. It is a figure which shows the arrangement | sequence of a color filter. (A)-(C) is a figure which shows the mode of the writing of the R, G, B component value to the two-dimensional memory 6r, 6g, 6b. (A) And (B) is a figure which shows the mode of the writing of the R and G component value to the two-dimensional memories 6r and 6g. It is a flowchart which shows operation | movement of the comparison circuit 7c. (A)-(E) are figures which show the mode of the edge in an image. 3 is a flowchart showing the operation of the coefficient determination circuit 11. 3 is a flowchart showing the operation of the coefficient determination circuit 11. (A) And (B) is a figure which shows the mode of the writing of the B and G component value to the two-dimensional memories 6b and 6g. (A)-(C) is a figure which shows the mode of the writing of the R, G, B component value to the two-dimensional memory 6r, 6g, 6b. (A)-(C) is a figure which shows the mode of the writing of the R, G, B component value to the two-dimensional memory 6r, 6g, 6b. (A)-(C) is a figure which shows the mode of the writing of the R, G, B component value to the two-dimensional memory 6r, 6g, 6b. (A)-(C) is a figure which shows the mode of the writing of the R, G, B component value to the two-dimensional memory 6r, 6g, 6b. (A)-(C) is a figure which shows the mode of the writing of the R, G, B component value to the two-dimensional memory 6r, 6g, 6b. It is a flowchart which shows operation | movement of the comparison circuit 7c. (A)-(C) is a figure which shows the mode of the writing of the R, G, B component value to the two-dimensional memory 6r, 6g, 6b. (A)-(C) is a figure which shows the mode of the writing of the R, G, B component value to the two-dimensional memory 6r, 6g, 6b. (A)-(C) are figures which show the mode of a change of J component and K component. (A)-(C) are figures which show the interpolation result by a prior art. It is a figure which shows the mode of a change of J component and K component. (A)-(I) is a figure which shows the mode of a change of J component, K component, and its high frequency component.

Explanation of symbols

8 correlation discrimination circuit, 8a high frequency component calculation circuit, 8b high frequency component calculation circuit, 11 coefficient determination circuit, 9a low frequency component calculation circuit, 9b low frequency component calculation circuit, 10 ratio calculation circuit, 10a change amount calculation circuit, 10b change amount Calculation circuit, 10c divider, 12 subtractor, 13 multiplier, 14 multiplier, 15 adder, 20 known color increment calculation circuit, 30 insufficient color increment calculation circuit.

Claims (24)

Among the first to Nth (N is a natural number of 2 or more) N different color components,
For a pixel having a Jth color component value (J is any one of 1 to N) (hereinafter referred to as an interpolation target pixel),
In an interpolation processing apparatus that obtains a color image by calculating a Kth color component value (K is any one of 1 to N excluding J),
An insufficient color low frequency component calculating means for outputting a low frequency component value of the Kth color component in the interpolation target pixel;
An insufficient color increment calculating means for outputting an increment value of the Kth color component in the interpolation target pixel;
Correlation determining means for determining whether a positive correlation, a negative correlation, or no correlation exists between the Jth color component and the Kth color component;
Based on the result of discrimination in the correlation discriminating means, the increment value of the Kth color component calculated by the insufficient color increment calculating means and the Kth color calculated by the insufficient color low frequency component calculating means. Computing means for computing the low-frequency component value of the component ;
From at least two or more directions around the interpolation target pixel, it has a edge detecting means for detecting the direction in which the presence of an edge,
An interpolation processing apparatus , wherein calculations performed by the insufficient color low frequency component calculating means, the insufficient color increment calculating means, and the correlation determining means are switched in accordance with a detection result of the edge detecting means . The computing means is
When the correlation determining means determines that a positive correlation exists between the J-th color component and the K-th color component, the increment value of the K-th color component is used as the K-th color component. Add to the low frequency component value of the component,
When the correlation determining means determines that a negative correlation exists between the J-th color component and the K-th color component, the increment value of the K-th color component is used as the K-th color component. Subtract from the low frequency component value of the component,
When the correlation determining unit determines that there is no correlation between the Jth color component and the Kth color component, the low frequency component value of the Kth color component is set to the Kth color component. The interpolation processing apparatus according to claim 1, wherein the increment value of the component is output without addition or subtraction. The lack of color a low-frequency component computing means, by calculating a weighted average using pixels arranged in the direction of the detected edge in said edge detection means, according to claim 1, characterized in that to calculate the low-frequency component The interpolation processing device described in 1. The correlation determination means includes
When the edge detection unit determines that there is no edge, when it is determined that an edge exists in the horizontal direction, and when it is determined that an edge exists in the vertical direction, the high-frequency component value of the Kth color component is When the absolute value is smaller than the predetermined threshold value, the coefficient value is set to 0. When the absolute value of the high frequency component value of the Kth color component is equal to or larger than the predetermined threshold value, the coefficient value is set to 1.
When the edge detecting unit determines that an edge exists in the diagonally upward direction and when it is determined that an edge exists in the diagonally upward direction, the absolute value of the high-frequency component value of the Kth color component is When the value is smaller than a predetermined threshold, the value of the coefficient is set to 1, the absolute value of the high frequency component value of the Kth color component is greater than or equal to the predetermined threshold, and the high frequency component value of the Jth color component and the first When the sign of the high-frequency component value of the K color component is different, the coefficient value is set to −1, the high-frequency component value of the K-th color component is equal to or greater than the predetermined threshold value, and the J-th color component When the sign of the high frequency component value and the high frequency component value of the Kth color component are not different, the value of the coefficient is set to 1.
The computing means is
If the coefficient is 1, the increment value of the Kth color component is added to the low frequency component value of the Kth color component;
If the coefficient is -1, the increment value of the Kth color component is subtracted from the low frequency component value of the Kth color component;
If the coefficient is 0, the low frequency component value of the Kth color component is interpolated without adding or subtracting the increment value of the Kth color component to the low frequency component value of the Kth color component. The interpolation processing apparatus according to claim 1 , wherein the interpolation processing apparatus outputs the processing result. The insufficient color increment calculating means includes
A known color increment calculating means for outputting an increment value of the Jth color component in the interpolation target pixel;
A ratio calculating means for outputting a ratio representing a ratio of a change amount of the Kth color component to a change amount of the Jth color component in the interpolation target pixel;
As increment color component of the first K, the increment of the color components of the first J, interpolation processing apparatus according to any one of claims 1 to 4, characterized in that outputs a value obtained by multiplying the ratio . The ratio calculation means calculates the amount of change in the Jth color component and the amount of change in the Kth color component using pixels arranged in the direction of the edge detected by the edge detection means. The interpolation processing apparatus according to claim 5 . The known color increment calculation means includes:
A known color low frequency component calculating means for calculating a low frequency component value of the Jth color component in the interpolation target pixel;
As an increment value of the Jth color component,
The interpolation processing apparatus according to claim 5 , wherein a value obtained by subtracting a low frequency component value of the Jth color component from a Jth color component value in the interpolation target pixel is output. The correlation determination means includes
A known color high-frequency component calculating means for outputting a high-frequency component value of the J-th color component in the interpolation target pixel;
A deficient color high frequency component calculating means for outputting a high frequency component value of the Kth color component in the interpolation target pixel;
Interpolation processing apparatus according to any one of claims 1 to 7, characterized in that to determine the correlation based on the frequency component values of the color components of the high-frequency component value and the first K color component of said first J. The correlation determination means includes
When the absolute value of the high-frequency component value of the Kth color component is smaller than a predetermined threshold (hereinafter referred to as “first condition”), the Jth color component and the Kth color component The interpolation processing apparatus according to claim 8 , wherein it is determined that there is no correlation between them. The correlation determining means, when the first condition is not satisfied, to claim 9, characterized in that determining that there is a positive correlation between the color components of the first K color components of said first J The interpolation processing apparatus described. The correlation determination means includes
The absolute value of the high-frequency component value of the J-th color component and the absolute value of the high-frequency component value of the K-th color component are greater than a predetermined threshold, and the high-frequency component value of the J-th color component and the When the sign of the high-frequency component value of the K color component does not match (hereinafter referred to as “second condition”),
The interpolation processing apparatus according to claim 9 , wherein it is determined that there is a negative correlation between the Jth color component and the Kth color component. The correlation determination unit determines that there is a positive correlation between the Jth color component and the Kth color component when the first condition and the second condition are not satisfied. Item 12. The interpolation processing device according to Item 11 . An interpolation processing method executed by an interpolation processing apparatus having an insufficient color low frequency component calculation means, an insufficient color increment calculation means, a correlation determination means, a calculation means, and an edge detection means,
Among the first to Nth (N is a natural number of 2 or more) N different color components,
For a pixel having a Jth color component value (J is any one of 1 to N) (hereinafter referred to as an interpolation target pixel),
In an interpolation processing method for calculating a K-th color component value (K is any one of 1 to N excluding J) to obtain a color image,
The insufficient color low frequency component calculating means for outputting a low frequency component value of the Kth color component in the interpolation target pixel;
An insufficient color increment calculating step in which the insufficient color increment calculating means outputs an increment value of the Kth color component in the interpolation target pixel;
A correlation determination step in which the correlation determination means determines whether a positive correlation, a negative correlation, or no correlation exists between the Jth color component and the Kth color component;
The calculation means is calculated in the increment value of the Kth color component calculated in the insufficient color increment calculation step and in the insufficient color low frequency component calculation step based on the determination result in the correlation determination step. A calculation step of calculating a low frequency component value of the Kth color component ;
It said edge detection means, at least two or more directions around the interpolation target pixel, possess an edge detection step of detecting a direction of the edge,
An interpolation processing method , wherein calculations performed in the insufficient color low frequency component calculation step, the insufficient color increment calculation step, and the correlation determination step are switched according to the detection result of the edge detection step . The calculation step includes:
When it is determined by the correlation determination step that a positive correlation exists between the Jth color component and the Kth color component, an increment value of the Kth color component is set as the Kth color component. Add to the low frequency component value of the component,
When it is determined by the correlation determination step that a negative correlation exists between the Jth color component and the Kth color component, an increment value of the Kth color component is calculated as the Kth color component. Subtract from the low frequency component value of the component,
When it is determined by the correlation determination step that there is no correlation between the Jth color component and the Kth color component, the low frequency component value of the Kth color component is set to the Kth color component. The interpolation processing method according to claim 13 , wherein the increment value of the component is output without addition or subtraction. The lack of color a low-frequency component calculating step, the weighted average using pixels arranged in the direction of the edge detected by the edge detection step to calculate the claim 13, characterized in that to calculate the low-frequency component The interpolation processing method described in 1. The correlation determination step includes
In the edge detection step, when it is determined that there is no edge, when it is determined that an edge exists in the horizontal direction, and when it is determined that an edge exists in the vertical direction, the high-frequency component value of the Kth color component is determined. When the absolute value is smaller than the predetermined threshold value, the coefficient value is set to 0. When the absolute value of the high frequency component value of the Kth color component is equal to or larger than the predetermined threshold value, the coefficient value is set to 1.
In the edge detection step, when it is determined that an edge exists in the diagonally upward direction, and when it is determined that an edge exists in the diagonally upward direction, the absolute value of the high-frequency component value of the Kth color component is When the value is smaller than a predetermined threshold, the value of the coefficient is set to 1, the absolute value of the high frequency component value of the Kth color component is greater than or equal to the predetermined threshold, and the high frequency component value of the Jth color component and the first When the sign of the high-frequency component value of the K color component is different, the coefficient value is set to −1, the high-frequency component value of the K-th color component is equal to or greater than the predetermined threshold value, and the J-th color component When the sign of the high frequency component value and the high frequency component value of the Kth color component are not different, the value of the coefficient is set to 1.
The calculation step includes:
If the coefficient is 1, the increment value of the Kth color component is added to the low frequency component value of the Kth color component;
If the coefficient is -1, the increment value of the Kth color component is subtracted from the low frequency component value of the Kth color component;
If the coefficient is 0, the low frequency component value of the Kth color component is interpolated without adding or subtracting the increment value of the Kth color component to the low frequency component value of the Kth color component. It outputs as a processing result. The interpolation processing method of Claim 13 characterized by the above-mentioned. The insufficient color increment calculating means includes a known color increment calculating means and a ratio calculating means,
The insufficient color increment calculation step includes:
A known color increment calculating step in which the known color increment calculating means outputs an increment value of the Jth color component in the interpolation target pixel;
The ratio calculating means includes a ratio calculating step of outputting a ratio representing a ratio of a change amount of the Kth color component to a change amount of the Jth color component in the interpolation target pixel;
The interpolation processing method according to any one of claims 13 to 16 , wherein a value obtained by multiplying the increment value of the Jth color component by the ratio is output as the increment value of the Kth color component. . In the ratio calculation step, the change amount of the J-th color component and the change amount of the K-th color component are calculated using pixels arranged in the direction of the edge detected in the edge detection step. The interpolation processing method according to claim 17 . The known color increment calculating means includes known color low frequency component calculating means;
The known color increment calculation step includes:
The known color low frequency component calculating means includes a known color low frequency component calculating step of calculating a low frequency component value of the J-th color component in the interpolation target pixel;
As an increment value of the Jth color component,
The interpolation processing method according to claim 17 , wherein a value obtained by subtracting a low-frequency component value of the J-th color component from a J-th color component value in the interpolation target pixel is output. The correlation determination means includes known color high frequency component calculation means and insufficient color high frequency component calculation means,
The correlation determination step includes
The known color high frequency component calculating means for outputting a high frequency component value of the Jth color component in the interpolation target pixel;
The insufficient color high-frequency component calculating means includes an insufficient color high-frequency component calculating step of outputting a high-frequency component value of the K-th color component in the interpolation target pixel;
The interpolation processing method according to claim 13 , wherein the correlation is determined based on a high-frequency component value of the J-th color component and a high-frequency component value of the K-th color component. The correlation determination step includes
When the absolute value of the high-frequency component value of the Kth color component is smaller than a predetermined threshold (hereinafter referred to as “first condition”), the Jth color component and the Kth color component The interpolation processing method according to claim 20 , wherein it is determined that there is no correlation between them. The correlation determination step, when said first condition is not satisfied, the claim 21, characterized in that to determine that between color components of the first K color components of said first J is a positive correlation The interpolation processing method described. The correlation determination step includes
The absolute value of the high-frequency component value of the J-th color component and the absolute value of the high-frequency component value of the K-th color component are greater than a predetermined threshold, and the high-frequency component value of the J-th color component and the When the sign of the high-frequency component value of the K color component does not match (hereinafter referred to as “second condition”),
The interpolation processing method according to claim 21 , wherein it is determined that there is a negative correlation between the Jth color component and the Kth color component. The correlation determination step determines that there is a positive correlation between the Jth color component and the Kth color component when the first condition and the second condition are not satisfied. Item 24. The interpolation processing method according to Item 23 .

Images