JP6559020B2 - Image processing apparatus, image processing method, and program - Google Patents

Description

  The present invention relates to an image processing apparatus, an image processing method, and a program for processing an image captured through an optical system.

  An image pickup apparatus such as a digital camera picks up an image by forming a subject image or the like passing through an optical system on an image pickup surface of an image pickup element. At this time, in the image formed on the imaging surface through the optical system, a phase shift of color components (a positional shift of each color component, hereinafter referred to as a color shift) occurs due to lateral chromatic aberration of the optical system. There are many. For this reason, a technique for improving the image quality by correcting the color shift due to the chromatic aberration of magnification through image processing has been proposed. As an example, Patent Document 1 discloses that the phase of the R and B color components of the photographed image is the amount of color shift due to the chromatic aberration of magnification obtained from the design value with respect to the G color component (corresponding to the phase shift amount of the color component). There is disclosed a technique for correcting an image by shifting it by the amount of color misregistration. Hereinafter, correction that shifts the image by the amount of phase shift of the color component due to chromatic aberration of magnification is referred to as color shift correction.

  In addition, the amount of color misregistration due to lateral chromatic aberration may not always match the amount of color misregistration obtained from the design value due to individual differences between cameras, and in this case, the accuracy of color misregistration correction may be reduced. For this reason, by detecting the amount of color misregistration due to chromatic aberration of magnification from the feature points (mainly the edge portion) of the photographed image, even when there are variations due to individual differences in the camera, it is possible to realize color misregistration correction with high accuracy. Technology has been proposed. For example, Patent Document 2 obtains a correlation value between a G signal acquired from a local window set around an edge portion of an image and an R and B signal acquired while shifting the window in the radial direction. A technique for estimating the color misregistration amount of the R and B signals with respect to the G signal is disclosed. At this time, the size of the window is, for example, about four times the width of the detected edge. Then, the technique described in Patent Document 2 adjusts the levels of the R and B signals so that the average values of the G signal and the R and B signals acquired from the window match, thereby detecting the color misregistration amount. Has improved.

JP 2008-15946 A Japanese Patent No. 4706635

  However, the technique described in Patent Document 2 described above requires that the coefficient for level adjustment be calculated every time the R and B signals are acquired by changing the window shift position, resulting in a longer processing time. There's a problem. In addition, the technique described in Patent Document 2 is based on the detection accuracy of the amount of color misregistration when the window includes a repeated pattern in which the same pattern or the like is repeated, or when a colored edge is included. There is a problem that will decrease.

  The present invention has been made in view of such a problem, and provides an image processing apparatus, an image processing method, and a program capable of detecting a color shift amount from a captured image in a short time and with high accuracy. Objective.

The image processing apparatus of the present invention divides each pixel of the photographed image into a plurality of color components from a photographed image captured through an optical system, and obtains images of a plurality of color planes in which each pixel has the same color component. Generating means for generating, based on the plurality of color planes, an edge region of the image, and a small change region in which each pixel value change amount is smaller than a predetermined change determination value around the edge region. a detection region detecting means, based on the value of the color component of each pixel before dangerous out edge regions and the small change area, for the color plane has been made with reference color among the plurality of color planes, than the reference color And a deviation detecting means for detecting a deviation amount of each color plane.

  According to the present invention, it is possible to detect the amount of color misregistration from a captured image in a short time and with high accuracy.

It is a figure which shows schematic structure of the digital camera of embodiment. It is a figure which shows the example of 1 structure of a color shift detection part. It is a figure used for description of a Bayer arrangement. It is a figure used for description of a color shift detection direction. It is a figure used for description of the color shift representative value according to image height. It is a figure used for description of input data interpolation processing. It is a figure used for description of a low pass filter in the case of input data interpolation processing. It is a flowchart of the edge detection process in 1st, 2nd embodiment. It is a figure which shows an example of the signal value of the color shift detection target in 1st Embodiment. It is a figure used for detailed explanation of color shift detection. It is a figure used for description of the filter for edge detection. It is a figure used for description of a correlation value acquisition result. It is a figure used for description of the color shift amount total result according to image height. It is the figure which showed the signal value of the color shift detection target in 2nd Embodiment. It is a flowchart of the edge detection process in 3rd Embodiment. It is a figure used for description of an edge map variable. It is a figure used for description of a detection direction control parameter.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.
In the following description, a digital camera is exemplified as an example of the image processing apparatus of the present embodiment, but for example, various information processing apparatuses such as a tablet terminal, a smartphone, an electronic game machine, a navigation apparatus, and a personal computer. Also good. In this case, the information processing apparatus acquires a photographed image photographed by a camera function mounted on the apparatus itself or a photographed image photographed by a digital camera or the like, and this embodiment described later on the photographed image. Each process according to the form is executed. The processing according to the present embodiment in the information processing apparatus is realized by executing a computer program in an internal CPU or the like. A computer program for realizing the processing according to the present embodiment is provided to the information processing apparatus via a recording medium, various networks, or a communication line.

  FIG. 1 shows a schematic configuration of a digital camera 100 as an example of an image processing apparatus of the present embodiment. FIG. 1 shows only main components according to the present embodiment, and illustration of other components included in a general digital camera is omitted. FIG. 2 shows a schematic configuration of the color misregistration detection unit 105 of the digital camera 100 of FIG. The description of the color misregistration detection unit 105 in FIG. 2 will be described later.

  The imaging lens 101 is an optical system including a lens group including a zoom lens and a focus lens. The imaging element 102 is a CCD or CMOS element for converting incident light from the imaging lens 101 into an electrical signal. A color filter for separating incident light into red (R), green (G), and blue (B) color components is disposed on the imaging surface of the imaging element 102. FIG. 3 shows an array of filter elements corresponding to R, G (G1, G2), and B color components in the color filter, and this array is called a so-called Bayer array. Each of these filter elements corresponds to each pixel position of the image sensor 102. Note that the example of FIG. 3 shows only a part of the color filter.

  The A / D converter 103 converts the analog image signal output from the image sensor 102 into a digital image signal. The digital image signal output from the A / D converter 103 is so-called RAW (raw) image data, and is stored in the RAM 104 which is a temporary storage memory.

  The color misregistration detection unit 105 reads out the RAW image data stored in the RAM 104, for example, in units of blocks each consisting of a plurality of pixels in the horizontal and vertical directions (x and y directions), and R, G, and B for each block. A color shift amount representing a phase shift amount between the color components is detected. Here, in the present embodiment, it is assumed that the amount of color misregistration caused by the chromatic aberration of magnification of the imaging lens 101 is detected. Since the chromatic aberration of magnification of an optical system such as the imaging lens 101 generally appears as point-symmetric chromatic aberration with the optical axis of the lens as the center, detection of the color misregistration amount is performed in a radial direction with the optical axis as the center.

  For this reason, in the present embodiment, the color misregistration detection unit 105 uses, for example, eight regions (regions d0 to d7) in which the central pixel of the read block is based on the center 41 of the RAW image 40 illustrated in FIG. ), The direction in which the amount of color misregistration is detected is determined. FIG. 4B shows one block for each of the areas d0 to d7 when the central pixel of the block is included in each of the areas d0 to d7 in FIG. The direction of the arrow in each block in FIG. 4B represents the direction in which the color misregistration amount is detected in each block, and the arrows drawn in the regions d0 to d7 in FIG. Correspond to each direction. As shown in FIGS. 4A and 4B, the color misregistration detection unit 105 detects the color misregistration amount only in one direction for each block. The detailed content of the color misregistration detection process performed by the color misregistration detection unit 105 will be described later.

  When the photographing lens 101 is, for example, a tilt lens or a shift lens, a lens with a so-called lens shift type camera shake correction function, the chromatic aberration is not a point-symmetric chromatic aberration. In addition, when the image that is the target of color misregistration detection is, for example, a trimmed image, the position corresponding to the optical axis of the lens may not be specified. In such a case, the direction of chromatic aberration is calculated based on the information on the lenses used, the information on the trimmed range, etc., and the amount of color misregistration can be detected based on the calculated direction of chromatic aberration. It may be done.

  Further, each cell in FIG. 4B represents a pixel, but in the example of FIG. 4B, the number of pixels in the horizontal and vertical directions constituting the block is an actual number to simplify the illustration. It is drawn less than the number of pixels. Such simplification of illustration is similarly applied to RAW images, planes, blocks, and the like shown in other figures described later, and the number of pixels and the number of cells in each figure, and the illustrated RAW images and planes are shown. The number of blocks and the number of cells are drawn less than the actual number.

  Further, as shown in FIG. 5A, the color misregistration detection unit 105 maintains the image height from the image center 51 for each color misregistration amount detected for each block represented by the squares in the drawing of the RAW image data 50. Are summed (totaled) for each of the image height sections h1 to h5 divided at intervals of. The color misregistration detection unit 105 sets the values obtained in the respective image height sections h1 to h5 as representative values of the color misregistration amounts in the image height sections h1 to h5. Then, as shown in FIG. 5B, the color misregistration detection unit 105 outputs and stores the representative values of the color misregistration amounts for the image height sections h1 to h5 together with the number of samples to the RAM 104.

  The color misregistration correction unit 106 reads out the RAW image data, the representative value of the color misregistration amount in each image height section h1 to h5, and information on the number of samples from the RAM 104. Then, the color misregistration correction unit 106 performs color misregistration correction on the RAW image data for each image height section h1 to h5 using the representative value of the color misregistration amount of each image height section h1 to h5. The color misregistration correction process for the lateral chromatic aberration is a process in which the phases of R, G, and B color components in the RAW image are shifted by a color misregistration amount (color component phase misregistration amount) to perform phase alignment (position alignment). is there. Such a color misregistration correction process is an existing technique described in, for example, Patent Document 1 and Patent Document 2 described above, and therefore detailed description thereof is omitted in the present embodiment.

  Note that it is considered that the reliability of color misregistration detection is low for an image height section in which the number of samples for color misregistration detection is extremely small. Therefore, the color misregistration correction unit 106 interpolates the color misregistration amount using the representative value of the color misregistration amount of the adjacent image height interval for the image height interval in which the number of samples is extremely small and the reliability is low, and the interpolation is performed. The color misregistration correction is performed using the color misregistration amount by. In addition, when a color shift amount is prepared in advance as design value data corresponding to each of the image height sections h1 to h5, the color shift correction unit 106 determines the image height section with an extremely small number of samples. Color shift correction may be performed using design value data.

  The image processing unit 107 performs various types of image processing such as development processing and noise reduction processing on the RAW image data after the color shift correction by the color shift correction unit 106. The image data after the image processing by the image processing unit 107 is stored in the external memory 109 or the like via the external memory control unit 108.

<First Embodiment>
Hereinafter, as the first embodiment, details of the color misregistration amount detection processing performed by the color misregistration detection unit 105 of the digital camera 100 illustrated in FIG. 1 will be described.
As shown in FIG. 2, the color misregistration detection unit 105 of the digital camera 100 shown in FIG. 1 includes an input data conversion unit 201, a detection unit 202, a representative value calculation unit 203, and a local memory 204. Yes.

  The input data conversion unit 201 performs color conversion processing as described below on the RAW image data as color plane generation means, and generates color plane data for each of the R, G, and B color components. Here, a plane is an image in which data at each pixel position is formed only from data of the same color components of R, G, and B, respectively. An example of color plane generation for each of the R, G, and B color components will be described with reference to FIGS. 6 (a) to 6 (d) and FIG.

  FIG. 6A shows a part of the RAW image read out from the RAM 104 in FIG. 1 by the input data conversion unit 201. The input data conversion unit 201 extracts pixel values of the same color component from the RAW image shown in FIG. 6A as shown in FIG. The example in FIG. 6B shows an example in which only the R component pixel value is extracted from the RAW image in FIG. After extracting the pixel values of the same color component as shown in FIG. 6B, the input data conversion unit 201 extracts a zero value for the pixel position where there is no signal value of the color component as shown in FIG. 6C. Insert.

  Next, the input data conversion unit 201 applies each filter as shown in FIG. 7 to the image after the zero value is inserted into the pixel position where there is no signal value of the color component as shown in FIG. Interpolation processing is performed using a low-pass filter in which coefficients (coefficients 1 to 4) are arranged. FIG. 6D shows an R plane generated by the input data conversion unit 201 performing an interpolation process using the low-pass filter of FIG. 7 on the image of FIG. The input data conversion unit 201 performs the same processing as described with reference to FIGS. 6B to 6D on the G and B color components of the RAW image in FIG. And B plane. Since the G color component of the RAW image is composed of G1 and G2, only one pixel value of G1 and G2 in FIG. 6A is used for the G plane in order to match the R and B planes. To generate. The color plane data of the R, G, B color components created in this way is stored in the local memory 204.

  The detection unit 202 is an example of a region detection unit and a deviation amount detection unit. The detection unit 202 sets one of the R, G, and B color planes stored in the local memory 204 as a reference color plane. In the present embodiment, for example, the G plane is a reference color plane, and the R and B planes are color planes other than the reference color. Further, the detection unit 202 acquires pixel values for one column extracted in the above-described color shift detection direction for each block for each of the R, G, and B color planes. Then, the detection unit 202 extracts the pixel value for one column extracted from the G plane block of the reference color and one column extracted from the block corresponding to the G plane block among the R and B planes other than the reference color. The amount of color misregistration is detected using the pixel value.

  Although details will be described later, in order to realize such detection of the color misregistration amount, the detection unit 202 uses each pixel value of the pixel column for one column extracted from each R, G, B color plane, An edge region (hereinafter referred to as an edge portion) is detected. Similarly, although details will be described later, the detection unit 202 is a small change region in which each pixel whose change in the pixel value is small from the both ends of the width of the edge portion in the color shift detection direction, for example, continues around the edge portion. (Hereinafter referred to as a solid part) is detected. Further, although details will be described later, the detection unit 202 detects the phase shift amount (color shift amount) of the planes other than the reference color (R, B) with respect to the reference color (G) plane from the solid portion. For example, the detection unit 202 matches the level of at least one of the G plane and the R and B planes according to the pixel value of the solid portion, and the G plane pixel value and R, The amount of deviation is detected based on the pixel value of the B plane. In the following description, both ends of the width of the edge portion in the color misregistration detection direction are simply expressed as “both ends of the edge portion”. Further, in the following description, a region outside from both ends of the edge portion is simply referred to as “outer region at both ends of the edge portion”.

  FIG. 8 is a flowchart showing the flow of processing in the detection unit 202 of the present embodiment. The contents of the processing of each step in FIG. 8 will be described by taking each signal value of R, G, B as shown in FIG. 9 as an example. FIG. 9 is a diagram showing R, G, and B signal values at each pixel position (values of each color component at each pixel at each pixel position).

  In the flowchart of FIG. 8, the detection unit 202 first performs R, R, and R of each pixel for one column extracted in the above-described color misregistration detection direction per block for each plane stored in the local memory 204 as processing in step S801. Edge detection is performed using G and B signals. Edge detection is performed using a luminance value (hereinafter referred to as a Y signal) created by the following equation (1) from the R, G, and B signal values at each pixel position as shown in FIG. .

  Y [n] = 0.3 × R [n] + 0.6 × G [n] + 0.1 × B [n] (1)

  Note that n in Expression (1) represents R, G, and B of one column extracted in the color misregistration detection direction for each block for each of the R, G, and B planes stored in the local memory 204. It is a number representing the pixel position of each signal. FIGS. 10A, 10B, and 10C are diagrams in which R, G, and B signal values at the pixel positions in FIG. 9 are arranged in correspondence with the pixel positions. FIG. 10D shows a diagram in which each value of the Y signal calculated using the equation (1) is arranged corresponding to each pixel position.

  Further, the detection unit 202 performs edge detection filter processing on the Y signal using an edge detection filter in which filter coefficients (−1, 0, 1) as shown in FIG. 11 are set. FIG. 10E shows a diagram in which signal values after edge detection filter processing for the Y signal are arranged in correspondence with the respective pixel positions. Furthermore, the detection unit 202 compares the absolute value of the filter output level of the edge detection filter with a preset edge level determination value EdgeThLevel. Then, the detection unit 202 obtains a pixel position where the absolute value of the filter output level after the filter processing is larger than the edge level determination value EdgeThLevel. In the following description, the absolute value of the filter output level after the edge detection filter processing is expressed as “signal absolute value”.

  Further, the detection unit 202 compares the width of the continuous region (length in the edge width direction) of each pixel position where the signal absolute value is larger than the edge level determination value EdgeThLevel with a preset edge length determination value EdgeThLength. To do. Then, the detection unit 202 has a region in which the width (length in the width direction) of the continuous regions at each pixel position is equal to or greater than the edge length determination value EdgeThLength (the width of the continuous region is the edge length determination value). The above continuous area) is detected as an edge portion. For example, when the edge level determination value EdgeThLevel is “300” and the edge length determination value EdgeThLength is “3”, in the example of FIG. 10E, the detection unit 202 detects the region of the pixel position “4-7”. Detect as an edge part. After step S801, the detection unit 202 advances the process to step S802.

  In step S802, the detection unit 202 continues a predetermined number or more of pixels each having a small amount of change in the Y signal value between pixels at adjacent pixel positions from the both ends of the edge portion detected in step S801 toward the outer side. It is determined whether or not a region (solid portion) exists. Specifically, the detection unit 202 has a width (length) set in advance as a solid portion length determination value Beta, which is a predetermined number, from a pixel position that is one pixel outside the pixel positions included at both ends of the edge portion. The pixel area (the number of pixels) up to the pixel position corresponding to (a) is determined. Furthermore, the detection unit 202 has a signal absolute value shown in FIG. 10E and a solid part preset as a predetermined change judgment value at each pixel position in the pixel region corresponding to the solid part length judgment value Beta. The level judgment value BetaThLevel is compared.

  When the signal absolute value is smaller than the solid portion level determination value BetaThLevel at all pixel positions in the pixel region of the solid portion length determination value Beta, the detection unit 202 determines that the edge portion detected in step S801 is an effective edge portion. It is determined that For example, when the solid part length determination value Beta is “3” and the solid part level determination value BetaThLevel is “100”, the detection unit 202 has an area of pixel positions “1 to 3” as illustrated in FIG. Then, the area of the pixel position “8 to 10” is detected as a solid part. Then, when the signal absolute value is smaller than the solid part level determination value BetaThLevel at all the pixel positions in the solid parts of the pixel positions “1 to 3” and “8 to 10”, the detection unit 202 sets the pixel positions “4 to 7”. Is determined to be an effective edge portion. In this way, when the signal absolute value is smaller than the solid portion level determination value BetaThLevel at all pixel positions in the solid region length determination value Beta, the outer region at both ends of the edge portion is a signal between adjacent pixel positions. It is thought that the change of the value is a small solid part. If the detection unit 202 detects that the outer region at both ends of the edge portion is a solid portion in step S802 and determines that the edge portion is valid, the detection unit 202 advances the process to step S803. The processing after step S803 is processing for detecting a deviation amount.

  On the other hand, in step S <b> 802, the detection unit 202 determines that at least one pixel position having a signal absolute value equal to or greater than the solid part level determination value BetaThLevel among the pixel positions in the pixel area of the solid part length determination value Beta exists. Then, the process of FIG. 8 ends. Here, the region where the pixel position where the signal absolute value is equal to or greater than the solid portion level determination value BetaThLevel may be an image region such as a repetitive pattern in which a fine pattern or the like is repeated. It is done. Therefore, if the detection unit 202 determines in step 802 that there is a pixel position where the signal absolute value is equal to or greater than the solid portion level determination value BetaThLevel, it is determined that there is no valid edge portion, and FIG. The process of the flowchart ends.

  When the processing proceeds to step S803, the detection unit 202 integrates the R, G, and B signal values for the solid portion in the outer region at both ends of the edge portion. Further, the detection unit 202 sets the ratio (gain value) of the R integrated value that is a color component other than the reference color to the G integrated value that is the color component of the reference color, and the G integrated value of the reference color. On the other hand, the ratio (gain value) of the integral value of B, which is a color component other than the reference color, is obtained. Then, the detection unit 202 uses the solid portion with the smaller pixel position value (starting point side in the color misregistration detection direction) across the edge portion as the first solid portion, and the larger pixel position value (color misregistration detection). The solid portion on the end point side of the direction is set as the second solid portion, and each gain value is obtained by Expressions (2) to (5).

gainR1 = sumG1 / sumR1 Formula (2)
gainR2 = sumG2 / sumR2 Formula (3)
gainB1 = sumG1 / sumB1 Formula (4)
gainB2 = sumG2 / sumB2 Formula (5)
sumG1: Integral value of the G signal in the first solid part sumG2: Integral value of the G signal in the second solid part sumR1: Integral value of the R signal in the first solid part sumR2: Integral value of the R signal in the second solid part sumB1: Integral value of B signal in first solid part sumB2: Integral value of B signal in second solid part gainR1: Ratio of R integral value to integral value of G in first solid part (gain value)
gainR2: Ratio of R integrated value to G integrated value in second solid portion (gain value)
gainB1: Ratio of B integrated value to G integrated value in first solid portion (gain value)
gainB2: Ratio of B integrated value to G integrated value in second solid portion (gain value)

  FIG. 10 (h) shows the calculation results of the integral values and gain values of the first and second solid portions in the examples of FIGS. 10 (a) to 10 (c). As an example, when the first solid part is given, sumR1 is “2400”, sumG1 is “3000”, sumB1 is “1500”, gainR1 is “1.25”, and gainB1 is “2.00”. After step S803, the detection unit 202 advances the process to step S804.

  In step S804, the detection unit 202 compares the gain value of the R signal in the outer region at both ends of the edge portion, and similarly compares the gain value of the B signal in the outer region at both ends of the edge portion. Here, for example, when the gain values of the R signal and B signal in the outer regions at both ends of the edge portion are greatly different, it is considered that the edge portion is likely to be colored. When the edge portion is colored in this way, it is considered that the correlation between the R and B signals with respect to the G signal is low. As described above, when the edge portion is colored and the correlation between the R and B signals with respect to the G signal is low in the outer region at both ends of the edge portion, the color shift amount detection accuracy for correcting the lateral chromatic aberration is corrected. Is likely to decline. For this reason, as shown in Equation (6) and Equation (7), the detection unit 202 determines that the absolute value of the difference between the gain values of the outer regions at both ends of the edge portion is relative to the gain value for each of the R signal and the B signal. It is determined whether or not the gain determination value is preset as a threshold value.

abs (gainR1-gainR2) <ThDiffGainR (6)
abs (gainB1-gainB2) <ThDiffGainB (7)

  Note that abs (gainR1-gainR2) in equation (6) is the gain value (gainR1 and gainR2) of the outer region at both ends of the edge portion in the R signal obtained by equations (2) and (3) described above. The absolute value of the difference. Also, abs (gainB1-gainB2) in equation (7) is the difference between the gain values (gainB1 and gainB2) of the outer regions at both ends of the edge portion in the B signal obtained by equations (3) and (4). Absolute value. In the equation (6), ThDiffGainR is a gain determination value set as a gain value threshold for the R signal, and ThDiffGainB is a gain determination value set as a gain value threshold for the B signal. . In the following description, the difference absolute values abs (gainR1-gainR2) and abs (gainB1-gainB2) in the R signal and the B signal are simply expressed as “difference absolute value abs”. Further, the gain determination values ThDiffGainR and ThDiffGainB are simply expressed as “gain determination values ThDiffGain”.

  Here, when the difference absolute value abs in the R and B signals is smaller than the corresponding gain determination value ThDiffGain, it is considered that the edge portion is not colored. In this case, it is considered that the color shift amount for correcting the lateral chromatic aberration can be detected with high accuracy. On the other hand, when at least one of the absolute difference values abs in the R and B signals is equal to or greater than the corresponding gain determination value ThDiffGain, it is considered that the edge portion is colored. In this case, it is considered that the color misregistration amount for correcting the lateral chromatic aberration cannot be detected with high accuracy. Therefore, if the detection unit 202 determines in step S804 that the absolute difference value abs in the R and B signals is smaller than the corresponding gain determination value ThDiffGain, the process proceeds to step S805. On the other hand, when the detection unit 202 determines in step S804 that at least one of the absolute difference values abs in the R and B signals is equal to or greater than the corresponding gain determination value ThDiffGain, the color shift amount of FIG. The detection process ends.

  In step S805, the detection unit 202 reads out the R and B signals from the local memory 204 while shifting the R and B signals by one pixel each in the color misregistration detection direction, and corresponds to the pixel value corresponding to the G plane of the reference color for the signal value of the edge portion. Correlation values with the signal values of are respectively obtained. In Expressions (8) and (9), a calculation expression of SAD values (sadR [k], sadB [k]) that is a sum of absolute differences as a correlation value with the signal value of the corresponding pixel of the G plane is used. Show.

  In Expressions (8) and (9), k is a value indicating the pixel shift position when acquiring the R and B signals, and the value is changed according to the maximum value of the amount of color misregistration to be detected. The Further, gainR is an average value of the gain values gainR1 and gainR2 respectively obtained by the above-described equations (2) and (3) for the R signal in the outer region at both ends of the edge portion. Similarly, gainB is an average value of the gain values gainB1 and gainB2 obtained by the above-described equations (4) and (5) for the B signal in the outer region at both ends of the edge portion. In the following description, the SAD value (sadR [k], sadB [k]) of the sum of absolute differences is simply expressed as “SAD value”. After step S805, the detection unit 202 advances the process to step S806.

In step S806, the detection unit 202 stores the SAD value obtained as described above as information on the detected amount of color misregistration in the local memory 204 together with information on the image height and the edge detection direction as shown in FIG. Let 12 corresponds to 0 to 7 in the areas d0 to d7 in FIG. 4 described above.
The detection unit 202 performs the above processes for each column of all blocks. When the process for each column of all the blocks is completed, the detection unit 202 ends the process of the flowchart of FIG.

Hereinafter, the representative value calculation unit 203 will be described in detail.
The representative value calculation unit 203 estimates the color misregistration amount with reference to the SAD value of each sample stored in the local memory 204 after the detection unit 202 detects the color misregistration amount for all the blocks.

  Specifically, the representative value calculation unit 203 first calculates the following equation (10) from the shift position K0 at which the SAD value is minimized and the SAD values at the shift positions (K0-1) and (K0 + 1) before and after the shift position. Is used to determine the amount of color misregistration for each R, G, B subpixel constituting one pixel.

x =-[S (K0 + 1) -S (K0-1)] / [2S (K0-1) -4S (K0) + 2S (K0 + 1)] (10)
Here, x is an estimated value of the color misregistration amount, and S (K) is sadR [k] or sadB [k] of the SAD value at each shift position.

  Further, the representative value calculation unit 203 aggregates (totals) the color misregistration amount calculated in units of subpixels as described above for each image height interval (h1 to h5) as shown in FIG. After the completion, an average value of color misregistration amounts for each image height section is calculated. Then, the representative value calculation unit 203 stores the calculated average value in the RAM 104 together with the number of samples as the representative value of the color misregistration amount shown in FIG.

  As described above, according to the present embodiment, the color misregistration detection unit 105 can reliably detect an effective edge portion as described above, and by detecting the effective edge portion, the outside of both ends of the edge portion can be detected. The solid part of the region can also be reliably detected. That is, according to the present embodiment, the gain value used for detecting the color misregistration amount needs to be calculated only once from the effective edge portion and the solid portion. As described above, according to the present embodiment, if the gain value calculation for level adjustment, which is conventionally performed every time the shift position is changed, is calculated only once from the solid portions outside the both ends of the edge portion. Therefore, the amount of color misregistration can be detected in a short time. Further, in this embodiment, by performing determination of the solid portion outside the both ends of the edge portion and comparison of the gain values, when the edge portion is colored or there is no solid portion, a repetitive pattern or the like is obtained. This area can be prevented from being used when detecting the amount of color misregistration. For this reason, according to this embodiment, the amount of color misregistration can be detected with high accuracy.

<Second Embodiment>
Hereinafter, the second embodiment will be described.
The digital camera 100 of the second embodiment has the same basic configuration as the digital camera 100 of FIG. 1 described above, but the color misregistration detection processing in the color misregistration detection unit 105 is different from the first embodiment. Different. In the case of the second embodiment, the color misregistration detection unit 105 improves the detection accuracy of the color misregistration amount by performing edge detection for each of the R, G, and B signals. Only the parts different from the first embodiment will be described below.

  In the first embodiment described above, edge detection is performed using the Y signal calculated from the R, G, and B signals. Here, for example, when any one of the R, G, B signals is greatly deviated from the other signals (for example, the amount of color misregistration is converted to a pixel position and reaches nearly 10 pixels). In such a case, there is a possibility that the edge portions of the R and B signals cannot be accurately detected with the Y signal. For example, FIG. 14 shows an example in which the B signal is largely deviated from, for example, the G and B signals among the pixel positions Nsg, Nsr, Neg, Ner, Nsb, and Nsb. In the case of the example of FIG. 14, if only the Y signal is used as in the first embodiment described above, the detection accuracy of the color misregistration amount may be lowered. When the amount of color misregistration is large in this way, when detecting the amount of color misregistration, a method of performing edge detection for each of the R, G, and B signals is effective as in the second embodiment.

  In the second embodiment, the overall process flow of the color misregistration detection process is substantially the same as the process shown in the flowchart of FIG. 8 described in the first embodiment. However, in the case of the second embodiment, the solid portion determination and the gain value calculation are performed with reference to the pixel regions outside the start point and end point pixel positions at both ends of the edge portion in each of R, G, and B. In the case of the example of FIG. 14, Nsg and Neg are the pixel positions of the start point and end point of the edge portion in G. Similarly, Nsr and Ner are the start point and end point of R, and Nsb and Neb are the pixel positions of the start point and end point of B It is. Therefore, in the second embodiment, the solid portion determination and the gain value are performed by referring to the R, G, and B signal values of the outer pixel area excluding the pixel positions Nsg, Nsr, Neg, Ner, Nsb, and Neb. Is calculated.

  In the case of the second embodiment, the color misregistration detection unit 105 calculates each integral value sumG1 for each of R, G, and B of the first solid part and the second solid part according to Expressions (11) to (16). , SumG2, sumR1, sumR2, sumB1, and sumB2.

Here, “Bata” in Expressions (11) to (16) is a solid portion length determination value at both ends of the edge, as in the case of the first embodiment.
Further, the color misregistration detection unit 105 uses the integrated values sumG1, sumG2, sumR1, sumR2, sumB1, and sumB2 to obtain the gain values gainR1, gainR2, gainB1, and gainB2 according to the above formulas (2) to (5). Ask. Then, the color misregistration detection unit 105 uses these gain values to obtain the color misregistration amount in the same manner as described in the first embodiment. Thereby, also in the second embodiment, the amount of color misregistration can be detected with high accuracy in a short time, as in the case of the first embodiment. Furthermore, in the case of the second embodiment, even when the amount of color misregistration is large in the R, G, and B signals, it is possible to detect the amount of color misregistration with high accuracy.

<Third Embodiment>
Hereinafter, a third embodiment will be described.
The digital camera 100 of the third embodiment has the same basic configuration as the digital camera 100 of FIG. 1 described above, but the color misregistration detection processing in the color misregistration detection unit 105 is different from the first embodiment. Different. The color misregistration detection unit 105 according to the third embodiment detects a plurality of edge portions using a plurality of columns of pixel values extracted in the color misregistration detection direction, and calculates an integral value of the solid portion detected at the plurality of edge portions. By calculating the gain value using the added value, the detection accuracy of the color misregistration amount is improved.

  In the first embodiment described above, gain calculation is performed using only the integral values of the R, G, and B signals acquired from the solid portions outside both ends of the detected edge portion. However, in the case of the first embodiment, for example, when the influence of noise is large, such as a captured image obtained by imaging with high sensitivity, the edge portion is not detected correctly, and the gain value There is a possibility that the accuracy of the calculation is lowered. When color misregistration detection is performed using a photographed image having a large influence of noise in this way, as in the third embodiment, a value obtained by summing up integral values acquired with a solid portion with respect to a plurality of edge portions is used. A method of calculating the gain value is effective. Only the parts different from the first embodiment will be described below.

  FIG. 15 shows a flowchart of processing in the detection unit 202 of the color misregistration detection unit 105 in the case of the third embodiment. In the flowchart of FIG. 15, the processing from edge detection in step S1501 to gain value comparison in step S1504 is the same as the processing from step S801 to step S804 in FIG. Further, step S1509 and step S1510 in FIG. 15 are the same processes as step S805 and step S806 in FIG.

  However, in the case of the third embodiment, when the detection unit 202 determines in step S1502 that there is no valid edge portion, the detection unit 202 advances the processing to step S1507. If the detection unit 202 determines in step S1504 that the difference absolute value abs of the gain values at both ends of the edge is greater than or equal to the gain determination values ThDiffGainR and ThDiffGainB, the process proceeds to step S1507. On the other hand, if the detection unit 202 determines in step S1504 that the absolute difference value abs is smaller than the gain determination values ThDiffGainR and ThDiffGainB, the process proceeds to step S1505.

  In the case of the third embodiment, in step S1505, the detection unit 202 adds the above-described integral values sumG1, sumG2, sumR1, sumR2, sumB1, and sumB2 to the variables sumG_blk, sumR_blk, and sumB_blk. The calculation formula at this time is expressed by formula (17) to formula (19). Note that the variables sumG_blk, sumR_blk, and sumB_blk are initialized to “0” before starting color misregistration detection for each block.

sumG_blk = sumG_blk + (sumG1 + sumG2) (17)
sumR_blk = sumR_blk + (sumR1 + sumR2) (18)
sumB_blk = sumB_blk + (sumB1 + sumB2) Expression (19)

After step S1505, the detection unit 202 advances the process to step S1506.
In step S1506, the detection unit 202 sets each of the small regions corresponding to the edge portion detected from the G plane with respect to the edge map variable EdgeMap as shown in FIG. 16 that has the same pixel size as the target block for color misregistration detection. “1” is set in the element of the coordinate position. Here, the edge map variable EdgeMap is a variable map formed by arranging each element two-dimensionally, and all the elements are initialized to “0” before the color shift detection for each block is started. Shall. After step S1506, the detection unit 202 advances the process to step S1507.

  In step S1507, the detection unit 202 determines whether or not the processing in steps S1501 to S1506 has been completed for the pixel values of all columns in the color misregistration detection direction in the block. In step S1507, if the detection unit 202 determines that the processing has been completed for all the columns in the block, the detection unit 202 proceeds to step S1508. If it is determined that the processing has not been completed, the detection unit 202 proceeds to step S1501. The process is returned and the next pixel column is processed.

  When the process proceeds to step S1508, the detection unit 202 calculates gain values gainR_blk and gainB_blk using Expression (20) and Expression (21). As a result, a plurality of integration values calculated for each of the plurality of edge portions are added in the small region where the element is “1” in the edge map variable EdgeMap, and the value obtained by adding the plurality of integration values is added. The gain value calculated from the above is obtained.

gainR_blk = sumG_blk / sumR_blk (20)
gainB_blk = sumG_blk / sumB_blk (21)

After step S1508, the detection unit 202 advances the process to step S1509.
In step S1509, the detection unit 202 uses the gain values gainR_blk, gainB_blk and the edge map variable EdgeMap, and calculates the SAD values (sadR [k], sadB [k]) in units of blocks using Expressions (22) and (23). Ask.

  Here, Nh is the number of pixels in the horizontal direction of the block, and Nv is the number of pixels in the vertical direction of the block. k is a shift position for acquiring the R and B signals. Further, α and β are parameters for determining the directions of the R and B signals acquired at the shifted position, and are values uniquely determined according to the color misregistration detection direction as shown in FIG. After step S1508, the detection unit 202 advances the process to step S1509, which is the same process as step S805 in FIG. The following processing is the same as steps S806 and S806 of FIG.

  Note that the detection unit 202 stores the gain value of each edge obtained in step S1504, and only when the variation of the gain value obtained in the entire block is within a predetermined range (can be determined as a uniform color in the entire block), in step S1508. The gain value may be calculated.

  As described above, in the third embodiment, the gain value is calculated using a result obtained by adding the integral values of the solid portions detected at the plurality of edge portions in units of blocks, thereby making the gain calculation sensitive to noise. The degree of detection can be suppressed and high detection accuracy can be maintained.

<Other embodiments>
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

  The above-described embodiments are merely examples of implementation in carrying out the present invention, and the technical scope of the present invention should not be construed as being limited thereto. That is, the present invention can be implemented in various forms without departing from the technical idea or the main features thereof.

  DESCRIPTION OF SYMBOLS 100 Digital camera, 101 Imaging lens, 102 Image sensor, 103 A / D converter, 104 RAM, 105 Color shift detection part, 106 Color shift correction part, 107 Image processing part, 108 External memory control part, 109 External memory, 201 Input data conversion unit, 202 Deviation amount detection unit, 203 Representative value calculation unit

Claims (14)

Generating means for dividing each pixel of the photographed image into a plurality of color components and generating images of a plurality of color planes, each pixel having the same color component, from a photographed image captured through an optical system;
An area detecting means for detecting, based on the plurality of color planes, an edge area of the image and a small change area in which each pixel has a change amount smaller than a predetermined change determination value around the edge area; ,
Based on the value of the color component of each pixel before dangerous out edge regions and the small change area, for the color plane has been made with reference color among the plurality of color planes, other than the reference color shift amount of each color plane An image processing apparatus comprising: a deviation detecting unit that detects the deviation. The region detecting means includes
Extracting from the plurality of color planes a pixel column composed of pixels in the detection direction corresponding to chromatic aberration caused by the optical system used when the captured image was captured, and each pixel value of the extracted pixel column To detect an edge region of the image based on
An area in which the amount of change in the pixel value between pixels adjacent in the detection direction continues from each end of the width of the detected edge region in the detection direction is smaller than a predetermined change determination value. The image processing apparatus according to claim 1, wherein the image processing apparatus detects the small change area. The area detection unit generates each luminance value corresponding to each pixel from a plurality of color components of each pixel of the extracted pixel row in the detection direction, and a predetermined edge detection filter for each of the generated luminance values An area in which pixels having an absolute value of a processed filter output level larger than a predetermined edge level determination value continuously exist for a predetermined edge length determination value or more is detected as the edge region. Item 3. The image processing apparatus according to Item 2. The region detecting means is configured such that the absolute value of the filter output level obtained by performing predetermined edge detection filter processing on the value of each color component of each pixel of the extracted pixel row in the detection direction is based on a predetermined edge level determination value. The image processing apparatus according to claim 2, wherein an area in which large pixels continuously exist for a predetermined edge length determination value or more is detected as the edge area.   The region detection means is configured to detect the predetermined value when the amount of change in the pixel value between the adjacent pixels from the both ends of the edge region is smaller than a predetermined change determination value. 5. The image processing apparatus according to claim 3, wherein an area composed of each pixel that continues for a number of times is detected as the small change area. 6. The region detecting means detects the detection only when the absolute value of the filter output level subjected to the predetermined edge detection filter processing is smaller than a predetermined change determination value for all the pixels of the detected small change region. The selected edge area as an effective edge area,
The image processing apparatus according to claim 5, wherein the deviation detection unit detects the deviation amount only when the edge area is an effective edge area. Said region detecting means, wherein each the corresponding For each pixel of the small change area the value of each color component by integrating respectively calculates the integrated value to the ends of the detected edge region, the color plane has been made to the reference color Obtaining a gain value representing a ratio of an integral value of a color component of a color plane other than the reference color to an integral value of the color component, and calculating an absolute difference between the gain values of the small change regions respectively corresponding to the both ends of the edge region Only when the value is smaller than a predetermined gain determination value, the detected edge region is set as a valid edge region,
The image processing apparatus according to claim 5, wherein the deviation detection unit detects the deviation amount only when the edge area is an effective edge area.   The deviation detecting means matches the level of at least one of the color plane of the reference color and the color plane other than the reference color according to the pixel value in the detected small change area, and the level is adjusted. 8. The shift amount is detected based on a pixel value of a color plane of the reference color and a pixel value of each color plane other than the reference color later. The image processing apparatus described.   The region detecting means extracts a plurality of pixel columns corresponding to the detection direction from a predetermined small region including the detected edge region, and a plurality of edges based on each pixel of the plurality of pixel columns. Detecting an area, calculating each of the integral values for the plurality of edge areas, adding a plurality of integral values calculated for the plurality of edge areas, and using the added integral value The image processing apparatus according to claim 7, wherein the gain value corresponding to a predetermined small area is obtained.   The region detecting means extracts a plurality of pixel columns corresponding to the detection direction from a predetermined small region including the detected edge region, and a plurality of edges based on each pixel of the plurality of pixel columns. Detecting an area, calculating each of the integral values for the plurality of edge areas, determining the gain value using the calculated integral values for the plurality of edge areas, and each of the plurality of edge areas. When the obtained variation of the plurality of gain values falls within a predetermined range, the plurality of integral values calculated for each of the plurality of edge regions are added, and the added integral value is used to The image processing apparatus according to claim 7, wherein the gain value corresponding to a predetermined small area is obtained. On the basis of the deviation amount detected by the displacement detecting means, before machine according to any one of claims 1 to 10, characterized in that it comprises a deviation correction means for performing phase alignment of the image of each color plane Image processing device. The image processing apparatus according to claim 1, wherein the plurality of color components are red, green, and blue. A step of generating a plurality of color plane images in which each pixel is composed of the same color component by dividing each pixel of the photographed image into a plurality of color components from a captured image captured via an optical system; ,
Based on the plurality of color planes, an area detection unit detects an edge area of the image and a small change area in which each pixel has a change amount of a pixel value smaller than a predetermined change determination value around the edge area. And steps to
Deviation detection means, based on the value of the color component of each pixel before dangerous out edge regions and the small change area, for the color plane has been made with reference color among the plurality of color planes, each other than the reference color color And a step of detecting an amount of plane misalignment. Program for causing a computer to function as each unit of the image processing apparatus according to any one of claims 1 to 12.

Images