KR20090103669A - Image processing apparatus, image processing method, and program for attaining image processing - Google Patents

Abstract

PURPOSE: An image processing apparatus, an image processing method, and a program for attaining image processing are provided to process an image at a high speed while simplifying the generation process of an edge direction map by properly interpolating the omitted color components of each pixel. CONSTITUTION: A digital camera(100) is equipped with an image processing apparatus(30). The digital camera includes an optical system(10) that has a group of multiple lenses, an imaging assembly(20) that converts an image of a subject formed by the optical system into electric signals, and the image processing apparatus that receives the electric signals from the imaging assembly and makes the received electric signals subjected to a predetermined series of image processing to generate color image data. The imaging assembly has an image sensor(24) with a two-dimensional arrangement of multiple fine imaging elements for converting the light intensities into electric signals. A color filter array(22) is provided before the image sensor and has a mosaic arrangement of fine color filters of R(Red), G(Green), and B(Blue). The R color filters, the G color filters, and the B color filters are constructed to allow transmission of R color light, transmission of G color light, and transmission of B color light, respectively. The image sensor captures image data having a mosaic arrangement of image parts responsive to the R light intensities, image parts responsive to the G light intensities, and image parts responsive to the B light intensities according to the mosaic arrangement of the R, G, and B color filters in the color filter array. The image processing apparatus mounted on the digital camera receives the image data of the mosaic arrangement from the imaging assembly and generates color image data with settings of the R component, the G component, and the B component in the respective pixels. In the image processing apparatus of the embodiment, a CPU, a ROM, a RAM, and a data IF(Input/output interface) are interconnected via a bus to enable mutual data transmission. The CPU performs a series of processing to generate the color image data according to a program stored in the ROM. The resulting color image data thus generated may be output to an external device via an external output terminal(40) or may be output to an external recording medium(50).

Description

Image processing apparatus, image processing method, and program for achieving image processing {IMAGE PROCESSING APPARATUS, IMAGE PROCESSING METHOD, AND PROGRAM FOR ATTAINING IMAGE PROCESSING}

The present invention relates to a technique for generating image data with an imaging device such as a digital camera. More specifically, the present invention relates to a color filter array having a mosaic arrangement of fine color filters of R (red), G (green), and B (blue) corresponding to the three primary colors of light. It relates to a technique for generating color image data by application.

With advances in digital technology, images are generally processed as digital data (image data). An imaging device such as a digital camera makes it possible to immediately output a captured image in the form of image data. Imaging devices are generally equipped with an electrical image sensor composed of small elements for converting light intensity into an electrical signal. The imaging device focuses the captured image of the subject on the image sensor by the optical system, and detects the intensity of light as an electrical signal in the individual elements to generate image data. Light incident on the optical system may be separated into three color components R, G, and B corresponding to three primary colors of light. Each color light of the three color components R, G, and B is focused on the image sensor, and an electrical signal indicative of the light intensity of each color component is output to generate color image data.

The simplest method of focusing each color light of the three color components R, G, and B obtained as the separation of the light incident on the optical system onto the image sensor is by using spectroscopic prism to focus the incident light into three color components. It separates into the color light of R, G, and B and focuses each color light on an image sensor, and produces the image data regarding each color component R, G, and B. FIG. This method undesirably requires three image sensors. One widely used method is to assign one of the color components of R, G, and B to each element constituting the image sensor to achieve detection of each color component R, G, and B by one image sensor. will be. The general construction of this technique is a small color filter allowing the transmission of only the R component to the front of the device specified for the detection of the R component, and a small color filter allowing the transmission of only the G component to the front of the device specified for the detection of the G component. A color filter, and a small color filter allowing the transmission of only the B component on the front of the device designated for the detection of the B component. This configuration enables simultaneous detection of image data of three color components R, G, and B with one image sensor. In the technique of detecting each color component R, G, and B with one image sensor, each element designated for the detection of a predetermined color component (e.g., R component) has a different color component (e.g., G component and B component). ) Cannot be detected. Thus, the resulting image data has a mosaic arrangement of pixels of the R component, pixels of the G component, and pixels of the B component. Interpolation of the missing color component of each pixel with the color component of the neighboring pixel enables generation of color image data having the settings of all the color components R, G, and B of all the pixels.

An imaging device that converts separate color light of three color components R, G, and B into an electrical signal depending on three image sensors to generate image data of each color component R, G, and B is sometimes called a '3 image. Sensor 'device. An imaging device that generates image data in a mosaic array using only one image sensor and calculates missing color components by interpolation is sometimes referred to as a "single image sensor" device. The process of generating color image data having the settings of all the color components R, G, and B by interpolating the missing color components of the image data of the mosaic array is sometimes referred to as "demosaicking process".

Single image sensor devices require interpolation of missing color components. This naturally takes time for interpolation and can result in the generation of pseudo colors due to interpolation errors. There are various proposed techniques aimed at minimizing the increase in time required for interpolation while preventing the generation of pseudo colors. One proposed technique computes the color difference component (e.g., the difference between the G component and the R component) of each pixel after the calculation of the missing color component, and maximizes from a pixel array of a preset number of pixels including the target pixel. Pixels having a chrominance component and pixels having a minimum chrominance component are removed as noise, and each color component of the target pixel is recalculated (see Japanese Patent Laid-Open No. 2005-167974). Another proposed technique applies a low-pass filter to the computed chrominance components of each pixel and recalculates each color component of the target pixel from the chrominance components after removal of noise (Japanese Patent Laid-Open No. 2005- 260908).

With the increasing demand of consumers for higher picture quality of imaging devices, the development of demosaicing techniques to prevent the generation of pseudo colors has been highly demanded. In order to meet consumer demand for higher picture quality, the number of pixels constituting each image captured by the imaging device is increasing. There is also a need for development of demosaicing technology that enables high speed processing.

In order to simultaneously meet these contradictory requirements for prior art demosaicing processing, there would be a need to provide a demosaicing technique that enables high speed processing while effectively preventing the generation of pseudo colors.

In order to achieve at least some of the aforementioned requirements, one aspect of the present invention provides mosaic image data having a setting of a single color component among three color components R, G, and B corresponding to the three primary colors of light of each pixel. An image processing apparatus for generating color image data having settings of all three color components R, G, and B of each pixel by subjecting the received mosaic image data to a series of image processing. The mosaic image data includes a pixel column having an alternating arrangement of pixels of the G component and pixels of the R component, which are different color components in the vertical direction, and alternating pixels of the B component, which are pixels of the G component and other color components in the vertical direction. A pixel column having an array, a pixel row having an alternating arrangement of pixels of the G component and pixels of the R component, which are different color components in the horizontal direction, and pixels of the component B, which are pixels of the G component and other color components in the horizontal direction Is represented by a combination of pixel rows having an alternating array of.

The image processing apparatus includes a color difference calculation module in two directions, an edge direction detection module, and a missing color component interpolation module. The vertical color difference component calculating module calculates the vertical color difference component between the G component of each pixel included in each pixel column of the mosaic image and the other color components, using the vertically neighboring available components. Obtain a G component at a pixel location or obtain a different color component at a G component pixel. The horizontal color difference component calculating module uses the horizontally neighboring color components to calculate the horizontal color difference component between the G component of each pixel included in each pixel row of the mosaic image and the other color components. Obtain a G component at a location or other color component at a G component pixel. The edge direction detection module selects the pixels of the R component and the pixels of the B component from the mosaic image, and at each pixel position, changes the vertical color difference component with respect to at least each of the selected pixels, and changes the horizontal color difference component. Detect at least the edge direction of the selected pixel. Finally, the missing color component interpolation module interpolates the missing color component of each pixel of the mosaic image data with the setting of one color component of each pixel of the mosaic image data with reference to the detected edge direction.

Another aspect of the present invention is an image processing method corresponding to the image processing apparatus of the above configuration. The image processing method receives mosaic image data having a setting of a single color component among three color components R, G, and B corresponding to three primary colors of light of each pixel, and performs a series of image processing on the received mosaic image data. To generate color image data having the settings of all three color components R, G, and B of each pixel.

The image processing method includes a pixel column having an alternating arrangement of pixels of the G component and an R component pixel which is another color component in the vertical direction, and an alternating arrangement of pixels of the G component and B component which are other color components in the vertical direction. A pixel column having an alternating arrangement of pixels of the G component and pixels of the R component, which are other color components in the horizontal direction, and alternating arrays of pixels of the B component, which are pixels of the G component and other color components in the horizontal direction Image data represented by a combination of pixel rows is received as mosaic image data. The image processing method is then obtained by interpolating the G component of each pixel included in each pixel column of mosaic image data by interpolation, and different from the G component of each pixel included in each pixel column in the vertical direction. Calculate the vertical color difference component between color components. Similarly, the image processing method obtains color components different from the G component of each pixel included in each pixel row of the mosaic image by interpolation, so as to differ from the G component of each pixel included in each pixel row in the horizontal direction. The horizontal color difference component between color components is calculated. The image processing method then selects the pixels of the R component and the pixels of the B component from the mosaic image data, and at each of the pixel positions, changes the amount of change in the vertical color difference component with respect to each of at least the selected pixels in the horizontal color difference component. At least the edge direction of the selected pixel is detected in comparison with the amount of change in. The image processing method interpolates the missing color component of each pixel of the mosaic image data with the setting of one color component of each pixel of the mosaic image data with reference to the detected edge direction.

In the image processing apparatus and the corresponding image processing method according to an aspect of the present invention, the procedure is performed for each pixel row constituting mosaic image data and the vertical color difference component of each pixel column constituting the received mosaic image data. Both sides of the horizontal color difference component are calculated. The mosaic image data includes a pixel column having an alternating arrangement of pixels of a G component and an R component pixel in a vertical direction, a pixel column having an alternating arrangement of pixels of a G component in a vertical direction and a pixel of a B component, and a G in a horizontal direction A pixel row having an alternating arrangement of pixels of a component and a pixel of an R component, and a pixel row having an alternating arrangement of pixels of a G component and a pixel of a B component in the horizontal direction. The color difference component between the G component and the R component or the color difference component between the G component and the B component is obtained as the vertical color difference component of each pixel column. Similarly, a color difference component between the G component and the R component or a color difference component between the G component and the B component is obtained as the horizontal color difference component of each pixel row. The procedure then selects pixels of the R component and B components from the mosaic image data, and compares the amount of change in the vertical color difference component with the amount of change in the horizontal color difference component with respect to at least each of the selected pixels. To detect the edge direction. As described above, each of the vertical color difference component and the horizontal color difference component represents a color difference component between the G component and the R component or a color difference component between the G component and the B component. In the pixel of the R component, both the vertical color difference component and the horizontal color difference component are color difference components between the G component and the R component. Thus, the comparison between the vertical color difference component and the horizontal color difference component enables accurate detection of the edge direction of each pixel of these R components. In the pixel of the B component, both the vertical color difference component and the horizontal color difference component are color difference components between the G component and the B component. Thus, the comparison between the vertical color difference component and the horizontal color difference component enables accurate detection of the edge direction of each pixel of these B components. The procedure interpolates the missing color component of each pixel of the mosaic image data with the setting of one color component of each pixel of the mosaic image data with reference to the correctly detected edge direction. Thus, such an arrangement makes it possible to generate proper color image data.

In the image processing apparatus according to one aspect of the present invention, the amount of change in the vertical color difference component and the amount of change in the horizontal color difference component used for detecting the edge direction of each pixel can be calculated in the following manner. According to one preferred application, the image processing apparatus first selects each target pixel as an object of detection in the edge direction. The amount of change in the vertical color difference component of the target pixel is calculated from the vertical color difference component of the target pixel column containing the target pixel and the vertical color difference component of the adjacent pixel column adjacent to the target pixel column. As an example, the amount of change in the vertical chrominance component of the target pixel may be calculated from the target pixel column including the target pixel and the vertical chrominance component of two pixel columns, that is, two adjacent pixel columns adjacent to the target pixel column. have. As another example, the amount of change in the vertical color difference component of the target pixel is the vertical color difference of the larger number of pixel columns, that is, at least the target pixel column including the target column and two adjacent pixel columns on both sides of the target pixel column. It can be calculated from the components. Similarly, the amount of change in the horizontal color difference component of the target pixel is calculated from the horizontal color difference component of the target pixel row containing the target pixel and the horizontal color difference component of the adjacent pixel row adjacent to the target pixel row. In this case, the amount of change in the horizontal chrominance component of the target pixel may be calculated from two pixel rows, that is, the horizontal chrominance component of the target pixel row including the target pixel and one neighboring pixel row adjacent to the target pixel row. Or from a horizontal color difference component of a larger number of pixel rows.

The calculation based on a target pixel column including a target pixel as an object of edge direction detection and a multi-pixel column including at least one neighboring pixel column adjacent to the target pixel column is higher than the calculation based on a single pixel column. The amount of change in the vertical color difference component is determined with accuracy. Similarly, a calculation based on a target pixel row containing a target pixel and a multiple pixel row including at least one neighboring pixel row adjacent to the target pixel row is horizontal with higher precision compared to a calculation based on a single pixel row. The amount of change in the directional color difference component is determined. The comparison between the precisely calculated vertical color difference component of the target pixel and the precisely calculated horizontal color difference component enables proper detection of the edge direction of the target pixel.

There are two different types of vertical color difference components of the pixel column, namely the color difference component between the G component and the R component and the color difference component between the G component and the B component. Vertical color difference components of pixel columns adjacent to each other may not have the same type of vertical color difference components. In this case, it is considered that the calculation of the amount of change from different types of color difference components is meaningless. However, it is empirically known that even different types of chrominance components have variations in the similar tendency of nearby positions of the image. Therefore, it is important even if the amount of change in the vertical color difference component of the target pixel calculated from the different types of color difference components. Therefore, the calculation from the vertical color difference component of multiple neighboring pixel columns determines the amount of change in the vertical color difference component of the target pixel with high precision. There are also two different types of horizontal color difference components of the pixel rows, namely, the color difference component between the G component and the R component and the color difference component between the G component and the B component. Therefore, even if the amount of change in the horizontal color difference component of the target pixel calculated from the different type of color difference components is important. Therefore, the calculation from the horizontal color difference component of the multiple neighboring pixel rows determines the amount of change in the horizontal color difference component of the target pixel with high precision. The comparison between the amount of change in the vertical color difference component of the target pixel and the amount of change in the horizontal color difference component enables proper detection of the edge direction of the target pixel.

According to another preferred application of the above aspect of the present invention, the image processing apparatus detects edge directions of only pixels of the R component and pixels of the B component included in the mosaic image data.

The pixels of the R component and the pixels of the B component of the mosaic image data require interpolation of the G components. The G component is closer to a particular wavelength where human vision is more sensitive to the amount of change in brightness, compared to the B and R components. Thus, the G component has a greater impact on the resulting picture quality. The detection of the edge direction of the pixel of the R component and the pixel of the B component of the mosaic image data enables the G component to be interpolated to these pixels with reference to the detected edge direction. This arrangement ensures effective and proper interpolation even if no edge direction is detected for every pixel.

According to another preferred application of the above aspect of the present invention, when the amount of change in the vertical color difference component of a pixel is larger than the amount of change in the horizontal color difference component of the pixel, the image processing apparatus makes the edge direction of each pixel in the horizontal direction. Detect. When the amount of change in the vertical color difference component of the pixel is smaller than the amount of change in the horizontal color difference component of the pixel, the image processing apparatus detects the edge direction of each pixel in the vertical direction. When the amount of change in the vertical color difference component of the pixel is equal to the amount of change in the horizontal color difference component of the pixel, the image processing apparatus detects the edge direction of each pixel as the vertical direction or the horizontal direction. In this case, the edge direction of the pixel can be fixed in the vertical direction or the horizontal direction, or can be switched alternately or randomly between the vertical direction and the horizontal direction.

The edge direction is vertical or horizontal. This setting significantly simplifies the interpolation of missing color components. In this application, interpolation is performed under the assumption that the vertical edge or the horizontal edge actually exists even in pixels with diagonal edges or even no edges. However, the assumption of the existence of vertical edges or horizontal edges of pixels without edges usually does not affect the resulting picture quality. In addition, interpolation by estimation of the vertical and horizontal edge directions of pixels having diagonal edges usually does not significantly deteriorate the resulting photographic quality due to using the precise interpolation concept described below.

According to another preferred application of the above aspect of the present invention, the image processing apparatus comprises a G component which is a missing color component of an interpolation target pixel, which is an object of interpolation, with a pair of upper and lower neighboring pixels above and below the interpolation target pixel. Interpolate with the color components of a pair of neighboring pixels selected between a pair of left and right neighboring pixels on the left and right of the interpolation target pixel.

When the edge direction detected in the interpolation target pixel is the vertical direction, the G component of the interpolation target pixel is interpolated with the color component obtained in the pair of upper and lower neighboring pixels above and below the interpolation target pixel. On the other hand, when the edge direction detected in the interpolation target pixel is the horizontal direction, the G component of the interpolation target pixel is interpolated with the color component obtained in the pair of left and right neighboring pixels on the left and right sides of the interpolation target pixel. If the interpolation target pixel does not have an edge, the G component of the interpolation target pixel is either a pair of upper and lower neighboring pixels above and below the interpolation target pixel or a pair of left and right neighboring pixels left and right of the interpolation target pixel. It is interpolated with the obtained color component. This arrangement allows for fast interpolation of the G components based on the color components of two neighboring pixels. Interpolation target pixels, which are objects of G component interpolation, are non-G pixels (R pixels and B pixels) without setting of the G component of the mosaic image data. Mosaic image data includes a relatively large number of said non-G pixels. Fast interpolation of the G component significantly accelerates the first half of the process of interpolating missing color components of the mosaic image data.

In a preferred embodiment of the above application for interpolation of the G component of the interpolation target pixel, the image processing apparatus is arranged so that the vertical color difference component or the horizontal direction color difference component of the interpolation target pixel is adjacent to the upper and lower portions of the upper and lower portions of the interpolation target pixel. Interpolate with the vertical chrominance component of the pair of pixels or the horizontal chrominance component of the pair of left and right neighboring pixels on the left and right of the interpolation target pixel. Then, the image processing apparatus calculates the G component of the interpolation target pixel based on the setting of the vertical color difference component or the horizontal color difference component interpolated in the interpolation target pixel and the color component of the interpolation target pixel of the mosaic image data. .

In an interpolation pixel in which the object of interpolation is a G component, an R component or a B component exists. The color difference component in the vertical direction and the color difference component in the horizontal direction are given as the difference between the G component and the existing component. The image processing apparatus having the above-described configuration includes the vertical color difference component or the horizontal color difference component of the interpolation target pixel and the vertical color difference component of the upper and lower neighboring pixels above and below the interpolation target pixel, or the left and right sides of the interpolation target pixel. Interpolate with the horizontal chrominance component of the right neighboring pixel. The G component of the interpolation target pixel can be computed immediately from the interpolated vertical color difference component or horizontal color difference component and the color component of the mosaic image data set of the interpolation target pixel. The calculation from the interpolated chrominance components in the vertical or horizontal direction ensures proper calculation of the G components and effectively prevents the generation of pseudo colors. A significant change in the difference between the G and R components or a significant change in the difference between the G and B components leads to a noticeable color change. Interpolation of each of the color components R, G, and B causes the difference between the color components of the interpolation target pixel to change significantly from the corresponding difference between the color components of the neighboring pixels. This can lead to the generation of pseudo colors. However, the calculation of the G component from the interpolation of the color difference component in the vertical direction or the horizontal direction and the interpolated color difference component prevents a large amount of change of the difference between the color components in the interpolating direction at least. This arrangement effectively prevents the generation of pseudo colors.

In another preferred embodiment of the application, the image processing apparatus interpolates the G components of all pixels without the setting of the G components of the mosaic data, and then interpolates other missing color components of the pixels.

As described above, the G component of each interpolation target pixel is interpolated quickly based on the raw components obtained at neighboring pixels above and below the interpolation target pixel or neighboring pixels to the left and right of the interpolation target pixel. After fast interpolation of the G component of all pixels without setting the G component of the mosaic image data, other missing color components can be interpolated immediately and quickly with the setting of the interpolated G component and the color component of the pixel of the mosaic image data. .

In another preferred application of the above aspect of the present invention, before interpolation of the missing color component, the image processing apparatus detects an edge direction of at least an R component pixel and a B component pixel included in the mosaic image data, thereby providing an edge direction map. Create a map. The image processing apparatus interpolates missing color components of each pixel of the mosaic image data with reference to the edge directions stored in the edge direction map.

In the image processing apparatus of such a configuration, detection of the edge direction is separated from interpolation of missing color components of each pixel, thus ideally simplifying the first half of the processing. Edge detection of multiple pixels can be obtained simultaneously by simply referencing an edge direction map. The missing color component of each pixel is appropriately interpolated based on the edge direction of the neighboring pixel. In the structure in which the edge direction is detected only with respect to the pixel of the R component and the pixel of the B component of the mosaic image data, there are pixels without detection of the edge direction. Even in the above case, the missing color component of each pixel may be interpolated based on the edge direction of the pixel estimated from the detected edge direction of the neighboring pixel.

In another application of the present invention, a computer reads out a program stored in a computer readable medium for realizing the above-described image processing method to achieve each function. Accordingly, another aspect of the present invention is to receive mosaic image data having a setting of a single color component among three color components R, G, and B corresponding to the three primary colors of light of each pixel and serially receive the received mosaic image data. A computer readable medium storing a program for causing a computer to realize a method of generating color image data having settings of all three color components R, G, and B of each pixel by performing image processing.

The program,

A pixel column having an alternating arrangement of pixels of the G component and an R component pixel which is another color component in the vertical direction, a pixel column having an alternating arrangement of pixels of the G component and a B component pixel which is another color component in the vertical direction, a G component A combination of pixels rows having an alternating arrangement of pixels of R and pixels of R components that are different color components in the horizontal direction, and pixels rows having an alternating arrangement of pixels of G components and pixels of B components, which are other color components in the horizontal direction. Receiving image data to be expressed as mosaic image data;

In order to calculate the vertical color difference component between the G component of each pixel included in each pixel column of the mosaic image data and the other color components, the G component of the pixel position of the other color component is used by using the available components that are vertically neighboring. Obtaining or obtaining another color component of the G component pixel position;

In order to calculate the horizontal color difference component between the G component of each pixel included in each pixel row of the mosaic image data and the other color components, horizontally neighboring color components are used to obtain the G components of pixel positions of other color components. Or to obtain another color component of the G component pixel position;

From the mosaic image data, pixels of the R component and pixels of the B component are selected, and at each of the pixel positions, at least the edge direction of the selected pixel is determined by comparing the change amount of the vertical color difference component with respect to the selected pixel with the change amount of the horizontal color difference component. Detecting; And

With reference to the detected edge direction, the computer causes the functions to interpolate the two missing color components of each pixel of the mosaic image data with the setting of one color component of each pixel of the mosaic image data.

The computer reads the program according to this aspect of the present invention to achieve each of the above functions and to appropriately interpolate missing color components of the mosaic image data. Thus, this arrangement ensures high speed generation of color image data and effectively prevents generation of pseudo colors.

The present invention ensures highly precise detection of even small sized edges and thus makes it possible to generate a precise edge direction map. The present invention properly interpolates the missing color components of each pixel with reference to such an accurate edge direction map, so that the generation process and the interpolation of the missing color components of the edge direction map can be executed at a simple and high speed. Therefore, the present invention enables high-speed generation of appropriate color image data from original image data having a mosaic arrangement of R, G, and B color components.

1 schematically illustrates a configuration of a digital camera equipped with an image processing apparatus of an embodiment of the present invention.

FIG. 2 is a conceptual diagram illustrating a structure of a color filter array and an image sensor included in the image processing device of FIG. 1.

Fig. 3 is a flowchart showing color image data generation processing executed by the image processing apparatus of the embodiment to generate color image data from raw image data.

4 conceptually illustrates a process of converting original image data into data of color difference components.

5 shows a calculation formula for calculating the color difference component from the original image data.

6 illustrates a low pass filter applied for the removal of noise included in a chrominance component.

FIG. 7 is a flowchart showing details of the edge direction map generation process executed in the color image data generation process of FIG. 3 by the image processing apparatus of the embodiment. FIG.

Fig. 8 shows the calculation of the change amount Vv in the vertical direction from the color difference component CDv in the vertical direction.

9 shows the calculation of the change amount Vh in the horizontal direction from the color difference component CDh in the horizontal direction.

10 shows a possible arrangement of neighboring pixels around a target pixel selected for detection of the edge direction.

 11 shows one method applicable to detection of the edge direction of a G pixel.

12 shows another method applicable to detection of the edge direction of a G pixel.

13 shows another method that can be applied to calculating the amount of change in the color difference component.

14 shows another method that can be applied to calculating the amount of change in the color difference component.

FIG. 15 shows another method applicable to calculating even small amounts of change in color difference components.

FIG. 16 is a flowchart showing details of the G component interpolation processing executed in the color image data generation processing of FIG. 3 by the image processing apparatus of the embodiment.

17 shows a processing flow for computing missing G components of non-G pixels.

FIG. 18 is a flowchart showing details of non-G pixel interpolation processing executed in the color image data generation processing of FIG. 3 by the image processing apparatus of the embodiment. FIG.

FIG. 19 is a flowchart showing details of the G pixel interpolation processing executed in the color image data generation processing of FIG. 3 by the image processing apparatus of the embodiment.

20 shows the estimation of the edge direction of the G interpolation target pixel from the edge direction of the neighboring pixel.

The present invention may be more fully understood with reference to the following preferred embodiment of the following sequence based on the accompanying drawings.

A. System Configuration

B. Outline of Color Image Data Generation Processing

C. Edge Direction Map Generation Processing

D. G component interpolation

E. Non-G Pixel Interpolation

F. G pixel interpolation

A. System Configuration

1 schematically illustrates the configuration of a digital camera 100 equipped with an image processing apparatus 30 of an embodiment of the present invention. As illustrated, the digital camera 100 includes an optical system 10 having a group of multiple lenses, an imaging assembly 20 for converting an image of a subject formed by the optical system 10 into an electrical signal, and an imaging assembly. And an image processing apparatus 30 that receives the electric signal from the 20 and performs a series of preset image processing on the received electric signal to generate color image data.

The imaging assembly 20 has an image sensor 24 having a two dimensional array of multiple fine imaging elements for converting light intensity into an electrical signal. The color filter array 22 is provided in front of the image sensor 24 and has a mosaic arrangement of fine color filters of R (red), G (green), and B (blue). The arrangement of the R, G, and B color filters constituting the color filter array 22 will be described later in detail. The R color filter, the G color filter, and the B color filter were configured to enable transmission of R color light, transmission of G color light, and transmission of B color light, respectively. The image sensor 24 includes an image portion sensitive to the R light intensity, an image portion sensitive to the G light intensity, and B, according to a mosaic arrangement of the R, G, and B color filters of the color filter array 22. Image data having a mosaic arrangement of image portions sensitive to light intensity is captured.

The image processing apparatus 30 mounted on the digital camera 100 receives image data in a mosaic array from the imaging assembly 20 and generates color image data having settings of R, G, and B components of each pixel. . In the image processing apparatus 30 of the present embodiment, the CPU, the ROM, the RAM, and the data input / output interface (I / F) are interconnected via a bus so as to enable mutual data transfer. The CPU executes a series of processes to generate color image data in accordance with the program stored in the ROM. Thus, the resulting color image data can be output to the external device via the external output terminal 40 or can be output to the external recording medium 50.

The image data having a mosaic arrangement of R, G, and B components captured by the image sensor 24 may be used to generate color image data having settings of the R, G, and B components of each pixel. It is used as source data referenced by 30). Therefore, the image data of the mosaic arrangement captured by the image sensor 24 may be referred to as 'raw image data' below.

2 is a conceptual diagram illustrating the structures of the color filter array 22 and the image sensor 22. As mentioned above, the image sensor 24 has a two-dimensional array of fine imaging elements for outputting an electrical signal corresponding to the light intensity. In the illustrated example of FIG. 2, the fine imaging elements are arranged in a lattice pattern. Each of the small rectangles of the lattice pattern of the image sensor 24 conceptually represents one imaging device.

The color filter array 22 has one R color filter, a G color filter, and a B color filter set corresponding to each position of the multiple imaging elements constituting the image sensor 24. In FIG. 2, a sparse hatched rectangle, a tightly hatched rectangle, and a blank hatched rectangle represent an R color filter, a B color filter, and a G color filter, respectively. In the arrangement of the R, G, and B color filters, the G color filters are first positioned diagonally to each other, forming a checkerboard pattern. That is, the G color filter occupies half of the area of the color filter array 22. The same number of R color filters and B color filters are then evenly arranged in the remaining half of the color filter array 22. The resulting color filter array 22 of this arrangement shown in FIG. 2 is referred to as a Bayer color filter array.

As mentioned above, the G color filter, the R color filter, and the B color filter are designed to enable transmission of only the G color light, transmission of only the R color light, and transmission of only the B color light, respectively. Thus, the image sensor 24 captures image data in a mosaic arrangement by the function of the Bayer color filter array 22 located in front of the image sensor 24 as shown in FIG. The image data of the mosaic arrangement cannot be processed in the same manner as ordinary image data, and cannot be represented directly as an image. The image processing apparatus 30 receives the image data (original image data) of the mosaic arrangement and generates normal color image data having the settings of the R, G, and B components of each pixel.

B. Outline of Color Image Data Generation Processing

3 is a flowchart showing color image data generation processing executed by the image processing apparatus 30 of the embodiment to generate color image data from original image data. In the structure of this embodiment, the CPU included in the image processing apparatus 30 executes such color image data generation by a software configuration. However, this is not essential and certain hardware elements, such as signal processors, may be used for the same purpose.

At the start of the color image data generation process, the CPU first receives the original image data which is the source data from the image sensor 24 (step S100), and converts the received original image data into the data of the color difference component as described later in detail. (Step S102).

4 conceptually illustrates a process of converting original image data into data of color difference components. 4 shows a conceptual diagram of the original image data read out from the image sensor 24. The original image data obtained by the Bayer color filter array has a mosaic arrangement of R, G, and B components. In the horizontal direction, two types of pixel rows of original image data: pixel rows having alternatingly arranged G pixels (pixels having G components) and R pixels (pixels having R components), and alternately arranged There are only pixel rows with G pixels and B pixels (pixels with B components). Similarly, in the vertical direction, only two types of pixel columns of original image data: pixel columns having alternately arranged G and R pixels, and pixel columns having alternately arranged G and B pixels. There is this. In the middle of Fig. 4, the original image data consists only of alternating G and R pixels in the horizontal and pixel rows of alternating G and B pixels, and alternating pixel and alternating pixel and R pixel columns. Conceptually illustrating what consists of only pixel columns of G pixels and B pixels.

Based on the results of such observation, the color image data generation processing of the present embodiment uses the original image data of the R, G, and B components to express the difference (the horizontal color difference component) of the color components of the pixel rows in the horizontal direction. And data representing the difference (color difference component in the vertical direction) of the color component in the pixel column in the vertical direction. For example, the original image data is converted into data of color difference components between G and R components with respect to pixel rows or pixel columns of alternating G and R pixels. Similarly, the original image data is converted into data of chrominance components between G and B components with respect to pixel rows or pixel columns of alternating G and B pixels. The formula for calculating the color difference component will be described later in detail.

The lower part of FIG. 4 conceptually illustrates the color difference components in the horizontal and vertical directions thus obtained. The lower left figure shows the vertical color difference component CDv obtained by processing the original image data in the vertical direction. Each of the hatched pixel columns is an alternating pixel column of G and B pixels. Therefore, each color difference component CDv of this pixel column represents a color difference component between the G component and the B component. Each of the pixel columns without the remaining hatch is a pixel column of alternating G and R pixels. Therefore, each color difference component CDv of this pixel column represents a color difference component between the G component and the R component. The original image data in the horizontal direction can be similarly processed. Each of the hatched pixel rows imparts a color difference component (CDh) between G and B pixels. Each of the pixel rows that do not have the remaining hatch gives the color difference component CDh between the G pixels and the R pixels.

5 shows a calculation formula for calculating the color difference component from the original image data. FIG. 5A shows a calculation formula for calculating the color difference component CDv in the vertical direction, and FIG. 5B shows a calculation formula for calculating the color difference component CDv in the horizontal direction. In these calculations, 'z' represents the value of the original image data obtained by the image sensor 24, and z (r, s) is the r-th pixel position and the original position downward from the position of the original set of images. Denotes a value of a specific position defined by the s-th pixel position in the rightward direction (see upper portion of FIG. 4).

Referring to FIG. 5 (a), the description relates to a calculation formula for calculating the color difference component CDv in the vertical direction. When the target pixel, which is the object of the calculation of the chrominance component CDv, is a pixel having the G component (G pixel), the upper calculation formula of Fig. 5A is adopted. The process first yields an average value of the upper and lower pixels above and below the target pixel (i.e., the second term surrounded by dashed lines) and an average calculated from the G component of the target pixel (i.e., surrounded by dashed lines). Subtraction) to calculate the color difference component CDv of the target pixel in the vertical direction. Since the target pixel for the calculation of the chrominance component CDv is a G pixel, the tone value z (r, s) of the target pixel naturally represents the G component. The element z (r-1, s) of the second term represents the hue value of the original image data of the upper pixel above the target pixel, while the element z (r + 1, s) of the second term represents the lower pixel of the lower pixel below the target pixel. The hue value of the original image data is shown. As described above with reference to FIG. 4, the top and bottom pixels above and below the target G pixel can be either R pixels or B pixels but are always pixels of the same color component. Therefore, the second term surrounded by the dashed-dotted line in the upper equation of Fig. 5A represents the R component or the B component calculated from the values of the upper and lower pixels (i.e., R pixels or B pixels). Application of the upper equation of Fig. 5 (a) determines the color difference component CDv in the vertical direction with respect to the pixel having the G component (G pixel), irrespective of the upper and lower pixels which are R pixels or B pixels.

It should be noted that the application of the same equation for the upper and lower pixels, R and B pixels, gives two different types of color difference components (CDv). When the upper and lower pixels are R pixels, the resulting color difference component CDv in the vertical direction represents the color difference component between the G component and the R component. On the other hand, when the upper and lower pixels are B pixels, the resulting color difference component CDv in the vertical direction represents the color difference component between the G component and the B component.

The color difference component CDv in the vertical direction can be similarly calculated with respect to pixels other than G pixels (ie, R pixels and B pixels). For example, if the target pixel, which is an object of the operation of the chrominance component CDv, is an R pixel, the upper and lower pixels above and below the target pixel are G pixels as described above with reference to FIG. The average value of the upper and lower pixels is considered to designate the G component of the target pixel, which is an object of the operation of the chrominance component CDv. The subtraction of the hue value of the target pixel (in this case, the R component) in the designated G component determines the chrominance component CDv. Each of the set of B pixels that is the target pixel receives the same series of processing. As for the pixels other than the G pixels, as shown in the lower calculation formula of FIG. 5 (a), the subtraction of the second term surrounded by the dashed line in the first term surrounded by the dashed line on the right indicates that the color difference component in the vertical direction ( CDv).

The same calculation can be applied to the calculation of the color difference component CDv in the vertical direction with respect to both the R pixel and the B pixel set which are target pixels other than the G pixel. However, it should be noted that the R and B pixels treated as target pixels impart two different types of color difference components (CDv). When the target pixel is an R pixel, the resulting color difference component CDv represents the color difference component between the G component and the R component. On the other hand, when the target pixel is a B pixel, the resulting color difference component CDv represents the color difference component between the G component and the R component.

The color image data generation process of this embodiment applies the exact same calculation formula for the advanced process with two different types of color difference components (CDv). The application of the same formula ideally achieves the simplicity and high speed of advanced processing with two different types of color difference components (CDv) which can be compared to conventional simple processing. However, unlike conventional simple processing, this advanced processing of the present embodiment enables proper demosaicing as well as prevention of generation of pseudo color.

The color difference component CDh in the horizontal direction may be calculated in the same manner as the color difference component CDv in the vertical direction described above. The calculation of the color difference component CDh is briefly described with reference to Fig. 5B. As shown in the upper calculation formula of Fig. 5B, the subtraction of the second term surrounded by the dashed line in the first term surrounded by the broken line on the right gives the color difference component CDh of the G pixel in the horizontal direction. In the formula, the element z (r, s) represents the hue value (that is, the G component) of the original image data of the target pixel that is the object of the calculation of the chrominance component. The element z (r, s-1) represents the hue value of the original image data of the left pixel to the left of the target pixel, while the element z (r, s + 1) represents the hue value of the original image data of the right pixel to the right of the target pixel. Indicates. The left and right pixels to the left and right of the target pixel can be either R pixels or B pixels but are always pixels of the same color component.

In the calculation of the color difference component CDh in the horizontal direction with respect to the target pixel other than the G pixel (ie, with respect to the R target pixel or the B target pixel), the left and right pixels on the left and right sides of the R or B target pixel are G. Pixels. As shown in the lower equation of Fig. 5 (b), the subtraction of the second term (the hue value of the target pixel) surrounded by the dashed line in the first term (average value of the left and right pixels) surrounded by the dashed line is horizontal. The color difference component CDh in the direction is given.

The same calculation calculates the horizontal color difference component (CDh) for a target G pixel having R pixels or B pixels as left and right pixels and for both a set of R pixels and B pixels that are target pixels other than G pixels. Can be applied to However, the operation gives two different types of color difference components (CDh), namely the color difference component between the G component and the R component and the color difference component between the G component and the B component. The application of completely identical calculations ideally achieves the simplicity and high speed of advanced processing with two different types of color difference components (CDh) in the horizontal direction, which can be compared to conventional simple processing.

As clearly understood from the comparison between Figs. 5 (a) and 5 (b), the equation used for the calculation of the color difference component CDv in the vertical direction is the equation used for the calculation of the color difference component CDh in the horizontal direction. Is quite similar to That is, the process flow can be normalized for the calculation of both of the color difference components (CDv, CDh) in different two directions. This further improves the simplicity and high speed of the overall process.

In step S102 of the flowchart of FIG. 3, the original image data captured by the image sensor 24 is converted into a vertical color difference component CDv and a horizontal color difference component CDh. Although not essential, it is desirable to apply a low pass filter to the vertical color difference component CDv and the horizontal color difference component CDh for the purpose of removing likely noise from these color difference components. As will be described later, the color image data generation processing of the present embodiment uses the color difference component for detection of the edge direction included in the image. Application of low pass filters is generally undesirable because low pass filters tend to fade edges and degrade edge detection accuracy. However, the procedure of this embodiment enables detection of the edge direction with extremely high precision as described below. Thus, slight edge blur does not significantly reduce the detection accuracy, and noise removal enables stable detection in the edge direction.

6 illustrates a low pass filter applied to remove noise included in a chrominance component. Fig. 6A shows a low pass filter applied to the color difference component CDv in the vertical direction and to the color difference component CDh in the horizontal direction. For a better understanding, Fig. 6 (b) shows the application of the low pass filter to the color difference component CDh in the horizontal direction. The target pixel, which is the object of low pass filter application, is indicated by the rectangle with hatch in Fig. 6 (b). The low pass filter of this embodiment sums the color difference components of the target pixel and the color difference components of two left pixels and two right pixels on the left and right sides of the target pixel. The color difference component of the target pixel is multiplied by the weight of four pixels, so that the summed color difference component corresponds to eight pixels. In this case, a simple three-bit shift of the sum to a lower bit imparts a color difference component of one pixel equal to the sum divided by eight. This weighting makes it possible to perform high speed processing by the low pass filter.

As described previously, each of the vertical color difference component CDv and the horizontal color difference component CDh may represent a color difference component between the G component and the R component or a color difference component between the G component and the B component. The low pass filter can be applied to the vertical color difference component CDv and the horizontal color difference component CDh without considering those different types of color difference components. Thus, the color image data generation processing of this embodiment achieves the extreme simplicity and high speed of the advanced processing.

Referring again to the flowchart of Fig. 3, after calculating the color difference component CDv in the vertical direction and the color difference component CDh in the horizontal direction, the color image data generation process of the present invention performs the data of these color difference components. A low edge direction map is generated (edge direction map generation process) (step S104). The edge direction map indicates the direction of each edge included in the image in the vertical direction or the horizontal direction. The edge direction may not be necessarily set for all pixels, but should be set for at least non-G pixels (ie R pixels and B pixels). The color image data generation processing of this embodiment detects the edge direction based on the color difference component CDv in the vertical direction and the color difference component CDh in the horizontal direction as described later. This ensures accurate and relatively high speed detection in the edge direction. Details of the edge direction map generation process will be described later.

After generation of the edge direction map, the CPU determines the hue value of the G component of each target pixel (i.e., R pixel or B pixel) other than the G pixel by interpolation (G component interpolation processing) (step S106). The G component interpolation process performs interpolation with the value of the G component of the neighboring pixel with reference to the edge direction map to determine an appropriate value of the G component of the non-G target pixel. Details of the G component interpolation processing will be described later.

The CPU simultaneously performs interpolation of the remaining color components of the non-G target pixels other than the G pixels (non-G pixel interpolation processing) (step S108). The G component of the non-G target pixel, which may be an R pixel or a B pixel, has already been determined by interpolation in the previous step S106. Thus, the process of step S108 interpolates the B component of the R target pixel while interpolating the R component for the B target pixel. Thus, pixels other than G pixels have all of the color components R, G, and B obtained. Details of the non-G pixel interpolation processing (interpolation of the remaining color components after interpolation of the G component for each non-G target pixel other than the G pixel) will be described later.

Upon completion of interpolation of pixels other than G pixels, the CPU interpolates the missing color components (R component and B component) of the original image data for each G target pixel (G pixel interpolation processing) (step S110). The G pixel interpolation process also performs interpolation with reference to the edge direction map to determine an appropriate value of the missing color component of the G target pixel. Details of this G pixel interpolation process will be described later.

Upon completion of the interpolation of missing color components of pixels other than G pixels (steps S106 and S108) and the interpolation of missing color components of G pixels (step S110), all pixels acquire all the obtained color components R, G, and B. Have Thereafter, the CPU outputs the obtained RGB color component as color image data generated from the original image data (step S112), and ends the color image data generation process of FIG.

As described above, the color image data generation process of this embodiment generates an edge direction map by detecting the edge direction based on the color difference component CDv in the vertical direction and the color difference component CDh in the horizontal direction. More specifically, the edge direction is divided into two different types (color difference component between G component and R component and color difference component between G component and B component) of vertical color difference component CDv and horizontal color difference component CDh. Is detected accordingly. This property enables the generation of edge direction maps with high precision. Application of highly precise edge direction maps allows for proper interpolation of missing color components to the original image data. Both the edge direction map generation process and the interpolation of missing color components are simple and can be executed at high speed. Therefore, the above series of processes of this embodiment enable the high speed generation of appropriate color image data from the original image data having a mosaic arrangement of R, G, and B color components. The following description sequentially describes edge direction map generation processing, G component interpolation processing (interpolation of G components of pixels other than G pixels), non-G pixel interpolation processing (interpolation of remaining color components of pixels other than G pixels), and It relates to the details of G pixel interpolation processing (interpolation of missing color components of G pixels).

C. Edge Direction Map Generation Processing

FIG. 7 is a flowchart showing details of the edge direction map generation process executed in step S104 of the color image data generation process of FIG. 3 by the image processing apparatus 30 of the present embodiment.

The edge direction map generation process first selects one target pixel as an object of edge direction detection (step S200), and calculates the change amount Vv of the color difference component in the vertical direction (step S202). The amount of change Vv in the vertical direction is naturally calculated from the color difference component CDv in the vertical direction.

Fig. 8 shows the calculation of the change amount Vv in the vertical direction from the color difference component CDv in the vertical direction. FIG. 8A shows a calculation formula and FIG. 8B conceptually shows the calculation of the change amount Vv in the vertical direction according to the calculation formula of FIG. 8A. The rectangle into which the hatch in Fig. 8B is inserted indicates the position of the target pixel. The calculation processing of the change amount Vv in the vertical direction first calculates the square of the difference between the color difference component CDv of the target pixel and the upper pixel above the target pixel. Similarly, the operation process calculates the square of the difference between the color difference component CDv of the target pixel and the lower pixel below the target pixel. The blank arrows shown between the target pixel and the top pixel with the hatch shown in FIG. 8 (b) and between the target pixel and the bottom pixel with the hatch indicate the calculation of the square of the difference between the chrominance components CDv of each pixel. .

After the calculation of the square of each difference between the chrominance components CDv of the target pixel and the upper and lower pixels, the operation process is similarly performed by the chrominance components CDv of the left pixel of the target pixel and the upper and lower portions of the left pixel. In addition to the square of each difference between the upper and lower pixels, the square of the color difference component CDv of the right pixel to the right of the target pixel and the difference between the upper and lower pixels of the upper and lower pixels of the right pixel is calculated. Fig. 8B shows six squares obtained from six pairs of pixels arranged in the vertical direction. The sum of these six squares is designated as the change amount Vv in the vertical direction with respect to the target pixel.

Referring again to the flowchart of Fig. 7, after the calculation of the change amount Vv in the vertical direction, the edge direction map generation process calculates the change amount Vh of the color difference components in the horizontal direction (step S204). The change amount Vh in the horizontal direction is calculated in the same manner as the calculation of the change amount Vv in the vertical direction except for the use of the color difference component CDh in the horizontal direction for calculation.

9 shows the calculation of the change amount Vh in the horizontal direction from the color difference component CDh in the horizontal direction. FIG. 9A shows a calculation formula and FIG. 9B conceptually shows the calculation of the change amount Vh in the horizontal direction according to the calculation formula of FIG. 9A. As shown by the empty arrows in Fig. 9B, the arithmetic processing is the square of each difference between the color difference component CDh of the target pixel and the left and right pixels of the left and right sides of the target pixel, the upper pixel above the target pixel. Square of each difference between the chrominance component CDh of the pixel and the left and right pixels of the left and right of the upper pixel, and the chrominance component CDh of the lower pixel below the target pixel and the left and right of the left and right of the lower pixel. The square of each difference between pixels is continuously calculated. The sum of these six squares is designated as the change amount Vh in the horizontal direction with respect to the target pixel.

Referring back to the flowchart of Fig. 7, the variation amount Vv in the vertical direction and the variation amount Vh in the horizontal direction calculated as described above are compared with respect to the target pixel (step S206). If the amount of change Vh in the horizontal direction is equal to or more than the amount of change Vv in the vertical direction (step S206; YES), it is determined that the edge is directed in the vertical direction through the target pixel. Then, the value "d" representing the edge direction is set equal to "1" (step S208). On the contrary, if the amount of change Vv in the vertical direction is equal to or more than the amount of change Vh in the horizontal direction (step S206; NO), it is determined that the edge faces the horizontal direction through the target pixel. Then, the value 'd' representing the edge direction is set equal to '3' (step S210). That is, pixels in the edge direction 'd' set to '1' have edges in the vertical direction, whereas pixels in the edge direction 'd' set to '3' have edges in the horizontal direction.

The square of the difference between the color difference components is calculated at the time of the determination of the change amount Vv in the vertical direction (see FIG. 8) and at the time of the determination of the change amount Vh in the horizontal direction (see FIG. 9). This calculation enables detection of the edge direction without taking into account the sign of the difference between the chrominance components. Thus, one possible variant may yield the absolute value of each difference between chrominance components instead of calculating their squares. The sum of the absolute values with respect to the neighboring pixels gives the change amount Vv in the vertical direction or the change amount Vh in the horizontal direction. However, squares enhance the difference between chrominance components compared to absolute values and thus increase the detection accuracy in the edge direction. It is to be understood that the present invention is not limited to absolute or squared differences and that other differences or similar means can be used instead to determine the edge direction.

Some care is required for comparison between the amount of change Vv in the vertical direction and the amount of change Vh in the horizontal direction. Therefore, there are some variations of the edge direction map generation process shown in FIG. 7 as described below.

10 shows a possible arrangement of neighboring pixels around a target pixel selected for detection of the edge direction. The neighboring pixels around the selected target pixel satisfy one of the four states shown in Figs. 10 (a) to 10 (d). In the state of Fig. 10A, the target pixel column of the target pixel and the upper and lower pixels in the vertical direction give each square of the two chrominance components CD _GR between the G component and the R component, The left and right pixel columns in the vertical direction to the left and the right of the target pixel column give each square of four chrominance components (CD _GB ) between the G and B components. Therefore, the amount of change Vv in the vertical direction is calculated as the sum of the squares of the two color difference components CD _GR and the squares of the four color difference components CD _GB . Similarly, the amount of change Vh in the horizontal direction is calculated as the sum of the squares of the two color difference components CD _GR and the squares of the four color difference components CD _GB . Thus, a simple comparison between the change amount Vv in the vertical direction and the change amount Vh in the horizontal direction enables proper detection of the edge direction of the target pixel.

In the state of FIG. 10 (b), each of the change amount Vv in the vertical direction and the change amount Vh in the horizontal direction is the square of the two color difference components CD _GR and the square of the four color difference components CD _GB . It is calculated as the sum. Thus, a simple comparison between the change amount Vv in the vertical direction and the change amount Vh in the horizontal direction enables proper detection of the edge direction of the target pixel.

However, this is not satisfied in the state of Fig. 10 (c) or the state of Fig. 10 (d). For example, in the state of FIG. 10C, the change amount Vv in the vertical direction is calculated as the sum of the squares of the two color difference components CD _GB and the squares of the four color difference components CD _GR . However, the horizontal change amount Vh is calculated as the sum of the squares of the two color difference components CD _GR and the squares of the four color difference components CD _GB . That is, the change amount Vv in the vertical direction cannot be simply compared with the change amount Vh in the horizontal direction. Even in the state of Fig. 10 (d), the amount of change Vv in the vertical direction cannot be easily compared with the amount of change Vh in the horizontal direction.

One possible means to prepare for this problem is to detect the edge direction only for pixels that meet the state of FIG. 10 (a) or the state of FIG. 10 (b). In this means, the edge direction map generation process of FIG. 7 selects the target pixel only among pixels other than G pixels, that is, R pixels and B pixels, in step S200. As previously described with respect to FIG. 2, the G pixels occupy half of the area of the Bayer color filter array. Detecting the edge direction only for R and B pixels halves the number of pixels to be processed to ensure fast generation of the edge direction map. In this case, the edge direction is not known for G pixels. However, application of an appropriate interpolation procedure prevents adverse effects of unknown edge direction of G pixels on the resulting picture quality.

The edge direction cannot be detected for G pixels, but can be detected according to the following procedure. The procedure of FIG. 11 applies twice the weight to the target pixel column or target pixel row containing the target pixel, compared to the weight applied to another pixel column or pixel row. Fig. 11A shows a calculation equation used for the determination of the change amount Vv in the vertical direction, and Fig. 11B shows the calculation equation used for the determination of the change amount Vh in the horizontal direction. According to the calculation formula, the change amount Vv in the vertical direction and the change amount Vh in the horizontal direction can be calculated with respect to all the pixels, that is, the G pixel, the R pixel, and the B pixel. Then, the edge direction can be determined for all these pixels by comparison between the change amount Vv in the vertical direction and the change amount Vh in the horizontal direction. In the equation of FIG. 11, the weight may be divided in half for a pixel column or pixel row without a target pixel.

Another method of calculation is the amount of change of two pixel columns and the amount of change of two pixel rows, that is, a target pixel column containing a target pixel and one neighboring pixel column adjacent to the target pixel column, and a target pixel row and a target including the target pixel. An edge direction of each target pixel may be detected based on one neighboring pixel row adjacent to the pixel row. FIG. 12A shows a calculation formula used to calculate a change amount Vv in the vertical direction with respect to the target pixel column including the target pixel in the vertical direction and the right pixel column in the vertical direction on the right side of the target pixel column. FIG. 12B conceptually shows the calculation of the change amount Vv in the vertical direction according to the calculation formula of FIG. 12A. As previously described with reference to FIG. 4, the alternately arranged pixel columns of the G and R pixels and the alternately arranged pixel columns of the G and B pixels are based on the Bayer color filter array. Alternately in the vertical direction. Therefore, the change amount Vv in the vertical direction is given as the sum of the squares of the two color difference components CD _GB and the square of the two color difference components CD _GR .

The amount of change Vh in the horizontal direction is similarly calculated for two pixel rows. 12 (c) shows a calculation formula used to calculate the change amount Vh in the horizontal direction, and FIG. 12 (d) conceptualizes the calculation of the change amount Vh in the horizontal direction according to the calculation formula in FIG. 12 (c). As shown. As previously described with respect to Fig. 4, the alternatingly arranged pixel rows of G and R pixels and the alternatingly arranged pixel rows of G and B pixels are obtained from the original image data based on the Bayer color filter array. It appears alternately in the horizontal direction. Therefore, as in the vertical change amount Vv according to the calculation formula of FIG. 12A, the horizontal change amount Vh according to the calculation formula of FIG. 12C is equal to the square of the two color difference components CD _GB . It is given as the sum of the squares of the two color difference components (CD _GR ). Therefore, the edge direction can be detected by comparison between the change amount Vv in the vertical direction and the change amount Vh in the horizontal direction.

Referring again to the flowchart of Fig. 7, as described above, the edge direction map generation process of this embodiment calculates the change amount Vv of the color difference components in the vertical direction and the change amount Vh of the color difference components in the horizontal direction ( Steps S202 and S204, the edge direction of the target pixel is detected based on the comparison between the change amount Vv in the vertical direction and the change amount Vh in the horizontal direction (steps S206 to S210). In the case of the calculation of the variation amounts Vv and Vh according to the methods shown in Figs. 8 and 9, the target pixel as the object of edge direction detection can be selected only from pixels other than G pixels, that is, R pixels and B pixels. (Step S200). On the other hand, in the case of the calculation of the variation amounts Vv and Vh according to one of the methods shown in Figs. 11 and 12, the target pixel can be selected from all the pixels (step S200). However, this is not limiting. Even in the case of the calculation of the variation amounts Vv and Vh according to one of the methods shown in FIGS. 11 and 12, the target pixel may be selected only from pixels other than G pixels, that is, R pixels and B pixels. After detection of the edge direction of the target pixel selected in step S200, the CPU determines whether the edge direction has been detected for all the pixels (step S212). If there are any unprocessed pixels (step S212: NO), the edge direction map generation process returns to step S200 to select another target pixel among the unprocessed pixels and execute a series of subsequent processes. Upon completion of detection of the edge direction with respect to all the pixels (step S212: YES), the edge direction map generation process in Fig. 7 is finished.

The edge direction map generation process of this embodiment enables detection of the edge direction included in the original image data with extremely high precision. Interpolation of missing color components of each pixel constituting the original image data is performed with reference to this highly precise edge direction map. This ensures proper interpolation without generating pseudo color. The reason for the generation of the highly precise edge direction map according to the edge direction map generation process of this embodiment is described below.

As shown in Figs. 2 and 4, the original image data based on the Bayer color filter array has data of certain color components R, G, and B only at separate pixel positions. The G component has data of one ratio (G pixels) per two pixels, and the R component and B components have data of one ratio (R pixels and B pixels) per four pixels. Due to the distinct appearance of this data, it is difficult to detect the edge with high precision from the amount of change in the hue value for each of the color components R, G, and B.

Thus, color difference components are used in place of the R, G, and B color components. For each pixel column or pixel row having an alternating arrangement of the G and R components, the row image data is converted into data of the color difference component between the G and R components. Regarding each of the pixel columns or pixel rows having an alternating arrangement of the G and B components, the row image data is converted into data of the color difference component between the G and B components. The lower part of Fig. 4 shows the conversion of the row image data into the vertical color difference component CDv and the horizontal color difference component CDh. Pixel columns and pixel rows having color difference components between the G and B components are shown as rectangles hatched in FIG. 4.

The amount of change in the chrominance component can be compared between neighboring pixel columns or neighboring pixel rows. For example, in an image region having a sudden change in brightness, there is a considerable amount of change in both the color difference component between the G component and the R component and the color difference component between the G component and the B component. Similarly, in an image region having a sudden change in color, there is also a considerable amount of change in either or both of the color difference component between the G component and the R component and / or the color difference component between the G component and the B component. Therefore, even the amount of change in the color difference component having different types can be compared. The edge can then be detected according to the chrominance component of the neighboring pixel column or the neighboring pixel row. This edge detection technique, based on the chrominance components of neighboring pixel columns or neighboring pixel rows, enables precise detection of even small edges.

The edge direction map generation process of this embodiment detects the edge direction with high precision based on the above consideration and thus gives a highly precise edge direction map. The specific procedure converts the original image data based on the Bayer color filter array into a vertical color difference component CDv and a horizontal color difference component CDh. As shown in the lower part of FIG. 4, the obtained data of the color difference component is obtained by the pixel column or pixel row of the color difference component (CD _GR ) between the G component and the R component, and the color difference component (CD _GB ) between the G component and the B component. It has an alternating array of pixel columns or pixel rows. Then, the change amount of the color difference component is calculated according to the formulas of FIGS. 8 and 9 or according to one of the formulas of FIGS. 11 and 12. The calculated amount of change is obtained from color difference components of neighboring pixel columns or neighboring pixel rows having different types. The amount of variation calculated up to the chrominance component having different types can be compared for edge detection as described above. This edge detection, based on the chrominance component of the neighboring pixel column or the neighboring pixel row, ensures highly precise detection even of small edges. Therefore, accurate detection of the edge direction of each pixel is made possible by the comparison between the change amount Vv in the vertical direction and the change amount Vh in the horizontal direction.

As is clearly understood from this description, in the edge direction map generation process of this embodiment, the amount of change Vv in the vertical direction and the amount of change Vh in the horizontal direction are calculated based on the color difference component of the adjacent pixel column or the adjacent pixel row. It is important to do. In other words, it is not so important what formula is used for the calculation of the change amount Vv in the vertical direction and the change amount Vh in the horizontal direction. Accordingly, various calculations may be applied instead of the calculations of FIGS. 8 and 9 or the calculations of FIGS. 11 and 12.

One modified procedure is to calculate the square of each difference between the color difference components of each pixel column or each pixel row of each pixel row as shown in FIG. 13 and sum the squares to determine the amount of change in the color difference component. have. Although FIG. 13 only shows calculation of the change amount Vh of a horizontal direction. The amount of change Vv in the vertical direction can be calculated in a similar manner. Another modified procedure calculates the square of each difference between the color difference components of each pixel column or each pixel row of each pixel row as well as neighboring pixels, as shown in FIG. Can be determined.

Another variation can determine the amount of change in the chrominance component by considering the difference between the chrominance components of the further distant pixels. For example, the modified procedure of FIG. 15 takes into account the difference between the color difference components of each pixel column or each pixel row of four pixels, as well as the difference between the color difference components of each pixel column or each pixel row of two pixels. The amount of change in the color difference component is determined. 14 and 15 only show the calculation of the change amount Vh in the horizontal direction, the change amount Vv in the vertical direction can be calculated in a similar manner.

D. G component interpolation

As described above with reference to the flowchart of Fig. 3, the color image data generation processing of this embodiment starts the G component interpolation processing after generation of the edge direction map (step S104), so that pixels other than G pixels (i.e., The G component of the R pixel and the B pixel is interpolated (step S106).

FIG. 16 is a flowchart showing the details of the G component interpolation process of interpolating G components of R pixels and B pixels. The G component interpolation process first specifies an edge direction 'd' of each interpolation target pixel, which is an object of interpolation, with reference to the generated edge direction map (step S300). As noted above, the edge direction map may have edge direction 'd' detected for all pixels or edge direction 'd' detected only for non-G pixels (ie, R and B pixels). For interpolation of the G component of a non-G pixel, the edge direction 'd' is always specified by referring to the edge direction map.

Thereafter, the G component interpolation process determines whether the specified edge direction 'd' is equal to '1' (step S302). If the specified edge direction 'd' is equal to '1' (step S302: YES), the chrominance components of the interpolation target pixel are interpolated with the chrominance components of the upper and lower pixels above and below the interpolation target pixel (step S304). On the other hand, if the specified edge direction 'd' is equivalent to '3' (step S302: NO), the color difference components of the interpolation target pixel are interpolated with the color difference components of the left and right pixels of the left and right sides of the interpolation target pixel (step S306). ). Then, the G component interpolation processing obtains original image data of the interpolation target pixel (step S308), and calculates the G component of the interpolation target pixel by adding the original image data to the interpolated chrominance component of the interpolation target pixel (step S310). ). This series of processes is described in more detail.

17 shows a processing flow for computing missing G components of non-G pixels. The interpolation target pixel is a non-G pixel, that is, an R pixel or a B pixel. When the interpolation target pixel is an R pixel, neighboring pixels are arranged as shown in Fig. 17A. On the other hand, when the interpolation target pixel is a B pixel, neighboring pixels are arranged as shown in Fig. 17B. Here, it is assumed that the interpolation target pixel, which is the interpolation object of the G component, is an R pixel and the specified edge direction 'd' is equivalent to '1'.

When the edge direction 'd' is equal to '1', the change amount Vh of the color difference component in the horizontal direction is more than the change amount Vv of the color difference component in the vertical direction. In general, interpolation at a smaller amount of change (closer to a stable state) is expected to have better interpolation results than interpolation at a larger amount of change. Therefore, a vertical color difference component having a smaller amount of change is used for the calculation of the G component. As clearly understood from Fig. 17A, in the R interpolation target pixel, both of the vertical color difference component and the horizontal color difference component are obtained by subtracting the R component from the G component. That is, the G component of the interpolation target pixel may be calculated from the color difference component in the vertical direction or the color difference component in the horizontal direction. For improved interpolation precision, the G component of the interpolation target pixel has a smaller amount of change in the vertical direction, as already indicated by the value '1' of the corresponding edge direction indicator 'd' of the interpolation target pixel. It is calculated from the components.

The color difference component in the vertical direction is obtained by subtracting the R component from the G component, and the original image data obtained from the interpolation target pixel is the R component. Therefore, addition of the original image data to the chrominance component immediately determines the G component of the interpolation target pixel. The color difference component used for calculating the G component may be a color difference component of an interpolation target pixel calculated according to the calculation equation of FIG. 5. Preferably, the color difference component used for calculating the G component may be a color difference component after noise is removed by the low pass filter shown in FIG. 6.

The chrominance component of the interpolation target pixel may be calculated from the chrominance component of neighboring pixels instead of being read directly. The modified procedure may read out the color difference components of the upper and lower pixels above and below the interpolation target pixel in the vertical direction and calculate the color difference components of the interpolation target pixel from the read color difference components. The G component interpolation processing of Fig. 16 calculates the color difference component of the interpolation target pixel from the color difference component of the neighboring pixel in step S304 or step S306. Another modified procedure may read the color difference component of the larger neighbor pixel of the interpolation target pixel to calculate the color difference component of the interpolation target pixel from the read color difference component.

After the calculation of the G component of all the interpolation target pixels as described above, the CPU ends the G component interpolation processing in FIG. 16 and returns to the color image data generation processing in FIG. Next, the color image data generation processing starts non-G pixel interpolation processing to interpolate the remaining color components of pixels other than the G pixels (step S108).

E. Non-G Pixel Interpolation Processing

18 is a flowchart showing details of non-G pixel interpolation processing. As previously described, after interpolation of the G component of a non-G pixel, the non-G pixel interpolation process interpolates the remaining color components of each non-G pixel, that is, the B component of the R pixel or the R component of the B pixel. do. The basic concept of non-G pixel interpolation processing is briefly described before processing details.

Non-G pixel interpolation processing interpolates the B component of an R pixel while interpolating the R component of a B pixel. In the case where the interpolation target pixel is an R pixel, all four pixels arranged diagonally adjacent to the interpolation target pixel are all B pixels as shown in Fig. 17A. The B component is obtained as original image data of these four neighboring pixels. On the other hand, when the interpolation target pixel is a B pixel, all four pixels arranged diagonally adjacent to the interpolation target pixel are all R pixels as shown in Fig. 17B. The R component is obtained as original image data of these four neighboring pixels. The simplest procedure is to perform interpolation with the original image data of four pixels arranged diagonally adjacent to the interpolation target pixel to determine the hue value of the remaining color components of the interpolation target pixel. For example, the R component of the interpolation target pixel can be obtained as an average of the four available R components of the diagonally surrounded pixel. Similarly, the B component of the interpolation target pixel can be obtained as the average of the four available circle B components of the diagonally surrounded pixel.

Direct interpolation of the remaining color components by the original image data of neighboring pixels can cause the generation of pseudo colors due to some deviation of the difference from other color components. In natural color images, RGB data is significantly correlated to both intra-channel and inter-channel sensing. This means that pixels in the local picture area usually have similar color difference characteristics. Therefore, it is preferable to interpolate the color difference component of the interpolation target pixel with the color difference component of the neighboring pixel, and to determine the hue value of the remaining color components of the interpolation target pixel based on the interpolated color difference component. Thus, instead of direct interpolation by the original image data of four pixels arranged diagonally adjacent to the interpolation target pixel, another available procedure calculates the color difference component of the interpolation target pixel from the color difference component of four neighboring pixels, and interpolates. The hue value of the remaining color components of the interpolation target pixel is determined based on the calculated chrominance components of the target pixel. For example, the interpolation object is the B component for the R interpolation target pixel in the state of FIG. 17 (a). A preferred procedure is to obtain the chrominance components between the G and B components of four B pixels arranged diagonally adjacent to the R interpolation target pixel, and use these four chrominance components to determine the corresponding chrominance components of the R interpolation target pixel. Calculate. For example, the color difference component of the interpolation target position may be calculated as an average of the available color difference components. The procedure then calculates the hue values of the remaining color components from the calculated color difference component and the interpolated G component of the R interpolation target pixel.

All pixels arranged diagonally adjacent to the R interpolation target pixel are B pixels. Both of the vertical color difference component CDv and the horizontal color difference component CDh computed with respect to these neighboring pixels according to the calculation formula of FIG. 5 are the color difference components between the G component and the B component. The corresponding color difference component of the interpolation target pixel may be calculated from a total of eight color difference components. One typical procedure calculates the average of each of the vertical color difference component CDv and the horizontal color difference component CDh of four pixels arranged diagonally adjacent to the interpolation target pixel as the representative color difference component of the pixel, The corresponding chrominance component of the interpolation target pixel can be determined from the representative chrominance component of four neighboring pixels. Another exemplary procedure may detect the edge direction of the interpolation target pixel and calculate the color difference component of the interpolation target pixel only from the chrominance component (obtained according to the calculation formula of FIG. 5) of the neighboring pixel that meets the detected edge direction. This method of determining the color difference component of the interpolation target pixel by considering both the vertical color difference component CDv and the horizontal color difference component CDh with respect to the neighboring pixels requires a complicated series of processing. On the other hand, the calculation based on the G component and the B component of four pixels arranged diagonally adjacent to the interpolation target pixel enables the determination of the higher speed of the color difference component (CD _GB ) of the interpolation target pixel.

Based on this basic concept, after interpolation of the G component, the non-G pixel interpolation process of the present embodiment shown in the flowchart of FIG. 18 interpolates the remaining color components of each non-G pixel. Specific procedures of the non-G pixel interpolation process are described with reference to the flowchart of FIG. 18.

The non-G pixel interpolation process first obtains the G component and the original image data of four pixels arranged diagonally adjacent to the interpolation target pixel (step S350). In the case where the interpolation target pixel is an R pixel, four neighboring pixels arranged diagonally are all B pixels as shown in Fig. 17A. Thus, original image data of these neighboring pixels is obtained with respect to the B component here. On the other hand, when the interpolation target pixel is a B pixel, four neighboring pixels arranged diagonally are all R pixels as shown in Fig. 17B. Thus, original image data of these neighboring pixels is obtained here with respect to the R component.

Then, the non-G pixel interpolation process calculates the color difference component of these four neighboring pixels by subtracting the G component from the original image data, and calculates the average of the four color difference components as the color difference component of the interpolation target pixel ( Step S352). In its application, the color difference component generally shows the subtraction of the R component or the B component from the G component as shown in FIG. However, the calculation of step S352 changes the sign of the color difference component by subtracting the G component from the original image data (R component or B component). This is a labor to change symbol in subsequent operations.

After the calculation of the color difference component of the interpolation target pixel, the non-G pixel interpolation process obtains the G component of the interpolation target pixel (step S534). Since the interpolation target pixel is a non-G pixel, the original image data of the interpolation target pixel is not related to the G component. However, the G component of the interpolation target pixel has already been determined by the G component interpolation process of FIG. 16 executed before this non-G pixel interpolation process.

The sum of the color difference component of the interpolation target pixel calculated in step S352 and the G component obtained in step S354 determines the remaining color components of the interpolation target pixel (step S356).

After the calculation of the remaining color components of all the non-G interpolation target pixels as described above, the CPU terminates the non-G pixel interpolation process in FIG. 18 and returns to the color image data generation process in FIG. Thereafter, the color image data generation processing starts the G pixel interpolation processing to interpolate missing color components of the G pixels (step S110).

F. G pixel interpolation

19 is a flowchart showing details of a G pixel interpolation process. The G pixel interpolation process first specifies the edge direction 'd' of each interpolation target pixel, which is an object of interpolation (step S400). The edge direction has already been detected and set in the edge direction map by the edge direction map generation process of FIG. As previously described, the edge direction can be detected for all pixels or only for non-G pixels alternately. If the edge direction is detected for all the pixels, the process of step S400 immediately specifies the edge direction of the interpolation target pixel by referring to the edge direction map. On the other hand, if the edge direction is detected only for non-G pixels, the process of step S400 estimates the edge direction of the interpolation target pixel from the detected edge direction of the neighboring pixel.

20 shows estimation of the edge direction of an interpolation target pixel from the edge direction of a neighboring pixel. The G pixel interpolation process interpolates the missing (ie R and B) color components of the G pixel. The original pixel of the original data surrounding the interpolation target pixel satisfies the arrangement of Fig. 20 (a) or the arrangement of Fig. 20 (b). In Figs. 20A and 20B, hatched rectangles represent interpolation target pixels. This estimation is performed under the assumption that the edge direction was detected only for non-G pixels. Thus, the edge direction was set to four neighboring pixels of the top, bottom, left and right of the interpolation target pixel.

20 (c) and 20 (d) show the arrangement of the edge direction 'd' of four neighboring pixels of the interpolation target pixel. In the arrangement of the edge direction 'd' of four neighboring pixels shown in Fig. 20 (c), the number of neighboring pixels having the edge direction 'd' set to '1' is the edge direction 'd' set to '3'. More than the number of neighboring pixels. Therefore, it is appropriate to estimate the edge direction 'd' of the interpolation target pixel as '1'. Meanwhile, in the arrangement of the edge direction 'd' of four neighboring pixels shown in FIG. 20 (d), the number of neighboring pixels having the edge direction 'd' set to '3' is the edge direction 'set to' 1 '. more than the number of neighboring pixels with d '. Therefore, it is appropriate to estimate the edge direction 'd' of the interpolation target pixel as '3'.

In this way, the estimation procedure counts the number of neighboring pixels with edge direction 'd' set to '1' and the number of neighboring pixels with edge direction 'd' set to '3' to count more counts. The edge direction 'd' is designated as the edge direction 'd' of the interpolation target pixel. Instead of counting the individual number of neighboring pixels, another estimation procedure may calculate the sum of the edge directions 'd' set in four neighboring pixels of the top, bottom, left and right of the interpolation target pixel. If the neighboring pixels having the edge direction 'd' set to '1' and the neighboring pixels having the edge direction 'd' set to '3' are the same number, the sum of the edge directions 'd' is equal to '8'. . If the sum of the edge directions 'd' is less than '8', it is expected that more neighboring pixels have the edge direction 'd' set to '1'. Therefore, the edge direction 'd' of the interpolation target pixel is designated as '1'. On the other hand, when the sum of the edge directions 'd' is larger than '8', it is expected that a larger number of neighboring pixels have the edge direction 'd' set to '3'. Therefore, the edge direction 'd' of the interpolation target pixel is designated as '3'. If the sum of the edge directions 'd' is equal to '8' (when the neighboring pixels with edge direction 'd' set to '1' and the neighboring pixels with edge direction 'd' set to '3' are the same number For example, the edge direction 'd' of the interpolation target pixel may be designated as '1' or '3'.

As described above, when the edge direction 'd' is set for all the pixels, the G pixel interpolation process of FIG. 19 simply reads the edge direction 'd' of the interpolation target pixel from the edge direction map in step S400. On the other hand, if the edge direction 'd' is set only for non-G pixels, the G pixel interpolation process reads the edge direction of the adjacent pixel adjacent to the interpolation target pixel from the edge direction map in step S400, and the Estimate the edge direction 'd'.

After the specification of the edge direction 'd' of the interpolation target pixel, the G pixel interpolation process determines whether the specified edge direction 'd' is equal to '1' (step S402). If the specified edge direction 'd' is equivalent to '1' (step S402: YES), it is determined that the edge is directed in the vertical direction through the interpolation target pixel. Thus, the G pixel interpolation process obtains the R component, G component, and B component of the upper and lower neighboring pixels above and below the interpolation target pixel. As shown in the arrangement of Figs. 20 (a) and 20 (b), the upper and lower neighboring pixels above and below the interpolation target pixel are always non-G pixels. The missing color component of the non-G pixel has already been computed by the G component interpolation process of FIG. 16 and the non-G pixel interpolation process of FIG. Thus, the R, G, and B components of these upper and lower neighboring pixels can be obtained immediately.

On the other hand, when the specified edge direction 'd' is not equal to '1' (step S402: NO), it is determined that the edge is directed in the horizontal direction through the interpolation target pixel. Thus, the G pixel interpolation process obtains the R component, G component, and B component of the left and right neighboring pixels on the left and right of the interpolation target pixel (step S406). Since the left and right neighboring pixels on the left and right of the interpolation target pixel are always non-G pixels, the R, G, and B components of these left and right neighboring pixels can be obtained immediately.

The G pixel interpolation process then calculates the color difference component of the neighboring pixel that is the object of acquisition of the R, G, and B components (step S408). When the edge direction 'd' of the interpolation target pixel is equal to '1' (step S402: YES), the G pixel interpolation process calculates the color difference components of the upper and lower neighboring pixels. On the other hand, when the edge direction 'd' of the interpolation target pixel is not equal to '1' (step S402: NO), the G pixel interpolation processing calculates the color difference components of the left and right neighboring pixels. In the non-G pixel interpolation process of Fig. 18, a symbol-changed color difference component is obtained in step S408 by subtracting the G component from the R component and subtracting the G component from the B component. This is an effort to change the symbol in the subsequent operation of the R and B components of the interpolation target pixel.

After calculating the color difference components of the upper and lower neighboring pixels or the left and right neighboring pixels (step S408), the G pixel interpolation process calculates the color difference components of the interpolation target pixels by performing interpolation with the calculated color difference components of the neighboring pixels (step S408). S410). As mentioned above, the calculated chrominance components of the upper and lower neighboring pixels or the left and right neighboring pixels subtract the G component from the chrominance component and the B component by subtracting the G component from the R component. Color difference component by one). Thus, the interpolated chrominance component is a preference changed chrominance component.

The G pixel interpolation process then obtains the G component of the interpolation target pixel (step S412). Since the interpolation target pixel is a G pixel, the original image data is directly related to the G component. Thereafter, the G pixel interpolation process calculates the R component and the B component of the interpolation target pixel from the interpolated color difference component of the interpolation target pixel and the obtained G component (step S414). Since the interpolated chrominance component is a symbol-changed (i.e., green component subtracted from the red component, and green component subtracted from the blue component) chrominance components, a simple addition of the G component to the interpolated chrominance component is equivalent to the R component of the interpolation target pixel. Determine the component B.

After the calculation of the missing color components (R component and B component) of all the G pixels as described above, the CPU ends the G pixel interpolation processing in FIG. 19 and returns to the color image data generation processing in FIG.

Upon completion of the G pixel interpolation process (step S110), the demosaicing process, that is, interpolation of missing color components, is terminated for all pixels. The CPU then outputs the R, G, and B components of each pixel as color image data generated from the original image data (step S112), and ends the color image data generation process of FIG.

The digital camera 100 of this embodiment executes demosaicing of the original image data to generate color image data in accordance with a series of processes described in detail below. The color image data generation process generates an edge direction map before the demosaicing process. By considering the amount of change in the color difference component CDv in the vertical direction and the color difference component CDh in the horizontal direction, two different types of color difference components (i.e., the color difference component between the G component and the R component) Color difference component between the G component and the B component). This ensures highly precise detection even of small sized edges and thus enables the generation of precise edge direction maps. Subsequent demosaicing processes refer to this precise edge direction map to properly interpolate the missing color component of each pixel.

Both the edge direction map generation process and the interpolation of missing color components are rather simple and can be executed at high speed. Therefore, the series of processes in this embodiment enables the high speed generation of appropriate color image data from the original image data having a mosaic arrangement of R, G, and B color components.

It is to be understood that the present invention may be embodied as an embodiment of a program of instructions stored in a computer readable medium executing the method of the present invention when executed by a computer. Examples of computer readable media include magnetic media such as hard disks, floppy disks, and magnetic tape; Optical media such as CD-ROMs and holographic devices; Magneto-optical media; And store, store, and execute code such as application specific integrated circuits (“ASICs”), programmable logic devices (“PLDs”), flash memory devices, ROMs, and RAM devices. Included but not limited to specially configured hardware devices.

The digital camera 100 of this embodiment generates color image data from original image data according to the procedure described above. The present invention is not limited to any of the above-described embodiments and variations thereof, and can be achieved in various applications and aspects without departing from the scope or spirit of the main features of the present invention.

Claims (10)

Receive mosaic image data having a setting of a single color component among the three color components R, G, and B corresponding to the three primary colors of light of each pixel, and perform a series of image processing on the received mosaic image data to display three of each pixel. An image processing apparatus for generating color image data having settings of all three color components R, G, and B, The mosaic image data includes a pixel column having an alternating arrangement of the pixels of the G component and the pixels of the R component, which are different color components in the vertical direction, and an alternating arrangement of the pixels of the B component, which are the pixels of the G component and other color components in the vertical direction. A pixel column having an alternating arrangement of pixels of the G component and pixels of the R component, which are other color components in the horizontal direction, and alternating arrays of pixels of the B component, which are pixels of the G component and other color components in the horizontal direction Represented by a combination of pixel rows, The image processing apparatus, In order to calculate the vertical color difference component between the G component and each other color component of each pixel included in each pixel column of the mosaic image, at the pixel position of the other color component using the available vertically neighboring components. A vertical color difference component calculating module for obtaining a G component or obtaining another color component at a G component pixel position; In order to calculate a horizontal color difference component between the G component of each pixel included in each pixel row of the mosaic image and another color component, horizontally neighboring color components are used to obtain a G component at a pixel position of another color component. A horizontal color difference component calculating module for obtaining another color component at the G component pixel position; From the mosaic image data, a pixel of the R component and a pixel of the B component are selected, and at each of the pixel positions, the variation of the vertical color difference component with respect to each of the selected pixels is equal to the variation of the horizontal color difference component. An edge direction detection module for comparing at least an edge direction of the selected pixel; And An image processing process comprising a missing color component interpolation module for interpolating a missing color component of each pixel of the mosaic image data with reference to the detected edge direction with a setting of one color component of each pixel of the mosaic image data Device. The method according to claim 1, wherein the edge direction detection module, A vertical value that calculates a change amount of the vertical color difference component of each target pixel, which is the detection object in the edge direction, from the vertical color difference component of the target pixel column including the target pixel and the vertical color difference component of the adjacent pixel column adjacent to the target pixel column. Direction change amount calculation module; And Calculating a change amount of a horizontal chrominance component of each target pixel that is a detection object in the edge direction from a horizontal chrominance component of a target pixel row including the target pixel and a horizontal chrominance component of a neighboring pixel row adjacent to the target pixel row An image processing apparatus having a horizontal direction variation amount calculation module. The image processing apparatus according to claim 1, wherein the edge direction detection module detects edge directions of only pixels of the R component and pixels of the B component included in the mosaic image data. The method according to claim 1, wherein the edge direction detection module detects the edge direction of each pixel in the horizontal direction when the change amount of the vertical color difference component of the pixel is larger than the change amount of the horizontal color difference component of the pixel, The edge direction detection module detects the edge direction of each pixel in the vertical direction when the change amount of the vertical color difference component of the pixel is smaller than the change amount of the horizontal color difference component of the pixel, And the edge direction detecting module detects the edge direction of each pixel in the vertical direction or the horizontal direction when the change amount of the vertical color difference component of the pixel is equal to the change amount of the horizontal color difference component of the pixel. The interpolation target pixel of claim 1, wherein the missing color component interpolation module comprises a G component, which is a missing color component of an interpolation target pixel, which is an object of interpolation, and a pair of upper and lower neighboring pixels above and below the interpolation target pixel and the interpolation target pixel. And interpolating with a color component of a pair of neighboring pixels selected between a pair of left and right neighboring pixels on the left side and the right side of. The method of claim 5, wherein the missing color component interpolation module, The vertical color difference component of the interpolation target pixel or the horizontal color difference component of the pair of upper and lower neighboring pixels of the upper and lower portions of the interpolation target pixel or the left and right vicinity of the left and right of the interpolation target pixel. A color difference component interpolation module for interpolating with a horizontal color difference component of a pair of pixels; And A G component that calculates a G component of the interpolation target pixel based on a setting of the interpolated vertical color difference component or the horizontal color difference component of the interpolation target pixel and the color component of the interpolation target pixel of the mosaic image data An image processing apparatus having a calculation module. The method according to claim 5, wherein the missing color component interpolation module interpolates the G component of all pixels without setting the G component of the mosaic data, and then interpolates the R and B components of the pixel with the interpolated G component and the mosaic. An image processing apparatus which interpolates with the setting of the color component of the pixel of the image data. The method of claim 1, wherein the edge direction detection module is configured to detect an edge direction of at least an R component pixel and a B component pixel included in the mosaic image data to generate an edge direction map, And the missing color component interpolation module interpolates missing color components of each pixel of the mosaic image data with reference to an edge direction stored in the edge direction map. Receive mosaic image data having a setting of a single color component among the three color components R, G, and B corresponding to the three primary colors of light of each pixel, and perform a series of image processing on the received mosaic image data to display three of each pixel. An image processing method for generating color image data having settings of all three color components R, G, and B, A pixel column having an alternating arrangement of pixels of the G component and an R component pixel which is another color component in the vertical direction, a pixel column having an alternating arrangement of pixels of the G component and a B component pixel which is another color component in the vertical direction, a G component A combination of pixels rows having an alternating arrangement of pixels of R and pixels of R components that are different color components in the horizontal direction, and pixels rows having an alternating arrangement of pixels of G components and pixels of B components, which are other color components in the horizontal direction. Receiving the image data to be expressed as the mosaic image data; Obtaining a color component different from the G component of each pixel included in each pixel column of the mosaic image data, and thereby generating a vertical color difference component between the G component and each other color component of each pixel included in the pixel column in the vertical direction. Calculating a; Obtaining a color component different from the G component of each pixel included in each pixel row of the mosaic image data, thereby generating a horizontal color difference component between the G component and each other color component of each pixel included in the pixel row in the horizontal direction; Calculating a; Selecting a pixel of the R component and a pixel of the B component from the mosaic image data, and comparing the amount of change in the vertical color difference component with respect to the amount of change in the horizontal color difference component at least for each of the selected pixels at each of the pixel positions, at least the Detecting an edge direction of the selected pixel; And And interpolating a missing color component of each pixel of the mosaic image data with the setting of one color component of each pixel of the mosaic image data with reference to the detected edge direction. Receive mosaic image data having a setting of a single color component among the three color components R, G, and B corresponding to the three primary colors of light of each pixel, and perform a series of image processing on the received mosaic image data to display three of each pixel. A computer readable medium having a program for causing a computer to realize a method of generating color image data having settings of all three color components R, G, and B, The program, A pixel column having an alternating arrangement of pixels of the G component and an R component pixel which is another color component in the vertical direction, a pixel column having an alternating arrangement of pixels of the G component and a B component pixel which is another color component in the vertical direction, a G component A combination of pixels rows having an alternating arrangement of pixels of R and pixels of R components that are different color components in the horizontal direction, and pixels rows having an alternating arrangement of pixels of G components and pixels of B components, which are other color components in the horizontal direction. A function of receiving represented image data as the mosaic image data; Obtaining a color component different from the G component of each pixel included in each pixel column of the mosaic image data, and thereby generating a vertical color difference component between the G component and each other color component of each pixel included in the pixel column in the vertical direction. A function of calculating; Obtaining a color component different from the G component of each pixel included in each pixel row of the mosaic image data, thereby generating a horizontal color difference component between the G component and each other color component of each pixel included in the pixel row in the horizontal direction; A function of calculating; Selecting a pixel of the R component and a pixel of the B component from the mosaic image data, and comparing the amount of change in the vertical color difference component with respect to the amount of change in the horizontal color difference component at least for each of the selected pixels at each of the pixel positions, at least the Detecting an edge direction of a selected pixel; And Referencing the detected edge direction, causing the computer to achieve functions of interpolating a missing color component of each pixel of the mosaic image data with the setting of one color component of each pixel of the mosaic image data Media available.