JP5348253B2 - Image processing apparatus and image processing method, imaging apparatus and imaging method, program, and recording medium - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To obtain an image in which noise is reduced even under a low-illuminance condition and blurs caused by chromatic aberration are reduced. <P>SOLUTION: When an X-axis direction is regarded as 0&deg; and attention is paid to pixels indicated as G in a color filter, the pixels indicated as G are disposed at every other pixel in any direction of 0&deg;, 45&deg;, 90&deg; and 135&deg;. All pixels adjacent to the pixels indicated as G in the direction of 0&deg; are pixels indicated as M. All pixels adjacent to the pixels indicated as G in the direction of 90&deg; are pixels indicated as W. All pixels adjacent to the pixels indicated as G in the direction of 45&deg; are pixels indicated as R or B. All pixels adjacent to the pixels indicated as G in the direction of 135&deg; are pixels indicated as R or B, being different from the pixels adjacent in the direction of 45&deg;. Demosaic processing is then carried out based on W-M=G. The present invention is applicable to an imaging apparatus, a camera signal processing unit, a demosaic processing unit, and a color filter. <P>COPYRIGHT: (C)2012,JPO&amp;INPIT

Description

  The present invention relates to an image processing device and an image processing method, an imaging device and an imaging method, a program, and a recording medium, and in particular, an image processing device and an image processing method capable of obtaining a blur-free image with canceled chromatic aberration, The present invention relates to an imaging device, an imaging method, a program, and a recording medium.

  In recent years, digital cameras have been widely used. In a digital camera, an image signal obtained by an image sensor (image sensor) such as a CCD or CMOS is digitized through a lens, and after appropriate image processing is performed, the obtained image signal is stored in a flash memory or the like. The information can be recorded on a recording medium or directly transferred to an information processing apparatus such as a personal computer by cable connection or infrared communication. In an information processing apparatus such as a personal computer, an image corresponding to the supplied image signal can be displayed on a monitor such as a CRT (Cathode Ray Tube) or a liquid crystal display.

  Conventional image sensors that have been widely used conventionally include three types of color filters of R (red), G (green), and B (blue). That is, each pixel of the image sensor can acquire only one of the wavelength components of R (red), G (green), or B (blue). That is, the image signal obtained by the image sensor includes a pixel group that can acquire the R (red) spectral component, a pixel group that can acquire the G (green) spectral component, and a pixel that can acquire the B (blue) spectral component. It consists of groups. At this time, the color arrangement used in the color filter is, for example, shown in FIG. 1 and is called a Bayer arrangement.

  When performing imaging processing using such an image sensor, if the subject is not very bright, that is, if the amount of light obtained from the subject is small, the output from the image sensor will be a small value, and the resulting image signal will be noise. Since the image was buried, the final image was also noisy.

  Therefore, conventionally, an image sensor using the color arrangement shown in FIG. 2, that is, a Bayer arrangement combining white (Y) as a luminance signal with red (R), green (G), and blue (B) is used. Thus, a signal processing technique for obtaining a high-resolution image is used (for example, see Patent Document 1). According to this technology, the color filter array in which the white pixels shown in FIG. 2 are arranged in a checkered pattern is applied, and on the assumption that all the pixels do not feel infrared light, a high resolution is obtained using the white pixels in the checkered arranged pixels. Has been made so that you can get.

  That is, according to the color filter arrangement shown in FIG. 2, since the Y pixels in the checkered array have sensitivity to almost the entire visible light, the green (G) pixels are checked as shown in FIG. 1 for the same subject. Compared with the arrangement, it is possible to obtain a stronger image signal. For this reason, by using the color filter shown in FIG. 2, the S / N ratio of the checkered signal that influences the resolution is compared with the case of using the green pixel checkered color filter shown in FIG. It becomes possible to improve.

Japanese Patent Laid-Open No. 4-88784

  The lens forms an image of incident light on the image sensor using the refraction effect of light. Since the refraction effect is used, a blur called chromatic aberration of the lens is generated depending on the wavelength of incident light. Usually, since the focus is adjusted in a state where the lens position is set so that the G (green) component is imaged, the G (green) component is an image captured in such a focused state. The image will be in focus, but the other components will be out of focus.

  The chromatic aberration will be described with reference to FIG.

  That is, conventionally, in an imaging apparatus having an image sensor such as a digital camera, the input light is refracted by the lens 1 and the lens 1 and the color filter are formed so as to form an image near the image sensor 2 having the color filter. An image sensor 2 is disposed.

  In general, the G (green) spectral component is imaged at the position of the image sensor 2. In this case, the spectral component of B (blue) having a wavelength shorter than that of G (green) is imaged on the lens side of the image sensor 2 as shown in FIG. 3A, and the spectral component of G (green) As shown in 3B, the image sensor 2 forms an image in focus, and R (red) spectral component having a wavelength longer than G (green), or the sum of R (red) and IR (infrared). 3C is imaged on the opposite side of the lens from the image sensor 2 as shown in FIG. 3C. Therefore, in such a state, components other than G (green) are not accurately imaged on the image sensor. That is, both B (blue), which has a shorter wavelength than G (green), and R (red) and IR (infrared), which have longer wavelengths than G (green), form a blurred picture on the image sensor.

  In order to solve this problem, a lens with little chromatic aberration may be used, but such a lens has a drawback that it becomes expensive.

  The present invention has been made in view of such a situation, and makes it possible to obtain an image with less noise and less blur due to chromatic aberration without using an expensive lens even under low illumination conditions. Is.

The image processing apparatus according to the first aspect of the present invention is an optical signal that is acquired by being incident on a predetermined color filter through a lens, and includes any one of a plurality of different spectral components for each pixel. In an image processing apparatus that receives input of an optical signal having a component and generates image data based on the optical signal for each pixel, the first pixel value corresponding to the optical signal in the predetermined pixel, Contrast component calculating means for calculating a contrast component of the image data based on a second pixel value obtained by an interpolation process executed using a pixel value of an adjacent pixel of a predetermined pixel, and the contrast component calculating means, said one of the first pixel value or the second pixel value, a pixel value corresponding to the first spectral component most frequency bandwidth is wide of the spectral components different, pre From the first spectral components, a pixel value corresponding to the second spectral components excluding the predetermined frequency band close to a frequency that causes no chromatic aberration of the lens, or a first weighting factor to said first spectral component Based on the arithmetic processing using the pixel value corresponding to the third spectral component represented by the linear sum of the multiplied value and the value obtained by multiplying the second spectral component by the second weighting factor, the image Find the contrast component of the data.

The contrast component of the image data calculated by the contrast component calculation means can include a pixel value corresponding to the third spectral component.

The third spectral component is a frequency component in a predetermined range corresponding to green.

  The contrast component calculation means includes a green component calculation means for calculating a pixel value of each pixel corresponding to green based on the first pixel value or the second pixel value, and a plurality of different spectral components. Among them, a red component calculation unit that calculates a pixel value of each pixel corresponding to red based on a pixel value corresponding to a predetermined frequency component corresponding to red and a calculation result by the green component calculation unit, and a plurality of red component calculation units Of the different spectral components, a blue component calculation that calculates a pixel value of each pixel corresponding to blue based on a pixel value corresponding to a predetermined frequency component corresponding to blue and a calculation result by the green component calculation means Means.

The image data may further include pattern direction estimation means for estimating the direction of the pattern near each pixel in the image data, and the contrast component calculation means includes the pattern direction estimation means. The contrast component of the image data can be calculated based on the estimation result of the pattern direction.

Pattern accuracy calculation for determining the accuracy of a pattern in each of the 0 degree direction, 45 degree direction, 90 degree direction, and 135 degree direction, assuming that one direction of the arrangement on the plane on which the pixels are arranged is 0 degree And a pattern direction estimating unit that estimates a pattern direction in the vicinity of each pixel in the image data based on a calculation result by the pattern accuracy calculating unit. be able to.

Based on the calculation result by the pattern accuracy calculation unit, the pattern direction estimation unit may determine whether the pattern direction near each pixel in the image data is close to 0 degrees or 90 degrees. Alternatively, it can be determined which of 45 degrees and 135 degrees is likely to be close.

  The first spectral component may include at least an infrared component.

Pixels corresponding to the third spectral component are in any of 0 degree direction, 45 degree direction, 90 degree direction, and 135 degree direction, with one direction on the plane in which the pixels are arranged as 0 degree. It is arranged every other pixel.

The plurality of different spectral components are five types of spectral components including the first spectral component, the second spectral component, and the third spectral component, and the first spectral component and the second spectral component The pixels corresponding to the spectral component and the fourth spectral component and the fifth spectral component are defined as 0 degree direction, 45 degree direction, where one direction of the arrangement on the plane where the pixels are arranged is 0 degree. It is arranged adjacent to the pixel corresponding to the third spectral component in either the 90-degree direction or the 135-degree direction, and corresponds to the first spectral component and the second spectral component. Are arranged in a direction orthogonal to the pixel corresponding to the third spectral component, and the pixels corresponding to the fourth spectral component and the fifth spectral component are the third spectral component. in a direction orthogonal to the corresponding pixel It is the column.

An image processing method according to a first aspect of the present invention receives an input of an optical signal having one of a plurality of different spectral components acquired by being incident on a predetermined color filter through a lens. In the image processing method of the image processing apparatus for generating image data based on the optical signal for each pixel, a first pixel value corresponding to the optical signal in the predetermined pixel is acquired, and the predetermined pixel Interpolation processing is performed using pixel values of adjacent pixels to obtain a second pixel value, and among the first pixel value and the second pixel value, the highest frequency among a plurality of different spectral components A pixel value corresponding to a first spectral component having a wide bandwidth, and a pixel corresponding to a second spectral component obtained by excluding a predetermined frequency band in the vicinity of a frequency at which no chromatic aberration occurs in the lens from the first spectral component Value, also , Corresponding to a third spectral component represented by a linear sum of a value obtained by multiplying the first spectral component by a first weighting factor and a value obtained by multiplying the second spectral component by a second weighting factor. A step of obtaining a contrast component of the image data by an arithmetic process using pixel values to be processed.

The program according to the first aspect of the present invention receives an input of an optical signal having one of a plurality of different spectral components obtained by being incident on a predetermined color filter via a lens, and a pixel A computer-executable program for controlling processing for generating image data based on each optical signal, and controlling acquisition of a first pixel value corresponding to the optical signal in a predetermined pixel, An interpolation process is performed using a pixel value of an adjacent pixel of the predetermined pixel to calculate a second pixel value, and a plurality of different spectral components among the first pixel value and the second pixel value are calculated. most pixel values bandwidth of the frequency corresponding to the wide first spectral component of the component, from the first spectral component, the second spectral excluding the predetermined frequency band close to a frequency that causes no chromatic aberration of the lens A pixel value corresponding to the minute or a linear sum of a value obtained by multiplying the first spectral component by a first weighting factor and a value obtained by multiplying the second spectral component by a second weighting factor. The computer is caused to execute a process including a step of obtaining a contrast component of the image data by an arithmetic process using a pixel value corresponding to the third spectral component.

In the first aspect of the present invention, a first pixel value corresponding to an optical signal in a predetermined pixel is acquired, an interpolation process is performed using a pixel value of an adjacent pixel of the predetermined pixel, and a second pixel value is obtained. A pixel value is acquired, and a pixel value corresponding to a first spectral component having the widest frequency bandwidth among a plurality of different spectral components among the first pixel value and the second pixel value, the first spectral component To a pixel value corresponding to the second spectral component excluding a predetermined frequency band in the vicinity of the frequency at which no chromatic aberration occurs in the lens, or a value obtained by multiplying the first spectral component by the first weighting factor, and the second The contrast component of the image data is obtained by arithmetic processing using the pixel value corresponding to the third spectral component represented by the linear sum of the spectral component multiplied by the second weighting coefficient.

The imaging device according to the second aspect of the present invention is an imaging device that captures an image, and the light incident through the lens is converted into light signals of a plurality of different spectral components by passing through a predetermined color filter for each pixel. An optical signal acquisition unit to acquire, a conversion unit to convert the optical signal acquired by the optical signal acquisition unit into a digital signal, and the digital signal converted by the conversion unit to process the predetermined signal in all pixels Image processing means for generating image data having pixel values corresponding to a plurality of color components, wherein the image processing means includes a first pixel value corresponding to the optical signal in the predetermined pixel, and Contrast for calculating the contrast component of the image data based on the second pixel value obtained by the interpolation processing executed using the pixel value of the adjacent pixel of the predetermined pixel Component calculating means, wherein the contrast component calculating means has a first spectral component having the widest frequency bandwidth among the plurality of different spectral components of the first pixel value or the second pixel value. Or a pixel value obtained by excluding a second spectral component corresponding to a predetermined frequency band in the vicinity of a frequency at which no chromatic aberration occurs in the lens from the first spectral component , or the first spectral component Using a pixel value corresponding to a third spectral component represented by a linear sum of a value obtained by multiplying the first weighting factor by a value obtained by multiplying the second spectral component by a second weighting factor Based on the processing, a contrast component of the image data is obtained.

The plurality of different spectral components are five types of spectral components including the first spectral component, the second spectral component, and the third spectral component, and the first spectral component and the second spectral component The pixels corresponding to the spectral component and the fourth spectral component and the fifth spectral component are defined as 0 degree direction, 45 degree direction, where one direction of the arrangement on the plane where the pixels are arranged is 0 degree. It is arranged adjacent to the pixel corresponding to the third spectral component in either the 90-degree direction or the 135-degree direction, and corresponds to the first spectral component and the second spectral component. Are arranged in a direction orthogonal to the pixel corresponding to the third spectral component, and the pixels corresponding to the fourth spectral component and the fifth spectral component are the third spectral component. in a direction orthogonal to the corresponding pixel It is the column.

An imaging method according to a second aspect of the present invention is an imaging method of an imaging apparatus that captures an image. Light that has entered through a lens is converted into optical signals having a plurality of different spectral components through a predetermined color filter. Acquiring for each pixel, converting the acquired optical signal into a digital signal, acquiring a first pixel value corresponding to the optical signal in the predetermined pixel from the converted digital signal, and Interpolation processing is performed using pixel values of pixels adjacent to the pixel to obtain a second pixel value, and the first pixel value and the second pixel value out of a plurality of different spectral components The pixel value corresponding to the first spectral component having the widest frequency bandwidth, and the second spectral component obtained by removing a predetermined frequency band in the vicinity of the frequency at which no chromatic aberration occurs in the lens from the first spectral component. Or a third sum represented by a linear sum of a value obtained by multiplying the first spectral component by a first weighting factor and a value obtained by multiplying the second spectral component by a second weighting factor. A step of obtaining a contrast component of the image data by a calculation process using a pixel value corresponding to the spectral component.

The program according to the second aspect of the present invention is a program that can be executed by a computer that controls a process of capturing an image, and is incident through a lens, and as a plurality of different spectral components through a predetermined color filter. Controlling acquisition of an optical signal obtained for each pixel, controlling conversion of the acquired optical signal into a digital signal, and corresponding to the optical signal in the predetermined pixel from the converted digital signal The second pixel value is calculated by performing an interpolation process using a pixel value of an adjacent pixel of the predetermined pixel, and obtaining the first pixel value and the second pixel value. Among the pixel values, the pixel value corresponding to the first spectral component having the widest frequency bandwidth among the plurality of different spectral components, and chromatic aberration does not occur in the lens from the first spectral component. A pixel value excluding a second spectral component corresponding to a predetermined frequency band in the vicinity of the frequency, or a value obtained by multiplying the first spectral component by a first weighting factor, and a second value added to the second spectral component. The computer is caused to execute a process including a step of obtaining a contrast component of the image data by an arithmetic process using a pixel value corresponding to the third spectral component represented by a linear sum with a value obtained by multiplying the weighting coefficient of.

In the second aspect of the present invention, an optical signal that is incident through a lens and obtained for each pixel as a plurality of different spectral components through a predetermined color filter is acquired, and the acquired optical signal is a digital signal. The first pixel value corresponding to the optical signal at the predetermined pixel is acquired from the converted digital signal, and the interpolation process is executed using the pixel value of the adjacent pixel of the predetermined pixel to obtain the second value. Of the first pixel value and the second pixel value, the pixel value corresponding to the first spectral component having the widest frequency bandwidth among the plurality of different spectral components, the first spectral value from a component, the second pixel value excluding the spectral components corresponding to a predetermined frequency band close to a frequency that causes no chromatic aberration in the lens, or, a value obtained by multiplying the first weighting factor to the first spectral component, the 2 The arithmetic processing using pixel values corresponding to the third spectral components represented by the linear sum of a value obtained by multiplying the second weighting factor to the spectral component, the contrast component of the image data is obtained.

  The imaging device may be an independent device or a block that performs imaging processing of the information processing device.

  According to the first aspect of the present invention, it is possible to obtain a contrast component, and in particular, it is possible to execute a demosaic process in which chromatic aberration is canceled using a pixel value having a spectral component in a wider frequency region than in the past.

  In addition, according to the second aspect of the present invention, an image can be captured, and in particular, in demosaic processing, image data in which chromatic aberration is canceled using pixel values having spectral components in a wider frequency region than in the past. Can be obtained.

It is a figure for demonstrating the conventional color filter arrangement | sequence. It is a figure for demonstrating the conventional color filter arrangement | sequence. It is a figure for demonstrating chromatic aberration. It is a block diagram which shows the structure of an imaging device. It is a figure for demonstrating the spectral component of a color filter. It is a figure for demonstrating the color arrangement | sequence of the color filter of FIG. It is a figure for demonstrating the spectral component of the color filter of FIG. It is a block diagram which shows the structure of the camera signal processing part of FIG. It is a figure for demonstrating the relationship between the direction of a pattern, and interpolation. It is a block diagram which shows the structure of the demosaic process part 44 of FIG. It is a flowchart explaining an imaging process. It is a flowchart explaining a mosaic process. It is a flowchart explaining a mosaic process. It is a flowchart explaining the pattern accuracy calculation process of a 0 degree direction. It is a figure for demonstrating the interpolation process of a 0 degree direction. It is a flowchart explaining the pattern accuracy calculation process of a 45 degree direction. It is a figure for demonstrating the interpolation process of a 45 degree direction. It is a flowchart explaining the pattern accuracy calculation process of a 90 degree direction. It is a figure for demonstrating the interpolation process of a 90 degree | times direction. It is a flowchart explaining the pattern accuracy calculation process of a 135 degree direction. It is a figure for demonstrating the interpolation process of a 135 degree | times direction. It is a flowchart explaining an angle interpolation appropriateness degree calculation process. It is a flowchart explaining an angle interpolation appropriateness degree calculation process. It is a flowchart explaining a 0 degree direction interpolation G component image generation process. It is a figure for demonstrating a principal component analysis. It is a flowchart explaining a 45 degree direction interpolation G component image generation process. It is a flowchart explaining 90 degree direction interpolation G component image generation processing. It is a figure for demonstrating a principal component analysis. It is a flowchart explaining 135 degree direction interpolation G component image generation processing. It is a flowchart explaining an R component pixel value calculation process. It is a figure for demonstrating a principal component analysis. It is a flowchart explaining a B component pixel value calculation process. It is a figure for demonstrating a principal component analysis. It is a figure for demonstrating the different color arrangement | sequence of the color filter of FIG. It is a block diagram which shows the structure of a personal computer.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  FIG. 4 is a block diagram illustrating a configuration of the imaging device 11 including the solid-state imaging device.

  Light incident through the optical lens 21 is applied to the color filter 22.

  The color filter 22 has an arrangement which will be described later with reference to FIG. 6, and transmits light having a predetermined frequency for each pixel to be incident on the solid-state image sensor 23.

  The solid-state image sensor 23 receives light that passes through the color filter 22 out of light incident through the optical lens 11, and converts the light energy obtained by receiving the light into an electric signal by photoelectric conversion for each pixel. An image signal that is photoelectrically converted by the solid-state image sensor 23 and output as an electrical signal is converted into a digital signal by the A / D converter 24 and then supplied to the camera signal processor 25.

  The camera signal processing unit 25 performs clipping processing, gamma correction, white balance correction, demosaic processing, and the like on the supplied digital signal, and supplies the digital signal to the display unit 26 or the image compression unit 27. Details of the demosaic process executed by the camera signal processing unit 25 will be described later.

  The display unit 26 includes, for example, a liquid crystal display element and a driver for driving the element, and an image corresponding to image data obtained by performing predetermined processing by the camera signal processing unit 25 as necessary. Is displayed.

  The image compression unit 27 performs a compression process on the supplied image signal, reduces the data amount of the image signal, converts the image signal into a predetermined recording image format, and outputs it to the recording unit 28 or the external output unit 29. The recording unit 28 records the converted image data on, for example, a storage element such as a hard disk or a semiconductor memory, or a removable recording medium attached to a predetermined driver. The external output unit 29 outputs the converted image data to another device by wired or wireless communication. Note that the image compression processing 27 does not necessarily have to be subjected to the recorded or externally output signal, but in recent years, the number of pixels of the image sensor has increased, and the device itself Since downsizing is required, it is preferable to perform image compression when image data is recorded or when it is externally output via a predetermined medium.

  The spectral characteristics of a general color filter and the color filter 12 of FIG. 4 will be described with reference to FIG.

  For example, in a general Bayer color filter as described with reference to FIG. 1, the filter corresponding to the B channel is a filter having a high transmittance of an optical signal having a wavelength of about 200 to 300 nm corresponding to blue. The filter corresponding to the G channel is a filter having a high transmittance of an optical signal having a wavelength of about 450 to 550 nm corresponding to green, and the filter corresponding to the R channel is a wavelength of about 550 to 650 nm corresponding to red. This is a filter having a high transmittance of the optical signal. These RGB-compatible filters have a property of hardly transmitting infrared light components having a wavelength of about 700 nm or more.

  On the other hand, the color filter 22 has five filters that acquire optical signals including spectral components in two wavelength bands in addition to RGB.

  The color filter 22 is a pixel group (hereinafter referred to as a first pixel) that can acquire an optical signal of a spectral component in a wider wavelength band (hereinafter referred to as a first spectral component) so as to cope with a case where the subject is not bright. For example, a filter for acquiring the entire visible light of RGB or for acquiring an optical signal including invisible light components such as visible light and infrared light is prepared. .

  Under low illumination conditions, a light source with a low color temperature and a large amount of infrared light is often used. In addition, if invisible light such as infrared light is used as auxiliary light, the atmosphere is hardly impaired. In view of the above, there is a demand for a technique for increasing effective photographing sensitivity under a light source containing a lot of invisible light such as infrared light.

  Therefore, it is preferable to acquire an optical signal including visible light and infrared light as the first spectral component. However, the entire visible light of RGB excluding infrared light may be used as the first spectral component. In addition, for example, a spectral component obtained by removing only a predetermined band from a wavelength band including visible light and infrared light, or a spectral component obtained by removing only a predetermined band from visible light may be used. If the wavelength region width included in one spectral component is wide, the imaging sensitivity can be improved accordingly.

  That is, even if the color filter 22 uses a filter that acquires the entire visible light as a filter that acquires the optical signal of the spectral component in the widest wavelength band, the SN ratio can be improved as compared with the conventional case. Is possible. However, even in this case, the sensitivity is sufficient if there is sufficient illuminance, but in the dark environment where sufficient illuminance cannot be obtained, low illuminance conditions where the light source contains a lot of infrared light, or low Under imaging conditions using infrared auxiliary light with illuminance, the sensitivity is still insufficient, and a high-quality image with reduced noise cannot be generated. On the other hand, a filter for acquiring an optical signal containing invisible light components such as visible light and infrared light as a filter for acquiring the optical signal of the first spectral component that is the spectral component in the widest wavelength band. In the case of darkness where sufficient illuminance cannot be obtained, in low illuminance conditions where the light source contains a lot of infrared light, or in shooting conditions that use low intensity illuminant auxiliary light However, it is possible to generate a high-quality image with reduced noise, which is preferable.

  For example, a filter for acquiring an optical signal containing invisible light components such as visible light and infrared light has a peak of about 530 nm as shown in FIG. It has the property of transmitting a signal and transmitting an infrared light component of 700 nm or more.

  In the following description, the color filter 22 is assumed to acquire an optical signal including invisible light components (hereinafter also referred to as W) such as visible light and infrared light as the first spectral component.

  Further, the color filter 22 can acquire a pixel group (hereinafter, referred to as a second spectral component) obtained by removing a wavelength component in a predetermined range from the first spectral component (hereinafter, referred to as a second pixel group). Can be obtained).

  The second spectral component may be a spectral component obtained by removing the wavelength component corresponding to any of RGB from the first spectral component, but there is no chromatic aberration in order to easily obtain an image that is out of focus. The spectral component obtained by removing the component from the first spectral component is preferably the second spectral component. Since the human eye is most sensitive to light with a wavelength near 555 nm, the lens is generally set so that chromatic aberration does not occur in the green component. It is preferable that the spectral component obtained by removing the wavelength component corresponding to G (green) from the component, that is, the spectral component corresponding to the magenta color component is the second spectral component.

  Of course, the spectral component obtained by removing the wavelength component corresponding to B (blue) from the first spectral component, that is, the yellow spectral component may be used as the second spectral component, or R (red) from the first spectral component. The spectral component excluding the wavelength component corresponding to, that is, the spectral component excluding the wavelength component corresponding to cyan may be used as the second spectral component.

  Hereinafter, the color filter 22 corresponds to a spectral component obtained by removing a wavelength component corresponding to G (green) from a wavelength region of an invisible light component such as visible light and infrared light as the second spectral component, that is, magenta. A description will be given assuming that an optical signal of a spectral component (hereinafter also referred to as M) is acquired.

  Thus, by using the color filter 22, the solid-state image sensor 23 can be used even when the subject is not bright, in addition to the pixel group for acquiring the RGB spectral components similar to the conventional color filter. From the first spectral component, a pixel group that can acquire a first spectral component that is a spectral component in a wavelength band wider than the wavelength band in each of RGB of the conventional color filter so that a sufficient amount of light enters 23. It becomes possible to acquire a pixel group that can acquire the second spectral component that is the spectral component excluding the wavelength component of the predetermined region.

  The pixel data of the second pixel group is subtracted from the pixel data of the first pixel group to the pixel data acquired in this way (in other words, weight W1 = 1, weight W2 = (− 1). ) To obtain the linear sum of the pixel values of the first pixel group and the second pixel group based on the weights W1 and W2, thereby obtaining pixel data of wavelength components in a predetermined region. When no chromatic aberration is generated in the pixel data of the wavelength component in the predetermined region, a clear image without blur can be obtained. In other words, the lens 21 is set so that there is no chromatic aberration in the wavelength of the predetermined region obtained by subtracting the pixel data of the second pixel group from the pixel data of the first pixel group, and this predetermined region By using the pixel data of the wavelength component, that is, the pixel data from which chromatic aberration is removed as the contrast component, pixel data without chromatic aberration can be created, and a clear image can be created with chromatic aberration cancelled. .

  An example of the color filter array in the color filter 22 and each pixel acquired in the solid-state image sensor 23 will be described with reference to FIG. FIG. 6 shows a filter arrangement of a part of the color filter 22.

  Each pixel of the solid-state image sensor 23 in which the spectral components that can be acquired by the color filters 22 are different is a pixel indicated by W, a pixel indicated by M, a pixel indicated by G, a pixel indicated by R, And it is classified into five types of pixels indicated by B.

  A pixel indicated by W (a pixel corresponding to the first pixel group that acquires the first spectral component described above) is a pixel that acquires a spectral component (the first spectral component described above) in a wide wavelength band. is there. Specifically, for example, the pixel indicated by W is a spectral component of B (blue), a spectral component of G (green), a spectral component of R (red), and a spectral component of IR (infrared). It is a pixel that can acquire all. In the color filter 22, normally, if nothing is processed on the front surface of the corresponding pixel of the solid-state image sensor 23, the solid-state image sensor 23 can obtain the first spectral component.

  A pixel indicated by M (a pixel corresponding to the second pixel group that acquires the second spectral component described above) is a wavelength component within a predetermined range (for example, G (green)) from the first spectral component. This is a pixel that acquires a spectral component (a second spectral component described above) excluding a corresponding predetermined range of wavelength components). Specifically, for example, the pixel can acquire a B (blue) spectral component, an R (red) spectral component, and an IR (infrared) spectral component. Note that, in the color filter 22, a pixel that can acquire the second spectral component in the solid-state image sensor 23 is usually obtained by adding a color filter that passes the magenta color to the front surface of the corresponding pixel of the solid-state image sensor 23.

  A pixel indicated by G is a pixel that can acquire a G (green) spectral component. In the color filter 22, a G (green) spectral component is usually acquired in the solid-state image sensor 23 by attaching a color filter that passes green and an infrared cut filter in front of the corresponding pixel of the solid-state image sensor 23. It becomes a pixel that can be.

  The pixel indicated by R is a pixel that can acquire a spectral component of R (red). Note that, in the color filter 22, an R (red) spectral component is usually acquired in the solid-state image sensor 23 by adding a color filter that passes red and an infrared cut filter in front of the corresponding pixel of the solid-state image sensor 23. It becomes a pixel that can be.

  The pixel indicated by B is a pixel that can acquire the B (blue) spectral component. In the color filter 22, a B (blue) spectral component is usually acquired in the solid-state image sensor 23 by attaching a color filter that passes blue and an infrared cut filter in front of the corresponding pixel of the solid-state image sensor 23. It becomes a pixel that can be.

  As shown in FIG. 6, for example, the color filter 22 uses a certain pixel indicated by W as a reference, that is, coordinates (0, 0) (2, 0) (0 , 2) (2, 2) has a pixel indicated by W, coordinates (1, 1) (1, 3) (3, 1) (3, 3) have pixels indicated by M, and coordinates A pixel indicated by G is arranged at (0,1) (0,3) (2,1) (2,3), and a pixel indicated by B is arranged at coordinates (1,0) (3,2). The filter arrangement is such that the minimum unit is a 4 × 4 matrix such that pixels indicated by R are arranged at coordinates (3, 0), (1, 2).

  That is, in the color filter 22, when the X axis direction is set to 0 degree and attention is paid to the pixel indicated by G, the pixel indicated by G is in the 0 degree direction, the 45 degree direction, the 90 degree direction, and the 135 degree direction. In any case, the pixels arranged every other pixel and adjacent to the pixel indicated by G in the 0 degree direction are all pixels indicated by M, and the pixels adjacent to the pixel indicated by G in the 90 degree direction are , All are pixels indicated by W, and pixels adjacent to the pixel indicated by G in the 45 degree direction are all pixels indicated by R or B, and the pixel indicated by G is in the 135 degree direction. The adjacent pixels are either one of R or B, which is different from the adjacent pixels in the 45 degree direction.

  Predetermined processing is performed by the camera signal processing unit 25 on the image data photographed using the color filter 22 and the solid-state imaging device 23, and a final output image can be obtained.

  In this color filter 22, even if the arrangement of the respective pixels of W and M is reversed, the demosaic process described later can be executed in the same manner, and even if the arrangement of the respective pixels of B and R is reversed. The demosaic process described later can be executed in the same manner.

  The five wavelength bands in the color filter 22 described with reference to FIG. 6 will be described with reference to FIG.

  In FIG. 7, the horizontal axis represents the wavelength. (A) in the figure shows the spectral component of B (blue), (b) in the figure shows the spectral component of G (green), and (c) in the figure shows R (red). The spectral component or the total spectral component of R (red) and IR (infrared) is shown.

  The wavelength component (that is, the first spectral component) acquired by the first pixel group indicated by W is the sum of the wavelength components (A), (B), and (C) in the figure. The wavelength component (that is, the second spectral component) that can be acquired by the second pixel group indicated by M is the sum of the wavelength components (a) and (c) in the figure.

  That is, when the optical signal acquired by the second pixel group indicated by M is subtracted from the optical signal acquired by the first pixel group indicated by W, there is no chromatic aberration, and G ( Green) spectral component can be obtained. By using the pixel data of the G (green) spectral component from which chromatic aberration has been removed as a contrast component, it is possible to create a sharp image with high contrast and with reduced chromatic aberration.

  As described above, the data input to the camera signal processing unit 25 is a pixel group corresponding to different spectral components indicated by W, M, G, B, and R output from the solid-state imaging device 23. That is, the pixels constituting the image data input to the camera signal processing unit 25 can be classified into pixel groups indicated by W, M, G, B, and R, and the pixel group indicated by W is arranged. In the pixel coordinates (X, Y), assuming that X is an even number and Y is an even number, the pixel coordinates (X and Y in which X is an odd number and Y is an odd number) In the coordinates (X, Y) of the pixel where the pixel group indicated by G is arranged, X is an even number, Y is an odd number, and the coordinates (X, Y) of the pixel where the pixel group indicated by B is arranged , X is an odd number, Y is an even number, and XY-1 is a multiple of 4. In the coordinates (X, Y) of the pixel where the pixel group indicated by R is arranged, X is an odd number, Y is Even numbers and XY-3 are multiples of 4.

  In other words, for example, when the pixel position of a certain pixel indicated by W is the reference point (0, 0), the pixels whose X and Y are even numbers in the data input to the camera signal processing unit 25. Corresponds to W, pixels where X and Y are both odd numbers correspond to M, pixels where X is even number and Y is odd number correspond to G, X is odd number, Y is even number, and X A pixel that is a multiple of 4 corresponds to B, -Y is an odd number, Y is an even number, and XY-3 is a pixel that is a multiple of 4.

  The image data output from the camera signal processing unit 25 is data composed of three components R (red), G (green), and B (blue) at all pixel positions, that is, image data after demosaicing. is there.

  FIG. 8 is a block diagram showing a more detailed configuration of the camera signal processing unit 25. The camera signal processing unit 25 includes a clipping processing unit 41, a gamma correction unit 42, a white balance correction unit 43, and a demosaic processing unit 44.

  The clipping processing unit 41 checks whether or not the pixel value of each pixel of the supplied image data is within the defined range, and if the pixel value is less than the lower limit value of the defined range, The value is corrected to the lower limit (clipping to the noise level), and if the pixel value exceeds the upper limit of the defined range, the pixel value is corrected to the upper limit (clipping to the saturation level).

  The gamma correction unit 42 performs gamma correction on each pixel value of each pixel of the input mosaic image.

  The white balance correction unit 43 corrects the white balance, that is, the difference in color depending on the light source at the time of shooting.

  The demosaic processing unit 44, based on the supplied mosaic image of the predetermined color filter array, when each pixel of the color filter array does not have any of RGB color components, An interpolation process, that is, a demosaic process is executed, and image data in which all pixels have RGB pixel values is generated.

  Here, it is described that the clipping processing unit 41, the gamma correction unit 42, the white balance correction unit 43, and the demosaic processing unit 44 are arranged in this order, and the image processing is executed in this order. Unit 41, demosaic processing unit 44, gamma correction unit 42, and white balance correction unit 43 are arranged in this order, and image processing may be executed in this order, and image processing is executed in this order. Even in the demosaic processing unit 44 described below, the same demosaic processing is executed.

  In the demosaic processing unit 44, attention is paid to each pixel constituting the image as a target pixel, and R, G, and B values in each pixel (coordinates (H, K), where H and K are integers) are used. However, only one of W, M, G, R, and B data exists in the pixel (H, K) of the input data. Therefore, the demosaic processing unit 44 performs an interpolation process for estimating a pixel value from pixels in the vicinity of the pixel of coordinates (H, K), and obtains R, G, and B values for each pixel.

  Depending on which direction the pixel value of the adjacent pixel is used for interpolation, that is, the pixel value obtained as a result of the interpolation process may differ depending on the direction of interpolation. A specific example will be described with reference to FIG.

  For example, as shown in FIG. 9, in an image having a black and white gradation pattern in the direction of a predetermined angle θ, it is assumed that the pixel of interest 101 is interpolated using neighboring pixels. Actually, interpolation is performed using adjacent pixels, but here, interpolation processing is performed using pixels located slightly apart so that the influence of the interpolation result due to the direction of the pixels used for interpolation appears extremely. A case will be described.

  Actually, it is a pixel in the vicinity of the target pixel with respect to the target pixel 101 of white or a color close to white, and is the pixel 102 and the pixel 103 having substantially the same color, or a pixel in the vicinity of the target pixel. Since the angle is 90 degrees with respect to the angle θ, a case will be considered in which interpolation processing is performed using either the pixel 104 or the pixel 105 having a dark color unlike the pixel of interest. Here, if the interpolation process is performed using the pixel 104 and the pixel 105, the pixel value of the pixel 101 becomes darker than the actual color. On the other hand, if interpolation processing is performed using the pixel 102 and the pixel 103, a correct result can be obtained.

  That is, the interpolation processing must be executed using neighboring pixels in an appropriate direction based on the pattern (including changes in color and brightness) in the vicinity of the target pixel.

  Therefore, in the demosaic processing unit 44, there is a possibility that the direction of the pattern in the vicinity of the pixel of interest is each of the 0 degree direction, the 45 degree direction, the 90 degree direction, and the 135 degree direction. A numerical value corresponding to the certainty (likeness) is obtained. The result is referred to as a 0 degree direction pattern accuracy, a 45 degree direction pattern accuracy, a 90 degree direction pattern accuracy, and a 135 degree direction pattern accuracy at each pixel position. Details of the pattern accuracy will be described later.

  The demosaic processing unit 44 then determines the appropriateness of the interpolation processing at each angle based on the pattern accuracy in the 0 degree direction, the pattern accuracy in the 45 degree direction, the pattern accuracy in the 90 degree direction, and the pattern accuracy in the 135 degree direction. Then, a numerical value indicating the possibility that the interpolation processing using the neighboring pixel values of the respective angles is appropriate is obtained.

  Here, the “pattern accuracy” at each angle is used as an index of the probability of determining whether the angle is correct without considering other angles. On the other hand, the “appropriateness” of the interpolation processing at each angle is used as an index of the probability of determining whether the angle is correct in consideration of other angles.

  Further, the demosaic processing unit 44 uses the interpolation pixel value of the G (green) component and the neighboring pixel value in the 45 degree direction obtained as a result of interpolation using the neighboring pixel value in the 0 degree direction with respect to the G (green) component. The interpolated pixel value of the G (green) component obtained as a result of interpolation, the interpolated pixel value of the G (green) component obtained as a result of interpolating using the neighboring pixel value in the 90 degree direction, and the neighboring pixel value in the 135 degree direction The interpolation pixel value of the G (green) component obtained as a result of interpolation is obtained, and weight addition is performed weighted with the appropriateness of interpolation at each angle, so that only the G (green) component considering the pattern direction is obtained. Image data can be obtained.

  Then, the demosaic processing unit 44, based on the image data of the G (green) component considering the direction of the pattern, the correlation coefficient of G (green) and R (red), G (green) and B (blue) The correlation coefficient is obtained, and the pixel value of each pixel of R (red) and B (blue) is obtained. Thereby, since the pixel values of R (red), G (green), and B (blue) at each pixel position are obtained, the demosaic processing unit 44 outputs this as a result image.

  FIG. 10 is a block diagram illustrating a detailed configuration example of the demosaic processing unit 44 of the camera signal processing unit 24 described with reference to FIG.

  The demosaic processing unit 44 includes a 0 ° direction pattern analysis processing unit 61, a 45 ° direction pattern analysis processing unit 62, a 90 ° direction pattern analysis processing unit 63, a 135 ° direction pattern analysis processing unit 64, and a pattern direction determination unit. 65, 0 degree direction interpolation G image calculation processing unit 71, 45 degree direction interpolation G image calculation processing unit 72, 90 degree direction interpolation G image calculation processing unit 73, 135 degree direction interpolation G image calculation processing unit 74, G image calculation processing Part 75, R image calculation processing part 81, and B image calculation processing part 82.

  The pattern analysis processing unit 61 in the 0-degree direction, the pattern analysis processing unit 62 in the 45-degree direction, the pattern analysis processing unit 63 in the 90-degree direction, and the pattern analysis processing unit 64 in the 135-degree direction are within a predetermined range from the target pixel. Based on the pixel value of the pixel, interpolation processing in each direction is performed according to the filter arrangement of the color filter 22. In the filter arrangement described with reference to FIG. 6, the X-axis direction is set to 0 degree, and in the 0 degree direction, one of adjacent G and M is interpolated on the other side, and in the 45 degree direction. Performs interpolation processing on one of adjacent W and M on the other side, and performs interpolation processing on the other side on either one of adjacent G and W in the direction of 90 degrees, and 135 degrees In the direction, interpolation processing is performed on one of adjacent W and M on the other side. In this way, in a predetermined range, in addition to the pixel values of the color components originally possessed by a plurality of pixels, the pixel values of the other color components are obtained by interpolation processing, and the pixel values of these two color components The pattern accuracy at each angle is obtained by executing the principal component analysis.

  Specifically, the pattern analysis processing unit 61 in the 0 degree direction pays attention to a pixel having a G component pixel value within a predetermined range from the target pixel, and the pixel in the 0 degree direction with respect to the X axis. Interpolation processing is performed using two adjacent M component pixel values, and dispersion of pixel values of a plurality of pairs of the G component pixel value and the M component interpolation pixel value is obtained to perform principal component analysis. Here, if the contribution ratio of the first principal component is high, it can be said that the pattern accuracy in the 0 degree direction is high. Note that the pattern analysis processing unit 61 in the 0 degree direction focuses on a pixel having a pixel value of an M component within a predetermined range from the target pixel, and is adjacent to the pixel in the 0 degree direction with respect to the X axis. Even if the principal component analysis is performed by performing the interpolation process using the pixel values of the two G components, it is possible to obtain the pattern accuracy in the 0 degree direction in the same manner.

  The pattern analysis processing unit 62 in the 45 degree direction focuses on a pixel having a pixel value of a W component within a predetermined range from the target pixel, and two M adjacent to the pixel in the 45 degree direction with respect to the X axis. Interpolation processing is performed using the pixel values of the components, and the principal component analysis is performed by obtaining the dispersion of the pixel values of a plurality of pairs of the W component pixel values and the M component interpolation pixel values. Here, if the contribution ratio of the first principal component is high, it can be said that the pattern accuracy in the 45-degree direction is high. Note that the pattern analysis processing unit 62 in the 45-degree direction focuses on a pixel having an M component pixel value within a predetermined range from the pixel of interest, and is adjacent to the pixel in the direction of 45 degrees with respect to the X axis. Even if the principal component analysis is performed by performing the interpolation processing using the pixel values of the two W components, it is possible to obtain the pattern accuracy in the 45 degree direction in the same manner.

  The pattern analysis processing unit 63 in the 90 degree direction focuses on a pixel having a G component pixel value within a predetermined range from the target pixel, and two W adjacent to the pixel in the 90 degree direction with respect to the X axis. Interpolation processing is performed using the pixel values of the components, and the principal component analysis is performed by obtaining the dispersion of the pixel values of a plurality of pairs of the G component pixel values and the W component interpolation pixel values. Here, if the contribution ratio of the first principal component is high, it can be said that the pattern accuracy in the 90-degree direction is high. The 90-degree direction pattern analysis processing unit 63 focuses on a pixel having a W component pixel value within a predetermined range from the target pixel, and is adjacent to the pixel in the 90-degree direction with respect to the X axis. Even if the principal component analysis is performed by performing the interpolation processing using the pixel values of the two G components, the pattern accuracy in the 90-degree direction can be obtained in the same manner.

  The pattern analysis processing unit 64 in the 135 degree direction focuses on a pixel having a W component pixel value within a predetermined range from the target pixel, and two M adjacent to the pixel in the 135 degree direction with respect to the X axis. Interpolation processing is performed using the pixel values of the components, and the principal component analysis is performed by obtaining the dispersion of the pixel values of a plurality of pairs of the W component pixel values and the M component interpolation pixel values. Here, if the contribution ratio of the first main component is high, it can be said that the pattern accuracy in the direction of 135 degrees is high. Note that the pattern analysis processing unit 64 in the 135 degree direction focuses on a pixel having an M component pixel value within a predetermined range from the target pixel, and is adjacent to the pixel in the 135 degree direction with respect to the X axis. Even if the principal component analysis is performed by performing the interpolation processing using the pixel values of the two W components, the pattern accuracy in the direction of 135 degrees can be obtained in the same manner.

  As described above, the “pattern accuracy” at each angle is used as an index of the probability of determining whether the angle is correct as the pattern direction without considering other angles. A detailed example of calculating each pattern accuracy will be described later with reference to FIGS.

  The pattern direction determination unit 65 is obtained by the pattern analysis processing unit 61 in the 0 degree direction, the pattern analysis processing unit 62 in the 45 degree direction, the pattern analysis processing unit 63 in the 90 degree direction, and the pattern analysis processing unit 64 in the 135 degree direction. Based on the obtained pattern accuracy at each angle, the appropriateness of 0 degree direction interpolation, the appropriateness of 45 degree direction interpolation, the appropriateness of 90 degree direction interpolation, and the appropriate degree of 135 degree direction interpolation are obtained.

  Here, as described above, the “appropriateness” of the interpolation processing at each angle is used as an index of the probability of determining whether the angle is correct in consideration of other angles. A detailed example of calculation of the appropriateness of interpolation in each direction will be described later with reference to the flowcharts of FIGS.

  The 0-degree direction interpolation G image calculation processing unit 71 generates image data of only the G (green) component by executing the interpolation process in the 0-degree direction for pixels that do not have the G component. Specifically, in the case where the color filter array described with reference to FIG. 6 is used, the 0 degree direction storage G image calculation processing unit 71 has the W pixel adjacent to the R and B pixels in the 0 degree direction. Since the G and M pixels are adjacent to each other in the 0 degree direction, the W component value can be estimated by adding the G and M pixel values to obtain the W component of all the pixels. . Then, by extracting a pixel having a G component in the input signal from pixels within a predetermined range near the target pixel, and performing a principal component analysis of the G component and the W component of the plurality of pixels. The G component of the target pixel is obtained. An example of a detailed calculation procedure of the processing will be described later with reference to FIGS.

  The 45 degree direction-interpolated G image calculation processing unit 72 generates the image data of only the G (green) component by executing the interpolation process in the 45 degree direction for the pixels that do not have the G component. Specifically, when the color filter array described with reference to FIG. 6 is used, the 45-degree direction-interpolated G image calculation processing unit 72 has R and B pixels adjacent to the G pixel in the 45-degree direction, Since the W and M pixels are adjacent to each other in the 45 degree direction, it is possible to estimate the G pixel value by subtracting the M pixel value from the W pixel value. G component is obtained. An example of a detailed calculation procedure of the processing will be described later with reference to FIG.

  The 90-degree direction interpolation G image calculation processing unit 73 generates image data of only the G (green) component by executing the interpolation process in the 90-degree direction for pixels that do not have the G component. Specifically, when the color filter array described with reference to FIG. 6 is used, the 90-degree direction interpolation G image calculation processing unit 73 has R and B pixels adjacent to M pixels in the 90-degree direction, Since the G and W pixels are adjacent to each other in the 90-degree direction, it is possible to estimate the M pixel value by subtracting the G pixel value from the W pixel value. Estimate the M component. Then, by extracting a pixel having a G component in the input signal from pixels within a predetermined range near the target pixel, and performing a principal component analysis of the G component and the M component of the plurality of pixels. The G component of the target pixel is obtained. An example of a detailed calculation procedure of the process will be described later with reference to FIGS.

  The 135 degree direction-interpolated G image calculation processing unit 74 generates image data of only the G (green) component by executing the interpolation process in the 135 degree direction for pixels that do not have the G component. Specifically, when the color filter array described with reference to FIG. 6 is used, the 135 degree direction-interpolated G image calculation processing unit 74 has the R and B pixels adjacent to the G pixel in the 135 degree direction. Since the W and M pixels are adjacent to each other in the direction of 135 degrees, it is possible to estimate the G pixel value by subtracting the M pixel value from the W pixel value. G component is obtained. An example of a detailed calculation procedure of the processing will be described later with reference to FIG.

  The G image calculation processing unit 75 includes a 0 degree direction interpolation G image calculation processing unit 71, a 45 degree direction interpolation G image calculation processing unit 72, a 90 degree direction interpolation G image calculation processing unit 73, and a 135 degree direction interpolation G image calculation processing unit. 74, the appropriateness of 0 degree direction interpolation and the appropriateness of 45 degree direction interpolation obtained by the process of the pattern direction determination unit 65 for the G component images generated by the interpolation processing in the respective directions. The G component image data in which the pattern direction is taken into consideration is obtained by weighting and adding the appropriateness of the degree, the appropriateness of 90 degree direction interpolation, and the appropriate degree of 135 degree direction interpolation.

  The R image calculation processing unit 81 performs principal component analysis based on the pixel value of the R component included in the imaging data and the G component image data obtained by the G image calculation processing unit 75 to obtain the R component Correlation with the G component is obtained, and based on this, image data of the R component is obtained. Details of the processing will be described later with reference to FIGS. 30 and 31.

  The B image calculation processing unit 82 performs principal component analysis based on the B component pixel value included in the imaging data and the G component image data obtained by the G image calculation processing unit 75 to obtain the B component A correlation with the G component is obtained, and based on this, image data of the B component is obtained. Details of the processing will be described later with reference to FIGS. 32 and 33.

  Next, imaging processing executed by the imaging device 11 will be described with reference to the flowchart of FIG.

  In step S1, the image sensor, that is, the color filter 22 and the solid-state imaging device 23 receive the light incident through the lens 21, and converts the obtained light energy into an electrical signal by photoelectric conversion for each pixel. The image data is acquired and supplied to the A / D converter 24. The color component of each pixel of the obtained image data is determined by the filter arrangement shown in FIG. The color filter 22 has the filter arrangement described with reference to FIG.

  In step S2, the A / D converter 24 converts the supplied analog image data into digital signal image data, that is, performs AD conversion.

  In step S <b> 3, the camera signal processing unit 25 performs clipping processing in the clipping processing unit 41, gamma correction in the gamma correction unit 42, and white balance correction in the white balance correction unit 43.

  In step S <b> 4, the camera signal processing unit 25 performs a demosaic process that will be described later using the flowcharts of FIGS. 12 and 13.

  In step S5, the camera signal processing unit 25 determines whether or not the processed image is displayed. If it is determined in step S5 that no image is displayed, the process proceeds to step S7.

  If it is determined in step S5 that an image is displayed, the camera signal processing unit 25 supplies the processed image data to the display unit 26 in step S6. The display unit 26 displays an image corresponding to the supplied image data.

  If it is determined in step S5 that no image is displayed, or after the processing in step S6 is completed, in step S7, the camera signal processing unit 25 determines whether or not the processed image is output externally. . If it is determined in step S7 that the image is not output externally, the process proceeds to step S9 described later.

  If it is determined in step S7 that the image is output to the outside, the camera signal processing unit 25 supplies the processed image data to the image compression unit 37 in step S8. The image compression unit 37 compresses the supplied image data as necessary, and supplies the compressed image data to the external output unit 29. The external output unit 29 externally outputs the supplied image data.

  If it is determined in step S7 that the image is not output externally, or after the processing in step S8 is completed, in step S9, the camera signal processing unit 25 determines whether or not the processed image is recorded. . If it is determined in step S9 that no image is recorded, the process ends.

  If it is determined in step S9 that the image is to be recorded, the camera signal processing unit 25 supplies the processed image data to the image compression unit 37 in step S10. The image compression unit 37 compresses the supplied image data as necessary, and supplies the compressed image data to the recording unit 28. The recording unit 28 records the supplied image data on, for example, a hard disk, a storage element such as a semiconductor memory, or a detachable recording medium attached to a predetermined driver, and the processing ends. .

  Through such processing, image data including pixels that acquire optical signals in a wide frequency band is captured, and various processing including demosaic processing is performed, and display, external output, or recording is performed as necessary. . In the demosaic process, which will be described later with reference to the flowcharts of FIGS. 12 and 13 in step S4, a contrast component is extracted without causing chromatic aberration.

  Next, the demosaic process executed in step S4 of FIG. 11 will be described with reference to the flowcharts of FIGS.

  In step S41, the demosaic processing unit 44 obtains image data in which each pixel has a pixel value of any component of R, G, B, W, and M based on the filter arrangement described with reference to FIG. get.

  In step S42, a pattern angle calculation process in the 0 degree direction, which will be described later using the flowchart of FIG. 14, is executed.

  That is, in step S42, the pattern analysis processing unit 61 in the 0 degree direction performs principal component analysis on the data interpolated in the 0 degree direction at the target pixel in the output image, and sets the first main pattern as the pattern accuracy in the 0 degree direction. Determine the contribution ratio of the component. This small contribution rate means that the pattern is not in the 0 degree direction, that is, there is a high possibility that the interpolation process using the adjacent pixels in the 0 degree direction is not appropriate. The large contribution ratio means that the pattern may change in the 90 degree direction, but at least the pattern does not change in the 0 degree direction, that is, the adjacent pixels in the 0 degree direction are used. This means that there is a high possibility that the interpolation processing performed is appropriate.

  In step S43, a pattern angle calculation process in the 45 degree direction, which will be described later using the flowchart of FIG. 16, is executed.

  That is, in step S43, the 45-degree direction pattern analysis processing unit 62 performs principal component analysis on the data interpolated in the 45-degree direction at the target pixel in the output image, and obtains the first main pattern accuracy as the 45-degree direction pattern accuracy. Determine the contribution ratio of the component. This small contribution rate means that the pattern is not in the 45 degree direction, that is, there is a high possibility that the interpolation process using the adjacent pixels in the 45 degree direction is not appropriate. And if this contribution ratio is large, the pattern may change in the direction of 135 degrees, but at least the pattern does not change in the direction of 45 degrees, that is, the adjacent pixels in the direction of 45 degrees are used. This means that there is a high possibility that the interpolation processing performed is appropriate.

  In step S44, a pattern angle calculation process in the 90-degree direction, which will be described later with reference to the flowchart of FIG. 18, is executed.

  That is, in step S44, the 90-degree direction pattern analysis processing unit 63 performs principal component analysis on the data interpolated in the 90-degree direction at the target pixel in the output image, and obtains the first main pattern as the pattern accuracy in the 90-degree direction. Determine the contribution ratio of the component. This small contribution rate means that the pattern is not in the 90-degree direction, that is, there is a high possibility that interpolation processing using adjacent pixels in the 90-degree direction is not appropriate. The large contribution ratio means that the pattern may change in the 0 degree direction, but at least the pattern does not change in the 90 degree direction, that is, the adjacent pixels in the 90 degree direction are used. This means that there is a high possibility that the interpolation processing performed is appropriate.

  In step S45, a pattern angle calculation process in the 135 degree direction, which will be described later with reference to the flowchart of FIG. 20, is executed.

  That is, in step S45, the pattern analysis processing unit 64 in the 135 degree direction performs principal component analysis on the data interpolated in the 135 degree direction at the target pixel in the output image, and obtains the first main pattern accuracy as the pattern accuracy in the 135 degree direction. Determine the contribution ratio of the component. This small contribution rate means that the pattern is not in the 135 degree direction, that is, there is a high possibility that the interpolation process using adjacent pixels in the 135 degree direction is not appropriate. And if this contribution ratio is large, the pattern may change in the 45 degree direction, but at least the pattern does not change in the 135 degree direction, that is, the neighboring pixels in the 135 degree direction are used. This means that there is a high possibility that the interpolation processing performed is appropriate.

  In step S46, angle interpolation appropriateness calculation processing, which will be described later with reference to the flowcharts of FIGS. 22 and 23, is executed.

  That is, in step S46, the pattern direction determination unit 65 determines the appropriateness of 0 degree direction interpolation for the target pixel based on the pattern accuracy in each of the four directions obtained by the processes in steps S42 to S45. The appropriateness of the degree direction interpolation, the appropriateness of the 90 degree direction interpolation, and the appropriate degree of the 135 degree direction interpolation are calculated.

  In step S47, 0 degree direction interpolation G component image generation processing, which will be described later with reference to the flowchart of FIG. 24, is executed.

  That is, in step S47, the 0 degree direction interpolation G image calculation processing unit 71 generates G (green) component image data obtained as a result of executing the 0 degree direction interpolation process.

  In step S48, a 45-degree direction-interpolated G component image generation process, which will be described later with reference to the flowchart of FIG. 26, is executed.

  That is, in step S48, the 45 degree direction interpolation G image calculation processing unit 72 generates G (green) component image data obtained as a result of executing the 45 degree direction interpolation process.

  In step S49, a 90-degree direction-interpolated G component image generation process, which will be described later with reference to the flowchart of FIG. 27, is executed.

  That is, in step S49, the 90-degree direction interpolation G image calculation processing unit 73 generates G (green) component image data obtained as a result of executing the 90-degree direction interpolation processing.

  In step S50, 135-degree direction-interpolated G component image generation processing, which will be described later using the flowchart of FIG. 29, is executed.

  That is, in step S50, the 135-degree direction-interpolated G image calculation processing unit 74 generates G (green) component image data obtained as a result of executing the 135-degree direction interpolation process.

  In step S51, the G image calculation processing unit 75 obtains the green component images obtained as a result of interpolation in the respective directions obtained by the processing in steps S47 to S50, and the respective angles obtained in step S46. The image data of the G (green) component in consideration of the pattern direction is obtained by performing weighted addition based on the correction appropriateness.

  Specifically, the G image calculation processing unit 75 sets the coordinates of the pixel of interest as (H, K), and at that pixel, the appropriateness of 0 degree direction interpolation × 0 degree direction interpolation G image pixel value + 45 degree direction interpolation Appropriate degree × 45 degree direction interpolation G image pixel value + 90 degree direction interpolation appropriate degree × 90 degree direction interpolation G image pixel value + 135 degree direction interpolation appropriate degree × 135 degree direction interpolation G image pixel value Is calculated as a pixel value at the coordinates (H, K) of the G (green) image in consideration of the direction of. Then, this calculation is executed for all the pixels, and G (green) component image data in consideration of the pattern direction is generated.

  The G (green) component image data generated here is W and M that receive light in a frequency band wider than the frequency band corresponding to any of the conventional R, G, and B color components. It is generated using the data. That is, the image data of the G (green) component generated here is a good image even in a dark subject as compared with the conventional image generated only with the respective color components of R, G, and B. Can be obtained. Further, since the G component without chromatic aberration is generated using the W and M data, the G component in which the blur due to chromatic aberration is suppressed despite the use of light in a wide frequency band. Image data can be obtained.

  In step S52, an R component pixel value calculation process, which will be described later using the flowchart of FIG. 30, is executed.

  That is, in step 52, the R image calculation processing unit 81 performs principal component analysis based on the R component pixel value included in the input signal and the G component image data obtained by the G image calculation processing unit 75. The correlation between the R component and the G component is obtained, and based on this, the image data of the R component is obtained.

  In step S53, a B component pixel value calculation process, which will be described later with reference to the flowchart of FIG. 32, is executed.

  That is, in step S 53, the B image calculation processing unit 82 performs principal component analysis based on the B component pixel values included in the input signal and the G component image data obtained by the G image calculation processing unit 75. The correlation between the B component and the G component is obtained, and the image data of the B component is obtained based on this.

  In step S54, the demosaic processing unit 44 calculates the G, R, and B pixels in each pixel calculated by the G image calculation processing unit 75, the R image calculation processing unit 81, and the B image calculation processing unit 82, respectively. The value is output as output image data, and the process returns to step S4 in FIG. 11 and proceeds to step S5.

  Through such processing, demosaic processing is executed, and image data in which RGB color components are aligned for each pixel can be obtained from a mosaic image composed of five RGBWM color components.

  Further, here, G image data having no chromatic aberration is obtained by using W and M pixel values having a wider frequency band than the respective RGB color components, that is, the amount of received light is larger than these color components. Since R and B image data are generated based on the generated image, a good image can be obtained even when the subject is dark, for example.

  In addition, since interpolation processing is performed in consideration of the direction of the pattern in the image, compared with the case where interpolation processing is performed using pixel values of neighboring pixels in all directions without considering the direction of the pattern at all. Better images can be obtained.

  Next, the pattern accuracy calculation process in the 0 degree direction executed in step S42 in FIG. 12 will be described with reference to the flowchart in FIG.

  In step S81, the pattern analysis processing unit 61 in the 0 degree direction selects one unprocessed target pixel. Here, for example, the pixel position of the target pixel is assumed to be coordinates (H, K).

  In step S <b> 82, the pattern analysis processing unit 61 in the 0 degree direction detects G pixels that exist within a predetermined range from the target pixel.

  For example, in the filter array described with reference to FIG. 6, there are pixels indicated by G at a plurality of coordinates (X, Y) where the X coordinate is an even number and the Y coordinate is an odd number. The pattern analysis processing unit 61 in the 0 degree direction detects the G pixel indicated by such coordinates. Here, the predetermined range is a range that is empirically or experimentally determined based on conditions such as the number of samples for principal component analysis and the strength of correlation due to separation from the pixel of interest. The predetermined range is indicated by coordinates (X, Y) that satisfy H-4 ≦ X ≦ H + 4 and K + 4 ≦ Y ≦ K + 4, for example, where the pixel position of the pixel of interest is the coordinates (H, K). It may be a range of 9 × 9 pixels or the like, or may be a size other than this, or a range of a predetermined number of pixels selected from pixels with a close distance between the pixels centered on the target pixel. There may be.

  In step S83, the pattern analysis processing unit 61 in the 0 degree direction interpolates all the detected G pixels (X, Y) with M adjacent in the 0 degree direction.

  That is, in the G pixel at the coordinates (X, Y) detected in step S82, as shown in FIG. 15, the coordinates (X-1, Y) and the coordinates (X + 1, Y) are adjacent to each other in the 0 degree direction. M pixels indicated by Y) are arranged. Therefore, by obtaining the average value of the M values of these two points, the M interpolated pixel values interpolated in the 0 degree direction at the coordinates (X, Y) can be obtained. That is, the pattern analysis processing unit 61 in the 0 degree direction obtains the value of M interpolated in the 0 degree direction at the coordinates (X, Y) where X is an even number and Y is an odd number, according to the expression (1). Can do.

M value interpolated in the 0-degree direction at coordinates (X, Y) where X is an even number and Y is an odd number = {(M value of coordinates (X-1, Y)) + (coordinates (X + 1, Y Y) M value)} ÷ 2
... (1)

  In step S84, the pattern analysis processing unit 61 in the 0-degree direction performs the G pixel value at the pixel position corresponding to G detected in the predetermined range in step S82 and the M interpolation obtained in step S83. Pixel values are plotted in a two-dimensional space of G and M, and principal component analysis is performed.

  That is, within the predetermined range, the input image at the position of the coordinates (X, Y) where the X coordinate is an even number and the Y coordinate is an odd number has a G pixel value. The interpolated pixel value of M in the 0 degree direction was obtained. Using this, a pair of G value and M value is established at a plurality of coordinates (X, Y) where the X coordinate is an even number and the Y coordinate is an odd number within a predetermined range. Therefore, the pattern analysis processing unit 61 in the 0 degree direction can perform principal component analysis by plotting the plurality of pairs in a two-dimensional space of G and M.

  In step S85, the pattern analysis processing unit 61 in the 0 degree direction obtains the contribution ratio of the first principal component in the principal component analysis in step S84 and sets the pattern accuracy in the 0 degree direction.

  The contribution ratio of the first principal component is obtained by (dispersion value of the first principal component) / (sum of variances of each variable), which is (dispersion value of the first principal component) / (scattering amount of the entire sample). ). That is, in the portion where the pattern does not change in the 0 degree direction, the pair that is correctly interpolated by interpolation in the 0 degree direction and plotted in the two-dimensional space of G and M should be on one straight line. In other words, when the pattern does not change in the 0 degree direction, the contribution ratio other than the first principal component is substantially 0 as a result of the principal component analysis. Therefore, the contribution ratio of the first principal component is synonymous with the pattern accuracy in the 0 degree direction at the target pixel. As a result of the principal component analysis, if the contribution ratio of the first principal component is small, it can be estimated that a correct interpolated image cannot be obtained by interpolation in the 0 degree direction because the pattern changes in the 0 degree direction.

  Here, for example, in a local region represented by H-4 ≦ X ≦ H + 4 and K + 4 ≦ Y ≦ K + 4, it is assumed that the values of G and M have a correlation. In general, the hue does not change at a position where the pattern changes. In other words, the G value and the M value are generally in a proportional relationship (the proportionality coefficient is positive). Therefore, when obtaining the first principal component, it may be obtained with a condition that it passes through the origin, or if the direction of the first principal component has a negative slope, the pattern accuracy value in the 0 degree direction. May be decreased at a predetermined rate.

  In step S86, the pattern analysis processing unit 61 in the 0 degree direction determines whether or not the pattern accuracy in the 0 degree direction for all pixels has been calculated. If it is determined in step S86 that the pattern accuracy in the 0 degree direction for all pixels has not been calculated, the process returns to step S81, and the subsequent processes are repeated. If it is determined in step S86 that the pattern accuracy in the 0 degree direction for all pixels has been calculated, the process returns to step S42 in FIG. 12 and proceeds to step S43.

  By such processing, the direction of the pattern near the target pixel is 0 degree direction using the pixel value of the pixel having the G color component and the pixel value of the pixel having the M color component in the input image data. In other words, the accuracy (likeness) that the pattern direction is 0 degree can be obtained.

  Next, the pattern accuracy calculation process in the 45 degree direction executed in step S43 in FIG. 12 will be described with reference to the flowchart in FIG.

  In step S111, the pattern analysis processing unit 62 in the 45 degree direction selects one unprocessed target pixel. Here, for example, the pixel position of the target pixel is assumed to be coordinates (H, K).

  In step S112, the pattern analysis processing unit 62 in the 45 degree direction detects W pixels that exist within a predetermined range from the target pixel.

  For example, in the filter array described with reference to FIG. 6, there are pixels indicated by W at a plurality of coordinates (X, Y) where X is an even number and Y is an even number. The 45-degree direction pattern analysis processing unit 62 detects W pixels indicated by such coordinates.

  In step S113, the 45-degree pattern analysis processing unit 62 interpolates all the detected W pixels (X, Y) with M adjacent in the 45-degree direction.

  That is, in the W pixel at the coordinate (X, Y) detected in step S112, as shown in FIG. 17, the coordinate (X-1, Y-1) and the coordinate ( M pixels indicated by X + 1, Y + 1) are arranged. Therefore, by obtaining the average value of the M values of these two points, the M interpolated pixel value interpolated in the 45 degree direction at the coordinates (X, Y) can be obtained. That is, the pattern analysis processing unit 62 in the 45-degree direction obtains the value of M interpolated in the 45-degree direction at a plurality of coordinates (X, Y) where X is an even number and Y is an even number, according to Expression (2). be able to.

M value interpolated in 45 degree direction of coordinates (X, Y) where X is an even number and Y is an even number = {(M value of coordinates (X-1, Y-1)) + (coordinates (X + 1 , Y + 1) of M)} ÷ 2
... (2)

  In step S114, the pattern analysis processing unit 62 in the 45-degree direction uses the pixel value of each W at the pixel position corresponding to W detected within the predetermined range in step S112, and the M interpolation obtained in step S113. The pixel values are plotted in a two-dimensional space of W and M, and a principal component analysis is performed.

  That is, within the predetermined range, an input image at a position of coordinates (X, Y) where the X coordinate is an even number and the Y coordinate is an even number has a W pixel value. The interpolated pixel value in the 45 degree direction of M was obtained. Using this, a pair of W value and M value is established at a plurality of coordinates (X, Y) where the X coordinate is an even number and the Y coordinate is an even number within a predetermined range. Therefore, the pattern analysis processing unit 62 in the 45 degree direction can perform principal component analysis by plotting the plurality of pairs in a two-dimensional space of W and M.

  In step S115, the pattern analysis processing unit 62 in the 45 degree direction obtains the contribution ratio of the first principal component in the principal component analysis in step S114 and sets the pattern accuracy in the 45 degree direction.

  The contribution ratio of the first principal component is obtained by (dispersion value of the first principal component) / (sum of variances of each variable), which is (dispersion value of the first principal component) / (scattering amount of the entire sample). ). That is, in a portion where the pattern does not change in the 45 degree direction, a pair that is correctly interpolated by interpolation in the 45 degree direction and plotted in the two-dimensional space of W and M should be on one straight line. In other words, when the pattern does not change in the 45 degree direction, the contribution ratio other than the first principal component is substantially 0 as a result of the principal component analysis. Therefore, the contribution ratio of the first principal component is synonymous with the pattern accuracy in the 45-degree direction at the target pixel. As a result of the principal component analysis, if the contribution ratio of the first principal component is small, the pattern changes in the 45 degree direction, so that it can be estimated that a correct interpolation image cannot be obtained by interpolation in the 45 degree direction.

  Here, for example, in a local region represented by H-4 ≦ X ≦ H + 4 and K + 4 ≦ Y ≦ K + 4, it is assumed that the values of W and M have a correlation. In general, the hue does not change at a position where the pattern changes. That is, the value of W and the value of M are generally in a proportional relationship (proportional coefficient is positive). Therefore, when obtaining the first principal component, it may be obtained with a condition that the first principal component passes through the origin, or if the direction of the first principal component has a negative slope, the pattern accuracy value in the 45 degree direction. May be decreased at a predetermined rate.

  In step S116, the 45-degree direction pattern analysis processing unit 62 determines whether or not the 45-degree pattern accuracy for all pixels has been calculated. If it is determined in step S116 that the pattern accuracy in the 45 degree direction for all pixels has not been calculated, the process returns to step S111, and the subsequent processes are repeated. If it is determined in step S116 that the pattern accuracy in the 45 degree direction for all pixels has been calculated, the process returns to step S43 in FIG. 12 and proceeds to step S44.

  By such processing, the direction of the pattern near the target pixel is 45 degrees using the pixel value of the pixel having the W color component and the pixel value of the pixel having the M color component in the input image data. In other words, the accuracy (likeness) of the pattern direction being 45 degrees can be obtained.

  Next, the pattern accuracy calculation process in the 90-degree direction, which is executed in step S44 of FIG. 12, will be described with reference to the flowchart of FIG.

  In step S141, the pattern analysis processing unit 63 in the 90-degree direction selects one unprocessed target pixel. Here, for example, the pixel position of the target pixel is assumed to be coordinates (H, K).

  In step S142, the pattern analysis processing unit 63 in the 90-degree direction detects G pixels existing within a predetermined range from the target pixel.

  As described above, in the filter array described with reference to FIG. 6, there are pixels indicated by G at a plurality of coordinates (X, Y) where the X coordinate is an even number and the Y coordinate is an odd number. The pattern analysis processing unit 63 in the 90-degree direction detects G pixels indicated by such coordinates.

  In step S143, the pattern analysis processing unit 63 in the 90-degree direction interpolates all the detected G pixels (X, Y) with W adjacent in the 90-degree direction.

  That is, in the G pixel at the coordinates (X, Y) detected in step S142, as shown in FIG. 19, the coordinates (X, Y + 1) and the coordinates (X, Y−) are adjacent to each other in the 90-degree direction. W pixels indicated by 1) are arranged. Therefore, by obtaining the average value of the W values of these two points, the interpolated pixel value of W interpolated in the 90-degree direction of the coordinates (X, Y) can be obtained. That is, the pattern analysis processing unit 63 in the 90-degree direction obtains the value of W interpolated in the 90-degree direction of the coordinates (X, Y) where X is an even number and Y is an odd number, using Expression (3). Can do.

The value of W interpolated in the 90-degree direction of coordinates (X, Y) where X is an even number and Y is an odd number = {(value of W of coordinates (X, Y + 1)) + (coordinates (X, Y−1)) W value)} ÷ 2
... (3)

  In step S144, the pattern analysis processing unit 63 in the direction of 90 degrees interpolates each G pixel value at the pixel position corresponding to G detected in the predetermined range in step S142 and W obtained in step S143. Pixel values are plotted in a two-dimensional space of G and W, and principal component analysis is performed.

  That is, within the predetermined range, an input image at a position of coordinates (X, Y) where the X coordinate is an even number and the Y coordinate is an odd number has a G pixel value, and in step S143, the corresponding position is The interpolated pixel value in the 90 degree direction of W was obtained. Using this, a pair of G value and W value is established at a plurality of coordinates (X, Y) where the X coordinate is an even number and the Y coordinate is an odd number within a predetermined range. Therefore, the pattern analysis processing unit 63 in the 90-degree direction can perform principal component analysis by plotting the plurality of pairs in a two-dimensional space of G and W.

  In step S145, the pattern analysis processing unit 63 in the 90 degree direction obtains the contribution ratio of the first principal component in the principal component analysis in step S144, and sets it as the pattern accuracy in the 90 degree direction.

  The contribution ratio of the first principal component is obtained by (dispersion value of the first principal component) / (sum of variances of each variable), which is (dispersion value of the first principal component) / (scattering amount of the entire sample). ). That is, in a portion where the pattern does not change in the 90-degree direction, a pair that is correctly interpolated by interpolation in the 90-degree direction and plotted in the two-dimensional space of G and W should be on one straight line. In other words, when the pattern does not change in the 90-degree direction, the contribution ratio other than the first principal component is substantially 0 as a result of the principal component analysis. Therefore, the contribution ratio of the first principal component is synonymous with the pattern accuracy in the 90-degree direction at the target pixel. As a result of the principal component analysis, if the contribution ratio of the first principal component is small, it can be estimated that a correct interpolated image cannot be obtained by interpolation in the 90 degree direction because the pattern changes in the 90 degree direction.

  Here, for example, in a local region represented by H-4 ≦ X ≦ H + 4 and K + 4 ≦ Y ≦ K + 4, it is assumed that the values of G and W have a correlation. In general, the hue does not change at a position where the pattern changes. In other words, the G value and the W value are generally in a proportional relationship (the proportionality coefficient is positive). Therefore, when obtaining the first principal component, it may be obtained with a condition that the first principal component passes through the origin, or if the direction of the first principal component has a negative slope, the pattern accuracy value in the 90 degree direction. May be decreased at a predetermined rate.

  In step S146, the pattern analysis processing unit 63 in the 90-degree direction determines whether or not the pattern accuracy in the 90-degree direction for all pixels has been calculated. If it is determined in step S146 that the pattern accuracy in the 90-degree direction for all pixels has not been calculated, the process returns to step S141, and the subsequent processes are repeated. If it is determined in step S146 that the pattern accuracy in the 90-degree direction for all pixels has been calculated, the process returns to step S44 in FIG. 12 and proceeds to step S45.

  By such processing, the direction of the pattern near the target pixel is 90 degrees using the pixel value of the pixel having the G color component and the pixel value of the pixel having the W color component in the input image data. In other words, the accuracy (likeness) of the pattern direction being 90 degrees can be obtained.

  Next, the pattern accuracy calculation process in the 135 degree direction executed in step S45 of FIG. 12 will be described with reference to the flowchart of FIG.

  In step S171, the pattern analysis processing unit 64 in the 135 degree direction selects one unprocessed target pixel. Here, for example, the pixel position of the target pixel is assumed to be coordinates (H, K).

  In step S172, the pattern analysis processing unit 64 in the 135-degree direction detects W pixels that exist within a predetermined range from the target pixel.

  For example, in the filter array described with reference to FIG. 6, there are pixels indicated by W at a plurality of coordinates (X, Y) where the X coordinate is an even number and the Y coordinate is an even number. The pattern analysis processing unit 64 in the 135 degree direction detects the W pixel indicated by such coordinates.

  In step S173, the 135 degree direction pattern analysis processing unit 64 interpolates all the detected W pixels (X, Y) with M adjacent in the 135 degree direction.

  That is, in the W pixel at the coordinate (X, Y) detected in step S172, as shown in FIG. 21, the coordinate (X-1, Y + 1) and the coordinate (X + 1, X) are adjacent to each other in the 135 degree direction. M pixels indicated by Y-1) are arranged. Therefore, by obtaining the average value of the M values of these two points, it is possible to obtain the M interpolated pixel values interpolated in the 135 degree direction of the coordinates (X, Y). That is, the pattern analysis processing unit 64 in the 135-degree direction obtains the value of M interpolated in the 135-degree direction at the coordinates (X, Y) where X is an even number and Y is an even number, using Expression (4). Can do.

M value interpolated in 135-degree direction at coordinates (X, Y) where X is an even number and Y is an even number = {(M value of coordinates (X-1, Y + 1)) + (coordinates (X + 1, Y−) 1) M value)} ÷ 2
... (4)

  In step S174, the pattern analysis processing unit 64 in the direction of 135 degrees uses the pixel value of each W at the pixel position corresponding to W detected within the predetermined range in step S172, and the M interpolation obtained in step S173. The pixel values are plotted in a two-dimensional space of W and M, and a principal component analysis is performed.

  That is, within the predetermined range, the input image at the position of the coordinates (X, Y) where the X coordinate is an even number and the Y coordinate is an even number has a pixel value of W, and in step S173, the corresponding position An interpolated pixel value of M in the 135 degree direction was obtained. Using this, a pair of W value and M value is established at a plurality of coordinates (X, Y) where the X coordinate is an even number and the Y coordinate is an even number within a predetermined range. Therefore, the pattern analysis processing unit 64 in the 135 degree direction can perform principal component analysis by plotting the plurality of pairs in a two-dimensional space of W and M.

  In step S175, the pattern analysis processing unit 64 in the 135 degree direction obtains the contribution ratio of the first principal component in the principal component analysis in step S174, and sets the pattern accuracy in the 135 degree direction.

  The contribution ratio of the first principal component is obtained by (dispersion value of the first principal component) / (sum of variances of each variable), which is (dispersion value of the first principal component) / (scattering amount of the entire sample). ). That is, in a portion where the pattern does not change in the 135 degree direction, the pair that is correctly interpolated by interpolating in the 135 degree direction and plotted in the two-dimensional space of W and M should be on one straight line. In other words, if the pattern does not change in the direction of 135 degrees, the contribution ratio other than the first principal component is substantially 0 as a result of the principal component analysis. Therefore, the contribution ratio of the first principal component is synonymous with the pattern accuracy in the direction of 135 degrees in the target pixel. If the contribution ratio of the first principal component is small as a result of the principal component analysis, it can be estimated that a correct interpolated image cannot be obtained by interpolation in the 135 degree direction because the pattern changes in the 135 degree direction.

  Here, for example, in a local region represented by H-4 ≦ X ≦ H + 4 and K + 4 ≦ Y ≦ K + 4, it is assumed that the values of W and M have a correlation. In general, the hue does not change at a position where the pattern changes. That is, the value of W and the value of M are generally in a proportional relationship (proportional coefficient is positive). Therefore, when obtaining the first principal component, it may be obtained with a condition that the first principal component passes through the origin, or if the direction of the first principal component has a negative slope, the pattern accuracy value in the 135 degree direction. May be decreased at a predetermined rate.

  In step S176, the pattern analysis processing unit 64 in the 135 degree direction determines whether or not the pattern accuracy in the 135 degree direction for all pixels has been calculated. If it is determined in step S176 that the pattern accuracy in the direction of 135 degrees for all pixels has not been calculated, the process returns to step S171, and the subsequent processes are repeated. If it is determined in step S176 that the pattern accuracy in the direction of 135 degrees for all the pixels has been calculated, the process returns to step S45 in FIG. 12 and proceeds to step S46.

  By such processing, the direction of the pattern near the target pixel is 135 degrees using the pixel value of the pixel having the W color component and the pixel value of the pixel having the M color component in the input image data. In other words, the accuracy (likeness) of the pattern direction being 135 degrees can be obtained.

  Next, the angle interpolation appropriateness calculation process executed in step S46 of FIG. 12 will be described with reference to the flowcharts of FIGS.

  Here, the pattern accuracy at each angle described with reference to FIGS. 14 to 21 is used as an index for determining whether the angle is correct as a pattern angle without considering other angles. . On the other hand, the term “appropriateness” is used as an index for determining whether the angle is correct as a pattern angle in consideration of other angles.

  In step S201, the pattern direction determination unit 65 selects one unprocessed target pixel.

  In step S202, the pattern direction determination unit 65 obtains a probability P0 of a pattern in the 0 degree direction between 0 degrees and 90 degrees based on the pattern accuracy in the 0 degree direction and the pattern accuracy in the 90 degree direction of the target pixel. . Specifically, the pattern direction determination unit 65 has a probability P0 = (pattern accuracy in the 0 degree direction) ÷ {(pattern accuracy in the 0 degree direction) + (0 degree direction between 0 degrees and 90 degrees) + ( The pattern accuracy in the 90-degree direction)} is obtained.

  In step S203, the pattern direction determination unit 65 obtains a probability P90 of a pattern in the 90 degree direction between 0 degree and 90 degrees based on the pattern accuracy in the 0 degree direction and the pattern accuracy in the 90 degree direction of the target pixel. . Specifically, the pattern direction determination unit 65 has a probability that there is a pattern in the 90 degree direction between 0 degrees and 90 degrees = (pattern accuracy in the 90 degree direction) / {(pattern accuracy in the 0 degree direction) + (90 Pattern accuracy in the direction of angle)}.

  In step S <b> 204, the pattern direction determination unit 65 obtains a probability P <b> 45 of a pattern in the 45 degree direction between 45 degrees and 135 degrees based on the pattern accuracy in the 45 degree direction and the pattern accuracy in the 135 degree direction of the target pixel. . Specifically, the pattern direction determination unit 65 has a probability P45 = (pattern accuracy in the 45 degree direction) ÷ {(pattern accuracy in the 45 degree direction) + (45 degree direction between 45 degrees and 135 degrees) + ( Pattern accuracy in the direction of 135 degrees)}.

  In step S <b> 205, the pattern direction determination unit 65 obtains a probability P <b> 135 having a pattern in the 135 degree direction out of 45 degrees and 135 degrees, based on the pattern accuracy in the 45 degree direction and the pattern accuracy in the 135 degree direction of the target pixel. . Specifically, the pattern direction determination unit 65 has a probability P135 = (pattern accuracy in the direction of 135 degrees) ÷ {(pattern accuracy in the direction of 45 degrees) + ( Pattern accuracy in the direction of 135 degrees)}.

  The accuracy P0 and the accuracy P90 obtained in steps S202 and S203 are values that do not take into account the pattern accuracy in the 45 degree direction and the pattern accuracy in the 135 degree direction, and the accuracy P45 and the accuracy P45 obtained in step S204 and step S205. The accuracy P135 is a value that does not consider the pattern accuracy in the 0 degree direction and the pattern accuracy in the 90 degree direction.

  In step S <b> 206, the pattern direction determination unit 65 uses the obtained accuracy P <b> 0 and accuracy P <b> 90 to obtain the accuracy Q <b> 090 that the pattern is in the direction of 0 degree or 90 degrees. Specifically, the pattern direction determination unit 65 calculates the accuracy Q090 that the pattern is in the direction of either 0 degree or 90 degrees by (absolute value of P0−P90) ÷ {(absolute value of P0−P90) + ( The absolute value of P45-P135)}.

  In step S207, the pattern direction determination unit 65 uses the obtained accuracy P45 and accuracy P135 to obtain the accuracy Q45135 that the pattern is in the direction of either 45 degrees or 135 degrees. Specifically, the pattern direction determination unit 65 sets the accuracy Q45135 that the pattern is in the direction of 45 degrees or 135 degrees by (absolute value of P45−P135) ÷ {(absolute value of P0−P90) + ( The absolute value of P45-P135)}.

  In step S208, the pattern direction determination unit 65 calculates the appropriateness of 0 degree interpolation based on the obtained accuracy P0 and the accuracy Q090. Specifically, the pattern direction determination unit 65 obtains the appropriateness of 0 degree direction interpolation = P0 × Q090.

  In step S209, the pattern direction determination unit 65 calculates the appropriateness of 90-degree interpolation based on the obtained accuracy P90 and accuracy Q090. Specifically, the pattern direction determination unit 65 obtains 90 degree direction interpolation appropriateness = P90 × Q090.

  In step S210, the pattern direction determination unit 65 calculates the appropriateness of 45-degree interpolation based on the obtained accuracy P45 and accuracy Q45135. Specifically, the pattern direction determination unit 65 obtains the appropriateness of 45 degree direction interpolation = P45 × Q45135.

  In step S211, the pattern direction determination unit 65 calculates the appropriateness of 135 degree interpolation based on the obtained accuracy P135 and accuracy Q45135. Specifically, the pattern direction determination unit 65 obtains the appropriateness of 135-degree direction interpolation = P135 × Q45135.

  In step S212, the pattern direction determination unit 65 determines whether or not the processing for all the pixels has been completed. If it is determined in step S212 that the processing has not been completed for all pixels, the processing returns to step S201, and the subsequent processing is repeated. If it is determined in step S212 that the process has not been completed for all pixels, the process returns to step S46 in FIG. 12 and proceeds to step S47.

  By such processing, first, the ratio of the appropriateness of the 0 degree direction interpolation and the appropriateness of the 90 degree direction interpolation is obtained from the pattern accuracy in the 0 degree direction and the pattern accuracy in the 90 degree direction. The ratio between the appropriateness of 45 degree direction interpolation and the appropriateness degree of 135 degree direction interpolation is obtained from the two of the pattern accuracy and the pattern accuracy in the 135 degree direction. Thus, it is estimated which is considered to be correct with respect to two orthogonal directions.

  Then, the ratio of the appropriateness of the 0 degree directional interpolation and the appropriateness of the 90 degree directional interpolation and the ratio of the appropriateness of 45 degree directional interpolation and the appropriateness of 135 degree directional interpolation is obtained. Finally, the appropriateness of 0 degree direction interpolation, the appropriateness of 45 degree direction interpolation, the appropriateness of 90 degree direction interpolation, and the appropriateness degree of 135 degree direction interpolation are obtained.

  It is difficult to determine a difference in direction shifted by 45 degrees, but it is relatively easy to determine a difference in direction shifted by 90 degrees. That is, obtaining the appropriateness of interpolation in the 0 degree direction and the appropriateness of interpolation in the 45 degree direction from the “ratio between the appropriateness of 0 degree direction interpolation and the 45 degree direction interpolation” means that the direction is 45 It is difficult because it is a slight difference of degrees. On the other hand, from the ratio of the appropriateness of 0 degree direction interpolation and the appropriate degree of 90 degree direction interpolation, the appropriateness of interpolation in the 0 degree direction and the 90 degree direction It is easy to calculate the appropriateness of interpolation because the angles are greatly different. Therefore, according to such a procedure, it is possible to obtain the appropriateness of interpolation for each angle without performing complicated calculation processing.

  As a specific example, when there is a pattern in the 0 degree direction, what kind of appropriateness is required for each angle will be described.

  First, since the pattern is in the 0 degree direction, the “pattern accuracy in the 0 degree direction” is large, and the “pattern accuracy in the 90 degree direction” orthogonal thereto is small. Therefore, the accuracy P0 and the accuracy P90 obtained in steps S201 and S202 are P0≈1 and P90≈0. Further, since the 45 degree direction and the 135 degree direction are respectively shifted by the same angle (45 degrees) with respect to the actual pattern direction, "45 degree direction pattern accuracy" ≒ "135 degree direction Pattern accuracy ". Therefore, the accuracy P45 and the accuracy P135 obtained in step S204 and step S205 are P45≈P135.

  Accordingly, the accuracy Q090≈1 / (1 + 0) ≈1 is obtained in step S206, and the accuracy Q45135≈0 ÷ (1 + 0) ≈0 is obtained in step S207. Then, in step S208, “appropriateness of 0 degree direction interpolation” ≈1 × 1≈1, and in step S209, “appropriate degree of 90 degree direction interpolation” ≈0 is obtained. In step S210, “appropriateness of 45 degree direction interpolation” ≈0, and in step S211, “appropriate degree of 135 degree direction interpolation” ≈0. Therefore, a demosaic process is performed by interpolation in the 0 degree direction, and a good image can be obtained.

  Further, for example, for a portion where a subject without a pattern is projected, all values of the pattern accuracy in the 0 degree direction, the pattern accuracy in the 45 degree direction, the pattern accuracy in the 90 degree direction, and the pattern accuracy in the 135 degree direction are comparable. As a result, the appropriateness of 0 degree direction interpolation ≒ 0.25, the appropriate degree of 45 degree direction interpolation ≒ 0.25, the appropriate degree of 90 degree direction interpolation ≒ 0.25, the appropriate degree of 135 degree direction interpolation ≈0.25, and the interpolation processing is executed with the same weight in all directions.

  Next, the 0 degree direction interpolation G component image generation process executed in step S47 of FIG. 13 will be described with reference to the flowchart of FIG.

  In the filter array of the color filter 22 described with reference to FIG. 6, every other line having an odd Y-axis value has G values in the input image, and therefore is indicated by M. In the pixel, the G interpolation pixel value can be calculated by interpolation in the 0 degree direction. However, since the G value does not exist in the input image for the line having an even Y-axis value, the G interpolation pixel value cannot be directly calculated by interpolation in the 0 degree direction. Therefore, first, an image of W in all pixels is calculated by interpolation in the 0 degree direction, and then G in all pixels is calculated by correlation between G and W.

  First, in step S241, the 0 degree direction interpolation G image calculation processing unit 71 selects one target pixel that has not been subjected to W value extraction or interpolation processing.

  In step S242, the 0 degree direction interpolation G image calculation processing unit 71 determines whether or not the target pixel is a pixel having a W pixel value. A coordinate (X, Y) where X is an even number and Y is an even number originally has a pixel indicated by W. If it is determined in step S242 that the pixel of interest is a pixel having a pixel value of W, this value may be set to the value of W at the coordinates (X, Y), and the process will be described later in step S251. Proceed to

  If it is determined in step S242 that the target pixel is not a pixel having a pixel value of W, in step S243, the 0 degree direction interpolation G image calculation processing unit 71 is adjacent to the target pixel in the 0 degree direction. It is determined whether there is a W pixel as a pixel. For pixels with coordinates (X, Y) where X is an odd number and Y is an even number, pixels with coordinates (X-1, Y) and coordinates (X + 1, Y) adjacent in the 0 degree direction are It has a pixel value. If it is determined in step S243 that there is no W pixel in the 0 degree direction with respect to the target pixel, the process proceeds to step S245 described later.

  If it is determined in step S243 that there is a W pixel in the 0 degree direction with respect to the target pixel, in step S244, the 0 degree direction interpolation G image calculation processing unit 71 determines the W of the W adjacent to the 0 degree direction. Based on the pixel value, an interpolation value of W is obtained. That is, the 0-degree direction interpolation G image calculation processing unit 71 calculates the average of the pixel value at the coordinates (X−1, Y) and the pixel value at the coordinates (X + 1, Y) and interpolates W at the coordinates (X, Y). The pixel value is used, and the process proceeds to step S251 to be described later.

  If it is determined in step S243 that there is no W pixel in the 0 degree direction with respect to the target pixel, in step S245, the 0 degree direction interpolation G image calculation processing unit 71 sets the G pixel value to the target pixel. It is determined whether or not the pixel is a pixel having the same. If it is determined in step S245 that the target pixel is not a pixel having the G pixel value, the process proceeds to step S248 described later.

  When it is determined in step S245 that the target pixel is a pixel having a G pixel value, in step S246, the 0-degree direction interpolation G image calculation processing unit 71 calculates the M pixel value adjacent in the 0-degree direction. Based on this, an M interpolation value of the target pixel is obtained. That is, there is a pixel indicated by G at coordinates (X, Y) where X is an even number and Y is an odd number. And there are pixels indicated by M at the position of the pixel adjacent to the pixel in the 0 degree direction, that is, the coordinates (X-1, Y) and the coordinates (X + 1, Y). The average value can be M interpolated pixel values at coordinates (X, Y).

  In step S247, the 0-degree direction interpolation G image calculation processing unit 71 calculates the G pixel value of the target pixel and the M interpolation value based on the fact that W pixel value = G pixel value + M pixel value. The pixel value of W is obtained by addition, and the process proceeds to step S251 described later.

  If it is determined in step S245 that the target pixel is not a pixel having a G pixel value, in step S248, the 0-degree direction interpolation G image calculation processing unit 71 determines that the target pixel is a pixel having an M pixel value. It is determined whether or not. If it is determined in step S248 that the target pixel is not a pixel having an M pixel value, the process proceeds to step S251 to be described later.

  If it is determined in step S248 that the target pixel is a pixel having an M pixel value, in step S249, the 0-degree direction interpolation G image calculation processing unit 71 is adjacent to the target pixel in the 0-degree direction. Based on the G pixel value of the pixel, the G interpolation value at the target pixel is obtained. That is, there is a pixel indicated by M at coordinates (X, Y) where X is an odd number and Y is an odd number. The pixel adjacent to the pixel in the 0 degree direction, that is, the pixel indicated by G at the position of the coordinates (X−1, Y) and the coordinates (X + 1, Y) is the average of these two points. The value can be the G interpolated pixel value at coordinates (X, Y).

  In step S250, the 0-degree direction interpolation G image calculation processing unit 71 calculates the M pixel value and the G interpolation value of the pixel of interest based on the fact that W pixel value = G pixel value + M pixel value. The pixel value of W is obtained by addition, and the process proceeds to step S251 described later.

  If it is determined in step S242 that the pixel of interest is a pixel having a pixel value of W, the pixel of interest has a pixel value of M in step S248 after the processing of step S244 or step S247 is completed. When it is determined that the pixel is not a pixel, or after the process of step S250 is completed, in step S251, the 0 degree direction-interpolated G image calculation processing unit 71 determines whether or not the value of W has been obtained for all the pixels. To do. If it is determined in step S251 that the value of W has not been obtained for all pixels, the process returns to step S241 and the subsequent processes are repeated.

  If it is determined in step S251 that W values have been obtained for all the pixels, in step S252, the 0-degree direction-interpolated G image calculation processing unit 71 sets unprocessed target pixels for which G value determination processing has not been performed. Select one.

  In step S253, the 0 degree direction-interpolated G image calculation processing unit 71 detects G pixels existing within a predetermined range from the target pixel. Here, the predetermined range is a range that is empirically or experimentally determined based on conditions such as the number of samples for principal component analysis and the strength of correlation due to separation from the pixel of interest. The predetermined range is indicated by coordinates (X, Y) that satisfy H-4 ≦ X ≦ H + 4 and K + 4 ≦ Y ≦ K + 4, for example, where the pixel position of the pixel of interest is the coordinates (H, K). It may be a range of 9 × 9 pixels or the like, or may be a size other than this, or a range of a predetermined number of pixels selected from pixels with a close distance between the pixels centered on the target pixel. There may be.

  In addition, the predetermined range here and the predetermined range in the 90-degree direction-interpolated G component image generation processing described later are patterns in the respective directions of 0 degree, 45 degrees, 90 degrees, and 135 degrees described above. It may be the same as or different from the predetermined range used in the accuracy calculation process.

  In step S254, the 0-degree direction-interpolated G image calculation processing unit 71 determines the G pixel value of each pixel detected in step S253 and the W value obtained at the pixel position as 2 of G and W. Plot in dimensional space and do principal component analysis.

  That is, since the values of W at all the pixel positions are obtained by the above-described processing, naturally, the pixel values of W are also obtained at the pixel positions of the detected G pixels. Therefore, when the predetermined range is, for example, H-4 ≦ X ≦ H + 4 and K + 4 ≦ Y ≦ K + 4 with respect to the coordinates (H, K) of the target pixel, X is an even number and Y is an odd number Since a plurality of G pixels indicated by coordinates (X, Y) are included in a predetermined range, a plurality of pairs of W values and G values can be created. By plotting the plurality of pairs in a two-dimensional space of W and G and performing principal component analysis, a straight line of the first principal component as shown in FIG. 25 can be obtained.

  Here, when the variance of the G pixel values detected in step S253 is varianceG, the average is AveG, the variance of the W pixel values at these pixel positions is varianceW, and the average is AveW, The formula of one principal component straight line is shown by the following formula (5).

  However, the coefficient S is +1 for a positive correlation and -1 for a negative correlation.

  In step S255, the 0 degree direction interpolation G image calculation processing unit 71 obtains a G pixel value based on the straight line obtained as a result of the principal component analysis and the W pixel value of the target pixel.

  Specifically, as shown in FIG. 25, the G pixel value corresponding to the W pixel value of the target pixel on the straight line of the first principal component is obtained. By substituting the value of W at the target pixel (H, K) into the equation, the value of G is obtained. However, here, it is assumed that there is a correlation between the values of W and G within a predetermined range such as H-4 ≦ X ≦ H + 4 and K + 4 ≦ Y ≦ K + 4.

  In step S256, the 0 degree direction-interpolated G image calculation processing unit 71 determines whether or not G values have been obtained for all the pixels. If it is determined in step S256 that G values have not been obtained for all pixels, the process returns to step S252, and the subsequent processes are repeated. If it is determined in step S256 that G values have been obtained for all the pixels, the process returns to step S47 in FIG. 13 and proceeds to step S48.

  By such processing, an image of W in all pixels is calculated by interpolation in the 0 degree direction, and G in all pixels is calculated by correlation between G and W using the result. An image of the G component can be obtained by interpolation.

  Next, the 45-degree direction-interpolated G component image generation process executed in step S48 of FIG. 13 will be described with reference to the flowchart of FIG.

  In the filter arrangement of the color filter 22 described with reference to FIG. 6, every other line in the input image includes G which is composed of coordinates (X, Y) where X + Y is an odd number and extends in the 45 ° direction. Therefore, in the pixel indicated by R or B, the G interpolation pixel value can be calculated by interpolation in the 45 degree direction. However, a line extending in the direction of 45 degrees constituted by coordinates (X, Y) where X + Y is an even number does not have a G value in the input image, but is constituted by W or M. With -M, the value of G can be calculated.

  First, in step S281, the 45-degree direction-interpolated G image calculation processing unit 72 selects one target pixel that has not been subjected to G value extraction or interpolation processing.

  In step S282, the 45-degree direction-interpolated G image calculation processing unit 72 determines whether or not the target pixel is a pixel having a G pixel value. There is a pixel originally indicated by G at coordinates (X, Y) where X is an even number and Y is an odd number. If it is determined in step S282 that the pixel of interest is a pixel having a G pixel value, this value may be set to the G value at the coordinates (X, Y), and the process will be described later in step S291. Proceed to

  When it is determined in step S282 that the target pixel is not a pixel having a G pixel value, in step S283, the 45 degree direction-interpolated G image calculation processing unit 72 is adjacent to the target pixel in the 45 degree direction. It is determined whether there is a G pixel as a pixel. For pixels with coordinates (X, Y) where X + Y is an odd number, the pixels with coordinates (X-1, Y-1) and coordinates (X + 1, Y + 1) that are adjacent in the 45-degree direction have G pixel values. doing. If it is determined in step S283 that there is no G pixel in the 45-degree direction with respect to the target pixel, the process proceeds to step S285 described later.

  If it is determined in step S283 that there is a G pixel in the 45 degree direction with respect to the target pixel, in step S284, the 45 degree direction interpolation G image calculation processing unit 72 uses the G pixel value in the 45 degree direction as a basis. Then, an interpolation value of G is obtained. That is, the 45-degree direction-interpolated G image calculation processing unit 72 calculates the average of the pixel value at the coordinates (X−1, Y−1) and the pixel value at the coordinates (X + 1, Y + 1) as G in the coordinates (X, Y). The processing proceeds to step S291 described later.

  When it is determined in step S283 that there is no G pixel in the 45 degree direction with respect to the target pixel, in step S285, the 45 degree direction interpolation G image calculation processing unit 72 sets the pixel value of W to the target pixel. It is determined whether or not the pixel is a pixel having the same. If it is determined in step S285 that the target pixel is not a pixel having a W pixel value, the process proceeds to step S288 described later.

  When it is determined in step S285 that the target pixel is a pixel having a W pixel value, in step S286, the 45-degree direction interpolation G image calculation processing unit 72 is based on the M pixel value in the 45-degree direction. , M interpolated values are obtained. That is, there is a pixel indicated by W at coordinates (X, Y) where X is an even number and Y is an even number. Then, since there are pixels adjacent to the pixel in the 45-degree direction, that is, the pixels indicated by M at the coordinates (X-1, Y-1) and coordinates (X + 1, Y + 1), these 2 The average value of the points can be M interpolated pixel values at the coordinates (X, Y).

  In step S287, the 45-degree direction-interpolated G image calculation processing unit 72 calculates the M interpolation value from the W pixel value of the target pixel based on the G pixel value = W pixel value−M pixel value. The pixel value of G is obtained by subtraction, and the process proceeds to step S291 described later.

  If it is determined in step S285 that the target pixel is not a pixel having a pixel value of W, in step S288, the 45-degree direction-interpolated G image calculation processing unit 72 determines that the target pixel has a pixel value of M It is determined whether or not. If it is determined in step S288 that the target pixel is not a pixel having an M pixel value, the process proceeds to step S291 described later.

  When it is determined in step S288 that the target pixel is a pixel having an M pixel value, in step S289, the 45-degree direction interpolation G image calculation processing unit 72 is adjacent to the target pixel in the 45-degree direction. An interpolated value of W is obtained based on the W pixel value of the pixel. That is, there is a pixel indicated by M at coordinates (X, Y) where X is an odd number and Y is an odd number. Since there are pixels adjacent to the pixel in the direction of 45 degrees, that is, the coordinates (X-1, Y-1) and the coordinates (X + 1, Y + 1) are indicated by W, these two points Can be the interpolated pixel value of W at the coordinates (X, Y).

  In step S290, the 45-degree direction-interpolated G image calculation processing unit 72 calculates the M pixel value from the W interpolation value of the target pixel based on the G pixel value = W pixel value−M pixel value. The pixel value of G is obtained by subtraction, and the process proceeds to step S291 described later.

  If it is determined in step S282 that the pixel of interest is a pixel having a G pixel value, the pixel of interest has an M pixel value in step S288 after the processing of step S284 or step S287 is completed. If it is determined that the pixel is not a pixel, or after the process of step S290 is completed, in step S291, the 45-degree direction-interpolated G image calculation processing unit 72 determines whether or not a G value has been obtained for all the pixels. To do.

  If it is determined in step S291 that the value of G is not obtained for all pixels, the process returns to step S281, and the subsequent processes are repeated. If it is determined in step S291 that G values have been obtained for all the pixels, the process returns to step S48 in FIG. 13 and proceeds to step S49.

  By such processing, an image of the G component can be obtained by interpolation in the 45 degree direction.

  Next, the 90-degree direction-interpolated G component image generation process executed in step S49 in FIG. 13 will be described with reference to the flowchart in FIG.

  In the filter arrangement of the color filter 22 described with reference to FIG. 6, every other line having an even X-axis value has a G value in the input image, and therefore is indicated by W. In the pixel, the interpolation pixel value of G can be calculated by interpolation in the 90 degree direction. However, since the G value does not exist in the input image for the line having an odd value on the X axis, the interpolation pixel value of G cannot be directly calculated by interpolation in the 90 degree direction. Therefore, first, M images at all pixels are calculated by interpolation in the direction of 90 degrees, and then G at all pixels is calculated by correlation between G and M.

  First, in step S321, the 90-degree direction-interpolated G image calculation processing unit 73 selects one target pixel that has not been subjected to M value extraction or interpolation processing.

  In step S322, the 90-degree direction-interpolated G image calculation processing unit 73 determines whether or not the target pixel is a pixel having an M pixel value. There is a pixel originally indicated by M at coordinates (X, Y) where X is an odd number and Y is an odd number. If it is determined in step S322 that the target pixel is a pixel having an M pixel value, this value may be set to the M value at the coordinates (X, Y), and the process will be described later in step S331. Proceed to

  If it is determined in step S322 that the target pixel is not a pixel having an M pixel value, in step S323, the 90-degree direction interpolation G image calculation processing unit 73 is adjacent to the target pixel in the 90-degree direction. It is determined whether or not there are M pixels as pixels. For pixels with coordinates (X, Y) where X is an odd number and Y is an even number, the pixels with coordinates (X, Y-1) and coordinates (X, Y + 1) adjacent in the 90 degree direction are M It has a pixel value. If it is determined in step S323 that there is no M pixel in the 90-degree direction with respect to the target pixel, the process proceeds to step S325 described later.

  If it is determined in step S323 that there are M pixels in the 90 degree direction with respect to the target pixel, in step S324, the 90 degree direction-interpolated G image calculation processing unit 73 is based on the M pixel value in the 90 degree direction. Then, an interpolation value of M is obtained. That is, the 90-degree direction interpolation G image calculation processing unit 73 calculates the average of the pixel value at the coordinate (X, Y−1) and the pixel value at the coordinate (X, Y + 1), and interpolates M at the coordinate (X, Y). The pixel value is set, and the process proceeds to step S331 described later.

  If it is determined in step S323 that there is no M pixel in the 90 degree direction with respect to the target pixel, in step S325, the 90 degree direction interpolation G image calculation processing unit 73 sets the G pixel value to the target pixel. It is determined whether or not the pixel is a pixel having the same. If it is determined in step S325 that the target pixel is not a pixel having the G pixel value, the process proceeds to step S328 described later.

  When it is determined in step S325 that the target pixel is a pixel having a G pixel value, in step S326, the 90-degree direction interpolation G image calculation processing unit 73 is based on the W pixel value in the 90-degree direction. , W interpolated values are obtained. That is, there is a pixel indicated by G at coordinates (X, Y) where X is an even number and Y is an odd number. Since there are pixels adjacent to the pixel in the 90-degree direction, that is, at the coordinates (X, Y-1) and coordinates (X, Y + 1), these two points are included. The average value can be the interpolated pixel value of W at the coordinates (X, Y).

  In step S327, the 90-degree direction-interpolated G image calculation processing unit 73 calculates the G pixel value from the W interpolation value of the target pixel based on the fact that the pixel value of M = the pixel value of W−the pixel value of G. The pixel value of M is obtained by subtraction, and the process proceeds to step S331 described later.

  If it is determined in step S325 that the target pixel is not a pixel having a G pixel value, in step S328, the 90-degree direction-interpolated G image calculation processing unit 73 determines that the target pixel has a W pixel value. It is determined whether or not. If it is determined in step S328 that the target pixel is not a pixel having a W pixel value, the process proceeds to step S331 to be described later.

  When it is determined in step S328 that the target pixel is a pixel having a pixel value of W, in step S329, the 90-degree direction interpolation G image calculation processing unit 73 is adjacent to the target pixel in the 90-degree direction. Based on the G pixel value of the pixel, the G interpolation value is obtained. That is, there is a pixel indicated by W at coordinates (X, Y) where X is an even number and Y is an even number. The pixel adjacent to the pixel in the direction of 90 degrees, that is, the pixel indicated by G at the position of the coordinates (X, Y−1) and the coordinates (X, Y + 1), and the average of these two points The value can be the G interpolated pixel value at coordinates (X, Y).

  In step S330, the 90-degree direction-interpolated G image calculation processing unit 73 calculates the G interpolation pixel value from the W pixel value of the target pixel based on the M pixel value = W pixel value−G pixel value. Is subtracted to obtain the pixel value of M, and the process proceeds to step S331 described later.

  If it is determined in step S322 that the pixel of interest is a pixel having a pixel value of M, the pixel of interest has a pixel value of W in step S328 after the processing of step S324 or step S327 is completed. When it is determined that the pixel is not a pixel, or after the process of step S330 is completed, in step S331, the 90-degree direction-interpolated G image calculation processing unit 73 determines whether or not the value of M has been obtained for all the pixels. To do. If it is determined in step S331 that the value of M has not been obtained for all the pixels, the process returns to step S321, and the subsequent processes are repeated.

  If it is determined in step S331 that the value of M has been obtained for all pixels, in step S332, the 90-degree direction-interpolated G image calculation processing unit 73 sets an unprocessed target pixel for which the G value determination process has not been performed. Select one.

  In step S333, the 90-degree direction-interpolated G image calculation processing unit 73 detects G pixels existing within a predetermined range from the target pixel. Here, the predetermined range is a range that is empirically or experimentally determined based on conditions such as the number of samples for principal component analysis and the strength of correlation due to separation from the pixel of interest. The predetermined range is indicated by coordinates (X, Y) that satisfy H-4 ≦ X ≦ H + 4 and K + 4 ≦ Y ≦ K + 4, for example, where the pixel position of the pixel of interest is the coordinates (H, K). It may be a range of 9 × 9 pixels or the like, or may be a size other than this, or a range of a predetermined number of pixels selected from pixels with a close distance between the pixels centered on the target pixel. There may be.

  In step S334, the 90-degree direction-interpolated G image calculation processing unit 73 calculates the G pixel value of each pixel detected in step S333 and the M value obtained at the pixel position as G and M 2. Plot in dimensional space and do principal component analysis.

  That is, since the M values at all the pixel positions are obtained by the above-described processing, the M pixel values are naturally obtained even at the detected pixel positions of the G pixels. Therefore, when the predetermined range is, for example, H-4 ≦ X ≦ H + 4 and K + 4 ≦ Y ≦ K + 4 with respect to the coordinates (H, K) of the target pixel, X is an even number and Y is an odd number Since a plurality of G pixels indicated by coordinates (X, Y) are included, a plurality of pairs of M and G values can be created. By plotting the plurality of pairs in a two-dimensional space of M and G and performing principal component analysis, a straight line of the first principal component as shown in FIG. 28 can be obtained.

  Here, when the variance of G pixel values detected in step S333 is variance G, the average is Ave G, the variance of M pixel values at these pixel positions is variance M, and the average is Ave M, The equation of the straight line of one principal component is expressed by the following equation (6).

  However, the coefficient S is +1 for a positive correlation and -1 for a negative correlation.

  In step S335, the 90-degree direction-interpolated G image calculation processing unit 73 obtains the G pixel value based on the straight line obtained as a result of the principal component analysis and the M pixel value of the target pixel.

  Specifically, as shown in FIG. 25, the G pixel value corresponding to the M pixel value of the pixel of interest on the straight line of the first principal component is obtained, and the straight line shown in Expression (6) By substituting the value of M at the target pixel (H, K) into the equation, the value of G can be obtained. However, here, it is assumed that there is a correlation between the values of M and G within a predetermined range such as H-4 ≦ X ≦ H + 4 and K + 4 ≦ Y ≦ K + 4.

  In step S336, the 90-degree direction-interpolated G image calculation processing unit 73 determines whether G values have been obtained for all the pixels. If it is determined in step S336 that G values have not been obtained for all pixels, the process returns to step S332, and the subsequent processes are repeated. If it is determined in step S336 that G values have been obtained for all the pixels, the process returns to step S49 in FIG. 13 and proceeds to step S50.

  By such processing, an image of M in all pixels is calculated by interpolation in the direction of 90 degrees, and G is calculated in all pixels by correlation between G and M using the result. An image of the G component can be obtained by interpolation.

  Next, the 135-degree direction-interpolated G component image generation process executed in step S50 of FIG. 13 will be described with reference to the flowchart of FIG.

  In the filter arrangement of the color filter 22 described with reference to FIG. 6, every other line in the input image is formed in a line extending in the direction of 135 degrees constituted by coordinates (X, Y) where XY is an odd number. Therefore, in the pixel indicated by R or B, the G interpolation pixel value can be calculated by interpolation in the 135 degree direction. However, the line extending in the direction of 135 degrees constituted by coordinates (X, Y) where XY is an even number does not have a G value in the input image, but is constituted by W or M. , W-M, the value of G can be calculated.

  First, in step S361, the 135-degree direction interpolation G image calculation processing unit 74 selects one target pixel that has not been subjected to G value extraction or interpolation processing.

  In step S362, the 135-degree direction-interpolated G image calculation processing unit 74 determines whether or not the target pixel is a pixel having a G pixel value. There is a pixel originally indicated by G at coordinates (X, Y) where X is an even number and Y is an odd number. If it is determined in step S362 that the pixel of interest is a pixel having a G pixel value, this value may be set to the G value at the coordinates (X, Y), and the process will be described later in step S371. Proceed to

  If it is determined in step S362 that the target pixel is not a pixel having a G pixel value, in step S363, the 135-degree direction interpolation G image calculation processing unit 74 is adjacent to the target pixel in the 135-degree direction. It is determined whether there is a G pixel as a pixel. The pixels at coordinates (X-1, Y + 1) and (X + 1, Y-1) adjacent in the 135 degree direction with respect to the pixels at coordinates (X, Y) where X + Y is an odd number have a pixel value of G. doing. If it is determined in step S363 that there is no G pixel in the direction of 135 degrees with respect to the target pixel, the process proceeds to step S365 described later.

  If it is determined in step S363 that there is a G pixel in the 135 degree direction with respect to the target pixel, in step S364, the 135 degree direction interpolation G image calculation processing unit 74 uses the G pixel value in the 135 degree direction as a basis. Then, an interpolation value of G is obtained. That is, the 135-degree direction interpolation G image calculation processing unit 74 calculates the average of the pixel value at the coordinates (X−1, Y + 1) and the pixel value at the coordinates (X + 1, Y−1) as G in the coordinates (X, Y). The processing proceeds to step S371 described later.

  If it is determined in step S363 that there is no G pixel in the 135-degree direction with respect to the target pixel, in step S365, the 135-degree direction interpolation G image calculation processing unit 74 sets the pixel value of W as the target pixel. It is determined whether or not the pixel is a pixel having the same. If it is determined in step S365 that the target pixel is not a pixel having a W pixel value, the process proceeds to step S368 described later.

  When it is determined in step S365 that the target pixel is a pixel having a W pixel value, in step S366, the 135-degree direction interpolation G image calculation processing unit 74 is based on the M pixel value in the 135-degree direction. , M interpolated values are obtained. That is, there is a pixel indicated by W at coordinates (X, Y) where X is an even number and Y is an even number. Since there are pixels adjacent to the pixel in the direction of 135 degrees, that is, at the coordinates (X-1, Y + 1) and the coordinates (X + 1, Y-1), these pixels are indicated by M. The average value of the points can be M interpolated pixel values at the coordinates (X, Y).

  In step S367, the 135-degree direction interpolation G image calculation processing unit 74 calculates the M interpolation value from the W pixel value of the target pixel based on the G pixel value = W pixel value−M pixel value. The pixel value of G is obtained by subtraction, and the process proceeds to step S371 described later.

  If it is determined in step S365 that the target pixel is not a pixel having a pixel value of W, in step S368, the 135-degree directional interpolation G image calculation processing unit 74 determines that the target pixel is a pixel having an M pixel value. It is determined whether or not. If it is determined in step S368 that the target pixel is not a pixel having an M pixel value, the process proceeds to step S371 described later.

  When it is determined in step S368 that the target pixel is a pixel having an M pixel value, in step S369, the 135-degree direction interpolation G image calculation processing unit 74 is adjacent to the target pixel in the 135-degree direction. An interpolated value of W is obtained based on the W pixel value of the pixel. That is, there is a pixel indicated by M at coordinates (X, Y) where X is an odd number and Y is an odd number. Since there are pixels adjacent to the pixel in the direction of 135 degrees, that is, the coordinates (X-1, Y + 1) and the coordinates (X + 1, Y-1) are indicated by W, these two points Can be the interpolated pixel value of W at the coordinates (X, Y).

  In step S370, the 135-degree direction-interpolated G image calculation processing unit 74 calculates the M pixel value from the W interpolation value of the target pixel based on the fact that the G pixel value = W pixel value−M pixel value. The pixel value of G is obtained by subtraction, and the process proceeds to step S371 described later.

  If it is determined in step S362 that the pixel of interest is a pixel having a G pixel value, the pixel of interest has an M pixel value in step S368 after the processing of step S364 or step S367 is completed. When it is determined that the pixel is not a pixel, or after the process of step S370 is completed, in step S371, the 135-degree direction-interpolated G image calculation processing unit 74 determines whether or not a G value has been obtained for all the pixels. To do.

  If it is determined in step S371 that the value of G is not obtained for all pixels, the process returns to step S361, and the subsequent processes are repeated. If it is determined in step S371 that G values have been obtained for all the pixels, the process returns to step S50 in FIG. 13 and proceeds to step S51.

  By such processing, an image of the G component can be obtained by interpolation in the 135 degree direction.

  Then, as described above, in step S51 in FIG. 13, the G component image corrected in the direction of each angle is multiplied by the accuracy correction appropriateness as a weighting coefficient, and added to each pixel, thereby adding a pattern. In addition, a G component image generated using the property of WM = G in the pixels of the W and M components is generated.

  Next, the R component pixel value calculation process executed in step S52 of FIG. 13 will be described with reference to the flowchart of FIG.

  In step S401, the R image calculation processing unit 81 selects one unprocessed target pixel.

  In step S402, the R image calculation processing unit 81 detects R pixels existing within a predetermined range from the target pixel. Here, the predetermined range is a range that is empirically or experimentally determined based on conditions such as the number of samples for principal component analysis and the strength of correlation due to separation from the pixel of interest. The predetermined range is indicated by coordinates (X, Y) that satisfy H-4 ≦ X ≦ H + 4 and K + 4 ≦ Y ≦ K + 4, for example, where the pixel position of the pixel of interest is the coordinates (H, K). It may be a range of 9 × 9 pixels or the like, or may be a size other than this, or a range of a predetermined number of pixels selected from pixels with a close distance between the pixels centered on the target pixel. There may be.

  Further, the predetermined range here and the predetermined range in the B component pixel value calculation process described later may be the same as or different from the predetermined range used in any of the above-described processes. good.

  For example, when the pixel position of the pixel of interest is the coordinates (H, K), H-4 ≦ X ≦ H + 4, K + 4 ≦ Y ≦ K + 4, X is an odd number, Y is an even number, and XY-3 is four. There is a pixel indicated by R at coordinates (X, Y) which is a multiple of.

  In step S403, the R image calculation processing unit 81 acquires G pixel values estimated by the process of step S51 of FIG. 13 for the pixel positions of all detected R pixels (X, Y). . That is, the R image calculation processing unit 81 acquires a plurality of pairs of R pixel values and G pixel values at the pixel positions of all detected R pixels (X, Y).

  In step S404, the R image calculation processing unit 81 plots the R value and G pair value of each pixel in a two-dimensional space of R and G as shown in FIG. 31, and performs principal component analysis. Do.

  Here, when the variance of R pixel values within a predetermined range is varianceR, the average is AveR, the variance of G pixel values at those pixel positions is varianceG, and the average is AveG, the straight line expression of the first principal component Is expressed by the following equation (7).

  However, the coefficient S is +1 for a positive correlation and -1 for a negative correlation.

  In step S405, the R image calculation processing unit 81 obtains an R value based on the straight line obtained as a result of the principal component analysis and the G value of the target pixel. That is, the R image calculation processing unit 81 obtains the value of R by substituting the G pixel value of the target pixel into the above-described equation (7).

  In step S406, the R image calculation processing unit 81 determines whether R has been obtained for all pixels. If it is determined in step S406 that R has not been obtained for all pixels, the processing returns to step S401, and the subsequent processing is repeated. If it is determined in step S406 that R has been obtained for all the pixels, the process returns to step S52 in FIG. 13 and proceeds to step S53.

  By such processing, R component pixel values for all pixels are obtained based on the R pixel value included in the imaging data and the G pixel value estimated by the processing in step S51 of FIG.

  Here, for example, in a predetermined region such as H-4 ≦ X ≦ H + 4 and K + 4 ≦ Y ≦ K + 4, it is assumed that the pixel values of the R component and the G component are correlated. Processing is being executed.

  Next, the B component pixel value calculation process executed in step S53 in FIG. 13 will be described with reference to the flowchart in FIG.

  In step S441, the B image calculation processing unit 82 selects one unprocessed target pixel.

  In step S442, the B image calculation processing unit 82 detects B pixels existing within a predetermined range from the target pixel. Here, the predetermined range is a range that is empirically or experimentally determined based on conditions such as the number of samples for principal component analysis and the strength of correlation due to separation from the pixel of interest. The predetermined range is indicated by coordinates (X, Y) that satisfy H-4 ≦ X ≦ H + 4 and K + 4 ≦ Y ≦ K + 4, for example, where the pixel position of the pixel of interest is the coordinates (H, K). It may be a range of 9 × 9 pixels or the like, or may be a size other than this, or a range of a predetermined number of pixels selected from pixels with a close distance between the pixels centered on the target pixel. There may be.

  For example, when the pixel position of the pixel of interest is the coordinates (H, K), H-4 ≦ X ≦ H + 4, K + 4 ≦ Y ≦ K + 4, X is an odd number, Y is an even number, and XY-1 is 4 There is a pixel indicated by B at coordinates (X, Y) which is a multiple of.

  In step S443, the B image calculation processing unit 82 acquires G pixel values estimated by the processing in step S51 of FIG. 13 for the pixel positions of all the detected B pixels (X, Y). . That is, the B image calculation processing unit 82 acquires a plurality of pairs of B pixel values and G pixel values at the pixel positions of all detected B pixels (X, Y).

  In step S444, the B image calculation processing unit 82 plots the B value and G pair value of each pixel in a two-dimensional space of B and G, as shown in FIG. Do.

  Here, when the variance of the B pixel values within a predetermined range is variance B, the average is Ave B, the variance of the G pixel values at those pixel positions is variance G, and the average is Ave G, the straight line expression of the first principal component Is expressed by the following equation (8).

  However, the coefficient S is +1 for a positive correlation and -1 for a negative correlation.

  In step S445, the B image calculation processing unit 82 obtains the B value based on the straight line obtained as a result of the principal component analysis and the G value of the target pixel. That is, the B image calculation processing unit 82 obtains the value of B by substituting the G pixel value of the target pixel into the above-described equation (8).

  In step S446, the B image calculation processing unit 82 determines whether or not B has been obtained for all the pixels. If it is determined in step S446 that B has not been obtained for all pixels, the process returns to step S441, and the subsequent processes are repeated. If it is determined in step S446 that B has been obtained for all the pixels, the process returns to step S52 in FIG. 13 and proceeds to step S53.

  By such processing, the B component pixel values for all the pixels are obtained based on the B pixel values included in the imaging data and the G pixel values estimated by the processing in step S51 of FIG.

  In this case, too, it is assumed that the pixel values of the B component and the G component are correlated within a predetermined region such as H-4 ≦ X ≦ H + 4 and K + 4 ≦ Y ≦ K + 4. Processing is being executed.

  In each of the processes described above, the second pixel having the second spectral component data from the pixel (for example, the pixel indicated by W) data of the first pixel group having the first spectral component data. Subtract the data of the group of pixels (for example, the pixel indicated by M) (in other words, the weight W1 = 1 and the weight W2 = (− 1), and the first pixel group and the second by the weights W1 and W2. A description has been given of obtaining pixel data of a spectral component (for example, G (green) component) in a predetermined frequency range free from chromatic aberration by obtaining a linear sum with a pixel group. That is, the case where there is a relationship of G = WM has been described. On the other hand, for example, when M ′ having a sensitivity of ½ with respect to the sensitivity of the M pixel is used, G = W−M ′ is not satisfied. Instead, G = W The demosaic process may be executed based on −2 × M ′. Specifically, for example, the output value of the pixel corresponding to M of the solid-state imaging device 23 may be doubled and supplied to the A / D conversion unit 24 to perform the same processing. . Similarly, when the sensitivity of the signal corresponding to one of the color components is inferior to that of the other, the output value of the corresponding pixel of the solid-state image sensor 23 is increased by a predetermined magnification, and then A The same processing may be executed by supplying the data to the / D conversion unit 24.

  That is, the signal in the frequency band of the first spectral component (or a value obtained by multiplying the signal by a predetermined coefficient) to the signal in the frequency band of the second spectral component (or the signal is multiplied by a predetermined coefficient). Is obtained on the basis of the signal component corresponding to the spectral component in the predetermined frequency region obtained by subtracting the value), and a demosaic process is performed based on the obtained image in the frequency region without chromatic aberration. Thus, for example, when the frequency band of the first spectral component and the frequency band of the second spectral component are other than the illustrated frequency band, a plurality of dissociated frequencies without being continuous The present invention can also be applied to a case where a band is used. Further, as in the illustrated process, if the spectral component in the predetermined frequency region obtained by subtracting the frequency band of the second spectral component from the frequency band of the first spectral component matches the frequency region without chromatic aberration. The processing is simple and preferable. For example, the frequency band of the first spectral component is the G (green) component and the infrared component, the frequency band of the second spectral component is the infrared component, or the frequency band of the first spectral component is the infrared component. The present invention can also be applied to a visible light component excluding components and a frequency band of the second spectral component as an R (red) component and a B (blue) component.

  By applying the present invention as described above, it is possible to perform demosaic processing while having pixels that can acquire a spectral component in a wide wavelength band and canceling out chromatic aberration of a lens.

  The arrangement of the color filters can be set as shown in FIG. 34. For example, the arrangement shown in FIG. 34 may be used. The color filter array shown in FIG. 34 corresponds to a configuration obtained by rotating the array shown in FIG. 7 by 45 degrees.

  That is, in the color filter shown in FIG. 34, when the X-axis direction is set to 0 degree and attention is paid to the pixel indicated by G, the pixel indicated by G has a 0-degree direction, a 45-degree direction, a 90-degree direction, and The pixels adjacent to the pixel indicated by G in the 0 degree direction are all pixels indicated by B or R, and 90 pixels are indicated by G. The pixels adjacent to each other in the degree direction are pixels indicated by B or R, and the pixels adjacent to the pixel indicated by G in the 45 degree direction are all pixels indicated by M and indicated by G. Any pixel that is adjacent to the pixel in the direction of 135 degrees is a pixel indicated by W.

  In the color filter array shown in FIG. 34, even if the arrangement of the W and M pixels is reversed, the above-described demosaic process can be executed in the same manner, and the arrangement of the B and R pixels. Even if reverse, the above-described demosaic process can be executed in the same manner.

  That is, when the X axis direction is set to 0 degree and attention is paid to the pixel indicated by G, the pixel indicated by G is one pixel in any of the 0 degree direction, the 45 degree direction, the 90 degree direction, and the 135 degree direction. Arranged adjacent to each other so that the pixels of the other color components R, B, W, and M are positioned so that R and B, and W and M are orthogonal to the G pixel. When the color filter to be used is used, the demosaicing process described above can be performed in the same manner.

  In the above-described processing, the demosaicing process is performed by obtaining pattern accuracy at four angles of 0 degrees, 45 degrees, 90 degrees, and 135 degrees and obtaining an image of the G component by interpolation processing in each direction. However, the present invention is applicable to processing other than four angles of 0 degrees, 45 degrees, 90 degrees, and 135 degrees, for example, 0 degrees and 90 degrees, or Needless to say, the present invention can also be applied to the case of two angles such as 45 degrees and 135 degrees.

  In general, when a landscape or the like is imaged, there are many cases in which a pattern appears in the 0 degree direction and the 90 degree direction with respect to the captured image. Compared with these, there are not many 45 degree or 135 degree patterns. Therefore, in order to reduce the circuit scale or reduce the calculation time, pattern accuracy is obtained at two angles of 0 degrees and 90 degrees, and an image of the G component is obtained by interpolation processing in each direction. The demosaic process may be executed.

  Of the series of processes described above, in particular, the processes executed in the camera signal processing unit 25 can be executed not only by hardware but also by software. The software is a computer in which the program constituting the software is incorporated in dedicated hardware, or various functions can be executed by installing various programs, for example, a general-purpose personal computer Installed from a recording medium. In this case, for example, the imaging apparatus 11 described with reference to FIG. 4 includes a personal computer 101 as shown in FIG.

  35, a CPU (Central Processing Unit) 111 performs various processes according to a program stored in a ROM (Read Only Memory) 112 or a program loaded from a storage unit 118 to a RAM (Random Access Memory) 113. Execute. The RAM 113 also appropriately stores data necessary for the CPU 111 to execute various processes.

  The CPU 111, ROM 112, and RAM 113 are connected to each other via a bus 114. An input / output interface 115 is also connected to the bus 114.

  The input / output interface 115 includes an input unit 116 including a keyboard and a mouse, an output unit 117 including a display and a speaker, a storage unit 118 including a hard disk, a communication unit 119 including a modem and a terminal adapter, And the imaging process part 120 is connected. The communication unit 119 performs communication processing via a network including the Internet.

  The imaging processing unit 120 includes the optical lens 21, the color filter 22, the solid-state imaging device 23, and the A / D conversion unit 24 described with reference to FIG. 4, or can perform a similar function. Based on the control of the CPU 111 having the functions of the camera signal processing unit 25 described with reference to FIG. 5 and the image compression unit 27 of FIG. 4, the same processing as described above is executed.

  In addition, a drive 121 is connected to the input / output interface 115 as necessary, and a magnetic disk 111, an optical disk 112, a magneto-optical disk 133, a semiconductor memory 134, or the like is appropriately mounted, and a computer program read from them is loaded. If necessary, it is installed in the storage unit 118.

  When a series of processing is executed by software, a program constituting the software may execute various functions by installing a computer incorporated in dedicated hardware or various programs. For example, it is installed from a network or a recording medium into a general-purpose personal computer or the like.

  As shown in FIG. 35, this recording medium includes a magnetic disk 111 (including a floppy disk) and an optical disk 112 (including a floppy disk) that are distributed to supply a program to a user separately from the apparatus main body. Package media including CD-ROM (compact disk-read only memory), DVD (including digital versatile disk), magneto-optical disk 133 (including MD (mini-disk) (trademark)), or semiconductor memory 134 In addition to being configured, it is configured by a ROM 112 in which a program is stored and a hard disk included in the storage unit 118 supplied to the user in a state of being incorporated in the apparatus main body in advance.

  Further, in the present specification, the step of describing the program recorded on the recording medium is not limited to the processing performed in chronological order according to the described order, but may be performed in parallel or It also includes processes that are executed individually.

  The embodiment of the present invention is not limited to the above-described embodiment, and various modifications can be made without departing from the gist of the present invention.

  DESCRIPTION OF SYMBOLS 11 Imaging device 21 Lens, 22 Color filter, 23 Solid-state image sensor, 24 A / D conversion part, 25 Camera signal processing part, 26 Display part, 27 Image compression part, 28 Recording part, 29 External output part, 44 Demosaic processing part , 610 degree pattern analysis processing section, 62 45 degree direction pattern analysis processing section, 63 90 degree direction pattern analysis processing section, 64 135 degree direction pattern analysis processing section, 65 pattern direction determination section, 710 degree Direction interpolation G image calculation processing unit, 72 45 degree direction interpolation G image calculation processing unit, 73 90 degree direction interpolation G image calculation processing unit, 74 135 degree direction interpolation G image calculation processing unit, 75 G image calculation processing unit, 81 R Image calculation processing unit, 82 B image calculation processing unit

Claims (18)

An optical signal obtained by being incident on a predetermined color filter through a lens and receiving an optical signal having any one of a plurality of different spectral components for each pixel, and for each pixel In an image processing apparatus that generates image data based on the optical signal of
Based on a first pixel value corresponding to the optical signal in the predetermined pixel, and a second pixel value obtained by an interpolation process performed using a pixel value of an adjacent pixel of the predetermined pixel, Contrast component calculation means for calculating the contrast component of the image data,
The contrast component calculation means includes the first pixel value or the second pixel value.
A pixel value corresponding to a first spectral component having the widest frequency bandwidth among the plurality of different spectral components;
A pixel value corresponding to a second spectral component obtained by removing a predetermined frequency band in the vicinity of a frequency at which no chromatic aberration occurs in the lens from the first spectral component;
Or
Corresponds to a third spectral component represented by a linear sum of a value obtained by multiplying the first spectral component by a first weighting factor and a value obtained by multiplying the second spectral component by a second weighting factor. An image processing apparatus that obtains a contrast component of the image data based on a calculation process using pixel values. The image processing apparatus according to claim 1, wherein the contrast component of the image data calculated by the contrast component calculation unit includes a pixel value corresponding to the third spectral component. The image processing apparatus according to claim 2, wherein the third spectral component is a frequency component in a predetermined range corresponding to green. The contrast component calculation means includes
Green component calculation means for calculating a pixel value of each pixel corresponding to green based on the first pixel value or the second pixel value;
A red color for calculating a pixel value of each pixel corresponding to red, based on a pixel value corresponding to a predetermined frequency component corresponding to red among the plurality of different spectral components and a calculation result by the green component calculating means Component calculation means;
Blue that calculates a pixel value of each pixel corresponding to blue based on a pixel value corresponding to a predetermined frequency component corresponding to blue among a plurality of different spectral components and a calculation result by the green component calculation means The image processing apparatus according to claim 3, further comprising: a component calculation unit. A pattern direction estimating means for estimating the direction of the pattern near each pixel in the image data;
The image processing apparatus according to claim 1, wherein the contrast component calculation unit calculates a contrast component of the image data based on a pattern direction estimation result in the vicinity of each pixel by the pattern direction estimation unit. Pattern accuracy calculation for determining the accuracy of a pattern in each of the 0 degree direction, 45 degree direction, 90 degree direction, and 135 degree direction, assuming that one direction of the arrangement on the plane on which the pixels are arranged is 0 degree Further comprising means,
The image processing apparatus according to claim 5, wherein the pattern direction estimation unit estimates a direction of a pattern near each pixel in the image data based on a calculation result by the pattern accuracy calculation unit. The pattern direction estimation means is based on the calculation result by the pattern accuracy calculation means, and the direction of the pattern near each pixel in the image data is likely to be close to 0 degrees or 90 degrees, Alternatively, the image processing apparatus according to claim 6, wherein the image processing apparatus obtains which of 45 degrees and 135 degrees is likely to be close. The image processing apparatus according to claim 1, wherein the first spectral component includes at least an infrared component. Pixels corresponding to the third spectral component are in any of 0 degree direction, 45 degree direction, 90 degree direction, and 135 degree direction, with one direction on the plane in which the pixels are arranged as 0 degree. The image processing apparatus according to claim 1, wherein the image processing apparatus is disposed every other pixel. The plurality of different spectral components are five types of spectral components including the first spectral component, the second spectral component, and the third spectral component,
Pixels corresponding to the first spectral component and the second spectral component, and the fourth spectral component and the fifth spectral component, respectively, have one direction of the array on the plane where the pixels are arrayed as 0. As a degree, it is arranged adjacent to the pixel corresponding to the third spectral component in any one of the 0 degree direction, the 45 degree direction, the 90 degree direction, and the 135 degree direction, and the first Pixels corresponding to the spectral component and the second spectral component are arranged in a direction orthogonal to the pixel corresponding to the third spectral component, and correspond to the fourth spectral component and the fifth spectral component The image processing device according to claim 1, wherein the pixels to be arranged are arranged in a direction orthogonal to the pixels corresponding to the third spectral component. Receives an optical signal having one of a plurality of different spectral components obtained by being incident on a predetermined color filter through a lens, and receives image data based on the optical signal for each pixel. In the image processing method of the image processing apparatus for generating
Obtaining a first pixel value corresponding to the optical signal at a given pixel;
An interpolation process is performed using a pixel value of an adjacent pixel of the predetermined pixel to obtain a second pixel value,
Of the first pixel value and the second pixel value,
A pixel value corresponding to a first spectral component having the widest frequency bandwidth among the plurality of different spectral components;
A pixel value corresponding to a second spectral component obtained by removing a predetermined frequency band in the vicinity of a frequency at which no chromatic aberration occurs in the lens from the first spectral component;
Or
Corresponds to a third spectral component represented by a linear sum of a value obtained by multiplying the first spectral component by a first weighting factor and a value obtained by multiplying the second spectral component by a second weighting factor. An image processing method including a step of obtaining a contrast component of the image data by arithmetic processing using pixel values. Receives an optical signal having one of a plurality of different spectral components obtained by being incident on a predetermined color filter through a lens, and receives image data based on the optical signal for each pixel. A computer-executable program for controlling the process of generating
Controlling the acquisition of a first pixel value corresponding to the optical signal at a given pixel;
An interpolation process is performed using a pixel value that an adjacent pixel of the predetermined pixel has to calculate a second pixel value,
Of the first pixel value and the second pixel value,
A pixel value corresponding to a first spectral component having the widest frequency bandwidth among the plurality of different spectral components;
A pixel value corresponding to a second spectral component obtained by removing a predetermined frequency band in the vicinity of a frequency at which no chromatic aberration occurs in the lens from the first spectral component;
Or
Corresponds to a third spectral component represented by a linear sum of a value obtained by multiplying the first spectral component by a first weighting factor and a value obtained by multiplying the second spectral component by a second weighting factor. A program for causing a computer to execute a process including a step of obtaining a contrast component of the image data by an arithmetic process using pixel values.   A recording medium on which the program according to claim 12 is recorded. In an imaging device that captures an image,
Optical signal acquisition means for acquiring light incident through the lens for each pixel as optical signals of a plurality of different spectral components through a predetermined color filter;
Conversion means for converting the optical signal acquired by the optical signal acquisition means into a digital signal;
Image processing means for processing the digital signal converted by the conversion means to generate image data in which pixel values corresponding to a plurality of predetermined color components are aligned in all pixels;
The image processing means includes
Based on a first pixel value corresponding to the optical signal in the predetermined pixel, and a second pixel value obtained by an interpolation process performed using a pixel value of an adjacent pixel of the predetermined pixel, Contrast component calculation means for calculating the contrast component of the image data,
The contrast component calculation means includes the first pixel value or the second pixel value.
A pixel value corresponding to a first spectral component having the widest frequency bandwidth among the plurality of different spectral components;
A pixel value obtained by removing a second spectral component corresponding to a predetermined frequency band in the vicinity of a frequency at which no chromatic aberration occurs in the lens from the first spectral component;
Or
Corresponds to a third spectral component represented by a linear sum of a value obtained by multiplying the first spectral component by a first weighting factor and a value obtained by multiplying the second spectral component by a second weighting factor. An imaging apparatus that obtains a contrast component of the image data based on a calculation process using a pixel value. The plurality of different spectral components are five types of spectral components including the first spectral component, the second spectral component, and the third spectral component,
Pixels corresponding to the first spectral component and the second spectral component, and the fourth spectral component and the fifth spectral component, respectively, have one direction of the array on the plane where the pixels are arrayed as 0. As a degree, it is arranged adjacent to the pixel corresponding to the third spectral component in any one of the 0 degree direction, the 45 degree direction, the 90 degree direction, and the 135 degree direction, and the first Pixels corresponding to the spectral component and the second spectral component are arranged in a direction orthogonal to the pixel corresponding to the third spectral component, and correspond to the fourth spectral component and the fifth spectral component The imaging device according to claim 14, wherein pixels to be arranged are arranged in a direction orthogonal to a pixel corresponding to the third spectral component. In an imaging method of an imaging device that captures an image,
The light incident through the lens is acquired for each pixel as optical signals of a plurality of different spectral components through a predetermined color filter,
Converting the acquired optical signal into a digital signal;
Obtaining a first pixel value corresponding to the optical signal at the predetermined pixel from the converted digital signal;
An interpolation process is performed using a pixel value of an adjacent pixel of the predetermined pixel to obtain a second pixel value,
Of the first pixel value and the second pixel value,
A pixel value corresponding to a first spectral component having the widest frequency bandwidth among the plurality of different spectral components;
A pixel value corresponding to a second spectral component obtained by removing a predetermined frequency band in the vicinity of a frequency at which no chromatic aberration occurs in the lens from the first spectral component;
Or
Corresponds to a third spectral component represented by a linear sum of a value obtained by multiplying the first spectral component by a first weighting factor and a value obtained by multiplying the second spectral component by a second weighting factor. An imaging method including a step of obtaining a contrast component of image data by an arithmetic process using pixel values. A computer-executable program that controls processing for capturing an image,
Controlling the acquisition of the optical signal that is incident through the lens and obtained for each pixel as a plurality of different spectral components through a predetermined color filter,
Control the conversion of the acquired optical signal into a digital signal;
Controlling the acquisition of a first pixel value corresponding to the optical signal at a predetermined pixel from the converted digital signal;
An interpolation process is performed using a pixel value that an adjacent pixel of the predetermined pixel has to calculate a second pixel value,
Of the first pixel value and the second pixel value,
A pixel value corresponding to a first spectral component having the widest frequency bandwidth among the plurality of different spectral components;
A pixel value obtained by removing a second spectral component corresponding to a predetermined frequency band in the vicinity of a frequency at which no chromatic aberration occurs in the lens from the first spectral component;
Or
Corresponds to a third spectral component represented by a linear sum of a value obtained by multiplying the first spectral component by a first weighting factor and a value obtained by multiplying the second spectral component by a second weighting factor. A program that causes a computer to execute a process including a step of obtaining a contrast component of image data by an arithmetic process using pixel values.   A recording medium on which the program according to claim 17 is recorded.

Images