JP2012142676A - Imaging device and image generation method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an imaging device, an image generation method, and the like capable of acquiring a high-resolution image from a low-resolution moving image.SOLUTION: An imaging device includes: an image sensor; a readout control section; an estimation calculation section; and an image output section. The image sensor includes a plurality of light receiving elements PD1 to PD4 and a filter section CF. The readout control section reads out light-receiving values from the plurality of the light receiving elements to acquire a low-resolution frame image. Based on the low-resolution frame image, the estimation calculation section estimates an estimated pixel value of which pixel pitch is smaller than that of the low-resolution frame image. The image output section outputs a high-resolution frame image having higher resolution than the low-resolution frame image based on the estimated pixel value. In the filter section CF, a plurality of filters c1 to c4 having different colors are arranged corresponding to one of the light receiving elements PD1.

Description

  The present invention relates to an imaging apparatus, an image generation method, and the like.

  Each light receiving element of the imaging element corresponds to a color filter of one color. For example, in a Bayer array image sensor, any one of a red filter, a green filter, and a blue filter corresponds to one light receiving element.

JP 2009-124621 A JP 2008-243037 A

  The inventor has developed a technique for generating a high-resolution image from a low-resolution image obtained by imaging. This technique is used, for example, to reduce the data amount of a captured image or to increase the resolution when displaying the data.

  Conventionally, as described above, an image obtained by an imaging device in which one color filter corresponds to one light receiving device is made to have high resolution. For example, Patent Documents 1 and 2 disclose a technique of capturing a low resolution image while performing pixel shift and generating a high resolution image from the captured low resolution image.

  According to some aspects of the present invention, it is possible to provide an imaging device, an image generation method, and the like that can acquire a high-resolution image from a low-resolution moving image.

  One embodiment of the present invention includes an imaging device having a plurality of light receiving elements and a filter unit, a read control unit that reads light reception values of the plurality of light receiving elements to obtain a low resolution frame image, and the low resolution frame image An estimation calculation unit that estimates an estimated pixel value having a pixel pitch smaller than the pixel pitch of the low-resolution frame image, and a higher resolution than the low-resolution frame image based on the estimated pixel value. An image output unit that outputs a resolution frame image, and the filter unit relates to an imaging device in which a plurality of filters of different colors are arranged as a filter corresponding to one light receiving element.

  According to one embodiment of the present invention, light transmitted through a plurality of filters of different colors is received by a light receiving element. Then, a low resolution frame image based on the received light value is acquired, an estimated pixel value is estimated based on the low resolution frame image, and a high resolution frame image is output based on the estimated pixel value. This makes it possible to estimate a high resolution image from a low resolution image. In addition, since the light that has passed through the plurality of filters is received by one light receiving element, it is possible to acquire a light reception value corresponding to the added value of the plurality of light reception values.

  In the aspect of the invention, each of the plurality of filters has a weighted transmittance, and the estimation calculation unit is configured to perform the estimation based on a light reception value obtained by the weighted transmittance. May be performed.

In the aspect of the invention, the filter unit includes a Bayer array filter as a filter corresponding to the one light receiving element, and the Bayer array filter includes a first green filter, a red filter, A blue filter and a second green filter, wherein the transmittance of the red filter and the blue filter is 1 / r (r is a real number of 1 or more) relative to the transmittance of the first green filter; The transmittance of the second green filter may be 1 / r ² with respect to the transmittance of the first green filter.

  In the aspect of the invention, the filter unit may include a color filter and an ND filter having the weighted transmittance.

  If it does in this way, the light which permeate | transmitted the several filter with which the transmittance | permeability was weighted can be received with one light receiving element. Accordingly, it is possible to obtain a higher resolution estimated pixel value as compared with the case where there is no weighting of the transmittance.

  In one embodiment of the present invention, the readout control unit samples a subject image by sequentially shifting pixels while superimposing pixels, performs each imaging operation with the imaging element while shifting the pixels, and performs each imaging operation. The light reception value of the pixel obtained by the above may be acquired as a low resolution frame image.

  In the aspect of the invention, the readout control unit may sequentially set the pixels to a first position and a second position next to the first position, perform the pixel shift, and perform the estimation. When the pixel at the first position and the pixel at the second position overlap, the arithmetic unit calculates a difference value between the light reception value of the pixel at the first position and the light reception value of the pixel at the second position. The estimated pixel value may be estimated based on the difference value.

  According to one embodiment of the present invention, pixels are sequentially shifted while superimposing pixels, and each imaging operation is performed by the imaging element while the pixels are shifted, whereby a light reception value is acquired and a low-resolution frame image is acquired. Then, an estimated pixel value is estimated based on a plurality of received light values obtained by sequentially shifting the pixels. At this time, the estimated pixel value is estimated based on the difference value of the light reception values of the pixels at the first and second positions to be superimposed. Thereby, it becomes possible to acquire a high resolution image from a low resolution moving image by a simple process.

  In the aspect of the invention, the estimation calculation unit may include a first intermediate pixel value that is a light reception value of a first light reception region obtained by removing a superposition region from the pixel at the first position, and the second A relational expression with a second intermediate pixel value that is a light receiving value of a second light receiving area obtained by excluding the superposed area from the pixel of the position is expressed using the difference value, and the first, A second intermediate pixel value may be estimated, and the estimated pixel value may be obtained using the estimated first intermediate pixel value.

  In this way, it is possible to estimate the intermediate pixel value from the received light value acquired while sequentially shifting the pixels while superimposing the pixels, and obtain the final estimated pixel value from the estimated intermediate pixel value. Thereby, pixel value estimation of a high resolution frame image can be simplified.

  In the aspect of the invention, the estimation calculation unit may use the intermediate pixel value of the intermediate pixel value pattern when the continuous intermediate pixel value including the first and second intermediate pixel values is an intermediate pixel value pattern. The relational expression between the pixels is expressed using the light reception value of the pixel, the intermediate pixel value pattern represented by the relational expression between the intermediate pixel values is compared with the light reception value of the pixel, and the similarity is evaluated, Based on the similarity evaluation result, each intermediate pixel value included in the intermediate pixel value pattern may be determined so that the similarity is the highest.

  In this way, the intermediate pixel value can be estimated based on a plurality of received light values acquired by pixel shifting while pixels are superimposed.

  In the aspect of the invention, the estimation calculation unit obtains an evaluation function representing an error between the intermediate pixel value pattern represented by a relational expression between intermediate pixel values and a light reception value of the pixel, and the evaluation function Each of the intermediate pixel values included in the intermediate pixel value pattern may be determined so that the value of.

  In this way, by determining the intermediate pixel value so that the value of the evaluation function is minimized, the intermediate pixel value is determined so that the similarity between the intermediate pixel value pattern and the received light value is the highest. it can.

  According to another aspect of the present invention, there is provided an image generation method in an image sensor in which the image sensor includes a plurality of light receiving elements and a filter unit, and the filter unit has different colors as filters corresponding to one light receiving element. When a plurality of filters are arranged, the light reception values of the plurality of light receiving elements are read to obtain a low resolution frame image, and based on the low resolution frame image, from the pixel pitch of the low resolution frame image The present invention relates to an image generation method that estimates an estimated pixel value with a smaller pixel pitch and outputs a high-resolution frame image having a higher resolution than the low-resolution frame image based on the estimated pixel value.

  According to still another aspect of the present invention, the subject image is sampled by sequentially shifting the pixels while superimposing the pixels, and each imaging operation is performed by the imaging element while the pixels are shifted, and obtained by each imaging operation. The light reception value of the pixel is acquired as a low-resolution frame image, the pixel is sequentially set to a first position and a second position next to the first position, and the pixel shift is performed. When the pixel at the first position and the pixel at the second position overlap, a difference value between the light reception value of the pixel at the first position and the light reception value of the pixel at the second position is obtained, and the difference value is The estimated pixel value may be estimated based on this.

2 is a configuration example of an image sensor of the present embodiment. 1 is a first detailed configuration example of an image sensor. The 2nd detailed structural example of an image pick-up element. The 3rd detailed structural example of an image pick-up element. 3 shows a detailed configuration example of an ND filter. The 4th detailed structural example of an image pick-up element. 2 is a basic configuration example of an imaging apparatus. Explanatory drawing of an estimation process block. FIG. 9A is an explanatory diagram of received light values, and FIG. 9B is an explanatory diagram of intermediate pixel values. Explanatory drawing of the estimation method of the intermediate | middle pixel value in a 1st estimation method. Explanatory drawing of the estimation method of the intermediate | middle pixel value in a 1st estimation method. Explanatory drawing of the estimation method of the intermediate | middle pixel value in a 1st estimation method. FIGS. 13A and 13B are explanatory diagrams of an estimation method of an estimated pixel value in the first estimation method. Explanatory drawing of the estimation method of the estimated pixel value in a 1st estimation method. Explanatory drawing of the estimation method of the estimated pixel value in a 1st estimation method. Explanatory drawing of the estimation method of the estimated pixel value in a 1st estimation method. Explanatory drawing of a 2nd estimation method. Explanatory drawing of a 2nd estimation method. Explanatory drawing of a 2nd estimation method. Explanatory drawing of a 2nd estimation method. Explanatory drawing of the search range setting in a 2nd estimation method. Outline | summary explanatory drawing of a 2nd estimation method. Explanatory drawing of a 3rd estimation method. Outline | summary explanatory drawing of a 3rd estimation method. Explanatory drawing of a 4th estimation process. An example of the configuration of a noise filter. Explanatory drawing of the 1st interpolation method of a light reception value. An example of a timing chart of interpolation processing. Explanatory drawing of the 2nd interpolation method of a light reception value. Explanatory drawing of the 2nd interpolation method of a light reception value. Explanatory drawing of the motion compensation with respect to a fusion frame. 1 is a first detailed configuration example of an imaging device and an image processing device. The 2nd detailed structural example of an imaging device and an image processing apparatus. FIG. 34A is an explanatory diagram of received light values in the fifth estimation method, and FIG. 34B is an explanatory diagram of intermediate pixel values in the fifth estimation method. Explanatory drawing of the estimation method of the intermediate | middle pixel value in a 5th estimation method. Explanatory drawing of the estimation method of the intermediate | middle pixel value in a 5th estimation method. Explanatory drawing of the estimation method of the intermediate | middle pixel value in a 5th estimation method. Explanatory drawing of the estimation method of the estimated pixel value in the 5th estimation method. Explanatory drawing of the estimation method of the intermediate | middle pixel value in a 6th estimation method. Explanatory drawing of the estimation method of the intermediate | middle pixel value in a 6th estimation method. Explanatory drawing of the estimation method of the intermediate | middle pixel value in a 6th estimation method. Explanatory drawing of the estimation method of the estimated pixel value in the 6th estimation method. Explanatory drawing of the estimation method of the estimated pixel value in the 6th estimation method. Explanatory drawing of the estimation method of the estimated pixel value in the 6th estimation method.

  Hereinafter, preferred embodiments of the present invention will be described in detail. The present embodiment described below does not unduly limit the contents of the present invention described in the claims, and all the configurations described in the present embodiment are indispensable as means for solving the present invention. Not necessarily.

1. Comparative Example First, a comparative example of this embodiment will be described. Among digital camera and video camera products, there are those in which a digital camera mainly for still image shooting has a moving image shooting function, and in which a video camera mainly for moving image shooting has a still image shooting function. Many of these cameras are used by switching between a still image shooting mode and a moving image shooting mode. Some of them enable high-definition still images to be shot at a high-speed frame rate comparable to that of moving image shooting, and high-speed continuous shooting in a short time is possible. If such a device is used, there is the convenience that a still image and a moving image can be shot with a single device.

  However, with these methods, there is a problem that it is difficult to obtain a high-quality still image without missing the shutter chance that many people demand. For example, in a method of instantaneously switching to a mode for shooting a high-quality still image during moving image shooting, there is a problem that the moving image is interrupted or a decisive moment has already been missed when the user notices.

  In order to solve the problem of not missing the photo opportunity, it is conceivable to capture all scenes by moving image shooting and obtain a decisive moment as a high-quality still image freely. In order to realize this method, it is necessary to be able to capture a high-definition image at a high frame rate.

  However, this method is not easy to realize. For example, in order to continuously capture images of 12 million pixels at 60 fps (fps: frame / second), an image sensor capable of performing ultra-high-speed imaging, a processing circuit for processing ultra-high-speed imaging data, and an ultra-high-speed data compression processing function Therefore, a recording means for recording enormous data is required. For this purpose, use of a plurality of image sensors, parallel processing, a large-scale memory, a high-capacity heat dissipation mechanism, and the like are required. However, this is impractical for consumer devices that require miniaturization and low cost. Although it is possible to realize a still image that is not high-definition with a resolution of about high-definition (2 million pixels) for moving image shooting, a resolution of about high-definition is insufficient as a still image.

  In addition, as a method of shooting a high frame rate moving image, a high-pixel image sensor capable of shooting a high-definition image with multiple pixels is used, and a low-resolution image is obtained by thinning-out readout of pixels or addition readout of adjacent pixels. A method realized by reducing data is conceivable. However, with this method, a high-definition image cannot be taken at a high frame rate.

  In order to solve this problem, it is necessary to obtain a high resolution image from a low resolution image captured at a high frame rate. As a technique for obtaining a high-resolution image from a low-resolution image, for example, a technique for generating a high-resolution image by performing so-called super-resolution processing on a low-resolution image captured by pixel shift can be considered.

  For example, as the technique, a technique using addition reading can be considered. That is, after the low-resolution image is read out while sequentially shifting the position, a high-definition image is temporarily assumed based on the plurality of position-shifted images. Then, the assumed image is degraded to generate a low resolution image, which is compared with the original low resolution image, the high definition image is deformed so that the difference is minimized, and the high definition image is estimated. As this super-resolution processing, an ML (Maximum-Likelihood) method, a MAP (Maximum A Postterior) method, a POCS (Projection Onto Convex Set) method, an IBP (Iterative Back Projection) method, and the like are known.

  Further, as a technique using super-resolution processing, there is a technique disclosed in Patent Document 1 described above. In this method, low-resolution images that have been pixel-shifted during moving image shooting are sequentially shot in time series, and a plurality of low-resolution images are combined to assume a high-resolution image. Then, the above-described super-resolution processing is performed on the assumed high-resolution image, and a high-resolution (likely) high-resolution image is estimated.

  However, in these methods, a general super-resolution process is used in which the estimation accuracy is increased by an iterative operation that frequently uses a two-dimensional filter. Therefore, the scale of processing becomes very large and the processing time increases, and there is a problem that it is difficult to apply to devices with processing capacity and cost limitations. For example, when applied to a small portable imaging device such as a digital camera, the scale of the processing circuit becomes large, and problems such as an increase in power consumption, generation of a large amount of heat, and a significant increase in cost occur.

  Further, Patent Document 2 described above discloses a technique for generating a high resolution image using a plurality of pixel-shifted low resolution images. In this method, a temporary pixel constituting a high-resolution image to be obtained is set as a sub-pixel, and the pixel value of the sub-pixel is set so that the average value of the sub-pixel matches the pixel value of the captured low-resolution image. Is estimated. In this pixel value estimation, initial values of a plurality of subpixels are set, pixel values of subpixels excluding the subpixels to be calculated are subtracted from pixel values of the low resolution image, and pixel values are obtained sequentially. Applies to pixels.

  However, with this method, there is a problem that the estimation error becomes very large unless the initial value is specified. In this method, in order to set the initial value, the pixel value change of the sub-pixel is small, and a portion where the average value thereof and the low-resolution pixel value covering them are substantially equal is found from the image. Therefore, it is difficult to estimate the initial value unless a part suitable for setting the initial value is found from the captured image. In addition, it is necessary to search for an appropriate part for setting the initial value.

2. In this embodiment, an image is picked up by an image pickup device in which a plurality of color filters having different colors are arranged in one pixel. Then, a low-resolution moving image with a high frame rate is picked up by pixel shift (pixel shifting), and a high-resolution image is generated from the low-resolution moving image by a simple pixel estimation method. Using the generated high-resolution image, a high-resolution still image at an arbitrary timing in the moving image is acquired, or a high-resolution moving image with a high frame rate is acquired.

  Note that a general color image is not picked up by the image pickup device of the present embodiment, but in this specification, a subject image picked up by the image pickup device of the present embodiment is appropriately referred to as an image or a moving image.

  Here, a case where pixel shift and estimation processing described later with reference to FIGS. 7 to 44 will be described as an example, but the present embodiment is not limited to this. For example, various known high resolution techniques may be applied to the low resolution image acquired by the image sensor of the present embodiment.

  FIG. 1 shows a configuration example of the image sensor of the present embodiment. FIG. 1 is a schematic diagram when the imaging device is viewed in plan from the filter unit CF side. The imaging element includes a filter portion CF and light receiving elements PD1 to PD4 (light receiving areas).

  The light receiving elements PD1 to PD4 are elements that perform photoelectric conversion of light transmitted through the filter portion CF, and are, for example, photodiodes or phototransistors. For example, the light receiving elements PD1 to PD4 are arranged in a matrix along the horizontal scanning direction and the vertical scanning direction. Hereinafter, the light receiving element PD1 will be described as an example, but the same applies to other light receiving elements.

The filter unit corresponding to the light receiving element PD1 is composed of a plurality of color filters cf1 to cf4. The filters cf1 to cf4 may be all different colors or may be partially the same color. The filters cf1 to cf4 and the light receiving element PD1 constitute pixels of the image sensor. As described below in FIG. 7, in the present embodiment is performed mechanically pixel shift, 9 as described later in (A), receiving values of the pixels while shifting pixels _{a 00, a 10, a 11} , a ₀₁ is acquired sequentially.

3. Detailed Configuration Example of Image Sensor FIG. 2 shows a first detailed configuration example of the image sensor. H1 in FIG. 2 is a plan view of one pixel, and H2 and H3 are side views. The imaging element shown in FIG. 2 includes a first green filter gr, a red filter r, a blue filter b, a second green filter gb, and a light receiving element PD1.

The filters gr, r, b, and gb are primary color filters that transmit G (green), R (red), B (blue), and G (green), respectively. If the received light values of the transmitted light of the filters gr, r, b, gb are v ₀₀ , v ₁₀ , v ₀₁ , v ₁₁ for convenience, the received light value of the light receiving element PD1 is a ₀₀ = v ₀₀ + v ₁₀ + v ₀₁ + v ₁₁ It is expressed.

When the estimation process described later with reference to FIGS. 7 to 44 is performed, an image of 12 megapixels is obtained using, for example, a 3 megapixel image sensor. The above-described received light values v ₀₀ , v ₁₀ , v ₀₁ , v ₁₁ correspond to pixel values of a 12 megapixel image. Specifically, the addition of the received light values v ₀₀ , v ₁₀ , v ₀₁ , v ₁₁ corresponds to the addition of intermediate pixel values and estimated pixel values, which will be described later with the following equations (1) and (5). That is, since the light that has passed through the plurality of filters gr, r, b, and gb is received by the common light receiving element PD1, a plurality of light reception values are added within the pixel.

  In the present embodiment, Gr, R, B, and Gb mosaic images are obtained by an estimation process that will be described later. By performing demosaicing processing on this mosaic image, a high-resolution color image can be obtained.

  According to this embodiment, since the light receiving elements are made common to a plurality of filters, the aperture ratio can be relatively improved by increasing the size of pixels when the process rule (wiring width) is the same. Further, the sensitivity can be improved by improving the aperture ratio. Moreover, the yield at the time of manufacturing a sensor can be improved because a pixel becomes large.

FIG. 3 shows a second detailed configuration example of the image sensor. The imaging element shown in FIG. 3 includes a first green filter g ₁ , a red filter r _{1 / r} , a blue filter b _{1 / r} , a second green filter gb _{1 / r} ² , and a light receiving element PD ₁ .

The filters g ₁ , r _{1 / r} , b _{1 / r} , and gb _{1 / r} ² have individual transmittances. Specifically, the transmittances of the filters r _{1 / r} and b _{1 / r} are weighted 1 / r (r is a real number satisfying r ≧ 1) with respect to the transmittance of the filter gb ₁ . Transmittance of the filter _{gb 1 /} ^{r 2} is the weighting of 1 / ^{r 2} with respect to the transmittance of the filter gb _1. In other words, the light-receiving values of the light receiving element PD1 is expressed as _{a 00 = v 00 + (1} / r) v 10 + (1 / r) v 01 + (1 / r 2) v 11. For example, the weighting of the transmittance can be realized by changing the color density of the color filter and the thickness of the color filter.

According to the present embodiment, it is possible to estimate a higher resolution estimated pixel value. Specifically, the pixel aperture can be considered as a window function for convolution in sampling of the subject image. When the above weighting is performed, the window function has more high frequency components than when the weighting is not performed (r = 1). For this reason, the received light value a ₀₀ includes more high-frequency components, and the pixel value estimated from the received light value a ₀₀ also has higher resolution.

  Further, according to the present embodiment, since the weighting coefficient for weighted addition can be set by the transmittance of the color filter, the circuits and software for weighted addition can be reduced. Specifically, when adding an electrical signal outside the light receiving element, it is necessary to set the gain of the amplifier in accordance with the weighting in the analog method, and calculations such as division and bit shift are performed in the digital method. There is a need. As a result, the circuit scale increases and the calculation speed decreases. In this regard, in this embodiment, since weighting is set by the transmittance of each color filter, such a problem does not occur.

  FIG. 4 shows a third detailed configuration example of the image sensor. The imaging element shown in FIG. 4 includes a first green filter gr, a red filter r, a blue filter b, a second green filter gb, an ND (Neutral Density) filter NF, and a light receiving element PD1.

The ND filter NF is a filter that exhibits a relatively constant transmittance over the entire wavelength range (for example, the visible light band). In the present embodiment, the above weighting is performed by the ND filter NF. FIG. 5 shows a detailed configuration example of the ND filter NF. The ND filter NF includes first to fourth filters nd1 to nd4. The filters nd1 to nd4 overlap with the filters gr, r, b, and gb, respectively, in plan view. The transmittances of the filters gr, r, b, and gb are all weighted 1. The transmittances of the filters nd2 and nd3 are weighted by 1 / r with respect to the transmittance of the filter nd1. The transmittance of the filter nd4 is weighted by 1 / r ² with respect to the transmittance of the filter nd1.

  In the present embodiment, the filter NF is not limited to the ND filter, and may be any filter that does not aim at wavelength selection.

  According to the present embodiment, the wavelength selection RGB filters gr, r, b, and gb can be manufactured by the same method as the conventional one. Therefore, it is not necessary to prepare a color filter material or process in order to perform weighted addition. Moreover, since the ND filter for the purpose of controlling the transmittance is a widely known and used filter, it is easy to realize weighting.

  FIG. 6 shows a fourth detailed configuration example of the image sensor. The imaging element shown in FIG. 6 includes a cyan filter c, a magenta filter m, a yellow filter y, a green filter g, and a light receiving element PD1.

  The filters c, m, y, and g are complementary color filters that transmit C (cyan), M (magenta), Y (yellow), and G (green). In the present embodiment, C, M, Y, and G mosaic images are obtained by an estimation process described later. A high-resolution color image can be obtained by performing RGB conversion and demosaicing processing on the mosaic image. It is also possible to weight the transmittance as described above. That is, the transmittance of the color filter may be adjusted, or the transmittance may be adjusted by adding an ND filter.

  As described above, the imaging apparatus according to the present embodiment includes the imaging element, the readout control unit, the estimation calculation unit, and the image output unit. As illustrated in FIG. 1, the imaging element includes a plurality of light receiving elements PD1 to PD4 and a filter unit CF. The read control unit reads the light reception values of the plurality of light receiving elements PD1 to PD4 and acquires a low resolution frame image. The estimation calculation unit estimates an estimated pixel value having a pixel pitch smaller than the pixel pitch of the low resolution frame image based on the low resolution frame image. The image output unit outputs a high-resolution frame image having a higher resolution than the low-resolution frame image based on the estimated pixel value. In this case, a plurality of filters cf1 to cf4 of different colors are arranged in the filter unit CF as filters corresponding to one light receiving element PD1.

  In the present embodiment, the read control unit corresponds to the superposition shift sampling unit 140 described later with reference to FIG. The estimation calculation unit corresponds to a pixel value estimation calculation unit 230 described later with reference to FIG. The image output unit corresponds to an anti-aliasing filter 240 described later with reference to FIG. Note that the filter unit CF and the filters cf1 to cf4 may be configured only by color filters, and may include color filters and other filters.

  According to this embodiment, the light transmitted through the plurality of filters cf1 to cf4 of different colors can be received by one light receiving element PD1. As a result, the aperture ratio and sensor sensitivity can be improved as described above. Further, according to the present embodiment, it is possible to capture a low resolution image and obtain a high resolution image from the low resolution image. That is, a high-resolution image can be estimated by performing an estimation process described later on the received light value. Thereby, since the data amount of captured image data can be reduced, it is possible to save storage and reduce the amount of communication data.

  In the present embodiment, as shown in FIG. 3, each of the plurality of filters has a weighted transmittance. As will be described later with reference to FIGS. 39 to 44, the estimation calculation unit performs estimation based on the received light value obtained from the weighted transmittance.

Specifically, the filter unit CF includes a Bayer array filter as a filter corresponding to one light receiving element PD1. The Bayer array filter includes a first green filter g ₁ , a red filter r _{1 / r} , a blue filter b _{1 / r,} and a second green filter gb _{1 / r} ² . The transmittance of the red filter r _{1 / r} and the blue filter b _{1 / r} is 1 / r with respect to the transmittance of the first green filter g ₁ . The transmittance of the second green filter gb _{1 / r} ² is 1 / r ² with respect to the transmittance of the first green filter g ₁ .

  Alternatively, as illustrated in FIG. 4, the filter unit CF may include color filters gr, r, b, and gb and ND filters nd1 to nd4 having weighted transmittances.

  In this way, light that has passed through a plurality of filters with weighted transmittance can be received by one light receiving element. Accordingly, as described above, it is possible to perform weighted addition within a pixel, and it is possible to increase the resolution of the estimated image compared to the case where there is no weighting. In addition, when the ND filter is used, the development load can be reduced because the conventional color filter can be used.

4). Pixel Shift Hereinafter, the estimation process performed by the present embodiment will be described in detail. First, the pixel shift performed by the present embodiment will be described with reference to FIG. FIG. 7 schematically illustrates a basic configuration example of the imaging apparatus. This imaging apparatus includes a lens 410 (imaging optical system in a broad sense) and an imaging element 420 (sensor, imager).

  The lens 410 forms an image of the subject Obj on the light receiving surface of the image sensor 420. As shown in C1, pixel shift control is performed on the lens 410 by a lens driving unit (not shown). Here, the pixel pitch of the image sensor 420 is assumed to be p. Then, the lens 410 is shifted by a shift amount p / 2 (shift amount s <p in a broad sense) in a direction along the x axis (or y axis) orthogonal to the optical axis. At this time, as the lens 410 is shifted by p / 2, the subject image on the light receiving surface is also shifted by the shift amount p / 2.

  The image sensor 420 captures an image of the subject Obj formed by the lens 410. Specifically, imaging is performed for each shift operation of the lens 410, and the subject image shifted by p / 2 is sampled. Thus, in the present embodiment, a low-resolution frame image having a pixel pitch p shifted by p / 2 by pixel shift is acquired.

  In the above configuration example, the case where the lens 410 is shifted by the shift amount p / 2 has been described as an example. However, in the present embodiment, the image sensor 420 may be shifted by the shift amount p / 2.

5. First Estimation Method In this embodiment, the pixel value of the pixel pitch p / 2 is estimated from the low resolution frame image of the pixel pitch p acquired as described above, and a high resolution frame image is generated. A method for estimating the pixel value of the pixel pitch p / 2 will be described with reference to FIGS. In the present specification, for the sake of simplicity, a pixel value acquired by a pixel of the image sensor is referred to as a received light value, and a pixel value obtained by estimation is referred to as an estimated pixel value.

  FIG. 8 is an explanatory diagram of an estimation processing block used for pixel estimation. In FIG. 8, pixels of the low-resolution frame image are indicated by solid-line squares, pixel positions in the horizontal direction are indicated by i, and pixel positions in the vertical direction are indicated by j (i and j are natural numbers).

As shown in FIG. 8, in this embodiment, estimation processing blocks Bk ₀₀ , Bk ₁₀ ,... With m × n pixels as one block are set. Then, pixel value estimation processing described later is performed for each estimation processing block. By this estimation process, an estimated pixel value from m × n pixels to 2m × 2n pixels is estimated. In the following, for the sake of simplicity, a case where 2 × 2 pixels are estimated from m × n = 1 × 1 pixels will be described as an example.

FIG. 9A and FIG. 9B are explanatory diagrams of received light values and intermediate pixel values used for estimation processing. As shown in FIG. 9A, the light receiving values a ₀₀ , a ₁₀ , a ₁₁ , a ₀₁ (pixel values) are obtained by shifting the pixels (light receiving units) with the pixel pitch p of the image sensor by p / 2. Are acquired sequentially. In the present embodiment, the estimated pixel value of the pixel pitch p / 2 is estimated from the four received light values. The light reception values a _{00 to} a ₁₁ may be the light reception values themselves acquired by imaging, or may be light reception values interpolated by time axis interpolation described later.

At this time, as shown in FIG. 9B, intermediate pixel values b _{00 to} b ₂₁ (intermediate estimated pixel values) having a horizontal pixel pitch of p / 2 are estimated from the received light values a _{00 to} a ₁₁ . Then, from the intermediate pixel values b _{00 to} b ₂₁ , an estimated pixel value having a horizontal / vertical pixel pitch of p / 2 is finally estimated.

This intermediate pixel value estimation method will be described by taking the intermediate pixel values b _{00 to} b ₂₀ in the first horizontal row shown in FIG. 10 as an example. The relationship of the following formula (1) is established between the light reception value and the intermediate pixel value.
a ₀₀ = b ₀₀ + b ₁₀ ,
a ₁₀ = b ₁₀ + b ₂₀ (1)

When the above equation (1) is transformed with b ₀₀ as an unknown (initial value, reference value), the intermediate pixel values b ₁₀ and b ₂₀ can be expressed as a function of b ₀₀ as shown in the following equation (2).
b ₀₀ = (unknown number),
b ₁₀ = a ₀₀ −b ₀₀ ,
b ₂₀ = b ₀₀ + δi ₀ = b ₀₀ + (a ₁₀ −a ₀₀ ) (2)

Here, as shown in the following equation (3), δi ₀ is a difference value between the received light values one shift away, and corresponds to the difference value between the intermediate pixel values b ₂₀ and b ₀₀ .
δi ₀ = a ₁₀ −a ₀₀
= (B ₁₀ + b ₂₀ ) − (b ₀₀ + b ₁₀ )
= B ₂₀ -b ₀₀ (3)

In this way, a combination pattern of high-definition intermediate pixel values {b ₀₀ , b ₁₀ , b ₂₀ } with b ₀₀ as an unknown is obtained. To determine the value of the intermediate pixel values represented as a function of the b _00, it is necessary to determine the unknown b _00.

As shown in FIG. 11, in the present embodiment, a pattern {a ₀₀ , a ₁₀ } based on pixel values detected by superposition shift sampling (pixel shift) and a pattern {b ₀₀ , b ₁₀ , b ₂₀ } based on intermediate pixel values are used. Compare Then, an unknown number b ₀₀ that minimizes the error is derived, and the derived unknown number b ₀₀ is set as the final intermediate pixel value b ₀₀ .

Specifically, as shown in the following equation (4) represents the evaluation function Ej of the error as a function of unknown b _00. Then, as shown in FIG. 12, an unknown number b ₀₀ = α that minimizes the evaluation function Ej (minimum value) is obtained in an exploratory manner (least square method).

As shown in the above equation (4), in the present embodiment, error evaluation is performed on the average value of the intermediate pixel values and the pattern {a ₀₀ , a ₁₀ } having a low frequency component. Thus, possible to prevent the pattern including many high frequency components as the estimated solution of the intermediate pixel values _{{b 00, b 10, b} 20} is derived. That is, even if the estimation of the unknown is inaccurate, an image including a large amount of low frequency components is generated. For this reason, it is possible to suppress generation of a pattern including an error in a high frequency component in which unnaturalness is more easily emphasized than in a low frequency component, and the natural appearance of the image is not lost. Thereby, rational pixel estimation is possible for a natural image having a small high frequency component compared to a low frequency component.

In this way, the value of the intermediate pixel value b ₀₀ is estimated, and the estimated value of b ₀₀ is substituted into the above equation (2) to determine the values of the intermediate pixel values b ₁₀ and b ₂₀ . Similarly, the intermediate pixel values b _{01 to} b ₂₁ in the second row are estimated using b ₀₁ as an unknown quantity.

Next, a method for _{obtaining the} final estimated pixel value v _ij using the estimated intermediate pixel value b _ij will be described. FIGS. 13A and 13B schematically illustrate the intermediate pixel value and the estimated pixel value. As shown in FIG. 13A, two columns of intermediate pixel values b _{00 to} b ₁₁ are used among the three columns of intermediate pixel values b _{00 to} b ₂₁ estimated by the above-described method. Then, as shown in FIG. 13B, estimated pixel values v _{00 to} v ₁₂ of the final pixel pitch p / 2 are estimated from the intermediate pixel values b _{00 to} b ₁₁ .

Hereinafter, in order to simplify the description, the pixel values v _{00 to} v ₀₂ in the first column shown in FIG. 14 will be described as an example. The pixel values v _{00 to} v ₀₂ are estimated by a method similar to the above-described intermediate pixel value estimation method. Specifically, the intermediate pixel values b ₀₀ and b ₀₁ are equivalent to values obtained by superimposing and sampling the pixel values v _{00 to} v ₀₂ while shifting the pixel values v _{00 to} v ₀₂ in units of 2 pixels in the vertical direction. Therefore, the relationship of the following formula (5) is established between the intermediate pixel value and the estimated pixel value.
b ₀₀ = v ₀₀ + v ₀₁ ,
b ₀₁ = v ₀₁ + v ₀₂ (5)

As shown in the following equation (6), the pixel values v ₀₁ and v ₀₂ can be expressed as a function of the unknown v ₀₀ .
v ₀₀ = (unknown number),
v ₀₁ = b ₀₀ −v ₀₀ ,
v ₀₂ = v ₀₀ + δj ₀ = v ₀₀ + (b ₀₁ −b ₀₀ ) (6)

Here, as shown in the following equation (7), δj ₀ is a difference value between the intermediate pixel values separated by one shift, and corresponds to the difference value between the pixel values v ₀₂ and v ₀₀ .
δi ₀ = b ₀₁ −b ₀₀
= (V ₀₁ + v ₀₂ )-(v ₀₀ + v ₀₁ )
= V ₀₂ -v ₀₀ (7)

As illustrated in FIG. 15, an unknown number v ₀₀ that minimizes the error between the pattern {b ₀₀ , b ₁₀ } based on the intermediate pixel value and the pattern {v ₀₀ , v ₀₁ , v ₀₂ } based on the estimated pixel value is derived. That is, as shown in the following equation (8), the error is expressed by an evaluation function Ei, and as shown in FIG. 16, an unknown number v ₀₀ = β that minimizes the evaluation function Ei is obtained in a search manner.

Then, the pixel values v _{10 to} v ₁₂ in the second column are obtained by the same process, and final estimated pixel values v ₀₀ , v ₀₁ , v ₁₀ , and v ₁₁ are determined. In the present embodiment, an appropriate noise reduction process may be performed on the image data composed of the final estimated pixel values to obtain a display image.

Here, although the case where the unknown numbers b ₀₀ and v ₀₀ are obtained in an exploratory manner has been described in the above-described embodiment, the unknown numbers b ₀₀ and v ₀₀ may be directly obtained in the present embodiment. That is, the error expression Ej shown in Equation (4) Since a quadratic function formula of b _00, are possible wherein deformation in the form of the following equation (9). Therefore, the minimum value α of b ₀₀ that minimizes Ej can be directly obtained. The minimum value β of v ₀₀ is obtained in the same manner.
Ej = (b ₀₀ −α) ² + ξ (9)

  As described above, in the method of switching between the still image shooting mode and the moving image shooting mode, there is a problem that when the user notices a photo opportunity, the decisive moment is often missed. In addition, in the method of synthesizing a high-resolution still image from a low-resolution moving image by super-resolution processing, since the super-resolution processing is a heavy load processing, there is a problem that the scale of the processing circuit increases. is there.

  In this regard, according to the present embodiment, the subject image is sampled by sequentially shifting the pixels while superimposing the pixels, and each imaging operation is performed by the imaging device while the pixels are shifted, and light reception of the pixels obtained by each imaging operation is performed. The value is acquired as a low resolution frame image. The acquired low-resolution frame image is stored, and an estimated pixel value having a pixel pitch smaller than the pixel pitch of the low-resolution frame image is estimated based on the plurality of stored low-resolution frame images. Based on the estimated pixel value, a high-resolution frame image having a higher resolution than the low-resolution frame image is output. At this time, the pixels to be pixel-shifted are sequentially set to the first position and the second position next to the first position. The pixels at the first and second positions overlap. And the difference value of the light reception value of the pixel of the 1st, 2nd position is calculated | required, and an estimated pixel value is estimated based on the difference value.

For example, as shown in FIG. 7, the pixel pitch p of the image sensor is shifted by p / 2 in the horizontal direction or the vertical direction. As shown in FIG. 9A, in this pixel shift, pixels are superimposed by p / 2. In the first to fourth frames, low-resolution frame images including the received light values a ₀₀ , a ₁₀ , a ₁₁ , a ₀₁ are sequentially acquired. These low-resolution frame images are acquired by, for example, the imaging device 10 shown in FIG. 32, input to the image processing device 20, and stored in the frame buffer 210 (storage unit). Then, the estimated pixel values v _{00 to} v ₁₁ having a pitch p / 2 smaller than the pixel pitch p are estimated by the pixel value estimation calculation unit 230 (estimation calculation unit) and increased by the anti-aliasing filter 240 (image output unit). A resolution frame image is output. At this time, as shown in FIG. 9A, in the first and second frames, pixels are set at the first and second positions where the received light values a ₀₀ and a ₁₀ are acquired. These pixels are overlapped in a region including the estimated pixels v ₁₀ and v ₁₁ . Then, as described above in FIG. 10 and the like, the difference value δi ₀ = a ₁₀ −a ₀₀ is obtained, and the estimated pixel values v _{00 to} v ₁₁ are estimated based on the difference value δi ₀ .

Thereby, it becomes possible to acquire a high resolution image from a moving image with a simple process. Specifically, the estimation process can be simplified by estimating the intermediate pixel value from the difference value δi ₀ . In addition, since the high-resolution still image can be generated at any timing of the low-resolution moving image, the user can easily obtain the high-resolution still image at the decisive moment. Also, during shooting, a low-resolution video (eg 3 megapixels) is acquired to shoot at a high frame rate (eg 60 frames), and a high-resolution still image (12 megapixels) or a high-definition video is displayed as needed. it can. In the case of digital zoom, a high-definition zoom image can be generated by compensating for the decrease in the number of pixels by estimation processing.

  More specifically, in the present embodiment, the first intermediate pixel value, which is the light reception value of the first light receiving region obtained by removing the overlap region from the pixel at the first position, and the overlap region from the pixel at the second position. A relational expression with the second intermediate pixel value, which is the light reception value of the second light receiving region excluding, is expressed using the difference value. Then, the first and second intermediate pixel values are estimated using the relational expression, and the pixel value of each pixel of the pixel is obtained using the estimated first intermediate pixel value.

For example, as described above with reference to FIGS. 9A and 9B, the first light receiving region corresponds to a region _from which the first intermediate pixel value b ₀₀ is acquired. The second light receiving region corresponds to a region for acquiring the second intermediate pixel value b _20. Then, as described above with reference to FIG. 10, the relational expression b ₂₀ = b ₀₀ + δi ₀ is obtained using the difference value δi ₀ . As described above in FIG. 11, the estimated unknown b _00, b ₂₀ are estimated using the relational expression. As described above with reference to FIG. 13A and the like, estimated pixel values v ₀₀ and v ₀₁ are obtained using b ₀₀ .

  In this way, the estimation process of the high resolution image can be simplified by once estimating the intermediate pixel value from the superposed shifted light reception value and obtaining the estimated pixel value from the superposed shifted intermediate pixel value. For example, compared with the above-described comparative example, complicated processing such as a repetitive calculation of a two-dimensional filter (Patent Document 1) and a search for an appropriate part for setting an initial value (Patent Document 2) is not required.

  Further, in the present embodiment, when successive intermediate pixel values including the first and second intermediate pixel values are used as the intermediate pixel value pattern, the relationship between the intermediate pixel values of the intermediate pixel value pattern The expression is expressed using the light reception value of the pixel. Then, the similarity is evaluated by comparing the intermediate pixel value pattern and the light reception value pattern (light reception value of the pixel), and based on the evaluation result, the intermediate pixel value of the intermediate pixel value pattern is set so that the similarity is the highest. The value of is determined.

For example, as described above with reference to FIG. 11, successive intermediate pixel values {b ₀₀ , b ₁₀ , b ₂₀ } correspond to the intermediate pixel value pattern, and {b ₀₀ , b ₁₀ , b ₂₀ } is expressed using received light values a ₀₀ and a ₁₀ . Then, the intermediate pixel value pattern {b ₀₀ , b ₁₀ , b ₂₀ } is compared with the received light value pattern {a ₀₀ , a ₁₀ }, and {b ₀₀ , so that the similarity represented by the evaluation function Ej is the highest. The values of b ₁₀ , b ₂₀ } are determined.

  Here, the intermediate pixel value pattern is a set of intermediate pixel values continuous in the horizontal direction (or vertical direction). The received light value pattern is a set of received light values that are continuous in the horizontal direction (or vertical direction) in a plurality of low-resolution frame images used for the estimation process.

  In this way, the intermediate pixel value can be estimated based on a plurality of received light values acquired by pixel shifting while pixels are superimposed.

  More specifically, in this embodiment, an evaluation function representing an error between the intermediate pixel value pattern and the received light value pattern expressed by a relational expression between intermediate pixel values is obtained. Then, the value of the intermediate pixel value of the intermediate pixel value pattern is determined so that the value of the evaluation function is minimized.

For example, the intermediate pixel value pattern {b ₀₀ , b ₁₀ , b ₂₀ } is expressed as a function of the unknown b ₀₀ as described above in the above equation (4) or the like, and the intermediate pixel value pattern {b ₀₀ , b ₁₀ , b _20} and the light-receiving value pattern _{{a 00,} the error between _{a 10}} is expressed by the evaluation function Ej. As described above in FIG. 12, evaluation value of the function Ej unknowns _{b 00} = _α (initial value) as the minimum sought, the value of _b 00 _{~b 20} by _{b 00} obtained is determined.

  In this way, the value of the intermediate pixel value can be estimated by expressing the error by the evaluation function and obtaining the intermediate pixel value corresponding to the minimum value of the evaluation function. For example, as described above, the initial value of the intermediate pixel estimation can be set with a simple process by obtaining the unknown using the least square method. Thereby, compared with the above-mentioned comparative example (patent document 2), the search of the image part suitable for initial value setting can be made unnecessary.

6). Second Estimation Method As described above with reference to FIG. 9B and the like, in the estimation processing block set to the intermediate pixel values b ₀₀ and b ₁₀ , the intermediate pixel value b _{20 is} estimated. This b ₂₀ corresponds to an unknown (initial variable) in the next estimation processing block set to the intermediate pixel values b ₂₀ and b ₃₀ . In the present embodiment, it is possible to speed up the estimation of the unknown b ₂₀ to be estimated next by using the already estimated intermediate pixel value b ₂₀ .

A second estimation method capable of speeding up the estimation of unknowns will be described with reference to FIGS. Hereinafter, the estimation of the intermediate pixel value (b ₂₀ etc.) will be described as an example, but the estimation pixel (v ₀₂ etc.) can be similarly estimated.

As shown in FIG. 17, in the first row in the horizontal direction detected by superposition shift sampling, the following formula (10) is established when attention is paid to the received light values a ₀₀ and a ₁₀ .
a ₀₀ = b ₀₀ + b ₁₀ ,
a ₁₀ = b ₁₀ + b ₂₀ (10)

The following equation (11) is established with b ₀₀ as an unknown.
b ₀₀ = (unknown number),
b ₁₀ = a ₀₀ −b ₀₀ ,
b ₂₀ = b ₀₀ + δi ₀ = b ₀₀ + (a ₁₀ −a ₀₀ ) (11)

Here, δi ₀ is expressed by the following formula (12).
δi ₀ = a ₁₀ −a ₀₀
= (B ₁₀ + b ₂₀ ) − (b ₀₀ + b ₁₀ )
= B ₂₀ -b ₀₀ (12)

As shown in FIG. 18, the received light value pattern {a ₀₀ , a ₁₀ } is compared with the intermediate pixel value pattern φ0 = {b ₀₀ , b ₁₀ , b ₂₀ }. Specifically, as shown in the following equation (13) represents the evaluation function Ej of the error as a function of unknown b _00. Then, as shown in FIG. 19, an unknown number b ₀₀ = α1 that minimizes the evaluation function Ej is obtained in a search manner, and the intermediate pixel values b ₁₀ and b ₂₀ are determined by the above equation (11).

Similarly, the next unknown b ₂₀ (initial variable) in the horizontal direction is obtained, and the intermediate pixel value pattern φ2 = {b ₂₀ , b ₃₀ , b ₄₀ } is obtained. That is, as shown in FIG. 20, when attention is paid to the received light values a ₂₀ and a ₃₀ , the following expression (14) is established.
a ₂₀ = b ₂₀ + b ₃₀ ,
a ₃₀ = b ₃₀ + b ₄₀ (14)

The following equation (15) is established with b ₂₀ as an unknown.
b ₂₀ = (unknown number),
b ₃₀ = a ₂₀ −b ₂₀ ,
b ₄₀ = b ₂₀ + δi ₂ = b ₂₀ + (a ₃₀ −a ₂₀ ) (15)

Here, δi ₂ is the difference value of the received light value one shift away, and is expressed by the following equation (16).
δi ₂ = a ₃₀ -a _{20
= (B 30 + b 40)} - (b 20 + b 30)
= B ₄₀ -b ₂₀ (16)

Next, as shown in FIG. 20, the received light value pattern {a ₂₀ , a ₃₀ } is compared with the intermediate pixel value pattern φ2 = {b ₂₀ , b ₃₀ , b ₄₀ }.

Specifically, as shown in the following equation (17) represents the evaluation function Ej of the error as a function of unknown b _20. At this time, by utilizing the fact that the value of _{b 20} by the estimation of the intermediate pixel value _b 00 _{~b 20} has already been obtained, as shown in FIG. 21, the _{b 20} already obtained in (= α1 + δi ₀₎ The neighborhood value range is set as the search range. Then, the unknown b ₂₀ is changed within the search range to obtain b ₀₀ = α2 that minimizes the evaluation function Ej. In this way, the number of operations of the following equation (17) can be significantly reduced, and the estimation process can be speeded up.

  According to the present embodiment, as shown in FIG. 22, the first pixel (first light receiving unit) and the second pixel (second light receiving unit) are adjacent pixels. Then, the first and second pixels are sequentially set to the first position and the second position next to the first position by the pixel shift. The first pixels at the first and second positions overlap in the first overlapping region, and the second pixels at the first and second positions overlap in the second overlapping region.

At this time, a region obtained by removing the first overlapping region from the first pixel at the first position is the first light receiving region. Further, a region obtained by removing the first overlapping region from the first pixel at the second position is the second light receiving region. Then, the first and second intermediate pixel values (for example, b ₀₀ , b ₂₀ ) that are the light receiving values of the first and second light receiving regions are estimated (b ₀₀ = α1, b ₂₀ = α1 + δi ₀ ). .

A region obtained by removing the second overlapping region from the second pixel at the first position is a third light receiving region. A region obtained by removing the second overlapping region from the second pixel at the second position is the fourth light receiving region. Then, the third intermediate pixel value (b ₂₀ ) that is the light reception value of the third light receiving region is an unknown number, and the fourth intermediate pixel value (b ₄₀ ) that is the fourth light reception value is an unknown number (b ₂₀ ) Using a relational expression (b ₄₀ = b ₂₀ + δi ₂ ).

The third light receiving region and the second light receiving region are the same region, and based on the second intermediate pixel value (b ₂₀ = α1 + δi ₀ ) obtained previously as the intermediate pixel value of the second light receiving region, A search range for an unknown (b ₂₀ ) that is the third intermediate pixel value is set. In the set search range, the unknown (b ₂₀ ) is exploratoryly calculated, and the third intermediate pixel value (b ₂₀ = α2) is estimated.

This makes it possible to estimate the intermediate pixel value at high speed. That is, the search range of the unknown (b ₂₀ ) estimated in the next estimation processing block can be set based on the previously determined second intermediate pixel value (b ₂₀ = α1 + δi ₀ ). Thus, limiting the scope of searching for unknown (b _20), it can be reduced search times.

7). Third Estimation Method In the above-described embodiment, the unknown number (b ₀₀ , b _20, etc.) is estimated only once. In this embodiment, the unknown number is estimated a plurality of times, and the unknown number is estimated based on the plurality of estimated values. The value may be determined with high accuracy. A third estimation method for determining an unknown value from a plurality of estimated values will be described with reference to FIG. Hereinafter, the intermediate pixel value (b ₂₀ ) will be described as an example, but the estimated pixel value (v _{02 and the} like) can be similarly estimated.

As shown in FIG. 23, in the received light value in the horizontal direction detected by superposition shift sampling, first, a high-definition intermediate pixel value φ ₀ = {b ₀₀ , b ₁₀ , b ₂₀ } is obtained from {a ₀₀ , a ₁₀ }. Ask. The intermediate pixel value φ ₀ = {b ₀₀ , b ₁₀ , b ₂₀ } is obtained by a method similar to the methods of the above formulas (10) to (13).

Next, a high-definition intermediate pixel value φ ₁ = {b ₁₀ , b ₂₀ , b ₃₀ } is obtained from {a ₁₀ , a ₂₀ } by the same method as the above formulas (14) to (17). Further, a high-definition intermediate pixel value φ ₂ = {b ₂₀ , b ₃₀ , b ₄₀ } is obtained from {a ₂₀ , a ₃₀ } by the same method as the above formulas (14) to (17).

In these three estimations, the intermediate pixel value b ₂₀ appears in each of the sets φ ₀ , φ ₁ , φ ₂ . Therefore, three estimated values of the intermediate pixel value b ₂₀ are obtained by three estimations. In the present embodiment, the final intermediate pixel value b ₂₀ is determined (determined) from these three estimated values. For example, the final intermediate pixel value b ₂₀ can be determined by the following first to fourth determination methods.

In the first determination method, an average value of the three b ₂₀ candidate values is set as the final determination value of b ₂₀ .

In the second determination method, two values having the smallest difference among the three b ₂₀ candidate values are specified, and an average value of the two specified values is set as a final determination value of b ₂₀ . According to this method, two estimated values that are close to each other are assumed to be probable estimated values, and the estimation accuracy of unknowns can be improved.

In the third determination method, pixel values of a large number of known high-resolution image samples are added to obtain a light reception value and an intermediate pixel value. An intermediate pixel value pattern having a high probability of occurrence with respect to the received light value pattern is specified in advance from the obtained received light value and intermediate pixel value. Then, an intermediate pixel value pattern having a high occurrence probability with respect to the received light value pattern acquired by photographing is obtained with reference to the correspondence relationship specified in advance. Among the three intermediate pixel value patterns φ ₀ , φ ₁ , φ ₂ , the one closest to the obtained intermediate pixel value pattern is determined, and b ₂₀ that is an element of the determined intermediate pixel value pattern is most probable. b Final decision value of ₂₀ . According to this method, it is possible to perform pixel value estimation reflecting a known image such as a natural image.

In the fourth determination method, b ₂₀ to be finally adopted is determined according to the degree of change in the difference values δi ₀ , δi ₁ , δi ₂ of the neighboring observation pixel values a _{00 to} a ₃₀ related to the derivation of the three b _20. To do. Here, the combination pattern of the degree of change applied to the determination of b ₂₀ is assumed to be ξ = {δi ₀ , δi ₁ , δi ₂ }. Pixel values of a large number of known high-resolution image samples are added to obtain element patterns and combination patterns ξ of φ ₀ , φ ₁ , and φ ₂ . An occurrence probability distribution of b ₂₀ for the combination pattern ξ is obtained in advance from the obtained element pattern and combination pattern ξ. In the final determination, the occurrence probability distribution of b ₂₀ with respect to the difference value pattern ξ obtained from the captured image is obtained with reference to the occurrence probability distribution obtained in advance. Then, using the occurrence probability distribution, it is determined which of the three b ₂₀ candidate values obtained by the estimation calculation has the highest probability of occurrence, and b ₂₀ is specified. That is, since the probability of the value generated by b ₂₀ with respect to the combination pattern ξ of the degree of change is known in advance, among the three b ₂₀ candidate values obtained by estimation using the combination pattern ξ in the estimation process. The most probable value of b ₂₀ can be determined from According to this method, it is possible to perform pixel value estimation reflecting a known image such as a natural image. Further, it is possible to perform pixel value estimation according to the degree of change in the pixel value regardless of the size of the pixel value.

  According to the present embodiment, as shown in FIG. 24, the first pixel (first light receiving unit) and the second pixel (second light receiving unit) are adjacent pixels. Then, the first and second pixels are sequentially set to the first position and the second position next to the first position by the pixel shift. The first pixels at the first and second positions overlap in the first overlapping region, and the second pixels at the first and second positions overlap in the second overlapping region.

At this time, a region obtained by removing the first overlapping region from the first pixel at the first position is the first light receiving region. Further, a region obtained by removing the first overlapping region from the first pixel at the second position is the second light receiving region. Then, a first intermediate pixel value pattern that is a continuous intermediate pixel value including first and second intermediate pixel values (for example, b ₀₀ , b ₂₀ ) that are light reception values of the first and second light receiving regions. (Φ ₀ = {b ₀₀ , b ₁₀ , b ₂₀ }) is estimated.

A region obtained by removing the second overlapping region from the second pixel at the first position is the third light receiving region. A region obtained by removing the second overlapping region from the second pixel at the second position is the fourth light receiving region. Then, a second intermediate pixel value pattern (φ which is a continuous intermediate pixel value including the third and fourth intermediate pixel values (b ₂₀ , b ₄₀ ) which are the light receiving values of the third and fourth light receiving regions. ₂ = {b ₂₀ , b ₃₀ , b ₄₀ }) is estimated.

The overlapping region of the first pixel at the second position and the second pixel at the first position is the fifth light receiving region. Then, successive intermediate pixel values that include the fifth intermediate pixel value (b ₂₀ ) that is the light reception value of the fifth light receiving region and do not include the first and fourth intermediate pixel values (b ₀₀ , b ₄₀ ). A third intermediate pixel value pattern (φ ₁ = {b ₁₀ , b ₂₀ , b ₃₀ }) is estimated.

The third and fifth light receiving areas are the same light receiving areas as the second light receiving areas. Then, the second, third, and second intermediate pixel values (b ₂₀ ) of the same light receiving region are obtained by estimating the _first to third intermediate pixel value patterns (φ ₀ , φ ₂ , φ ₁ ). Finally, it is determined based on the intermediate pixel value (b ₂₀ ) of 5.

  In this way, it is possible to estimate the pixel value with high accuracy. That is, the final pixel value can be determined based on the three (plural) estimated values obtained by estimating three (plural times). For example, by using a determination method using a known image as described above, it is possible to estimate a pixel value in accordance with a pixel value pattern of an actual image.

8). Fourth Estimation Method In the above estimation method, the pixel value is estimated using the least square method, but in the present embodiment, the pixel value may be estimated using a neural network (nonlinear estimation method). The fourth estimation method will be described with reference to FIG. In the following, the estimation of the intermediate pixel value (b ₀₀ etc.) will be described, but the estimated pixel value (v ₀₀ etc.) can be similarly estimated.

In this estimation method, pixel values of known high-resolution images are added, and the received light value {a _m0 , a _{(m + 1) 0} } and the intermediate pixel value φ _m = {b _m0 , b corresponding to this received light value. _{(M + 1) 0} , b _{(m + 2) 0} } is obtained (m is an integer equal to or greater than zero). With these as learning data, the weighting factor W is calculated in advance by learning calculation of the neural network shown in FIG. Specifically, the weighting factor W is calculated so that the error evaluation value E shown in the following formula (18) becomes zero (including substantially zero, which is a predetermined value in a broad sense). As a learning method of the neural network, a general neural network learning method may be used.

And when estimating from a picked-up image, it estimates using the weighting coefficient W computed beforehand. In other words, the relational expression of the intermediate pixel value is expressed by the method described in the above formula (2) and the like, b _m0 (= α) that minimizes the error evaluation value E shown in the following formula (18) by changing the unknown b _m0. Ask for.

According to the above embodiment, the correspondence relationship of the intermediate pixel value pattern φ _m = {b _m0 , b _{(m + 1) 0} , b _{(m + 2) 0} } with respect to the light reception value {a _m0 , a _{(m + 1) 0} } of the pixel is Based on the known high-resolution frame image, the similarity between the light reception value of the pixel and the intermediate pixel value pattern is acquired as pre-determined information. Then, based on the acquired foresight information, the intermediate pixel value pattern φ _m = {b _m0 , b _{(m + 1) 0} , b _{(m + 2) 0} } represented by the relational expression between the intermediate pixel values is acquired. Similarity with the received light value {a _m0 , a _{(m + 1) 0} } of the pixel is evaluated.

  In this way, the similarity is evaluated based on the foreseeing information acquired based on the known high-resolution frame image, so image information (for example, spatial frequency characteristics) included in the known image such as a natural image. Can be estimated.

For example, according to the fourth estimation method, the image processing apparatus has a neural network. This neural network uses, as look-ahead information, a node weighting factor W obtained by learning based on a known high-resolution frame image. Then, the neural network receives the intermediate pixel value pattern φ _m = {b _m0 , b _{(m + 1) 0} , b _{(m + 2) 0} } and the received light value {a _m0 , a _{(m + 1) 0} } of the pixel, and the similarity The evaluation result E is output. Based on the similarity evaluation result E from this neural network, the value of each intermediate pixel value of the intermediate pixel value pattern φ _m = {b _m0 , b _{(m + 1) 0} , b _{(m + 2) 0} } is determined.

  In this way, the similarity between the intermediate pixel value pattern and the light reception value of the pixel can be evaluated based on the foresight information by the neural network using the node weighting factor W obtained by learning.

Further, according to the third estimation method described above, an intermediate pixel value pattern having a high probability of occurrence with respect to the received light value pattern is obtained as look-ahead information (third determination method). Further, the occurrence probability distribution of b ₂₀ for the combination pattern ξ = {δi ₀ , δi ₁ , δi ₂ } of the degree of change may be obtained as look-ahead information (fourth determination method).

  In this way, the pixel value can be estimated using the occurrence probability distribution of the pixel value as foresight information. Thereby, a pixel value having a high probability of occurrence in a known image such as a natural image can be used as an estimated value.

9. Adaptive Noise Reduction Processing In this embodiment, noise filtering processing (filtering processing in a broad sense) corresponding to the pixel position may be performed on the estimated pixel value v _ij . FIG. 26 shows a configuration example of the noise filter.

For example, when pixel estimation is performed by superposition shift sampling by pixel shift, how noise is generated varies depending on the positions of the final estimated pixels v _{ij to} v _{(i + 1) (j + 1)} . Therefore, as shown in FIG. 26, a solid image with a fixed gradation (fixed gradation chart) is imaged, estimation processing is performed on the captured image, and final estimated pixels v _{ij to} v _{(i + 1} ) at different estimated positions. _{) (J + 1)} is compared by the noise comparison unit NH. Then, based on the comparison result, the filter coefficient is set by the filter coefficient setting unit FK so as to enhance the noise reduction effect of the noise filters F1 to F4 at the pixel positions where the generated noise is large. As a result, the filter coefficients are set so that the noise appearance is the same (substantially the same) regardless of the position of v _ij to v _{(i + 1) (j + 1)} . By performing the filtering process in which the filter coefficient is set in this manner, the image quality of the predicted pixel image can be improved.

According to the noise filter processing described above, filter processing with different filter coefficients is performed on the estimated pixel values v _{ij to} v _{(i + 1) (j + 1)} obtained by the estimation according to the pixel position of the estimated pixel.

  In this way, even when different noise is generated depending on the pixel position of the estimated pixel, the noise can be reduced by performing the noise filter process according to the pixel position of the estimated pixel.

10. First Interpolation Method In the present embodiment, as a light reception value (a ₀₀ or the like) used for pixel value estimation, a light reception value acquired by superposition sampling may be used as it is, or a light reception value generated by interpolation is used. Also good. The received light value interpolation method in this embodiment will be described with reference to FIGS. FIG. 27 is an explanatory diagram of the first interpolation method.

  Note that the frame used in the following description is, for example, the timing at which one low-resolution frame image is captured by the image sensor or the timing at which one low-resolution frame image is processed in image processing. Alternatively, one low-resolution frame image or high-resolution frame image in image data is also referred to as a frame as appropriate.

  In this first interpolation method, the received light value of the interpolation target frame is interpolated using the received light value acquired in another frame (time axis interpolation).

Specifically, as shown by A1 in FIG. 27, received light values a _ij , a _{(i + 1) j} , a _{(i + 1) (j + 1)} , and a _{i (j + 1)} are sequentially acquired in frames fx to fx + 3. In the frame fx + 4, the light reception value a _ij is acquired again. That is, as shown in A2, a low-resolution frame image composed of the received light value a _ij is acquired every four frames in frames fx, fx + 4, fx + 8,. As shown in A3, a time-axis interpolation filter process (filter process) is performed on the time-series data of the received light value a _ij . Then, as indicated by A4, the light reception value a ′ _ij is generated in all frames (each frame) by this time axis interpolation filter processing.

In this way, each received light value is acquired every four frames, and time axis interpolation filter processing is performed, and the received light values a ′ _ij , a ′ _{(i + 1) j} , a ′ _{(i + 1) (j + 1)} , a ′ _{i (j + 1)} is generated.

  FIG. 28 shows a timing chart example of the interpolation processing. As shown in B1 of FIG. 28, in each frame, the pixel setting position is shifted by ½ pixel. As shown in B2, the fusion frames F1, F2,... (Low-resolution frame images. Fusion-frame) are photographed in each frame. As shown in B3, a low-resolution moving image frame is generated, and live view display and recording are performed. As shown in B4, time axis interpolation of the received light value is performed, pixel value estimation is performed using the received light value after interpolation, and a high-resolution still image frame is generated. As shown in B5, a high-resolution still image or a high-resolution moving image is output from the ninth frame in which all received light values are interpolated.

  Now, since the low-resolution frame images are sequentially acquired by pixel shift, four frames are required to acquire all the received light values used for pixel value estimation. Therefore, if the photographed received light value is used for pixel value estimation as it is, the received light value of a different frame is used for estimation, and the image quality may be deteriorated when the subject is moving.

In this regard, according to the above-described embodiment, the pixel shift is performed in each frame fx, fx + 1,..., And the pixel is shifted to a plurality of positions (for example, four positions; P1 to P4 shown in FIG. 27 described above). Are set sequentially. Then, the pixel is set at the same position for each of a plurality of frames (every 4 frames). In this way, the received light values a _{ij to} a _{(i + 1) (j + 1)} corresponding to the respective positions in time series are acquired, and low resolution frame images are acquired in time series using the acquired received light values (continuous low Resolution frame image).

A process of interpolating received light values a ′ _{ij to} a ′ _{(i + 1) (j + 1)} of pixels at a plurality of positions P1 to P4 is performed on the obtained continuous low-resolution frame image. Specifically, the light-receiving value (a ′ _ij ) of the pixel at the missing position (for example, P1) in the continuous low-resolution frame image of the frame to be interpolated (for example, fx + 1) is time-axis interpolated. That is, the light reception value (a ′ _ij ) is received by the pixels in the continuous low-resolution frame images of the frames (fx, fx + 4) before and after the interpolation target frame (fx + 1) at the same position (P1) as the missing position. Time axis interpolation is performed using the value (a _ij ).

Then, based on the continuous low-resolution frame image interpolated by the time axis interpolation, the pixel value v _ij of each pixel in each frame fx, fx + 1,... Is estimated.

  In this way, the light reception value is acquired by the pixel shift, and the light reception value at the missing position is interpolated using the acquired light reception value, and the final estimated pixel value can be obtained from the light reception value after the interpolation. As a result, since the light reception values at all positions can be interpolated in each frame, the pixel value is estimated from the light reception values in the same frame, and image quality deterioration can be suppressed even when the subject is moving.

For example, in the above embodiment, the time axis interpolation is performed by the time axis interpolation filter.
As a result, the received light value (a _ij of fx, fx + 4) at each position (for example, P1) is time-axis interpolated to generate the received light value (a ′ _ij ) of the missing position (P1 of fx + 1 to fx + 3).

11. Second Interpolation Method In this embodiment, adaptive received light value interpolation may be performed according to the movement of the subject. A second interpolation method for adaptively interpolating received light values will be described with reference to FIGS. 29 and 30. FIG.

  In the second interpolation method, the received light values of the frames before and after the neighborhood are applied to a portion where there is no motion or a small amount (light reception value), and the light reception value is superimposed on the adjacent light reception value in the same frame for a portion where the motion is large or large Interpolate the shifted received light value. However, since it is determined that all pixels have movement when panning or the like is performed, motion compensation processing may be performed in advance.

Hereinafter, a case where a high-resolution still image of this frame is generated by focusing on the frame fx + 1 from the low-resolution image frames fx to fx + 4 shown in FIG. 29 will be described as an example. FIG. 30 shows a part of the received light value acquired as the imaging data in the frame fx + 1 for the sake of simplicity. In FIG. 30, the acquired light reception values are indicated by a ₁₀ (x + 1), a _{(−1) 0} (x + 1), a _{(−1) 2} (x + 1), and a ₁₂ (x + 1).

As shown in FIG. 30, for example _a 10 (x + 1) to determine the four high resolution pixels constituting this _a 10 (x + 1) pixel pitch p / 2 shifted by three light receiving values _{a 00,} based on the It is necessary to obtain (x + 1), a ₀₁ (x + 1), and a ₁₁ (x + 1) by interpolation. Below, the method of calculating | requiring these three light reception values is demonstrated.

First, as shown in D1 of FIG. 29, a ₀₀ (x) and a ₀₀ (x + 4) of the neighboring frames fx and fx + 4 acquired at the corresponding position of a ₀₀ (x + 1) are compared. When the difference is smaller than the predetermined threshold δ, it is determined that the subject (a part of the moving subject) has not passed through the pixel position corresponding to a ₀₀ (x + 1) between the frames fx and fx + 4. To do. In this case, as shown in _D2, assign _a 00 (x + 1) a as a value of 00 (x) or _a 00 (x + 4).

On the other hand, when the difference between a ₀₀ (x) and a ₀₀ (x + 4) is larger than the predetermined threshold δ, it is assumed that the subject passes through the pixel position corresponding to a ₀₀ (x + 1) between the frames fx and fx + 4. to decide. In this case, as indicated by D3, an interpolated value is obtained by using received light values a ₁₀ (x + 1) and a _{(−1) 0} (x + 1) close to a ₀₀ (x + 1) within the same frame, and a ₀₀ ( x + 1). For example, an average value of a ₁₀ (x + 1) and a _{(−1) 0} (x + 1) is set as the value of a ₀₀ (x + 1). However, in the present embodiment, interpolation may be performed using a larger number of peripheral light reception values instead of the average of two adjacent light reception values.

The above-described interpolation method of a ₀₀ (x + 1) is summarized by the following equation (19).
When | a ₀₀ (x) −a ₀₀ (x + 4) | ≦ δ,
a ₀₀ (x + 1) = a ₀₀ (x) = a ₀₀ (x + 4)
When | a ₀₀ (x) −a ₀₀ (x + 4) |> δ,
a ₀₀ (x + 1) = {a ₁₀ (x + 1) + a _{(−1) 0} (x + 1)} / 2
(19)

Similarly, as indicated by D4 and D5, a ₁₁ (x + 1) is interpolated by the following equation (20).
When | a ₁₁ (x−2) −a ₁₁ (x + 2) | ≦ δ,
a ₁₁ (x + 1) = a ₁₁ (x−2) = a ₁₁ (x + 2)
When | a ₁₁ (x−2) −a ₁₁ (x + 2) |> δ,
a ₁₁ (x + 1) = {a ₁₀ (x + 1) + a ₁₂ (x + 1)} / 2
(20)

As shown in D6 and D7, a ₀₁ (x + 1) is interpolated by the following equation (21).
When | a ₀₁ (x−1) −a ₀₁ (x + 3) | ≦ δ,
a ₀₁ (x + 1) = a ₀₁ (x−1) = a ₀₁ (x + 3)
When | a ₀₁ (x−1) −a ₀₁ (x + 3) |> δ,
a ₀₁ (x + 1) = {a ₁₀ (x + 1) + a _{(−1) 2} (x + 1)} / 2
(21)

  Note that the threshold value δ may be set by evaluating the quality of the generated image and setting an allowable level threshold value δ. For example, it may be set to a level at which it is not determined that there is movement due to noise in spite of still images.

According to the second interpolation method, the difference value (a ₀₀ (x) −a ₀₀ (x + 4)) of the light reception values of the pixels in the frames (for example, fx, fx + 4) before and after the frame to be interpolated (for example, fx + 1). Is required. When the difference value is smaller than the predetermined threshold δ, the received light value (a ₀₀ (x + 1)) of the pixel at the missing position (position) in the interpolation target frame (fx + 1) is the previous or next frame (fx, Interpolation is performed using received light values (a ₀₀ (x), a ₀₀ (x + 4)) of pixels at the same position as the missing position at fx + 4). On the other hand, when the difference value is larger than the predetermined threshold δ, the received light value (a ₀₀ (x + 1)) of the pixel at the missing position in the interpolation target frame (fx + 1) is acquired in the interpolation target frame (fx + 1). The received light values (a ₁₀ (x + 1), a _{(−1) 0} (x + 1)) are interpolated.

In this way, the received light value can be adaptively interpolated according to the movement of the subject. Specifically, in a pixel with small movement, the received light value acquired at the same position is used, so that a positional error can be reduced. On the other hand, in a pixel with a large motion, since a received light value acquired in the same frame is used, a temporal error can be reduced. For example, it is impossible to know from the received light values a ₀₀ (x) and a ₀₀ (x + 4) when the subject has passed between the frames fx and fx + 4. Therefore, when interpolating from a ₀₀ (x) and a ₀₀ (x + 4), the influence of the movement of the subject appears even at the timing before the passage timing of the subject. In this embodiment, when it is determined that the subject has passed, interpolation is performed using the light reception value of the same frame, so that the passage timing of the subject can be accurately reflected.

  Further, according to the second interpolation method, it is possible to suppress image quality deterioration due to erroneous detection of motion. For example, if there is a mere change in brightness exceeding a predetermined threshold δ within the four-frame imaging period, there is a possibility that a mere change in brightness is erroneously detected as the movement of the subject. In this regard, according to the second interpolation method, even if erroneous detection is performed, it is only possible to switch to intra-frame interpolation, so that it is possible to prevent the image quality from deteriorating significantly.

  In the first and second interpolation methods, the received light value may be interpolated by motion compensation. For example, as shown in FIG. 31, four continuous fusion frames photographed at a high frame rate (1/60 seconds) are used, and motion compensation is performed on each fusion frame to produce one still image (low frame). A high-definition image taken at a rate of 1/15 seconds) may be generated.

12 Imaging Device and Image Processing Device FIGS. 32 and 33 show detailed configuration examples of an imaging device and an image processing device that perform the above estimation processing and time axis interpolation processing.

  FIG. 32 shows a first detailed configuration example of an imaging apparatus and an image processing apparatus that perform zoom selection before pixel shift. The imaging apparatus 10 includes a low-pass optical LPF 100 (LPF: low-pass filter), a normal sampling unit 110, a wide-pass optical LPF 120, a zoom region selection unit 130, and a superposition shift sampling unit 140 (reading control unit). The image processing apparatus 20 includes an anti-aliasing filter 200, a frame buffer 210 (storage unit), a time-axis pixel interpolation unit 220 (interpolation processing unit), a pixel value estimation calculation unit 230 (estimation processing unit), and an anti-aliasing filter 240. Including.

  The imaging apparatus and image processing apparatus of the present embodiment are not limited to this configuration, and various modifications such as omitting some of the components or adding other components are possible. . FIG. 32 illustrates an example in which the image processing device 20 is provided outside the imaging device 10, but in the present embodiment, the image processing device 20 may be provided inside the imaging device 10.

  The imaging device 10 is, for example, a digital camera or a video camera. The image processing apparatus 20 is realized by, for example, an image processing engine (IC) or a PC (computer). The imaging device 10 and the image processing device 20 operate in a first mode for performing wide shooting (normal shooting and non-zoom shooting) and a second mode for performing digital zoom shooting. These modes are set by a control unit (not shown) of the imaging device 10 and the image processing device 20, for example. In the following description, an example of using a 12 Mpix (Mpix: megapixel) image sensor will be described.

  When the imaging device 10 is set to the first mode, the low-pass optical LPF 100 is set. The low-pass optical LPF 100 is a low-pass filter having a pass band corresponding to a resolution of 12 Mpix of the image sensor. Then, the normal sampling unit 110 captures a 12 Mpix image not by pixel shift imaging but by normal wide imaging. For example, the shooting rate is 15 fps.

  The anti-aliasing filter 200 of the image processing apparatus 20 performs anti-aliasing processing on the image and outputs a 12 Mpix still image or moving image.

  On the other hand, when the imaging apparatus 10 is set to the second mode, the wide-pass optical LPF 120 is set. The wide-pass optical LPF 120 is a low-pass filter having a pass band corresponding to a high resolution image by pixel shift. For example, when pixel shifting is performed at a ½ pixel pitch, it has a pass band that can be imaged at a resolution of 4 × 12 = 48 Mpix. Then, the zoom area selection unit 130 sets a reading area. For example, in the case of double digital zoom, a readout area of 12/4 = 3 Mpix is set. The superposition shift sampling unit 140 reads the received light value from the reading area while performing pixel shift control. For example, the shooting rate is 15 × 4 = 60 fps.

  The frame buffer 210 of the image processing apparatus 20 buffers captured images of 8 frames necessary for the interpolation process. The time-axis pixel interpolation unit 220 performs an interpolation process on the received light value from the 8-frame image. Then, the pixel value estimation calculation unit 230 obtains an estimated pixel value from the light reception value subjected to the interpolation process, and outputs an image of 3 × 4 = 12 Mpix. The anti-aliasing filter 240 performs anti-aliasing processing on the image, and outputs a 12 Mpics high-definition double digital zoom image.

  FIG. 33 shows a second detailed configuration example of an imaging apparatus and an image processing apparatus that perform zoom selection after pixel shift. The imaging apparatus 10 includes a wide-pass optical LPF 120 and a superposition shift sampling unit 140. The image processing apparatus 20 includes a frame buffer 210, a time axis pixel interpolation unit 220, a pixel value estimation calculation unit 230, an anti-aliasing filter 240, and a zoom region selection unit 250.

  The imaging device 10 and the image processing device 20 perform digital zoom after high definition by estimation processing. Specifically, the wide-pass optical LPF 120 has a pass band that can be imaged with a resolution of 4 × 12 = 48 Mpix. The superposition shift sampling unit 140 performs imaging with a pixel number of 12 Mpix of the image sensor while performing pixel shift control. For example, the shooting rate is 60 fps.

  Then, the frame buffer 210 buffers an image of 8 frames, the time axis pixel interpolation unit 220 interpolates the received light value, the pixel value estimation calculation unit 230 obtains an estimated pixel value, and an image of 4 × 12 = 48 Mpix is obtained. Output. The zoom area selection unit 250 sets a zoom area according to the zoom magnification, and trims the image in that area. For example, in the case of no zoom, a 48 Mpix image is output as it is, and in the case of a double digital zoom, a 48/4 = 12 Mpix image is trimmed. The anti-aliasing filter 240 outputs a 12 Mpix image by downsampling in the case of no zooming, and outputs a 12 Mpics high-definition 2x digital zoom image in the case of 2x digital zoom.

13. Fifth Estimation Method In the first to fourth estimation methods described above, pixel values are estimated by performing pixel shift in the horizontal direction and vertical direction. However, in this embodiment, pixel values are obtained by performing pixel shift in an oblique direction. May be estimated. The fifth estimation method will be described with reference to FIGS.

As shown in FIG. 34A, the received light value a _k (k is an integer equal to or greater than 0) is acquired while shifting the pixels in the diagonal direction at a p / 2 pitch. Here, the diagonal direction is a direction intersecting the horizontal direction and the vertical direction of the pixel array, for example, a direction along a diagonal line of a rectangular pixel.

When the pixel shift is performed in the diagonal direction as described above, a relational expression of the intermediate pixel value b _k (three-pixel addition value) in the diagonal direction is obtained. As shown in FIG. 34B, the received light values are set to a ₀ , a ₁ , a _{2 in the} order of shift, and intermediate pixel values b _{0 to} b ₂ , b ₁ ′, b are added so that three estimated pixels are added. Define a ₂ 'light receiving area. At this time, as shown in FIG. 35, the following expression (22) is established.
b ₀ = (unknown number),
b ₁ ′ = b ₀ + δ ₀ = b ₀ + (a ₁ −a ₀ ),
b ₂ ′ = b ₁ + δ ₁ = b ₁ + (a ₂ −a ₁ ) (22)

'Because the overlapping portion of the light receiving region of many, _{b 1} ≒ _{b 1'} b ₁ and _{b 1} Assuming that the following equation (23) holds.
b ₀ = (unknown number),
b ₁ = b ₀ + δ ₀ = b ₀ + (a ₁ −a ₀ ),
b ₂ = b ₁ + δ ₁ = b ₁ + (a ₂ −a ₁ ) (23)

Thus, b ₀ can be an unknown and the intermediate pixel values b ₁ and b ₂ can be obtained as a function of b ₀ . Here, δ ₀ and δ ₁ are the difference values of the sampling pixel values separated by one shift.

Next, an unknown b ₀ in the oblique direction is obtained, and an intermediate pixel value is obtained. As shown in FIG. 36, a pattern {a _k } based on sampling pixel values detected by superposition shift sampling and a pattern based on intermediate pixel values {b _k } are compared. Then, as shown in FIG. 37, the unknown b ₀ is derived and set with a state where the error E is minimized.

Specifically, the error Ek is obtained by the following equation (24). Then, an unknown number b ₀ (= γ) that minimizes Ek is obtained, and b ₀ is set to that value.

If the value of b ₀ can be estimated, b ₁ and b ₂ are naturally determined by substituting b ₀ into the above equation (23). If this is obtained for other pixels, all intermediate pixel values {b _k } can be derived.

As shown in FIG. 38, the final estimated pixel value v _k is the pixel value of the overlapping region of the pixels shifted in the diagonal direction. The following equation (25) is established for the estimated pixel value, and the estimated pixel value v _k is obtained by substituting the light reception value a _k and the intermediate pixel value b _k described above.
v ₀ = a ₀ −b ₀ ,
v ₁ = a ₁ -b ₁ (25)

According to the above embodiment, the pixels are shifted while being superimposed in an oblique direction, and a low-resolution frame image (an image composed of the light reception value _ak ) is acquired. Then, the light reception value v _k of the overlapping region of the pixels shifted in the diagonal direction is estimated as the estimated pixel value.

  In this way, the mechanical pixel shift can be performed only in one direction, and the pixel shift can be performed more easily than when performing the pixel shift in the horizontal direction and the vertical direction.

14 Sixth estimation method (weighted addition)
In the above estimation method, the case where there is no weighting of the filter transmittance (r = 1) has been described as an example. However, in this embodiment, the estimated pixel value is obtained from the light receiving value of the weighted (r> 1) transmittance. May be. The sixth estimation method will be described with reference to FIGS. 39 to 44.

The weighting coefficients of the transmittance of the filter cf1~cf4 shown in FIG. 1 and _{c 1, c 2, c 3} , c 4. Assuming that c ₁ = 1, the weighting coefficient takes the ratio relationship rule shown in the following equation (26) (r is a real number with r> 1).
c ₁ = 1, c ₂ = 1 / r, c ₃ = 1 / r, c ₄ = 1 / r ² (26)

In the following, in order to simplify the explanation, r = 2 is assumed and the following equation (27) is assumed.
c ₁ = 1, c ₂ = 1/2, c ₃ = 1/2, c ₄ = ¼ (27)

As shown in FIG. 39, attention is paid to the first horizontal row detected by superposition shift sampling, and pixel values are set to a ₀₀ , a ₁₀ , and a _{20 in the} order of shift. At this time, the following expression (28) holds.
a ₀₀ = c ₁ v ₀₀ + c ₂ v ₀₁ + c ₃ v ₁₀ + c ₄ v _{11
a 10 = c 1 v 10 +} c 2 v 11 + c 3 v 20 + c 4 v 21 ··· (28)

Further, b ₀₀ , b ₁₀ , and b ₂₀ are defined as shown in the following formula (29), and the above formula (27) is substituted.
b ₀₀ = c ₁ v ₀₀ + c ₂ v ₀₁ = v ₀₀ + (1/2) v ₀₁
b ₁₀ = c ₁ v ₁₀ + c ₂ v ₁₁ = v ₁₀ + (1/2) v _{11
b 20 = c 1 v 20 +} c 2 v 21 = v 20 + (1/2) v 21 ··· (29)

Next, when the above equation (28) is transformed using the above equations (27) and (29), the following equation (30) is established.
a ₀₀ = v ₀₀ + (1/2) v ₀₁ + (1/2) v ₁₀ + (1/4) v ₁₁
= B ₀₀ + (1/2) b _{10
a 10 = v 10 + (1/2} ) v 11 + (1/2) v 20 + (1/4) v 21
= B ₁₀ + (1/2) b ₂₀ (30)

In the above equation (30), when a ₀₀ and a ₁₀ are multiplied by a predetermined coefficient (predetermined weighting coefficient) to obtain a difference δi ₀ and transformed using the above equation (29), the following equation (31) is established.
δi ₀ = a ₁₀ -2a _{00
= (1/2) v 20 + (} 1/4) v 21 - (2v 00 + v 01)
= (1/2) b ₂₀ -2b ₀₀ (31)

If b ₀₀ is an unknown, the intermediate pixel values b ₁₀ and b ₂₀ can be obtained as a function of b ₀₀ as shown in the following equation (32).
b ₀₀ = (unknown number),
b ₁₀ = 2 (a ₀₀ −b ₀₀ ),
b ₂₀ = 4b ₀₀ + 2δi ₀ = 4b ₀₀ +2 (a ₁₀ −2a ₀₀ ) (32)

In this way, a high-definition combination pattern of intermediate pixel values {b ₀₀ , b ₁₀ , b ₂₀ } is obtained with b ₀₀ as an unknown (initial variable). Similarly, combination patterns of intermediate pixel values {b ₀₁ , b ₁₁ , b ₂₁ }, {b ₀₂ , b ₁₂ , b ₂₂ } are obtained with b ₀₁ and b ₀₂ as unknowns in the second and third lines. It is done.

Next, a description will be given of a method of obtaining the unknown _{b 00.} As shown in FIG. 40, a pattern {a ₀₀ , a ₁₀ } based on sampling pixel values detected by weighted superposition shift sampling is compared with a pattern based on intermediate pixel values {b ₀₀ , b ₁₀ , b ₂₀ }. Then, an unknown number b ₀₀ that minimizes the error E is derived and set as the intermediate pixel value b ₀₀ .

At this time, as shown in the above equation (30), the sampling pixel values {a ₀₀ , a ₁₀ } are added values of adjacent values by different weighting of the intermediate pixel values {b ₀₀ , b ₁₀ , b ₂₀ }. Therefore, even if these are simply compared, a correct estimated value cannot be obtained. Therefore, as shown in FIG. 40, the comparison is performed by weighting the intermediate pixel values. Specifically, the intermediate pixel values weighted _{{b ij, b (i +} 1) j} _is, the use of it is _{c 3 = c 1/2,} c 4 = c 2/2, the following equation (33) You can see that it holds.
a _ij = b _ij + (1/2) b _{(i + 1) j} (33)

Considering the weighting by the above equation (33), the evaluation function Ej shown in the following equation (34) is obtained. Then, the similarity between the pattern {a ₀₀ , a ₁₀ } and the intermediate estimated pixel value {b ₀₀ , b ₁₀ , b ₂₀ } is evaluated using this evaluation function Ej.

When the above equation (32) is used, the evaluation function Ej is represented by a function having b ₀₀ as an initial variable. Therefore, as shown in FIG. 41, the unknown b ₀₀ (= α) that minimizes Ej is obtained, and the value of b ₀₀ can be determined. Then, the estimated value of b ₀₀ is substituted into the above equation (32) to obtain b ₁₀ and b ₂₀ . Since the range of values that b ₀₀ can take is 0 ≦ b ₀₀ ≦ a ₀₀ , the minimum value of the evaluation function Ej may be obtained within this range. Similarly, in the second and third lines, a combination pattern of intermediate pixel values {b ₀₁ , b ₁₁ , b ₂₁ } and {b ₀₂ , b ₁₂ , b ₂₂ } is obtained with b ₀₁ and b ₀₂ as unknowns. It is done.

Next, a method for _{obtaining the} final estimated pixel value v _ij using the obtained intermediate pixel value b _ij will be described. In the following description, the leftmost vertical column (i = 0 column) shown in FIGS. 13A and 13B will be described as an example. As shown in FIG. 42, the relationship between the intermediate pixel values {b ₀₁ , b ₀₁ , b ₀₂ } and the final estimated pixel values {v ₀₀ , v ₀₁ , v ₀₂ } is expressed by the following equation (35).
b ₀₀ = c ₁ v ₀₀ + c ₂ v ₀₁ = v ₀₀ + (1/2) v ₀₁ ,
b ₀₁ = c ₁ v ₀₁ + c ₂ v ₀₂ = v ₀₁ + (1/2) v ₀₂ (35)

When the difference δj ₀ is obtained by multiplying b ₀₀ and b ₀₁ by a predetermined coefficient, the following equation (36) is established.
δj ₀ = b ₀₁ -2b ₀₀
= (V ₀₁ + (1/2) v ₀₂ ) − (2v ₀₀ + v ₀₁ )
= (1/2) v ₀₂ -2v ₀₀ (36)

If v ₀₀ is an unknown (initial variable), the final estimated pixel values v ₀₁ and v ₀₂ are obtained as a function of v ₀₀ using the above equations (35) and (36). The function is shown in the following formula (37).
v ₀₀ = (unknown number),
v ₀₁ = 2 (b ₀₀ −v ₀₀ ),
v ₀₂ = 4v ₀₀ + 2δj ₀ = 4v ₀₀ +2 (b ₀₁ -2b ₀₀ ) (37)

The estimated pixel value pattern {v ₀₀ , v ₀₁ , v ₀₂ } of the above equation (37) is compared with the intermediate pixel value pattern {b ₀₀ , b ₀₁ }, and an unknown v ₀₀ that minimizes the error Ei is derived. . At this time, using the fact that the weight of the final estimated pixel value {v _ij , v _{(i + 1) j} } is c ₂ = c _1/2 , the following equation (38) is established.
b _ij = v _ij + (1/2) v _{i (j + 1)} (38)

As shown in FIG. 43, the patterns are compared in consideration of the weighting shown in the above equation (38). Specifically, an evaluation function Ei shown in the following formula (39) is obtained.

Then, as shown in FIG. 44, an unknown number v ₀₀ (= β) that minimizes the evaluation function Ei is obtained, and the obtained v ₀₀ is substituted into the above equation (37) to obtain the final estimated pixel values v ₀₁ and v ₀₂ . Ask. Similarly, also in the second column, a combination pattern of final estimated pixel values {v ₁₀ , v ₁₁ , v ₁₂ } is obtained with v ₁₀ as an unknown.

  Although the present embodiment has been described in detail as described above, it will be easily understood by those skilled in the art that many modifications can be made without departing from the novel matters and effects of the present invention. Accordingly, all such modifications are intended to be included in the scope of the present invention. For example, a term (pixel value, superposition shift sampling, etc.) described together with a different term (light reception value, pixel shift, etc.) in a broader sense or the same meaning at least once in the specification or drawing is used anywhere in the specification or drawing. Can also be replaced by their different terms. Further, the configurations and operations of the image sensor, the readout control unit, the estimation processing unit, the image output unit, the imaging device, and the like are not limited to those described in the present embodiment, and various modifications can be made.

10 imaging device, 20 image processing device, 100 low-pass optical LPF,
110 Normal sampling unit, 120 Wide-pass optical LPF,
130 zoom region selection unit, 140 superposition shift sampling unit,
200 anti-aliasing filter, 210 frame buffer,
220 time axis pixel interpolation unit, 230 pixel value estimation calculation unit,
240 anti-aliasing filter, 250 zoom area selection unit,
410 lens, 420 imaging device,
Bk ₀₀ estimation processing block, CF filter unit, Ej evaluation function,
F1-F4 noise filter, FK filter coefficient setting unit, NH noise comparison unit,
P1 to P4 positions, PD1 to PD4 light receiving elements,
W Neural network coefficients,
a _ij light reception value, b ₀ unknown, b _ij intermediate pixel value, cf1 to cf4 filter,
fx frame, NF ND filter, p pixel pitch, p / 2 shift amount,
v ₀₀ unknown, v _ij estimated pixel value, δi _i difference value, φ _i intermediate pixel value pattern

Claims (11)

An image sensor having a plurality of light receiving elements and a filter unit;
A readout control unit that reads out light reception values of the plurality of light receiving elements to obtain a low-resolution frame image;
Based on the low-resolution frame image, an estimation calculation unit that estimates an estimated pixel value having a pixel pitch smaller than the pixel pitch of the low-resolution frame image;
An image output unit that outputs a high-resolution frame image having a higher resolution than the low-resolution frame image based on the estimated pixel value;
Including
In the filter part,
An image pickup apparatus, wherein a plurality of filters of different colors are arranged as filters corresponding to one light receiving element. In claim 1,
Each filter of the plurality of filters is
Having weighted transmission,
The estimation calculation unit includes:
An imaging apparatus, wherein the estimation is performed based on a light reception value obtained by the weighted transmittance. In claim 2,
The filter unit is
As a filter corresponding to the one light receiving element, it has a Bayer array filter,
The Bayer array filter is:
A first green filter, a red filter, a blue filter, and a second green filter;
The transmittance of the red filter and the blue filter is
1 / r (r is a real number of 1 or more) with respect to the transmittance of the first green filter,
The transmittance of the second green filter is
An image pickup apparatus having a transmittance of 1 / r ² with respect to the transmittance of the first green filter. In claim 2,
The filter unit is
A color filter,
An ND (Neutral Density) filter having the weighted transmittance;
An imaging device comprising: In any one of Claims 1 thru | or 4,
The read control unit
The subject image is sampled by sequentially shifting the pixel while superimposing the pixels, and each imaging operation is performed by the imaging element while the pixel is shifted, and the light reception value of the pixel obtained by each imaging operation is a low resolution frame image An imaging device characterized by being acquired as follows. In claim 5,
The read control unit
The pixel shift is performed by sequentially setting the pixels to a first position and a second position next to the first position,
The estimation calculation unit includes:
When the pixel at the first position and the pixel at the second position overlap, the difference value between the light reception value of the pixel at the first position and the light reception value of the pixel at the second position is obtained, and the difference An image pickup apparatus that estimates the estimated pixel value based on a value. In claim 6,
The estimation calculation unit includes:
A first intermediate pixel value that is a light receiving value of a first light receiving region obtained by removing a superimposed region from the pixel at the first position, and a second light receiving region obtained by removing the superimposed region from the pixel at the second position. The relational expression with the second intermediate pixel value that is the received light value is expressed using the difference value,
An imaging apparatus, wherein the first and second intermediate pixel values are estimated using the relational expression, and the estimated pixel value is obtained using the estimated first intermediate pixel value. In claim 7,
The estimation calculation unit includes:
When successive intermediate pixel values including the first and second intermediate pixel values are used as an intermediate pixel value pattern, a relational expression between the intermediate pixel values of the intermediate pixel value pattern is obtained using a light reception value of the pixel. Represent,
The intermediate pixel value pattern represented by the relational expression between the intermediate pixel values is compared with the light reception value of the pixel to evaluate similarity,
An image pickup apparatus that determines each intermediate pixel value included in the intermediate pixel value pattern based on the similarity evaluation result so that the similarity becomes the highest. In claim 8,
The estimation calculation unit includes:
An evaluation function representing an error between the intermediate pixel value pattern represented by the relational expression between the intermediate pixel values and the light reception value of the pixel is obtained, and the intermediate pixel value pattern is set so as to minimize the value of the evaluation function. An imaging apparatus characterized by determining each included intermediate pixel value. An image generation method in an image pickup apparatus in which an image pickup element includes a plurality of light receiving elements and a filter unit,
When a plurality of filters of different colors are arranged as filters corresponding to one light receiving element in the filter unit, a light reception value of the plurality of light receiving elements is read to obtain a low resolution frame image,
Based on the low resolution frame image, estimate an estimated pixel value of a pixel pitch smaller than the pixel pitch of the low resolution frame image,
An image generation method comprising: outputting a high-resolution frame image having a higher resolution than the low-resolution frame image based on the estimated pixel value. In claim 10,
The subject image is sampled by sequentially shifting the pixel while superimposing the pixels, and each imaging operation is performed by the imaging element while the pixel is shifted, and the light reception value of the pixel obtained by each imaging operation is a low resolution frame image Get as
The pixel shift is performed by sequentially setting the pixels to a first position and a second position next to the first position,
When the pixel at the first position and the pixel at the second position overlap, the difference value between the light reception value of the pixel at the first position and the light reception value of the pixel at the second position is obtained, and the difference An image generation method, wherein the estimated pixel value is estimated based on a value.

Images