JP4531231B2 - Imaging device - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates to an imaging apparatus capable of realizing an image with high resolution.
[0002]
[Prior art]
In an imaging device using a two-dimensional image sensor such as a CCD, the relative position between the image and the image sensor is changed slightly, and the images before and after the change are input to synthesize each image to increase the apparent number of pixels. What is known to increase resolution is known. In order to improve the resolution by such a method, as described in JP-A-7-236086, a transparent flat plate member is disposed in parallel between the photographing optical system and the two-dimensional image pickup device, and the transparent flat plate is provided. Displace incident light from the photographic optical system by using one of the three tilting means not on a straight line arranged on the member as a support part and using the other two points as an operation part for operating the transparent flat plate member. Thus, the image on the image sensor is slightly displaced.
[0003]
FIG. 44 is a block diagram showing a conventional imaging device disclosed in, for example, Japanese Patent Laid-Open No. 7-236086, and FIG. 45 is a perspective view showing a mechanism of the imaging device. 44 and 45, 1 is an imaging lens that forms an image, 2 is an imaging element such as a CCD that is two-dimensionally arranged to photoelectrically convert the image, and 103 is substantially between the imaging lens 1 and the imaging element 2. A transparent flat plate member 104 that is arranged in parallel and causes a slight displacement in the incident angle of incident light from the imaging lens 1 to the imaging element 2, and 104 is a base unit that supports the imaging lens 1 and the transparent flat plate member 103.
[0004]
Reference numerals 105a, 105b, and 105c fix the transparent flat plate member 103 to the base unit 104, and selectively operate two points to incline the transparent flat plate member 103. Reference numerals 106a, 106b, and 106c denote corresponding compression springs 105a, 105c. A spring pressing plate 107a for pressing 105b and 105c is provided together with a screw that penetrates the transparent flat plate member 103 and the compression spring 105a, and a portion near the transparent flat plate member 103 is displaced in the optical axis direction by driving to the transparent flat plate member 103. It is a motor that provides tilt. Although not explicitly shown in FIG. 45, a motor similar to the motor 107a is provided together with a screw that penetrates the transparent flat plate member 103 and the compression spring 105b.
[0005]
108 is a displacement device, 108a is a transparent flat plate member 103, an operating portion including a compression spring 105a and a spring pressing plate 106a, 108b is an operating portion including the transparent flat plate member 103, the compression spring 105b and the spring pressing plate 106b, and 109 is a transparent flat plate member. 103, a support portion including a compression spring 105c and a spring pressing plate 106c. The support portion 109 supports the transparent flat plate member 103 when the operation portions 108a and 108b are operated. These are integrally fixed to a housing (not shown), and, as shown in FIG. 44, a predetermined image processing circuit for processing a photoelectrically converted image signal, an image synthesis memory 110, an image memory 7, etc. Is connected.
[0006]
Next, the operation will be described.
First, imaging is performed in a state where neither of the two operating units 108a and 108b is operated, and an image is stored in the image memory 7 at the subsequent stage.
Next, by operating one of the two operating portions 108a and 108b, the transparent flat plate member 103 is rotated about the line connecting the other operating portion 108b and the support portion 109 as a rotation axis.
As a result, the image transmitted through the transparent flat plate member 103 is moved by the inclination of the transparent flat plate member 103 to form an image on the image sensor 2, and an image slightly shifted is stored in the image memory 7.
[0007]
Further, if one operating unit 108a is operated, images are sequentially moved in the same direction, and images are sequentially formed and stored on the image sensor 2. In addition, when the operation unit 108b is driven, the transparent flat plate member 103 is inclined with a line connecting the operation unit 108a and the support unit 109 as a rotation axis, and the image is moved in a direction different from the above.
By appropriately combining these movements in the two directions, a two-dimensional pixel shift to an arbitrary position is performed. After that, by interpolating a plurality of captured images stored in the image memory 7 for each corresponding pixel in consideration of the pixel shift execution direction, an image in which the number of pixels is optically increased is stored in the image synthesis memory 110. can get.
[0008]
Next, a signal processing method for generating a high-definition image from an image obtained by conventional pixel shifting will be described.
The signal array shown in FIG. 2 is an image sensor having a Bayer type array that is most often used in a single-plate image pickup apparatus. The R signal, the G signal, or the B signal in FIG. Color signal to be sampled.
FIG. 12 is an explanatory diagram in which the color signal of the image captured by shifting the image sensor shown in FIG. In the figure, R1, G1, and B1 are color signals of the first photographed image, and R2, G2, and B3 are color signals obtained from the second photographed image subjected to pixel shifting.
[0009]
When the image shown in FIG. 12 is generated as a high-resolution image, it is necessary to generate a full-color signal having four times the number of pixels because both horizontal and vertical are doubled compared to FIG. It is necessary to generate the non-photographed color signal from the peripheral pixel signal and to interpolate all the color signals of blank pixels in the same manner.
[0010]
For example, when focusing only on the G signal, there are G signals (G1, G2) obtained by imaging only at the positions shown in FIG. In the conventional technique, G1 ′ and G2 ′ are interpolated from the average value of the front and rear, left and right signals, and then further interpolated from the G signal obtained by interpolation as shown in FIG. A G signal can be obtained.
[0011]
Further, focusing on the B signal, as shown in FIG. 48, the B1 ′ signal is interpolated from the left and right B1 signals, and further the B1 ″ signal is interpolated from the interpolated B1 ′ signal. The same applies to the B2 signal. 49, the remaining pixels are interpolated from the interpolated B1 ′, B1 ″, B2 ′, B2 ″ as shown in FIG. 49. Also for the R signal, signals for all pixels are interpolated in the same manner as the B signal. By the above method, R, G, and B signals for all pixels can be obtained.
[0012]
[Problems to be solved by the invention]
Since the conventional imaging apparatus is configured as described above, it is necessary to mechanically drive and control the transparent flat plate member 103 by two motors in order to change the optical path two-dimensionally. For this reason, there is a problem that it is difficult to realize a pixel shift amount with high accuracy. In particular, the pixel pitch of recent solid-state imaging devices is mainly several microns, and a complicated control system is required to obtain a mechanical accuracy that is a fraction of that. In addition, since control is performed using mechanical vibration, a design that fully considers vibration and repeated life is required.
[0013]
Furthermore, the conventional signal processing for generating a high-resolution image from an image captured by the above method is simply linear interpolation, and therefore, it is not possible to obtain a resolution corresponding to the number of pixels obtained by performing pixel shifting. First, since the image in FIG. 2 on the single-plate image sensor has one color filter on each pixel, R, G, B signals corresponding to the number of pixels cannot be obtained, and only the number of pixels is obtained. The first cause is that the resolution is not obtained, and there is a problem in that when the high resolution image is obtained from the two images, the linear interpolation method cannot expect the improvement of the resolution.
[0014]
The present invention has been made in order to solve the above-described problems, and an imaging apparatus capable of realizing a high-resolution image by performing minute pixel shifting without mechanically vibrating the imaging optical system. The purpose is to obtain.
[0016]
[Means for Solving the Problems]
An imaging apparatus according to the present invention is The pixel coordinates of the imaging color signal of interest as the triangle formed by the imaging color signal that is the reference color among the imaging color signals that constitute the captured image of the imaging element before and after the displacement stored in the image storage means, and the above imaging The pixel shift direction and the pixel coordinates of the two image pickup color signals existing in the vertical direction with respect to the color signal are arranged on the horizontal axis, and the signal levels of the three image pickup color signals are arranged on the vertical axis. A first triangle having the arrangement position as a vertex is formed, and three non-imaging color signals existing in the same direction as the reference image color signal are formed as triangles formed by the non-imaging color signal that is an interpolation color. The pixel coordinates are arranged on the horizontal axis, the signal levels of the three non-imaging color signals are arranged on the vertical axis, and a second triangle whose apex is the arrangement position is formed, and the first triangle and Similarity ratio of the second triangle Performing color interpolation in consideration, and an interpolation and synthesis means for synthesizing the captured image of the imaging device before and after the displacement, is the interpolation and combining means, When color interpolation is performed on a blank pixel at a position where no image pickup pixel exists, it is determined whether or not a portion corresponding to an edge in the image passes through the blank pixel. A blank pixel is color-interpolated.
[0017]
An image pickup apparatus according to the present invention includes a compression / expansion unit that compresses an image captured by an image sensor, stores the compressed image in a compressed image memory, expands the compressed image, and outputs the compressed image to an image storage unit It is.
[0018]
The image pickup apparatus according to the present invention is provided with signal level correction means for correcting the signal level of the image pickup color signal constituting the picked-up image of the image pickup element before and after displacement stored in the image storage means.
[0019]
The imaging apparatus according to the present invention calculates the average signal level of the color signal in a predetermined area in the captured image of the image sensor before and after the displacement, and corrects the signal level of the color signal so that both average signal levels coincide. It is what I did.
[0020]
The image pickup apparatus according to the present invention controls the intensity of a magnetic field applied to a magneto-optical element disposed in an optical path that guides incident light to the image pickup element, and displaces the position of the incident light with respect to the image pickup element. .
[0021]
In the image pickup apparatus according to the present invention, pixel shifting means is arranged between the image pickup optical system and the image pickup element.
[0022]
In the image pickup apparatus according to the present invention, pixel shifting means is arranged in the previous stage of the image pickup optical system.
[0023]
The image pickup apparatus according to the present invention controls the intensity of an electric field applied to an electro-optical element disposed in an optical path that guides incident light to the image pickup element, and displaces the position of the incident light with respect to the image pickup element. .
[0024]
The image pickup apparatus according to the present invention controls the intensity of a voltage applied to a liquid crystal plate arranged in an optical path that guides incident light to the image pickup device, and displaces the position of the incident light with respect to the image pickup device.
[0025]
DETAILED DESCRIPTION OF THE INVENTION
An embodiment of the present invention will be described below.
Embodiment 1 FIG.
FIG. 1 is a block diagram showing an imaging apparatus according to Embodiment 1 of the present invention. In the figure, reference numeral 61 denotes a pixel shifting optical system (pixels) that modulates the optical path of a subject image incident on an imaging lens 1 (see FIG. 3). (Shifting means) 2 includes a rectangular charge-coupled device (hereinafter referred to as a CCD) formed in a planar shape with RGB primary color filters arranged in a Bayer-type arrangement (see FIG. 2), and an optical image is provided. An image pickup device for photoelectric conversion, 3 is an A / D conversion circuit for digitally converting an analog signal output from the image pickup device 2, 4 is a pixel shift optical system 61 and an image pickup unit control circuit for controlling the image pickup device 2, and 5 is a pixel shift. The imaging unit includes an optical system 61, an imaging device 2, an A / D conversion circuit 3, and an imaging unit control circuit 4.
[0026]
Reference numeral 6 denotes a camera control unit such as a CPU that controls the entire image pickup apparatus while communicating with the image pickup unit control circuit 4, 7 denotes an image memory (image storage means) that receives and temporarily stores a digital image signal, and 8 denotes A / D. Memory control means such as a direct memory access controller (hereinafter referred to as DMA controller) for writing a digital image signal from the conversion circuit 3 to the image memory 7, 9 performs digital image processing on the image data stored in the image memory 7. An image processing unit configured by software or an electronic circuit to be applied.
[0027]
10 is a signal level correction processing unit (signal level correction means) for performing white balance correction, black level correction, gamma correction, etc. on the image signal, and 11 is pixel interpolation for interpolating and synthesizing the captured image with high resolution by digital image processing. A synthesis processing unit (interpolation / synthesis unit) 12 removes noise generated in the optical system or circuit system from one high-resolution image obtained by the pixel interpolation / synthesis processing unit 11. An image correction processing unit including a filter, 13 is an image compression processing unit that encodes a finally obtained full-color image by an image compression method such as JPEG (Joint Photographic Experts Group) method that is an international standardization method, and 14 is an image pickup unit 5 And the image finally obtained through the image processing unit 9 is secondary storage such as liquid crystal display or flash memory. An interface unit for performing data interface or man-machine interface such as a shutter switch the transmission path, such as a storage or a serial interface or an infrared communication stage.
[0028]
FIG. 3 is a block diagram showing the internal configuration of the pixel shifting optical system 61. In FIG. 3, reference numeral 1 denotes an imaging lens (imaging optical system), 15 denotes a magnetic field generation circuit, and 16 denotes a voltage applied by the magnetic field generation circuit 15. A coil for generating a magnetic field therein, 17 is a Faraday element that produces a magneto-optic effect by the magnetic field generated in the coil 16, 18 is a polarizing plate, and 19 is a birefringent plate.
In the example of FIG. 3, the imaging lens 1 is arranged in front of the deflection plate 18, but the present invention is not limited to this. For example, the imaging lens 1 is arranged between the birefringent plate 19 and the imaging element 2. Also good.
[0029]
Next, the operation will be described.
Here, the case where it implement | achieves as a digital still camera which can image | photograph a still image is demonstrated.
First, a schematic operation of the entire imaging apparatus will be described, and detailed operations of each unit will be described later. Here, a procedure for obtaining two high-definition images having a pixel density 2 × 2 times the actual number of pixels of the CCD by photographing two images shifted by a half pixel in the 45 ° direction using the pixel shifting optical system 61 Will be described.
[0030]
When shooting in the high-definition mode is set by the photographer from the interface unit 14 and a depression of a release switch (not shown) is transmitted from the interface unit 14 to the camera control unit 6, the imaging unit 5 in the magnetic field generation circuit 15 An imaging operation is performed under a magnetic field application condition (for example, no applied magnetic field). Thus, the captured image is digitized and transmitted from the imaging unit 5 to the memory control unit 8.
[0031]
In the memory control means 8, the input image signal is stored in the image memory 7 according to a rule described later.
Next, an image photographed with a shift of ½ pixel in the −45 degree direction compared to the first photographed image due to the second applied magnetic field condition in the image sensing section control circuit 4 passes through the transmission path and is image sensing section 5. To the image processing unit 9 and similarly stored in the image memory 7.
Next, the image processing unit 9 reads the first and second captured images stored in the image memory 7 with the end of the input of the second captured image, and the signal level correction processing unit 10 White balance correction processing is performed by multiplying the signal level value of each RGB color of both images by an intensity correction coefficient.
[0032]
Next, the pixel interpolation / combination processing unit 11 performs resolution enhancement processing described later on the first and second captured images, and combines the two captured images into one high-resolution image.
Furthermore, the image data is transferred to the interface unit 14 for display on a liquid crystal display (not shown) through the subsequent image correction processing unit 12 and image compression processing unit 13 or for storage in a secondary storage medium such as a compact flash memory card. The
[0033]
Hereinafter, the pixel shift optical system 61 in the imaging unit 5, the image storage method in the image memory 7 by the memory control means 8, and the pixel interpolation / synthesis processing unit 11 will be described in detail.
First, the operation of the pixel shifting optical system 61 will be described with reference to FIGS. 4 is an operation principle diagram of FIG. 3, FIG. 5 is an explanatory diagram viewed from line AA in FIG. 4, FIG. 6 is an explanatory diagram viewed from line BB in FIG. 4, and FIG. It is explanatory drawing seen from CC line of 4.
[0034]
When light from the imaging lens 1 for forming an image is incident on the polarizing plate 18, linearly polarized light in the amplitude direction Wa shown in FIG. 5 is obtained. 4 indicates the direction of the magnetic field applied to the Faraday element 17. When the linearly polarized light in the vibration direction Wa shown in FIG. 5 enters the Faraday element 17 such as lead glass, the polarization plane of the linearly polarized light is rotated by the applied magnetic field in the magnetic field direction Ha. When linearly polarized light traveling in the magnetic field direction Ha is incident on the Faraday element 17, the plane of polarization of the transmitted light rotates, and the rotation angle θ is obtained by the following equation.
θ = R × l × H (1)
Here, l is the thickness of the Faraday element 17, H is the strength of the magnetic field, and R is a Verdet constant. The above formula (1) is described in, for example, “Optical Measurement Handbook” issued by Asakura Shoten Co., Ltd.
[0035]
In FIG. 4, the magnetic field strength at which the rotation angle θ is 0 degree is represented by Hθ. ₀ Then, when H = Hθ, linearly polarized light in the vibration direction Wa is obtained, and H = Hθ ₀ Is the vibration direction Wθ shown in FIG. ₀ Linearly polarized light is obtained.
Q shown in FIG. 7 is the optical axis of the birefringent plate 19. When linearly polarized light in the vibration direction Wa is incident on the birefringent plate 19, an ordinary ray L shown in FIG. _o Is obtained. Also, the vibration direction Wθ ₀ Is incident on the birefringent plate 19, the extraordinary ray L shown in FIG. _E Is obtained. Ordinary ray L _o And extraordinary ray L _E And P in the birefringent plate 19 shown in FIG. 4 is selected as P = PH / 2 (PH is the horizontal pixel pitch of the image sensor 2).
[0036]
The phase of the change in the magnetic field strength H applied to the Faraday element 17 is matched with the field shift pulse shown in FIG. With the above-described operation, in the imaging apparatus according to the present invention, signal charge accumulation in the A and B fields can be performed at a position separated by PH / 2 with respect to the relative position between the incident image and the pixel of the imaging device 2. . In other words, the spatial sampling region can be increased by temporally changing the strength H of the magnetic field applied to the Faraday element 17 and temporally changing the relative positional relationship between the incident optical image and the imaging element 2. .
Correspondingly, the timing of the signal readout pulse shown in FIG. 8 (3) is also shifted by a time T corresponding to PH / 2. As a result, as shown in FIG. 8 (5), the imaging apparatus according to the present invention can obtain a high pixel density image in one cycle with the A and B fields as one frame.
[0037]
In the first embodiment, the horizontal direction, that is, the coordinates with respect to the one-dimensional space, has been described as the pixel shift direction for the sake of simplification. However, the pixel shift position can be two-dimensionally performed according to the same principle. In that case, n sets of optical element groups including the polarizing plate 18, the first optical element having a magneto-optical effect, and the birefringent plate 19 are arranged in the imaging system, and between each optical element group, for example, What is necessary is just to insert n-1 quarter wave plates for converting linearly polarized light into circularly polarized light. By setting binary voltages to be applied to n optical elements having n magneto-optical effects in the n sets of optical element groups, 2 n pixel shifted images can be obtained.
[0038]
Based on the imaging principle described above, the Bayer type single-plate color imaging device shown in FIG. A pixel-shifted image can be taken according to the second applied magnetic field condition represented by a circle-shifted pixel. The storage method in the image memory 7 and the high resolution processing in the pixel interpolation / synthesis processing unit 11 in this case will be described in detail with reference to the drawings.
[0039]
10 interpolates the shaded points (hereinafter referred to as “blank pixels”) for the two image pickup pixels in FIG. 9, so that the image pickup element 2 is doubled in the main scanning direction and the sub-scanning direction, respectively. It is a schematic diagram which shows producing | generating the image which has the pixel number of 4 times for convenience. Each captured image is controlled by the memory control means 8 in the image memory 7 at a corresponding position in a matrix of 2 × 2 pixels, for example, each pixel of the first captured image is placed at the upper left pixel position of each matrix. Each pixel of the captured image is temporarily stored in the lower right pixel position of each matrix (a position shifted in the −45 degree direction compared to the first corresponding pixel).
[0040]
Actually, the DMA controller configured as the memory control means 8 can transfer data to a continuous area of the memory at a high speed, so that the physical addresses of the actual memory are arranged as shown in FIG. That is not always a good idea. In this case, each captured image is stored in a continuous area of the image memory 7, and the following pixel interpolation / combination processing unit 11 performs addressing so as to maintain the relative positional relationship of FIG. Can be processed.
[0041]
Each image stored in the image memory 7 is subjected to correction processing by the signal level correction processing unit 10. The correction processing includes, for example, white balance correction processing for correcting the sensitivity characteristic of the color filter corresponding to each color of RGB in the image sensor 2 and correcting white to the correct signal level, and the dynamic range of the signal in the low sensitivity region or the high sensitivity region. A gamma correction process or the like for correcting the image is selected as necessary.
[0042]
FIG. 11 is a flowchart showing a processing procedure of the pixel interpolation / synthesis processing unit 11. The pixel interpolation / combination processing unit 11 realizes color interpolation and high resolution by generating a non-imaging color in the photographic pixel and generating all color components in the blank pixel.
First, a G component generation process is performed on a pixel (hereinafter referred to as an R / B pixel position) in which a photographic pixel is an R component or a B component that has passed through the R color filter or the B color filter in FIG. 2 (step ST101). Further, the G component generation processing at the blank pixel position in FIG. 10 is performed (step ST102), thereby generating the G component in all the pixels in FIG.
[0043]
Next, the B component or R component generation processing at the R / B pixel position is performed (step ST103), the R color component and B color component generation processing at the G pixel position is performed (step ST104), and then in FIG. By generating the R color component and the B color component at the blank pixel position (step ST105), the R component and the B component in all the pixels in FIG. 10 are generated.
In each process, a target pixel (hereinafter referred to as a target pixel) for generating a color component is raster-scanned sequentially in the x direction (main scan) with respect to the pixel position stored in the image memory 7, and one line processing is performed. If completed, the raster scan advanced by one pixel in the y direction is repeated.
[0044]
Next, specific operations in each step will be described.
FIG. 12 shows the existing image signal components before the execution of the pixel interpolation / synthesis processing unit 11 and the positional relationship thereof, which shows the positional relationship of the center of gravity in FIG. In the drawing, the color signal with 1 subscript is obtained from the first photographed image, and the color signal with 2 subscript is obtained from the second photographed image. The pixel interpolation / combination processing unit 11 generates other color signals using the first and second photographed color signals.
[0045]
First, in the G component generation process (step ST101) at the R / B pixel position, the G component and the R component (or B component) existing in the direction perpendicular to the pixel shifting direction in the pixel shifting optical system 61 are used to obtain similarity. The G component at the target pixel is calculated. For example, when the G color signal is generated by interpolation among the non-imaging color signals at the position of the pixel whose imaging color signal is R (hereinafter referred to as the R pixel position), the pixel centered on the R pixel position shown in FIG. Among the pixels present in the window, in addition to the R color signal of interest, the position is at the coordinates of two R color signals (R1, R2) and two G color signals (G2, G3) that exist in the pixel shift direction and the vertical direction. Four imaging color signal level values are used. In FIG. 16, a triangular shape formed by these points is indicated by a broken line in a two-dimensional space including a reference direction and a signal level value axis for each color signal.
[0046]
In the first embodiment, the interpolation signal of the G color signal is obtained from the similarity ratio when the triangle formed by the imaging color signal R that is the reference color and the triangle that is formed by the non-imaging color signal G that is the interpolation color are similar figures. gR is calculated.
[0047]
Here, the triangle (R · R1 · R2) and R1 and R2 are averaged R _AVE It is clear that the triangles formed by the points R1H, R2H, and R replaced by (R · R1H · R2H) have the same geometrical area. In the image signal, the integral value (area of the geometric figure) obtained from the signal distribution corresponds to the image energy, and if the integral value of the same area is the same, the average luminance in that region is preserved and there is no visual problem. 16 can be replaced by a solid isosceles triangle (R, R1H, R2H).
[0048]
Similarly, the triangles (gR, G2, G3) formed by the G color signal as the interpolation color are replaced with isosceles triangles (gR, G2H, G3H), and isosceles triangles (R, R1H, R2H) and isosceles triangles ( By determining the interpolation signal value gR so that (gR · G2H · G3H) becomes a similar figure, color interpolation according to the ratio of image energy in a minute region is possible.
[0049]
When color interpolation by the conventional linear interpolation method is performed, the reproduced gR is a point of (G2 + G3) / 2, and the G color component shows a flat transition even though the R color component is a convex shape. , The resolution of the G color component is not sufficient and a false color is generated due to the protrusion of the R color signal. On the other hand, when this method is used, since the transition of the G color signal is reproduced well, color interpolation with high resolution and few false colors can be realized. Further, the calculation in this case is performed by using Expression (2).
gR (m, n) = G _AVE (M, n)
+ (R (m, n) -R _AVE (M, n)) / 4 (2)
However, in equation (2), the (x, y) coordinate of the R pixel position to be interpolated is (m, n), and G _AVE (M, n) is the average value of G2 and G3, R _AVE (M, n) represents an average value of R1 and R2, and is multiplied by ¼ as a line segment similarity ratio of two isosceles triangles.
[0050]
In the above example, the G color signal component is interpolated at the R pixel position. However, by replacing R with B, the G color signal component at the B pixel position can be similarly interpolated.
[0051]
In the above example, the G component at the R pixel position is color-interpolated. However, the same concept can be applied to the K color interpolation at an arbitrary J pixel position. 14 and 15 show the relationship between the reference pixel position of the reference color and the reference pixel position of the interpolation color when performing other color component interpolation at other pixel positions. FIG. 14 shows the case of B component interpolation at the R pixel position, and the above model is applied to the triangle shown in FIG. 17 for each color component. That is, since these three points form a triangle, an isosceles triangle is created by replacing the two points other than the pixel of interest with their average value, and a line is formed so that the interpolation color forms a similar isosceles triangle. Multiply by 1/2, the similarity ratio. Furthermore, it is possible to perform R component interpolation at the B pixel position by exchanging the relationship between R and B.
[0052]
FIG. 15 shows the state of the reference pixel position at the time of R component interpolation at the G pixel position. In this case, the distance from the G pixel position as the target pixel to the R pixel position in the reference direction seen from the target pixel is not an object centered on the G pixel position, but has been generated in the previous steps shown in FIG. The R component at the B pixel position is used as a point forming the interpolation color triangle, the G component at the R and B pixel positions is used as the point forming the reference color triangle, and the line segment similarity ratio of each isosceles triangle is 1 By setting / 1, it can be handled in the same way as the previous examples.
[0053]
Based on the above description, generalizing equation (2) can be described as equation (3).
K (m, n) = K _AVE (M, n)
+ (J (m, n) -J _AVE (M, n)) × Cd (3)
In Equation (3), K (m, n) is a non-imaging color signal to be interpolated at coordinates (m, n), J is an imaging color signal as a reference color at coordinates (m, n), and Cd is The line segment similarity ratio of the isosceles triangle formed by the K color with respect to the isosceles triangle formed by the J color is shown.
[0054]
In this way, it is possible to execute the color interpolation processing for the photographic pixels for one screen by repeatedly performing all color component interpolation in each pixel while sequentially scanning the pixel position to be color-interpolated in the horizontal and vertical directions. .
[0055]
Next, the G component generation (step ST102) at the blank pixel position will be described in detail. The shaded portion in FIG. 19 shows the positional relationship of the G component present in the image at this stage. When interpolating the G component at the blank pixel position, it is determined whether or not the portion corresponding to the edge in the image passes through the target pixel with reference to the surrounding pixels, and in that case, the pixel interpolation is performed in the ridge line direction of the edge line segment. By doing so, sharp edge reproduction is possible. A direction indicated by an arrow in FIG. 19 indicates a line segment direction that can be detected based on a reference pixel window including 7 × 7 pixels centered on the target pixel.
[0056]
The line segment angle detection is executed according to the following procedure.
The signal level average value Dav is calculated for the G component existing in the target pixel and the eight neighboring pixels.
Next, the signal value of each G pixel in the 7 × 7 pixel window is binarized to 1 or 0 using the calculated signal level average value Dav as a threshold value. The G pixel in the binarized window is compared with a plurality of predetermined patterns, and the line segment angle passing through the central pixel and the line segment on which side the line segment is bright or dark due to pattern matching Angle information is recognized. Based on the line segment angle information, the G component of the target pixel is calculated by linearly interpolating a plurality of pixel values existing in the ridge line direction of the line segment.
[0057]
For example, when the pixels in the 7 × 7 pixel window are binarized into the hatched portion (0) and the added portion (1) in FIG. 20, the edge of the image passes in the direction of the arrow in the figure by pattern matching. Is detected, and the G component of the target pixel is interpolated by a simple average or a weighted average with respect to the distance from the target pixel for the four pixel values indicated by the thick frame in the figure.
[0058]
By detecting the angle of the line segment and interpolating in the ridge line direction, it is possible to interpolate the edge sharply and the line segment more smoothly than in the case of component interpolation using a simple average value of eight neighboring pixels. Thereby, it is possible to give a high resolution to the observer of the image.
With the above method, the G components of all the pixels are generated with a high resolution, and the R / B component can be interpolated and generated with a high resolution by referring to this result.
[0059]
Regarding R / B component interpolation at the blank pixel position (step ST105), interpolation is performed by detecting a ridge line vector for a line segment using pattern matching similar to the G component interpolation at the blank pixel position. At this time, in order to prevent erroneous interpolation caused by the difference in the pattern matching result of the R / G / B component, the R / B component interpolation may be performed based on the line segment angle result detected during the G component interpolation. Good.
[0060]
As described above, a four-pixel high-resolution image having the number of pixels 2 × 2 times that of the image sensor 2 is generated by the pixel interpolation / synthesis processing unit 11, and the generated full-color image is generated by the image correction processing unit 12. Processing such as a noise removal filter that corrects image distortion in the optical system and the imaging system is performed, and the image compression processing unit 13 encodes the image in JPEG format and then sends the image to the interface unit 14. In this way, a high-resolution image exceeding the Nyquist frequency of the image sensor 2 is formed from the two pixel-shifted images.
[0061]
In the first embodiment, an example in which high resolution processing is performed using two images captured by shifting the pixels by 1/2 pixel in the −45 degree direction using the pixel shifting optical system 61 has been described. However, the present invention is not limited to this. . In other words, a finite number of light receiving elements of the image pickup element 2 are arranged in a plane, and incident light on the light receiving element surface is output as an integrated value, so that the spatial frequency of the captured image is limited (Nyquist frequency). It can be explained from the sampling theorem. In contrast, according to the “ultra-high-definition image acquisition method using two cameras” (Komatsu, Aizawa, Saito: Television Society Journal, Vol. 45, No. 10, pp. 1256 to 1262), imaging with an aperture ratio of 100% It is clearly shown that when the pixel-shifted images are integrated using elements, the resolution can be improved up to twice. By using this method and taking into account the aperture ratio of the actual image sensor, the shape of the light receiving element, and the like, and optimizing the shift direction, shift amount, and number of shots, the resolution in the composite image can be optimized.
[0062]
In the first embodiment, line pattern detection in 8 directions is performed using a window of 7 × 7 pixels at the time of pattern matching. However, the present invention is not limited to this. In order to reduce this, the window size may be reduced and the correlation of edge line segments between pixels in four directions in 45 degree increments or in two directions only in the horizontal and vertical directions may be used.
[0063]
Further, in the first embodiment, after the two images with the imaging condition of the pixel shifting optical system 61 changed are photographed and stored in the image memory 7, the processing in the subsequent image processing unit 9 is started. This is not limited to this. In other words, the signal level correction processing unit 10 in the image processing unit 9 can be implemented collectively for two captured images using the same white correction coefficient or gamma correction table. It may be executed individually, and in this case, the configuration is such that the first signal level correction process is performed while the second photographing operation is being performed, which leads to an increase in the speed of all processes.
[0064]
In the first embodiment, an example in which the second captured image is shifted in the −45 degree direction with respect to the first captured image has been described. It goes without saying that the same processing can be applied even if the direction or the direction of −135 degrees is shifted.
[0065]
Embodiment 2. FIG.
FIG. 21 is a block diagram showing an image pickup apparatus according to Embodiment 2 of the present invention. In the figure, the same reference numerals as those in FIG.
Reference numeral 20 denotes a fixed-length compression circuit that encodes each captured image data using a fixed-length compression method. Reference numeral 21 denotes the number of lines used when the fixed-length compression circuit 20 compresses the imaged digital signal output from the imaging unit 5. A line buffer for delay-accumulating minutes, 22 is a compressed image memory for storing image data compressed by the fixed-length compression circuit 20, and 23 is a fixed-length expansion processing unit for expanding image data stored in the compressed image memory 22. . The fixed length compression circuit 20, the line buffer 21, the compressed image memory 22 and the fixed length expansion processing unit 23 constitute compression / expansion means.
[0066]
Next, the operation will be described.
The operation of the imaging unit 5 including the pixel shifting mechanism is the same as that in the first embodiment. A captured image input from the imaging unit 5 using the pixel shifting optical system 61 is temporarily stored in the line buffer 21 for image compression by the fixed length compression circuit 20. Hereinafter, the compression operation in the fixed length compression circuit 20 will be described in detail.
[0067]
As the fixed-length encoding algorithm in the fixed-length compression circuit 20, for example, an algorithm that performs an encoding method that eliminates redundancy of adjacent pixel information for each block having 4 × 4 pixels as one unit is used. In this case, in the pixel array of FIG. 2, pixels of the same color do not exist at adjacent pixel positions, so the same color pixels are blocked in the fixed-length compression circuit 20.
[0068]
FIG. 22 is an explanatory diagram showing how the pixel array of FIG. 2 is rearranged by the fixed length compression circuit 20. The captured image data is sequentially stored in the line buffer 21 from the imaging unit 5, and the rearrangement of the pixel arrangement is started when the unprocessed data is arranged for 8 lines, and the encoding in the fixed length compression circuit 20 is performed. Pixel rearrangement is performed in units of 8 × 8 pixels, and each color 4 × 4 pixel is blocked as a unit block. The rearranged image signals are arranged by collecting G components at addresses in the upper left direction and addresses in the lower right direction, collecting B components at addresses in the lower left direction, and R components at addresses in the upper right direction. The
[0069]
An image encoding method in the fixed length compression circuit 20 will be described.
FIG. 23 is an explanatory diagram showing addresses when the fixed-length compression circuit 20 encodes the color image signals in the rearranged unit blocks. That is, FIG. 23 shows an address of m rows and n columns in the vertical direction (m, n is a natural number of 0 <m, n ≦ 4) among the unit blocks which are the same color image signal blocks of 4 pixels in the vertical and horizontal directions. This indicates that a quantization level described later is attached to the image signal.
[0070]
FIG. 24 shows the quantization level at which the intensity of the image signal of each pixel is hierarchized (quantized). In the figure, Lmin is the minimum value in the image signal intensity of 16 pixels shown in FIG. 23, Lmax is the maximum value in the image signal intensity of the same 16 pixels, and P1 is 8 etc. between the maximum value Lmax and the minimum value Lmin. 1/8 from the bottom, P2 is 1/8 from the top, Q1 is an average value of pixels having a signal intensity of Lmin to P1, and Q8 is a pixel having a signal intensity of Lmax and less than P2. Average value.
[0071]
LD is a gradation width index in the unit block, and is equal to Q8-Q1. L1 to L7 are obtained by arranging the values obtained by dividing the gradation width index LD into eight equal parts in ascending order. LA is equal to (Q1 + Q8) / 2 at the average level of image data in the unit block. φijk represents the quantization level for each pixel.
[0072]
25 and 26 are flowcharts showing the encoding procedure according to the second embodiment. The encoding procedure will be described below with reference to this flowchart.
First, the fixed length compression circuit 20 reads the image data in the unit block rearranged as shown in FIG. 22 (2) (step ST1).
Next, the signal intensity of the read image data for 4 × 4 pixels is calculated, and the values of P1, P2, Q1, Q2, LA, LD, L1 to L7 are sequentially obtained according to the following equations (step ST2 to step ST2). ST13).
[0073]
P1 = (Lmax + 7Lmin) / 8
P2 = (7Lmax + Lmin) / 8
Q1 = Ave (Xmn ≦ P1)
Q2 = Ave (Xmn> P2)
LA = (Q1 + Q8) / 2
LD = Q8-Q1
L1 = LA-3LD / 8
L2 = LA-LD / 4
L3 = LA-LD / 8
L5 = LA + LD / 8
L6 = LA + LD / 4
L7 = LA + 3LD / 8
[0074]
The expression Q1 means obtaining an average value of pixels having a signal intensity of Lmin or more and P1 or less, and the expression Q2 means obtaining an average value of pixels having a signal intensity greater than or equal to Lmax and P2.
[0075]
In this way, after sequentially obtaining the values of P1, P2, Q1, Q2, LA, LD, and L1 to L7, the fixed length compression circuit 20 sets n = 1 and m = 1 (steps ST14 and ST15), It is determined whether or not the pixel value (that is, pixel value X11) that is the signal intensity Xmn of the pixel at the address (m, n) at this time is equal to or less than L1 (step ST16).
[0076]
If the pixel value X11 is equal to or less than L1, the quantization level φijk of this pixel value is set to binary 000 (step ST17). Next, m is incremented by 1 (step ST31), and it is determined whether m is 4 or less (step ST32). When m is 4 or less, the pixel value of the pixel is compared with L1 again (step ST16).
[0077]
When m is larger than 4, n is incremented by 1 (step ST33), and it is determined whether or not the incremented n is 4 or less (step ST34). When n is 4 or less, the pixel value of the pixel is compared with L1 again (step ST16).
[0078]
If the pixel value Xmn is greater than L1, it is determined whether or not it is less than or equal to L2 (step ST18). If Xmn is less than or equal to L2, the quantization level φijk of this pixel is set to binary 001. (Step ST19). Next, m is incremented by 1 (step ST31), and it is determined whether m is 4 or less (step ST32). When m is 4 or less, the pixel value of the pixel is compared with L1 again (step ST16). When m is larger than 4, n is incremented by 1 (step ST33), and it is determined whether or not the incremented n is 4 or less (step ST34). When n is 4 or less, the pixel value of the pixel is compared with L1 again (step ST16).
[0079]
Similarly, it is determined whether the pixel value has a value between L1 and L2, between L2 and L3, between L3 and LA, between LA and L5, between L5 and L6, and between L6 and L7 (step ST16). , ST18, ST20, ST22, ST24, ST26, ST28), and according to the values, quantization levels φijk = 000, 001, 010, 011, 100, 101, 110, 111 are allocated to the pixels (steps ST17, ST19). , ST21, ST23, ST25, ST27, ST29, ST30).
[0080]
In this way, the quantization level is assigned to all the pixels in the same unit block, and the encoding is completed. The encoded data of the unit block is LA, LD, and φijk for each pixel.
[0081]
These processes are repeated for the number of unit blocks for the entire screen. When the digital signal from the A / D conversion circuit 3 is 10 bits per pixel (when the bits are not padded, it corresponds to 2 bytes), the image compression rate by fixed length encoding is 10/32.
In this way, each captured image data is subjected to fixed-length encoding in the fixed-length compression circuit 20 for each block composed of only pixels of the same color sequentially, and after the amount of image data is reduced, the memory control means 8 is stored in the compressed image memory 22.
[0082]
After the two photographed image data are accumulated in the compressed image memory 22 in this way, image processing in the image processing unit 9 is performed. First, the image data encoded in the fixed length compression circuit 20 is decoded. An image decoding method in the fixed length expansion processing unit 23 will be described with reference to the drawings. FIG. 27 is a flowchart showing the operation of the fixed length extension processing unit 23. Hereinafter, the fixed-length decoding operation of the fixed-length decompression processing unit 23 will be described with reference to this flowchart.
[0083]
When the fixed length decoding operation is started, first, the vertical coordinate value n is set to 1 (step ST40), and the horizontal coordinate value m is set to 1 (step ST41). That is, the address of the coordinate value (1, 1) in a certain unit block is designated by the operations of step ST40 and step ST41.
[0084]
Next, it is determined how many quantization levels Φijk of the designated address are (steps ST42, ST44, ST46, ST48, ST50, ST52, ST54), and the average value level LA and the respective quantization levels Φijk are determined. Based on the gradation width index LD, the signal intensity Ymn (Y11 if the pixel has the coordinate value (1, 1)) is obtained (steps ST43, ST45, ST47, ST49, ST51, ST53, ST55, ST56). ).
[0085]
In each step, the signal intensity Ymn is obtained from the average value level LA and the gradation width index value LD according to the following arithmetic expressions.
Ymn = LA-LD / 2 (Step ST43)
Ymn = LA-5LD / 14 (Step ST45)
Ymn = LA-3LD / 14 (Step ST47)
Ymn = LA-LD / 14 (step ST49)
Ymn = LA + LD / 14 (Step ST51)
Ymn = LA + 3LD / 14 (Step ST53)
Ymn = LA + 5LD / 14 (Step ST55)
Ymn = LA + LD / 2 (Step ST56)
After obtaining the signal strength of the pixel (1, 1), the pixel is moved one by one in the horizontal direction (steps ST57 and ST58), and the signal strength of the pixel (2, 1) is decoded by the same procedure (step ST57). ST42 to ST56).
[0086]
After decoding the signal strength for the uppermost pixel in the unit block in this way (step ST58), the vertical coordinate value is incremented by 1 (step ST59), and the signal is similarly applied to the pixel of the next stage. The strength is decoded (steps ST42 to ST58).
In this way, the signal strength is decoded for all the pixels in the unit block (steps ST41 to ST60), and the decoding operation is terminated.
[0087]
Next, the decoded data is subjected to a reverse rearrangement process of the data rearranged by the fixed length compression circuit 20 as shown in FIG. 28, and rearranged in the same order as when the pixel signals are read in the scanning line direction. Is stored in the image memory 7.
[0088]
As described above, the image data equivalent to the image data before the fixed length compression is stored on the image memory 7, and the image processing similar to that in the first embodiment is performed in other blocks of the image processing unit 9. And output to the interface unit 14.
[0089]
As described above, when a plurality of images are captured, each image data is compressed using the fixed length compression circuit 20, so that an image signal output from the A / D conversion circuit 3 of the imaging unit 5 is obtained. Is stored in the image memory 7, a smaller amount of data can be supplied to the data bus and stored in the storage means, and a high-speed memory control circuit for shortening the continuous shooting interval or a high-speed for at least two frames There is no need to use a writable semiconductor memory, and the apparatus can be configured at a low price.
[0090]
In other words, by using a small amount of the high-speed compressed image memory 22, it is possible to increase the apparent data storage speed in the storage means, and the first image capturing sequence is completed in a short time. By doing so, the second image capturing operation can be started in a short time. Therefore, it is possible to minimize the temporal change of the subject or the camera shake of the photographer that occurs between the two photographed images, and the same subject can be photographed with a ½ pixel shift with high accuracy. effective.
[0091]
In the second embodiment, the image memory 7 and the compressed image memory 22 are provided separately. However, the present invention is not limited to this, and the image memory 7 and the compressed image memory 22 may be provided on a semiconductor memory having the same storage area.
In the second embodiment, the fixed-length compression circuit 20, the line buffer 21, and the compressed image memory 22 are separately provided. However, the present invention is not limited to this, and two or more of these components are integrated circuits such as LSIs. It is good also as a structure included in one stone.
[0092]
Embodiment 3 FIG.
The third embodiment will be described below with reference to the drawings. The configuration and operation of the imaging unit 5 including the pixel shifting optical system 61 are the same as in the first and second embodiments. The operation of the signal level correction processing unit 10 of the image processing unit 9 which is characteristic in the third embodiment will be described in detail.
[0093]
Two image signals captured by the imaging unit 5 are stored in the image memory 7 of the image processing unit 9 via the A / D conversion circuit 3. The accumulated image data is subjected to individually applicable signal correction processing such as white balance correction processing and gamma correction processing in the signal level correction processing section 10. At this time, since the first image and the second image are taken in time series, for example, when a fluorescent lamp that emits light at a relatively low frequency of 50 to 60 MHz is taken as a light source, the light emission of the light source There may be a difference in the amount of incident light between the two images due to the relationship between the timing and the shutter time. That is, there is a difference in the average brightness of the two captured images, which may cause image quality deterioration when the pixel interpolation / synthesis processing unit 11 in the subsequent stage performs image synthesis.
[0094]
FIG. 29A shows the lightness characteristics when the same subject is photographed when the illumination light amount in the second imaging condition is reduced by about 30% compared to the first imaging condition. Each signal level when photographing under constant illumination causes a decrease in average brightness and a reduction in dynamic range when the illumination conditions change. In the figure, the Max and Min values are the maximum signal level and the minimum signal level in each image. The signal level correction processing unit 10 according to the third embodiment performs signal level correction processing for two images in addition to signal level correction processing in each image.
[0095]
That is, the maximum signal level value and the minimum signal level value in each image are detected, and processing for adjusting the other to one of the signal level characteristics is performed. Normally, automatic exposure correction or shutter speed correction is performed at the time of photographing the first image, and photographing is performed under optimum conditions, so that the second image is matched with the brightness distribution characteristic of the first image. If each signal level in the second image before and after correction is D (j) (j is each lightness level value under constant illumination) and D ′ (j), correction is first performed according to equation (4). (See FIG. 29 (b)).
D ′ (j) = D (j) + (Min1−Min2) (4)
[0096]
Subsequently, the dynamic range of the first image and the second image corrected by the equation (4) is corrected according to the equation (5), and the final correction signal level D ″ (j) of the second image is obtained. Calculate (see FIG. 29C).
D ″ (j) = D ′ (j) (Max1−Min1)
/ (Max2'-Min2 ') (5)
[0097]
By configuring the signal level correction processing unit 10 according to the above procedure, the brightness characteristics can be matched well even when the illumination conditions of the two captured images are different, and high-accuracy pixel interpolation / synthesis is possible. Processing is feasible.
When the imaging apparatus is configured as shown in FIG. 21 (Embodiment 2) and the captured image is encoded using the fixed-length compression circuit 20, LA in each code indicates an average signal level in a block composed of 16 pixels. Since it is a representative value, it can be easily explained that an equivalent effect can be realized equivalently even if all the above procedures are performed on LA. When configured in this manner, it is possible to reduce the computation scale related to the maximum / minimum signal level detection and signal level correction, which has been performed for all the pixels, to 1/16, and to realize high-speed processing. There is a special effect.
[0098]
In the third embodiment, the expressions (4) and (5) are sequentially applied to the second photographed image. However, the present invention is not limited to this, and these can be performed by one image data scan. Needless to say.
[0099]
Embodiment 4 FIG.
Hereinafter, the fourth embodiment will be described with reference to the drawings. The overall configuration in the fourth embodiment is the same as that in FIG. 1 in the first embodiment or FIG. 21 in the second embodiment. The fourth embodiment is different from the first and second embodiments in that an electro-optical element that changes the refraction phenomenon of transmitted light according to the strength of the electric field is used as the pixel shifting optical system 61. FIG. 30 is a block diagram showing a pixel shifting optical system 61 in the fourth embodiment. In the figure, 30 is an electric field generating circuit for generating an electric field by applying a voltage, and 31 is an electro-optical element.
[0100]
Next, the operation will be described.
FIG. 32, FIG. 33, and FIG. 34 show the electro-optic element 31 and the polarizing plate 18 in FIG. 31 as viewed from the directions of AA, BB, and CC, respectively. . In FIG. 31, natural light incident on the electro-optic element 31 from the imaging lens 1 for forming an image is two polarized components L orthogonal to each other as shown in FIG. _X , L _Y Can be expressed as
[0101]
When no electric field is applied to the electro-optic element 31, L in FIG. _X0 , L _Y0 The polarization component shown in FIG. _X , L _Y Is observed at a straight line. When a predetermined electric field E1 is applied to the electro-optic element 31, the electro-optic element 31 exhibits a birefringence phenomenon, and L in FIG. _X1 , L _Y1 The polarization component shown in FIG. Where L _X1 Is L as an ordinary ray in the birefringence phenomenon. _X Is observed at a straight line, L _Y1 Is L as an extraordinary ray of birefringence phenomenon. _Y Is observed at a position shifted by a distance P. However, in this case, the electric field generation circuit 30 applies an electric field E1 such that P = PH / 2 (PH is the horizontal pixel pitch of the image sensor 2) as shown in FIG.
[0102]
FIG. 8 is an operation timing chart in the imaging apparatus. 8A is a field shift pulse, FIG. 8B is the intensity of the electric field E applied to the electro-optic element 31, FIG. 8C is a signal readout pulse, and FIG. FIG. 8 (5) shows an output signal when the A and B fields are viewed as one frame. The phase of the change in the intensity of the electric field E is matched with the field shift pulse in FIG.
[0103]
With the above-described operation, the image pickup apparatus according to the fourth embodiment performs signal charge accumulation in the A and B fields at a position separated by PH / 2 with respect to the relative position between the incident image and the pixel of the image pickup device 2. Can do. Correspondingly, the timing of the signal readout pulse shown in FIG. 8 (3) is also shifted by a time T corresponding to PH / 2. As a result, as shown in FIG. 8 (5), this imaging apparatus can obtain an image with a high pixel density in one cycle with the A and B fields as one frame.
[0104]
The configuration and operation of the image processing unit 9 in the fourth embodiment can be configured using any of those described in the previous embodiments.
[0105]
Embodiment 5 FIG.
The fifth embodiment will be described below with reference to the drawings. The overall configuration in the fifth embodiment is the same as FIG. 1 in the first embodiment or FIG. 21 in the second embodiment. In the fifth embodiment, a case where a polarizing plate that produces linearly polarized light and a liquid crystal plate that changes the azimuth angle of incident light are used as the pixel shifting optical system 61 will be described as an example.
FIG. 35 is a block diagram showing the pixel shifting optical system 61 in the fifth embodiment. In FIG. 35, reference numeral 91 denotes a drive voltage generation circuit for controlling the voltage applied to the liquid crystal plate 92, and 92 is generated by the polarizing plate 18. It is a liquid crystal plate that can change the azimuth angle of linearly polarized light under voltage application conditions.
[0106]
Next, the operation will be described.
FIG. 36 is an operation principle diagram of FIG. 36, AA is the incident side of the polarizing plate 18, BB is the outgoing side of the polarizing plate 18, CC is the outgoing side of the liquid crystal plate 92, and DD is the vertical side of the outgoing side of the birefringent plate 19. The position of the direction is shown. 37 to 40 show the polarization directions of light at positions AA to DD shown in FIG.
[0107]
When an object is imaged, the light incident on the optical system is non-polarized light, and can be represented by a vertical polarization component and a horizontal polarization component as shown in FIG. The polarizing plate 18 emits light linearly polarized from the incident light. Further, as shown in FIG. 38, the polarization direction of the polarizing plate 18 is such that the azimuth is 45 ° with respect to the reference angle when the horizontal pixel array of the image sensor 2 is used as the reference angle. Tilt the axis.
[0108]
The liquid crystal plate 92 is a plate having a liquid crystal layer inside and sandwiching the liquid crystal between two electrode substrates. When a voltage (electric field) is applied from the outside, an electro-optic effect associated with a change in the molecular arrangement of the internal liquid crystal is obtained. cause. The liquid crystal plate 92 used in the fifth embodiment changes the polarization direction of incident linearly polarized light by applying a voltage to the liquid crystal plate 92 or switching the applied voltage off. is there. As an example for realizing the liquid crystal plate 92, there is an optical rotation effect by a twisted nematic (TN) type liquid crystal, which is a typical field effect type, among electro-optical effects. The operation is shown in FIG. 41 and FIG.
[0109]
The nematic liquid crystal exhibiting the TN mode has positive dielectric anisotropy, and the molecular arrangement of the element is shown in FIGS. In FIG. 41 and FIG. 42, 93 is a liquid crystal molecule. When a voltage is applied to the substrate, as shown in FIG. 41, the molecules are arranged so that the molecular long axis direction is perpendicular to both substrate surfaces, so that the linearly polarized light incident through the polarizing plate 18 is emitted as it is. . When no voltage is applied, the alignment of the liquid crystal molecules in the device is continuously twisted by 90 °, and optically causes a 90 ° optical rotation effect. Therefore, the linearly polarized light incident from the polarizing plate 18 is twisted by 90 °. The outgoing light has a 90 ° polarization direction different from that of the incident light.
[0110]
The polarization of the light emitted from the liquid crystal plate 92 is shown in FIG. In FIG. 39, c1 is a state in which a voltage is applied to the liquid crystal plate 92, and when no voltage is applied, it becomes linearly polarized light having an azimuth angle of 135 ° with respect to the reference angle as in c2. Now, let c1 linearly polarized light be the first deviation direction and c2 linearly polarized light be the second deviation direction. The light emitted from the liquid crystal plate 92 is then incident on the birefringent plate 19.
[0111]
The birefringent plate 19 is a material having a so-called birefringence in which the refractive index is not uniform depending on the polarization direction. In the imaging apparatus, when the spatial frequency of the subject exceeds the sampling frequency obtained from the pixel pitch of the imaging device 2, a repetitive noise is generated. Since it appears in the image as (aliasing noise), a quartz plate or the like is often used as an optical low-pass filter. The quartz plate, that is, the birefringent plate 19 separates incident light into ordinary rays and extraordinary rays, and the separation distance can be adjusted by the thickness of the birefringent plate 19.
[0112]
When a birefringent plate is provided so that the incident light in the first deviation direction is an ordinary ray and the incident light in the second deviation direction is an extraordinary ray, the extraordinary ray is indicated by a dotted line (see FIG. 40). ), The optical axis is different from that of ordinary light, and incident light having respective deflection directions forms images at different positions on the image sensor 2 as indicated by d1 and d2 in FIG. Further, as shown in FIG. 40, the thickness of the birefringent plate 19 is abnormal by wx / 2 which is half of the pixel pitch wx of the horizontal image sensor and wy / 2 which is half of the pixel pitch wy of the vertical image sensor. The light beam is provided different from the normal light beam.
[0113]
Therefore, the two images, the first image obtained by imaging the image formed on the image sensor 2 in the first deviation direction and the second image taken in the second deviation direction, are mutually Both horizontal and vertical are relatively shifted by 1/2 pixel.
[0114]
From the above, the image pickup apparatus outputs a drive voltage from the drive voltage generation circuit 91 to the liquid crystal plate 92 when the image pickup device 2 is driven by the control from the image pickup unit control circuit 4. When the voltage is applied to the liquid crystal plate 92, the liquid crystal molecular arrangement shown in FIG. 43 is formed, and incident light in the first deviation direction is imaged on the image sensor 2. The imaging device 2 captures an image with a drive signal from the drive voltage generation circuit 91 under the control of the imaging unit control circuit 4 to obtain a first image.
[0115]
Next, after taking the first image, the drive voltage generation circuit 91 sets the voltage applied to the liquid crystal plate 92 to zero. The liquid crystal molecular arrangement of the liquid crystal plate 92 is as shown in FIG. 42, and incident light in the second deflection direction is imaged on the image sensor 2. The image sensor 2 captures an image with a drive signal from the drive voltage generation circuit 91 to obtain a second image.
[0116]
FIG. 43 shows voltage pulses applied to the liquid crystal plate 92. The imaging unit control circuit 4 ends the imaging of the first image during the state 1 in FIG. 43 and ends the imaging of the second image during the state 2 in FIG.
By operating as described above, it is possible to obtain a first image and a second image that are shifted by 1/2 pixel both horizontally and vertically.
[0117]
The first image obtained as described above and the second image photographed with a shift of 1/2 pixel are sequentially converted into digital signals by the A / D conversion circuit 3 and then sent to the image processor 9. Sent.
[0118]
In the fifth embodiment, an example has been described in which the second captured image is shifted in the direction of 45 degrees as the direction in which the second captured image is shifted by 1/2 pixel. However, the present invention is not limited to this. Further, by appropriately combining the arrangement of the birefringent plate 18 or the voltage applied to the liquid crystal plate 92, a 135 degree direction, a -45 degree direction, a -135 degree direction, and the like can be realized.
[0119]
In the fifth embodiment, the TN type liquid crystal is exemplified as the liquid crystal plate 92 that changes the polarization direction of the incident linearly polarized light. However, if the liquid crystal plate 92 has an optical rotation effect by the photoelectric effect, the same applies. Can have an effect.
In the fifth embodiment, in the pixel shifting optical system 61 shown in FIG. 35, the imaging lens 1 is disposed between the liquid crystal plate 92 and the birefringent plate 18, but the imaging lens 1 has a pixel shifting optical system. You may arrange | position in any position in 61. FIG.
[0120]
Moreover, in all the embodiments described above, the configuration example of the digital still camera capable of continuously capturing still images has been described. However, the high-definition still image capturing mode is configured in the digital camcorder capable of capturing moving images. It goes without saying that it is possible.
[0121]
In all of the above embodiments, the configuration example in which the pixel interpolation / synthesis processing can be executed inside the imaging apparatus has been described. However, the present invention is not limited to this, and the imaging apparatus such as a personal computer or a color printer may be directly or directly connected to the imaging apparatus. You may comprise on the apparatus which can be connected via a storage medium indirectly.
[0122]
【The invention's effect】
As described above, according to the present invention, The pixel coordinates of the imaging color signal of interest as the triangle formed by the imaging color signal that is the reference color among the imaging color signals that constitute the captured image of the imaging element before and after the displacement stored in the image storage means, and the above imaging The pixel shift direction and the pixel coordinates of the two image pickup color signals existing in the vertical direction with respect to the color signal are arranged on the horizontal axis, and the signal levels of the three image pickup color signals are arranged on the vertical axis. A first triangle having the arrangement position as a vertex is formed, and three non-imaging color signals existing in the same direction as the reference image color signal are formed as triangles formed by the non-imaging color signal that is an interpolation color. The pixel coordinates are arranged on the horizontal axis, the signal levels of the three non-imaging color signals are arranged on the vertical axis, and a second triangle whose apex is the arrangement position is formed, and the first triangle and Similarity ratio of the second triangle Performing color interpolation in consideration, to synthesize the captured image of the imaging element before and after displacement Since it comprised as mentioned above, there exists an effect which can implement | achieve an image with high resolution.
In addition, when color interpolation is performed on a blank pixel at a position where there is no image pickup pixel, it is determined whether or not the portion corresponding to the edge in the image passes through the blank pixel. Further, since the blank pixels are color-interpolated, the image resolution can be increased.
[0124]
According to the present invention, there is provided a compression / expansion unit that compresses an image captured by the image sensor, stores the compressed image in the compressed image memory, decompresses the compressed image, and outputs the decompressed image to the image storage unit. Therefore, there is an effect that the continuous shooting interval can be shortened.
[0125]
According to the present invention, since the signal level correcting means for correcting the signal level of the image pickup color signal constituting the picked-up image of the image pickup element before and after the displacement stored in the image storage means is provided, the image resolution is reduced. There is an effect that can be enhanced.
[0126]
According to the present invention, the average signal level of the imaged color signal in the predetermined area in the captured image of the image sensor before and after the displacement is calculated, and the signal level of the imaged color signal is corrected so that the average signal level of both is the same. Since configured, there is an effect that the signal level can be accurately corrected.
[0127]
According to the present invention, the intensity of the magnetic field applied to the magneto-optical element disposed in the optical path for guiding the incident light to the image sensor is controlled, and the position of the incident light with respect to the image sensor is displaced. There is an effect that a minute pixel shift can be performed without mechanically vibrating the system.
[0128]
According to the present invention, since the pixel shifting means is arranged between the imaging optical system and the imaging device, there is an effect that the position of incident light with respect to the imaging device can be displaced.
[0129]
According to the present invention, since the pixel shifting means is arranged in the preceding stage of the imaging optical system, there is an effect that the pixel shifting means can be externally attached.
[0130]
According to the present invention, the configuration is such that the position of the incident light with respect to the image sensor is displaced by controlling the intensity of the electric field applied to the electro-optical element disposed in the optical path that guides the incident light to the image sensor. There is an effect that a minute pixel shift can be performed without mechanically vibrating the system.
[0131]
According to the present invention, the configuration is such that the intensity of the voltage applied to the liquid crystal plate disposed in the optical path that guides the incident light to the image pickup device is controlled to displace the position of the incident light with respect to the image pickup device. There is an effect that a minute pixel shift can be performed without mechanically vibrating the.
[Brief description of the drawings]
FIG. 1 is a configuration diagram illustrating an imaging apparatus according to Embodiment 1 of the present invention.
FIG. 2 is an explanatory diagram showing an image sensor having a Bayer type array.
3 is a configuration diagram showing an internal configuration of a pixel shifting optical system 61. FIG.
4 is an operation principle diagram of FIG. 3. FIG.
FIG. 5 is an explanatory diagram viewed from the line AA in FIG. 4;
6 is an explanatory diagram viewed from the line BB in FIG. 4;
7 is an explanatory view seen from the line CC in FIG. 4;
FIG. 8 is an explanatory diagram showing timings of various signals.
FIG. 9 is an explanatory diagram illustrating pixel positions of imaging pixels.
FIG. 10 is a schematic diagram illustrating image generation.
FIG. 11 is a flowchart showing a processing procedure of a pixel interpolation / synthesis unit 11;
12 is an explanatory diagram in which the color signal of the image captured by shifting the image sensor shown in FIG. 2 to the lower right of the diagonal ½ pixel and the previous color signal are displayed in an overlapping manner.
FIG. 13 is an explanatory diagram showing interpolation generation of color signals.
FIG. 14 is an explanatory diagram showing interpolation generation of color signals.
FIG. 15 is an explanatory diagram illustrating a relationship between a reference pixel position of a reference color and a reference pixel position of an interpolation color.
FIG. 16 is an explanatory diagram showing a relationship between a reference direction and a signal level.
FIG. 17 is an explanatory diagram showing a relationship between a reference direction and a signal level.
FIG. 18 is an explanatory diagram showing a relationship between a reference direction and a signal level.
FIG. 19 is an explanatory diagram showing a positional relationship of G components existing in an image.
FIG. 20 is an explanatory diagram showing interpolation by edge vector detection.
FIG. 21 is a block diagram showing an imaging apparatus according to Embodiment 2 of the present invention.
FIG. 22 is an explanatory diagram showing a state in which the pixel array is rearranged.
FIG. 23 is an explanatory diagram showing addresses when a color image signal in a unit block is encoded by the fixed-length compression circuit 20;
FIG. 24 is an explanatory diagram showing a quantization level for hierarchizing (quantizing) the intensity of an image signal of each pixel.
FIG. 25 is a flowchart showing an encoding procedure according to the second embodiment.
FIG. 26 is a flowchart showing an encoding procedure according to the second embodiment.
27 is a flowchart showing the operation of the fixed length extension processing unit 23. FIG.
FIG. 28 is an explanatory diagram showing a reverse rearrangement process for data rearranged by the fixed-length compression circuit 20;
FIG. 29 is an explanatory diagram showing lightness characteristics when a subject is photographed.
30 is a block diagram showing a pixel shifting optical system 61 in the fourth embodiment. FIG.
31 is an operation principle diagram of FIG. 30. FIG.
32 is an explanatory diagram viewed from the line AA in FIG. 31. FIG.
33 is an explanatory diagram viewed from the line BB in FIG. 31. FIG.
34 is an explanatory diagram viewed from the line CC of FIG. 31. FIG.
FIG. 35 is a block diagram showing a pixel shifting optical system 61 in the fifth embodiment.
36 is an operation principle diagram of FIG. 35. FIG.
FIG. 37 is an explanatory diagram showing the polarization direction of light at the position AA in FIG. 36;
38 is an explanatory diagram showing the polarization direction of light at the position BB in FIG. 36. FIG.
FIG. 39 is an explanatory diagram showing the polarization direction of light at the position CC in FIG. 36.
40 is an explanatory diagram showing the polarization direction of light at the position DD in FIG. 36. FIG.
FIG. 41 is a perspective view showing a molecular arrangement of an element.
FIG. 42 is a perspective view showing the molecular arrangement of the device.
FIG. 43 is an explanatory diagram showing liquid crystal molecule alignment.
FIG. 44 is a block diagram illustrating a conventional imaging device.
FIG. 45 is a perspective view illustrating a mechanism of the imaging apparatus.
FIG. 46 is an explanatory diagram showing G signals (G1, G2) obtained by imaging.
FIG. 47 is an explanatory diagram showing interpolation of a G signal.
FIG. 48 is an explanatory diagram showing interpolation of B1 ′ signals from left and right B1 signals.
FIG. 49 is an explanatory diagram showing interpolation of the remaining pixels from B1 ′, B1 ″, B2 ′, B2 ″.
[Explanation of symbols]
DESCRIPTION OF SYMBOLS 1 Imaging lens (imaging optical system) 2 Imaging element 3 A / D conversion circuit 4 Imaging part control circuit 5 Imaging part 6 Camera control part 7 Image memory (image storage means) 8 Memory control means 9 Image processing unit, 10 signal level correction processing unit (signal level correction unit), 11 pixel interpolation / synthesis processing unit (interpolation / synthesis unit), 12 image correction processing unit, 13 image compression processing unit, 14 interface unit, 15 magnetic field generation Circuit, 16 coils, 17 Faraday element, 18 polarizing plate, 19 birefringent plate, 20 fixed length compression circuit (compression / expansion means), 21 line buffer (compression / expansion means), 22 compressed image memory (compression / expansion means) , 23 fixed length expansion processing unit (compression / expansion means), 30 electric field generation circuit, 31 electro-optic element, 61 pixel shift optical system (pixel shift means), 91 drive voltage generation circuit, 9 Liquid crystal plate, 93 liquid crystal molecules.

Claims (9)

Pixel shift means for optically displacing the position of incident light with respect to the image sensor by half a pixel, image storage means for storing captured images of the image sensor before and after displacement by the pixel shift means, and image storage means stored in the image storage means Among the captured color signals constituting the captured image of the image sensor before and after the displacement, as the triangle formed by the captured color signal that is the reference color, the pixel coordinates of the focused captured color signal and the pixel with respect to the captured color signal The pixel coordinates of the two image pickup color signals existing in the shift direction and the vertical direction are arranged on the horizontal axis, and the signal levels of the three image pickup color signals are arranged on the vertical axis, and these arrangement positions are set as vertices. The first triangle is formed, and the pixel coordinates of the three non-imaging color signals existing in the same direction as the reference image color signal are plotted on the horizontal axis as the triangle formed by the non-imaging color signal that is the interpolation color. Arrangement As well as, by placing the signal level of the three non-imaging color signal on the vertical axis, forming a second triangle having the apexes of these positions, similar to the first triangle and the second triangle performing color interpolation in consideration of the ratio, and an interpolation and synthesis means for synthesizing the captured image of the imaging device before and after the displacement, the interpolation and combining means color interpolating blank pixels locations where the imaging pixels are not present In this case, it is determined whether a portion corresponding to an edge in an image passes through a blank pixel, and when passing through the blank pixel, the blank pixel is color-interpolated in the ridge line direction of the edge line segment. .   2. A compression / decompression unit that compresses a captured image of an image sensor, stores the compressed image in a compressed image memory, decompresses the compressed image, and outputs the decompressed image to an image storage unit. Imaging device.   3. The imaging according to claim 1, further comprising signal level correction means for correcting a signal level of a color image signal constituting a picked-up image of the image pickup device before and after displacement stored in the image storage means. apparatus.   The signal level correction means calculates the average signal level of the color signal in a predetermined area in the captured image of the image sensor before and after the displacement, and corrects the signal level of the color signal so that both average signal levels match. The imaging apparatus according to claim 3, wherein:   2. The pixel shifting means controls the intensity of a magnetic field applied to a magneto-optical element disposed in an optical path that guides incident light to the image sensor, and displaces the position of the incident light with respect to the image sensor. The imaging device according to claim 4.   6. The image pickup apparatus according to claim 5, wherein pixel shifting means is disposed between the image pickup optical system and the image pickup element.   6. The image pickup apparatus according to claim 5, wherein a pixel shifting unit is disposed in a front stage of the image pickup optical system.   The pixel shifting means controls the intensity of an electric field applied to an electro-optical element disposed in an optical path that guides incident light to the image sensor, and displaces the position of the incident light with respect to the image sensor. The imaging device according to claim 4.   The pixel shifting means controls the intensity of a voltage applied to a liquid crystal plate disposed in an optical path that guides incident light to the image pickup device, and displaces the position of the incident light with respect to the image pickup device. Item 5. The imaging device according to any one of Items 4 to 4.

Images