JP2012029199A - Stereoscopic imaging device and imaging method for the same - Google Patents

Abstract

PROBLEM TO BE SOLVED: To suppress increase in device scale.SOLUTION: A stereoscopic imaging device comprises: two imaging parts for taking images of the same subject; a parallax calculation part which detects corresponding points between images taken by the two imaging parts and calculates parallax information between the images taken by the two imaging parts: and a composition processing part which, with reference to the respective view points of the two imaging parts and on the basis of the parallax information and the images taken by the two imaging parts, composes images including a larger number of pixels than those in the taken images, thereby generating two systems of images including a large number of pixels.

Description

  The present invention relates to a stereoscopic imaging device and an imaging method thereof.

  In recent years, stereoscopic image devices have been actively developed in order to enhance the power and presence of images. As a technique for generating a stereoscopic image, there is known a technique in which two imaging devices for the left channel (L) and the right channel (R) are arranged on the left and right, and the subject is simultaneously photographed by the two imaging devices. Yes. Further, as a technique for displaying a stereoscopic image, a left channel (L) image and a right (R) channel image are alternately displayed on a single display screen for each pixel, and a kamaboko-shaped lens is provided in a predetermined manner. The left and right eyes of the viewer using a special optical system such as a lenticular lens arranged at intervals, a parallax barrier with fine slits arranged at a predetermined interval, and a patterning phase difference plate with regularly arranged fine polarizing elements A technique is known in which only the left channel (L) image is visible to the viewer's left eye and only the right channel (R) image is visible to the right eye.

  Furthermore, in recent years, a technique for generating a high-definition video using a plurality of cameras has been developed. For example, a thin color camera having a sub-pixel resolution by combining four sub-cameras including an imaging lens, a color filter, and a detector array has also been proposed (for example, see Patent Document 1). As shown in FIG. 18, the thin color camera includes four lenses 22 a to 22 d, four color filters 25 a to 25 d, and a detector array 24. The color filter 25 includes a filter 25a that transmits red light (R), filters 25b and 25c that transmit green light (G), and a filter 25d that transmits blue light (B). Take green and blue images. With this configuration, a high-resolution composite image is formed from two green images having high sensitivity in the human visual system, and a full-color image can be obtained by combining red and blue.

Special table 2007-520166

  However, the conventional technology for capturing a stereoscopic image requires two imaging devices for the left channel (L) and the right channel (R) in order to obtain one stereoscopic image. There is a problem of becoming a scale. For example, when the imaging devices described in Patent Document 1 are used for the left channel and the right channel, the two imaging devices are arranged side by side. Since the imaging apparatus described in Patent Literature 1 includes four sub cameras, eight sub cameras that are twice that number are required.

  The present invention has been made in view of such circumstances, and an object thereof is to provide a stereoscopic imaging apparatus and an imaging method thereof in which an increase in apparatus scale is suppressed.

(1) The present invention has been made to solve the above-described problems, and the stereoscopic imaging apparatus according to the present invention can cope with two imaging units that capture the same subject and the captured images of the two imaging units. Based on the parallax information and the captured images of the two imaging units based on the viewpoint of each of the two imaging units, and a parallax calculation unit that detects points and calculates parallax information of the captured images of the two imaging units And a synthesis processing unit for synthesizing an image having a larger number of pixels than the photographed image and generating two systems of images having the larger number of pixels.

(2) The stereoscopic imaging apparatus of the present invention is the above-described stereoscopic imaging apparatus, wherein the imaging unit captures an optical system that forms an image of a subject on an imaging surface, and captures the subject that is imaged on the imaging surface. An image pickup device that generates a video signal, and the one image pickup unit has a position of the image pickup device relative to the optical system that is higher or lower than a half of the image pickup pixel of the image pickup device compared to the other image pickup unit. It is characterized by being shifted to either.

(3) Moreover, the stereoscopic imaging device of the present invention is the above-described stereoscopic imaging device, which includes three or more imaging units, and among the imaging units, between the side-by-side imaging units, the optical system The position of the image sensor is shifted either up or down by half of the imaging pixel of the image sensor, and the position of the image sensor with respect to the optical system is between the vertically aligned image capture units among the image capture units. It is characterized in that it is shifted to the left or right by half of the imaging pixel of the image sensor.

(4) Further, the stereoscopic imaging device of the present invention is the stereoscopic imaging device described above, wherein the four imaging units are arranged at the vertices of a square along which each side is either horizontal or vertical, and the parallax The calculation unit calculates parallax information of captured images of two imaging units arranged at adjacent vertices of the square among the four imaging units, and the synthesis processing unit is configured to combine the videos. The parallax information is used for horizontal and vertical parallax correction of the captured video to be performed.

(5) Moreover, the stereoscopic imaging device of the present invention is the stereoscopic imaging device described above, and includes at least three imaging units, and the composition processing unit is based on the viewpoints of at least three imaging units. Based on the parallax information and at least two captured images of the imaging unit, an image having a larger number of pixels than the captured image is synthesized to generate at least three lines of the image having the larger number of pixels.

(6) Moreover, the imaging method of the present invention is an imaging method in a stereoscopic imaging apparatus including two imaging units that capture the same subject, and detects corresponding points between the captured images of the two imaging units, Based on the first process of calculating parallax information of the captured images of the two imaging units and the viewpoint of each of the two imaging units, the captured video is based on the parallax information and the captured images of the two imaging units. And a second step of synthesizing an image having a larger number of pixels and generating two systems of images having the larger number of pixels.

  According to the present invention, an increase in device scale can be suppressed.

1 is an overview diagram showing an overview of a stereoscopic imaging apparatus 10 according to a first embodiment of the present invention. It is a schematic block diagram which shows the structure of the three-dimensional imaging device 10 in the embodiment. It is a figure which shows the example of arrangement | positioning of the imaging lens and imaging device in the embodiment. It is a figure which shows another example of arrangement | positioning of the imaging lens and imaging device in the embodiment. It is a schematic block diagram which shows the structure of the parallax calculation part 21 in the embodiment. It is a figure showing reference picture RG in the embodiment. It is a figure which shows the reference | standard image BG in the embodiment. It is a figure which shows the structure of the reference attention block RB in the same embodiment. It is a figure which shows the structure of the reference attention block BB in the embodiment. It is a flowchart explaining the calculation process of the parallax data in the embodiment. It is a schematic block diagram which shows the function structure of the high resolution synthetic | combination process part 20 in the embodiment. It is a figure explaining operation | movement of the synthetic | combination process part 906 in the embodiment. It is a general-view figure which shows the general appearance of the three-dimensional imaging device 111 in 2nd embodiment of this invention. It is a schematic block diagram which shows the function structure of the three-dimensional imaging device 111 in the embodiment. It is a figure explaining arrangement of an image pick-up lens and an image sensor in the embodiment. It is a schematic block diagram which shows the function structure of the multi-view high resolution synthetic | combination process part 121 in the embodiment. It is a figure explaining operation | movement of the multi-view synthesis process part 131 in the embodiment. It is a schematic block diagram which shows the structure of the camera which produces | generates the conventional high definition image | video.

[First embodiment]
A first embodiment of the present invention will be described in detail with reference to the drawings. An overview of a stereoscopic imaging apparatus 10 according to an embodiment of the present invention is shown in FIG. 1, and a schematic block diagram showing its functional configuration is shown in FIG. As illustrated in FIGS. 1 and 2, the stereoscopic imaging device 10 includes an imaging unit 101, an imaging unit 102, a parallax calculation unit 21, and a high-resolution composition processing unit 20 that are arranged side by side. The imaging unit 101 includes an imaging lens 11-1 and an imaging element 12-1. The imaging unit 102 includes an imaging lens 11-2 and an imaging element 12-2. Note that the imaging unit 101 and the imaging unit 102 are arranged so that their optical axes are parallel so as to capture the same subject.

  Since the imaging lens 11-1 and the imaging lens 11-2 have the same configuration except for the installed position, the description of the common part will be described as the imaging lens 11. In addition, since the image pickup device 12-1 and the image pickup device 12-2 have the same configuration except for the position where they are installed, the description of the common parts will be described as the image pickup device 12. The imaging lens 11 forms an image of light from the subject on the imaging element 12. As shown in FIG. 2, the imaging element 12 (optical system) may be composed of a single convex lens or an optical system composed of a plurality of lenses. The image sensor 12 is a CMOS image sensor or the like, and photoelectrically converts the image formed and outputs it as a video signal. Hereinafter, the video signal output from the imaging device 12-1 of the imaging unit 101 is referred to as a video signal R, and the video signal output from the imaging device 12-2 of the imaging unit 102 is referred to as a video signal L.

  Two systems of video signals (video signal R and video signal L) output by the imaging unit 101 and the imaging unit 102 are input to the parallax calculation unit 21 and the high-resolution composition processing unit 20. The parallax calculation unit 21 searches for corresponding points between the two input video signals, and R reference parallax data RS based on the viewpoint of the imaging unit 101 and the viewpoint of the imaging unit 102 based on the search result. The L reference parallax data LS is calculated and output to the high resolution composition processing unit 20. The high-resolution composition processing unit 20 synthesizes the two input video signals based on the parallax data (disparity information), and outputs a right-eye video signal RC and a left-eye video signal LC.

  FIG. 3 is a diagram illustrating an arrangement example of the imaging lens and the imaging element. In FIG. 3, the x axis is taken in the horizontal direction (lateral direction), the y axis is taken in the vertical direction (up and down direction), and the z axis is taken in the depth direction. That is, FIG. 3 shows the arrangement of the imaging lens and the imaging element when the stereoscopic imaging device 10 is viewed from the front. As shown in FIG. 3, the imaging lens 11-1 and the imaging lens 11-2 are disposed at the same position in the y-axis direction. On the other hand, the image sensor 12-1 is displaced from the image sensor 12-2 by py / 2 in the y-axis direction (vertical direction). Here, py is the length of the pixel in the image sensor 12 in the y-axis direction. That is, the image sensor 12-1 and the image sensor 12-2 are arranged so as to be shifted by half the height of the pixel of the image sensor 12 in the y-axis direction (vertical direction).

  As a result, in the imaging unit 101, the position of the imaging element with respect to the imaging lens is shifted upward by half of the imaging pixel of the imaging element, as compared with the imaging unit 102. Conversely, the image sensor 12-1 may be arranged to be shifted downward by py / 2 in the y-axis direction (vertical direction) from the image sensor 12-2. In that case, the arrangement order of the pixels when the composition processing is performed in the high-resolution composition processing unit 20 described later is reversed.

  FIG. 4 is a diagram illustrating another arrangement example of the imaging lens and the imaging element. In FIG. 4, the x axis is taken in the horizontal direction (lateral direction), the y axis is taken in the vertical direction (up and down direction), and the z axis is taken in the depth direction. In the example illustrated in FIG. 4, the imaging lens 11-1 is arranged to be shifted by py / 2 in the y-axis direction (vertical direction) from the imaging lens 11-2. On the other hand, the image sensor 12-1 and the image sensor 12-2 are disposed at the same position in the y-axis direction. That is, the imaging lens 11-1 and the imaging lens 11-2 are arranged so as to be shifted in the y-axis direction (vertical direction) by half the pixel height of the imaging element 12.

  Thereby, in the imaging unit 101, the position of the imaging element 12 with respect to the imaging lens 11 is shifted upward by half of the shooting pixel of the imaging element 12 compared to the imaging unit 102. Conversely, the image sensor 12-1 may be arranged to be shifted downward by py / 2 in the y-axis direction (vertical direction) from the image sensor 12-2. In that case, the arrangement order of the pixels when the composition processing is performed in the high-resolution composition processing unit 20 described later is reversed.

  Next, the parallax calculation unit 21 will be described in detail with reference to FIG. FIG. 5 is a schematic block diagram illustrating the configuration of the parallax calculation unit 21. The parallax calculation unit 21 calculates parallax data from the video signal R output from the imaging unit 101 in FIG. 1 and the video signal L output from the imaging unit L102. The parallax calculation unit 21 includes coordinate conversion units 31 and 32, a right camera parameter storage unit 30R, a left camera parameter storage unit 30L, and a corresponding point search unit 33.

The right camera parameter storage unit 30R holds camera parameters including internal parameters such as a focal length and lens distortion parameters specific to the imaging unit 101, and external parameters representing the positional relationship between the two imaging units 101 and 102. . Similarly, the left camera parameter storage unit 30L holds camera parameters specific to the imaging unit 102.
The coordinate conversion unit 31 performs geometric conversion (coordinate conversion) on the video represented by the video signal R output from the imaging unit 101 by a known method for the purpose of placing the video of the imaging unit 101 and the imaging unit 102 on the same plane. To make the epipolar line parallel. At this time, the coordinate conversion unit 31 uses the camera parameters stored in the right camera parameter storage unit 30R. The coordinate conversion unit 32 geometrically converts the video represented by the video signal L output from the imaging unit 102 by an known method for the purpose of placing the video images of the imaging unit 101 and the imaging unit 102 on the same plane. Is parallelized. At this time, the coordinate conversion unit 32 uses the camera parameters stored in the left camera parameter storage unit 30L.

  The corresponding point search unit 33 searches for a corresponding pixel between the images in which the epipolar lines are parallelized by the coordinate conversion unit 31 and the coordinate conversion unit 32, and generates disparity data representing the parallax between the viewpoint of the imaging unit 101 and the viewpoint of 102. Ask. The corresponding point search unit 33 is composed of blocks that calculate two types of parallax data. One is an R reference parallax calculation unit 34, and the other is an L reference parallax calculation unit 35. The R reference parallax calculation unit 34 corresponds to each pixel of the reference image, using the image of the imaging unit 101 with the epipolar line parallelized as a reference image and the image of the imaging unit 102 with the epipolar line parallelized as a reference image. The reference video pixel is searched to calculate R-standard parallax data RS. The L reference parallax calculation unit 35 uses the video of the imaging unit 102 with the epipolar line parallelized as a reference video and the video of the imaging unit 101 with the epipolar line parallelized as a reference video, and references corresponding to each pixel of the standard video The pixel of the video is searched to calculate L reference parallax data. The R-standard parallax calculation block 34 and the L-standard parallax calculation block 34 have the same corresponding point search operation except that the standard video and the reference video are reversed.

  Next, a processing operation for searching for a corresponding pixel by the R reference parallax calculation unit 34 will be described with reference to FIGS. FIG. 6 is a diagram illustrating the reference image RG. FIG. 7 is a diagram illustrating the reference image BG. As described above, the epipolar lines are parallelized in both the reference image RG and the standard image BG. In order to obtain a pixel (corresponding pixel) on the reference image RG corresponding to a pixel on the standard image BG, a method of moving the target pixel on the standard image BG will be described with reference to FIG. The R reference parallax calculation unit 34 sets a block centered on the target pixel on the reference image BG (hereinafter referred to as a reference target block BB) from the upper left end (reference start block BS) of the reference image BG to the right along the line. If the moved reference attention block BB reaches the right end of the line, the reference attention block BB is moved pixel by pixel from the left end under one line to the right along the line. This is repeated until the block at the lower right corner of the reference image BG (reference end block BE) is reached.

  Next, the processing operation for the R-standard parallax calculation unit 34 to search for a block on the reference image RG similar to a certain target block (standard target block BB) on the standard image BG shown in FIG. 7 will be described with reference to FIG. The description will be given with reference. The R reference parallax calculation unit 34 first refers to a block (reference start block RS) on the reference image RG having the same coordinates as the coordinates (x, y) of the reference target block BB on the reference image BG shown in FIG. The target block RB is set, and thereafter, the reference target block is moved pixel by pixel along the line to the right. Then, as shown in FIG. 6, it is moved from the reference start block RS to a reference end block RE that is separated to the right by the search range. The search range (search range) here is a value corresponding to the maximum parallax of the photographed subject, and the shortest distance to the subject for which parallax data can be calculated is determined by the set search range. The R reference parallax calculation unit 34 searches the reference attention block BB on the reference image BG illustrated in FIG. 7 for the reference attention block described above.

  Next, a reference attention block determination method similar to the reference attention block by the R reference parallax calculation unit 34 will be described with reference to FIGS. 8 and 9. FIG. 9 is a diagram illustrating the configuration of the reference block of interest BB. The reference attention block BB is a block having a size of horizontal M × vertical N around the target pixel on the reference image BG. FIG. 8 is a diagram illustrating a configuration of the reference attention block RB. The reference attention block RB is a block having a size of horizontal M × vertical N around the target pixel on the reference image RG. In order to represent an arbitrary pixel on the reference target block RB shown in FIG. 8 and the reference target block BB shown in FIG. 9, pixel values of coordinates (i, j) where the horizontal direction is i and the vertical direction is j are respectively R Let (i, j), T (i, j).

The R reference parallax calculation unit 34 calculates a similarity for each combination of the reference attention block BB and the reference attention block RB, and determines a reference attention block RB similar to each reference attention block BB. As a method for obtaining the similarity, generally used SAD (Sum of Absolute Difference) is used. SAD calculates the absolute value of the difference between R (i, j) and T (i, j) for all the pixels of the block as in the similarity determination formula shown in equation (1), and sums them (S _SAD ). The R reference parallax calculation unit 34 _selects a reference attention block having the smallest value of S _{SAD in} the expression (1) among the reference attention blocks RG in the search range on the reference image RG corresponding to a certain reference attention block BB. , It is determined that it is similar to the reference attention block. Then, the R reference parallax calculation unit 34 sets a pixel at the center of the reference attention block RB similar to the reference attention block BB as a pixel corresponding to the attention pixel at the center of the reference attention block BB.

  The processing operation for searching for a corresponding pixel by the L reference parallax calculation unit 35 is substantially the same as that of the R reference parallax calculation unit 34, but the search range is different. In the case of the L reference parallax calculation unit 35, the reference start block RS is a block that is separated from the same coordinates as the reference target block BB by a search range to the left side, and the reference end block RE is a block at the same coordinates as the reference target block BB. It is.

  In the above description, the parallax data calculation method is described for each processing unit. Next, description will be given along the sequence of input pixels with reference to the processing flows shown in FIGS. The R reference parallax calculation unit 34 first sets the reference attention block at the head (reference start block BS) of the reference image BG (FIG. 7) (step S900). Then, all pixel values of the reference target block BB are read from the reference image BG (step S901). Next, the R standard parallax calculation unit 34 sets the block in the reference image RG (FIG. 6) having the same coordinates as the standard target block BB, that is, the head of the reference image RG (reference start block RS) as the reference target block RB. (Step S902). Then, all pixel values of the reference attention block RB are read from the reference image RG (step S903). The R reference parallax calculation unit 34 calculates and stores the SSAD values of the pixel values of the read reference attention block BB and reference attention block RB according to the equation (1) (step S904).

  Next, the R reference parallax calculation unit 34 determines whether or not the search range has ended (step S905). If the search range has not ended, the R reference parallax calculation unit 34 moves the reference block of interest to the right along the line direction by one pixel (step S906). ), Step S903 and Step S904 are performed again. These steps S903 to S906 are repeated while the reference block of interest RB is within the search range, and all SSAD values within the search range are calculated. Based on these calculation results, the R reference parallax calculation unit 34 detects the reference block of interest RB having the smallest SSAD value (step S907). Here, the minimum SSAD value calculated in step S907 is not necessarily a correct similar block. For example, the parallax cannot be detected correctly when the reference target block BB has no pattern (texture) or contour as a feature point, or when the search area on the reference image RG is an occlusion area. Whether the parallax has been detected correctly can be determined from how small the minimum SSAD value is.

  Therefore, the R reference parallax calculation unit 34 compares the minimum SSAD value with the threshold (step 908), and when the SSAD value is equal to or lower than the threshold (that is, when the similarity is high), the center of the reference block of interest BB (reference The difference between the x coordinate of the target pixel on the image BG) and the x coordinate of the center of the detected reference target block RB (the corresponding pixel on the reference image RG) is output as parallax data of the target pixel (step S909). ). On the other hand, when the SSAD value exceeds the threshold (that is, when the degree of similarity is low), the R reference parallax calculation unit 34 determines that the parallax has not been detected, and sets the parallax data to 0 or a unique value as the error flag. And output (step 910).

  Finally, the R reference parallax calculation unit 34 determines whether or not the reference block of interest BB has reached the reference end block BE, that is, whether or not the processing has ended (step S911). The block BB is moved to the right by one pixel along the line direction (step S912), and steps S901 to S910 are performed again. If it is determined in step S911 that the process has ended, the process ends. In this way, the steps S901 to S910 are repeated until the reference target block BB becomes the search end block of the reference image BG, and the parallax data of each pixel on the reference image BG is obtained.

  In the above description, as an example of the processing method of the corresponding point search unit 33 in FIG. 5, the pixel on the reference image similar to the target pixel on the base image is searched using the SAD similarity evaluation function. The parallax data may be obtained using any technique as long as it is a technique for searching for similar pixels on the standard image and the reference image.

  Next, the detailed configuration and operation of the high resolution composition processing unit 20 shown in FIG. 2 will be described with reference to FIGS. 11 and 12. FIG. 11 is a schematic block diagram showing a functional configuration of the high resolution composition processing unit 20. As shown in FIG. 11, the high-resolution composition processing unit 20 includes a left-eye composition unit 908 that generates a left-eye video signal LC, a right-eye composition unit 909 that generates a right-eye video signal RC, and a right camera parameter storage unit 902R. And a left camera parameter storage unit 902L. Each of the left-eye composition unit 908 and the right-eye composition block unit 909 includes an alignment correction processing unit 901, a correction processing unit 903, and a composition processing unit 906. Since the basic operation is the same for the left-eye synthesis unit 908 and the right-eye synthesis unit 909 except that the combination of input parallax data is different, the operation of the left-eye synthesis unit 908 will be described here. Description is omitted.

  In the left-eye combining unit 908, the video signal R of the imaging unit 101 is input to the alignment correction processing unit 901, and the video signal L of the imaging unit 102 is input to the correction processing unit 903. The alignment correction processing unit 901 corrects the lens distortion of the video represented by the video signal R based on the camera parameters indicating the lens distortion state of the imaging unit 101 stored in the right camera parameter storage unit 902R, and then the parallax calculation unit 21 Based on the L reference parallax data LS input from, and the camera parameters indicating the orientation and orientation of the image capturing unit 101 stored in the right camera parameter storage unit 902R, each pixel of the image with the lens distortion corrected is an image of the image capturing unit 102 ( Alignment is performed so as to capture the same subject position as a pixel having the same coordinates in the image corrected by the correction processing unit 903.

  However, in this alignment, when the image pickup unit 101 described in FIGS. 3 and 4 is compared with the image pickup unit 102, the position of the image pickup device 12-1 with respect to the image pickup lens 11-1 is taken by the image pickup device 12-1. Correction is not performed for a state of being shifted upward by half of the pixel. That is, the pixel at the coordinate (x, y) as the processing result of the alignment correction processing unit 901 is the pixel at the coordinate (x, y) as the processing result of the correction processing unit 903 and the pixel at the coordinate (x, y−1). The subject position between is captured. On the other hand, the correction processing unit 903 corrects the lens distortion of the video represented by the video signal L based on the camera parameters indicating the lens distortion state stored in the left camera parameter storage unit 902L.

  That is, the alignment correction processing unit 901 and the correction processing unit 903 perform epipolar line parallelization using camera parameters and parallax correction using L-reference parallax data LS. Here, the parallelization of the epipolar line can be performed by a known method. In addition, as the parallax correction, the alignment correction processing unit 901 shifts the pixels of the video obtained by parallelizing the epipolar line to the video represented by the video signal R by the amount corresponding to the parallax indicated by the L reference parallax data LS. Move to. For example, if the L reference parallax data LS at the coordinates (x, y) is d, the pixel at the coordinates (x + d, y) is moved to the coordinates (x, y). Note that the video signal R and the camera parameters of the imaging unit 101 are input to the correction processing unit 903 of the right-eye synthesis unit 909, and the video signal L and the camera of the imaging unit 102 are input to the alignment correction processing unit 901. A parameter and R reference parallax data RS are input. Further, in the parallax correction in the alignment correction processing unit 901 of the right-eye synthesis unit 909, the pixel is moved to the right by the amount of parallax indicated by the parallax data RS.

  Next, the operation of the composition processing unit 906 of the left-eye composition unit 908 will be described with reference to FIG. In FIG. 12, the horizontal axis indicates the spread (size) of the space in the y-axis direction, and the vertical axis indicates the light amplitude (light intensity). In the graph 40a, an image of the subject is formed by the imaging lenses 11-1 and 11-2 among the imaging elements 12-1 and 12-2 of the imaging units 101 and 102, and is incident on a certain vertical row of pixels. Shows the distribution of light. A graph denoted by reference numeral 40e indicates an output distribution of the alignment correction processing unit 901 corresponding to the pixel on which the light of the graph 40a is incident among the pixels of the imaging element 12-1 of the imaging unit 101. A graph denoted by reference numeral 40f indicates an output distribution of the correction processing unit 903 corresponding to the pixel on which the light of the graph 40a is incident among the pixels of the imaging element 12-2 of the imaging unit 102. The graph of reference numeral 40g shows the distribution of the output of the synthesis processing unit 906 with respect to the distribution of reference numerals 40e and 40f. Here, for simplicity of explanation, the relationship between the graphs will be described ignoring the effects of correction by the alignment correction processing unit 901 and the correction processing unit 903.

  Of the vertical lines in the figure, the solid line is the boundary line of the pixels of the image sensor 12-1 of the imaging unit 101, and the broken line is the boundary line of the pixels of the image sensor 12-2 of the imaging unit 102. That is, reference numerals 40b and 40c in the figure are pixels of the imaging unit 101 and the imaging unit 102, respectively, and the relative positional relationship is shifted by an offset indicated by an arrow 40d. In addition, it is preferable to set this offset to the magnitude | size of the half of the pixel (code | symbol 40b, code | symbol 40c) of the image pick-up element 12-1 and the image pick-up element 12-2. A half-pixel offset makes it possible to generate the highest definition image. Since the image sensors 12-1 and 12-2 integrate the light intensity in units of pixels, when an image of a subject indicated by reference numeral 40a is taken by the image pickup element 12-1, a video signal having a light intensity distribution indicated by reference numeral 40e. When an image is captured by the image sensor 12-2, a video signal having a light intensity distribution indicated by reference numeral 40f is obtained.

  The composition processing unit 906 synthesizes an image by alternately arranging the output of the alignment correction processing unit 901 and the output of the correction processing unit 903 in the y-axis direction, and compared with the outputs of the imaging units 101 and 102. A left-eye video signal LC in which the resolution in the y-axis direction is doubled is generated. At this time, the imaging unit 101 considers that the position of the imaging device 12-1 with respect to the imaging lens 11-1 is shifted upward by half of the imaging pixel of the imaging device 12-1, as compared with the imaging unit 102. The pixel b of the correction processing unit 903 derived from the imaging unit 102 and having the same coordinates as the pixel a of the alignment correction processing unit 901 derived from the imaging unit 101 is disposed below the pixel a. That is, when the origin is the upper right of the image, the x-axis is the right direction, and the y-axis is the lower direction, the pixel at the coordinates (x, y) being output from the alignment correction processing unit 901 is The pixel at the coordinates (2x, 2y) being outputted, and the pixel at the coordinates (x, y) being outputted by the correction processing unit 903 is a pixel at the coordinates (2x, 2y + 1) being outputted by the synthesis processing unit 906.

The composition processing unit 906 can reproduce a high-definition image close to the graph 40a indicated by reference numeral 40g by combining the two images in this way.
Note that the synthesis processing unit 906 of the right-eye synthesis unit 909 operates in the same manner as the synthesis processing unit 906 of the left-eye synthesis unit 908, but is a pixel having the same coordinates as the pixel c of the correction processing unit 903 derived from the imaging unit 101. Therefore, the pixel d of the alignment correction processing unit 901 derived from the imaging unit 102 is arranged below the pixel c.

  The above combining operation is executed by the left eye combining unit 908 and the right eye combining unit 909. As a result, the left-eye compositing unit 908 outputs a left-eye video signal LC, which is a high-definition video signal taken from the position of the imaging unit 102 (ie, viewed from the left eye), and the right-eye compositing unit 909 captures an image. A right-eye video signal RC that is a high-definition video signal shot from the position of the unit 102 (that is, viewed from the right eye) is output. By displaying the left-eye and right-eye images on the stereoscopic display device, it is possible to display a stereoscopic image having a resolution twice that of the imaging units 101 and 102.

  As described above, the left-eye video signal LC and the right-eye video signal RC output from the stereoscopic imaging device 10 have twice the resolution as compared with the video signals R and L output from the imaging units 101 and 102. ing. Therefore, the stereoscopic imaging device 10 can generate a stereoscopic video image having a resolution twice that of a stereoscopic imaging device having a resolution equivalent to that of the video signal output by the imaging unit 101.

[Second Embodiment]
A second embodiment of the present invention will be described in detail with reference to the drawings. In addition, about the part of the same function as 1st embodiment, description is abbreviate | omitted using the same code | symbol. FIG. 13 is an overview diagram illustrating an overview of the stereoscopic imaging apparatus according to the present embodiment. As shown in FIG. 13, the stereoscopic imaging apparatus 111 according to the present embodiment differs from the first embodiment in the number of imaging units, from two eyes (reference numerals 101, 102) to four eyes (reference numerals 101, 102, 103, 104). ) And increasing. That is, the stereoscopic imaging device 111 includes an imaging unit 101, an imaging unit 102, an imaging unit 103, and an imaging unit 104. The imaging unit 101 includes an imaging lens 11-1 and an imaging element 12-1. Similarly, the imaging unit 102 includes an imaging lens 11-2 and an imaging element 12-2, the imaging unit 103 includes an imaging lens 11-3 and an imaging element 12-3, and the imaging unit 104 includes an imaging lens. 11-4 and an image sensor 12-4.

  FIG. 14 is a schematic block diagram illustrating a functional configuration of the stereoscopic imaging device 111 according to the present embodiment. The stereoscopic imaging device 111 includes imaging units 101, 102, 103, 104, a parallax calculation unit 21, and a multi-view high-resolution composition processing unit 121. The video signals R and L output from the imaging unit 101 and the imaging unit 102 are input to the multi-view high resolution synthesis processing unit 121 and the parallax calculation unit 21. The video signal R ′ output from the imaging unit 103 and the video signal L ′ output from the imaging unit 104 are input to the multi-view high-resolution composition processing unit 121. The processing of the parallax calculation unit 21 is the same as that of the first embodiment, and calculates the R reference parallax data and the L reference parallax data, and outputs them to the multi-view high resolution synthesis processing unit 121. The high-resolution synthesis processing unit 121 performs synthesis processing on the input four video signals based on the two parallax data, and outputs a right-eye video signal RC and a left-eye video signal LC.

  FIG. 15 is a diagram illustrating the arrangement of the imaging lens and the imaging element in the present embodiment. In FIG. 15, the x axis is taken in the horizontal direction (lateral direction), the y axis is taken in the vertical direction (up and down direction), and the z axis is taken in the depth direction. That is, FIG. 15 shows the arrangement of the imaging lens and the imaging element when the stereoscopic imaging device 111 is viewed from the front. As shown in FIG. 15, the imaging lens 11-1 and the imaging lens 11-2 are disposed at the same position in the y-axis direction. The imaging lens 11-3 and the imaging lens 11-4 are disposed at the same position in the y-axis direction. The imaging lens 11-1 and the imaging lens 11-3 are disposed at the same position in the x-axis direction. The imaging lens 11-2 and the imaging lens 11-4 are disposed at the same position in the x-axis direction. A distance Dx from the center of the imaging lens 11-1 to the center of the imaging lens 11-2 is equal to a distance Dy from the center of the imaging lens 11-1 to the center of the imaging lens 11-3. As described above, in the present embodiment, the imaging units 101 to 104 are arranged at the vertices of a square along which each side is either horizontal or vertical.

  On the other hand, the image sensor 12-1 is displaced from the image sensor 12-2 by py / 2 in the y-axis direction (vertical direction). In addition, the image sensor 12-3 is arranged to be shifted by py / 2 in the y-axis direction (vertical direction) from the image sensor 12-4. Here, py is the length of the pixel in the image sensor 12 in the y-axis direction. Further, the image pickup device 12-1 is arranged to be shifted to the left by px / 2 in the x-axis direction (lateral direction) from the image pickup device 12-3. Further, the image sensor 12-2 is arranged to be shifted to the left by px / 2 in the x-axis direction (lateral direction) from the image sensor 12-4. Here, px is the length of the pixel in the x-axis direction in the image sensor.

As a result, in the imaging unit 101, the position of the imaging element with respect to the imaging lens is shifted upward by half of the imaging pixel of the imaging element, as compared with the imaging unit 102. Similarly, in the imaging unit 103, as compared with the imaging unit 104, the position of the imaging element with respect to the imaging lens is shifted upward by half of the imaging pixel of the imaging element. Further, in the imaging unit 101, the position of the imaging element with respect to the imaging lens is shifted to the left by half of the imaging pixel of the imaging element, as compared with the imaging unit 103. Similarly, in the imaging unit 102, the position of the imaging element with respect to the imaging lens is shifted to the left by half of the imaging pixel of the imaging element, as compared with the imaging unit 104.
Here, an example in which the imaging elements are displaced is shown, but the imaging lens may be displaced as shown in FIG. 4 described in the first embodiment.

  Next, the detailed configuration and operation of the multi-view high resolution composition processing unit 121 will be described with reference to FIGS. 16 and 17. The multi-view high-resolution composition processing unit 121 includes a left-eye multi-view composition unit 130 that generates a left-eye high-definition image, a right-eye multi-view composition unit 132 that generates a right-eye high-definition image, and the imaging unit 101. A camera parameter storage unit 902 for storing 104 camera parameters. The left-eye composition unit 130 and the right-eye composition block 130 include an alignment correction processing unit 901, a correction processing unit 903, a vertical / horizontal alignment correction processing unit 904, a vertical alignment correction processing unit 905, and a multi-view synthesis processing unit 131, respectively. It comprises. Since the left eye synthesis unit 130 and the right eye synthesis unit 132 have the same basic operation except that the combination of the input video signal and the parallax data is different, the operation of the left eye synthesis block 131 will be described here.

  The video signal R output from the imaging unit 101 is input to the alignment correction processing unit 901. Similar to the first embodiment, the alignment correction processing unit 901 performs the correction processing and alignment of the video represented by the video signal R, the camera parameters of the imaging unit 101 stored in the camera parameter storage unit 902, and the L reference parallax. Based on the data LS, an image from the viewpoint of the imaging unit 102 is generated. The video signal L output from the imaging unit 102 is input to the correction processing unit 903. Similar to the first embodiment, the correction processing unit 903 performs correction processing of the video represented by the video signal L based on the camera parameters of the imaging unit 102 stored in the camera parameter storage unit 902.

  The video signal R ′ output from the imaging unit 103 is input to the vertical / horizontal alignment correction processing unit 904. The vertical / horizontal alignment correction processing unit 904 performs correction processing and alignment of the video represented by the video signal R ′ based on the camera parameters of the imaging unit 103 stored in the camera parameter storage unit 902 and the L reference parallax data LS. An image from the viewpoint of the imaging unit 102 is generated. At this time, the distance Dx between the center of the imaging lens 11-1 of the imaging unit 101 and the center of the imaging lens 12-1 of the imaging unit 102 is equal to the center of the imaging lens 11-1 of the imaging unit 101 and the imaging unit 103. Since it is equal to the distance Dy from the center of the imaging lens 12-3, L reference parallax data LS is used as vertical parallax data. That is, the vertical / horizontal alignment correction processing unit 904 performs alignment by applying the L reference parallax data LS in the vertical direction and the horizontal direction. For example, the pixel value of the coordinate (x, y) in the image aligned by the vertical / horizontal alignment correction processing unit 904 is the video signal output by the imaging unit 103 when the L reference parallax data LS at the coordinate is d. Is a pixel value of coordinates (x + d, yd) in the image obtained by correcting the image represented by the camera parameter.

  The video signal L ′ output from the imaging unit 104 is input to the vertical alignment correction processing unit 905. The vertical alignment correction processing unit 905 performs correction processing and alignment of the video represented by the video signal L ′ based on the camera parameters of the imaging unit 104 stored in the camera parameter storage unit 902 and the L reference parallax data LS. An image from the viewpoint of the imaging unit 102 is generated. That is, the vertical alignment correction processing unit 905 performs alignment by applying the L reference parallax data LS in the vertical direction. For example, the pixel value of the coordinates (x, y) in the image aligned by the vertical alignment correction processing unit 905 is the video signal output by the imaging unit 104 when the L reference parallax data LS at the coordinates is d. Is a pixel value at coordinates (x, yd) in the image obtained by correcting the image represented by the camera parameter.

  Next, the operation of the multi-eye synthesis processing unit 131 in the left-eye synthesis unit 130 and the right-eye synthesis unit 132 will be described with reference to FIG. In the first embodiment, high-resolution composition processing using two images is performed, but in this embodiment, the multi-eye composition processing unit 131 includes four systems obtained by the four imaging units 101, 102, 103, and 104. High-resolution synthesis using the video signal. The multi-view synthesis processing unit 131 of the left-eye synthesis unit 130 generates and outputs a left-eye video signal LC ′ that is a signal representing video from the viewpoint of the imaging unit 102. In addition, the multi-view synthesis processing unit 131 of the right-eye synthesis unit 132 generates and outputs a right-eye video signal RC ′ that is a signal representing video from the viewpoint of the imaging unit 101. The four-line high-resolution synthesis is the same as the principle described in the light intensity distribution of FIG. 12, but more specifically, the resolution of the four imaging units 101, 102, 103, and 104 is VGA (640 × 480). A description will be given of a case where a high-resolution composition process is performed on a quad-VGA pixel (1280 × 960 pixels) that is four times the number of pixels.

  As shown in FIG. 17, four pixels adjacent to Quad-VGA pixels (1280 × 960 pixels) can obtain a high-resolution image by assigning and synthesizing pixels captured by different imaging units. It is. Here, the image pickup device 11-1 of the image pickup unit 101 is arranged so as to be shifted upward by half a pixel with respect to the image pickup device 11-2 of the image pickup unit 102. On the other hand, the image pickup device 11-4 of the image pickup unit 104 is shifted to the left by half a pixel with respect to the image pickup device 11-2 of the image pickup unit 102. ing. Therefore, in accordance with the direction in which the image sensor is displaced, the corrected image MR derived from the imaging unit 101, the corrected image ML derived from the imaging unit 102, and the imaging unit 103, as shown in FIG. The multi-eye synthesis processing unit 131 uses pixels G11, G21, G31, and G41, which are pixels having the same coordinates in each of the corrected image MR ′ derived from and the corrected image ML ′ derived from the imaging unit 104. Arrange as follows. That is, the pixel G31 is arranged on the right side of the pixel G11, the pixel G21 is arranged on the lower side of the pixel G11, and the pixel G41 is arranged on the right side of the pixel G32.

  In the left-eye composition unit 131, the corrected video MR derived from the imaging unit 101 is an image generated by the alignment correction processing unit 901, and the corrected video ML derived from the imaging unit 102 is the correction processing unit. The corrected image MR ′ derived from the image capturing unit 103 is an image generated by the image capturing unit 103, and the corrected image ML ′ derived from the image capturing unit 104 is the vertical position. This is an image generated by the alignment correction processing unit 905. In the right-eye composition unit 132, the corrected video MR derived from the imaging unit 101 is an image generated by the correction processing unit 903, and the corrected video ML derived from the imaging unit 102 is processed by the alignment correction processing unit 901. The corrected image MR ′ derived from the imaging unit 103, which is the generated image, is the image generated by the vertical alignment correction processing unit 905, and the corrected video ML ′ derived from the imaging unit 104 is the vertical / horizontal alignment correction. It is an image generated by the processing unit 904.

  The above combining operation is executed by the left eye combining unit 130 and the right eye combining unit 132. As a result, the left-eye combining unit 130 outputs a high-definition image obtained by combining four images captured from the position of the imaging unit 102 (that is, viewed from the left eye), and the right-eye combining unit 132 outputs the imaging unit 101. Compared with the output of the image pickup unit, four images taken from the position (that is, viewed from the right eye) are combined, and an image having a resolution four times as high is output. By displaying the high-definition video for the left eye and the right eye on the stereoscopic display device, it is possible to display the high-definition stereoscopic video.

  As described above, the left-eye video signal LC ′ output from the stereoscopic imaging device 111 and the right-eye video signal RC ′ have four times higher resolution than the video signals output from the imaging units 101, 102, 103, and 104. Have. That is, the left-eye video signal LC ′ output from the stereoscopic imaging device 111 and the right-eye video signal RC ′ are a video signal obtained by combining the video signal output from the imaging unit 101 and the video signal output from the imaging unit 103. In comparison, it has four times the resolution. Therefore, the stereoscopic imaging device 111 has a device scale equivalent to that of a stereoscopic imaging device having a resolution equivalent to a video signal obtained by combining the video signal output from the imaging unit 101 and the video signal output from the imaging unit 103, and twice that size. 3D images can be generated.

  In the present embodiment, an example in which the video signals input to the parallax calculation unit 21 are two video signals output by the imaging units 101 and 102 has been described. However, the number of input video signals can be increased. For example, it is also possible to input four systems of video output from the imaging units 101, 102, 103, and 104, calculate four systems of parallax with each as a reference, and perform multi-view high-resolution synthesis with each parallax data. is there. In this case, a high-definition video of a plurality of viewpoints other than those for the right eye and the left eye is output, and as a result, a multi-view stereoscopic video can be generated without resolution deterioration.

  In addition, FIG. 2 shows a program for realizing the functions of the parallax calculation unit 21 and the high-resolution synthesis processing unit 20 or the parallax calculation unit 21 and the multi-view high-resolution synthesis processing unit 121 in FIG. 14 on a computer-readable recording medium. The processing of each unit may be performed by recording, reading the program recorded on the recording medium into a computer system, and executing the program. Here, the “computer system” includes an OS and hardware such as peripheral devices.

  The “computer-readable recording medium” refers to a storage device such as a flexible medium, a magneto-optical disk, a portable medium such as a ROM and a CD-ROM, and a hard disk incorporated in a computer system. Furthermore, the “computer-readable recording medium” dynamically holds a program for a short time like a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line. In this case, a volatile memory in a computer system serving as a server or a client in that case, and a program that holds a program for a certain period of time are also included. The program may be a program for realizing a part of the functions described above, and may be a program capable of realizing the functions described above in combination with a program already recorded in a computer system.

  The embodiment of the present invention has been described in detail with reference to the drawings. However, the specific configuration is not limited to this embodiment, and includes design changes and the like within a scope not departing from the gist of the present invention.

DESCRIPTION OF SYMBOLS 10, 111 ... Three-dimensional imaging device 11-1, 11-2, 11-3, 11-4 ... Imaging lens 12-1, 12-2, 12-3, 12-4 ... Imaging element 20 ... High-resolution composition process part DESCRIPTION OF SYMBOLS 21 ... Parallax calculation part 30R ... Right camera parameter storage part 30L ... Left camera parameter storage part 31, 32 ... Coordinate conversion part 33 ... Corresponding point search part 34 ... R reference | standard parallax calculation part 35 ... L reference | standard parallax calculation part 101,102, DESCRIPTION OF SYMBOLS 103,104 ... Image pick-up part 121 ... Multi-view high-resolution composition processing part 131 ... Multi-eye composition processing part 901 ... Positioning correction processing part 902 ... Camera parameter storage part 902R ... Right camera parameter storage part 902L ... Left camera parameter storage part 903 ... Correction processing unit 904 ... Vertical / horizontal alignment correction processing unit 905 ... Vertical alignment correction processing unit 906 ... Composition processing unit 908, 130 ... Left eye composition Part 909, 132 ... right eye composition part

Claims (6)

Two imagers that image the same subject;
A parallax calculation unit that detects corresponding points between the captured images of the two imaging units and calculates parallax information of the captured images of the two imaging units;
Based on the viewpoints of each of the two image capturing units, based on the parallax information and the captured images of the two image capturing units, an image having a larger number of pixels than the captured image is synthesized, and two systems of images having the larger number of pixels A stereoscopic processing apparatus comprising: a synthesis processing unit that generates The imaging unit
An optical system that forms an image of a subject on the imaging surface;
An image sensor that generates a signal of a photographic image of a subject imaged on the imaging surface,
The one image pickup unit is characterized in that the position of the image pickup device with respect to the optical system is shifted up or down by half of the image pickup pixel of the image pickup device as compared with the other image pickup unit. Item 3. The stereoscopic imaging device according to Item 1. Including three or more imaging units,
Among the imaging units, between the side-by-side imaging units, the position of the imaging element with respect to the optical system is shifted up or down by half of the imaging pixel of the imaging element,
The position of the said image pick-up element with respect to the said optical system has shifted | deviated to either the right or left by the half of the imaging pixel of this image pick-up element between the image pick-up parts arranged in a line among the said image pick-up parts. The three-dimensional imaging device described in 1. Four of the imaging units are arranged at the vertices of a square along which each side is either horizontal or vertical,
The parallax calculation unit calculates parallax information of captured images of two imaging units arranged at adjacent vertices of the square among the four imaging units,
The stereoscopic imaging apparatus according to claim 1, wherein the synthesis processing unit uses the parallax information for parallax correction in the horizontal direction and the vertical direction of the captured video performed when the video is synthesized. Comprising at least three imaging units;
The synthesis processing unit synthesizes a video having a larger number of pixels than the captured video based on the parallax information and the captured video of the at least two imaging units based on the viewpoints of each of the at least three imaging units, and at least The three-dimensional imaging device according to claim 1, wherein an image having a large number of pixels of three systems is generated. An imaging method in a stereoscopic imaging device including two imaging units for imaging the same subject,
A first step of detecting corresponding points between the captured images of the two imaging units and calculating parallax information of the captured images of the two imaging units;
Based on the viewpoints of each of the two image capturing units, based on the parallax information and the captured images of the two image capturing units, an image having a larger number of pixels than the captured image is synthesized, and two systems of images having the larger number of pixels And a second process for generating the imaging method.

Images