JP6684118B2 - Binocular stereoscopic image generation device and its program - Google Patents

Description

  The present invention relates to a binocular stereoscopic image generation device that generates a binocular stereoscopic image by using an integral photography (IP: Integral Photography) type stereoscopic imaging device, and a program thereof.

  In the conventional integral photography method, a single camera images a subject through a lens array in which a plurality of minute lenses are arranged. At this time, since the camera images the focal plane of the lens array, each minute lens forming the lens array has the same function as a minute camera. As a result, the image of the subject captured through the lens array becomes an element image group in which minute images (element images) corresponding to the positions of the minute lenses are arranged. This element image group is an image in which light ray information from the subject is recorded, and the number of light rays that can be recorded depends on the resolution of the camera. For this reason, a high-resolution camera is required for IP imaging.

  Further, conventionally, a technique for generating an IP type element image group by using computer graphics (CG) has been studied. In this conventional technique, a virtual three-dimensional space is generated in a computer, a three-dimensional subject and a background (set) are reproduced as in the real world, and rays from the subject are traced in the virtual three-dimensional space. Generate a group of elemental images. The methods described in the following Patent Documents 1 and 2 and Non-Patent Documents 1 and 2 are known as methods for generating an element image by CG. These elemental image generation methods trace the light rays from the subject through a depth control lens that controls the light rays from the subject. In these methods, it is necessary to capture images with the camera from the observer side as many times as the number of pixels of the element image group. That is, since the element image generation method of Non-Patent Document 1 uses the ray tracing method for each pixel, it is equivalent to a camera that captures an image for each pixel, and the number of captures is the same as the number of pixels of the element image group. .

  Therefore, in order to further increase the speed, as a conventional technique, there is also known a method of capturing an image of a CG object with a camera by orthographic projection and converting these images into a group of element images (Patent Document 3 and Non-Patent Document 3). In such an element image conversion method, the number of times of imaging with the camera is the number of pixels forming the element image.

  Further, in the method described in Patent Document 4, it is possible to observe an artificially converted IP stereoscopic image as a stereoscopic stereoscopic image by adding the resolution characteristics of the IP method to the image captured by the stereoscopic stereoscopic camera. I am trying.

Japanese Patent No. 4567422 Re-table 00/059235 gazette (Japanese Patent Application No. 2000-608621) JP, 2011-234142, A JP, 2005-104107, A

Spyros S. Athineos et al., “Physical modeling of a microlens array setup for use in computer generated IP,” Proc. Of SPIE-IS & T Electronic Imaging, Vol. 5664 pp.472-479, 2005 Nakajima Kan et al., “High-speed 3D display image creation method using the principle of Integral Photography”, Institute of Image Information and Television Engineers Vol. 54, No. 3, pp.420-425 (2000) M.Katayama et al., “A method for converting three-dimensional models into auto-stereoscopic images based on integral photography”, Proc of SPIE-IS & T Vol.6805 68050Z-1 68050Z-8

  In the IP method, the resolution decreases as the distance from the display screen increases, so that the stereoscopic image positioned in front of or behind the display screen is blurred. At this time, the degree of blurring of the stereoscopic image cannot be known unless the image is actually captured. However, in the IP method, imaging cannot be performed unless the lens array and the depth control lens are arranged at appropriate positions. As a result, in the IP method, imaging is repeatedly performed in order to obtain a desired degree of blurring in a stereoscopic image, which is extremely time-consuming.

  On the other hand, the twin-lens stereoscopic method can generate a stereoscopic image in a short time as compared with a general elemental image conversion method as well as a general IP method. Therefore, for example, since the imaging time can be shortened in order to confirm the blurring of the stereoscopic image in the IP system, the use of the twin-lens stereoscopic system will be examined.

  In the first place, the IP method has a great difference in reproduction of motion parallax and resolution characteristics of a stereoscopic image as compared with the two-eye stereoscopic method. Under the condition that the observer observes at a specific position without moving, the influence of motion parallax disappears. That is, under the conditions described above, the IP method has a large difference only in the resolution characteristics of the stereoscopic image as compared with the twin-lens stereoscopic method. From this, by generating a twin-lens stereoscopic image from the IP stereoscopic image, the video creator can confirm blurring of the IP stereoscopic image on the stereoscopic display device.

  Therefore, it is an object of the present invention to provide a twin-lens stereoscopic image generation apparatus and a program thereof that can efficiently generate a twin-lens stereoscopic image from an IP stereoscopic image.

  In view of the above-mentioned problems, the binocular stereoscopic image generation device according to the present invention uses an integral photography type stereoscopic imaging device including an optical element array in which optical elements are two-dimensionally arranged and an imaging element, A twin-lens stereoscopic image generation apparatus that generates a twin-lens stereoscopic image, and is configured to include a viewpoint position determination unit, a pixel position calculation unit, an image sensor control unit, and a pixel value allocation unit.

According to this configuration, the binocular stereoscopic image generation device determines the viewpoint positions corresponding to both eyes of the observer who observes the binocular stereoscopic image by the viewpoint position determination means.
The twin-lens stereoscopic image generation apparatus uses the pixel position calculation means to perform optical conversion from the viewpoint position based on the viewpoint position, the optical element information regarding the preset optical element array, and the image sensor information regarding the preset image sensor. The pixel position for the right eye and the pixel position for the left eye of the image pickup element at which the straight line passing through the optical principal point of the element intersects are calculated.

In the binocular stereoscopic image generation device, the image sensor control unit drives the pixels at the right eye pixel position and the left eye pixel position among all the pixels of the image sensor.
In the binocular stereoscopic image generation device, the pixel value allocating unit acquires the pixel values of the driven pixels at the right eye pixel position and the left eye pixel position, and allocates the pixel values as the pixel values of the binocular stereoscopic image. . Then, the pixel value assigning unit generates a parallax image representing the binocular parallax of the observer as a binocular stereoscopic image, and outputs the generated parallax image to the stereoscopic display device of the polarization glasses system or the time division glasses system.
As described above, the binocular stereoscopic image generation device handles only the pixels at the right-eye pixel position and the left-eye pixel position necessary for generating a binocular stereoscopic image, out of all the pixels of the image sensor, so that the calculation amount is reduced. Can be suppressed.

  Further, in view of the above-mentioned problems, the twin-lens stereoscopic image generating apparatus according to the present invention is arranged between an optical element array in which optical elements are two-dimensionally arranged, an imaging element, and the optical element array and a subject. Binocular stereo for generating a binocular stereoscopic image by using an integral photography type stereoscopic imaging device including a shielding means having two openings corresponding to both eyes of an observer who observes a binocular stereoscopic image The image generating apparatus is configured to include a viewpoint position determining unit, a pixel position calculating unit, an aperture position controlling unit, and a pixel value assigning unit.

According to this configuration, the binocular stereoscopic image generation device determines the viewpoint positions corresponding to both eyes of the observer by the viewpoint position determination means.
The twin-lens stereoscopic image generation apparatus uses the pixel position calculation means to perform optical conversion from the viewpoint position based on the viewpoint position, the optical element information regarding the preset optical element array, and the image sensor information regarding the preset image sensor. The pixel position for the right eye and the pixel position for the left eye of the image pickup element at which the straight line passing through the optical principal point of the element intersects are calculated.

In the binocular stereoscopic image generation device, the aperture position control unit drives the shielding unit so that the light ray from the subject enters the right eye pixel position and the left eye pixel position of the image sensor through the aperture.
In the binocular stereoscopic image generation device, the pixel value allocating unit acquires the pixel values of the pixels at the right eye pixel position and the left eye pixel position and allocates the pixel values as the pixel values of the binocular stereoscopic image. Then, the pixel value assigning unit generates a parallax image representing the binocular parallax of the observer as a binocular stereoscopic image, and outputs the generated parallax image to the stereoscopic display device of the polarization glasses system or the time division glasses system.
As described above, the binocular stereoscopic image generation device handles only the pixels at the right-eye pixel position and the left-eye pixel position necessary for generating a binocular stereoscopic image, out of all the pixels of the image sensor, so that the calculation amount is reduced. Can be suppressed.

According to the present invention, the following excellent effects are exhibited.
Since the binocular stereoscopic image generation device according to the present invention handles only the pixels at the right eye pixel position and the left eye pixel position necessary for generating a binocular stereoscopic image among all the pixels of the image sensor, the calculation amount is It is possible to suppress, and it is possible to efficiently generate a binocular stereoscopic image.

FIG. 1 is an overall configuration diagram of a binocular stereoscopic image generation system according to a first embodiment of the present invention. It is a block diagram which shows the structure of the viewpoint position determination means of FIG. It is a block diagram which shows the structure of the pixel position calculation means of FIG. 6A and 6B are explanatory diagrams illustrating calculation of pixel positions in the first embodiment of the present invention, in which FIG. 9A illustrates an optical system of the IP image pickup device, and FIG. 9B illustrates a virtual optical system. FIG. 3 is an explanatory diagram illustrating driving of the image sensor in the first embodiment of the present invention. It is a block diagram which shows the structure of the pixel value allocation means of FIG. FIG. 3 is an explanatory diagram illustrating generation of a binocular stereoscopic image corresponding to a lenticular stereoscopic display device in the first embodiment of the present invention. FIG. 3 is an explanatory diagram illustrating generation of a binocular stereoscopic image compatible with an IP stereoscopic display device in the first embodiment of the present invention. FIG. 3 is an explanatory diagram illustrating a right-eye image area and a left-eye image area included in a binocular stereoscopic image in the first embodiment of the present invention. 3 is a flowchart showing the operation of the twin-lens stereoscopic image generation device in FIG. 1. It is a whole block diagram of the twin-lens stereoscopic image generation system which concerns on 2nd Embodiment of this invention. FIG. 11 is an explanatory diagram illustrating control of the shielding unit in the second embodiment of the present invention, showing a case where the IP image pickup device does not include a convex lens. FIG. 9 is an explanatory diagram illustrating control of the shielding unit in the second embodiment of the present invention, showing a case where the IP imaging device includes a convex lens. 12 is a flowchart showing the operation of the twin-lens stereoscopic image generation device in FIG. 11. It is a block diagram which shows the modification of a viewpoint position determination means.

  Hereinafter, each embodiment of the present invention will be described in detail with reference to the drawings as appropriate. In addition, in each embodiment, the same reference numerals are given to the same means and members, and description thereof is omitted.

(First embodiment)
[Overall Configuration of Binocular Stereoscopic Image Generation System]
The configuration of the twin-lens stereoscopic image generation system 1 according to the first embodiment of the present invention will be described with reference to FIG.
The binocular stereoscopic image generation system 1 is a system for generating a binocular stereoscopic image of a subject 90 by using an IP type stereoscopic imaging device (hereinafter, IP imaging device) 10. As illustrated in FIG. 1, the binocular stereoscopic image generation system 1 includes an IP imaging device 10 and a binocular stereoscopic image generation device 20. In the present embodiment, the IP imaging device 10 has the built-in twin-lens stereoscopic image generation device 20.

The IP image pickup device 10 picks up an image of the subject 90, and includes a convex lens 101, a lens array (optical element array) 102, and an image pickup element 103.
The convex lens 101 is, for example, an objective lens that collects light rays from the subject 90 on the lens array 102.
The lens array 102 is an array of convex lenses (optical elements) 102a arranged two-dimensionally (for example, vertically and horizontally or in a barrel shape). The light rays that have passed through this lens array 102 enter the image sensor 103.
The image pickup element 103 picks up an image of the light beam that has passed through the lens array 102. In the present embodiment, the image sensor 103 drives only the pixel 103a at a predetermined position according to the control signal input from the twin-lens stereoscopic image generation device 20, and only the driven pixel 103a captures an image. Then, the image sensor 103 outputs the captured image to the binocular stereoscopic image generation device 20.
Since the IP imaging device 10 has a general configuration, further description will be omitted.

[Configuration of Binocular Stereoscopic Image Generation Device]
The configuration of the twin-lens stereoscopic image generation device 20 will be described below.
The binocular stereoscopic image generation device 20 generates a binocular stereoscopic image of the subject 90 using the IP imaging device 10. As shown in FIG. 1, the binocular stereoscopic image generation device 20 includes a viewpoint position determination unit 201, a pixel position calculation unit 202, an image sensor control unit 203, and a pixel value allocation unit 204.

The viewpoint position determination means 201 determines the viewpoint position corresponding to both eyes of an observer who observes a binocular stereoscopic image. As shown in FIG. 2, the viewpoint position determination unit 201 includes a viewpoint position storage unit 2011. Then, the viewpoint position determination unit 201 reads out the viewpoint position stored in the viewpoint position storage unit 2011 and outputs it to the pixel position calculation unit 202.
The viewpoint position storage unit 2011 is a general storage device such as a memory or an HDD (Hard Disk Drive) that stores the viewpoint position in advance. This viewpoint position is manually set by the user of the twin-lens stereoscopic image generation system 1, for example.

  The pixel position calculation means 202 is based on the viewpoint position, the lens array information, and the image sensor information, and the right eye pixel position and the left pixel position of the image sensor 103 where a straight line passing from the viewpoint position to the optical principal point of the convex lens 102a intersects. The pixel position for the eye is calculated. As shown in FIG. 3, the pixel position calculation unit 202 includes an image sensor information storage unit 2021, an imaging lens array information storage unit 2022, and a pixel position calculation unit 2023.

  The image pickup device information storage unit 2021 is a general storage device such as a memory or an HDD that stores image pickup device information regarding the image pickup device 103 included in the IP image pickup device 10 in advance. For example, the image sensor information includes the pixel interval and the number of pixels of the image sensor 103. This image sensor information is manually set by the user of the twin-lens stereoscopic image generation system 1, for example.

  The imaging lens array information storage unit 2022 is a general storage device such as a memory or an HDD that stores in advance lens array information regarding the lens array 102 included in the IP imaging device 10. The lens array information includes, for example, the focal length, the interval, the number, and the arrangement of the convex lenses 102a forming the lens array 102. This lens array information is manually set by the user of the twin-lens stereoscopic image generation system 1, for example.

  The pixel position calculation unit 2023 includes the viewpoint position input from the viewpoint position determination unit 201, the image pickup device information stored in the image pickup device information storage unit 2021, and the lens array information stored in the image pickup lens array information storage unit 2022. With reference to, the pixel positions of the image sensor 103 that contribute as an image to the observer's right and left eyes are calculated as the right-eye pixel position and the left-eye pixel position.

<Calculation of pixel position for right eye and pixel position for left eye>
Calculation of the right eye pixel position and the left eye pixel position by the pixel position calculation unit 202 (pixel position calculation unit 2023) will be described with reference to FIG. 4 (see FIGS. 1 and 3 as appropriate).
First, the pixel position calculation unit 202 acquires the image pickup device information of the image pickup device information storage unit 2021 and the lens array information of the image pickup lens array information storage unit 2022.

Based on the lens array information and the image pickup element information, the pixel position calculation unit 202 simulates the optical system of the IP image pickup apparatus 10 shown in FIG. 4A as a virtual optical system as shown in FIG. 4B. it can. In FIG. 4A, the IP imaging device 10 of FIG. 1 is illustrated as a 4f optical system.
Further, the lens array 105 of FIG. 4A is one in which the position of the lens array 102 is optically shifted by the effect of the convex lenses 101 and 104. The lens array 105 is equivalent to a lens array in an IP type stereoscopic display device (hereinafter, IP display device). Therefore, the position where the observer observes the lens array 105 is the position of the both eyes 91 illustrated on the left side in FIG. Then, in the IP imaging device 10, the position of the both eyes 91 (viewpoint position) illustrated on the right side in FIG. 4B corresponds to the position of the both eyes 91.

The light ray reaching the left eye 92 _L is a light ray from the pixel 103 _{L at the} pixel position for the left eye, which is illustrated by the broken line in FIG. 4B. The light ray reaching the right eye 92 _R is the light ray from the pixel 103 _R at the right eye pixel position shown by the solid line in FIG. 4B. These ray groups draw a straight line connecting the both eyes 92 of the observer and the principal point α of the convex lens 102a. That is, the pixel position calculating means 202 calculates the positions where the extended lines of these straight lines intersect the image sensor 103 as the left-eye pixel position and the right-eye pixel position.

  After that, the pixel position calculation unit 202 outputs the calculated right eye pixel position and left eye pixel position to the image sensor control unit 203 and the pixel value allocation unit 204. At this time, the pixel position calculation unit 202 may convert the pixel position for the left eye and the pixel position for the right eye for each element image in point symmetry with reference to the center coordinates of the element image in order to prevent the depth inversion. .

Returning to FIG. 1, the description of the configuration of the twin-lens stereoscopic image generation device 20 will be continued.
The image sensor control unit 203 drives the pixels at the right eye pixel position and the left eye pixel position input from the pixel position calculation unit 202 among all the pixels of the image sensor 103.

  For example, when generating a binocular stereoscopic image, the pixels at the right-eye pixel position and the left-eye pixel position are driven for each element image, so the number of pixels to be driven is twice the total number of element images. At this time, as shown in FIG. 5, the pixel position for the right eye may be located between the pixels 103a in the image sensor 103 (the same applies to the pixel at the pixel position for the left eye).

Therefore, the pixel values of the pixels at the right-eye pixel position and the left-eye pixel position may be approximated by the pixel value of the pixel at the closest position in the image sensor 103. In this case, the number of driven pixels is twice the total number of element images.
Further, the pixel values of the pixels at the right-eye pixel position and the left-eye pixel position may be interpolated by the pixel values of the pixels 103a near the periphery 4 in the image sensor 103. In this case, the number of driven pixels is eight times the total number of element images.

  After that, the image sensor control unit 203 generates a control signal for driving the pixels at the pixel position for the right eye and the pixel position for the left eye, and outputs the generated control signal to the IP image capturing apparatus 10 (image sensor 103). In accordance with this control signal, the captured image captured by the image sensor 103 is input to the pixel value allocation means 204.

  The pixel value allocating means 204 acquires the pixel value of each pixel of the captured image input from the image sensor 103 and allocates it as the pixel value of the pixel of the two-eye stereoscopic image to generate a two-eye stereoscopic image. is there. As shown in FIG. 6, the pixel value allocation unit 204 includes a display device information storage unit 2041 and a twin-lens stereoscopic image generation unit 2042.

The display device information storage unit 2041 is a general storage device such as a memory or an HDD that stores in advance display device information representing the system of the stereoscopic display device. That is, in the present embodiment, the pixel value assigning unit 204 generates a binocular stereoscopic image corresponding to the system of the stereoscopic display device.
This display device information represents, for example, any one of a polarization glasses system (time division glasses system), a time division glasses system, a lenticular system, and an IP system. Here, in the case of the lenticular system and the IP system, it means that a stereoscopic display device of these systems displays a two-eye stereoscopic image.

  The binocular stereoscopic image generation means 2042 refers to the display device information stored in the display device information storage means 2041 and generates a binocular stereoscopic image corresponding to the system of the stereoscopic display device. Hereinafter, each method of the stereoscopic display device will be described in detail.

<Polarizing glasses method>
In the polarized glasses method, for example, a vertically polarized image for the right eye and a horizontally polarized image for the left eye are displayed. For the observer, a polarization filter that transmits only the image for the right eye (vertical polarization) is attached to the lens for the right eye, and a polarization filter that transmits only the image for the left eye (horizontal polarization) is attached to the lens for the left eye. Wear glasses and observe the image for the right eye and the image for the left eye. Therefore, the observer observes the image for the right eye only with the right eye and the image for the left eye only with the left eye. As a result, a stereoscopic image based on binocular parallax is obtained.

  Here, the binocular stereoscopic image generation unit 2042 acquires the pixel value of the pixel at the pixel position for the right eye input from the pixel position calculation unit 202 from the captured image input from the image sensor 103, and the image for the right eye is acquired. Assigned as the pixel value of the pixel (the same applies to the pixel position for the left eye). Then, the binocular stereoscopic image generation unit 2042 outputs the generated right-eye image and left-eye image to the polarizing glasses-type stereoscopic display device as a binocular stereoscopic image.

<Time division glasses method>
In the time division glasses method, for example, the image for the left eye and the image for the right eye are switched and displayed for each frame. The observer wears glasses that open the right-eye shutter while displaying the right-eye image, and opens the left-eye shutter while displaying the left-eye image, and observes the right-eye image and the left-eye image. . Therefore, the observer observes the image for the right eye only with the right eye and the image for the left eye only with the left eye. As a result, a stereoscopic image based on binocular parallax is obtained.

Here, the binocular stereoscopic image generation unit 2042 generates a binocular stereoscopic image by a method similar to the polarization glasses method.
When the stereoscopic display device is compatible with high-definition, the number of pixels of each binocular stereoscopic image needs to be equal to the number of pixels (1920 × 1080) defined by high-definition. For example, if the number of pixels of the binocular stereoscopic image is less than the number of pixels of high-definition, the binocular stereoscopic image generation unit 2042 enlarges the binocular stereoscopic image by interpolation processing. On the other hand, when the number of pixels of the binocular stereoscopic image exceeds the number of pixels of high-definition, the binocular stereoscopic image generation unit 2042 reduces the binocular stereoscopic image.
After that, the twin-lens stereoscopic image generation unit 2042 outputs the generated twin-lens stereoscopic image to the time-division glasses type stereoscopic display device.

<Lenticular method>
Consider a case where the stereoscopic display device 30 including the lenticular lens 301 and the display element 302 is viewed from the front at an infinite viewing distance as shown in FIG. 7. In this case, in the left half of the elemental image (longitudinal rectangular image) displayed on the display element 302 located on the back surface of the lenticular lens 301, a ray of any one pixel is incident on the right eye of the observer (left eye. The same). That is, the ratio (boundary) between the image region for the right eye and the image region for the left eye in the element image varies depending on the position of the element image and the viewing distance of the observer.

  Therefore, the binocular stereoscopic image generating unit 2042 causes the display element 302 to have an extended line of a straight line connecting the midpoint β of the line segment connecting the pupil centers of the both eyes 92 of the observer and the principal point γ of each lenticular lens 301. The intersecting position is determined as the boundary between the right-eye image area and the left-eye image area. Then, the binocular stereoscopic image generation unit 2042 acquires the pixel value of the pixel at the pixel position for the right eye input from the pixel position calculation unit 202 in the captured image input from the image sensor 103, and the image region for the right eye. Assigned as the pixel value of the pixel (the same applies to the image area for the left eye). After that, the binocular stereoscopic image generation unit 2042 outputs the element image group including the right eye image area and the left eye image area to the stereoscopic display device 30 as a binocular stereoscopic image.

<IP method>
Consider a case where the IP display device 40 including the lens array 401 in which the convex lenses 401a are arranged and the display element 402 are observed from the front as shown in FIG. Here, the binocular stereoscopic image generation unit 2042 uses a position where a straight line passing through the midpoint β and the principal point γ of the convex lens 401a intersects the display element 402 as a boundary between the right-eye image region and the left-eye image region. Ask. Then, the binocular stereoscopic image generation unit 2042 acquires the pixel value of the pixel at the pixel position for the right eye input from the pixel position calculation unit 202 in the captured image input from the image sensor 103, and the image region for the right eye. Assigned as the pixel value of the pixel (the same applies to the image area for the left eye).

For example, consider a case where the element image group reconstructed for the left eye is FIG. 9A and the element image group reconstructed for the right eye is FIG. 9B. In this case, the binocular stereoscopic image generation unit 2042 generates an element image group in which the left side is the image for the right eye and the right side is the image for the left eye, as illustrated in FIG. 9C. Then, the generated element image group is output to the IP display device 40 as a twin-lens stereoscopic image.
In FIG. 9, the image area for the right eye is illustrated by hatching, and the image area for the left eye is illustrated by dots.

[Operation of Binocular Stereoscopic Image Generating Device]
The operation of the twin-lens stereoscopic image generation device 20 will be described with reference to FIG. 10 (see FIG. 1 as appropriate).
As shown in FIG. 10, the binocular stereoscopic image generation device 20 determines the viewpoint positions corresponding to both eyes of the observer who observes the binocular stereoscopic image by the viewpoint position determination unit 201 (step S1).
The binocular stereoscopic image generation device 20 calculates the pixel position (pixel position for the right eye and pixel position for the left eye) of the image sensor 103 by the pixel position calculation unit 202 (step S2).

In the binocular stereoscopic image generation device 20, the pixel at the pixel position (pixel position for the right eye and pixel position for the left eye) calculated by the pixel position calculation unit 202 among all the pixels of the image sensor 103 by the image sensor control unit 203. Is driven (step S3).
The binocular stereoscopic image generation device 20 allocates the pixel value of each pixel of the captured image input from the image sensor 103 as the pixel value of the pixel of the binocular stereoscopic image by the pixel value allocation unit 204 (step S4). In this way, the binocular stereoscopic image generation device 20 generates a binocular stereoscopic image.

  As described above, in the binocular stereoscopic image generation device 20 according to the first embodiment of the present invention, only the pixels at the right eye pixel position and the left eye pixel position necessary for generating a binocular stereoscopic image are captured by the IP imaging device. Since it is driven by 10, the amount of calculation is suppressed, and a two-lens stereoscopic image can be efficiently generated.

(Second embodiment)
[Overall Configuration of Binocular Stereoscopic Image Generation System]
With reference to FIG. 11, a binocular stereoscopic image generation system 1B according to the second embodiment of the present invention will be described regarding differences from the first embodiment.

  In the first embodiment, the image sensor 103 drives only the pixels at the right-eye pixel position and the left-eye pixel position (FIG. 1). On the other hand, the second embodiment differs from the first embodiment in that the image sensor 103 images only the light rays that have passed through the opening 106a of the shielding means 106.

As shown in FIG. 11, the binocular stereoscopic image generation system 1B includes an IP imaging device 10B and a binocular stereoscopic image generation device 20B.
The IP image pickup device 10B includes a convex lens 101, a lens array 102, an image pickup element 103, and a shielding unit 106.
Since the convex lens 101, the lens array 102, and the image sensor 103 are the same as those in the first embodiment, the description thereof will be omitted.

  The shielding unit 106 controls (shields) the light rays from the subject 90 that have entered the convex lens 101. In the present embodiment, the shielding means 106 is arranged between the convex lens 101 and the lens array 102 and on the subject 90 side. Further, the shielding means 106 moves in the front-back direction (depth direction) shown by the block arrow.

  Further, the shielding means 106 has two openings 106a corresponding to both eyes of the observer. The distance between the openings 106a is proportional to the distance between the pupils of the observer. For example, the distance between the openings 106a may be a value proportional to the general pupil distance (65 mm) of an adult.

It should be noted that the IP image pickup apparatus 10B and the stereoscopic display apparatus have a relationship in which the pixel size is different but the number of pixels is the same, so the expression proportional is used. For example, if the IP imaging device 10B is
It is assumed that the imaging device 103 has 1920 × 1080 pixels with a pixel interval of 10 μm and the lens array 102 with a lens interval of 100 μm. In addition, for example, it is assumed that the stereoscopic display device includes a display element of 1920 × 1080 pixels with a pixel interval of 100 μm, and a lens array with a lens interval of 1 mm. In this case, the stereoscopic display device uses an optical system that is 10 times larger than the IP image pickup device 10B.

  The position of the shielding means 106 in the depth direction is controlled in proportion to the viewing distance of the observer. That is, the shielding unit 106 moves in the front-rear direction or changes the interval of the openings 106a and drives the diameter of the openings 106a (opening diameter) according to the control signal from the IP imaging device 10B.

  The shielding means 106 is not limited in its material and structure as long as it shields light rays, except for the place where the opening 106a is formed. For example, the shielding unit 106 may mount a liquid crystal element that transmits light rays at the position of the opening 106a and shields light rays at other positions on a drive mechanism that moves on a rail laid in the front-rear direction.

[Configuration of Binocular Stereoscopic Image Generation Device]
The configuration of the twin-lens stereoscopic image generation device 20B will be described below.
As shown in FIG. 11, the binocular stereoscopic image generation device 20B includes a viewpoint position determination unit 201, a pixel position calculation unit 202, a pixel value allocation unit 204B, and an aperture position control unit 205.
Note that the viewpoint position determination unit 201 and the pixel position calculation unit 202 are the same as those in the first embodiment, and therefore their explanations are omitted.

The aperture position control means 205 will be described first, and then the pixel value allocation means 204B will be described.
The aperture position control unit 205 drives the shielding unit 106 so that the light beam from the subject 90 enters the pixel position for the right eye and the pixel position for the left eye of the image sensor 103 through the aperture 106a. In the present embodiment, the aperture position control unit 205 stores the viewpoint position input from the viewpoint position determination unit 201, the image sensor information of the image sensor information storage unit 2021, and the lens array information of the imaging lens array information storage unit 2022. Referring to, the drive control of the shielding means 106 is performed.

<Control of shielding means>
Control of the shielding means 106 will be described with reference to FIGS. 12 and 13 (see FIG. 11 as appropriate).
To simplify the description, consider a case where the IP imaging device 10B does not include the convex lens 101, as shown in FIG. Here, a pinhole model in which light rays from the subject 90 pass only at the principal points of the convex lens 102a is applied.

Note that, in FIG. 12, only two openings 106 a provided in the shielding unit 106 are illustrated for easy viewing of the drawing.
Further, the viewing area is θ, the focal length of the convex lens 102a is f0, the distance from the lens array 102 to the start point where the viewing area is formed is L1, and the distance from L1 to the opening 106a is Lx. The distance between the two openings 106a is Hp.

The opening position control unit 205 refers to a preset reference parameter and calculates either the distance Lx or the interval Hp using the following formula (1) or formula (2).
Lx = LxHp / PD-L1 ... Formula (1)
Hp = PD × (Lx + L1) / L ... Formula (2)

When the distance Lx is calculated, the opening position control unit 205 moves the opening 106a in the front-rear direction so that the opening 106a is located at the distance Lx while not changing the interval Hp.
When the distance Hp is calculated, the opening position control unit 205 does not change the distance Lx, but changes the distance between the openings 106a so that the opening 106a has the distance Hp.
In any case, the light ray from the subject 90 passes through the opening 106a (the light ray shown by the thick line in FIG. 12 overlaps the opening 106a).

  Here, when calculating the distance Lx, the interval Hp is preset as a reference parameter. On the other hand, when the distance Hp is calculated, the distance Lx is preset as a reference parameter. In addition, as reference parameters, for example, the visual distance L, the distance PD between the eyes of the observer, and the pupil diameter P of the observer are also set in advance.

Next, the opening position control unit 205 calculates the diameter S of the opening 106a using the following formula (3) or formula (4). Then, the opening position control unit 205 changes the opening diameter of the opening 106a so that the calculated diameter S is obtained.
That is, when the distance Hp is calculated, the equation (3) is used, and when the distance Lx is calculated, the equation (4) is used.
S = P × Hp / PD Equation (3)
S = P * (Lx + L1) / L ... Formula (4)

As shown in FIG. 13, since the IP image pickup apparatus 10B actually includes the convex lens 101, consider a case where the lens array 102 is located on the focal plane of the convex lens 101. In this case, it is necessary to satisfy 0 <Lx <(f1-L1). In FIG. 13, the focal length of the convex lens 101 is f1.
After that, the opening position control means 205 moves the shielding means 106 in the front-rear direction or changes the distance between the openings 106a to generate a control signal for changing the diameter of the opening 106a, and outputs the generated control signal to the shielding means 106. To do.

Returning to FIG. 11, the pixel value assigning means 204B will be described.
The pixel value allocating means 204B acquires the pixel value of each pixel of the captured image input from the image sensor 103 and allocates it as the pixel value of the pixel of the twin-lens stereoscopic image to generate a twin-lens stereoscopic image. is there.

  As described above, since only the light ray that has passed through the opening 106a of the shielding unit 106 is incident on the image sensor 103, only the pixels at the pixel positions for the left eye and the pixel positions for the right eye are included in the captured image from the image sensor 103. Has a pixel value, and the pixel values of the other pixels are 0 (zero). That is, the image sensor 103 outputs the pixel values of all the pixels to the pixel value assigning means 204B.

In other respects, the pixel value assigning means 204B is the same as that of the first embodiment, and therefore further description is omitted.
Note that the binocular stereoscopic image generation device 20B further includes an image sensor control unit 203 (FIG. 1), as in the first embodiment, and a captured image captured by only the pixels driven by the image sensor control unit 203 has a pixel value. You may output to the allocation means 204B.

[Operation of Binocular Stereoscopic Image Generating Device]
The operation of the twin-lens stereoscopic image generation device 20B will be described with reference to FIG. 14 (see FIG. 11 as appropriate).
The processing of steps S1 and S2 is the same as that of the first embodiment, and thus the description thereof is omitted.

The binocular stereoscopic image generation apparatus 20B drives the shielding unit 106 so that the light beam from the subject 90 passes through the opening 106a and enters the image sensor 103 by the opening position control unit 205 (step S5).
The binocular stereoscopic image generation device 20B allocates the pixel value of each pixel of the captured image input from the image sensor 103 as the pixel value of the pixel of the binocular stereoscopic image by the pixel value allocation unit 204B (step S4B).

  As described above, in the binocular stereoscopic image generation device 20B according to the second embodiment of the present invention, only the pixels at the right eye pixel position and the left eye pixel position necessary for generating a binocular stereoscopic image are captured by the IP imaging device. Since the image is picked up by 10, the amount of calculation is suppressed, and a two-lens stereoscopic image can be efficiently generated.

  Although the respective embodiments of the invention of the present application have been described in detail above, the invention of the present application is not limited to the above-described embodiments, and includes design changes and the like within a range not departing from the gist of the invention of the present application.

(Modification 1)
A modified example of the viewpoint position determination method will be described with reference to FIG.
As shown in FIG. 15, the viewpoint position determination unit 201 includes a viewpoint position detection camera 2012, a viewpoint position detection control unit 2013, and a viewpoint position detection unit 2014.

  The viewpoint position detection camera 2012 is a general imaging camera that images the observer in order to detect the viewpoint position of the observer. The viewpoint position detection camera 2012 outputs a viewpoint image including both eyes of the observer to the viewpoint position detection control unit 2013. Further, the viewpoint position detection camera 2012 changes its imaging range (zoom) and frame rate according to the control signal input from the viewpoint position detection control means 2013.

  The viewpoint position detection control unit 2013 controls the imaging range (zoom) and the frame rate of the viewpoint position detection camera 2012 so that the observer is included in the viewpoint image. In addition, the viewpoint position detection control unit 2013 outputs the viewpoint image input from the viewpoint position detection camera 2012 to the viewpoint position detection unit 2014.

For example, the viewpoint position detection control unit 2013 obtains the moving speed of the observer based on the difference between frames of continuous viewpoint images. Then, the viewpoint position detection control unit 2013 controls the imaging range and frame rate of the viewpoint position detection camera 2012 according to the moving speed of the observer. That is, the viewpoint position detection control unit 2013 narrows the imaging range of the viewpoint position detection camera 2012 and lowers the frame rate as the moving speed of the observer becomes slower. On the other hand, the viewpoint position detection control unit 2013 increases the imaging range of the viewpoint position detection camera 2012 and increases the frame rate as the observer moves faster. After that, the viewpoint position detection control unit 2013 outputs a control signal of the imaging range and the frame rate to the viewpoint position detection camera 2012.
Note that the viewpoint position detection control unit 2013 may control only one of the imaging range and the frame rate of the viewpoint position detection camera 2012.

  The viewpoint position detection unit 2014 detects the viewpoint position from the viewpoint image input from the viewpoint position detection control unit 2013. For example, the viewpoint position detection unit 2014 detects the viewpoint position of the observer by performing known face detection processing and pupil detection processing on the viewpoint image. Then, the viewpoint position detecting means 2014 outputs the detected viewpoint position to the pixel position calculating means 202.

(Other modifications)
In each of the above-described embodiments, an image (still image) has been described, but the present invention is not limited to this. That is, the binocular stereoscopic image generation device can also generate a binocular stereoscopic image (moving image).

  In each of the above-described embodiments, it is described that the stereoscopic display device of the polarization glasses system, the time-division glasses system, the lenticular system, or the IP system displays the twin-eye stereoscopic image, but the present invention is not limited to this.

Although the binocular stereoscopic image generation device is described as independent hardware in each of the above-described embodiments, the present invention is not limited to this. For example, hardware resources such as a CPU, a memory, and a hard disk included in the computer may be realized by a twin-lens stereoscopic image generation program that causes the twin-lens stereoscopic image generation device to operate cooperatively. This program may be distributed via a communication line, or may be written and distributed in a recording medium such as a CD-ROM or a flash memory.

1,1B Binocular stereoscopic image generation system 10,10B IP imaging device (stereoscopic imaging device)
20, 20B Binocular stereoscopic image generation device 101 Convex lens 102 Lens array (optical element array)
103 image sensor 106 shielding means 106a aperture 201 viewpoint position determining means 202 pixel position calculating means 203 image sensor controlling means 204, 204B pixel value assigning means 205 aperture position controlling means

Claims (6)

A twin-lens stereoscopic image generating apparatus for generating a twin-lens stereoscopic image by using an integral photography type stereoscopic imaging device including an optical element array in which optical elements are two-dimensionally arranged, and an imaging element,
Viewpoint position determining means for determining viewpoint positions corresponding to both eyes of an observer who observes the binocular stereoscopic image;
A straight line that passes through the optical principal point of the optical element from the viewpoint position based on the viewpoint position , preset optical element information about the optical element array , and preset imaging element information about the image sensor. A pixel position calculating means for calculating the pixel position for the right eye and the pixel position for the left eye of the image pickup device where
Image sensor control means for driving pixels at the right eye pixel position and the left eye pixel position among all pixels of the image sensor,
Pixel value assigning means for obtaining the pixel values of the driven pixels for the right-eye pixel position and the left-eye pixel position and assigning the pixel values as the pixel values of the pixels of the binocular stereoscopic image,
Equipped with
The pixel value assigning means, the generated parallax images representing the binocular disparity of the observer as a two-eye stereo images, and wherein also be output from the parallax image generated on the stereoscopic display device of the polarizing glass method Binocular stereoscopic image generation device. An optical element array in which optical elements are two-dimensionally arranged, an image pickup element, and two openings corresponding to both eyes of an observer who observes a two-eye stereoscopic image and is arranged between the optical element array and a subject. A binocular stereoscopic image generation device for generating a binocular stereoscopic image using an integral photography type stereoscopic imaging device including a shielding unit having
Viewpoint position determining means for determining viewpoint positions corresponding to both eyes of the observer,
A straight line that passes through the optical principal point of the optical element from the viewpoint position based on the viewpoint position , preset optical element information about the optical element array , and preset imaging element information about the image sensor. A pixel position calculating means for calculating the pixel position for the right eye and the pixel position for the left eye of the image pickup device where
An aperture position control unit that drives the shielding unit so that a light ray from the subject enters the pixel position for the right eye and the pixel position for the left eye of the image sensor through the aperture,
Pixel value assigning means that obtains pixel values of pixels at the right-eye pixel position and the left-eye pixel position and assigns the pixel values as pixel values of the binocular stereoscopic image,
Equipped with
The pixel value assigning means, the generated parallax images representing the binocular disparity of the observer as a two-eye stereo images, and wherein also be output from the parallax image generated on the stereoscopic display device of the polarizing glass method Binocular stereoscopic image generation device. A twin-lens stereoscopic image generating apparatus for generating a twin-lens stereoscopic image by using an integral photography type stereoscopic imaging device including an optical element array in which optical elements are two-dimensionally arranged, and an imaging element,
Viewpoint position determining means for determining viewpoint positions corresponding to both eyes of an observer who observes the binocular stereoscopic image;
A straight line that passes through the optical principal point of the optical element from the viewpoint position based on the viewpoint position, preset optical element information about the optical element array, and preset imaging element information about the image sensor. A pixel position calculating means for calculating the pixel position for the right eye and the pixel position for the left eye of the image pickup device where
Image sensor control means for driving pixels at the right eye pixel position and the left eye pixel position among all pixels of the image sensor,
Pixel value assigning means for obtaining the pixel values of the driven pixels for the right-eye pixel position and the left-eye pixel position and assigning the pixel values as the pixel values of the pixels of the binocular stereoscopic image,
Equipped with
The pixel value assigning unit generates a parallax image representing the binocular parallax of the observer as the binocular stereoscopic image, and outputs the generated parallax image to a time-division glasses type stereoscopic display device. Binocular stereoscopic image generation device. An optical element array in which optical elements are two-dimensionally arranged, an image pickup element, and two openings corresponding to both eyes of an observer who observes a two-eye stereoscopic image and is arranged between the optical element array and a subject. A binocular stereoscopic image generation device for generating a binocular stereoscopic image using an integral photography type stereoscopic imaging device including a shielding unit having
Viewpoint position determining means for determining viewpoint positions corresponding to both eyes of the observer,
A straight line that passes through the optical principal point of the optical element from the viewpoint position based on the viewpoint position, preset optical element information about the optical element array, and preset imaging element information about the image sensor. A pixel position calculating means for calculating the pixel position for the right eye and the pixel position for the left eye of the image pickup device where
An aperture position control unit that drives the shielding unit so that a light ray from the subject enters the pixel position for the right eye and the pixel position for the left eye of the image sensor through the aperture,
Pixel value assigning means that obtains pixel values of pixels at the right-eye pixel position and the left-eye pixel position and assigns the pixel values as pixel values of the binocular stereoscopic image,
Equipped with
The pixel value assigning unit generates a parallax image representing the binocular parallax of the observer as the binocular stereoscopic image, and outputs the generated parallax image to a time-division glasses type stereoscopic display device. 2 stereoscopic image generating apparatus you. The viewpoint position determining means,
Viewpoint position detection control means for controlling the imaging range of the viewpoint position detection camera so that the observer is included in the viewpoint image,
Viewpoint position detection means for detecting the viewpoint position from the viewpoint image captured by the viewpoint position detection camera,
2 stereoscopic image generating apparatus according to any one of claims 1 to 4, characterized in that it comprises a. A binocular stereoscopic image generation program for causing a computer to function as the binocular stereoscopic image generation device according to any one of claims 1 to 5 .

Images