JPH08126034A - Method and device for displaying stereoscopic image - Google Patents

Abstract

PURPOSE: To avoid a sense of incongruity by devising a method such that the size of an observed image is felt identically to be an actual size even when the interval of optical axes differs between image pickup and observation. CONSTITUTION: The display device is provided with a left image pickup system 1L and a right image pickup system 1R whose optical axes are arranged in parallel at a prescibed interval 10, an image input section 2, image memories 3L, 3R, an image interpolation section 4, an image output section 6 and an HMD 7. One image is displayed through image interpolation from two images so that the parallax of the images in the case of HMD display is equal to a parallax ▵1 when an object is actually viewed. Reference point extract processing is conducted by using left image data and right image data being inputs to an image interpolation section 4. A small area of the left image is segmented as a template moved in parallel to detect a position at which a total sum of differences from the right image data is least and the result is used for the pixel position of the corresponding right image. The distance information is expressed as 1/Z=(xL-xR)/(f.10)and the X-coordinate of an interpolation image is expressed as x=xL-f.11/Z based on the pixel positions xL, xR of the corresponding left and right images and the image pickup parameters.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a stereoscopic image display device for observing an image having parallax obtained by a plurality of image pickup means as a stereoscopic image.

[0002]

2. Description of the Related Art Conventionally, as a display device for observing a stereoscopic image, images corresponding to both eyes are picked up by two cameras on the left and right sides, and these images are alternately switched and displayed for each field of a television and synchronized with the images. It is known that stereoscopic viewing is performed using shutter glasses that are opened and closed. However, this display device has a problem that a stereoscopic effect cannot be obtained unless it is observed near the center of the television.

In order to solve this, two display portions for independently observing the left and right images with the left and right eyes of the observer are provided on the head, and the so-called HM having the display portion mounted on the head is provided.
A method of directly inputting video signals of two left and right cameras to a D (Head Monted Display) to observe a stereoscopic image is known.

[0004]

However, the above-mentioned conventional example has a problem that the stereoscopic effect is uncomfortable when the distance between the optical axes of the two cameras and the distance between the display portions of the HMD are different during photographing. Especially, the average distance between human eyes is 6
It is about 5 mm, and the interval of the display part of the HMD is also adjusted to about 65 mm, but since the interval of the camera is limited by the size of the camera, the interval during imaging is generally larger than the interval during observation. Therefore, a so-called boxyard effect occurs in which the observed image feels smaller than it actually is.

According to the present invention, a stereoscopic image capable of obtaining a stereoscopic image in which the size of the observed image is the same as the actual size even when the optical axis interval is different between when the image is taken and when the image is observed can be obtained. An object is to provide a display device.

[0006]

In order to achieve the above object, a stereoscopic image display device according to claim 1 of the present invention comprises display means for displaying respective images taken by two left and right cameras, In a stereoscopic image display device for observing the displayed image as a stereoscopic image with both eyes, a generation means for generating at least one of the images displayed on the display unit by interpolation using the other image is provided. The means includes an interpolating means that interpolates based on at least the viewpoint interval at the time of image capturing and the optical axis interval at the time of image display.

A stereoscopic image display device according to a second aspect of the present invention is a stereoscopic image display device according to the first aspect, in which a displacement amount detection is performed to detect a displacement amount of pixel positions corresponding to respective images photographed by the left and right two cameras. The interpolating means has a ratio of a displacement amount of each of the detected pixel positions and a displacement amount of an image displayed on the display unit to a viewpoint interval during image capturing and an optical axis interval during image display. Interpolate to be equal to the ratio of.

A stereoscopic image display device according to a third aspect of the present invention is a stereoscopic image display device according to the first aspect, in which a displacement amount detection is performed to detect a displacement amount of pixel positions corresponding to respective images captured by the left and right two cameras. The interpolation means is provided with a ratio of the detected deviation amount of each pixel position and the deviation amount of the image displayed on the display unit to a viewpoint interval during image capturing and an optical axis interval during image display. Interpolate to be equal to a constant multiple of the ratio.

A stereoscopic image display device according to a fourth aspect is the stereoscopic image display device according to any one of the first to third aspects, in which the two cameras are set with convergence on the optical axis, and the setting is performed. An image conversion unit that converts an image captured by the two captured cameras into an image in which the optical axes are parallel to each other is provided, and the interpolation unit interpolates the converted image.

A stereoscopic image display method according to a fifth aspect is a stereoscopic image display method for displaying respective images photographed by two left and right cameras and observing the displayed images as a stereoscopic image with both eyes. When at least one of the images displayed on the display unit is generated by interpolation using the other image, the interpolation is performed based on at least the viewpoint interval at the time of image capturing and the optical axis interval at the time of image display.

[0011]

In the stereoscopic image display device according to claim 1 of the present invention, when the respective images taken by the two left and right cameras are displayed by the display means and the displayed image is observed as a stereoscopic image with both eyes. In addition, at least one of the images displayed on the display unit by the generation unit is generated by interpolation using the other image, and based on at least the viewpoint interval at the time of image capturing and the optical axis interval at the time of image display by the interpolation unit. Interpolate.

In the stereoscopic image display device according to the second aspect, the shift amount detecting means detects the shift amount of the pixel position corresponding to each image photographed by the left and right two cameras, and the interpolating means detects the shift amount. Interpolation is performed so that the ratio between the displacement amount of each pixel position and the displacement amount of the image displayed on the display unit becomes equal to the ratio of the viewpoint interval during image capturing and the optical axis interval during image display.

In the three-dimensional image display device according to the third aspect, the shift amount detecting means detects the shift amount of the pixel position corresponding to each image photographed by the two left and right cameras, and the interpolating means detects the shift amount. Interpolation is performed so that the ratio of the displacement amount of each pixel position and the displacement amount of the image displayed on the display unit is equal to a constant multiple of the ratio of the viewpoint interval during image capturing and the optical axis interval during image display. .

A stereoscopic image display device according to a fourth aspect of the present invention sets the two cameras with convergence on an optical axis, and an image captured by the two cameras set by the image conversion unit,
The image is converted into an image in which the optical axes are parallel, and the interpolation means interpolates the converted image.

[0015]

EXAMPLE An example of the stereoscopic image display device of the present invention will be described.

[First Embodiment] FIG. 1 is a block diagram showing the arrangement of a stereoscopic image display apparatus according to the first embodiment. In the figure, 1L and 1R are a left image pickup system and a right image pickup system, respectively.
The optical axes are arranged in parallel with each other with a predetermined gap 10 therebetween.

The image input section 2 includes a left image pickup system 1L and a right image pickup system 1
The video signal output from the R camera is converted into digital image data for each frame, and the converted image signal is stored in the image memory 3 respectively.
Store in L and 3R. The image interpolator 4 includes an image memory 3L,
The image displayed on the right display portion of the HMD 7 is interpolated from the image data stored in 3R, and the image data is output to the output image memory 5. The image output unit 6 converts the digital image data stored in the image memory 3L and the output image memory 5 into video signals and outputs the video signals to the HMD 7.

FIG. 2 is an explanatory diagram showing the principle of the method for displaying a stereoscopic image. In the figure, X represents the horizontal direction, Y (not shown) represents the vertical direction, and Z represents the depth direction. IL and IR are image planes of the left imaging system and the right imaging system, respectively, located at the focal length f from the viewpoint, and the distance between the viewpoints is 10. At this time, parallax occurs only in the X direction, so only the X coordinate will be described below. The Y coordinate of the same point is unchanged for the left and right images and the interpolated image to be displayed.

O is an object point on the subject, which is separated from the viewpoints of both cameras by a distance Z. xL and xR are object points O
Image points on the respective image planes IL and IR of the above, and are represented by Formula (1).

[0020]

## EQU00001 ## xL = f.X / Z, xR = f. (X-10) / Z Therefore, when the images are taken by the cameras separated by l0, the parallax .DELTA.0 is expressed by Formula (2).

[0021]

## EQU00002 ## .DELTA.0 = xL-xR = f.multidot.10 / Z On the other hand, when an image taken under such conditions is observed with an HMD having an optical axis interval l1, the parallax .DELTA.1 when the subject is actually observed is the same. Is expressed by equation (3).

[0022]

## EQU00003 ## .DELTA.1 = f.l1 / Z Therefore, an image having a parallax larger than the actual one is observed, and the distance in the depth direction is felt to be small.

In the present embodiment, one image is displayed by image interpolation obtained from two images so that the parallax of the image during HMD display becomes the parallax Δ1 when the subject is actually observed.

FIG. 3 is an explanatory diagram showing an example of image interpolation. In the figure, a and c are a left image and a right image, respectively, and b is an interpolated image after parallax correction.

FIG. 4 is a flowchart showing the image interpolation process. First, the corresponding point extraction processing is performed using the left image data and the right image data input to the image interpolation unit 4 (step S401). The corresponding point extraction process is a process of obtaining pixel positions of the right image corresponding to all the pixels of the left image,
This is done as follows.

First, a small region of the left image is cut out as a template, a position offset is applied to the template, and the template is moved in parallel to detect the position where the sum of the differences from the right image data is the smallest, and the pixel position of the corresponding right image is detected. And If the difference between the template and the image data does not fall below a certain threshold, or if the corresponding point cannot be specified because there are multiple positions having the minimum difference value, a flag indicating that the corresponding point cannot be found is set. The detected pixel position of the right image is converted into the image plane coordinate of the right imaging system 1R and output.

Next, the distance is calculated from the pixel positions of the corresponding left and right images and the imaging parameters (step S402). The distance information is obtained from Equation (4) from Equation (1) from the coordinates xL and xR of the pixel positions of the corresponding left and right images.

[0028]

## EQU00004 ## 1 / Z = (xL-xR) / (f.multidot.10) Further, the x coordinate in the interpolated image displayed at this time can be obtained from the mathematical expression (5).

[0029]

## EQU00005 ## x = xL-f.multidot.l1 / Z Thus, the distance information of all the pixels of which corresponding points are found in the interpolated image displayed by the processing of step S402 is obtained. However, since distance information cannot be obtained in a region where pixel positions in the left image and the right image do not correspond, distance interpolation processing is performed (step S403).

FIG. 5 is an explanatory diagram showing the contents of the distance interpolation processing. In the figure, the hatched area is an area that cannot be matched in the left and right images because it is hidden by a short-distance object.
Of these, the area indicated by OL is on the left side of the short-range object,
By comparing the distance information on the left and right sides of this area, it is recognized that the right side area has a longer distance, and the distance is interpolated from the distance information on the right side area. Through the above processing, all distance information in the interpolated image to be displayed can be obtained.

From the distance information obtained in step S403 and the image pickup parameters, the pixel coordinates of the left image and the right image of all the pixels on the interpolated image are obtained by the coordinate conversion equation shown in equation (6).

[0032]

## EQU00006 ## xL = x + f.multidot.l1 / Z, xR = xL-f.multidot.10 / Z Image data interpolation processing is performed from the pixel coordinates of the left and right images obtained in step S404 and the left and right image data (step S405). Interpolation of image data is performed by processing such as hyperbolic interpolation. At this time, the area where the corresponding points are obtained and the area shown by OL in FIG. 5 give the interpolation value from the left image, and the area shown by OR gives the interpolation value from the right image. Step S4
The interpolation image data displayed by 05 is output to the output image memory 5.

Therefore, in the present embodiment, the relationship between the parallax Δ0 picked up by the left and right cameras and the parallax Δ1 when the subject is actually observed satisfies the above-mentioned formulas (2) and (3), and Δ Since 0: Δ1 = l0: l1, it is possible to obtain a stereoscopic image in which the size of the observed image is the same as the actual size and does not cause a sense of discomfort.

[Second Embodiment] A stereoscopic image display apparatus according to the second embodiment will be described below. In the first embodiment,
If the optical axis intervals of the image pickup systems 1L and 1R become large to some extent, a common subject image cannot be obtained in the left and right images. Therefore, it is necessary to set the optical axes of the left and right image pickup systems 1L and 1R not parallel to each other but with some convergence. .

In the second embodiment, a display image for such a case is interpolated and generated. FIG. 6 is the second
It is a block diagram which shows the structure of the stereo image display apparatus of an Example. 1L and 1R are a left imaging system and a right imaging system,
The optical axes are separated by a predetermined distance l2, and the optical axes are arranged so as to intersect at a convergence angle 2θ.

The image input section 2 includes a left image pickup system 1L and a right image pickup system 1
The video signal output from the R camera is converted into digital image data frame by frame. Image conversion units 8L, 8
R is converted from the left and right image data converted into digital image data into image data obtained by imaging the left and right imaging systems 1L and 1R in parallel with each other, and the image memory 3
Store in L and 3R. The image interpolating unit 10 uses the image memory 3
An image to be displayed on the right display section of the HMD 7 is generated by interpolation from the image data stored in L and 3R, and the image data is output to the output image memories 9L and 9R. Image output unit 6
Converts the digital image data stored in the output image memories 9L and 9R into video signals, and outputs the HMD7.
Output to.

FIG. 7 is an explanatory diagram showing the processing contents of the image conversion units 8L and 8R. In the figure, IL and IR are the image planes of the left image pickup system and the right image pickup system which are respectively converged by θ with respect to the subject, and IL ′ and IR ′ are the images of the left image pickup system and the right image pickup system when the optical axes are parallel. The surface. The image conversion unit 8L converts the image so that the image point xL of the image plane IL of the left imaging system with respect to the object point O corresponds to the image point xL ′ of the image plane IL ′ of the left imaging system when the optical axis is parallel. To do.

Since there is no viewpoint movement, the correspondence between xL and xL 'can be matched using the focal length f and the convergence angle 2θ as parameters. Similarly, in the image conversion unit 8R, the image plane I of the right image pickup system when the image points of the image plane IR of the right image pickup system have their optical axes parallel to each other.
The image is converted so as to correspond to the image point of R '.

FIG. 8 is an explanatory diagram showing an example of image conversion.
Reference characters a and b are input images from the left imaging system and the right imaging system, respectively, and trapezoidal distortion occurs due to congestion. Images c and d are output images from the image conversion unit and are images when the optical axes are parallel. However, the shaded areas in the figure are areas where no image data has been input, and the optical axis centers are displaced in the X direction.

FIG. 9 is an explanatory diagram showing the processing contents of the image interpolation unit 10. IL ′ and IR ′ are the image planes of the left image pickup system and the right image pickup system, respectively, when the optical axes are parallel, and the optical axis interval l
An image having a parallax of 2 is input from the image memories 3L and 3R. The image to be displayed on the HMD 7 has an optical axis interval of l1, and the display image output to the output image memory 9L is an image having a parallax Δ2 shown in Expression (7) with respect to the image in the image memory 3L. Yes, this is the image interpolation unit 1
When it is 0, interpolation is performed based on the left and right image data and the imaging parameter.

[0041]

## EQU00007 ## .DELTA.2 = f.multidot. (L2-l1) / (2Z) Similarly, the display image to be output to the output image memory 9R is the parallax Δ shown in the equation (8) with respect to the image in the image memory 3L.
The image becomes 3 and is interpolated and generated by the image interpolation unit 10 from the left and right image data and the imaging parameter.

[0042]

[Expression 8] Δ3 = f · (l2 + l1) / (2Z) The image interpolation method performs the same processing as in the first embodiment shown in FIG. 4, and generates two interpolated images.

As described above, in the stereoscopic image display apparatus according to the second embodiment, the HM is applied to the image captured due to the congestion.
It is possible to obtain a stereoscopic image without a sense of discomfort when observed at D.

[Third Embodiment] Next, a stereoscopic image display apparatus according to a third embodiment will be described. When capturing a stereoscopic image, a miniature model of an actual subject may be captured and displayed. In this case, the observed image can be displayed so as to appear larger than it actually is by using the image interpolation process.

The stereoscopic image display device of the third embodiment is the same as the second embodiment.
It has the same configuration as the embodiment. FIG. 10 shows an image interpolating unit 10 for making a captured miniature model image look larger.
It is explanatory drawing which shows the processing content in.

Similar to FIG. 9, IL 'and IR' are the image planes of the left image pickup system and the right image pickup system, respectively, in the case where the optical axes are parallel, and an image having parallax corresponding to the optical axis interval l2 is generated. It is input from the image memories 3L and 3R. The image to be displayed on the HMD has an optical axis interval of 11, but the captured image is actually β
The parallax becomes 1 / β in order to display double, so l3 =
The display image to be output to the output image memory 9L as 11 / β is an image having a parallax Δ4 as shown in Expression (9) with respect to the image in the image memory 3L, and this is displayed by the image interpolating unit 10. Interpolation and generation are performed from the left and right image data and the imaging parameter.

[0047]

[Expression 9] Δ4 = f · (l2−l3) / (2Z) Similarly, the display image to be output to the image memory 9R for output is as shown in Formula (10) with respect to the image in the image memory 3L. The image has a parallax Δ5, which is interpolated and generated by the image interpolation unit 10 from the left and right image data and the imaging parameter.

[0048]

[Formula 10] Δ5 = f · (l2 + l3) / (2Z) As described above, in the present embodiment, the size of the miniature model when observed by the HMD with respect to the image of the actual miniature model of the subject. It is possible to obtain a stereoscopic image having a desired magnification of.

In the above-described embodiment, the HMD is used for display. However, two shot images are alternately switched and displayed for each field of the television, and shutter glasses are opened and closed in synchronization with the images. It is needless to say that the same effect can be obtained when stereoscopically viewed using.

[0050]

According to the three-dimensional image display device of the first aspect of the present invention, each image captured by the left and right two cameras is displayed by the display means, and the displayed image is a three-dimensional image with both eyes. At the time of observation, at least one of the images displayed on the display unit by the generation unit is generated by interpolation using the other image, and at least the viewpoint interval at the time of image capturing and the optical axis at the time of image display by the interpolation unit. Since the interpolation is performed based on the interval, even when the optical axis interval is different between the time of photographing and the time of observation, it is possible to obtain a stereoscopic image in which the size of the observed image is the same as the actual size and does not cause a feeling of strangeness.

According to the three-dimensional image display device of the second aspect, the shift amount detecting means detects the shift amount of the pixel position corresponding to each image photographed by the left and right two cameras, and the interpolating means is operable to detect the shift amount of the pixel position. Interpolation is performed so that the ratio of the detected deviation amount of each pixel position and the deviation amount of the image displayed on the display unit becomes equal to the ratio of the viewpoint interval during image capturing and the optical axis interval during image display. It is possible to obtain a stereoscopic image in which the size of the observed image is the same as the actual size and does not cause discomfort.

According to the three-dimensional image display device of the third aspect, the shift amount detecting means detects the shift amount of the pixel position corresponding to each image photographed by the left and right two cameras, and the interpolating means detects the shift amount. The ratio of the detected deviation amount of each pixel position and the deviation amount of the image displayed on the display unit is equal to a constant multiple of the ratio of the viewpoint interval at the time of image capturing and the optical axis interval at the time of image display. Since the interpolation is performed, it is possible to obtain a stereoscopic image in which an image of a miniature model of an actual subject is sensed at a desired magnification of the size of the miniature model when observed with an HMD, for example.

According to the three-dimensional image display device of the fourth aspect, the two cameras are set by converging the optical axis, and the images converted by the two cameras set by the image conversion unit are displayed. , The image is converted into an image in which the optical axes are parallel, and the interpolating means interpolates the converted image, so that a stereoscopic image which does not feel uncomfortable even when the image is picked up due to congestion is observed, for example, in the HMD. Images can be obtained.

According to the stereoscopic image display method of the fifth aspect, in the stereoscopic image display method of displaying the respective images photographed by the left and right two cameras and observing the displayed images as a stereoscopic image with both eyes. When at least one of the images displayed on the display unit is generated by interpolation using the other image, the interpolation is performed based on at least the viewpoint interval during image capturing and the optical axis interval during image display. Even when the optical axis interval is different between time and observation, it is possible to obtain a stereoscopic image in which the size of the observed image is the same as the actual size and does not cause a sense of discomfort.

[Brief description of drawings]

FIG. 1 is a block diagram showing a configuration of a stereoscopic image display device according to a first embodiment.

FIG. 2 is an explanatory diagram showing the principle of a method for displaying a stereoscopic image.

FIG. 3 is an explanatory diagram showing an example of image interpolation.

FIG. 4 is a flowchart showing image interpolation processing.

FIG. 5 is an explanatory diagram showing the contents of distance interpolation processing.

FIG. 6 is a block diagram showing a configuration of a stereoscopic image display device according to a second embodiment.

FIG. 7 is an explanatory diagram illustrating processing contents of image conversion units 8L and 8R.

FIG. 8 is an explanatory diagram showing an example of image conversion.

9 is an explanatory diagram showing the processing contents of the image interpolation unit 10. FIG.

FIG. 10 is an explanatory diagram showing the processing contents in the image interpolating unit 10 for making a captured image of a miniature model look large.

[Explanation of symbols]

 1L ... Left imaging system 1R ... Right imaging system 2 ... Image input unit 4 ... Image interpolation unit 7 ... HMD 8L, 8R ... Image conversion unit

Claims (5)

[Claims] 1. A stereoscopic image display device comprising display means for displaying respective images photographed by two left and right cameras, wherein the displayed image is viewed as a stereoscopic image with both eyes, and is displayed on the display section. Image generation means for generating at least one of the two images by interpolation using the other image, and the generation means includes an interpolation means for interpolating at least the viewpoint interval at the time of image capturing and the optical axis interval at the time of image display. A stereoscopic image display device characterized by being provided. 2. A displacement amount detecting unit for detecting a displacement amount of a pixel position corresponding to each image captured by the two left and right cameras, wherein the interpolation unit is a displacement amount of each of the detected pixel positions. The interpolation is performed such that the ratio of the shift amount of the image displayed on the display unit to the shift amount of the image displayed on the display unit is equal to the ratio of the viewpoint interval during image capturing and the optical axis interval during image display. Stereoscopic image display device. 3. A shift amount detecting means for detecting a shift amount of a pixel position corresponding to each image photographed by the two left and right cameras, wherein the interpolation means is a shift amount of each detected pixel position. Interpolation is performed so that the ratio of the shift amount of the image displayed on the display unit to the displacement amount of the image displayed on the display unit is equal to a constant multiple of the ratio of the viewpoint interval during image capturing and the optical axis interval during image display. 1. The stereoscopic image display device according to 1. 4. Image conversion in which the two cameras are set with convergence on the optical axis, and the images taken by the two set cameras are converted into images in which the optical axes are parallel. The stereoscopic image display device according to claim 1, further comprising a unit, wherein the interpolation unit interpolates the converted image. 5. A stereoscopic image display method for displaying images captured by two left and right cameras and observing the displayed images as a stereoscopic image with both eyes, wherein at least the image displayed on the display unit is A stereoscopic image display method, characterized in that, when one is generated by interpolation using the other image, at least one is interpolated based on a viewpoint interval during image capturing and an optical axis interval during image display.