JP2023059137A - Image composite device, image composite method, and program - Google Patents

Abstract

To provide an image composite device capable of appropriately composing a stereo image and a three-dimensional object.SOLUTION: An image composite device 1 comprises: a storage unit 11 into which a first stereo image having a first right-eye image and a first left-eye image is stored; an acceptance unit 12 which accepts operations for a three-dimensional object; an arrangement unit 13 which arranges the three-dimensional object correspondingly to the operations in a three-dimensional virtual space where both the first right-eye image and the first left-eye image are arranged so that a straight line connecting a right-eye viewpoint and a point corresponding to infinity on the first right-eye image intersects a straight line connecting a left-eye viewpoint and a point corresponding to infinity on the first left-eye image; a generation unit 14 which generates a second stereo image including a second right-eye image representing an image from the right-eye viewpoint while the first right-eye image is displayed and a second left-eye image representing an image from the left-eye viewpoint while the first left-eye image is displayed, in the three-dimensional virtual space in which the three-dimensional object is arranged; and an output unit 15 which outputs the second stereo image.SELECTED DRAWING: Figure 1

Description

The present invention relates to an image synthesizing device and the like for synthesizing a stereo image and a three-dimensional object.

2. Description of the Related Art Conventionally, by displaying a stereo image on a head-mounted display (HMD), a stereoscopic image is provided to a user wearing the HMD (see, for example, Patent Document 1). Also known is three-dimensional computer graphics (3DCG), such as rendering processing for arranging a three-dimensional object in a three-dimensional virtual space and generating a two-dimensional image from an arbitrary viewpoint.

Japanese Patent Application Laid-Open No. 2020-071718

However, no technique has been known for properly synthesizing a three-dimensional object with a stereo image. For example, if one two-dimensional image generated by 3DCG is combined with the right-eye image and the left-eye image included in the stereo image, the three-dimensional object will be displayed two-dimensionally. was there.

SUMMARY OF THE INVENTION An object of the present invention is to provide an image synthesizing apparatus and the like for appropriately synthesizing a three-dimensional object with a stereo image.

To achieve the above object, an image synthesizing device according to an aspect of the present invention includes a storage unit storing a first stereo image having a first right-eye image and a first left-eye image; and a first straight line connecting the right-eye viewpoint and a point corresponding to infinity in the first right-eye image, and the left-eye viewpoint and the first left-eye image. A three-dimensional virtual space in which the first right-eye image and the first left-eye image are arranged so that the second straight line connecting the point corresponding to infinity of the image intersects (however, the point corresponding to infinity is an arrangement unit for arranging a three-dimensional object according to the operation received by the reception unit (excluding the three-dimensional virtual space in which the first right-eye image and the first left-eye image are arranged so as to overlap); In the three-dimensional virtual space in which the dimensional object is arranged, a second right-eye image and a first left-eye image, which are images from a right-eye viewpoint in a state where the first right-eye image is displayed, are displayed. and an output unit configured to output the second stereo image.

With such a configuration, for example, according to the arrangement of the first right-eye image and the first left-eye image in the three-dimensional virtual space, the three-dimensional object to be operated is not located behind both images. can be obtained. Therefore, the three-dimensional object and the first stereo image can be appropriately synthesized. Further, for example, even when the three-dimensional object is located on the back side of the first stereo image, the three-dimensional object can be moved to the first stereo image by moving the first right eye image and the first left eye image by a shorter distance. can be arranged so as not to be located on the back side of the stereo image of the

Further, in the image synthesizing device according to one aspect of the present invention, the first right-eye image and the first left-eye image may be arranged such that points corresponding to a predetermined distance overlap in the three-dimensional virtual space.

With such a configuration, when the three-dimensional object is positioned closer to the viewer than the predetermined distance, the three-dimensional object is not positioned behind the first stereo image.

Further, the image synthesizing device according to one aspect of the present invention further includes a specifying unit that specifies a distance of a predetermined object included in the first stereo image, and the predetermined distance is the longest distance specified by the specifying unit. may exceed.

With such a configuration, when the three-dimensional object is located on the front side of the predetermined object whose distance is specified by the specifying unit, the three-dimensional object is not located on the back side of the first stereo image. Become.

Further, in the image synthesizing device according to one aspect of the present invention, the predetermined objects may be all objects included in the first stereo image.

With such a configuration, when the three-dimensional object is located on the front side of the farthest object included in the first stereo image, the three-dimensional object is not located on the back side of the first stereo image. It will be.

Further, in the image synthesizing device according to the aspect of the present invention, the receiving unit may also receive information designating a predetermined object, and the predetermined object may be an object designated by the information received by the receiving unit. .

With such a configuration, when the three-dimensional object is located on the front side of the object specified by the information received by the reception unit, the three-dimensional object is not located on the back side of the first stereo image. become.

Further, in the image synthesizing device according to the aspect of the present invention, when the distance to the three-dimensional object approaches the predetermined distance, the placement unit may increase the predetermined distance and place the first stereo image. good.

With such a configuration, by rearranging the first stereo image, it is possible to avoid positioning the three-dimensional object on the back side of the first stereo image.

Further, the image synthesizing device according to one aspect of the present invention includes a storage unit storing a first stereo image having a first right-eye image and a first left-eye image; An arrangement for arranging a three-dimensional object according to the operation accepted by the accepting unit in a three-dimensional virtual space in which the first right-eye image and the first left-eye image are arranged. and a second right-eye image and a first left-eye image, which are images from the right-eye viewpoint in a state where the first right-eye image is displayed in the three-dimensional virtual space in which the three-dimensional object is arranged. a generation unit that generates a second stereo image including a second left-eye image that is an image from a left-eye viewpoint in a state where is displayed; and an output unit that outputs the second stereo image. is.

With such a configuration, a stereo image and a three-dimensional object can be appropriately synthesized. That is, it is possible to generate a second stereo image in which both the object included in the first stereo image and the three-dimensional object are stereoscopically displayed.

Further, in the image synthesizing device according to the aspect of the present invention, the first right-eye image and the first left-eye image may be arranged such that points corresponding to infinity overlap in the three-dimensional virtual space.

With such a configuration, for example, it is possible to overlap an object included in the first stereo image and a three-dimensional object.

Further, in the image synthesizing device according to the aspect of the present invention, the first stereo image may be captured by a pair of cameras arranged so that their optical axes are parallel.

Further, in the image synthesizing device according to the aspect of the present invention, the first stereo image may be captured by a pair of cameras arranged so that their optical axes intersect.

Further, in the image synthesizing device according to one aspect of the present invention, the first stereo image and the second stereo image may be moving images.

With such a configuration, for example, when the first stereo image is an image of a surgical operation, it is possible to superimpose an instrument such as forceps, which is a three-dimensional object, on an instrument such as forceps included in the first stereo image. Thus, a novice operator can imitate the operation of an experienced operator in the virtual space, and can train for surgery.

Further, in the image synthesizing device according to one aspect of the present invention, the three-dimensional virtual space of the object included in the first stereo image is obtained from the first right-eye image and the first left-eye image arranged in the three-dimensional virtual space. may further include a determination unit that specifies a position in the 3D object and determines the relationship between the specified position and the position of the three-dimensional object, and the output unit may also output the determination result of the determination unit.

With such a configuration, for example, a user operating a three-dimensional object can know whether the three-dimensional object is being operated appropriately.

Further, an image synthesizing method according to an aspect of the present invention includes a step of accepting an operation of a three-dimensional object placed in a three-dimensional virtual space; A first straight line connecting the right-eye viewpoint and a point corresponding to infinity on the first right-eye image, and a first straight line connecting the left-eye viewpoint and a point corresponding to infinity on the first left-eye image. A three-dimensional virtual space arranged so that the two straight lines intersect (except for the three-dimensional virtual space where the first right-eye image and the first left-eye image are arranged so that the points corresponding to infinity overlap) ), placing a three-dimensional object in accordance with the accepted operation; and an image viewed from the right-eye viewpoint in a state in which the first right-eye image is displayed in the three-dimensional virtual space in which the three-dimensional object is arranged. and a second stereo image including a second left-eye image that is an image from a left-eye viewpoint in a state where the first left-eye image is displayed; and outputting two stereo images.

According to the image synthesizing device and the like according to one aspect of the present invention, it is possible to appropriately synthesize a stereo image and a three-dimensional object.

1 is a block diagram showing the configuration of an image synthesizing device according to an embodiment of the present invention; 3 is a flow chart showing the operation of the image synthesizing device according to the same embodiment; FIG. 4 is a diagram for explaining stereo image shooting in the same embodiment; FIG. 4 is a diagram for explaining generation of stereo images in the same embodiment; FIG. 4 is a diagram showing an example of display of stereo images after synthesis in the same embodiment; FIG. 4 is a diagram showing an example of display of stereo images after synthesis in the same embodiment; A diagram for explaining a point corresponding to infinity in the same embodiment. FIG. 4 is a diagram for explaining generation of stereo images in the same embodiment; FIG. 4 is a diagram showing an example of display of stereo images after synthesis in the same embodiment; FIG. 4 is a diagram for explaining generation of stereo images in the same embodiment; A diagram for explaining a point corresponding to infinity in the same embodiment. A diagram for explaining a point corresponding to infinity in the same embodiment. FIG. 4 is a diagram showing an example of a stereo image and a three-dimensional object arranged in a three-dimensional virtual space according to the same embodiment; FIG. 4 is a diagram showing an example of a stereo image and a three-dimensional object arranged in a three-dimensional virtual space according to the same embodiment; FIG. 3 is a block diagram showing another configuration of the image synthesizing device according to the same embodiment; A diagram showing an example of a stereo image arranged in a three-dimensional virtual space in the same embodiment. A diagram showing an example of a stereo image arranged in a three-dimensional virtual space in the same embodiment. A diagram showing an example of a stereo image arranged in a three-dimensional virtual space in the same embodiment. FIG. 3 is a block diagram showing another configuration of the image synthesizing device according to the same embodiment; Schematic diagram showing an example of the appearance of a computer system according to the same embodiment A diagram showing an example of a configuration of a computer system according to the same embodiment.

An image synthesizing apparatus and an image synthesizing method according to the present invention will be described below using embodiments. In the following embodiments, constituent elements and steps with the same reference numerals are the same or correspond to each other, and repetitive description may be omitted. The image synthesizing device and image synthesizing method according to the present embodiment generate a second stereo image by synthesizing the first stereo image and the three-dimensional object arranged in a three-dimensional virtual space. .

FIG. 1 is a block diagram showing the configuration of an image synthesizing device 1 according to this embodiment. The image synthesizing device 1 according to this embodiment includes a storage unit 11 , a reception unit 12 , an arrangement unit 13 , a generation unit 14 and an output unit 15 . The image synthesizing device 1 may be, for example, a general-purpose device such as a personal computer, a smartphone, or a tablet terminal, or may be a dedicated device that synthesizes a stereo image and a three-dimensional object. Further, the image synthesizing device 1 may be, for example, a stand-alone device or a server device in a server-client system. In the latter case, the reception unit 12 and the output unit 15 may receive inputs and output information via communication lines such as the Internet. In this embodiment, the case where the image synthesizing device 1 is a stand-alone device will be mainly described.

The storage unit 11 stores a first stereo image having a first right-eye image and a first left-eye image. The first stereo image may be, for example, a moving image or a still image. In this embodiment, a case where the first stereo image is a moving image will be mainly described. The first stereo image may be, for example, a photographed image or an image generated by 3DCG or the like. In this embodiment, a case where the first stereo image is a photographed stereo image will be mainly described.

In this embodiment, the case where all the data of the first stereo image, which is the moving image, is stored in the storage unit 11 will be mainly described, but this does not have to be the case. For example, the first stereo image, which is a moving image, may be received in real time by the image synthesizing device 1 and only part of the data to be processed may be stored in the storage unit 11 . Even in such a case, since each frame constituting the first stereo image, which is a moving image, is stored at least temporarily in the storage unit 11, the first stereo image is stored in the storage unit 11. can be said to be

In addition, in the present embodiment, a case where the first stereo image, which is a photographed image, is photographed by a pair of cameras arranged so that their optical axes are parallel will be mainly described. A case in which images are taken by a pair of cameras arranged so as to intersect will be described later.

The process by which the first stereo image is stored in the storage unit 11 does not matter. For example, the first stereo image may be stored in the storage unit 11 via a recording medium, or the first stereo image transmitted via a communication line or the like may be stored in the storage unit 11. Alternatively, the storage unit 11 may store the first stereo image input from an imaging device or the like. The storage unit 11 is preferably implemented by a non-volatile recording medium, but may be implemented by a volatile recording medium. The recording medium may be, for example, a semiconductor memory, a magnetic disk, an optical disk, or the like.

The accepting unit 12 accepts an operation of a three-dimensional object placed in a three-dimensional virtual space. This operation may be, for example, an operation related to the posture, position, shape, etc. of the three-dimensional object. The operation accepted by the accepting unit 12 may be any operation as long as it causes a change in the orientation, position, shape, etc. of the three-dimensional object placed in the three-dimensional virtual space. For example, when the three-dimensional object is forceps used in surgery, the operation related to the shape of the three-dimensional object may be an operation of opening or closing the tip of the forceps. The reception unit 12 receives an operation of a three-dimensional object from, for example, a controller of a VR (Virtual Reality) headset (for example, Oculus Quest 2, etc.) outputting a second stereo image, which will be described later, from the image synthesizing device 1 . Alternatively, the operation of the three-dimensional object may be received from other input devices such as a joystick or mouse. The acceptance of the operation of the three-dimensional object may be, for example, the acceptance of the result of detecting changes in position and angle in the three-dimensional real space using a sensor such as an acceleration sensor or a gyro sensor. It may be acceptance of input to an input device such as. The reception unit 12 normally receives an operation of a three-dimensional object in real time from an input device such as the HMD 2 or a controller such as a VR headset. The accepting unit 12 may accept an operation of a three-dimensional object via an input device, a communication line, or the like. Note that the receiving unit 12 may or may not include a device for receiving (for example, an input device or a communication device). Further, the reception unit 12 may be realized by hardware, or may be realized by software such as a driver for driving a predetermined device.

The arrangement unit 13 arranges a three-dimensional object according to the operation received by the reception unit 12 in the three-dimensional virtual space in which the first right-eye image and the first left-eye image are arranged. Since the position, posture, shape, etc. of the 3D object change according to the received operation, the arrangement unit 13 changes the 3D object in the 3D virtual space according to the operation. Since the placement of the three-dimensional object by the placement unit 13 is performed in accordance with the operation accepted by the acceptance unit 12, for example, when the operation is continuously accepted, the placement of the three-dimensional object in the three-dimensional virtual space Placement will continue to change. A method of arranging a three-dimensional object arranged in a three-dimensional virtual space in accordance with an operation is already known, and a detailed description thereof will be omitted. Information on the three-dimensional object arranged in the three-dimensional virtual space may be stored in the storage unit 11, for example.

Note that the relationship between the arrangement positions of the first right-eye image and the first left-eye image included in the first stereo image and the right-eye viewpoint and left-eye viewpoint in the three-dimensional virtual space is usually predetermined. there is Although a specific relationship between the two will be described later, for example, (A) the first right-eye image and the first left-eye image may be arranged on the front side in the parallel line-of-sight direction. Also, for example, (B) the arrangement of the first right-eye image and the first left-eye image may be arranged such that points corresponding to infinity overlap.

A three-dimensional object is usually placed between the first stereo image and the viewpoint. This is because when the first stereo image exists between the three-dimensional object and the viewpoint, the second stereo image after rendering does not include the three-dimensional object.

The generation unit 14 generates a second right-eye image and a first left-eye image, which are images from a right-eye viewpoint in a state where the first right-eye image is displayed, in a three-dimensional virtual space in which a three-dimensional object is arranged. A second stereo image is generated that includes a second left-eye image that is an image from the left-eye viewpoint in a state where the for-eye image is displayed. That is, when generating the second right-eye image, it is preferable to display the first right-eye image and not display the first left-eye image in the three-dimensional virtual space. Also, when generating the second left-eye image, it is preferable to display the second right-eye image and not display the second left-eye image in the three-dimensional virtual space. If the first left-eye image does not fit in the second right-eye image, the first left-eye image may be displayed during rendering using the right-eye viewpoint. The same is true when rendering using the left-eye viewpoint. The second stereo image is preferably generated so that the stereoscopically displayed object and the three-dimensional object included in the first stereo image are also stereoscopically displayed in the second stereo image. is. Also, the angle of view during rendering may be, for example, the same as the angle of view of the camera that captured the first stereo image, or may be different.

Each position of the right-eye viewpoint and the left-eye viewpoint in the three-dimensional virtual space may be determined in advance, or may be changed in real time, for example. In the latter case, for example, each position of the right-eye viewpoint and the left-eye viewpoint may be changed according to the position and orientation of the HMD 2 . Also, the arrangement of the first stereo image in the three-dimensional virtual space may be changed according to the change of the viewpoint. This is because the relative relationship between the position of the viewpoint and the positions where the first right-eye image and the first left-eye image are arranged is usually determined in advance. Note that changing the viewpoint according to the position and orientation of the HMD 2 is already known, and detailed description thereof will be omitted. Moreover, it is preferable that the line-of-sight direction is directed toward the center of each corresponding image. For example, the line-of-sight direction of the right-eye viewpoint is preferably directed toward the center of the right-eye image. In this embodiment, for convenience of explanation, the case where each line-of-sight direction of the right-eye viewpoint and left-eye viewpoint is the horizontal direction will be mainly described, but each line-of-sight direction may be a direction other than the horizontal direction. Needless to say.

When the first stereo image is a moving image, the second stereo image is generated by synthesizing each frame of the moving image with the three-dimensional object. become an image. In this case, for example, the first stereo image is reproduced in a three-dimensional virtual space, and the first stereo image being reproduced with the arrangement of the three-dimensional object changed according to the accepted operation, and a second stereo image containing the three-dimensional object modified in response to the manipulation may be generated. In addition, in the second right-eye image and the second left-eye image included in the second stereo image, the frame (that is, outer edge) of the first stereo image may be displayed, or otherwise. good. In the latter case, the frame of the first right-eye image (or the first left-eye image) is the frame of the second right-eye image (or the second left-eye image). stereo images may be generated.

Note that the placement unit 13 and the generation unit 14 can also be implemented using a real-time 3D development platform such as Unity (Unity Technologies). 8A and 8B are diagrams showing a first stereo image and surgical forceps, which are three-dimensional objects, placed in a three-dimensional virtual space of Unity. In FIG. 8A, the positions of the first right-eye image and the first left-eye image are shifted. For example, in the case of (A) above, they are arranged in this manner. Although both images are originally on the same plane, they are slightly shifted for explanation. In FIG. 8B, the first right-eye image and the first left-eye image are superimposed. For example, in the case of (B) above, they are arranged in this manner. The generation of images by the generation unit 14 is known as rendering processing in 3DCG, and detailed description thereof will be omitted. Needless to say, texture mapping or the like may be performed at the time of rendering.

The output unit 15 outputs the second stereo image generated by the generation unit 14 . Here, this output may be, for example, display on a display device (e.g., HMD, VR headset, etc.) that displays stereo images, transmission to a predetermined device via a communication line, or transmission to a recording medium. It may be stored or handed over to another component. In this embodiment, as described above, the case where the output unit 15 outputs the second stereo image to the HMD 2 will be mainly described. Note that the output unit 15 may or may not include a device for output (for example, a display device or a communication device). Also, the output unit 15 may be realized by hardware, or may be realized by software such as a driver for driving those devices.

Next, the operation of the image synthesizing apparatus 1 according to the present embodiment, that is, the image synthesizing method will be described with reference to the flowchart of FIG. In this flowchart, a second stereo image is generated by synthesizing a three-dimensional object corresponding to an operation received by the reception unit 12 with a first stereo image, which is a moving image stored in the storage unit 11. A case will be described.

(Step S101) The generator 14 determines whether to generate the second stereo image. If the second stereo image is to be generated, the process proceeds to step S102. If not, the process of step S101 is repeated until it is determined that the second stereo image is to be generated. Note that the generation unit 14 may determine, for example, to generate the second stereo image periodically. More specifically, the generation unit 14 may determine to generate the second stereo image for each time interval between frames of the second stereo image.

(Step S102) The reception unit 12 determines whether or not the operation of the three-dimensional object has been received. If the operation has been accepted, the process proceeds to step S103; otherwise, the process proceeds to step S104.

(Step S<b>103 ) The placement unit 13 places the three-dimensional object in the three-dimensional virtual space according to the operation accepted by the acceptance unit 12 . The placement of the 3D object may be to change the placement of the 3D object placed in the 3D virtual space, for example, to change the position, angle, shape, etc. of the 3D object.

(Step S104) The generation unit 14 displays the first right-eye image in the three-dimensional virtual space and generates the second right-eye image in a state in which the first left-eye image is not displayed. A second left-eye image is displayed and a second left-eye image is generated in a state in which the first right-eye image is not displayed, thereby generating a second stereo image. Note that the generation of the second right-eye image and the second left-eye image may be generation of one frame for the right eye and one frame for the left eye. In this case, the first right-eye image and the first left-eye image arranged in the three-dimensional virtual space may be changed frame by frame each time the process of step S104 is executed. The generated pair of frames may be stored in a recording medium (not shown) or the storage unit 11, for example.

(Step S105) The output unit 15 outputs the second stereo image generated in step S104. This output may be, for example, one frame for the right eye and one frame for the left eye. Then, the process returns to step S101. In this way, by repeating the processing of steps S101 to S105, the HMD 2 displays a second stereo image obtained by synthesizing the first stereo image and the three-dimensional object whose arrangement is changed according to the operation. It will be.

In addition, in the flowchart of FIG. 2, for example, the processing of steps S101, S104, and S105 and the processing of steps S102 and S103 may be performed in parallel. In addition, in the flowchart of FIG. 2, the processing ends when the power is turned off or an interrupt for processing end occurs.

Next, photographing of the first stereo image and synthesis of the first stereo image and the three-dimensional object will be specifically described.

FIG. 3 is a diagram for explaining the shooting of the first stereo image. In FIG. 3, the first stereo image is captured using the right-eye camera 21a and the left-eye camera 21b. That is, the first right-eye image 22a is captured by the right-eye camera 21a, and the first left-eye image 22b is captured by the left-eye camera 21b. The right-eye camera 21a and the left-eye camera 21b usually have the same specifications, and as a result, the first right-eye image 22a and the first left-eye image 22b also have the same number of pixels. In FIG. 3, it is assumed that the first stereo image is captured by a pair of cameras 21a and 21b arranged so that their optical axes are parallel. It is also assumed that the optical axes of both cameras 21a and 21b are both in the horizontal plane. The distance between the optical axis of the right-eye camera 21a and the optical axis of the left-eye camera 21b may be, for example, about the average human interpupillary distance. In this example, it is assumed that the object 20 to be photographed is a cone arranged so that the central axis is in the vertical direction. It is assumed that the object 20 is included in the first right-eye image 22a and the first left-eye image 22b. Note that FIG. 3 shows the state in which the cameras 21a and 21b are viewed from the vertical direction perpendicular to the horizontal plane. Strictly speaking, the object 20 should have a circular shape. is displayed in a triangle. Suppose that it is the same also in other drawings.

Next, when the first right-eye image 22a and the first left-eye image 22b are arranged in the three-dimensional virtual space (A) on the front side of the parallel viewing direction, (B) they correspond to infinity. Each of the cases where points are arranged so as to overlap will be described.

(A) Arranging each on the front side of the parallel line-of-sight direction FIG. 4A shows the first right-eye image 22a and the first left-eye image 22b on the front side of the parallel line-of-sight direction in the three-dimensional virtual space. It is a figure for demonstrating the case where it arrange|positions. In FIG. 4A, the right-eye viewpoint 23a and the left-eye viewpoint 23b are arranged apart by the distance between the pupils and the distance between the optical axes of the stereo cameras 21a and 21b. Also, the line-of-sight directions indicated by arrows are parallel and face the same direction (upward in FIG. 4A). In addition, in FIG. 4A, it is assumed that the line-of-sight direction exists in the horizontal plane. That is, in FIG. 4A, as in FIG. 3, it is assumed that the viewpoints 23a and 23b and the arranged images 22a and 22b are viewed from the vertical direction perpendicular to the horizontal plane. Also, the first right-eye image 22a and the first left-eye image 22b are orthogonal to the image on the front side of the right-eye viewpoint 23a and the left-eye viewpoint 23b, respectively, and the direction of the line of sight is directed toward the center of the image. are arranged as It is also assumed that the positional relationship between the right-eye viewpoint 23a and the first right-eye image 22a is the same as the positional relationship between the left-eye viewpoint 23b and the first left-eye image 22b. Also, the first right-eye image 22a and the first left-eye image 22b are arranged so as to be included in the same plane. The distance from the right-eye viewpoint 23a and the left-eye viewpoint 23b to the plane containing the first right-eye image 22a and the first left-eye image 22b is, for example, U/(2tan(φ/2)). good too. Here, U is the horizontal length of the first right-eye image 22a and the first left-eye image 22b, and φ is the horizontal angle of view of the right-eye viewpoint 23a and the left-eye viewpoint 23b. . Note that FIG. 4A shows the case where the first right-eye image 22a and the first left-eye image 22b are arranged apart from each other on the right side and the left side, but the two images are arranged such that they partially overlap each other. (see, eg, FIG. 8A).

In FIG. 4A, a second right-eye image is generated from the right-eye viewpoint 23a while the first right-eye image 22a is displayed, and a left-eye image 22b is displayed while the first left-eye image 22b is displayed. Assume that second left-eye images from viewpoint 23b are generated and their second stereo images are output to HMD 2 . Then, the user viewing the second stereo image will see the object 20 behind the arranged first right-eye image 22a and first left-eye image 22b, as indicated by the dashed lines in FIG. 4A. appears to exist.

In FIG. 4A, it is assumed that the user wearing the HMD 2 manipulates the three-dimensional object, which is the robot hand, using the controller to place the three-dimensional object 25a at the position indicated by the three-dimensional object 25a. Then, as shown in FIG. 4B, the user can see the robot hand on the front side and the object 20 on the back side. Therefore, when the user tries to touch the object 20 with the robot hand, it is necessary to move the three-dimensional object farther away from the viewpoint. Then, when the user operates the controller to move the three-dimensional object to the back side and place it at the position indicated by the three-dimensional object 25b, the front end side of the three-dimensional object 25b becomes the rear surface of the first right-eye image 22a. It will be positioned on the side and disappear from the viewpoint. As a result, the user cannot see the tip side of the robot hand, as shown in FIG. 4C. This is because the first right-eye image 22a is normally opaque. In addition, in FIG. 4C, the portion of the robot hand that cannot be seen is indicated by a dashed line. In this way, when the first right-eye image 22a and the first left-eye image 22b are respectively arranged on the front side of the parallel line-of-sight direction, the object displayed in the first stereo image (for example, the object 20 ) and the three-dimensional object. That is, it becomes difficult to overlap both in the second stereo image. 4B and 4C are diagrams showing stereo images on a two-dimensional plane, and are different from the actual second stereo images. The same applies to other figures showing stereo images.

Here, the disadvantages of arranging the right-eye image and the left-eye image respectively on the front side of the parallel line-of-sight direction have been described. In a situation where the two are not related, there is no demerit in synthesizing the stereo image and the three-dimensional object as described above. For example, if the first stereo image is an image of a landscape and the three-dimensional object is a character that dances in response to a user's operation, the character may dance in the foreground of the landscape. Synthetic methods can be used.

(B) Case where points corresponding to infinity are arranged so as to overlap FIG. 5 is a diagram for explaining points corresponding to infinity in a stereo image. As shown in FIG. 5, assuming that the right-eye imaging position, the left-eye imaging position, the angle of view, and the imaging direction indicated by the arrow are set, the right-eye image and the left-eye image are obtained by the right-eye imaging. An image is obtained by projecting an object or the like within the range of the angle of view that can be seen in the shooting direction from the position onto the virtual plane in the drawing. Note that the virtual plane is a plane perpendicular to the shooting direction. In this case, a point existing at infinity on the forward side in the imaging direction is projected onto the positions of points α and β on the virtual plane. That is, the point α of the image for the right eye and the point β of the image for the left eye are points corresponding to infinity. Note that the points α and β are the points where the right-eye image and the imaging direction extending from the right-eye imaging position intersect, and the points where the left-eye image and the imaging direction extending from the left-eye imaging position intersect, respectively. Therefore, when arranging the first right-eye image and the first left-eye image in the three-dimensional virtual space so that the points corresponding to infinity overlap, both are arranged at the same position. That is, both images should be superimposed and arranged.

FIG. 6A is a diagram for explaining a case in which the photographed first right-eye image 22a and first left-eye image 22b are arranged so that points corresponding to infinity overlap in the three-dimensional virtual space. be. In FIG. 6A, similarly to FIG. 4A, the right-eye viewpoint 23a and the left-eye viewpoint 23b are separated by the distance between the pupils and the distance between the optical axes of the stereo cameras 21a and 21b. In this case, the line-of-sight directions indicated by the arrows face the centers of the first right-eye image 22a and the first left-eye image 22b. Also in FIG. 6A, it is assumed that the line-of-sight direction exists in the horizontal plane. Moreover, FIG. 6A is a view seen from the vertical direction. The first right-eye image 22a and the first left-eye image 22b shown in FIG. 6A are each moved by the same distance from the position shown in FIG. 4A until they overlap in the plane direction of the images. .

In FIG. 6A, the first right-eye image 22a is displayed, and the second right-eye image is generated from the right-eye viewpoint 23a while the first left-eye image 22b is not displayed. A second left-eye image is generated from the left-eye viewpoint 23b in a state in which the left-eye image 22b is displayed and the first right-eye image 22a is not displayed, and these second stereo images are displayed on the HMD 2. Suppose it is output. Then, the user viewing the second stereo image will see the object 20 in front of the arranged first right-eye image 22a and first left-eye image 22b, as indicated by broken lines in FIG. 6A. appears to exist.

In FIG. 6A, it is assumed that the user wearing the HMD 2 manipulates the three-dimensional object, which is the robot hand, using the controller to place it at the position indicated by the three-dimensional object 25c. Then, as shown in FIG. 6B, the user sees the robot hand in the vicinity of the object 20 . Therefore, unlike the situation (A) above, the user can operate the controller to place the tip of the robot hand on the object 20 . In this way, by arranging the first right-eye image 22a and the first left-eye image 22b so that the points corresponding to infinity overlap, an object displayed in the first stereo image (for example, the object 20 ) and the three-dimensional object can be easily adjusted. That is, an object displayed as a stereo image and a three-dimensional object can be superimposed. This is because the point on the first right-eye image 22a corresponding to infinity and the point on the first left-eye image 22b corresponding to infinity overlap, so the position corresponding to infinity is This is because the three-dimensional object can be placed anywhere from the closest point to the point of view to infinity by placing the three-dimensional object at any point within the range from the point of view to the image. .

Next, a case where the first stereo image is captured by a pair of cameras arranged so that their optical axes intersect will be described. For example, when stereo images are captured using a surgical assistance robot, da Vinci, the optical axes are sometimes arranged so as to cross each other. In this case, as shown in FIG. 7A, the imaging direction indicated by the arrow from the right-eye imaging position (that is, the optical axis of the camera) and the imaging direction indicated by the arrow from the left-eye imaging position are the lines of sight. will intersect on the front side of (that is, the front side of the camera). The positions corresponding to infinity between the right-eye image and the left-eye image photographed in this manner are points α1 and β1 in the figure. These points are the intersections of two parallel straight lines extending from the right-eye imaging position and the left-eye imaging position and the right-eye image and the left-eye image so that the angles formed with the imaging direction are equal. Since the situation on the right-eye side and the situation on the left-eye side are in contrast, if the point α1 can be specified in the right-eye image, the point β1 can also be specified in the left-eye image. Therefore, the identification of the position of the point α1 in the image for the right eye will be described below.

As shown in FIG. 7B, let X be the distance from the intersection of the shooting direction and the right-eye image to point α1, let L be the distance from the right-eye shooting position to the shot image, and let L be the horizontal length of the right-eye image. is U, the angle of view in the horizontal direction is φ, and the angle between the photographing direction and the straight line connecting the photographing position for the right eye and the point α1 is θ, the following equation is obtained.
tan(φ/2)=U/(2L)
tan θ=X/L

Therefore, X becomes as follows.
X=U×tan θ/(2 tan (φ/2))

Here, φ is known because it is determined by the camera used for photographing. θ is also known because 2θ is the angle formed by the line-of-sight directions of the right eye side and the left eye side in FIG. 7A. Therefore, when arranging the first right-eye image 22a and the first left-eye image 22b photographed as shown in FIG. 7A so that points corresponding to infinity overlap, the horizontal width of the images and θ and φ are used to calculate X as described above. , and the point β1, which is horizontally away from the center by X, should be placed on top of each other. When arranging the first right-eye image 22a and the first left-eye image 22b in the three-dimensional virtual space, the right-eye imaging position in FIG. 7B may be the right-eye viewpoint. That is, the first right-eye image 22a and the first left-eye image 22a are placed on a plane at a distance L from the right-eye viewpoint and the left-eye viewpoint arranged in the three-dimensional virtual space so that the points α1 and β1 overlap each other. You may arrange|position the image 22b for. Here, L is L=U/(2tan(φ/2)) from the above equation. In addition, the first right-eye image 22a and the first left-eye image 22b pass through the midpoint of the line segment connecting the right-eye viewpoint 23a and the left-eye viewpoint 23b, and are symmetrical about a plane perpendicular to the line segment. is preferably arranged so that That is, when the images arranged as shown in FIG. 6A are viewed from the front side, the first right-eye image 22a is moved to the left by X in the plane including the images, and the first left-eye image 22b is moved to the left. , to the right by X in the plane containing the image. As a result, the first right-eye image 22a and the first left-eye image 22b are arranged as shown in FIG. 6C. In addition, in FIG. 6C, for convenience of explanation, the first right-eye image 22a and the first left-eye image 22b are shifted and displayed. image 22b exists on the same plane. Also in this case, the line-of-sight direction is set so as to face the center of each corresponding image. In this way, even when the first stereo image is captured by a pair of cameras arranged so that their optical axes intersect, the first right-eye image and The first left-eye image can be arranged, and as a result, the positional relationship between the object included in the first stereo image and the three-dimensional object can be easily adjusted.

As described above, according to the image synthesizing device 1 and the image synthesizing method according to the present embodiment, the first right-eye image and the first left-eye image included in the first stereo image in the three-dimensional virtual space, 3D objects are arranged, a new right-eye image is generated while the first right-eye image is displayed, and a new left-eye image is generated while the first left-eye image is displayed. By doing so, it is possible to generate a second stereo image by synthesizing the first stereo image and the three-dimensional object. By synthesizing the two in this way, it is possible to generate a second stereo image capable of stereoscopically displaying both the object included in the stereo image and the three-dimensional object. Also, by arranging the first right-eye image and the first left-eye image so that points corresponding to infinity overlap in the three-dimensional virtual space, the object included in the stereo image and the three-dimensional object It becomes possible to adjust the positional relationship appropriately. For example, it becomes possible to display a three-dimensional object superimposed on an object included in a stereo image. Also, when the 3D object and the object included in the stereo image are superimposed, it is possible to avoid a situation in which a part of the 3D object is positioned behind the stereo image and becomes invisible.

Note that, in the present embodiment, the case where the first and second stereo images are moving images has been mainly described, but this does not have to be the case. The first and second stereo images may be still images. In this case, the second stereo image, which is a still image, is generated by combining the first stereo image, which is a still image, with the three-dimensional object arranged according to the operation.

Next, a usage example of the image synthesizing device 1 according to the present embodiment will be described.

The first stereo image is a moving image obtained by photographing the action of an expert, the three-dimensional object corresponds to an instrument or the like used by the expert, and the user operating the three-dimensional object is an expert. It may be a beginner who imitates the movement of the person. In this case, a second stereo image is generated by synthesizing the three-dimensional object operated by the beginner with the first stereo image showing the action of the expert. More specifically, the expert's motion may be a surgical motion, a motion for playing a sport, or a motion for playing a musical instrument. Three-dimensional objects are, for example, instruments such as forceps used in surgery, instruments such as baseball bats, golf clubs, tennis rackets, badminton rackets, and table tennis rackets used in sports, and musical instruments used in music performances. There may be. In this way, a novice can imitate the actions of an expert and practice surgery, playing sports, playing a musical instrument, and the like. In this case, it is considered important to superimpose the three-dimensional object on the object included in the first stereo image. Preferably a synthesis is performed.

The first stereo image is a moving image of a person such as an actor, an actress, or a singer, and the three-dimensional object is a user's hand or the like. You may be able to shake hands with an actor or the like. In this case, the second stereo image is a combination of a person such as an actor and a three-dimensional object such as a hand operated by the user. In this way the user will be able to shake hands with a celebrity, for example. In this case as well, it is considered important to superimpose the three-dimensional object corresponding to the hand of the user on the position of the hand of a celebrity or the like included in the first stereo image. Preferably, a synthesis is performed between the first stereo image and the three-dimensional object.

The first stereo image is a photographed image of the interior of the building or a still image such as CG, and the three-dimensional object is furniture. A second stereo image may be generated showing the situation of the furniture placed inside. This second stereo image enables the user to stereoscopically see the situation after the furniture has been placed. In this case as well, it is considered important to superimpose the 3D object on the object included in the first stereo image. Preferably a synthesis is performed.

The first stereo image is a moving image captured by a stereo camera worn by a user in a real space such as a construction site or a repair site. A pointer indicating the position of or an object corresponding to an instrument used in the real space, and the second stereo image is a three-dimensional object synthesized with the first stereo image in the real space. good. The second stereo image may be viewed by experts and users in real space. By manipulating the three-dimensional object, the expert can give instructions to the user in the real space, or demonstrate what the user in the real space should do using the three-dimensional object. Also, the user in the real space can perform appropriate tasks in the real space by viewing the second stereo image including the three-dimensional object operated by the expert and performing the task. In this case, the image synthesizing device 1 may receive the first stereo image in real time and output the second stereo image in real time. In this case as well, it is considered important to superimpose the 3D object on the object included in the first stereo image. Preferably a synthesis is performed.

Although some usage examples have been shown here, it is obvious that the image synthesizing apparatus 1 can be used for other examples of synthesizing the first stereo image and the three-dimensional object, and the image synthesizing apparatus 1 can be used in situations other than the above. It goes without saying that the device 1 may also be used.

Further, the image synthesizing device 1 may determine the positional relationship between the object included in the first stereo image and the three-dimensional object arranged according to the operation received by the receiving unit 12 . The determination regarding the positional relationship may be, for example, a collision determination that determines whether the object included in the first stereo image and the three-dimensional object are in contact, or an overlap determination that determines whether the two overlap. or other determinations relating to the positional relationship between the two. In this case, the image synthesizing device 1 may further include a determining section 16 as shown in FIG. Also, in this case, it is important to adjust the positional relationship between the object included in the first stereo image and the three-dimensional object. is preferably performed.

The determination unit 16 identifies the position in the three-dimensional virtual space of the object included in the first stereo image from the first right-eye image and the first left-eye image arranged in the three-dimensional virtual space. This position identification is, for example, identifying the corresponding points of the first right-eye image and the first left-eye image, and determining the positions of the corresponding points in the three-dimensional virtual space and the positions of the right-eye viewpoint and the left-eye viewpoint. may be performed using For example, the position in the three-dimensional virtual space corresponding to the pair of corresponding points of the first right-eye image and the first left-eye image is the same as the corresponding point of the first right-eye image and the right-eye viewpoint in the three-dimensional virtual space. and a straight line connecting the corresponding point of the first left-eye image and the left-eye viewpoint. If the two straight lines do not intersect due to an error or the like, the midpoint of the position on the two straight lines where the distance between the two straight lines is the shortest is the corresponding pair of the first right-eye image and the first left-eye image. It may be a position in a three-dimensional virtual space corresponding to a point. It should be noted that the process of specifying the points in the three-dimensional space corresponding to the corresponding points on the stereo image in this manner is already known, and detailed description thereof will be omitted. By performing such a process on each corresponding point of the first stereo image, the determination unit 16 determines the corresponding points in the three-dimensional virtual space corresponding to each pixel included in the first right-eye image and the first left-eye image. can be located. Note that the positions specified in this way usually form a plane in the three-dimensional virtual space. This is because stereo images do not show the structure on the back side. Alternatively, the determining unit 16 may identify objects included in the image by performing contour extraction or the like on the first stereo image, and identify the position of each identified object.

After identifying the position in the three-dimensional virtual space of the object included in the first stereo image, the determination unit 16 determines the relationship between the identified position and the position of the three-dimensional object in the three-dimensional virtual space. This determination may be at least one of hit determination and overlap determination. For example, when a plane formed by the position of an object included in the first stereo image in the three-dimensional virtual space and at least part of the three-dimensional object in the three-dimensional virtual space overlap, It may be determined that both are hit. Further, the determining unit 16 may determine, for example, at least a partial surface formed by the position of the object in the three-dimensional virtual space included in the first stereo image and at least a partial surface of the three-dimensional object in the three-dimensional virtual space. and overlap, it may be determined that both overlap. In this case, for example, an index indicating the degree of overlap may be obtained. The index is, for example, the maximum overlap between the plane formed by the position of the object in the three-dimensional virtual space included in the first stereo image and the plane of the three-dimensional object in the three-dimensional virtual space. is 100%, and may be an index indicating the degree of overlap.

The output unit 15 may also output the determination result of the determination unit 16 regarding the positional relationship between the object included in the first stereo image and the three-dimensional object. The output may be, for example, the result of collision judgment or the result of overlap judgment. If the result of collision judgment indicates that both are in collision, an output indicating that both are in collision. Alternatively, if the result of the overlap determination indicates that the two overlap, the output may be an output indicating that the two overlap or an index indicating the degree of overlap. More specifically, the output unit 15 may superimpose a display corresponding to the determination result on the second stereo image and output it, and vibrate the controller operated by the user according to the determination result. Alternatively, a sound may be output according to the determination result, or other output may be performed according to the determination result. For example, as shown in FIG. 6B, the user is operating a robot arm that is a three-dimensional object, and when the robot arm hits the cone-shaped object included in the first stereo image, both hit A display (for example, blinking of the entire screen, etc.), or a vibration output or a sound output indicating that the two have hit each other may be performed. Also, for example, as shown in FIGS. 8A and 8B, the user is operating a forceps that is a three-dimensional object, and the forceps overlaps the forceps included in the first stereo image that is the video of the simulated surgery. Sometimes, display indicating that the two overlap each other (for example, display of a graphic indicating that they overlap) or an index indicating the degree of overlap between the two may be output. In this way, by determining the relationship between the position of the object included in the first stereo image and the position of the three-dimensional object, the user operating the three-dimensional object can, for example, It will be possible to check the position and know whether the three-dimensional object is properly manipulated.

In (B) above, the case where the first right-eye image 22a and the first left-eye image 22b are arranged so that the points corresponding to infinity overlap in the three-dimensional virtual space has been described. It doesn't have to be. For example, when a three-dimensional object exists on the near side (that is, on the viewpoint side) of a position a predetermined distance away from the viewpoint, the first right-eye image 22a and the first left-eye image 22b are three-dimensional objects. In the virtual space, points corresponding to the predetermined distance may be arranged so as to overlap. For example, in FIG. 10, a first right-eye image 22a-1 and a first left-eye image 22b-1 indicated by dashed lines are arranged in a three-dimensional virtual space so that points corresponding to infinity overlap. Then, as shown by the first right-eye image 22a-2 and the first left-eye image 22b-2 in FIG. 10, the first right-eye image 22a-1 and the first left-eye image 22b-1 may be displaced by the same distance in the directions indicated by the arrows respectively in their plane directions. More specifically, the first right eye image 22a-2 is shifted to the right from the first right eye image 22a-1, and the first left eye image 22b-2 is shifted to the first left eye image 22b. It may be moved left from -1. The first left-eye image 22b-2 and the first left-eye image 22b-2 are arranged in the three-dimensional virtual space so that points corresponding to positions N meters away from the viewpoints 23a and 23b overlap. shall be where N is a positive real number. Also, in FIG. 10, it is assumed that the line-of-sight direction exists in the horizontal plane and the situation seen from the vertical direction is shown. 11 and 12 are assumed to be the same. Also, although different codes are used for each position, such as the first right-eye images 22a-1 and 22a-2, if the positions are not particularly distinguished, the first right-eye image 22a and the first right-eye image 22a are used. sometimes called. The same is true for the first left-eye image 22b.

Arranging the first stereo image so that points corresponding to positions N meters away from the viewpoints 23a and 23b are superimposed means, for example, that when the first stereo image is captured, an object existing N meters ahead of the camera is The first left-eye image 22b-2 and the first left-eye image 22b-2 may be arranged so as to overlap each other. Note that the points corresponding to the positions separated by N meters in the first right-eye image 22a and the first left-eye image 22b are specified using, for example, the parallax, the focal length of the stereo camera, and the distance between the stereo cameras. may be The distance may be, for example, the distance from a straight line connecting the viewpoints 23a and 23b. N meters may also be specified by the user, for example. For example, information indicating N meters may be received by the receiving unit 12, and the first stereo image may be arranged such that points corresponding to positions separated by N meters overlap accordingly in the three-dimensional virtual space. . This placement may be performed by the placement unit 13, for example. In this case, the points corresponding to the positions separated by N meters in the first right-eye image 22a and the first left-eye image 22b are specified using, for example, parallax, focal length, and distance between viewpoints. may be identified by other methods. Also, for example, the distance is associated with the degree of overlap between the first right-eye image 22a and the first left-eye image 22b when arranging the first right-eye image 22a and the first left-eye image 22b so that points corresponding to positions separated by the distance overlap. When the information is stored in the storage unit 11, the degree of overlap between the two images corresponding to N meters is acquired using the information, and the first right-eye image 22a is adjusted so as to achieve the degree of overlap. and the first left-eye image 22b may be arranged so as to overlap each other. Further, in FIG. 10, the positions of the points corresponding to infinity (that is, the center points of the images) are indicated by black circles in each of the first right-eye image 22a and the first left-eye image 22b. Originally, the first left-eye image 22b-2 and the first left-eye image 22b-2 exist on the same plane (that is, on the same straight line in the figure), but for the sake of explanation, they are moved slightly vertically. staggered. These things are assumed to be the same in FIGS. 11 and 12 as well.

When the first stereo images are arranged like the first right-eye image 22a-2 and the first left-eye image 22b-2 shown in FIG. All objects within N meters from the viewpoint, which are included in the images, will appear stereoscopically in the first left-eye image 22b-2 and in front of the first left-eye image 22b-2. Therefore, even when a three-dimensional object attempts to touch an object at a distance of up to N meters contained in the first stereo image, at least a part of the three-dimensional object is located on the far side of the first stereo image. You can avoid invisible situations. Thus, in the three-dimensional virtual space, instead of arranging the first right-eye image 22a and the first left-eye image 22b so that points corresponding to infinity overlap, points corresponding to a predetermined distance The first image for the right eye 22a and the first image for the left eye 22b may be arranged so that they overlap each other.

Also, as shown in FIG. 11, the first right-eye images 22a-2, 22a-3, and 22a-4 are arranged so that the angles of view from the right-eye viewpoint 23a do not change. Note that, as is clear from FIG. 11, the first right-eye image 22a-3 arranged on the far side with respect to the viewpoint is larger than the first right-eye image 22a-2 arranged on the front side. The first right-eye image 22a-4, which is an image and is arranged on the back side, is an image larger than the first right-eye image 22a-3, which is arranged on the front side. In addition, it is preferable that the size of each image is changed so that the aspect ratio is not changed. The same applies to the first left-eye images 22b-2, 22b-3, and 22b-4. Further, in FIG. 11, straight lines 26a and 26b connecting the viewpoints 23a and 23b and points corresponding to infinity in each image are indicated by dashed-dotted lines. The first right-eye image 22a-2 and the first left-eye image 22b-2 overlap at points corresponding to a predetermined distance (for example, N meters), and the first right-eye image 22a-3 and the first left-eye image 22b-2 overlap each other. In the first left-eye image 22b-3, the points corresponding to infinity overlap, and in the first right-eye image 22a-4 and the first left-eye image 22b-4, there is no overlapping point. In this way, a first straight line 26a connecting the right-eye viewpoint 23a and a point corresponding to infinity on the first right-eye image 22a and a line corresponding to the left-eye viewpoint 23b and infinity on the first left-eye image 22b. In the three-dimensional virtual space, when the first right-eye image 22a and the first left-eye image 22b are arranged in the three-dimensional virtual space so that the second straight line 26b connecting the two points intersects , arranging the first right-eye image 22a and the first left-eye image 22b so that the point corresponding to infinity or the point corresponding to the predetermined distance overlaps the same situation as the first It can also be realized by arranging the right-eye image 22a and the first left-eye image 22b. Therefore, the first right-eye image 22a and the first left-eye image 22b may be arranged in the three-dimensional virtual space such that the first straight line 26a and the second straight line 26b intersect. Even in this case, for example, an object that is included in the first stereo image and that is touched or superimposed by a three-dimensional object arranged according to an operation received by the receiving unit 12 is The first right-eye image 22a and the first left-eye image 22b are displayed in three-dimensional virtual space so as to be stereoscopically displayed in front of the first right-eye image 22a and the first left-eye image 22b. is preferably arranged. For example, if the distance to an object that is touched or superimposed by a three-dimensional object is less than N meters, the point corresponding to N meters in the three-dimensional virtual space, or the distance exceeding N meters The first stereo image may be arranged such that the points overlap, or the first and second stereo images may be arranged such that the first right eye image 22a-4 and the first left eye image 22b-4. The first stereo image may be arranged on the far side (that is, on the far side from the viewpoint) of the intersection of the straight lines 26a and 26b.

Next, the effect of moving the first right-eye image 22a and the first left-eye image 22b in the planar direction will be described. In FIG. 12, the first right-eye image 22a-2, the first left-eye image 22b-2, and the first right-eye image obtained by moving them by the same distance in the direction of the right-pointing arrow and the direction of the left-pointing arrow, respectively. 22a-5 and a first left-eye image 22b-5. A point 31a is the intersection of straight lines 26a and 26b connecting points corresponding to infinity in the first right-eye image 22a-2 and the first left-eye image 22b-2 and the viewpoints 23a and 23b. A point 31b is the intersection of straight lines 26a and 26b connecting points corresponding to infinity in the right-eye image 22a-5 and the first left-eye image 22b-5 and the viewpoints 23a and 23b. The points 31a and 31b are the positions at infinity corresponding to the first right-eye image 22a-2 and the first left-eye image 22b-2 respectively arranged in the three-dimensional virtual space, and the first right-eye image 22a -5 and the position of infinity corresponding to the first left-eye image 22b-5. Therefore, when the first stereo images are arranged like the first right-eye image 22a-2 and the first left-eye image 22b-2, the user can see the difference between the viewpoints 23a and 23b and the point 31a. In between, all objects contained in the first stereo image will appear to be present. Further, when the first stereo images are arranged like the first right-eye image 22a-5 and the first left-eye image 22b-5, the user will not be able to see the difference between the viewpoints 23a and 23b and the point 31b. In between, all objects contained in the first stereo image will appear to be present. Therefore, by moving the first right-eye image 22a and the first left-eye image 22b in a direction in which the overlap is reduced or in a direction in which both images are separated, the depth of the display range of the object included in the first stereo image is increased. will be longer. On the other hand, by moving the first right-eye image 22a and the first left-eye image 22b in a direction in which the overlap increases or in a direction in which both images approach each other, the depth of the display range of the object included in the first stereo image is increased. will be shorter. Therefore, for example, when it is desired to superimpose a three-dimensional object to be manipulated on various objects included in the first stereo image, the depth of the display range of the objects included in the first stereo image is shortened. , the first right-eye image 22a and the first left-eye image 22b may be arranged. With such an arrangement, it is possible to reduce the amount of movement of the three-dimensional object when superimposing the three-dimensional object on various objects contained in the first stereo image. For example, the reception unit 12 may receive information from the user that instructs to change the distance from the viewpoints 23a and 23b to the intersection of the straight lines 26a and 26b. Then, the placement unit 13 changes the placement of the first right-eye image 22a and the first left-eye image so that the position of the intersection of the straight lines 26a and 26b is changed according to the received information. may In this way, for example, it is also possible to change the depth of the first stereo image in response to an input from the user. By arranging the first right-eye image 22a and the first left-eye image 22b so that the straight lines 26a and 26b intersect, the sense of depth of the first stereo image can be adjusted in this way.

Next, a case will be described in which the first right-eye image 22a and the first left-eye image 22b are arranged in the three-dimensional virtual space according to the distance of the object included in the first stereo image. In this case, the image synthesizing device 1 may further include a specifying unit 17 as shown in FIG. 9, the image synthesizing apparatus 1 shown in FIG. 13 may further include a determination unit 16, and the output unit 15 may also output the result of determination by the determination unit 16. .

The specifying unit 17 specifies the distance of a predetermined object included in the first stereo image. This distance is calculated using, for example, the parallax between the first right-eye image 22a and the first left-eye image 22b regarding a predetermined object, the focal length of the cameras 21a and 21b, the distance between the cameras 21a and 21b, and the like. may In this way, once the object whose distance is to be measured included in the first stereo image is determined, the distance to that object can be calculated. Note that the calculated distance may be converted into a distance in a three-dimensional virtual space and used in subsequent processing, for example.

Objects whose distances are to be measured may be, for example, all objects included in the first stereo image. In this case, for example, the specifying unit 17 may specify each object included in the first stereo image and specify the distance for each of the specified objects. Object identification may be performed, for example, by contour extraction, image segmentation, or the like. Note that the specifying unit 17 does not have to specify, for example, the background such as the sky or the road included in the first stereo image as an object.

Also, the object whose distance is to be measured may be, for example, an object designated by the user. In this case, the receiving unit 12 may receive information designating the object. Then, the object specified by the information received by the receiving unit 12 may be the object whose distance is to be measured. The information designating the object may be, for example, information designating the position of the object in the image, or information designating the type of object (for example, "hand" or "ball"). The designation of the position of the object may be performed by, for example, designation of a point-like position or designation of a rectangular area. Also, the reception of information specifying the type of object may be, for example, reception from an input device such as a keyboard, or reading of information from a recording medium. It may be reception of the voice recognition result, or reception of other information specifying the type of object. In this case, the reception unit 12 may perform, for example, a plurality of types of reception, that is, reception of operation of the three-dimensional object and reception of information designating the object.

When the information specifying the object is the information specifying the position of the object, the specifying unit 17, for example, specifies the object displayed at the specified position, and specifies the distance to the specified object. Alternatively, a position in the first right-eye image 22a and the first left-eye image 22b corresponding to the specified position may be specified, and the distance related to the specified position may be specified. Further, when the information specifying an object is information specifying the type of object, the specifying unit 17 specifies the type of object in the first right-eye image 22a and the first left-eye image 22b, for example. and determine the distance of the identified object. Examples of the method of identifying the accepted type of object include a method of identifying the position of an object in an image by pattern matching, and a method of identifying the accepted type from the types corresponding to each pixel of the image identified by image segmentation. , object detection to identify areas in the image for objects of the accepted type, and the like.

If the information specifying the object is information specifying the position of the object, the distance may be specified, for example, in the frame in which the position is specified. Further, when the information designating the object is information designating the type of the object, the identification of the object whose distance is to be measured is performed, for example, when the first stereo image is a moving image, for a specific frame. may be performed, or may be performed repeatedly for multiple frames. In the former case, for example, when information designating a predetermined object is received by the receiving unit 12 during output of a moving image that is the second stereo image, the object is specified for the frame at that time. may In the latter case, that is, when the object is specified for a plurality of frames, the object may be specified for each frame at a predetermined interval, for example. Note that when the first stereo image is accepted in real time and the second stereo image is generated in real time using the accepted first stereo image, for example, the identification of the object is also performed in real time. may On the other hand, when the first stereo image is stored in the storage unit 11 in advance, for example, the object may be specified in advance before the second stereo image is generated.

In the three-dimensional virtual space, the first right-eye image 22a and the first left-eye image 22b may be arranged such that points corresponding to distances exceeding the longest distance specified by the specifying unit 17 overlap. By doing so, the object specified by the specifying unit 17 is displayed in front of the first stereo image in the three-dimensional virtual space. be able to In addition, when the distances of a plurality of objects are specified, it is preferable that the first stereo image is arranged so that the point corresponding to the distance exceeding the longest distance among the plurality of distances overlaps. Further, when the object distance is specified for a plurality of frames, for example, the first stereo image is arranged so that the point corresponding to the distance exceeding the longest distance among the specified distances overlaps. is preferred. Note that the distance exceeding the specified longest distance may be, for example, a distance obtained by adding a predetermined distance to the specified longest distance. The predetermined distance to be added is, for example, a length equal to or greater than a positional deviation between the objects included in the first stereo image that may occur when performing an operation to superimpose the three-dimensional object on the object. may By doing so, it is possible to prevent the three-dimensional object from being located on the back side of the first stereo image even if such positional deviation occurs. Note that such placement of the first stereo images may be performed by the placement unit 13, for example.

Note that the three-dimensional object may be prevented from being positioned behind the first right-eye image 22a and the first left-eye image 22b without using the specifying unit 17. FIG. In this case, when the first stereo image is arranged in the three-dimensional virtual space so that the points corresponding to the predetermined distance overlap, the distance to the three-dimensional object to be operated is the predetermined distance. (that is, when the three-dimensional object approaches the first stereo image), the placement unit 13 may increase the predetermined distance and place the first stereo image. For example, when the distance obtained by adding a predetermined distance to the distance from the viewpoints 23a and 23b to the three-dimensional object becomes the predetermined distance, the placement unit 13 sets the distance longer than the predetermined distance. The first stereo image may be rearranged in the three-dimensional virtual space such that corresponding points overlap. A distance longer than the predetermined distance may be, for example, a distance obtained by adding a predetermined distance to the predetermined distance. Note that the placement unit 13 may rearrange the first stereo image in real time, for example, while the second stereo image is being generated or output. Therefore, the rearrangement of the first stereo image may be performed repeatedly according to the manipulation of the three-dimensional object, for example.

As a method of rearranging the first stereo image, for example, as shown in FIG. 12, the first right-eye image 22a and the first left-eye image 22b are moved to the plane of the image without changing the distance from the viewpoints 23a and 23b. There is a method of moving in a direction. In the latter case, by changing the arrangement of the first stereo image, the sense of depth of the objects included in the first stereo image changes, and accordingly the objects are arranged in the three-dimensional virtual space. The composite position of the three-dimensional object will also change. Therefore, from the viewpoint of not causing these changes, the first stereo image is rearranged by a method of moving the arrangement position of the first stereo image to the back side without changing the angle of view as shown in FIG. is preferred. It should be noted that moving the first stereo image by a shorter distance can prevent the three-dimensional object from being positioned behind the first stereo image by moving the first stereo image so that the straight lines 26a and 26b intersect. This is because stereo images are arranged. Therefore, in this case as well, it is preferable to rearrange the first stereo image so that the straight lines 26a and 26b intersect.

When rearranging the first stereo image to prevent the three-dimensional object from being positioned on the back side of the first stereo image, for example, the arrangement unit 13 rearranges the first image for the right eye. 22a and the first left-eye image 22b are arranged in a three-dimensional virtual space so that points corresponding to a predetermined distance overlap, the predetermined distance is lengthened according to the position of the three-dimensional object. Even if the position of the three-dimensional object approaches the viewpoints 23a and 23b, the first stereo image is not rearranged to shorten the predetermined distance. may

Alternatively, the first stereo image may be arranged in the three-dimensional virtual space so that the first straight line 26a and the second straight line 26b intersect, or points corresponding to a predetermined distance may overlap in the three-dimensional virtual space. arranging the stereo images so that the predetermined distance exceeds the longest distance specified by the specifying unit 17, or when the three-dimensional object approaches the first stereo image, the first Needless to say, the rearrangement of one stereo image may be performed, for example, on the first stereo image taken by a pair of cameras arranged so that their optical axes intersect.

Further, in the above embodiments, each process or function may be implemented by centralized processing by a single device or single system, or may be implemented by distributed processing by multiple devices or multiple systems. It may be realized by

Further, in the above-described embodiment, when the information is passed between the components, for example, when the two components that exchange the information are physically different, one of the components output of information and reception of information by the other component, or one component if the two components that pass the information are physically the same from the phase of processing corresponding to the other component to the phase of processing corresponding to the other component.

In the above embodiments, information related to processing executed by each component, for example, information received, acquired, selected, generated, transmitted, or received by each component Also, information such as thresholds, formulas, addresses, etc. used by each component in processing may be stored temporarily or for a long period of time in a recording medium (not shown), even if not specified in the above description. Further, each component or an accumulation section (not shown) may accumulate information in the recording medium (not shown). Further, each component or a reading unit (not shown) may read information from the recording medium (not shown).

Further, in the above embodiment, if the information used in each component etc., for example, information such as thresholds, addresses and various set values used in processing by each component may be changed by the user, the above The user may or may not be able to change such information as appropriate, even if not explicitly stated in the description. If the information can be changed by the user, the change is realized by, for example, a reception unit (not shown) that receives a change instruction from the user and a change unit (not shown) that changes the information according to the change instruction. may The reception of the change instruction by the reception unit (not shown) may be, for example, reception from an input device, reception of information transmitted via a communication line, or reception of information read from a predetermined recording medium. .

Further, in the above embodiment, when two or more components included in the image synthesizing apparatus 1 have communication devices, input devices, etc., the two or more components may physically have a single device. or may have separate devices.

Further, in the above embodiments, each component may be configured by dedicated hardware, or components that can be realized by software may be realized by executing a program. For example, each component can be realized by reading and executing a software program recorded in a recording medium such as a hard disk or a semiconductor memory by a program execution unit such as a CPU. During the execution, the program execution unit may execute the program while accessing the storage unit or recording medium. The software that implements the image synthesizing device 1 in the above embodiment is the following program. In other words, this program causes a computer capable of accessing a storage unit in which a first stereo image having a first right-eye image and a first left-eye image is stored to a three-dimensional object arranged in a three-dimensional virtual space. a first straight line connecting the right-eye viewpoint and a point corresponding to infinity on the first right-eye image, and a point corresponding to infinity on the left-eye viewpoint and the first left-eye image A three-dimensional virtual space in which the first right-eye image and the first left-eye image are arranged so that a second straight line connecting an arrangement unit for arranging a 3D object according to the operation received by the reception unit, a 3D virtual space in which the 3D object is arranged (excluding the 3D virtual space in which the image and the first image for the left eye are arranged); In space, a second right-eye image, which is an image from the right-eye viewpoint when the first right-eye image is displayed, and an image from the left-eye viewpoint when the first left-eye image is displayed. A program for functioning as a generation unit that generates a second stereo image including a certain second left-eye image and an output unit that outputs the second stereo image.

Further, the software that realizes the image synthesizing device 1 in the above embodiment may be the following program. This program manipulates a three-dimensional object placed in a three-dimensional virtual space by a computer having access to a storage unit in which a first stereo image having a first right-eye image and a first left-eye image is stored. a receiving unit for receiving; a placing unit for placing a three-dimensional object according to an operation received by the receiving unit in a three-dimensional virtual space in which the first right-eye image and the first left-eye image are placed; In the three-dimensional virtual space in which the first right-eye image is displayed and the second right-eye image, which is an image from the right-eye viewpoint, and the first left-eye image are displayed A program for functioning as a generation unit that generates a second stereo image including a second left eye image that is an image from a left eye viewpoint, and an output unit that outputs the second stereo image.

In the program, the functions realized by the program do not include functions that can be realized only by hardware. For example, functions that can be realized only by hardware such as a modem or an interface card, such as a reception unit that receives information and an output unit that outputs information, are not included in the functions realized by the program.

Also, this program may be executed by being downloaded from a server or the like, and the program recorded on a predetermined recording medium (for example, an optical disk such as a CD-ROM, a magnetic disk, a semiconductor memory, etc.) may be read. may be performed by Also, this program may be used as a program constituting a program product.

Also, the number of computers executing this program may be singular or plural. That is, centralized processing may be performed, or distributed processing may be performed.

FIG. 14 is a schematic diagram showing an example of the appearance of a computer that implements the image synthesizing device 1 according to the above embodiment by executing the above program. The above embodiments can be implemented by computer hardware and computer programs executed thereon.

In FIG. 14, computer system 900 comprises computer 901 including CD-ROM drive 905 , keyboard 902 , mouse 903 and monitor 904 .

FIG. 15 is a diagram showing the internal configuration of the computer system 900. As shown in FIG. In FIG. 15, a computer 901 includes a CD-ROM drive 905, an MPU (Micro Processing Unit) 911, a ROM 912 for storing programs such as a boot-up program, and a ROM 912 for storing programs such as a boot-up program. A RAM 913 for temporarily storing and providing a temporary storage space, a hard disk 914 for storing application programs, system programs and data, and a bus 915 for interconnecting the MPU 911, ROM 912 and the like are provided. Note that the computer 901 may include a network card (not shown) that provides connection to a LAN, WAN, or the like.

A program that causes computer system 900 to execute the functions of image synthesizing apparatus 1 according to the above embodiment may be stored in CD-ROM 921 , inserted into CD-ROM drive 905 , and transferred to hard disk 914 . Alternatively, the program may be transmitted to computer 901 via a network (not shown) and stored in hard disk 914 . Programs are loaded into RAM 913 during execution. The program may be loaded directly from the CD-ROM 921 or network. Also, the program may be read into the computer system 900 via another recording medium (eg, DVD, etc.) instead of the CD-ROM 921 .

The program does not necessarily include an operating system (OS) or a third party program that causes the computer 901 to execute the functions of the image synthesizing apparatus 1 according to the above embodiment. A program may contain only those portions of instructions that call the appropriate functions or modules in a controlled manner to produce the desired result. How the computer system 900 operates is well known and will not be described in detail.

Moreover, it goes without saying that the present invention is not limited to the above-described embodiments, and that various modifications are possible and are also included within the scope of the present invention.

REFERENCE SIGNS LIST 1 image synthesizing device 11 storage unit 12 reception unit 13 placement unit 14 generation unit 15 output unit 16 determination unit 17 identification unit

Claims (12)

a storage unit storing a first stereo image having a first right-eye image and a first left-eye image;
a reception unit that receives an operation of a three-dimensional object placed in a three-dimensional virtual space;
A first straight line connecting the right-eye viewpoint and a point corresponding to infinity on the first right-eye image, and a second straight line connecting the left-eye viewpoint and a point corresponding to infinity on the first left-eye image. A three-dimensional virtual space in which the first right-eye image and the first left-eye image are arranged so that the straight lines of an arrangement unit that arranges a three-dimensional object according to the operation received by the reception unit (excluding the three-dimensional virtual space in which the first left-eye image is arranged);
In the three-dimensional virtual space in which the three-dimensional object is arranged, a second right-eye image and the first left-eye image, which are images from a right-eye viewpoint in a state where the first right-eye image is displayed. a generation unit that generates a second stereo image including a second left-eye image that is an image from a left-eye viewpoint in a state where is displayed;
and an output unit that outputs the second stereo image. 2. The image synthesizing device according to claim 1, wherein said first right-eye image and said first left-eye image are arranged such that points corresponding to a predetermined distance overlap in said three-dimensional virtual space. further comprising a specifying unit that specifies the distance of a predetermined object included in the first stereo image,
3. The image synthesizing device according to claim 2, wherein said predetermined distance exceeds the longest distance specified by said specifying unit. 4. The image synthesizing device according to claim 3, wherein said predetermined objects are all objects included in said first stereo image. The receiving unit also receives information designating the predetermined object,
4. The image synthesizing device according to claim 3, wherein said predetermined object is an object designated by information received by said receiving unit. 3. The image synthesizing device according to claim 2, wherein when the distance to said three-dimensional object approaches said predetermined distance, said placement unit increases said predetermined distance and places said first stereo image. 7. The image synthesizing apparatus according to claim 1, wherein said first stereo image is captured by a pair of cameras arranged so that their optical axes are parallel. 7. The image synthesizing device according to claim 1, wherein said first stereo image is captured by a pair of cameras arranged so that their optical axes intersect. 9. The image synthesizing device according to claim 1, wherein said first stereo image and said second stereo image are moving images. specifying a position in the three-dimensional virtual space of an object included in the first stereo image from the first right-eye image and the first left-eye image arranged in the three-dimensional virtual space; and a determination unit that determines the relationship between the position of the three-dimensional object and the position of the three-dimensional object,
10. The image synthesizing device according to claim 1, wherein said output unit also outputs a result of determination by said determination unit. receiving an operation of a three-dimensional object placed in a three-dimensional virtual space;
A first right-eye image and a first left-eye image included in a first stereo image are a first straight line connecting a right-eye viewpoint and a point corresponding to infinity on the first right-eye image, and a left eye. A three-dimensional virtual space arranged so that a second straight line connecting the viewpoint and a point corresponding to infinity in the first image for left eye intersects arranging the 3D object according to the accepted operation in a 3D virtual space (excluding the 3D virtual space in which the first right-eye image and the first left-eye image are arranged);
In the three-dimensional virtual space in which the three-dimensional object is arranged, a second right-eye image and the first left-eye image, which are images from a right-eye viewpoint in a state where the first right-eye image is displayed. generating a second stereo image including a second left-eye image that is an image from a left-eye viewpoint when is displayed;
and outputting the second stereo image. a computer accessible to storage in which a first stereo image having a first right eye image and a first left eye image is stored;
a reception unit that receives an operation of a three-dimensional object placed in a three-dimensional virtual space;
A first straight line connecting the right-eye viewpoint and a point corresponding to infinity on the first right-eye image, and a second straight line connecting the left-eye viewpoint and a point corresponding to infinity on the first left-eye image. A three-dimensional virtual space in which the first right-eye image and the first left-eye image are arranged so that the straight lines of an arrangement unit that arranges a three-dimensional object according to the operation received by the reception unit (excluding the three-dimensional virtual space in which the first left-eye image is arranged);
In the three-dimensional virtual space in which the three-dimensional object is arranged, a second right-eye image and the first left-eye image, which are images from a right-eye viewpoint in a state where the first right-eye image is displayed. A generation unit that generates a second stereo image including a second left-eye image that is an image from a left-eye viewpoint in a state where is displayed;
A program for functioning as an output unit that outputs the second stereo image.

Images