JP2005165614A - Device and method for synthesizing picture - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a stereoscopic photograph print system, etc., for allowing a user to easily synthesize without a uncomfortable feeling an actually photographed stereoscopic picture with a virtual space picture arranged in a virtual space and to obtain a stereoscopic photograph with high definition. <P>SOLUTION: A picture synthesizing device comprises: a virtual space organizing means for determining the movable range of a three-dimensional (3D) model, based on information of depth to a subject in an actually photographed picture; an operating means for moving, rotating, magnifying, and reducing, etc., the 3D model within the determined range; a multi-view-point picture sequence generating means for synthesizing the virtual space picture generated from the 3D model with the actually photographed picture at a position determined by the operating means, etc.; and a 3D picture integrating means for integrating the synthesized multi-view-point sequence picture with a 3D picture. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

  The present invention relates to an image composition apparatus and an image composition method used for stereoscopic image formation.

  Conventionally, it is possible to prepare a subject for photographing, and it is possible to easily obtain a high-quality stereoscopic image, and a good subject stereoscopic image can be observed by superimposing an optical member such as a lenticular plate on the stereoscopic image. A stereoscopic photo print system that can be used is disclosed in Japanese Patent Laid-Open No. 2001-346226. In this proposal, a stereo image at a plurality of viewpoint positions is input from a camera equipped with a stereo adapter, and a parallax map extraction unit that extracts a parallax map indicating a depth distribution of a subject from the stereo image, the stereo image and the parallax map And multi-view image sequence generating means for generating a multi-view image sequence of a subject from a plurality of viewpoints, and 3D image synthesizing means for synthesizing a 3D image based on the multi-view image sequence. Since the apparatus can automatically generate a multi-viewpoint image from a captured stereo image, a high-quality stereoscopic image can be obtained. The outline of the proposal will be described below.

  FIG. 9 is a block diagram illustrating a configuration of a stereoscopic photo print system described in JP-A-2001-346226. Reference numerals 901, 902, and 903 denote a subject, a stereo adapter, and a camera, respectively. An example of a camera 903 equipped with a stereo adapter 902 is shown in FIG. In FIG. 9, reference numerals 901, 902, and 903 denote the same subject, stereo adapter, and camera as in FIG. Reference numeral 1002 denotes a camera lens, 1003 denotes a photographing surface, 1004 denotes a prism, and 1005 and 1006 denote mirrors. Further, o is the lens center of the photographic lens 1002, l is the optical axis of the photographic lens 1002, and m and n are chief rays of light beams passing through the centers of the left-eye screen and the right-eye screen on the photographic surface 1003, respectively. As shown in the figure, it is symmetrical with respect to the optical axis l of the photographing lens.

  Next, the flow of processing described in JP-A-2001-346226 will be described with reference to FIG.

  First, in step S1101, in order to make a stereo image data that can be handled by a processing program, the stereo image is taken into a memory of a PC that is an image processing apparatus. At this time, a stereo image file is instructed by an input device such as a keyboard or a mouse (not shown), and the designated file is read into the program.

  Next, in step S1102, the trapezoidal distortion that has occurred during the shooting of the stereo image is corrected. First, the acquired image is divided into stereo image left and right image data. That is, when the image of each RGB channel of the stereo image data is a two-dimensional array of width and height of M × N pixels, the width and height are M / 2 × N with a vertical line passing through the center of the image as a boundary. Divide into image data. Then, according to the shooting parameters of the camera and adapter, trapezoidal distortion correction is performed in the opposite direction by the same angle on the left and right so that the virtual imaging planes after trapezoidal distortion correction are parallel to each other around the screen center of the left and right image data. I do. The trapezoidal distortion correction process is a well-known geometric transformation based on a three-dimensional rotation matrix of an image.

  Next, in step S1103, a pair of left and right stereo images of a predetermined size is acquired from the corrected left and right image data. First, the amount of positional deviation between the left and right image data is obtained. Although it is desirable that the same part of the subject in the left and right image data is captured, the amount of horizontal displacement differs depending on the distance between the camera and the subject at the time of capture and the distribution in the depth direction, so trapezoidal distortion correction This is because it is necessary to adjust the image position in each piece of image data. In addition, since the trapezoidal distortion correction processing in step S1102 is performed according to the shooting parameters of a predetermined camera and adapter, the state of completely parallel viewing is affected by the influence of the camera and adapter mounting state and the displacement of each component of the adapter. Since the right and left images may not be obtained, adjustment in the vertical direction is also necessary.

  In step S1104, corresponding points representing the same subject portion as the point-to-point correspondence between the acquired stereo image pairs are extracted. Corresponding points are extracted by applying template matching to each point.

  Next, in step S1105, a parallax map is extracted from the corresponding points. Here, the parallax map represents a horizontal position shift (parallax) with respect to the corresponding point position of the right image for each pixel of the left image with the left image of the stereo image pair as a reference image. Although there are pixels that can be obtained directly from the difference in the horizontal position by the corresponding point extraction, the parallax is also obtained by interpolation for uncorresponding points and points that were not originally searched for corresponding points. Thus, a parallax map is obtained when the parallax is obtained for all pixels.

  In step S1107, a multi-viewpoint image sequence is generated using the parallax map. The image generated here is a multi-viewpoint image constituting a three-dimensional stripe image, and the size of each image depends on the number of images and the resolution and size of printing.

  The image to be generated is generated by deforming using the left image and the parallax map. The viewpoint position of the image to be generated is determined so that a predetermined amount of parallax is generated. Further, the viewpoint positions are determined so that the viewpoints are arranged symmetrically at equal intervals around the viewpoint position of the left image. This is to minimize the amount of deformation of the image by observing a stable stereoscopic image by arranging the viewpoint positions of the image sequence at equal intervals and arranging the viewpoint positions symmetrically around the viewpoint position of the left image, This is to stably obtain a high-quality stereoscopic image even if an error occurs in the parallax map depending on the subject and shooting conditions.

  From the parallax map obtained in step S1106, the parallax corresponding to the subject position closest to the shooting position and the parallax corresponding to the subject position farthest from the shooting position are obtained. Then, the viewpoint position is determined so that the closest subject is observed at a predetermined distance from the printing surface at the time of observation, and the farthest subject is observed at a predetermined distance from the printing surface.

  Next, a method for generating each viewpoint image will be described.

First, a new viewpoint image is generated by forward mapping using pixels of the left image. That is, first, the position (x _N , y _N ) in the new viewpoint image that maps each pixel of the left image is the parallax d at the pixel position (x, y) of the left image, the ratio r representing the viewpoint position, the left image And the new viewpoint image size are obtained from Equation (1).

_{x N = H / h × (} x + r × (d-sh)), y N = y ---- (1)
Then, the pixel at the pixel position (x, y) of the left image is copied to the position (x _N , y _N ) of the new viewpoint image. This process is repeated for all pixels of the left image. Next, among the pixels of the new viewpoint image, the hole filling process is performed on the pixels to which no pixel is assigned from the left image. In this hole filling process, an effective pixel that is a predetermined distance away from the pixel to be obtained is searched, and a weighted average value is assigned using the distance of the pixel value as a parameter. If there is no effective pixel, the search is repeated with the search range expanded.

  Through the above processing, a new viewpoint image in which all pixels are valid is generated. This process is repeated for the number of viewpoints to form a multi-viewpoint image sequence.

Next, in step S1108, a three-dimensional stripe image is synthesized from the multi-viewpoint image sequence. At this time, the three-dimensional image is synthesized so that the same coordinate pixel of each image of the multi-viewpoint image sequence is arranged as an adjacent pixel according to the viewpoint arrangement of the image. When the pixel value at the j-th viewpoint is P _jmn (where m and n are the indices of the pixel array in the horizontal and vertical directions, respectively), the j-th image data is represented as the following two-dimensional array.

P _j00 P _j10 P _j20 ...
P _j01 P _j11 P _j21 ...
P _j02 P _j12Pj22 ...
In the synthesis, the images of the respective viewpoints are decomposed into strips for each line in the vertical direction, and are synthesized by the number of viewpoints in the reverse order of the viewpoint positions. Therefore, the combined image is a stripe image as shown below.

P _N00 ·· P ₂₀₀ P ₁₀₀ P _N10 · · P ₂₁₀ P ₁₁₀ P _N20 · · P ₂₂₀ P ₁₂₀ · ·
P _N01 ·· P ₂₀₁ P ₁₀₁ P _N11 · · P ₂₁₁ P ₁₁₁ P _N21 · · P ₂₂₁ P ₁₂₁ · · ·
P _N02 ·· P ₂₀₂ P ₁₀₂ P _N12 · · P ₂₁₂ P ₁₁₂ P _N22 · · P ₂₂₂ P ₁₂₂ · ·
However, viewpoint 1 represents the left end image and N represents the right end image. Here, the order of arrangement of the viewpoint positions is reversed because, when observing with the lenticular plate, the image is observed in the left and right direction within one pitch of the lenticular. This three-dimensional stripe image has a size of X (= N × H) × v when the original multi-viewpoint image is an N viewpoint image having a size of H × v.

  Next, the pitch is aligned with the lenticular plate for this three-dimensional stripe image. Since there are N RPdpi pixels for one pitch, the pitch is 1 pitch N / RPinch. However, since the pitch of the lenticular plate is RLinch, the pitch is adjusted by multiplying the image by RL × RP / N in the horizontal direction. In addition, since the number of pixels in the vertical direction needs to be (RL × RP / N) × Y pixels at that time, the magnification is set to (RL × RP × Y) / (N × v) times in the vertical direction. . Accordingly, the scaling process in the horizontal and vertical directions as described above is performed on the three-dimensional stripe image to obtain image data for printing.

  In step S1109, printing is performed on the output image in step S1109.

  As described above, the processing in steps S1101 to S1109 makes it possible to observe a good stereoscopic image by superimposing the lenticular plate on the printed image.

  On the other hand, with the advancement of computer graphics (hereinafter abbreviated as CG) technology, virtual space images are generated from multiple virtual viewpoint positions of a three-dimensional model composed of geometric information and surface attribute information arranged in the virtual space. As a technique for generating and printing a three-dimensional image from the plurality of virtual space images, there is a technique disclosed in US Pat. No. 5,643,231. This technology is a method of generating a 3D photograph using only virtual images of a plurality of viewpoint positions generated on a computer without using an actual captured image, rather than generating a 3D photograph using a captured image. It is said that it is possible to obtain a stable stereoscopic photograph by using CG.

  As described above, a system for obtaining a stereoscopic photograph from each of a real photographed image or a virtual space image generated and expressed by CG has been proposed. However, due to recent technological progress, devices that easily acquire digital images, such as digital cameras, scanners, and digital videos, are generally used. Regarding CG, general personal computers and games are also used. A high-quality virtual space image can be generated and displayed on a device. However, the types of 3D models that can be generated in the virtual space are limited, and it is virtually impossible to make a 3D model so that all existing objects can be handled by CG. From such a background, there was a demand to generate a more realistic and entertaining image by simply synthesizing a virtual space image with a three-dimensional real image, but the user can easily and high-quality. An image synthesizing apparatus capable of synthesizing has not been realized yet.

  In view of the above problems, the present invention fuses a stereoscopic image actually captured and a virtual space image generated from a three-dimensional model arranged in the virtual space easily and three-dimensionally without any sense of incongruity. It is an object of the present invention to provide a stereoscopic photo print system that can obtain a high-quality stereoscopic photo.

  In order to achieve the above object, in the image composition device according to claim 1, virtual space construction means for determining a movable range of a virtual space and a three-dimensional model from depth information to a subject, and a three-dimensional model as the virtual A three-dimensional model moving means for moving within a range determined by the space construction means, and a tertiary for performing at least one of movement, rotation, enlargement and reduction within the range determined by the virtual space construction means for the three-dimensional model A multi-viewpoint image that generates a composite multi-viewpoint image sequence by synthesizing a virtual space image generated from a three-dimensional model arranged in a virtual space determined by the three-dimensional model operation means and a live-action image. Sequence generation means and three-dimensional image integration means for integrating the combined multi-view image generated by the multi-view image sequence generation means into a three-dimensional image Characterized in that it has a.

  According to a second aspect of the present invention, in the image synthesizing apparatus according to the first aspect, the multi-viewpoint image sequence generating unit generates the real-shot multi-viewpoint image and the virtual space multi-viewpoint image, and then synthesizes them. It is characterized by doing.

  According to a third aspect of the present invention, in the image synthesizing apparatus according to the first aspect, when generating the virtual space multi-viewpoint image, the depth information is referred to, and a pixel from the depth information is three-dimensional. If it is determined whether or not the model is drawn, and if it is determined that the 3D model is being drawn, it is determined that the 3D model is not drawn using the same pixel information as that pixel of the virtual space multi-viewpoint image. In this case, a synthesized image is generated using the same pixel information as that of the pixel of the live-action multi-viewpoint image.

  According to a fourth aspect of the present invention, in the image synthesizing apparatus according to the first or third aspect, the multi-view image sequence generating means generates depth information when generating the virtual space multi-view image. , It is determined whether a certain pixel is drawing a three-dimensional model from the depth information, and if it is determined that a three-dimensional model is being drawn, the same pixel information as that pixel of the virtual space multi-viewpoint image is obtained. If it is determined that the 3D model is not drawn, the same pixel information as the pixel is generated from the stereoscopic image information, and a composite image is generated.

  According to a fifth aspect of the present invention, in the image synthesizing apparatus according to the first, second, third, or fourth aspect, the multi-view image sequence generating means includes a real-shot multi-view image and a multi-view image. The virtual space multi-view image sequence is characterized by generating and synthesizing multi-view images in the same shooting viewpoint order.

  According to a sixth aspect of the present invention, there is provided the image synthesizing apparatus according to the first aspect, wherein the three-dimensional model moving means contacts or exceeds the moving range of the three-dimensional model determined by the virtual space construction means. In such a case, the operator is notified to that effect.

  In the image composition device according to the seventh aspect of the present invention, the virtual space image generation parameter determining means for generating a virtual space image from the stereoscopic image observation parameters, and the parameters determined by the virtual space image generation parameter determining means are used. A virtual space multi-view image sequence generating unit that generates a virtual space multi-view image sequence based on the virtual space multi-view image sequence generating unit, and the virtual space multi-view image sequence and the multi-view image sequence generated by the virtual space multi-view image sequence generating unit The image synthesizing unit includes an image synthesizing unit, and a three-dimensional image integrating unit that integrates the synthesized image synthesized by the actual virtual space image synthesizing unit as a three-dimensional image.

  An image composition device according to an eighth aspect of the present invention is the image composition device according to the seventh aspect, wherein the virtual space image generation parameter determination means arranges the three-dimensional model of the virtual space in front of the real space. It is characterized by.

  The image composition device according to the invention described in claim 9 is the image composition device according to claim 7, wherein the virtual space image generation parameter determination means uses the nearest distance and the farthest distance of the three-dimensional model arranged in the virtual space. A parameter is determined as a virtual space area for reproducing the image.

  In the image processing method according to the tenth aspect of the present invention, a virtual space construction step for determining a movable range of the virtual space and the three-dimensional model from the depth information to the subject, and a three-dimensional model by the virtual space construction step. A three-dimensional model moving step for moving within the determined range, and a three-dimensional model operation step for performing at least one of movement, rotation, enlargement, and reduction within the range determined by the virtual space construction step for the three-dimensional model And a multi-viewpoint image sequence generation step of generating a composite multi-viewpoint image sequence by combining the virtual space image generated from the three-dimensional model arranged in the virtual space determined in the three-dimensional model operation step and the real image. A three-dimensional image integration step of integrating the combined multi-view image sequence generated in the multi-view image sequence generation step into a three-dimensional image. The features.

  An image processing method according to an eleventh aspect of the present invention is the image processing method according to the tenth aspect, wherein the multi-view image sequence generation step generates a real-view multi-view image and a virtual space multi-view image, respectively, and then synthesizes them. It is characterized by doing.

  The image processing method according to a twelfth aspect of the present invention is the image processing method according to the tenth aspect, wherein the multi-view image sequence generation step refers to depth information to be generated when generating a virtual space multi-view image. Then, it is determined from the depth information whether or not the pixel is a 3D model, and if it is a 3D model, a virtual space multi-viewpoint image is used. It is characterized by doing.

  The image processing method according to claim 13 is the image processing method according to claim 10 or 12, wherein the multi-view image sequence generation step generates a depth when generating a virtual space multi-view image. Refer to the information to determine whether the depth information at the same coordinates as the pixel to be calculated is a three-dimensional model. If the pixel is not a three-dimensional model, use a virtual space multi-viewpoint image. The pixel is generated from stereoscopic image information to generate a composite image.

  In an image processing method according to a fourteenth aspect of the present invention, in the image processing method according to the tenth, eleventh, twelfth, or thirteenth aspect, the multi-view image sequence generation step includes: The virtual space multi-view image sequence is characterized by generating and synthesizing multi-view images in the same shooting viewpoint order.

  The image processing method according to claim 15 is the image processing method according to claim 10, wherein the three-dimensional model moving step touches or exceeds the moving range of the three-dimensional model determined in the virtual space construction step. In such a case, the operator is notified to that effect.

  In the image processing method according to the sixteenth aspect of the present invention, a virtual space image generation parameter determination step for generating a virtual space image from a stereoscopic image observation parameter, and a parameter determined by the virtual space image generation parameter determination step are used. A virtual space multi-view image sequence generation step for generating a virtual space multi-view image sequence based on the virtual space multi-view image sequence generation step, and the virtual space multi-view image sequence and the multi-view image sequence generated in the virtual space multi-view image sequence generation step It has an image composition process, and a 3D image integration process for integrating the composite image synthesized by the live-action virtual space image composition process as a 3D image.

  The image processing method according to claim 17 is the image processing method according to claim 16, wherein in the virtual space image generation parameter determination step, a three-dimensional model of the virtual space is arranged on the near side of the real space. It is characterized by that.

  The image processing method according to an eighteenth aspect of the present invention is the image processing method according to the sixteenth aspect, wherein in the virtual space image generation parameter determination step, the nearest distance and the farthest distance of the three-dimensional model arranged in the virtual space. A parameter is determined as a virtual space region for reproducing the distance.

  As described in detail above, according to the image synthesizing apparatus according to the first to sixth aspects of the present invention, the real image stereoscopic image and the virtual image generated from the three-dimensional model configured on the virtual space can be displayed in a three-dimensional manner without any sense of incongruity. It becomes possible to synthesize an easy three-dimensional image.

  According to the image synthesizing apparatus of the seventh to ninth aspects, it is possible to easily synthesize a three-dimensional image without a sense of incongruity three-dimensionally even when a virtual space matched with a real image is not constructed.

  According to the image synthesizing method according to any one of claims 10 to 15, a three-dimensional image that is three-dimensionally comfortable and easy to sense a stereoscopic image and a virtual image generated from a three-dimensional model configured on a virtual space. It is possible to synthesize.

  According to the image synthesizing apparatus of the sixteenth to eighteenth aspects, it is possible to easily synthesize a three-dimensional image without a sense of incongruity three-dimensionally even when a virtual space matched with a photographed image is not constructed.

(First embodiment)
Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

  FIG. 1 is a block diagram showing a configuration of an image composition system according to an embodiment of the present invention. A stereoscopic image input device 100 is a device for inputting stereo images taken at a plurality of viewpoint positions. For example, the stereo image input device may be configured by mounting a stereo adapter on a camera described in JP-A-2001-346226. If the subject does not move, the left and right images may be output by attaching the camera to a pan that slides to the left and right. FIG. 2 shows an outline in the case of acquiring left and right two-line images. 201 is a subject, L202 and R203 are center points of left and right viewpoints separated by a base line length b, respectively, and are schematically shown in front of the camera as opposed to a focal plane that originally forms image planes IL and IR separated by a focal length f. It represents. Reference numeral 103 denotes an image composition device, which combines a stereo image taken from a plurality of viewpoint positions from the stereoscopic image input device 100 and a virtual space image generated from a three-dimensional model arranged in the virtual space to obtain a tertiary. An apparatus for generating an original integrated image, for example, a general-purpose personal computer. A two-dimensional display device 101 includes a CRT display that displays an image input from the stereoscopic image input device 100 and an image processed or combined by the image combining device 103. Reference numeral 105 denotes an operation input device, which is a pointing device that allows an operator to instruct an operation command to the image composition device 103, and to rotate, move, enlarge, or reduce the 3D model displayed on the 2D display device 101. Consists of mouse and joystick. A printing apparatus 104 is configured to print image data and the like generated and displayed by the image composition apparatus 103. Note that the stereoscopic image input device 100, the printing device 104, and the operation input device 105 are connected to the image composition device by using an interface such as a USB (Universal Serial Bus).

  Next, the configuration of an internal block diagram in the image composition apparatus 103 will be described.

  Reference numeral 1031 denotes a circuit that generates a parallax map or the like from the stereo image input from the stereoscopic image input apparatus 100. Reference numeral 1032 denotes a virtual space image generation unit, an image generated from a three-dimensional model arranged in the virtual space, a reference image input from the stereoscopic image input device 100 and set by a stereoscopic image information generation unit 1031 described later, and And a circuit for displaying the composite image on the two-dimensional display device 101. Reference numeral 1033 denotes a three-dimensional model storage unit which includes an auxiliary storage device that stores geometric shape information and surface attribute information or surface texture image information. Reference numeral 1035 denotes a virtual space construction, which includes a circuit that constructs a virtual space using parameters such as a parallax map from the stereoscopic image information generation unit 1031. Reference numeral 1036 denotes a three-dimensional model operation unit, and information input from the operation input unit 105 that inputs an operation on the three-dimensional model from the operator is used to move, rotate, enlarge, reduce, and the like to the three-dimensional model in the actual virtual space. It is composed of a circuit that converts it into an operation amount and presents it to the operator when a predetermined operation range is exceeded. A multi-view image sequence generation unit 1034 generates a real-shot multi-view image sequence using a stereo image input from the stereoscopic image input apparatus 100 as will be described later, and also generates a virtual space multi-view image sequence. Consists of. Reference numeral 1037 denotes a three-dimensional image synthesis unit, which includes a circuit that synthesizes a real-shot multi-view image sequence generated by the multi-view image sequence generation unit and a virtual space multi-view image sequence as a three-dimensional image.

  Next, the flow of the processing program in the image composition apparatus 103 in this embodiment will be described in detail with reference to FIG.

  S301 is a stereoscopic image information generation step, in which a stereo image input by the stereoscopic image input apparatus 100 and camera parameters at the time of shooting (baseline length, focal length, etc. between viewpoints) are input, and a parallax map representing a depth distribution to the subject Is generated. At this time, the left image of the stereo images is determined as the reference image. There is no problem even if the right image is used as the reference image, but at this time, the reference image for calculating the parallax of the parallax map needs to be recalculated as the right image. The stereoscopic image information generation step S301 is configured by the method described in JP-A-2001-346226.

S302 is a virtual space construction step, in which a virtual space is constructed from the parallax map generated in the stereoscopic image information generation step S301 and the camera parameters from the stereo camera. An example of the constructed virtual space is shown in FIG. 401 represents the subject, and 402 represents the center of the virtual camera in the virtual space. The maximum / minimum distance is calculated from the camera parameters input from the stereoscopic image input device and the parallax map calculated by the stereoscopic image information generation unit 101. The distance to the maximum parallax d _max is set as Z _nr , and the distance to the minimum parallax d _min is set as Z _fr , which is calculated from the relationship between the parallax and the camera base line length according to Expression (2).

Z _nr = b · f / d _max , Z _fr = b · f / d _min ---- (2)
Next, the region on which the perspective projection is performed on the virtual camera is matched with the aspect ratio A = W / H of the reference image determined by the stereoscopic image information generation unit 101. Here, W and H actually represent the number of horizontal pixels and the number of vertical pixels of the reference image of the stereoscopic image information generation unit. At this time, in the space obtained by extending the perspective projection center 402 to the virtual camera and the quadrangular pyramid extending from the perspective projection area to the farthest distance Z _fr , the area from Z _{nr to} Z _{fr in the} depth direction from the virtual camera is represented in the virtual space. A view volume 405 is constructed. A three-dimensional model arranged in the view volume 405 is drawn as a virtual space image.

  S303 is a live-action / CG composite display step, which generates a composite image of the image in which the three-dimensional model is arranged in the virtual space constructed in the virtual space construction step S302 and the reference image determined in the stereoscopic image information generation step S301. The composite image is displayed on the two-dimensional display device 101. A bird's-eye view at the time of synthesis is shown in FIG. This shows a state in which the three-dimensional model 406 is arranged in the virtual space constructed in FIG. FIG. 5 shows an image synthesized by perspective projection in the view model. In order to generate a composite image, first, a live-action image determined as a reference image by the stereoscopic image information generation unit 101 as a background 501 is generated as a background on a virtual space, and then the three-dimensional models 502 and 503 are perspectively projected. The images are generated in the order of going. Thus, a live-action / CG composite image as shown in FIG. 5 is displayed on the two-dimensional image display device 106.

S304 is a CG model layout step, in which a three-dimensional model of a composite image displayed on the two-dimensional display device 101 is selected via the operation input device 105, and is performed via operations such as movement, rotation, enlargement, and reduction. The outline will be described with reference to FIG. 5. When a triangular pyramid 502 is selected from among a plurality of three-dimensional models, a circumscribed rectangle 504 of the selected three-dimensional model of the triangular pyramid 502 is displayed. The 3D model selected by the operator is clearly specified, and the selected 3D model is operated. The movement in the depth direction will be described with reference to FIG. FIG. 6A is an image that is perspective-projected as shown in FIG. 5 and shows an outline of the XZ plane of the view volume displayed in FIG. 4B. A main subject, 602 and 603 are three-dimensional models, and 604 represents a circumscribed rectangle of the selected three-dimensional model 602. Here, when the three-dimensional model 602 is moved in the depth direction, it can be arranged behind the main subject 601 on the editing screen as shown in FIG. For this reason, when a three-dimensional composite image is finally generated, the three-dimensional model image that should be positioned on the back side is stereoscopically overlapped with the main subject in front, and correct depth information cannot be reproduced. Therefore, the region surrounded by oblique lines in FIG. 6 (a), the words, as a space three-dimensional model movable space 605 from the distance Z _lr to the front side of the main object to the shortest distance Z _n of the view volume, that region To check that the circumscribed rectangle 604 of the three-dimensional model does not go out of the area. If the selected 3D model collides with the boundary of the 3D model movable space 605, even if it is notified that the 3D model cannot be moved in the depth direction by shining the screen or outputting sound. Good. In addition, a region in which a region that is not hidden behind the main subject 601 with respect to the photographing position is added as a region where the three-dimensional model can be moved, such as a region 606 indicated by diagonal lines in FIG. Good. Here, the depth direction of the three-dimensional model movable space is set to the distance Z _lr to the main subject. However, when various subjects exist before the main subject, the distance to the subject may be used. Further, it may be possible to edit or operate the three-dimensional model movable space 605 by presenting a diagram as shown in FIG. 6 to the user.

  S305 is a multi-viewpoint image sequence generation step, which is arranged and displayed in the virtual space constructed in the virtual space construction step S302 and the real-image multi-viewpoint image sequence based on the reference image and the parallax map calculated in the stereoscopic image information generation step S301. When displayed on the device 106, a multi-viewpoint image sequence is generated by combining a virtual space multi-viewpoint image sequence generated from a virtual space generated from a three-dimensional model.

  FIG. 7 shows the flow of processing in the multi-viewpoint image sequence generation step S305. Step S701 is a live-action multi-view image sequence generation unit that generates a multi-view image sequence from the reference image and the generated parallax map in the stereoscopic image information generation step S301. The processing of the multi-viewpoint image sequence generation unit can be realized using a technique for generating a multi-viewpoint image sequence described in JP-A-2001-346226.

  Step S702 is a virtual space multi-view image generation parameter determination step, which uses the virtual space parameters displayed on the two-dimensional display device 101, the parallax map generated in the stereoscopic image information generation step S301, and the like. Determine the parameters in the virtual space to generate.

Parameter generation in the virtual space will be described with reference to FIG. FIG. 8 shows the XZ plane of the view volume laid out as shown in FIG. 4B. The three-dimensional models 801 and 802 are arranged as shown in the figure, and the virtual camera 804 at that time is It is assumed that it is arranged at the position X0. Also, when generating a stereoscopic image as commonly observed in such a lenticular plate, since in many cases to adjust the parallax so that the vicinity of the main object on the disparity 0, disparity near depth Z _0r the main subject exists 0 Therefore, it is necessary to generate a virtual space multi-viewpoint image sequence. First, _let Z _nr , Z _fr of the view volume and the distance Z _0r to adjust the parallax to zero. In addition, the camera base line length b when actually captured by the stereoscopic image input apparatus 100 may be stored as a predetermined fixed value, or a typical value is selected by the operator when the virtual space is generated. Also good. The movement range of the virtual camera is set so that the virtual camera position parameter is generated so that an image is generated by arranging or moving the virtual camera symmetrically about the position X0 and at equal intervals.

Step S703 is a virtual space multi-view image sequence generation step, which generates a virtual space multi-view image sequence. In this case, unlike the case where the image is displayed on the above-described two-dimensional display device 106 (step S303 in FIG. 3), a projection image is generated by directly perspective-transforming only the virtual space without drawing a reference image as a background. I will do it. At this time, in the subsequent image synthesizing step S704, the real multi-view image sequence generated in the real multi-view image sequence generating step and the virtual space multi-view image sequence generated in the virtual space multi-view image sequence generating step S703 are combined. Therefore, the chroma key composition can be performed by initializing the background of the virtual space multi-viewpoint image generated in step S703 with a predetermined color or a predetermined luminance value. Further, for each of the generated virtual space multi-viewpoint images, the images are translated and adjusted so that the parallax becomes 0 at the depth distance _Z0r . As described in Japanese Patent Laid-Open No. 2001-346226, the order of generating the virtual space multi-viewpoint image sequence is reversed in consideration of arranging images on stripes in reverse order in consideration of observation with a lenticular plate. (Right side → left side), or vice versa. That is, by generating in the same order as the live-action multi-viewpoint image sequence generating means, it is possible to facilitate the control in the subsequent image composition S704.

  Step S704 is an image synthesis step, in which the multi-view image sequences generated in the live-action multi-view image sequence generation step S701 and the virtual space multi-view image sequence generation step S704 are combined. Specifically, as described above, it is possible to easily obtain a composite image by duplicating a pixel portion initialized with a predetermined color or the like in a virtual space image from a live-action multi-viewpoint image, and other portions from the virtual space image. Can do. Alternatively, when the virtual space image is generated in the virtual space multi-viewpoint image sequence generation step S703, the depth information at each viewpoint position is stored, and only the portion that is the farthest point of the depth information is captured in the image composition step S704. If it is configured so that it is duplicated from the image and the others are duplicated from the virtual space image, if there is a predetermined color or brightness in the 3D model as in chroma key composition, the live-action image will be mistaken for that part. There is an effect that problems such as synthesis do not occur. Alternatively, when the virtual viewpoint image is generated in the multi-viewpoint image sequence generation unit, the depth information at each viewpoint position is stored, and the part of the pixel at the farthest point in the depth information is displayed as the stereoscopic image information. You may make it perform forward mapping directly from the reference | standard image produced | generated by production | generation step S301, and a depth map. As a result, it is not necessary to generate a live-action image multi-viewpoint image sequence for each viewpoint position, and only a live-action image at a required position can be generated, thereby reducing the storage memory capacity and increasing the processing speed. There is.

  S306 is a three-dimensional image integration step, in which the multi-view image sequence generated in the multi-view image sequence generation step S307 is combined with a three-dimensional image and printed by the printing apparatus 104. The three-dimensional image integration step S306 can be realized by the three-dimensional image synthesizing means described in JP-A-2001-346226 previously proposed by the present applicant.

  Step S307 is a printing step, and the three-dimensional image synthesized in the three-dimensional image integration step S306 is printed. By observing an optical member such as a lenticular plate superimposed on the printing result, a good stereoscopic image can be observed.

  As described above, the user can lay out the captured stereoscopic image on the two-dimensional display device without actually printing it, and can obtain a high-quality stereoscopic image that is not three-dimensionally uncomfortable.

(Second embodiment)
In the first embodiment, in the virtual space construction step S302 for constructing the virtual space, the virtual space is generated using the depth information to the subject such as the parallax map generated in the stereoscopic image information generation step S301. However, it is possible to configure without using the generated parallax map or the like. That is, it is also possible to synthesize a real image and a virtual space image by an approach of synthesizing images so as to maintain consistency in a three-dimensional space constituted by a lenticular plate to be finally observed. Hereinafter, this method will be described.

In the present embodiment, there is no change in the apparatus configuration as in the first embodiment, and this can be realized by changing the virtual space multi-viewpoint image generation parameter determination step S702 in FIG. The outline of this change will be described with reference to FIG. FIG. 12A shows an XZ plane of a three-dimensional space reproduced by a lenticular plate, and represents optimum maximum / minimum parallax d _max and d _min when observing with the lenticular plate to be observed. When observing an image with a parallax larger or smaller than the maximum / minimum optimum for observation, there is a high possibility that an appropriate stereoscopic effect may not be obtained or that the observer may become tired. Therefore, it is common to provide a restriction such as the optimum maximum / minimum parallax. In the figure, d ₀ means parallax 0 that does not sink or rise on the printing surface. Here, in order to keep the depth direction of the consistency with the live-action and virtual spaces as described also in the first embodiment, of the three-dimensional space to be reproduced by the lenticular plate as in FIG, d _max ~ d _min '1202 in the virtual _{space, d min'} by to assign _{to d} min 1201 live-action space, it is possible to synthesize a three-dimensional photographs in three dimensions is inaccurate not occur even if it is reproduced .

Next, a method for determining a virtual camera movement range for generating a virtual space image so as to have a predetermined parallax within the virtual space multi-viewpoint will be described with reference to FIG. In the figure, Z _nv and Z _fv are the nearest / farthest distances laid out in the virtual space, respectively, and are the nearest / farthest planes on which the three-dimensional model is arranged. This can be easily calculated from the arrangement information of the three-dimensional model in the virtual space. Z _0v represents d _{0 in} FIG. 12A, that is, a distance at which the parallax is desired to be zero. In this arrangement, a space from Z _nv to Z _fv is represented by a virtual camera movement range B such that parallax d _max and d _min ′ are represented as shown in FIG. A movement amount Δp that translates the virtual image for setting the parallax to 0 at the depth Z _0v when it is generated is calculated.

Now, _assuming that the focal length in the virtual space is f ′, and the original parallaxes at the depths Z _nv , Z _fv , and Z _0v are d _n , d _f , and d ₀ , each parallax is expressed by Equation (3) Can express.

_{d n = Bf '/ Z nv} , d f = Bf' / Z fv, d 0 = Bf '/ Z 0v ---- (3)
In addition, since the generated virtual image is appropriately translated so that the parallax at Z _0v becomes ₀ , d ₀ may be subtracted from the actual parallax at each position, and therefore, it can be expressed as in Expression (4).

_{d n '= d n -d 0} , d f' = d f -d 0, d 0 '= 0 ---- (4)
Now, the camera movement amounts B and Z ₀ are calculated when given as d _n ′ = d _max and d _f ′ = d _min ′, and when Z _nv and Z _{fv in the} virtual space are given. Now, if Z _0v = αZ _nv and Z _fv = AZ _nv (α, A: constant), then Equation (5) is obtained.

_{d n '= Bf' / Z} nv -Bf '/ αZ nv = Bf' (1-1 / α) / Z nv
_{d f '= Bf' / AZ} nv -Bf '/ αZ nv = Bf' (1 / A-1 / α) / Z nv ---- (5)
From the above relationship, the distance B of the camera photographing position can be calculated by Expression (6).

_{B = Z nv × d n '} / (f' (1-1 / α))
= Z _nv × d _n '/ (f' (1 / A-1 / α)) ---- (6)
However, (alpha) is calculated by Formula (7).

α = (d _n ′ −d _f ′) / (d _n ′ / A−d _f ′)
_{= (D n '-d f'} ) / (d n 'Z nv / Z fv -d f') - --- (7)
Here, when the number of images to be generated is N, since the camera shooting interval ΔB = B / (N−1), the parallel movement amount Δp at the i-th shooting position (i: 0 to N−1) is expressed by the formula ( 8) (however, it is calculated by Z _0v = αZ _nv ).

_{Δp = Δbf '× i / Z} 0v ---- (8)
The virtual multi-viewpoint image sequence generated by setting the parameters described above and the real-shot multi-viewpoint image sequence are combined and printed as a three-dimensional image in the same manner as in the first embodiment, and the lenticular plate is superimposed on the print result. As a result, it is possible to finally obtain a stereoscopic photograph without a sense of incongruity without matching the virtual space multi-viewpoint image sequence generation parameter with the three-dimensional spatial information of the real image.

  In addition, if the virtual space multi-view image generation parameter determination unit according to the present embodiment is used, a virtual space multi-view image sequence is possible even when only a live-shot multi-view image sequence is input, and there is no sense of incongruity stereoscopically. This means that a three-dimensional image can be obtained. Therefore, it is possible to generate and save parallax maps from live-action stereo images within this image synthesizer, and to generate good 3D synthesized images, simplifying the processing speed and internal device configuration The effect that can be obtained.

  In the embodiments described so far, the three-dimensional model is configured so that only geometric operations such as movement, rotation, enlargement, and reduction can be executed. However, in order to generate a natural virtual space image in terms of image quality, The type and parameters of the light source in the space may be changed while observing the composite image with the real image.

  In this embodiment, the three-dimensional model is stored in the three-dimensional model storage unit. However, the three-dimensional model may be input from a network.

  The present invention is not limited to the apparatus of the above-described embodiment, and may be applied to a system constituted by a plurality of devices or an apparatus constituted by one device. A storage medium storing software program codes for realizing the functions of the above-described embodiments is supplied to a system or apparatus, and a computer (or CPU or MPU) of the system or apparatus reads and executes the program codes stored in the storage medium. Needless to say, it will be completed by doing.

  In this case, the program code itself read from the storage medium realizes the functions of the above-described embodiments, and the storage medium storing the program code constitutes the present invention. As a storage medium for supplying the program code, for example, a floppy (registered trademark) disk, hard disk, optical disk, magneto-optical disk, CD-ROM, CD-R / RW, magnetic tape, nonvolatile memory card, and ROM are used. I can do it. In addition, by executing the program code read by the computer, not only the functions of the above-described embodiments are realized, but also the OS running on the computer based on the instruction of the program code performs the actual processing. Needless to say, a case where the function of the above-described embodiment is realized by performing part or all of the processing is also included.

  Furthermore, after the program code read from the storage medium is written to the memory provided in the function expansion board inserted in the computer or the function expansion unit connected to the computer, the program code is expanded based on the instruction of the next program code. It goes without saying that the functions of the above-described embodiments may be realized by performing some or all of the actual processing by the CPU or the like provided on the expansion board or the expansion unit.

1 is a block diagram illustrating a configuration of an image composition device according to a first embodiment of the present invention. It is a figure which shows the outline of the image acquisition of a stereo camera. It is a flowchart of the processing program of the image composition device concerning an embodiment. It is the figure which showed the bird's-eye view of the three-dimensional space (b) synthesize | combined with the three-dimensional space (a) and virtual space of a real image. FIG. 3 is a diagram showing a perspective projection image that is displayed on the two-dimensional image device 106 by combining a real image and a virtual space image. It is a figure of the XZ plane explaining the area | region which can move a 3D model with the 3D model operation part 105. FIG. It is a flowchart which shows the detail of a multiview image sequence production | generation step. It is a figure of the XZ plane which shows the outline of the virtual image generation in virtual space. It is a block diagram which shows the outline | summary of the process in a prior art example. It is a figure explaining the stereo adapter in a prior art example. It is a flowchart which shows the processing flow in a prior art example. It is a figure for demonstrating a virtual space image generation parameter determination part.

Explanation of symbols

DESCRIPTION OF SYMBOLS 100 Stereoscopic image input apparatus 101 Two-dimensional display apparatus 102 Image composition apparatus 103 Printing apparatus 104 Operation input apparatus

Claims (18)

An image synthesis device that generates a stereoscopic photograph by synthesizing a real image and a virtual space image generated from a three-dimensional model image arranged in a virtual space,
A virtual space construction means for determining a movable range of the virtual space and the three-dimensional model from the depth information to the subject;
3D model operation means for performing at least one of movement, rotation, enlargement, and reduction within the range determined by the virtual space construction means for the 3D model;
Multi-viewpoint image sequence generation means for generating a composite multi-viewpoint image sequence by combining a virtual space image generated from a three-dimensional model arranged in a virtual space determined by the three-dimensional model operation means and a real image;
3D image integration means for integrating the synthesized multi-viewpoint sequence image generated by the multi-viewpoint image sequence generation means into a 3D image;
An image synthesizing apparatus comprising:   The image synthesizing apparatus according to claim 1, wherein the multi-view image sequence generation unit generates and synthesizes each of the photographed multi-view image and the virtual space multi-view image.   The multi-viewpoint image sequence generation means refers to depth information when generating a virtual space multi-viewpoint image, determines whether or not a certain pixel is drawing a three-dimensional model from the depth information, and determines the three-dimensional model. If it is determined that drawing is performed, the same pixel information as that of the pixel of the virtual space multi-viewpoint image is used. If it is determined that the three-dimensional model is not drawn, the same pixel information as that of the real-shot multi-viewpoint image is used. The image composition apparatus according to claim 1, wherein a composite image is generated.   The multi-viewpoint image sequence generation means refers to depth information when generating a virtual space multi-viewpoint image, determines whether or not a certain pixel is drawing a three-dimensional model from the depth information, and determines the three-dimensional model. If it is determined that the pixel is drawn, the same pixel information as the pixel of the virtual space multi-viewpoint image is used. If it is determined that the three-dimensional model is not drawn, the same pixel information as the pixel is generated from the stereoscopic image information. 4. The image composition apparatus according to claim 1, wherein a composite image is generated.   4. The multi-view image sequence generation unit generates and synthesizes a multi-view image in the same view order for a live-action multi-view image and a virtual space multi-view image sequence. Alternatively, the image composition apparatus according to claim 4.   2. The image composition according to claim 1, wherein when the three-dimensional model moving means touches or exceeds the moving range of the three-dimensional model determined by the virtual space construction means, the fact is presented to the operator. apparatus. In an image composition device that creates a stereoscopic photograph by synthesizing a real-view multi-viewpoint image sequence photographed or generated at a plurality of photographing positions and a virtual space image generated from a three-dimensional model arranged in the virtual space,
Virtual space image generation parameter determination means for generating a virtual space image from the stereoscopic image observation parameters;
Virtual space multi-viewpoint image sequence generation means for generating a virtual space multi-viewpoint image sequence based on the parameters determined by the virtual space image generation parameter determination means;
Image synthesizing means for synthesizing the virtual space multi-view image sequence generated by the virtual space multi-view image sequence generating means and the multi-view image sequence;
3D image integration means for integrating the synthesized image synthesized by the real-life virtual space image synthesis means as a 3D image;
An image synthesizing apparatus comprising:   The image synthesizing apparatus according to claim 7, wherein the virtual space image generation parameter determining unit arranges the three-dimensional model of the virtual space on the front side of the real space.   8. The image synthesizing apparatus according to claim 7, wherein the virtual space image generation parameter determination means determines a parameter as a virtual space region that reproduces the nearest distance and the farthest distance of the three-dimensional model arranged in the virtual space. . An image combining method for generating a stereoscopic photograph by combining a real image and a virtual space image generated from a three-dimensional model image arranged in a virtual space,
A virtual space construction process for determining the movable range of the virtual space and the three-dimensional model from the depth information to the subject;
A three-dimensional model operation step of performing at least one of movement, rotation, enlargement, and reduction within the range determined by the virtual space construction means for the three-dimensional model;
A multi-viewpoint image sequence generation step of generating a composite multi-viewpoint image sequence by combining a virtual space image generated from a three-dimensional model arranged in a virtual space determined in the three-dimensional model operation step and a real image;
A three-dimensional image integration step of integrating the composite multi-view sequence image generated by the multi-view image sequence generation means into a three-dimensional image;
An image synthesizing method characterized by comprising:   11. The image synthesizing method according to claim 10, wherein the multi-view image sequence generating step generates a real-shot multi-view image and a virtual space multi-view image, respectively, and then combines them.   The multi-viewpoint image sequence generation step refers to depth information when generating a virtual space multi-viewpoint image, determines whether a pixel is drawing a three-dimensional model from the depth information, and determines the three-dimensional model. If it is determined that drawing is performed, the same pixel information as that of the pixel of the virtual space multi-viewpoint image is used. If it is determined that the three-dimensional model is not drawn, the same pixel information as that of the real-shot multi-viewpoint image is used. The image composition method according to claim 10, wherein a composite image is generated.   The multi-viewpoint image sequence generation step refers to depth information when generating a virtual space multi-viewpoint image, determines whether a pixel is drawing a three-dimensional model from the depth information, and determines the three-dimensional model. If it is determined that the pixel is drawn, the same pixel information as the pixel of the virtual space multi-viewpoint image is used. If it is determined that the three-dimensional model is not drawn, the same pixel information as the pixel is generated from the stereoscopic image information. 13. The image composition method according to claim 10 or 12, wherein a composite image is generated.   The multi-view image sequence generation step generates and synthesizes the multi-view images in the same shooting viewpoint sequence for the real-shot multi-view image and the virtual space multi-view image sequence. The image composition method according to claim 12 or claim 13.   11. The image composition according to claim 10, wherein when the three-dimensional model movement step touches or exceeds the movement range of the three-dimensional model determined in the virtual space construction step, the fact is presented to the operator. Method. In an image composition method for creating a stereoscopic photograph by synthesizing a real-shot multi-viewpoint image sequence photographed or generated at a plurality of photographing positions and a virtual space image generated from a three-dimensional model arranged in the virtual space,
A virtual space image generation parameter determination step for generating a virtual space image from the stereoscopic image observation parameters;
A virtual space multi-view image sequence generation step for generating a virtual space multi-view image sequence based on the parameters determined by the virtual space image generation parameter determination step;
An image synthesis step of synthesizing the virtual space multi-view image sequence and the multi-view image sequence generated in the virtual space multi-view image sequence generation step;
A three-dimensional image integration step of integrating the synthesized image synthesized by the live-action virtual space image synthesis step as a three-dimensional image;
An image synthesizing method characterized by comprising:   The image composition method according to claim 16, wherein in the virtual space image generation parameter determination step, a three-dimensional model of the virtual space is arranged on the near side of the real space. 18. The image composition method according to claim 17, wherein, in the virtual space image generation parameter determination step, parameters are determined as a virtual space region that reproduces the nearest distance and the farthest distance of the three-dimensional model arranged in the virtual space. .

Images