JP7393092B2 - Virtual viewpoint image generation device, method and program - Google Patents

Description

The present invention provides a virtual viewpoint video generation device that can provide a high-quality virtual viewpoint video that does not feel strange even in an environment where a pulling camera and a closer camera coexist when generating a 3D model of a subject and synthesizing the virtual viewpoint video. , relating to methods and programs.

Free viewpoint (virtual viewpoint) video technology is a technology that acquires video images from multiple cameras and enables video viewing from any viewpoint, including a viewpoint where no camera is present. As one method for realizing a free-viewpoint video, there is a 3D model-based free-viewpoint video generation method based on a visual volume intersection method disclosed in Non-Patent Document 1.

The visual volume intersection method, as shown in Figure 9, projects a binary silhouette image into 3D space by extracting only the object part from each camera image, and leaves only the part that becomes the intersection set as a 3DCG model. This is a method for generating 3D models.

The visual volume intersection method uses the full model free viewpoint method (a method that faithfully represents the shape of a 3D model) disclosed in Patent Document 1, and the billboard method free viewpoint method (method that uses a 3D model as a billboard) disclosed in Non-Patent Document 2. It is used as a basic technology to realize a billboard that is created in the shape of a billboard and maps the texture from a nearby camera onto the billboard.

In the free viewpoint production disclosed in Non-Patent Document 1, first, a 3D space in which a free viewpoint video is to be produced is filled with a voxel grid divided by a cubic grid. Next, the three-dimensional position of each voxel grid is back-projected onto the silhouette image of each camera, and the silhouette image at the corresponding position is referred to. Then, a voxel grid whose silhouette is determined to be white (the subject is present) by many cameras is modeled.

This type of free-viewpoint video can be used for applications such as watching and enjoying sports interactively in real time from any viewpoint, and by taking advantage of its ability to create video from any viewpoint, it can be used to create realistic scenes based on determined camera work. It has been used for purposes such as creating impressive replay videos.

Japanese Patent Application Publication No. 2018-063635 Patent application No. 2020-133176

Laurentini, A. "The visual hull concept for silhouette based image understanding.", IEEE Transactions on Pattern Analysis and Machine Intelligence, 16, 150-162, (1994). H. Sankoh, S. Naito, K. Nonaka, H. Sabirin, J. Chen, "Robust Billboard-based, Free-viewpoint Video Synthesis Algorithm to Overcome Occlusions under Challenging Outdoor Sport Scenes", Proceedings of the 26th ACM international conference on Multimedia, pp. 1724-1732, (2018) J. Chen, R. Watanabe, K. Nonaka, T. Konno, H. Sankoh, S. Naito, "A Fast Free-viewpoint Video Synthesis Algorithm for Sports Scenes", 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems ( IROS 2019), WeAT17.2, (2019). Qiang Yao, Hiroshi Sankoh, Nonaka Keisuke, Sei Naito. "Automatic camera self-calibration for immersive navigation of free viewpoint sports video," 2016 IEEE 18th International Workshop on Multimedia Signal Processing (MMSP), 1-6, 2016. C. Stauffer and W. E. L. Grimson, "Adaptive background mixture models for real-time tracking," 1999 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 246-252 Vol. 2 (1999). R. S. Yang et al., "Multi-Kinect scene reconstruction: Calibration and depth inconsistencies," 2013 28th International Conference on Image and Vision Computing New Zealand (IVCNZ 2013), Wellington, 2013, pp. 47-52. Shuang Sun et al.,"Parametric Human Shape Reconstruction via Bidirectional Silhouette Guidance," Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2019.

Camera placement is important when producing free-viewpoint videos. For example, if you have a camera that is very close to the subject, you can create a camera work that starts from the angle of view of the pulled camera and uses the close camera as the goal. It is possible to achieve this with a texture with a certain degree of precision.

The closer the camera is to the subject, the sharper the texture will be obtained, but the closer the camera is to the subject, the smaller the area on the stadium that will be reflected by the camera. In particular, in a case where only a specific camera is moving very close to the subject, the subject that is outside the field of view of the closer camera will disappear without being formed into a 3D model.

FIG. 10 is a diagram comparing the range in which the intersection set is formed when all cameras are pulling cameras [Figure 10(a)] and when one moving camera is included [Figure 10(b)]. In the figure (b), the range of the intersection set is smaller than in the figure (a), and it can be seen that the 3D model of the object outside the field of view of the closer camera may disappear without being formed.

These technical issues can be solved by changing the threshold for the number of cameras involved in 3D model generation, such as creating a 3D model for the part that is visible from N-1 cameras in an environment with N cameras.

Alternatively, as the inventors of the present invention have invented and already applied for a patent (Patent Document 2), when a 3D voxel model of the subject is generated by the visual volume intersection method using the camera images of the approaching camera and the retracting camera. , As shown in Figure 11, even if the 3D model is prevented from disappearing by processing the object as if it exists outside the viewing angle range of the closer camera, and processing it as if the object does not exist with the pull camera. good.

On the other hand, even if it is possible to generate a 3D model in an environment where a pulling camera and a closer camera coexist by making full use of each of the above methods, the subject that is photographed by the closer camera will only be seen from the closer camera and only by the pulling camera. A rendering process is required to map each subject from the camera.

In addition, when textures are mapped from each camera to an object that exists at the field of view boundary between the closer camera and the closer camera, as shown in Figure 12, the difference in the resolution of each camera and the difference in the cameras being mapped cause problems. However, there was a problem in that the break at the border of the viewing angle was clearly noticeable, creating an unnatural feeling in the viewing quality.

An object of the present invention is to map textures to a 3D model created in an environment where a pulling camera and a leaning camera coexist to generate a virtual viewpoint image, and to determine the positional relationship with the viewing angle boundary of the leaning camera for each 3D model. An object of the present invention is to provide a virtual viewpoint video generation device, method, and program that do not create a break in the field angle boundary by performing mapping from only one of a pulling camera and a closer camera.

In order to achieve the above object, the present invention provides a virtual viewpoint video generation device that generates a virtual viewpoint video based on camera images taken of a subject from a plurality of viewpoints, and is characterized in that it includes the following configuration. .

(1) A means for generating a 3D model based on each camera image of a closer camera and a closer camera, a means for determining whether or not a 3D model of a subject exists on the field of view boundary of a close camera, and a means for generating a 3D model The mapping means includes a means for mapping a texture from each camera image, and the mapping means includes a means for mapping a texture from the camera image of the pulling camera, including a 3D model existing on the field of view boundary, from the camera image of the pulling camera. mapped.

(2) The determining means is to determine whether or not the 3D bounding box containing the 3D model exists on the field of view boundary of the camera.

(3) The means for generating a 3D model employs the visual volume intersection method based on camera images from the approaching camera and the retracting camera, and a means for constructing a low-resolution voxel model based on the silhouette of the subject; means for constructing a high-resolution voxel model in the region of the low-resolution voxel model based on the 3D model; It was decided whether or not to do so.

(4) If the 3D model is a polygon model, occlusion information is generated that records whether each polygon of the 3D model is visible or invisible from each camera, and occlusion information is generated that records whether each polygon of the 3D model is visible or invisible from each camera. The means for rewriting and mapping the occlusion information of the closer camera to the invisible one is to map a texture to each polygon from a camera that makes the polygon visible.

(1) The virtual viewpoint image generation device of the present invention performs texture mapping on a 3D model that exists on the field of view boundary of the close camera from the camera image of the pull camera, including within the field of view of the close camera. By doing this, we can prevent the quality degradation that can occur when textures are mapped from both the local camera and the pull camera to a single 3D model.

(2) Determine whether the 3D model of the subject exists on the field of view boundary of the close camera based on whether the 3D bounding box containing the 3D model exists on the field of view boundary of the close camera. Therefore, if all eight vertices of the 3D bounding box are within the view angle range of the closer camera, or if all eight vertices are outside the view angle range of the closer camera, it can be determined that the 3D bounding box does not exist on the view angle boundary. Therefore, very fast determination is possible.

(3) Whether or not the 3D model of the subject exists on the field of view boundary of the camera is determined at the time of the low-resolution voxel model, so there is no need to wait for the results of high-resolution voxel model generation. It becomes possible to perform boundary determination in parallel with the above, and it becomes possible to perform processing at high speed.

(4) The occlusion information of the closer camera regarding the polygons of the 3D model existing on the boundary of the view angle is rewritten to be invisible, so you can obtain an appropriate camera image by simply referring to the occlusion information without referring to the boundary determination result. You can now map textures from

FIG. 1 is a functional block diagram of a virtual viewpoint video generation device according to a first embodiment of the present invention. FIG. 7 is a diagram showing an example of dividing a 3D model into each subject using bounding boxes. FIG. 7 is a diagram showing a method of performing texture mapping to each 3D model from camera images according to the angle of view. FIG. 2 is a functional block diagram of a virtual viewpoint video generation device according to a second embodiment of the present invention. FIG. 3 is a diagram showing an example of camera parameters. FIG. 3 is a functional block diagram of a virtual viewpoint video generation device according to a third embodiment of the present invention. FIG. 7 is a functional block diagram of a virtual viewpoint video generation device according to a fourth embodiment of the present invention. FIG. 7 is a diagram illustrating an example of rewriting occlusion information based on the result of boundary determination. FIG. 2 is a diagram showing a method of generating a 3D model using a visual volume intersection method. FIG. 7 is a diagram illustrating an example in which a subject outside the viewing angle range of a close-up camera disappears without a 3D model being formed. FIG. 2 is a diagram illustrating a 3D model production method according to Patent Document 2. FIG. 3 is a diagram showing an example in which image quality deteriorates when textures are mapped from a closer camera and a puller camera to one 3D model.

Embodiments of the present invention will be described in detail below with reference to the drawings. FIG. 1 is a functional block diagram showing the configuration of the main parts of a virtual viewpoint video generation device 1 according to the first embodiment of the present invention, and the main components are a 3D model production server 10 and a rendering server 20. Here, we will explain a case where a sports scene is photographed using N cameras Cam1 to CamN, some of which are close-up cameras and the rest are rear cameras.

Such a virtual viewpoint video generation device 1 can be configured by installing applications (programs) that realize each function on a general-purpose computer or server. Alternatively, it can be configured as a dedicated machine or single-function machine in which part of the application is converted into hardware or software.

The 3D model production server 10 includes a silhouette image acquisition section 101, a 3D model generation section 102, and a boundary determination section 103, and generates a 3D model for each subject and provides it to the rendering server 20. Furthermore, the positional relationship with the viewing angle boundary of the close-up camera is determined for each 3D model, and the determination result is provided to the rendering server 20.

The silhouette image acquisition unit 101 acquires from the silhouette image database 30 silhouette images used for 3D model generation using the visual volume intersection method from each camera image of the approaching camera and the pulling camera. In order to generate a 3D model using the visual volume intersection method, it is desirable to obtain silhouette images from three or more cameras.

The silhouette image is given in the form of a binary mask image in which the subject area for which the 3D model is generated is represented in white (=1), and the other areas are represented in black (=0). Any existing method, such as the background subtraction method disclosed in Non-Patent Document 5, can be used to generate such a silhouette image.

The 3D model generation unit 102 calculates a 3D voxel model of the subject by a visual volume intersection method using N silhouette images, based on the silhouette image acquired by the silhouette image acquisition unit 101 and separately provided camera classification information. . Here, the camera classification information is information that identifies whether each camera is a closer camera or a closer camera.

The visual volume intersection method obtains the common part of the visual cone when N silhouette images are projected onto the 3D world coordinates as the visual volume (Visual Hull) VH(I), and is expressed by the following formula (1). It is indicated by. Here, set I is a set of silhouette images of each camera, and Vi is a viewing cone calculated from the silhouette images obtained from the i-th camera.

The voxel model generated in this manner may be treated as voxels, or may be converted into a polygon model based on the Martin Cube method or the like. Here, the explanation will be continued assuming that it is converted to a polygon model.

The boundary determining unit 103 determines whether the 3D model generated by the 3D model generating unit 102 exists on the view angle boundary of the close camera. In this embodiment, as shown in Figure 2, a 3D bounding box that includes each independent 3D model block is defined, and it is determined whether the 3D model exists on the view angle boundary for each 3D bounding box. do.

When determining a 3D bounding box, if all 8 vertices are within the field of view of the camera or all 8 vertices are outside the field of view of the camera, the 3D bounding box does not exist on the field of view boundary. It is determined that If the determination is made on a bounding box basis, only 8 vertices need to be checked, making it possible to perform extremely fast determination.

On the other hand, the 3D bounding box is not exactly the same as the shape of the 3D model. Therefore, even though the included 3D model is within the field of view of the close camera, if only the vertices of the 3D bounding box leak outside the field of view of the close camera, errors may occur in boundary determination.

Considering accuracy, it is desirable to use a voxel model included in a 3D bounding box to make a determination, rather than using a 3D bounding box unit. For example, if the center points of all voxels in the voxel model are back-projected in the direction of the closer camera, and there are center points that fall within the field of view of the closer camera and center points that do not, then this object exists in the boundary area. Judgment is made as a matter of fact. If there are multiple closer cameras, the boundary determination result may be calculated by the number of subjects x the number of closer cameras.

The rendering server 20 uses the shape information of the 3D model of the subject produced by the 3D model production server 10 and each camera image (texture) to render a composite image seen from a virtual viewpoint. In this embodiment, free viewpoint rendering is performed using a full model.

Note that the rendering server 20 may be configured on the same computer as the 3D model production server 10, or may be configured as a separate server. Generally, a 3D model only needs to be calculated once for a specific frame, so it is calculated quickly on a high-end PC and saved, and then this 3D model is transferred to a virtual viewpoint viewing terminal equipped with a rendering function. By configuring it for distribution, it is possible to distribute video to multiple terminals, including one high-end PC and low-spec terminals.

In the rendering server 20, the virtual viewpoint selection unit 201 detects a viewpoint selection operation by an operator and acquires the position and orientation of the virtual viewpoint _pv . The boundary dependent mapping unit 202 maps the texture from each camera image to each polygon of the 3D model based on the virtual viewpoint p _v and the result of boundary determination. The virtual viewpoint video output unit 203 outputs the rendered composite video as a virtual viewpoint video.

FIG. 3 is a diagram schematically showing a texture mapping method by the boundary-dependent mapping unit 202. For a 3D model that is determined to be within the field of view of the close-up camera without straddling the field-of-view boundary, only the texture extracted from the camera image of the close-up camera is mapped. In addition, only the texture extracted from the camera image of the pull camera is mapped to a 3D model that is determined to be within the view angle of the pull camera without crossing the view angle boundary.

In addition, for 3D models that are determined to be within the field of view of the camera without straddling the field of view boundary, those 3D models that are also within the field of view of the camera will be automatically The texture may be mapped.

On the other hand, for a 3D model that is determined to be on the field of view boundary of the close camera, only the texture extracted from the camera image of the pull camera, including the area within the field of view of the close camera, is displayed. mapped. This makes it possible to prevent the deterioration of video quality (Fig. 12) that may occur due to the textures of both the closer camera and the closer camera being mapped to the 3D model on the field-of-view boundary.

Although the first embodiment described above has been described on the assumption that camera classification information is provided separately, the present invention is not limited to this, and as in the second embodiment shown in FIG. A camera classification unit 104 that outputs camera classification information based on parameters may be provided so that the classification result changes adaptively in accordance with the focal length that changes due to a zoom operation or the like.

The camera classification unit 104 automatically classifies the N cameras as close cameras or close cameras by using camera parameters given by the following equation (2).

Camera parameters are used to convert a point (X, Y, Z) on the world coordinates to a 2D point (u, v) on the camera image, and r ₁₁ to r ₃₃ are rotation matrices that indicate the camera orientation. , t ₁ to t ₃ are translation matrices representing the camera position, and together they are called the camera's extrinsic parameters.

f _x and f _y are focal lengths in pixel units that indicate the degree of zooming, and c _x and c _y are principal points of the image, which are usually the center of the image. Parameters such as the focal length and principal point are called internal parameters of the camera (which often includes parameters related to distortion caused on the image by the camera lens, but are omitted here for simplicity).

s is a variable used for scaling to [u, v, 1]. These camera parameters can be calculated in advance using the technique disclosed in Non-Patent Document 4. Figure 5 shows an example of camera parameters that are actually input.

Here, f _x and f _y are focal lengths in pixel units that indicate the degree of zooming, so a camera with a large value is likely to have been zoomed greatly. Therefore, the camera classification unit 104 can automatically classify close cameras by checking f _x and f _y .

The number of cameras classified as closer cameras is not limited to one; several cameras (≦N cameras) with f _x , f _y larger than a certain value may all be classified as closer cameras, or f _x , f N1 units (≦N units) from the side with the largest _y may be classified as closer cameras. Furthermore, cameras with L% (L is an arbitrary constant from 0 to 100) of the total number of cameras may be classified as closer cameras, preferentially starting from the one with larger f _x and f _y . Furthermore, classification may be performed based on the camera position calculated from external parameters instead of f _x and f _y .

Alternatively, each camera may be classified by measuring the size of a 3D model of a subject created in the previous frame or a general-purpose 3D model prepared in advance, such as a goal post, reflected in each camera. For example, a 3D model prepared in advance can be placed in the area captured by all cameras, including the close-up camera, and the 3D model can be classified based on the size of the silhouette that appears when this 3D model is back-projected toward the camera. .

FIG. 6 is a functional block diagram showing the configuration of the main parts of the virtual viewpoint video generation device 1 according to the third embodiment of the present invention, and the same reference numerals as above represent the same or equivalent parts. The present embodiment is characterized in that the 3D model generation unit 102 includes a low-resolution voxel model generation unit 102a and a high-resolution voxel model generation unit 102b, and boundary determination is performed based on the low-resolution voxel model. It has characteristics.

The low-resolution voxel model generation unit 102a generates a voxel model for a coarse voxel grid with a unit voxel size of M. The unit voxel size M is a larger value than the unit voxel size L in the high-resolution voxel generation unit 102b, and is set to, for example, M=5 cm. In this embodiment, a voxel grid with unit voxel size M is placed in the target range of 3D model generation (for example, in the case of a sports video, the field where the sport is played, etc.), and a 3D model is formed using this voxel grid as the target. It is determined based on the visual volume intersection method.

Next, for the formed coarse voxel model, labeling processing is performed for each chunk of the coarse voxel model by repeating a process in which connected voxels are considered to be the same subject.

Next, define a 3D bounding box for the mass obtained in this way so as to include it, generate a voxel grid of unit voxel size L only inside this 3D bounding box, and do the same as above. Performs detailed voxel generation. Such a two-step voxel generation method is disclosed in Non-Patent Document 3. The boundary determination unit 103 performs boundary determination for each 3D bounding box generated by the low-resolution voxel model generation unit 102a.

In this way, if the judgment is made at the time of the low-resolution voxel model, the boundary judgment can be made in parallel without waiting for the results of high-resolution voxel model generation, which speeds up the processing. be able to. However, there is nothing in the present invention that prevents boundary determination based on a high-resolution voxel model.

In this way, if boundary determination is performed using a high-resolution voxel model, a more precise model shape can be obtained than when a low-resolution voxel model is used, so that boundary determination can be performed more accurately.

FIG. 7 is a functional block diagram showing the configuration of the main parts of the virtual viewpoint video generation device 1 according to the fourth embodiment of the present invention, and the same reference numerals as in the third embodiment represent the same or equivalent parts. Therefore, its explanation will be omitted.

In this embodiment, the 3D model production server 10 includes an occlusion information generation unit 105, rewrites the occlusion information based on the determination result by the boundary determination unit 103, and the boundary dependent mapping unit 202 of the rendering server 20 The feature is that texture mapping is performed based on occlusion information.

The occlusion information generation unit 105 generates occlusion information that classifies each vertex of the 3D model into a visible camera and an invisible camera. In an environment where N cameras exist as in this embodiment, N pieces of occlusion information are calculated for each vertex of the 3D model, and occlusion information is set to "1" for a visible camera, "0" for an invisible camera, etc. Information is recorded.

In a soccer competition scene, when two players overlap and player A covers player B in a certain camera image, it is necessary to map the texture so that player A's texture is not reflected in player B's 3D model. In such a case, the occlusion information regarding the camera is recorded as "invisible" for the vertices of the occluded portion of player B's 3D model. This occlusion information is calculated using, for example, a method using a depth map as disclosed in Patent Document 1.

The boundary-dependent mapping unit 202 selects two cameras (c ₁ , c ₂ ) near the virtual viewpoint according to the result of the boundary determination, and maps these camera images onto the polygon g of the 3D model. That is, if the 3D model to be mapped is not on the field of view boundary and all of it is within the field of view of the closer camera, two cameras near the virtual viewpoint are selected for the closer camera. On the other hand, if the 3D model to be mapped is on the field of view boundary or all of it is within the field of view of the pulling camera, two cameras near the virtual viewpoint will be selected for the pulling camera. be done.

In addition, in this embodiment, as preprocessing, the visibility of a certain polygon g is determined using occlusion information of the three vertices that make up the polygon (the three vertices are when the 3D model is formed of triangular polygons, and in reality depends on the number of vertices that make up each polygon).

For example, if the visibility determination flag of polygon g for camera c_1 is expressed as g_(c_1 ), if all three vertices of polygon g are visible, g_(c_1 ) is visible, and if any of the three vertices is invisible. If so, g_(c_1 ) is set as invisible. In this embodiment, texture mapping is performed as follows according to the result of polygon visibility determination for each camera.

Case 1: When visibility determination flags g _c1 and g _c2 of cameras c ₁ and c ₂ regarding polygon g are both “visible” Mapping by alpha blending is performed based on the following equation (3).

Here, texture _c1 (g) and texture _c2 (g) indicate camera image areas to which polygon g corresponds in cameras c ₁ and c ₂ , and texture (g) indicates a texture mapped to the polygon. The alpha blend ratio a is calculated according to the ratio of the distance (angle) between the virtual viewpoint p _v and each camera position p_(c_1 ), p_(c_2 ).

Case 2: When only one of visibility determination flags g _c1 and g _c2 is visible Rendering is performed using only the texture of the camera that makes polygon g visible. That is, in the above equation (3), the value of the ratio a corresponding to the visible camera texture_(c_i) is set to 1. Alternatively, the third camera c_3, which is next closest to the virtual viewpoint p_v, is referred to instead of the invisible one camera, and mapping is performed by alpha blending based on the above equation (3) as in case 1.

Case 3: When both visibility determination flags g _c1 and g _c2 are invisible At least one of the visibility determination flags indicates that another camera near the virtual viewpoint p _v (generally, one with a close angle) is to be selected. Repeatedly until , the texture at the reference pixel position of each camera image is mapped onto the polygon g by alpha blending based on the above equation (3), as in case 1.

Note that in the above embodiment, the number of nearby cameras to be initially referred to is two, but this may be changed by user settings. In that case, according to the number of initial reference cameras b, the above equation (1) is expanded to a linear sum of b cameras (total sum of weights is 1). Also, textures are not mapped for polygons that are invisible to all cameras.

Here, in this embodiment, the occlusion information is determined by the boundary determination unit 103 so that the boundary-dependent mapping unit 202 can map texture from an appropriate camera image based on occlusion and boundary conditions only by referring to the occlusion information. It can be rewritten according to the results.

Figure 8 is a diagram showing an example of rewriting occlusion information. Here, the visibility/invisibility (occlusion) of the closer camera is assigned to the least significant bit, and for each polygon that makes up the 3D model located on the boundary. is always rewritten to "0" indicating a shielded state, regardless of whether each vertex is visible or invisible (shielded).

In each of the above embodiments, the 3D model generation unit 102 generates the 3D model by the visual volume intersection method using silhouette images based on camera images of the approaching camera and the pulling camera. However, the present invention is not limited to this, and can also be applied to texture mapping to a 3D model generated based on a depth sensor (Non-Patent Document 6) or a 3D model generated based on a neural network (Non-Patent Document 7). Applicable.

In Non-Patent Document 6, the 3D shape of an object is acquired by using a depth sensor that can acquire the distance to the object, such as Kinect (registered trademark). By combining multiple Kinects (registered trademark), it is possible to obtain highly accurate shapes from 360 degrees. In addition, Kinect (registered trademark) comes with an RGB camera in addition to a depth sensor, so the RGB camera of Kinect (registered trademark) can be used for texture mapping. In other words, only the shape can be acquired with a depth sensor, and a normal camera can be used for texture mapping.

Non-Patent Document 7 discloses a method of estimating the shape of a 3D model using a neural network using a single camera or multiple cameras.

1...Virtual viewpoint video generation device, 10...3D model production server, 20...Rendering server, 101...Silhouette image acquisition section, 102...3D model generation section, 102a...Low resolution voxel model generation section, 102b...High resolution voxel Model generation section, 103... Boundary determination section, 104... Camera classification section, 105... Occlusion information generation section, 201... Virtual viewpoint selection section, 202... Boundary dependent mapping section, 203... Virtual viewpoint video output section

Claims (12)

In a virtual viewpoint video generation device that generates a virtual viewpoint video based on camera images taken of a subject from multiple viewpoints,
means for generating a 3D model based on each camera image of the approaching camera and the pulling camera;
means for determining whether or not the 3D model of the subject exists on the field of view boundary of the camera;
means for mapping a texture from each camera image to the 3D model,
The determining means includes a 3D model other than a 3D model in which all vertices of a 3D bounding box containing the 3D model are within the viewing angle range of the approaching camera or outside the viewing angle range of the approaching camera. It is determined that it exists above the
The virtual viewpoint image generation is characterized in that the mapping means maps a texture from the camera image of the pulling camera, including the area within the viewing angle of the closer camera, for a 3D model existing on the field angle boundary of the closer camera. Device. The means for generating the 3D model generates the 3D model by a visual volume intersection method based on camera images of the approaching camera and the pulling camera,
means for constructing a low resolution voxel model based on a silhouette of a subject;
means for constructing a high-resolution voxel model in the region of the low-resolution voxel model based on a silhouette of the subject;
The virtual viewpoint video generation device according to claim 1 , wherein the determining means determines for each 3D model whether or not the low-resolution voxel model exists on a field-of-view boundary of a close-up camera. . The virtual model according to claim 2 , wherein the means for generating the 3D model determines whether or not a 3D bounding box containing each low-resolution voxel model exists on a view angle boundary of a close camera. Viewpoint video generation device. the 3D model is a polygon model,
comprising means for generating occlusion information recording whether each polygon of the 3D model is visible or invisible from each camera;
The means for generating occlusion information rewrites occlusion information of a closer camera regarding polygons of the 3D model existing on the view angle boundary to invisible,
4. The virtual viewpoint video generation device according to claim 1 , wherein the mapping means maps a texture to each polygon from a camera that makes the polygon visible. comprising means for classifying each camera as a closer camera or a closer camera;
5. The determining means determines whether or not each 3D model of the object exists on a view angle boundary of a close-up camera based on the classification result. virtual viewpoint image generation device. 6. The virtual viewpoint video generation device according to claim 5 , wherein the classifying means classifies each camera as a close camera or a backward camera based on camera parameters. In a virtual viewpoint video generation method in which a computer generates a virtual viewpoint video based on camera images taken of a subject from multiple viewpoints,
Generates a 3D model of the subject based on camera images from the approaching camera and pulling camera,
Determine whether the 3D model of the subject exists on the field of view boundary of the camera,
Mapping the texture from each camera image to the 3D model,
When making the above determination, all vertices of the 3D bounding box containing the 3D model are within the field of view range of the camera or outside the field of view of the camera. It is determined that it exists above the
A virtual viewpoint characterized in that, when mapping the texture, for a 3D model that exists on the field angle boundary of the close camera, texture is mapped from the camera image of the pull camera including the area within the view angle of the close camera. Video generation method. The 3D model is generated by a visual volume intersection method based on camera images of a approaching camera and a pulling camera,
Build a low-resolution voxel model based on the subject's silhouette,
Build a high-resolution voxel model in the area of the low-resolution voxel model based on the silhouette of the subject,
8. The virtual viewpoint image generation method according to claim 7 , further comprising determining, for each 3D model, whether or not the low-resolution voxel model exists on the field-of-view boundary of the close-up camera. the 3D model is a polygon model,
Generate occlusion information that records whether each polygon of the 3D model is visible or invisible from each camera,
Rewrite the occlusion information of the closer camera regarding polygons of the 3D model existing on the field of view boundary to invisible,
9. The virtual viewpoint video generation method according to claim 7 , wherein when mapping the texture to each polygon, the texture is mapped from a camera that makes the polygon visible. In a virtual viewpoint video generation program that generates a virtual viewpoint video based on camera images taken of a subject from multiple viewpoints,
A procedure for generating a 3D model of a subject based on camera images from a close camera and a pull camera;
a step of determining whether the 3D model of the subject exists on the field of view boundary of the camera;
causing a computer to execute a procedure of mapping textures from each camera image to the 3D model,
In the step of determining, all 3D models other than 3D models in which all vertices of the 3D bounding box containing the 3D model are within the field of view range of the camera or outside the field of view of the camera are located at the field of view boundary of the camera. It is determined that it exists above the
In the mapping step, a texture is mapped from the camera image of the pulling camera to the 3D model existing on the field of view boundary of the closer camera, including the area within the view angle of the closer camera. program. The procedure for generating the 3D model includes generating a 3D model by a visual volume intersection method based on camera images of a closer camera and a puller camera,
steps for constructing a low-resolution voxel model based on a subject's silhouette;
constructing a high-resolution voxel model in the region of the low-resolution voxel model based on a silhouette of the subject;
11. The virtual viewpoint video generation program according to claim 10 , wherein the determining step determines, for each 3D model, whether or not the low-resolution voxel model exists on a view angle boundary of a close camera. . the 3D model is a polygon model,
A step of generating occlusion information recording whether each polygon of the 3D model is visible or invisible from each camera,
The procedure for generating the occlusion information includes rewriting the occlusion information of the closer camera regarding polygons of the 3D model existing on the view angle boundary to invisible;
12. The virtual viewpoint video generation program according to claim 10 , wherein in the mapping step, a texture is mapped to each polygon from a camera that makes the polygon visible.

Images