JP7536531B2 - Image processing device, image processing method, image processing system, and program. - Google Patents

Description

The present invention relates to coloring three-dimensional models of objects when generating virtual viewpoint images.

Recently, a technology that has attracted attention involves installing multiple real cameras (real cameras) in different positions, capturing images synchronously from multiple viewpoints, and using the multiple viewpoint images obtained by the capture to generate a virtual viewpoint image that represents the view from a non-existent virtual camera (virtual camera). With this virtual viewpoint image, for example, it is possible to view highlight scenes of a soccer or basketball game from various angles, providing the user with a higher sense of realism than with normal video.

In generating a virtual viewpoint image, first, images captured by multiple real cameras are collected on a server or the like, and a three-dimensional model of the object (subject) in the image is generated. This three-dimensional model is then colored (texture is applied) based on the specified virtual viewpoint, and two-dimensional transformation such as projective transformation is further performed to obtain a two-dimensional virtual viewpoint image. In the above generation process, a viewpoint-dependent texture mapping technique is often used to color the three-dimensional model (Non-Patent Document 1). In viewpoint-dependent texture mapping, when determining the color of a point of interest on an object, the colors of corresponding points on the images captured by multiple real cameras are blended with weighting. In this way, discontinuity in color changes caused by color differences between captured images corresponding to each viewpoint, errors in the three-dimensional model shape, camera parameters, etc. are suppressed, and a more natural texture expression is achieved.

P.E.Debevec, C.J.Taylor, and J.Malik: “Modeling and Rendering Architecture from Photographs: A Hybrid Geometry-and Image-Based Approach,” SIGG RAPH’96, pp.11-20, 1996.

In the viewpoint-dependent texture mapping, an image captured by a real camera located close to the virtual camera is used. Therefore, if there is a structure between the virtual camera and the object, the color of the structure is used to color the object, which can result in an unnatural appearance. For example, in a soccer game, if a real camera located close to the virtual camera captures a player through the goal net, a texture including the goal net is pasted onto the 3D model. A virtual viewpoint image with such coloring not only differs from the actual appearance, but also has the problem that the player's movements and details are difficult to understand. In addition, if a virtual camera is set in a position to view a player from inside the goal, the texture of the goal net, which should not be visible from the virtual camera, is pasted onto the 3D model of the player, resulting in an unnatural virtual viewpoint image.

The technology disclosed herein has been developed in consideration of the above problems, and its purpose is to properly reproduce the color of an object in a virtual viewpoint image even when a structure that blocks the object is present.

The image processing device according to the present disclosure is an image processing device that performs a rendering process using shape data representing a three-dimensional shape of an object to generate a virtual viewpoint image corresponding to a virtual viewpoint, and is equipped with a determination means for determining a processing method for coloring the object in the rendering process, and a rendering means for coloring the object according to the determined processing method, the shape data being generated based on a plurality of first captured images obtained by capturing images using a plurality of first imaging devices arranged around an imaging target area, a structure being present in the imaging target area, and the determination means determining whether color information for the coloring is to be obtained from captured images included in the plurality of first captured images, or from a second captured image obtained by capturing images using a second imaging device arranged in a position where the object can be captured without being obstructed by the structure, based on the positional relationship between the virtual viewpoint, the structure, and the object.

The technology disclosed herein makes it possible to properly reproduce the color of an object in a virtual viewpoint image, even when a structure that blocks the object is present.

FIG. 1 is a block diagram showing an example of the configuration of an image processing system. FIG. 2 is a diagram showing an example of the arrangement of imaging modules. FIG. 2 is a diagram illustrating an example of a hardware configuration of a server. FIG. 2 is a block diagram showing the functional configuration of a server. FIG. 13 is a diagram showing an example of virtual camera settings. FIG. 4 is a diagram showing an imaging range of a virtual camera. 11 is a flowchart showing the flow of a rendering process. 4A and 4B are diagrams for explaining the positional relationship between an object and each camera. 6A and 6B are diagrams showing an example of a result of a rendering process. FIG. 13 is a diagram showing the position of the virtual camera at which coloring using an image captured by an external camera is not possible.

The following describes an embodiment of the present invention with reference to the drawings. Note that the following embodiment does not limit the present invention, and not all of the combinations of features described in the present embodiment are necessarily essential to the solution of the present invention. Note that the same components are described with the same reference numerals.

[Embodiment 1]
(Basic system configuration)
FIG. 1 is a block diagram showing an example of the configuration of an image processing system according to this embodiment. The image processing system 100 includes imaging modules 110a to 110j and 120, a switching hub 115, a server 116, a database (DB) 117, a control device 118, and a display device 119. In each of the imaging modules 110a to 110j, cameras 111a to 111j and camera adapters 112a to 112j are connected by internal wiring (a to j are ten units: a, b, c, d, e, f, g, h, I, and j). Similarly, in the imaging module 120, a camera 121 and a camera adapter 122 are connected by internal wiring. Each of the imaging modules 110a to 110j and 120 transmits data via a network cable. A switching hub (hereinafter, referred to as "HUB") 115 is a device that performs routing between each network device. The imaging modules 110a to 110j and 120 are connected to the HUB 115 by network cables 113a to 113k. Similarly, the server 116 and the control device 118 are connected to the HUB 115 by network cables 113l and 113n, respectively. The server 116 and the Db 117 are connected by a network cable 113m. The control device 118 and the display device 119 are connected by a video cable 114. The cameras 111a to 111j and 121 capture images in high-precision synchronization with each other based on a synchronization signal. In this embodiment, the imaging modules 110a to 110j are installed so as to surround the soccer field, which is the imaging target area, as shown in FIG. 2(a). FIG. 2(a) is a diagram of the field as viewed from directly above, and the imaging modules 110a to 110j are installed at a certain constant and the same height from the ground. On the other hand, as shown in FIG. 2B, the imaging module 120 is installed from inside the soccer goal, which is a structure whose position is fixed, toward the center of the field (i.e., the direction in which objects such as players are present). Note that FIG. 2B is an enlarged view of the part surrounded by the dashed line in FIG. 2A. In the following description, the imaging modules 110a to 110j installed around the periphery of the field will be referred to as "external cameras", and the imaging module 120 installed inside the soccer goal will be referred to as "internal camera". Note that in this embodiment, the number of external cameras 110 is 10, and the number of internal cameras 120 is 1, but this is merely an example and is not limited to this.

The server 116 processes the captured image sent from the external camera 110, generates a three-dimensional model of the object, and colors the generated three-dimensional model (also called "texture pasting" or "texture mapping"). In this embodiment, the object for which the three-dimensional model is generated is a moving object such as a player or a ball. Here, the "three-dimensional model" means shape data that represents the three-dimensional shape of the object. The server 116 also has a time server function that generates a time synchronization signal for time synchronization of the present system. The database (hereinafter referred to as "DB") 117 accumulates data such as image data processed by the server 116 and the generated three-dimensional model, and provides the accumulated data to the server 116. The control device 118 is an information processing device that controls the external camera 110, the internal camera 120, and the server 116. The control device 118 is also used to set the virtual camera (virtual viewpoint). The display device 119 displays a setting user interface screen (UI screen) for the user to specify a virtual viewpoint in the control device 118, and displays a UI screen for viewing the generated virtual viewpoint image.

(Operation of Image Processing System)
Next, the general operation of the image processing system 100 will be described. The captured image obtained by the external camera 110a is subjected to predetermined image processing such as foreground/background separation, and then transmitted to the external camera 110b. Similarly, the external camera 110b transmits the captured image obtained by the external camera 110b together with the captured image received from the external camera 110a to the external camera 110c. By continuing such an operation, 10 sets of captured images (including a foreground image) are transmitted from the external camera 110j to the HUB 115, and then transmitted to the server 116. In addition, the captured image obtained by the internal camera 120 is subjected to predetermined image processing such as foreground/background separation, and then transmitted to the server 116 via the HUB 115.

The server 116 generates a three-dimensional model of the object and performs rendering processing based on captured image data from different viewpoints (hereinafter referred to as "multiple viewpoint image data") acquired from the external camera 110j and the internal camera 120. The server 116 also transmits a time and a synchronization signal to each of the external cameras 110a to 110j and the internal camera 120. Upon receiving the time and synchronization signal, each of the external cameras 110a to 110j and the internal camera 120 captures images using the received time and synchronization signal, and performs frame synchronization of the captured images. In other words, each of the external cameras 110 and the internal camera 120 captures images on a frame-by-frame basis, synchronized to the same time.

(Server hardware configuration)
Next, the hardware configuration of the server 116 that is responsible for generating the virtual viewpoint image will be described with reference to Fig. 3. The DB 117, the control device 118, the camera adapters 112 and 122 each basically have the same hardware configuration. The server 116 has a CPU 201, a ROM 202, a RAM 203, an auxiliary storage device 204, a communication I/F 205, and a bus 206.

The CPU 201 uses the programs and data stored in the ROM 202 and the RAM 203 to control the entire server 116 and realizes each functional unit shown in FIG. 4 described later. The server 116 may have one or more dedicated hardware different from the CPU 201, and at least a part of the processing by the CPU 201 may be executed by the dedicated hardware. Examples of the dedicated hardware include an ASIC (application specific integrated circuit), an FPGA (field programmable gate array), and a DSP (digital signal processor). The ROM 202 stores programs that do not require modification. The RAM 203 temporarily stores programs and data provided from the auxiliary storage device 204, and data provided from the outside via the communication I/F 205. The auxiliary storage device 204 is composed of, for example, an HDD or SSD, and stores various data and programs such as input data such as image data and audio data, tables referenced in various processes described later, and various application programs. The communication I/F 205 is used for communication with an external device. For example, when the server 116 is connected to an external device by wire, a communication cable is connected to the communication I/F 205. When the server 116 has a function of wirelessly communicating with an external device, the communication I/F 205 is equipped with an antenna. The bus 206 connects the above-mentioned components to transmit data and signals. In this embodiment, the server 116 generates a virtual viewpoint image based on information (virtual viewpoint information) indicating the position and orientation of a virtual camera (virtual viewpoint) set by the control device 118, and the user views the image on the display device 119. However, the system configuration is not limited to this, and for example, a single information processing device having both a user interface function for inputting virtual viewpoint information and a user interface function for viewing a virtual viewpoint image may be incorporated.

(Server Functional Configuration)
4 is a block diagram showing the functional configuration of the server 116 as an image processing device that generates a three-dimensional model of an object and performs rendering processing based on virtual viewpoint information according to this embodiment. The server 116 has a three-dimensional model generation unit 401, a processing method determination unit 402, and a rendering unit 403. The server 116 realizes the functions of each unit shown in FIG. 4 by the CPU 201 executing a predetermined program stored in the RAM 203 or the ROM 202. In this embodiment, the server 116 is configured to realize the functions of each unit described above by one server 116, but the functions of each unit may be realized in a distributed manner by a plurality of servers. For example, the server may be divided into a server that performs the function of the three-dimensional model generation unit 401 and a server that performs two functions, that is, the processing method determination unit 402 and the rendering unit 403.

The multiple viewpoint image data (images captured corresponding to multiple viewpoints captured by synchronous shooting) input to the server 116 is first input to the three-dimensional model generation unit 401.

The three-dimensional model generating unit 401 first extracts silhouettes of objects from each captured image with different viewpoints. Then, using the obtained silhouette image (foreground image) and the camera parameters of each external camera 110 input separately, a three-dimensional model is generated as shape data representing the three-dimensional shape of the object by a volume intersection method or the like. The camera parameters include external parameters, internal parameters, and distortion parameters. The internal parameters refer to parameters that represent the focal length and the optical center called the principal point. The external parameters are parameters that represent the position and line of sight of the camera, and the rotation angle around the line of sight. The distortion parameters are coefficients that represent radial image distortion caused by differences in the refractive index of the lens, and circumferential distortion caused by the lens and the image plane not being parallel. In this embodiment, a soccer game in which multiple players move around on a vast field is assumed as the captured scene. In other words, the three-dimensional model generated by the three-dimensional model generating unit 401 is data that represents the three-dimensional shape of each object, such as the player and the ball. In the three-dimensional model of each object, the three-dimensional shape is represented, for example, as a collection (block) of unit cubes called voxels. Note that the three-dimensional shape may be represented in other formats, such as a point cloud format or a polygon format, instead of the voxel format.

The processing method determination unit 402 determines the processing method at the time of rendering based on the virtual viewpoint information indicating the position and attitude of the virtual camera, the position information of the structures existing in the captured scene, and the position information of the objects such as players. Specifically, it analyzes the relative positions of the virtual camera, the structures, and the objects, and determines whether to use the color information (texture information) of the image captured by the external camera 110 or the color information of the image captured by the internal camera 120 for coloring the three-dimensional model. "Coloring the three-dimensional model" can be rephrased as a process of selecting pixels of the captured image based on the three-dimensional positions of the voxels that constitute the three-dimensional model and determining the pixel values in the virtual viewpoint image. Here, the virtual viewpoint information is generated by the user setting a virtual camera in the captured space using a joystick or the like in the control device 118, for example, and is input from the control device 118 to the server 116. The structures existing in the captured scene are stationary objects that are installed in the captured space and do not move unless an external force is applied, and in this embodiment, soccer goals including a goal net are assumed. Since the positions of the structures are fixed, it is sufficient to obtain their position coordinates in advance at a timing such as before the start of the match and store them in the auxiliary storage device 204 or the like. The object position information is generated when the above-mentioned 3D model is generated and is obtained from the 3D model generation unit 401. The position information of the structure or object includes position coordinates (world coordinates) that specify the 3D position in the imaging space and information that specifies the size.

The rendering unit 403, in accordance with the rendering processing method determined by the processing method determination unit 402, applies color (applies texture) to the three-dimensional model generated by the three-dimensional model generation unit 401 based on the virtual viewpoint information, and generates an image that represents the view from the virtual viewpoint.

(Details of the rendering process)
Next, the rendering process in this embodiment will be described in detail with a specific example. FIG. 5 is a diagram showing a camera arrangement in FIG. 2, in which the virtual camera 303 is set at a position such that the goalkeeper 302 is captured over the soccer goal 301, as viewed from directly above the field. FIG. 6 shows the field of view (angle of view) from the virtual camera 303. As is clear from FIG. 6, when the goalkeeper 302 is viewed from the virtual camera 303, the goal net 301a of the soccer goal 301 is present so as to block the goalkeeper 302. Assuming such a case, the operation of the processing method determination unit 402 and the rendering unit 403 of the server 116 will be described with reference to the flowchart shown in FIG. 7. Note that the series of processes shown in the flowchart in FIG. 7 are realized by the CPU 201 expanding a predetermined control program stored in the auxiliary storage device 204 into the RAM 203 and executing the program when the generation of the three-dimensional model of the object is completed in the three-dimensional model generation unit 401. 7, the processes of S707, S708, S710, and S711 are performed by the rendering unit 403, and the remaining steps are performed by the processing method determination unit 402. In the following description, the symbol "S" represents a step.

In S701, an object of interest to be subjected to the rendering process is determined from one or more objects present in the field of view (angle of view) of the virtual camera specified by the virtual viewpoint information input from the control device 118.

In the next step S702, a nearby external camera within a certain range of the virtual camera is determined from the external cameras 110a to 110j. The position of the virtual camera is specified by the virtual viewpoint information described above, and the position of each of the external cameras 110a to 110j is specified by the camera parameters described above. Here, it is assumed that one external camera located closest to the set virtual camera is selected. Specifically, one external camera whose field of view (angle of view) overlaps the field of view (angle of view) of the virtual camera at the maximum rate is selected from among the ten external cameras 110a to 110j. For example, in the case of the virtual camera 303 set at the position shown in FIG. 5, the external camera 110f is selected. FIG. 8(a) is a diagram showing the positional relationship of the goalkeeper 302, soccer goal 301, external camera 110f, internal camera 120, and virtual camera 303 in the camera arrangement shown in FIG. 5, as viewed from the longitudinal side of the field. FIG. 8B is a diagram showing the positional relationship when the virtual camera 303' is set at a position inside the goal net 301a on an extension of the optical axis direction of the virtual camera 303. In both cases of FIG. 8A and FIG. 8B, the fields of view of the virtual cameras 303 and 303' largely overlap with the field of view of the external camera 110f. Once the external camera closest to the virtual camera is determined in this manner, the process proceeds to S703. Note that in this embodiment, only one nearest external camera is determined, but this is not limited to this. For example, multiple external cameras within a certain range of the virtual camera may be determined, and the following process may be performed by weighting each of them according to their distance from the virtual camera.

In S703, a voxel of interest to be colored is determined from among the voxels constituting the three-dimensional model of the target object. The voxel of interest may be a group of multiple voxels (a group of target voxels). In the next step S704, it is determined whether or not a structure is present at a position that blocks the voxel of interest determined in S703 when the voxel of interest is viewed from the external camera determined in S702. When making the determination, the position information of the structure located at a predetermined position and the position information of the external camera determined in S702 are read from the RAM 203 or the auxiliary storage device 204 and used. For example, in the case of the above-mentioned Figures 8(a) and (b), it is determined whether or not color information of the corresponding pixel position in the image captured by the external camera 110f can be used to color the voxel of interest in the three-dimensional model of the goalkeeper 302. In other words, in this step, it is determined whether a structure at a fixed position (here, the goal net 301a) covers even a part of the target object (goalkeeper 302). Now, in both cases of FIG. 8(a) and (b), the outer camera closest to the virtual camera is 110f. Therefore, as shown in the figure, since the soccer goal 301 exists between the goalkeeper 302 and the outer camera 110f, the goal net 301a is reflected in front of the goalkeeper 302. In other words, in both cases of FIG. 8(a) and (b), the goalkeeper 302 is in a state where it is blocked by the goal net 301a. Therefore, in the cases of FIG. 8(a) and (b), it is determined that a structure (hereinafter, referred to as "blocking object") that blocks the target voxel exists. If the result of the determination is that a blocking object exists for the target voxel, proceed to S705, and if not, proceed to S709.

In S705, when it is determined in S704 that an obstruction exists, it is determined whether color information for coloring the target voxel can be obtained from the image captured by the internal camera installed inside the obstruction (here, the goal net 301a). Here, "color information can be obtained" means that the original color of the target object is appropriately obtained from the corresponding pixel position in the captured image without other objects, including the obstruction, being reflected. In the cases of FIG. 8(a) and (b) above, there are no obstructing objects, including other players, between the internal camera 120 and the goalkeeper 302. Therefore, it is determined that color information for coloring the target voxel can be obtained from the image captured by the internal camera 120. If color information can be obtained from the image captured by the internal camera 120 as a result of the determination, proceed to S706, and if not, proceed to S711. In S706, it is determined whether color information from the image captured by the internal camera is used to color the target voxel. This determination may be based on user selection via a UI screen (not shown), or may be determined automatically according to certain conditions. One possible method for determining the conditions is based on the attributes of the object of interest (information indicating the object type, such as player, referee, or ball). If color information from the image captured by the internal camera is used for coloring, the process proceeds to S707, and if not, the process proceeds to S708. Note that when it is determined that color information from the image captured by the internal camera can be acquired (Yes in S705), the process may skip the determination in S706 and proceed to S707.

In S707, the voxel of interest is colored using color information obtained from the corresponding pixel position in the image captured by the internal camera. In S708, the voxel of interest is colored using color information obtained from the corresponding pixel position in the image captured by one of the external cameras (for example, the external camera closest to the virtual camera determined in S702). After the rendering process in S707 or S708 is completed, the process proceeds to S712.

In S709, if it is determined in S704 that no obstruction exists, it is determined whether color information for coloring the target voxel can be obtained from an image captured by any of the external cameras (including the external camera determined in S702). Since there are multiple players and referees on the field in addition to structures such as soccer goals, even if it is determined in S704 that no obstruction exists, there may be other players between the target object and the object. Taking this into consideration, it is determined whether color information to be used for coloring the target voxel can be obtained from an image captured by an external camera. Specifically, first, the positional relationship between the virtual camera, the external camera, and each object is calculated based on the position coordinates of the virtual camera, the position coordinates of all objects present within the field of view (angle of view) of the virtual camera, and the camera parameters of all the external cameras. Then, it is determined whether there is an image captured by an external camera from which color information of the target object (here, the goalkeeper's original color) can be obtained. If the result of the determination is that the original color information of the target object can be obtained from the image captured by any of the external cameras 110a to 110j, the process proceeds to S710; if not, the process proceeds to S711.

In S710, coloring is performed on the voxel of interest by acquiring color information from the corresponding pixel position of the image captured by the outer camera for which it was determined in S709 that color information can be acquired. In addition, in S711, if color information cannot be acquired from the image captured by any of the real cameras (all outer cameras and inner camera), the voxel of interest is colored by correction (interpolation process) using color information from the surrounding area of the voxel of interest. Alternatively, the voxel of interest may not be colored at all and may be processed as a transparent voxel with no color information. Then, after the coloring process of S710 or S711 is completed, the process proceeds to S712.

In S712, it is determined whether or not coloring has been completed for all voxels that make up the target object (whether or not there are any uncolored voxels). If coloring has been completed for all voxels, the process proceeds to S713, and if there are any unprocessed voxels, the process returns to S703 to determine the next target voxel and continue processing.

In S713, it is determined whether rendering has been completed for all objects present in the field of view (angle of view) of the virtual camera. If it has been completed, this process ends. On the other hand, if there are unprocessed objects, the process returns to S701 to determine the next object of interest and continue processing.

The above is the content of the rendering process according to this embodiment. Figures 9(a) and (b) respectively show examples of the results of the rendering process of this embodiment when the goalkeeper 302 in Figures 5 and 6 described above is used as the target object. Figure 9(a) shows the result when the rendering process is finally performed in S708, in which the goal net 304 is reflected in the goalkeeper 302. On the other hand, Figure 9(b) shows the result when the rendering process is finally performed in S707, in which the goal net 304 is not reflected in the goalkeeper 302, and the goalkeeper 302 is colored in its original color. This rendering process results in a virtual viewpoint image in which the three-dimensional model is appropriately colored.

<Modification>
In the flowchart of FIG. 7, color information for coloring is acquired from the image captured by the internal camera only when there is an obstruction. In other words, it is assumed that when there is no obstruction when the target object is viewed from the virtual camera, color information is preferentially acquired from the image captured by the external camera and the three-dimensional model is colored. However, the present invention is not limited to such a configuration. For example, even when there is no obstruction to the target object, color information may be acquired from the image captured by the internal camera and the three-dimensional model may be colored. Furthermore, the image captured by the internal camera may be used together with the image captured by the external camera when generating the three-dimensional model.

In addition, among the areas where the virtual camera (virtual viewpoint) can be set, the areas that are likely to be blocked by structures may be calculated in advance, the information may be stored, and the information may be used to determine the presence or absence of a blocking object. FIG. 10 is a diagram in which an area indicating the position of the virtual camera where appropriate coloring by the image captured by the outer camera is not possible is superimposed on the schematic diagram shown in FIG. 5 described above. In FIG. 10, if the virtual camera is set within the range of the hatched area 1000, the goal net will always be reflected in the images captured by the outer cameras 110e, 110f, and 110g. This means that when the goalkeeper 302 is viewed from within the hatched area 1000, the goal net will always be in the way, and the original color information of the goalkeeper 302 cannot be obtained from the images captured by the outer cameras 110e, 110f, and 110g. In other words, if a virtual camera is set within the hatched area 1000, if you want to use the original color of the goalkeeper 302 to color the three-dimensional model, you will inevitably use the image captured by the inner camera 120. Therefore, when making the judgment in S704 of the flowchart in FIG. 7 described above, if the virtual camera specified by the virtual viewpoint information is set within a specific area such as the above-mentioned shaded area 1000, it may be determined that there is an obstruction and the process may proceed to S705.

In this embodiment, the example of a net-like structure being an obstruction has been described with the generation of a virtual viewpoint image of a shooting scene in a soccer game as viewed from the back side of the goalkeeper in mind, but the present invention is not limited to this. First, the above embodiment can be applied as is to sports in which a goal net is used like soccer, such as handball, futsal, and water polo. This embodiment can also be applied to sports in which opponents are on either side of a net, such as volleyball, tennis, badminton, and table tennis. In this case, it is sufficient to install internal cameras on both sides of the net to capture the player's side. In addition to the goal net in basketball, it is also possible to treat acrylic goal boards as obstructions. Furthermore, it is also possible to apply the present invention to sports in which a protective net is used, such as baseball and throwing events in track and field. In this case, it is sufficient to install internal cameras inside the playing area (closer to the player than the protective net).

As described above, according to this embodiment, when coloring a three-dimensional model of an object, the object can be appropriately colored even if a structure is present in a position that occludes the object.

Other Examples
The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

Claims (12)

An image processing device that performs rendering processing using shape data representing a three-dimensional shape of an object to generate a virtual viewpoint image corresponding to a virtual viewpoint,
a determining means for determining a processing method for coloring the object in the rendering process;
a rendering means for coloring the object in accordance with the determined processing method;
Equipped with
the shape data is generated based on a plurality of first captured images obtained by performing imaging using a plurality of first imaging devices arranged around an imaging target area;
a structure is present in the imaging target area,
The determining means is
determining whether color information for the coloring is to be acquired from a captured image included in the plurality of first captured images, or from a second captured image acquired by capturing an image with a second imaging device disposed at a position where the object can be captured without being obstructed by the structure, based on a positional relationship between the virtual viewpoint, the structure, and the object;
13. An image processing device comprising: The image processing device according to claim 1, characterized in that the determining means determines that, when the object is obstructed by the structure when viewed from a specific imaging device among the plurality of first imaging devices that is located within a certain range from the virtual viewpoint, the color information for the coloring is obtained from the second captured image. The image processing device according to claim 2, characterized in that the specific imaging device is an imaging device that has a field of view closest to the field of view of the virtual viewpoint among the plurality of first captured images. The image processing device according to claim 1, characterized in that the determining means determines that, when the virtual viewpoint is between the object and the structure, the color information for the coloring is obtained from a second captured image. The image processing device according to claim 1, characterized in that the determination means determines that, if the structure is present in the field of view of the virtual viewpoint and is between the object and the virtual viewpoint, the color information for the coloring is obtained from a second captured image. The determining means is
displaying a user interface screen for allowing a user to select whether or not to acquire color information for the coloring from the second captured image, when color information for the coloring can be acquired from the second captured image;
determining whether color information for the coloring is to be acquired from a captured image included in the plurality of first captured images or from the second captured image based on a user selection via the user interface screen;
6. The image processing device according to claim 1, The image processing device according to any one of claims 1 to 6, characterized in that the structure has a net-like shape. a means for storing information on a specific area in which the object is likely to be blocked by the structure, among the areas in which the virtual viewpoint can be set;
When the virtual viewpoint is set within the specific area indicated by the stored information, the determination means determines that color information for the coloring is to be acquired from a second captured image.
2. The image processing device according to claim 1, the shape data represents a three-dimensional shape of the object in voxels or a point cloud;
The determining means determines a processing method for the coloring in units of the voxels or the point clouds,
The rendering means performs the coloring in units of the voxels or points.
9. The image processing device according to claim 1, The image processing device according to any one of claims 1 to 9, characterized in that the number of the first imaging devices is greater than the number of the second imaging devices. An image processing method for generating a virtual viewpoint image according to a virtual viewpoint by performing a rendering process using shape data representing a three-dimensional shape of an object, comprising:
a determining step of determining a processing method for coloring the object in the rendering process;
a rendering step of coloring said object in accordance with the determined processing method;
having
the shape data is generated based on a plurality of first captured images obtained by performing imaging using a plurality of first imaging devices arranged around an imaging target area;
A structure is present in the imaging target area,
In the determining step, whether color information for the coloring is to be obtained from a captured image included in the plurality of first captured images or from a second captured image obtained by capturing an image with a second imaging device disposed at a position where the object can be captured without being obstructed by the structure is determined based on a positional relationship between the virtual viewpoint, the structure, and the object.
13. An image processing method comprising: A program for causing a computer to function as an image processing device according to any one of claims 1 to 10.

Images