JP2019149112A - Composition device, method, and program - Google Patents

Abstract

To provide a composition device of free viewpoint images capable of performing high-speed processing and coping with occlusion problems.SOLUTION: A composition device 10 includes: a calculation unit 3 that finds likelihood images of an area of a photographed subject from viewpoint images of a multi-viewpoint image; a backprojection unit 6 that obtains backprojection data obtained by back projecting the likelihood images onto a plurality of respective backprojection planes arranged in a three-dimensional space; and a drawing unit 7 that combines free viewpoint images of the subject by rendering the subject by drawing and back projecting texture of all or part of viewpoint images of the multi-viewpoint image for the backprojection data to the plurality of backprojection planes constituting the backprojection data in order.SELECTED DRAWING: Figure 2

Description

  The present invention relates to a free viewpoint image composition apparatus, method, and program capable of high-speed processing and capable of coping with the occlusion problem.

  2. Description of the Related Art Conventionally, there has been proposed a technique for generating a video from a free viewpoint that is not captured by a camera (hereinafter referred to as a free viewpoint video) for a sports scene or the like. This technology enables video viewing from various viewpoints by synthesizing videos from virtual viewpoints that are not arranged based on videos taken by multiple cameras and displaying the results on the screen. It is what.

  Here, among free viewpoint video synthesis technologies, there is an existing technology that synthesizes a high-quality free viewpoint video by generating a 3DCG model of a subject using a principle called a view volume intersection method. In this full model method, the outline information of the subject obtained from multiple cameras is back-projected into a three-dimensional space and described in a large number of point cloud data to accurately reproduce the outline of the subject. Yes (Depending on the technique, polygon data may be created by a technique called marching cubes, but the point of using enormous point cloud data in the middle is unchanged). Using the 3DCG model of the subject generated in advance, the position of the virtual viewpoint is determined and rendered on the display, thereby generating a free viewpoint video.

Japanese Patent Application No. 2017-167472

  In contrast to the full model method, the present applicant has proposed a technique for realizing a visual volume intersection method using a virtual plane group without using point cloud data (Patent Document 1). This technology eliminates the need to access an enormous amount of data, and "can perform from the subject model generation to composite video display at once (compositing without discharging intermediate data)", so point cloud data There is an advantage that free viewpoint video composition can be performed at a much higher speed than the method using the. Also, in this patent document, the memory of a GPU (Graphics Processing Unit) in an actual computer is adaptively changed by changing the density and coordinates of a virtual plane group according to the coordinates of a virtual viewpoint selected by the user. A method for image synthesis suitable for the region size has also been proposed.

  However, there is room for improvement in the method of Patent Document 1. Specifically, in Patent Document 1, since the depth of the subject viewed from a plurality of cameras is not explicitly calculated, the context of the subject in the camera image cannot be considered, resulting in an unnatural video composition. There was a case.

  More specifically, as shown in a schematic example in FIG. 1, for example, when viewed from a certain camera CA, two objects (objects) (for example, fields) on the xy plane in the world coordinate system indicated by the xyz coordinate system. Two people such as the above athletes are assumed, but here, as a schematic example, two “cylinders” are shown side by side, and one (gray cylinder CLB) is the other Consider a scene that is occluded by (white cylinder CLF) (the occlusion in the camera image of such objects is hereinafter referred to as occlusion). As shown in the figure, when viewed from the camera CA (and the camera CV corresponding to the virtual viewpoint), the + x direction is the front side and the -x direction is the rear side. In this case, the image (texture) PA taken by the camera CA should be pasted (mapped) to the unshielded object (white cylinder CLF) on the near side, and the object behind it that cannot be seen by occlusion It should not be mapped to (gray cylinder CLB). However, as described above, Patent Document 1 does not calculate the subject's front-rear relationship (in order to achieve high-speed calculation), so that the camera image is pasted to all subjects regardless of the presence or absence of occlusion. For this reason, there were cases in which an unnatural composite image was obtained.

  That is, when the virtual viewpoint used to obtain the composite image is at a position such as the camera CV in FIG. 1 (and therefore the camera CV is not for obtaining a live-action image), the camera corresponding to the virtual viewpoint. Assuming that there is an actual camera at the position, the composite video in CV is the state where only the gray cylinder CLB that is the object behind is taken like the image PV, It is desired that the gray cylinder CLB is not shielded by the white cylinder CLF. However, if an image of the camera CA is used as a live-action image for generating the composite image, a state in which occlusion occurs in the gray cylinder CLB by the white cylinder CLF in the foreground (as shown in the image PA) The image PVA in a state equivalent to that seen from the PV is synthesized, and the image PV that should originally be synthesized is not synthesized.

  The present invention has been made in view of the above-described problems of the prior art, and an object of the present invention is to provide a synthesis apparatus, method, and program capable of dealing with the occlusion problem by high-speed processing in accordance with the framework of Patent Document 1.

  In order to achieve the above object, the present invention is a synthesis device, which calculates a likelihood image of a target area captured from each viewpoint image of a multi-viewpoint image, and three-dimensionally each of the likelihood images. A backprojection unit that obtains backprojection data backprojected to a plurality of backprojection planes arranged in space, and textures of all or part of the viewpoint images of the multi-viewpoint image with respect to the backprojection data, A rendering unit that synthesizes a free viewpoint image of the object by rendering the object by rendering the object by back projecting sequentially onto the plurality of back projection planes constituting backprojection data; To do. Further, the present invention is characterized by being a method and a program corresponding to the apparatus.

  According to the present invention, the textures of all or some of the viewpoint images of the multi-viewpoint image are backprojected in order onto the backprojection plane, and are drawn in order to synthesize the free viewpoint image with high-speed processing. It becomes possible to do.

It is a figure which shows the model example of having room for improvement regarding the synthetic | combination image | video by the method of patent document 1 depending on the image | video used in the case of composition | combination, and the aspect of the positional relationship of object. It is a functional block diagram of the synthesizing | combining apparatus which concerns on one Embodiment. It is a schematic diagram which shows that matching of the coordinate of a camera image and the coordinate of a world coordinate system is attained by camera calibration. It is a figure which shows the schematic example of the background difference method applied by the extraction part. It is a figure which shows the example of a model which concerns on one Embodiment of the surface group and order which are respectively set by a surface group setting part and an order setting part. It is a figure which shows the process of a back projection part typically. It is a flowchart of operation | movement of the drawing part which concerns on one Embodiment, a reprojection part, and the provision part. It is a figure which shows the model example of the process in which a reprojection part acquires a reprojection area | region. It is a figure which shows the model example of the drawing by a drawing part.

FIG. 2 is a functional block diagram of a synthesis device according to an embodiment. As illustrated, the synthesis device 10 includes a calibration unit 1, an extraction unit 2, a calculation unit 3, a surface group setting unit 4, a sequence setting unit 5, a back projection unit 6, a drawing unit 7, a reprojection unit 8, and a grant unit 9. Prepare. As illustrated, the back projection unit 6, the drawing unit 7, the reprojection unit 8, and the assigning unit 9 constitute a rendering unit 20. As an overall operation, the synthesizer 10 receives a plurality of camera images V _{c, t} (u, v) as multi-viewpoint images as input, and the user receives the camera images V _{c, t} (u, v). The composite video SY _t (u, v) at the virtual viewpoint CV designated by the input or the like (that is, the free viewpoint CV) is output.

As a main additional configuration (or configuration for performing additional processing) with respect to the configuration shown in Patent Document 1, the composition device 10 in FIG. 2 includes a sequence setting unit 5, a drawing unit 7, a reprojection unit 8, and a grant unit 9. Is. Each of the additional processing by the functional unit of the additional configuration and the processing in each of the other functional units linked to the additional processing obtains a composite video SY _t (u, v) in consideration of occlusion in the present invention. Contribute to making this possible.

Here, notations relating to video data and the like used in the description of the present invention will be described. The input multi-view video “V _{c, t} (u, v)” means time t (t = 1, 2, 3,...) Of a plurality of N cameras c (c = 1, 2,..., N). The video V is expressed as a function of variables (and indexes) c, t, u, v as representing the pixel value at the pixel position (u, v) (where u, v are integers). Similarly, the synthesized video “SY _t (u, v)” to be output also represents the synthesized video SY as a function, representing the pixel value at the pixel position (u, v) at the time t. . Similarly, each data appearing in the following description is also an index that distinguishes camera c, time t, and virtual plane k (described later), with the uppercase part representing the data function name, followed by the lowercase part. Subsequent (u, v) and (i, j) are indexes for distinguishing positions. Some of the indexes may not exist.

The synthesizing device 10 generates a multi-viewpoint video V _{c, t} (u, v) at each time t = 1, 2, 3,... On the video (if the time t is fixed, the multi-viewpoint image V _{c, t} (u, v )), A composite image SY _t (u, v) (or a composite image SY _t (u, v) when time t is fixed) is output, but the composition processing is performed at any time t. It is common. Therefore, in the following description, it is assumed that the process is an arbitrary time t in the input video V _{c, t} (u, v), and in some cases, the time t is not particularly mentioned. The outline of the processing of each part of the synthesizing apparatus 10 in FIG. 1 is as follows.

First, the multi-viewpoint image V _{c, t} (u, v) as an input to the synthesizing device 10 is input to the calibration unit 1, the extraction unit 2, and the drawing unit 7. Here, the images of the input multi-viewpoint images V _{c, t} (u, v) in each camera c are subjected to time synchronization between different camera images (that is, the time t is different). It is assumed that the same object is taken from the shooting position of the camera c (assuming that it is the synchronized common time between the camera images).

<Calibration section 1>
The calibration unit 1 performs so-called camera calibration. The multi-viewpoint image (the image of each camera c) V _{c, t} (u, v) is used as an input, and the ground (field) of the real space for each camera c. Associating coordinates (x, y, z) with camera image V _{c, t} (u, v) and backprojecting external parameters of the obtained calibration data (ie, camera parameters) Output to the unit 6 and the drawing unit 7. Each of the parts 2, 6, and 7 can be processed by using the obtained calibration data. (Note that, as is well known, an internal parameter for eliminating lens distortion can be obtained by calibration, but each part of the synthesizing apparatus 10 may use distortion-corrected data using the internal parameter. (The flow of data exchange concerning is omitted in FIG. 2.)

If the calibration operation by the calibration unit 1 is based on a fixed camera, it is only necessary to perform the operation once at a certain time t in each camera c, and at subsequent times t + 1, t + 2,. May use calibration data already obtained at time t. Further, when the calibration data is already given to the multi-viewpoint image V _{c, t} (u, v), the calibration unit 1 may be omitted. (In this case, it can be considered that the calibration unit 1 exists as an external configuration of the synthesis apparatus 10).

3, the image V _c of the camera _{c, t} (u, v) coordinates (u, v) of _c and the world coordinate system a point (x, y, z) is correspondence between made possible by the camera calibration An example of this will be shown. The association is possible in the form that a point (x, y, z) exists on a straight line L (u, v) passing through the camera center of the camera c and the coordinates (u, v) _c . The world coordinate system (x, y, z) by the camera calibration is common to all cameras c.

For the camera calibration in the calibration unit 1, any existing method may be used. A marker that can detect, for example, a feature point or a line segment automatically and / or manually is a known position in the world coordinates (x, y, z). Then, the correspondence between the feature point coordinates and the line segment related coordinates in the coordinates (u, v) _c of the camera c is obtained, and the camera parameters may be acquired.

<Extractor 2>
The extraction unit 2 classifies the background and foreground in the image using the background difference method that is an existing method for the image V _{c, t} (u, v) of each camera c, and expresses the classification. Alternatively, a mask image M _{c, t} (u, v) having the pixel value as the likelihood of the foreground (which may be given in grayscale gradation or the like) is obtained and output to the calculation unit 3. Here, the classified foreground is an area of an object (for example, a person) photographed in the image V _{c, t} (u, v).

  FIG. 4 is a diagram illustrating a schematic example of extraction processing in the extraction unit 2, and by applying the background difference method to an original image [1] at a certain time t including a person as a target (object), An image [2] as a background difference result is generated. This image [2] is a mask image in which the foreground pixel portion corresponding to the subject person is white (pixel value is 1) and the other background pixel portions are black (pixel value is 0).

  In FIG. 4, the texture image [3] of the object including the texture information of the person (target) and the background pixel portion being black can be obtained from the mask image [2] and the original image [1]. It has been shown to be. In this way, by separating the image background and its foreground, it becomes possible to roughly extract the object (image information) such as a person.

Here, when applying the background subtraction method in the extraction unit 2, the background image BG _{c, t} (u, v) for the image V _{c, t} (u, v) of each camera c is given in advance. To do. If the camera c is fixed and the light source conditions are unchanged, the background image may be a still image. In addition, when the extraction unit 2 obtains the mask image _{Mc, t} (u, v) as a likelihood map related to the foreground instead of a binary map (only foreground / background distinction), any type of existing target (such as an object) ) A likelihood calculation method may be used, for example, as a saliency map.

As for the extraction unit 2, as in the calibration unit 1, the mask image M _{c, t} (u, v) is associated with the input image V _{c, t} (u, v) in advance. For example, the extraction unit 2 may have an external configuration omitted from the synthesizing device 10.

<Calculation unit 3>
The calculation unit 3 uses the mask image M _{c, t} (u, v) generated by the extraction unit 2, and is a back projection unit that is a functional unit (functional unit responsible for subsequent processing) located on the rear stage side. The alpha value α1 _{c, t} (u, v) used for backprojecting to the backprojection plane group by 6 is calculated for each camera image c as each viewpoint image, and the alpha value α1 _{c, t} (u, v ) Is output to the backprojection unit 6. The calculated alpha value plays a role as a parameter for adjusting the remaining degree of the contour of the object in the free viewpoint image SY _t (u, v) finally synthesized in the drawing unit 7 on the subsequent stage side. It becomes. Specifically, the calculation unit 3 can calculate the alpha value α1 _{c, t} (u, v) in each of the following embodiments.

In the first embodiment, as the simplest calculation method, the pixel value of the mask image _{Mc, t} (u, v) obtained from the extraction unit 2 is directly adopted as the alpha value. For example, if the mask image _{Mc, t} (u, v) is extracted as a binary (0 or 1) image, the pixel value may be used as a binary alpha value as it is. That is, the alpha value of the foreground (pixel value = 1) region is 1, and the alpha value of the background (pixel value = 0) region is 0. Also, if the mask image _{Mc, t} (u, v) is given as a likelihood map of the foreground, the likelihood map is left as it is or normalized so that the value is 0 or more and 1 or less As a result, an alpha value may be obtained.

In the first embodiment, the calculation unit 3 is omitted from the synthesizing device 10 (the mask image M _{c, t} (u, v) obtained by the calculation unit 3 has an alpha value α1 _{c, t} (u, v ) Can be regarded as a configuration directly input to the back projection unit 6). (When normalizing the likelihood map, the normalization may be performed at the extraction unit 2 stage.)

The first embodiment is effective when the extraction unit 2 correctly extracts the foreground subject mask image _{Mc, t} (u, v). However, in practice, when a mask image is extracted from a real-world video, a mask image (in the case of a binary value) in which, for example, a part of the subject is missing is rarely extracted due to the influence of noise. Absent. As described above, there are the following second and third embodiments as other embodiments suitable for situations where extraction of a correct mask image cannot be expected.

In the second and third embodiments, it is assumed that the foreground and the background of the mask image _{Mc, t} (u, v) are binaryly distinguished. When the mask image M _{c, t} (u, v) is given as a foreground likelihood with a step value (or continuous value) larger than two values such as grayscale gradation, the likelihood is A foreground or background distinction may be obtained from the threshold determination.

In the second embodiment, the foreground pixels in the mask image M _{c, t} (u, v) adopt the same value (alpha value is 1) as in the first embodiment, but the background pixels have their alpha values changed. The alpha value α1 _{c, t} (u, v) is calculated by setting a value τ that is non-zero and greater than zero and less than 1 (0 <τ <1, for example, τ = 0.5).

According to the second embodiment, when the back projection is performed in the back projection unit 6 to be performed next and the alpha value α1 _{c, t} (u, v) is superimposed, the background pixel value τ is not zero. Therefore, it is expected that the boundary between the foreground and the background is likely to remain slightly, and the processing is performed to reduce the adverse effect due to the inaccuracy of the mask image _{Mc, t} (u, v) extracted by the extraction unit 2. It is possible to proceed.

The third embodiment changes the background alpha value τ, which is a fixed value regardless of the position in the second embodiment, according to the position, and changes the mask image _{Mc, t} (u, As the distance from the boundary between the foreground and background in v) to the pixel for which the alpha value is determined increases, the alpha value set for the background pixel decreases, and the alpha value α1 _{c, t} Calculate (u, v). Specifically, the distance (for example, perpendicular distance) from the background pixel whose alpha value is to be determined to the mask boundary in the vicinity of the pixel is d, and the alpha value α is expressed by, for example, the following equation (1) α = θ · f (d)
Can be used to calculate. Here, f (d) is a monotonically decreasing function of d that returns an alpha value, and θ is an attenuation rate of the alpha value.

  Compared with the second embodiment, the third embodiment makes it possible to convert a subject (target) into a free viewpoint image with a more natural appearance, but on the other hand, the distance d to the mask boundary in the vicinity of each pixel is reduced. Since it is necessary to calculate, it can be said that the amount of calculation and the calculation time are increasing.

  Further, in addition to the embodiments described above, various methods can be used as long as the calculation method is such that, for example, the alpha value of the foreground pixel is 1 and the alpha value of the background pixel is less than 1. It can be adopted.

<Face group setting unit 4>
The plane group setting unit 4 is used as a projection destination to be used by the back projection unit 6 or the like of the rear side processing unit in a three-dimensional model space that models the world coordinate system xyz of the multi-viewpoint image V _{c, t} (u, v). The back projection plane group P that is a plurality of planes is set, and the setting result (that is, the coordinate position / range that each plane constituting the plane group P occupies in the three-dimensional model space) is sent to the order setting unit 5. Output. As will be described in detail later, this backprojection plane group P is used as a reference for backprojecting the input multi-viewpoint image V _{c, t} (u, v) to the three-dimensional model space.

Specifically, the surface group setting unit 4 can set the backprojection surface group P as the one corresponding to the position (and the direction of the line of sight) of the virtual viewpoint CV in the three-dimensional model space specified by the user input or the like. it can. FIG. 5 is a diagram showing a schematic example for explaining an embodiment to be set. (FIG. 5 is also a schematic example of one embodiment of the order setting unit 5 described later.) In one embodiment, the position of the virtual viewpoint CV in the three-dimensional model space designated by the user input and the like and the position thereof A plane group P = {P _{t, k} | k = 1,2 consisting of K planes each having a predetermined size passing through a straight line of the camera axis Lc with respect to the viewing direction Lc (that is, the camera axis Lc of the virtual viewpoint CV) ,..., K}, and the planes P _{t, k} can be set to be parallel to each other at a predetermined angle with respect to the camera axis Lc.

Here, with respect to the orientation of each plane P _{t, k to be} set in the three-dimensional model space, the following embodiments are possible. That is, the angle formed by each plane P _{t, k and the} camera axis Lc may be an arbitrary predetermined angle, but in one embodiment, the angle may be a right angle. Also, instead of setting the orientation of each plane P _{t, k based on} the camera axis Lc, set the orientation based on the xyz coordinates (same as the world coordinate system xyz) set in the 3D model space. You may make it do. For example, assuming that the xy plane is the ground (field), a plane group P parallel to the xy plane may be set, or a plane group P parallel to the yz plane or the zx plane may be set.

Note that the world coordinate system xyz described with reference to FIG. 3 and the like with respect to the calibration unit 1 (that is, the world coordinate system xyz as a space in which the multi-viewpoint image V _{c, t} (u, v) is captured) is combined with the composite image. What is modeled for rendering to obtain SY _t (u, v) is a three-dimensional model space. Here, since the position of each point in the world coordinate system xyz and the position of each point in the three-dimensional model space correspond one-to-one, the coordinate system for the three-dimensional model space is the same as the world coordinate system “xyz”. In the following, the description will be given.

In one embodiment, the plane group P may be set assuming that the planes P _{t, k} parallel to each other are adjacent to each other with a predetermined distance d. In another embodiment, the distance between adjacent ones of the planes P _{t, k} is not a constant value d and may vary. For example, a setting may be used in which the distance is reduced as the plane P _{t, k} exists near a position where the object is likely to exist in the three-dimensional model space.

In the present invention, the plane group P set by the plane group setting unit 4 plays the role of the three-dimensional voxel (point group) in the visual volume intersection method of the prior art without the need for depth information. By realizing in the plane group P), it is possible to synthesize a free viewpoint synthesized video SY _t (u, v) at high speed while suppressing memory consumption. Accordingly, the scope of the plane group P are arranged in the three-dimensional model space xyz is multiview image _{V c, t (u, v} ) the subject region mask M _c by the extraction unit 2 _{from, t} (u, v) As long as it covers the range in which the object to be extracted (the object that is finally rendered from the free viewpoint in the rendering unit 7) can exist. The information on the range that can exist is given in advance as information linked to the multi-viewpoint image V _{c, t} (u, v), and the plane group setting unit 4 sets the plane group P so as to cover the range. do it. (The associated information may be further given in advance in association with the calibration information of the calibration unit 1.) That is, regarding the plane group P, the number K of planes to be configured, the distance d between planes, A setting related to memory consumption such as the size of the surface may be set so as to cover the range.

  Note that when the user or the like designates the viewpoint position (and orientation) of the virtual viewpoint CV, any existing information input interface may be used. For example, it may be directly input as a numerical value, or the numerical value may be obtained from an existing line-of-sight position detection technique (detection of the position of the user's pupil with respect to the pupil photographing camera). The numerical value may be calculated and acquired from a mouse operation or an operation on a touch panel. A menu selection may be used to input from a plurality of viewpoint position candidates.

<Order setting part 5>
The order setting unit 5 sets the order for each plane P _{t, k} constituting the surface group P set as described above by the surface group setting unit 4, and the set surface group P = {P _{t, k} | k = 1, 2,..., K} is output to the drawing unit 7. The drawing unit 7 described later performs processing using the surface group P in accordance with the output order.

Specifically, in the order setting unit 5, the surface group P = {P _{t, k} | k = 1,2,..., K} set by the surface group setting unit 4 is designated by the user input or the like in the surface group setting unit 4 The order based on the relationship with the position of the virtual viewpoint CV can be set. As a preferred embodiment, the order may be given in the order in which each surface is closer to the virtual viewpoint CV. In the following, when the notation of the surface group P = {P _{t, k} | k = 1,2,..., K} is used, the index k for distinguishing each surface is given by the order setting unit 5 It shall represent the order. Moreover, the case where the said close order k is set is demonstrated as an explanatory example. That is, the plane P _{t, k} in the surface group P represents a plane whose distance from the virtual viewpoint CV is kth, and the smaller k is, the closer to the virtual viewpoint CV is. , It is assumed that the larger k is, the farther the position is from the virtual viewpoint CV.

For example, the schematic example in FIG. 5 sets a plane group P (a group of planes P composed of K = 3 planes) perpendicular to the camera axis direction Lc with respect to the specified virtual viewpoint CV. Set in the unit 4, and further, in the order setting unit 5 by setting the order k = 1, 2, 3 in order from the closest to the virtual viewpoint CV, the three planes P _{t in the} closest order as the plane group P _{, 1} , P _{t, 2} , P _{t, 3} are set in the three-dimensional model space xyz so as to be separated from each other by a distance d. Further, the set three planes P _{t, 1} , P _{t, 2} , P _{t, 3} cover the range where the target OB can exist.

<Rendering unit 20>
The rendering unit 20 includes the calibration data obtained by the calibration unit 1, the alpha value α1 _{c, t} (u, v) calculated by the calculation unit 3, and the order-assigned surface group set by the order setting unit 5. By using P = {P _{t, k} | k = 1,2, ..., K}, the object in the multi-view image V _{c, t} (u, v) as an input to the synthesis device 10 is a free viewpoint. As a result of rendering into the display area, a composite image SY _t (u, v) is obtained.

More specifically, the rendering unit 20 can be realized by using, for example, a GPU as hardware, and a plane group P = {P _{t, k} | k = 1, 2,. , K} to the GPU vertex shader in the order k, and on the display based on the virtual viewpoint CV information (viewpoint position coordinates and line-of-sight information) specified by the user etc. in the face group setting unit 4 The target free viewpoint image as that at the virtual viewpoint CV is rendered to obtain a composite image SY _t (u, v). (It may be rendered in pixel units by the GPU pixel shader.) Here, 3DCG data, which is background information other than the target, is also read and simultaneously rendered in parallel based on a known method. The final free viewpoint image can be synthesized. In addition, 3DCG data as background information is converted to a predetermined background prepared by the extraction unit 2 when applying the background subtraction method to a virtual viewpoint CV (planar projection conversion for each plane part, etc.) May be synthesized.

  Hereinafter, the back projection unit 6, the drawing unit 7, the reprojection unit 8, and the assigning unit 9 that perform element processing for realizing the rendering process in the rendering unit 20 will be described. Here, after explaining the individual processes of the respective units 6, 7, 8, 9 and 9, the respective units 7, 8, 9 after the drawing unit 7 are further accompanied by repetitive processing / update processing in cooperation with each other. The operation flow will be described with reference to FIG.

<Back projection unit 6>
FIG. 6 schematically illustrates the processing of the backprojection unit 6 with respect to the case where the camera is composed of three (c = 1, 2, 3) and the surface group P is composed of three surfaces (k = 1, 2, 3). It is shown as an example. The back projection unit 6 obtains the alpha value α1 _{c, t} (u, v) for each camera c (c = 1, 2,..., N) obtained from the calculation unit 3 by the surface group P obtained by the order setting unit 5. = {P _{t, k} | k = 1,2, ..., K} Back-projected onto each surface P _{t, k} and then integrated, so that the accumulated alpha value on each surface P _{t, k} Obtain α2 _{t, k} (i, j).

Here, in order to clarify the distinction between the contents of each data, the following distinctions are used. That is, the alpha value obtained by the calculation unit 3 is “α1”, the alpha value obtained by integrating these with respect to each camera c by the back projection unit 6 is “α2”, and the names (function names) are distinguished. Further, the alpha value α1 of the calculation unit 3 corresponds to the position (u, v) of the input image V _{c, t} (u, v), and therefore is described as the pixel position (u, v), while the back projection unit 6 Since the alpha value α2 is given as a distribution on each plane P _{t, k} arranged in the xyz space, the position on the plane is set to (i, j) in distinction from (u, v) . Note that the position (i, j) is generally designated by a real number, unlike the pixel position (u, v).

Although shown schematically in Figure 6, the back projection plane P _t specified by the inverse projection unit 6 index _k, with respect to _k, the face P _t by the following steps 1 to _3, the _k The accumulated alpha value α2 _{t, k} (i, j) at can be obtained. Note that (in contrast to the drawing unit 7 described later), the backprojection unit 8 relates to each backprojection plane P _{t, k} in any order (not limited to the order specified by the index k). 3 may be implemented. Moreover, you may implement in parallel regarding several plane _{Pt, k} .

(Procedure 1) The alpha value image α1 _{c, t} (u, v) is arranged at a position corresponding to the arrangement position in the three-dimensional model space xyz of the camera c that has obtained the image.

(Procedure 2) By projecting the alpha value image α1 _{c, t} (u, v) arranged in the three-dimensional model space xyz from the camera center of the corresponding camera c toward the plane P _{t, k} , Back projection position (i _{[u, v]} , j _{[u, v]} ) _c on the plane P _{t, k} of each pixel position (u, v) of the alpha value image α1 _{c, t} (u, v) obtain. Here, the back-projected range in the space xyz is the top of the camera c of the camera c, and the alpha value image α1 _{c, t} (u, v) is the bottom surface (the cut surface of the cone, the bottom surface area in binary) It is expressed by a cone CN _{c, t} . In FIG. 6, the cone CN _{c, t} is schematically shown by a broken line with respect to c = 1,2,3. The projection result toward the plane P _{t, k} is the product set “P _{t, k} ∩CN _{c, t} ”.

(Procedure 3) Corresponding on the obtained back projection position (i _{[u, v]} , j _{[u, v]} ) _c (the position is also a position in the xyz space depending on the arrangement of the plane P _{t, k} ) Alpha value α1 _{c, t} (u, v) of each camera c (c = 1, 2,..., N that can be backprojected) (that is, alpha value α1 _{c, The} alpha value α2 _{t, k} (i, j) is obtained by accumulating _t (u, v)). Thus, for example, when the alpha value α1 _{c, t} (u, v) is a binary mask, α2 _{t, k} (i, j) at a location of 1 in all the cameras c is 1 (target), If even one contains 0, it becomes 0 (not the target area). Further, when the value of 0 to 1 is continuously set to α1 _{c, t} (u, v) in the calculation unit 3, an effect that the boundary gradually approaches 0 is obtained.

<Drawing part 7>
The drawing unit 7 synthesizes the synthesized image SY _t (u, v) as a free viewpoint image viewed from the virtual viewpoint CV designated by the user input or the like in the surface group setting unit 4. Here, the object captured in the multi-viewpoint image V _{c , t} (u, v) is rendered as the foreground texture TX _{c, t} (u, v) as viewed from the virtual viewpoint CV, and has already been described. A synthesized image SY _t (u, v) is obtained by synthesizing the background BG _t (u, v) as viewed from the virtual viewpoint CV by a known method as described above.

Here, when the drawing unit 7 specifically draws the foreground texture TX _{c, t} (u, v), the surface group P = {P _{t, k} | k = 1,2,. , K} in the order of the back projection plane P _{t, k} according to the order k. The multi-viewpoint image V _{c, t} (u, v) used in the drawing is located at the specified virtual viewpoint CV (and among the all cameras c distinguished by indexes c = 1, 2,..., N) (and (N) (n ≦ N) determined to be close. Note that the use of the n pieces corresponds to, for example, the image V _{c, t} (u, v) of the camera c that is reversed from the virtual viewpoint CV when the target is viewed from the side opposite to the virtual viewpoint CV. Therefore, there is a high possibility that the texture necessary for drawing is not included.

  For example, in the case where each camera c is arranged so as to surround the object in a circumferential shape or a spherical shape and is photographed facing the object, the virtual viewpoint CV is also taken from the vicinity of the circumference or the sphere. If it is set to watch the image, the proximity of the position and the proximity of the direction are linked, so it is only necessary to select n pieces of either the position or the direction that are close. If the arrangement of each camera c is arbitrary, n cameras c determined to be close to the virtual viewpoint CV may be selected in consideration of both the position and the orientation.

  Hereinafter, for the sake of explanation, it is assumed that the indices of n cameras determined to be close to the position (and direction) are c = 1, 2, 3,.

<Reprojection unit 8>
Further, in the drawing related to the back projection plane P _{t, k} in the drawing unit 7, each of the regions S to be actually used for the drawing from the multi-viewpoint images V _{c, t} (u, v) in the n cameras. _The reprojection unit 8 sets _{c, t, k} (u, v). The reprojection unit 8 obtains the region _{Sc, t, k} (u, v) and outputs it to the drawing unit 7 and the assigning unit 9.

<Granting part 9>
In addition, each pixel position (u, v) in the region S _{c, t, k} (u, v) (corresponding to a part of the image V _{c, t} (u, v)) used by the drawing unit 7 for drawing. For drawing, information that reflects the drawing history such as whether or not the texture TX _{c, t} (u, v) has already been used when drawing in the order of the back projection plane P _t, k according to the order k. Is held and updated as the control value d _{c, t, k-1} (u, v), and the drawing unit 7 draws in consideration of the control value d _{c, t, k-1} (u, v). Do. The assigning unit 9 obtains a control value d _{c, t, k-1} (u, v) that is considered in the drawing and outputs the control value d _{c, t, k-1} (u, v) to the drawing unit 7.

In the assigning unit 9, for each pixel position (u, v), the control value d _{c, t} is assumed to reflect the drawing history when drawing in the order of the back projection plane P _{t, k} according to the order k. _{, k-1} (u, v). For example, the control value d _{c, t, k−1} (u, v) may be obtained as the number of times the drawing has been performed. In the following description, it is assumed that the control value d _{c, t, k-1} (u, v) is the number of times the drawing has been performed.

The outline of the individual processing of the drawing unit 7, the reprojection unit 8, and the grant unit 9 has been described above. FIG. 7 is a flowchart according to an embodiment of an operation in which the drawing unit 7, the reprojection unit 8, and the assigning unit 9 draw the texture TX _{c, t} (u, v) in cooperation with each other. Hereinafter, the details of the operations of the respective units 7, 8, and 9 will be described while explaining the steps of FIG. Here, it can be seen that the flow structure of FIG. 7 takes the configuration of an iterative process, which expresses that the iterative process is drawn in the order of the backprojection planes P _{t, k} according to the order k = 1, 2,. It is a thing. Accordingly, in the description of FIG. 7, the index k is used not only as the meaning of the identifier k of the backprojection plane P _{t, k} but also as the meaning of the number counter k of the repetition process.

  When the flow of FIG. 7 is started, the counter k is set to an initial value k = 1, and the process proceeds to step S1.

<Step S1>
In step S1, the assigning unit 9 uses the initial value d _{c, t, 0} (u, v) at the initial value “k = 1” of the control value d _{c, t, k-1} (u, v) for drawing. Set as corresponding to each pixel position (u, v) of the image V _{c, t} (u, v) of each target camera c = 1,2, ..., n, then go to step S2 move on. (Note that there is no backprojection plane P _{t, k with} “k = 0” regarding the description of “initial value d _{c, t, 0} (u, v)”, but “control value as described below. d _{c, t, k-1} (u, v) ”(k ≧ 1) is a control value used for drawing on the backprojection plane P _{t, k} (k ≧ 1). ("Initial value d _{c, t, 0} (u, v)" exists as a control value used on the backprojection plane P _{t, 1.} )

Here, since drawing has not yet been performed at the time of step S1, the assigning unit 9 sets the values of the initial values d _{c, t, 0} (u, v) to all the cameras c = 1, 2,. The position (u, v) may be given as 0 (the number of drawing times is zero).

<Step S2>
In step S2, the reprojection unit 8 performs the reprojection area S _{c, t, k} as a partial area in the image V _{c, t} (u, v) in each camera c = 1, 2 _,. After (u, v) is set, the process proceeds to step S3.

FIG. 8 is a diagram illustrating a schematic example of processing in which the reprojection unit 8 obtains the reprojection region _{Sc, t, k} (u, v). In FIG. 8, the processing indicated by the arrow line L1 from [1] to [2] indicates the back projection processing by the back projection unit 6 which has already been described with reference to FIG. _The alpha value α2 _{t, k} (i, j) back-projected on _{t, k} is shown as an elliptical region. In FIG. 8, the process indicated by the arrow L2 from [2] to [3] is a schematic example of processing by the reprojection unit 8.

Here, in describing the processing of the reprojection unit 8 and its significance, the terms are defined as follows. As already described in the schematic example of FIG. 6 and the like, the backprojected alpha value α2 _{t, k} (i, j) obtained by the backprojection unit 6 is distributed on all backprojection planes P _{t, k.} {Α2 _{t, k} (i, j) | k = 1,2,..., K} arranged in the space xyz corresponds to the visual hull in the prior art visual volume intersection method. It was obtained as a thing. Therefore, the arranged data {α2 _{t, k} (i, j) | k = 1,2,..., K} corresponding to the visual hull is referred to as “backprojection data”.

Alpha value α2 _{t, k} (i, j) of one plane P _{t, k} taken out from k in the back projection data {α2 _{t, k} (i, j) | k = 1,2, ..., K} Corresponds to a “cross section” _obtained by slicing an object (such as a person) whose shape is represented by the backprojection data along the plane P _{t, k} . (Note that [2] in FIG. 8 schematically illustrates the cross section as an elliptical shape.)

In the reprojection unit 8, the alpha value α2 _{t, k} (i, j) as the cross section in the plane P _{t, k} designated by the index k (processing order k) is assigned to each camera c = 1, 2,. , n is re-projected onto the image plane (u, v) _c to obtain a re-projection region S _{c, t, k} (u, v) in the corresponding image V _{c, t} (u, v). Here, the reprojection region S _{c, t, k} (u, v) is obtained by reprojecting the region of which alpha value α2 _{t, k} (i, j) has a value greater than 0 and is determined to be the foreground. What should I do? As is apparent, the reprojection in the reprojection unit 8 is the reverse of the backprojection (u, v) → (x, y, z) by the backprojection unit 6, that is, normal projection (x, y, z) → (u, v) is a process for obtaining an area formed on the image plane (u, v) _c when a cross-sectional area of the alpha value α2 _{t, k} (i, j) is photographed by the camera c. .

The reprojection region S _{c, t, k} (u, v) in each camera image V _{c, t} (u, v) acquired as described above is backprojected, as is clear from the processing contents of the reprojection. It includes a texture for drawing a cross section of an object on the plane plane P _{t, k} . Therefore, in the next step S3, the drawing unit 7 performs drawing on the cross section.

<Step S3>
In step S3, the drawing unit 7 determines the texture of the reprojection area S _{c, t, k} (u, v) in each camera image V _{c, t} (u, v) (c = 1,2, ..., n). the reverse projection plane P _t which is designated by the index _k, backprojection to _k by _{((u, v) c →} (x, y, projection of z)), the back projection plane P _t, the _k After drawing the texture, the process proceeds to step S4. (The drawn range is the cross-sectional area of the alpha value α2 _{t, k} (i, j).)

  FIG. 9 is a diagram illustrating a schematic example when the drawing is performed with two images of the cameras c = 1,2.

At the time of drawing, the texture TX _{t, k} (i, j) at the same position (i, j) on the backprojection plane P _{t, k} is converted into a plurality of camera images V _{c, t} (u, v) ( c = 1,2, ..., n) are drawn from the corresponding positions (u _{[i, v]} , v _{[i, v]} ) _{c of the} reprojection region S _{c, t, k} (u, v) in _c ) Will be obtained. Accordingly, the drawing unit 7 obtains the texture TX _{t, k} (i, j) by distributing the pixels from the plurality of cameras c used for the drawing as schematically shown in the following addition formula. In accordance with the distribution, the drawing is performed. In the following addition formula, “E _c ” is a coefficient representing a distribution ratio of the pixel “V _{c, t} (u _{[i, v]} , v _{[i, v]} )” of the camera c.
TX _{t, k} (i, j) = Σ _c E _c * V _{c, t} (u _{[i, v]} , v _{[i, v]} )

  Here, various embodiments are possible with respect to the technique of distributing and drawing.

In the first embodiment, the distribution coefficient E _c camera c, the position of the camera c (and orientation), and the position of the virtual viewpoint CV specified (and orientation), the closer the value of the coefficient The pixels of the near camera c may be allocated with a greater focus. Note that the evaluation of the proximity of the position (and orientation) of the camera may be performed using the same evaluation as when the cameras c = 1, 2,..., N close to the virtual viewpoint CV are determined.

In the second embodiment, the assigning unit 9 assigns the distribution coefficient E _c of the camera c to the pixel “V _{c, t} (u _{[i, v]} , v _{[i, v]} )” of the camera c used for the drawing. The control value d _{c, t, k-1} (u _{[i, v]} , v _{[i, v]} ) given from, that is, the smaller the number of times already used for drawing Can do. According to the second embodiment, the pixels used for drawing at a certain point of time have less influence on the drawing thereafter. As a special case, if it has been used for drawing even once, a distribution coefficient E _c = 0 can be set so that the flag is not used thereafter for drawing. Similarly, the distribution coefficient E _c = 0 may be set when a predetermined upper limit number is reached.

In the second embodiment, the distribution coefficient E _c of the camera c is set for each drawing source position (u _{[i, v]} , v _{[i, v]} ) corresponding to the drawing destination position (i, j). Coefficient E _c (u _{[i, v]} , v _{[i, v]} ). In addition, in the first drawing of k = 1, the drawing state is not yet drawn (the control value d _{c, t, 0} is the initial value given in step S1), so that the distribution coefficient E _c according to the second embodiment is all Will be equal.

The first embodiment and the second embodiment relating to the distribution can be combined. The distribution coefficient E _c is obtained in the first embodiment and / or the second embodiment, and is normalized so that the sum of all the cameras c = 1, 2,. It is only necessary to perform drawing.

That is, with respect to the method of distributing and drawing, as described above, it may be specifically overlapped by alpha blending with respect to taking the weighted sum with the distribution coefficient E _c . (Note that for each distribution coefficient E _c , the range of 0 ≦ E _c ≦ 1 is set, and the value of the sum Σ _c E _c of the coefficients E _c for obtaining the texture TX _{t, k} (i, j) is 1. (The weighted sum that is normalized to corresponds to the alpha blend.)

In this case, the alpha value α _{c, t, k-1} (u _{[i, v]} ) is determined from the control value d _{c, t, k-1} (u _{[i, v]} , v _{[i, v]} ) using a predetermined function g _{. v]} , v _{[i, v]} ) are obtained as follows, and the alpha value α _{c, t, k-1} (u _{[i, v]} , v _{[i, v]} ) is used to determine the alpha What is necessary is just to make it blend. The predetermined function g is the control value d _{c, t, k-1} (u _{[i, v]} , v _{[i, v]} ), that is, already used for drawing, almost the same as in the second embodiment related to the distribution coefficient E _c . A function that increases the transparency as the number of times increases is used.
α _{c, t, k-1} (u _{[i, v]} , v _{[i, v]} ) = g (d _{c, t, k-1} (u _{[i, v]} , v _{[i, v]} ))

Here, the _calculated alpha value α _{c, t, k-1} (u _{[i, v]} , v _{[i, v]} ) is 1 for all cameras c = 1, 2,..., N. You may standardize as follows.

For example, regarding the example of the camera c = 1, 2 in FIG. 7, when alpha blending is performed in consideration of the distances m1, m2 from the camera (similar to the first embodiment regarding the distribution coefficient E _c ), The alpha blend result can be obtained as follows.
TX _{t, k} (i, j) = α * {1-m1 / (m1 + m2)} * V _{1, t} (u _{[i, v]} , v _{[i, v]} )
+ [1-m2 / (m1 + m2) + (1-α) * {1- m1 / (m1 + m2)}] * V _{2, t} (u _{[i, v]} , v _{[i, v]} )
Here, α = α _{1, t, k−1} (u _{[i, v]} , v _{[i, v]} ), that is, α is an alpha value of the camera c = 1.

This alpha blending example distributes the amount reduced by the alpha value of camera c = 1 (the amount of transparency of the texture drawing of camera c = 1) to camera c = 2, and increases the opacity of the texture. It is an example. In exactly the same manner, generally, the following equations may be used with α and β as the alpha values of the cameras c = 1 and 2.
TX _{t, k} (i, j) = [α * {1-m1 / (m1 + m2)} + (1-β) * {1- m2 / (m1 + m2)}] * V _{1, t} (u _{[i, v]} , v _{[i, v]} )
+ [β * {1-m2 / (m1 + m2)} + (1-α) * {1- m1 / (m1 + m2)}] * V _{2, t} (u _{[i, v]} , v _{[i , v]} )
As is apparent from the above equation, each of V _{1, t} (u _{[i, v]} , v _{[i, v]} ) and V _{2, t} (u _{[i, v]} , v _{[i, v]} ) Is a normalized alpha value that takes into account both the distinction of cameras c = 1,2 and their distances m1, m2.
Similarly, when there are three or more cameras, the texture of the camera made transparent by the alpha value may be distributed to the other cameras.

<Step S4>
In step S4, based on the drawing result in the drawing unit 7 in the latest step S3, the assigning unit 9 uses the control values d _{c, t, k} for the drawing unit 7 to use in the next (k + 1) step S3. _{After obtaining +1} (u, v), the process proceeds to step S5. As described above, the control value d _{c, t, k + 1} (u, v) is used for drawing by the pixel (u, v) of the image V _{c, t} (u, v) of the camera c up to the time point. What is necessary is just to obtain | require as the number of times obtained.

<Step S5>
In step S5, it is determined whether or not drawing has been completed for all backprojection planes Pt, k (i, j), and if completed, that is, if k = K at that time, the process proceeds to step S7. If not completed, that is, if k <K, the process proceeds to step S6.

<Step S6>
In step S6, the next value k + 1 is set to k, that is, the value of k is incremented by 1, and the process returns to step S2.

<Step S7>
In step S7, the textures TX _{t, k} (i, j) obtained for all backprojection planes P _{t, k} (i, j) (k = 1, 2,..., K) in the above K _iterations. ) (The texture is the result of rendering the target in xyz space), and the rendering unit 7 projects the target (foreground) rendering result onto the image plane (u, v) of the virtual viewpoint CV. And the background is rendered by a known method as described above to obtain a synthesized video SY _t (u, v), and the flow ends.

As described above, according to the present invention, rendering on the backprojection plane P _{t, k} (i, j) in the order specified by the index k, and rendering for obtaining the composite video SY _t (u, v) By allocating between cameras by alpha value etc., the problem of quality degradation due to occlusion is solved without impairing the high-speed processing form (real-time property) of free viewpoint video composition in Patent Document 1 as a prior method. can do. Hereinafter, supplementary explanations in the present invention will be described.

(1) In the process by the drawing unit 7 (step S7 in FIG. 7), textures TX _{t, k} (at a plurality of different positions (i, j) (positions of real numbers i, j) on the backprojection plane P _{t, k} i, j) may be back-projected to the same pixel (u, v) (integer) in the composite image SY _t (u, v) and correspond to it. In such a case, it may be handled by processing according to the implementation of the GPU or the like. For example, it is obtained as an average value of textures at the plurality of positions (i, j) (for example, three positions (0.1, 0.1), (0.11, 0.09), (0.09, 0.11) close to each other). Good.

(2) The synthesizing device 10 of the present invention can be realized as a computer having a general configuration. That is, from users such as CPU (Central Processing Unit) and GPU (Graphic Processing Unit), main storage that provides work area to the CPU, auxiliary storage that can be configured with hard disk, SSD, etc., keyboard, mouse, touch panel, etc. The compositing apparatus 10 may be configured by a general computer including an input interface for receiving input, a communication interface for communication by connecting to a network, a display for displaying, a camera, and a bus for connecting them. it can. The processing of each unit of the synthesizing apparatus 10 can be realized by a CPU and / or GPU that reads and executes a program for executing the processing, but any part of the processing is realized by a separate dedicated circuit or the like. You may make it do.

  DESCRIPTION OF SYMBOLS 10 ... Composition apparatus, 1 ... Calibration part, 2 ... Extraction part, 3 ... Calculation part, 4 ... Surface group setting part, 5 ... Order setting part, 6 ... Back projection part, 7 ... Drawing part, 8 ... Reprojection part, 9 ... Granting part, 20 ... Rendering part

Claims (11)

A calculation unit for obtaining a likelihood image of a target area captured from each viewpoint image of the multi-viewpoint image;
A backprojection unit for obtaining backprojection data obtained by backprojecting each of the likelihood images onto a plurality of backprojection planes arranged in a three-dimensional space;
The texture of all or part of the viewpoint images of the multi-viewpoint image is drawn on the backprojection data by sequentially backprojecting the texture onto the plurality of backprojection planes constituting the backprojection data. And a rendering unit that synthesizes the target free viewpoint image by rendering.   The drawing unit considers a history regarding whether or not each pixel of the viewpoint image is already used for the drawing when the texture of the whole or a part of the viewpoint image is drawn by the back projection. The synthesizer according to claim 1.   In the drawing unit, when the texture of all or a part of the viewpoint images of the multi-viewpoint image is backprojected and drawn, the number of times is based on the number of times the pixels of the viewpoint image are already used for the drawing. The composition device according to claim 1, wherein the ratio of pixels used for drawing is reduced for drawing.   The synthesizing apparatus according to claim 3, wherein the drawing unit performs the drawing using an alpha value that increases a degree of transmission as the number of times increases.   In the drawing unit, when the textures of all or part of the viewpoint images are backprojected and drawn, the sum of the alpha values in the texture used to draw the same portion is constant. The composition apparatus according to claim 4, wherein the composition is drawn after normalization.   In the rendering unit, when the back-projected and rendered textures of all or part of the viewpoint images of the multi-viewpoint image, the position of the virtual viewpoint of the free viewpoint image to be synthesized and the position of the viewpoint of each viewpoint image 6. The composition apparatus according to claim 1, wherein drawing is performed according to a difference from the composition.   The composition device according to claim 6, wherein the drawing unit reduces a ratio used for drawing for a viewpoint image having a larger difference.   The composition apparatus according to claim 1, wherein the order is an order corresponding to a positional relationship between a virtual viewpoint of the combined free viewpoint image and the plurality of back projection planes.   9. The synthesizing apparatus according to claim 8, wherein the order is earlier as each of the plurality of back projection planes is closer to the virtual viewpoint of the synthesized free viewpoint image. A calculation step for obtaining a likelihood image of a target area captured from each viewpoint image of the multi-viewpoint image;
Backprojecting to obtain backprojection data obtained by backprojecting each of the likelihood images onto a plurality of backprojection planes arranged in a three-dimensional space;
The texture of all or part of the viewpoint images of the multi-viewpoint image is drawn on the backprojection data by sequentially backprojecting the texture onto the plurality of backprojection planes constituting the backprojection data. A rendering step of rendering the target free viewpoint image by rendering. A computer, a synthesizer,
A calculation unit for obtaining a likelihood image of a target area captured from each viewpoint image of the multi-viewpoint image;
A backprojection unit for obtaining backprojection data obtained by backprojecting each of the likelihood images onto a plurality of backprojection planes arranged in a three-dimensional space;
The texture of all or part of the viewpoint images of the multi-viewpoint image is drawn on the backprojection data by sequentially backprojecting the texture onto the plurality of backprojection planes constituting the backprojection data. A program that functions as a synthesizing device that includes a drawing unit that synthesizes the target free viewpoint image by rendering.

Images