JP6894873B2 - Image processing equipment, methods and programs - Google Patents

Description

INDUSTRIAL APPLICABILITY The present invention can apply a free-viewpoint image to a billboard system that can be obtained at high speed, and can quickly and quickly cover a target area on an image corresponding to a virtual viewpoint, which is required to generate the billboard. It relates to an image processing apparatus, a method and a program that can be obtained with high accuracy.

The free-viewpoint video technology enables the viewer to interactively and freely control the viewpoint to generate an image viewed from an arbitrary viewpoint. According to this technology, the viewer can set a desired viewpoint to view a realistic image without the viewpoint being determined in advance by the content creator or the like as in ordinary video content that is not a free viewpoint. It is possible to do. In this technique, it is generally performed to generate a high-quality free-viewpoint image from a multi-viewpoint image obtained by shooting with a plurality of cameras so as to surround the target scene of the image. Here, the approach for generating a free-viewpoint image can be divided into two types. One is based on a complete 3D reconstruction and the other does not necessarily utilize the 3D reconstruction.

Non-Patent Document 1 introduces a free-viewpoint video technology by automatically performing three-dimensional reconstruction and texture mapping, and a visual hull (product set) generated as an intersection (intersection) of visual volumes based on a silhouette image. It has been pointed out that Visual Hull) gives an approximate shape to the extraction target and enables high-quality rendering by performing viewpoint-dependent texture mapping. Here, the shape of the visual hull needs to be accurate so that there is no noticeable error in the texture mapping.

Non-Patent Document 2 introduces a method of semi-automatically generating a free-viewpoint image without performing complete 3D reconstruction. Specifically, a building as an approximation by a 2D region of a target 3D model. By performing target detection, target tracking, and target separation in order to perform texture mapping on the board (billboard), it is possible to generate a free-viewpoint video of a sports game using an application on a mobile terminal without compromising video quality. Has been done.

In Non-Patent Document 3, as a method of obtaining the target visual hull as a polyhedral model, (1) the viewing edge is specifically calculated, (2) the pyramid intersection edge is calculated, and (3) A method consisting of the steps of identifying the faces of a polyhedron is introduced. Firstly, the method can generate a polyhedral model that can correspond to precise and various shapes in a verifiable manner, and secondly, it can generate the polyhedral model more efficiently than the existing latest method. Has been done.

Grau O, Hilton A, Kilner J, et al. A free-viewpoint video system for visualization of sport scenes [J]. SMPTE motion imaging journal, 2007, 116 (5-6): 213-219. Sabirin, Houari, et al. "Semi-Automatic Generation of Free Viewpoint Video Contents for Sport Events: Toward Real-time Delivery of Immersive Experience." IEEE MultiMedia (2018). Franco J S, Boyer E. Efficient polyhedral modeling from silhouettes [J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2009, 31 (3): 414-427.

However, the above-mentioned conventional techniques still have room for improvement in obtaining free-viewpoint images efficiently and with high quality.

That is, in Non-Patent Document 1, in order to obtain a high-quality free-viewpoint image, a visual hull as a highly accurate three-dimensional model must be generated, which significantly increases the amount of calculation and thus reduces efficiency. I had no choice but to do it. Further, in Non-Patent Document 2, it is avoided to obtain a complete three-dimensional model by the billboard method for efficiency, but the billboard method approximates the three-dimensional model in two dimensions to cause occlusion (occlusion). There is a problem that the finally generated free-viewpoint image becomes unnatural when there is a shielded part that cannot be seen from the camera due to the overlapping of the objects.

Further, there is room for improvement in Non-Patent Document 3. For example, although the polyhedron model achieves a higher speed than that of Non-Patent Document 1, the polyhedron model also involves a considerable load in the calculation.

In view of the above problems of the prior art, the present invention provides a virtual viewpoint required to generate the billboard, which can be applied to a billboard system capable of obtaining a free-viewpoint image at high speed. It is an object of the present invention to provide an image processing apparatus, method and program capable of obtaining a target area on a corresponding image at high speed and with high accuracy.

In order to achieve the above object, the present invention is an image processing device, and a point of obtaining target point cloud data from a multi-viewpoint image based on a viewpoint position in each image and a point on the boundary between the foreground and the background. A group generation unit, a sorting unit that clusters the point cloud data into point cloud data for each individual object, a neighborhood acquisition unit that acquires neighborhood points of each point in the point cloud data for each individual object, and the individual unit. The back projection unit that back-projects the point cloud data for each object onto the image plane at the designated virtual viewpoint and the neighborhood points of the acquired points are defined as the neighborhood points to which the points on the image plane are connected. As a result, the region defined by the connecting relationship is characterized by including an extraction unit for extracting an individual target region in the image plane. Further, it is characterized in that it is a method and a program corresponding to the image processing apparatus.

According to the present invention, it is possible to obtain a region of an individual object captured in an input multi-viewpoint image in an image at a designated virtual viewpoint at high speed and with high accuracy.

It is a functional block diagram of the image processing apparatus which concerns on one Embodiment. It is a figure for demonstrating the processing in a point group generation part. It is an auxiliary explanatory view of FIG. It is a figure for demonstrating the processing in a point group generating part and a sorting part. It is a figure for demonstrating the processing in the back projection part and the extraction part.

FIG. 1 is a functional block diagram of an image processing device according to an embodiment. As shown in the figure, the image processing device 10 includes a point group generating unit 1, a sorting unit 2, a neighborhood acquisition unit 3, a back projection unit 4, an extraction unit 5, and a drawing unit 6. As its overall operation, the image processing device 10 is viewed from the input multi-viewpoint image and the virtual viewpoint input by the user (viewer) from the designated virtual viewpoint based on the billboard method. It generates a free-viewpoint image.

Further, as shown in the figure, the schematic functions of the respective parts 1 to 6 of the image processing apparatus 10 are as follows. The point cloud generator 1 reflects the arrangement of the objects (generally one or more objects) captured in the multi-viewpoint image in the three-dimensional space from the input multi-viewpoint image. , Generates point cloud data in the three-dimensional space, and outputs the point cloud data to the sorting unit 2. It should be noted that the input multi-viewpoint image is given a distinction between the foreground and the background in advance in the image of each viewpoint, and the point cloud generation unit 1 also uses the given distinction between the foreground and the background to make a point. Generate point cloud data.

The sorting unit 2 obtains the point cloud data sorted for each individual object by clustering the point cloud data obtained from the point cloud generating unit 1, and acquires the point cloud data for each individual target in the neighborhood. Output to unit 3 and back projection unit 4.

The neighborhood acquisition unit 3 provides information on which of the points corresponding to the neighborhood is in the same individual target point cloud data for each point belonging to the point cloud data for each individual object obtained from the sorting unit 2. Is acquired, and the information about the points in the vicinity of each acquired point is output to the extraction unit 5 for each point cloud data of the individual target.

The back projection unit 4 back-projects the point cloud data for each individual object obtained from the sorting unit 2 onto the image plane corresponding to the virtual viewpoint set by the user, and the result of the back projection, that is, on the image plane. The point cloud data for each individual object in is output to the extraction unit 5. In the back projection process in the back projection unit 4, that is, assuming that there is a virtual camera in the virtual viewpoint set by the user, the arrangement on the camera image when the point cloud data for each individual object is taken by the virtual camera is obtained. Corresponds to processing.

The extraction unit 5 is obtained from the back projection unit 4 by using the information of the points in the vicinity of each point of the point group data (data in the three-dimensional space) for each individual object obtained from the neighborhood acquisition unit 3. Regarding the point group data for each individual object as the arrangement on the image plane (data on the two-dimensional image plane), the information of the area occupied by the corresponding individual object on the image plane is obtained, and the obtained image plane. The information of the area of the individual target above is output to the drawing unit 6. In the extraction unit 5, the information obtained from the neighborhood acquisition unit 3 can obtain the information in the region, assuming that the points that are considered to be points in the vicinity of each other are connected to each other on the region.

The drawing unit 6 uses an image of one or more viewpoints determined to be close to the virtual viewpoint set by the user among the multi-viewpoint images input to the image processing device 10, and the image plane obtained by the extraction unit 5. By drawing (rendering) to the area of each individual object (corresponding to the foreground occupied by the individual object) on (corresponding to the plane of the camera image of the virtual viewpoint set by the user), the freedom seen from the virtual viewpoint set by the user Generate a viewpoint image.

Although the outline processing contents of each part 1 to 6 of the image processing apparatus 10 have been described above, the processing can be performed in real time by executing each frame at each time on the video. That is, the image processing device 10 has a multi-viewpoint image MVP (t) as a frame of each time t = 1, 2, 3, ... Of the multi-viewpoint video given at a predetermined frame rate, and each time t set by the user. The virtual viewpoint V (t) in the above is received as an input, and the free viewpoint image FVP (t) as seen at the virtual viewpoint V (t) can be output in real time as a video at each time t. The high-speed processing by the image processing apparatus 10 of the present invention enables the real-time processing, and even when occlusion occurs between objects in the multi-viewpoint image, a high-quality free-viewpoint image can be obtained. It becomes possible to obtain.

The details of the processing of each part 1 to 6 of the image processing apparatus 10 will be described below. 2 to 5 show data and the like handled in the processing of each part by a schematic example, and in the following description, these will be referred to as examples as appropriate. FIG. 2 is a diagram for explaining the processing in the point group generating unit 1, and FIG. 3 is an auxiliary explanatory diagram of FIG. FIG. 4 is a diagram for explaining the processing in the point group generating unit 1 and the sorting unit 2 separately from [a] to [f]. FIG. 5 is a diagram for explaining the processing in the back projection unit 4 and the extraction unit 5 separately from [g] to [k].

<Point group generator 1>
The role of the point cloud generator 1 is based on the billboard method, which corresponds to a visual hull given in the form of a polygon mesh or the like as a result of complete three-dimensional reconstruction as in Non-Patent Document 1 described above. This is required at high speed in the image processing device 10 that generates a free viewpoint image. As a result, the image processing device 10 can secure occlusion resistance, which was difficult with the conventional billboard method. Specifically, the point group generator 1 obtains a point cloud corresponding to the visual hull.

Here, the multi-viewpoint image as the input for the point cloud generator 1 to obtain the point cloud is N images P1, P2,…, PN taken by N cameras C1, C2,…, CN. It is assumed that the foreground and the background are distinguished in advance in each image P1, P2, ..., PN by applying an arbitrary existing method such as the background subtraction method. Here, the foreground area in each of the images P1, P2, ..., PN corresponds to the area of the object being photographed, which is the object of rendering in the free-viewpoint image generated by the image processing device 10. It is also assumed that the calibration information between each camera and each camera is known.

The point group generating unit 1 generates a point group as an approximation of the visual hull from the multi-viewpoint image in which the foreground and the background are distinguished in the image of each viewpoint as described above. Specifically, the point cloud generation unit 1 obtains the line-of-sight edge (viewing edge) of the polygon mesh generated as an approximation of the visual hull in Non-Patent Document 3 described above, and further, the present invention. In one embodiment, a point cloud can be generated at high speed by extracting only the points at both ends of the line-of-sight edge as the line segment.

Here, a method of obtaining the line-of-sight edge as a process in the point group generating unit 1 will be described. For the line-of-sight edge (generally present in large numbers), select one of the input N multi-viewpoint images P1, P2,…, PN as the reference image, and use the other as the target image. After selecting, it can be obtained as corresponding to the reference image and the target image. VE (Pt) is the line-of-sight edge set obtained from the target image Pt and the reference image Pr (where t = 1, 2,…, N, r = 1, 2,…, N, t ≠ r). , Pr), in the point group generator 1, all combinations of the selection method of the target image Pt and the reference image Pr (multi-viewpoint image) as given by the following equation (1). In the case of N images, the total number of combinations is N (N-1)), and _all of the line-of-sight edges are obtained as VE all, which is a collection of line-of-sight edge sets VE (Pt, Pr). Then, as described above, the point cloud generator 1 outputs a set of points at both ends of each edge (line segment) belonging to the _{obtained edge set VE all as a point cloud.}

FIG. 2 is a diagram for explaining the process of obtaining the line-of-sight edge set VE (Pt, Pr) from the target image Pt and the reference image Pr. Here, Pt = P1 and Pr = P2 (t = 1 and r = 2). ) Is shown as an example, but the process of obtaining the edge set VE (Pt, Pr) is common regardless of the selection method. In FIG. 2, the positions of the camera C1 and the camera C2 that capture the target image P1 and the reference image P2, respectively, are indicated by points (white circles ◯) as the positions of the optical centers thereof. (Therefore, in the description of FIG. 2, the optical center of the camera is referred to as "optical center C1" or "optical center C2".) In FIG. 2, a group of straight lines passing through the optical center C1 and the optical center C2 is drawn. However, the depicted arrangement is similar to the well-known arrangement in epipolar geometry for stereovision.

The line-of-sight edge set VE (P1, P2) for P1 and P2 in FIG. 2 can be obtained by the flow of the following repeating steps.
(Procedure 1) A new point pi on the boundary between the foreground area R1 of the target image P1 (the area shown in gray in the image P1 in FIG. 2) and the background area is newly selected, and the process proceeds to step 2.
(Procedure 2) A plane (epipolar surface) stretched by a line segment VL1 from the optical center C1 of the target image P1 toward the point pi, a straight line passing through the optical centers C1 and C2 of each image P1 and P2, and a reference image P2. The line segment (generally a plurality) in the foreground region R2 is obtained as the intersection with the foreground region R2 (the region shown in gray in the image P2 in FIG. 2), and the line segment at the intersection is obtained from the optical center C2. Is projected onto the line-of-sight VL1 to obtain a line segment belonging to the line-of-sight edge set VE (P1, P2), and then the process proceeds to step 3.
(Procedure 3) If all the point pis have been selected in the series of steps 1 repeated up to this point, the line-of-sight edge set VE (P1, P2) obtained in the series of steps 2 up to this point is output. End the flow. If all the point pis have not been selected, the process returns to step 1 and a new point pi that has not been selected so far is selected.

Here, regarding step 2, in the example of FIG. 2, two line segments where the epipolar surface and the foreground region R2 intersect are obtained on the intersection line (epipolar line) L2 between the epipolar surface and the reference image P2. Assuming that this is projected onto the line of sight VL1 from the optical center C2, a line segment having two points pi-1 and pi-2 at both ends and a line segment having two points pi-3 and pi-4 at both ends, The two line segments of are found to belong to the line-of-sight edge set VE (P1, P2).

Further, an example of a point pi selected from the boundary between the foreground region R1 and the background region in step 1 is shown in FIG. 3 as corresponding to the example of FIG. In FIG. 3, eight points p1, p2,…, and p8 are selected as individual pis. Therefore, in the case of the example of FIG. 3, by repeating the above steps 1 to 3 only eight times, the processing is completed for all the point pis, and the flow is completed.

In one embodiment of the present invention, selecting the point pi on the boundary in step 1 is also a method different from that of Non-Patent Document 3. That is, in Non-Patent Document 3, the boundary is approximated by a polygon in the target image, and a point on the boundary is not particularly selected. On the other hand, in step 1, each point pi on the boundary is discrete on the target image as being on the boundary and at regular intervals from each other in pixel units, that is, at a distance on the target image. Select. By selecting each point pi discretely in pixel units, there is an effect that the distribution of the obtained line-of-sight edges can be made uniform.

Regarding steps 1 and 2, the foreground area R1 (and background area) of the image P1 and the foreground area R2 (and the background area) of the image P2 are, as described above, the foreground and background in the input multi-viewpoint image in advance. It is determined by the fact that the distinction between is given.

FIGS. 2 and 3 have described the process of obtaining the line-of-sight edge set VE (Pt, Pr) for one selection method of t = 1 and r = 2, but as described in the above equation (1), regarding the other selection methods. Similarly, by obtaining the line-of-sight edge set VE (Pt, Pr), the line-of-sight edge set VE _all can be obtained as a whole of the line-of-sight edge set VE (Pt, Pr) obtained for all selection methods. In FIG. 4, by dividing into [a] to [d], the process of increasing the line-of-sight edge set VE (Pt, Pr) according to the selection method, that is, each time the selection method is covered, the equation (1) An example of the process of increasing the union and finding it is shown. [a] is an example in which only VE (P1, P2) is plotted in a three-dimensional space with camera C1 as the target side and camera C2 as the reference side. [B] is an example in which camera C1 is the target side and cameras C2 to C4 are respectively. This is an example of plotting VE (P1, P2) to VE (P1, P4) together as the reference side. [C] shows VE (P1, P2) with camera C1 as the target side and cameras C2 to C8 as the reference side. This is an example of plotting ~ VE (P1, P8) together. [D] summarizes VE (P1, P2) ~ VE (P1, P10) with camera C1 as the target side and cameras C2 to C10 as the reference side. This is an example plotted. We can see the process of increasing the union of equation (1) from [a] to [d] (that is, _{approaching the final VE all).}

[E] in FIG. 4 is an example of point cloud data in the three-dimensional space output by the point cloud generator 1 as points at both ends of each edge (line segment) of _{the finally obtained viewpoint edge set VE all.} As shown in the figure, the point cloud is composed of two individual objects OB1 and OB2 (for example, individual objects as two soccer players on a soccer field), and it turns out that the point cloud is composed. Is due to the processing of the next sorting unit 2.

<Separation part 2>
In the sorting unit 2, clustering is performed on the point cloud data (multiple point coordinate data in the three-dimensional space) obtained by the point cloud generation unit 1, and each cluster as a result is individually photographed in the multi-viewpoint image. Obtain the clustering result as applicable to the target. In one embodiment, clustering may be performed directly on the three-dimensional point cloud data PG ∋ {(x, y, z)} obtained by the point cloud generation unit 1. _{In another embodiment, the target is the two-dimensional data P XY} ∋ {(X, Y)} obtained by projecting the three-dimensional point cloud data PG obtained by the point cloud generation unit 1 onto a predetermined plane (referred to as the XY plane). As a result, clustering may be performed. (In this case, the clustering result after projecting to the 2D data will be adopted as the clustering result for the original 3D point cloud data.)

Here, regarding the predetermined plane XY for projection, a plane stretched along the direction in which the individual target moves (for example, when the individual target is each soccer player on the soccer field, the soccer field as the ground). Plane) or a plane used for calibrating the input multi-viewpoint image (for example, a plane on which a plane marker for calibration is arranged) can be adopted. Moreover, other planes may be adopted.

Assuming that each cluster of the clustering result is cluster o _i (i = 1, 2,…, n), the processing in the sorting part 2 is formally performed by the clustering function clustering as in the following equation (2A) or (2B). Can be represented. Equation (2A) is for direct clustering of 3D data, and equation (2B) is for clustering after projecting onto 2D data. Here, the number of clusters n may be given in advance as the number of known objects in the input multi-viewpoint image.

As the clustering method, any existing method such as k-means method (k-means method) or the like may be used. Here, the distance (similarity) between the points to be clustered may be evaluated by the Euclidean distance or the like.

In the example of FIG. 4, as shown in [f], by applying the embodiment of Eq. (2B) with n = 2, clustering results o ₁ and o ₂ are obtained as corresponding to the target OBs 1 and OB 2. ing. In this way, as shown in [e], the clustering result corresponding to the target OB1 and OB2 can be obtained even in the original 3D point cloud data.

<Neighborhood acquisition unit 3>
The role of the neighborhood acquisition unit 3 is to obtain information on which point the neighborhood point of each point is, as information for enabling the extraction unit 5 on the latter stage side to appropriately extract the target area. .. The significance of this will be explained in the explanation of the extraction unit 5 described later. In the vicinity acquisition unit 3 specifically, three-dimensional points of the point cloud data (each individual target for each sorting unit 2 obtained in individual subjects o _i (as described above cluster o _i corresponds to the individual subject o _i) for each point t of data coordinates), a predetermined number k nearest neighbors pt _1, pt ₂ in the same cluster o _i, ..., it acquires information of pt _k (∈o _i). At this time, _{after calculating the distance from each point t to other points in the same cluster o i} , the upper k points having a short distance may be selected. That is, the processing of the neighborhood acquisition unit 3 can be formally expressed as the following equation (3) by the function find_nearest_point that finds the k-nearest neighbor of the point t in _{the cluster o i.} At this time, any existing k-nearest neighbor method (KNN) may be used.

_{In one embodiment of the present invention, the set {t, pt 1} , pt ₂ , ..., pt _k } of each point t obtained by the neighborhood acquisition unit 3 and its k neighborhood points forms a convex hull. Based on the above premise, the area extraction by triangulation is performed in the extraction unit 5 on the latter stage side. That is, in some cases, it may not be a convex hull (rarely), but based on such a premise, the region can be extracted by triangulation, and the accuracy of the extracted region can be ensured as a result.

By generating the point groups densely to some extent in the point group generation unit 1, the distance between each point t obtained by the neighborhood acquisition unit 3 and its k neighborhood points in the three-dimensional space can be determined. When it becomes smaller and is back-projected onto the image plane by the back-projection unit 4 on the rear stage side, the set {t, pt ₁ , pt ₂ ,…, pt _k } is back-projected to the same pixel. It may exist. Therefore, the value of k is assumed to be greater than 3, for example, so as to increase the possibility that at least 3 pixels from the set {t, pt ₁ , pt ₂ ,…, pt _{k} can be obtained by back projection (k ≧ 4).} ) Is desirable. For example, k = 6 or k = 7. In addition, in order to determine the area by triangulation in the extraction unit 5 on the latter stage side, at least three different pixels determined to be in the neighborhood by the neighborhood acquisition unit 3 are required.

In the example of FIG. 5, regarding the point cloud of a certain individual target OB1 as shown in [g], a certain point pt1 (one example of the point t) as shown in [h] as a part of the point cloud is enlarged and displayed. The six neighborhood points pt2 to pt7 obtained with k = 6 are shown. In this way, for all the points t belonging to the point cloud of the individual target OB1 shown in [g], the neighborhood acquisition unit 3 obtains the points in the vicinity thereof.

<Back projection 4>
The back projection unit 4 back-projects _{the point cloud data for each individual object o i} obtained from the sorting unit 2 onto the image plane corresponding to the virtual viewpoint set by the user. The result of the back projection can be expressed as a binary image in the image on the image plane (u, v) corresponding to the virtual viewpoint set by the user. That is, with respect to a certain pixel position (u, v) on the image plane, if at least one point belonging to the point cloud data is back-projected, the value is set to 1 (white), and there must be no back-projected point. For example, the output of the back projection unit 4 can be obtained as a binary image with a value of 0 (black).

^{Here, as is well known about the back projection process, by multiplying the perspective projection matrix T i} ₃₄ (3 rows and 4 columns) corresponding to the virtual viewpoint set by the user, as given by the following equation (4). It can be realized. ^{The superscript i} of T _{i 34} is the identifier of the virtual viewpoint camera. In Eq. (4), [x yz 1] and [ru rv r] (both are column vectors by transposition) are the points (x, y, z) of the point cloud to be back-projected and their images. It is a homogeneous coordinate representation of the pixel position (u, v) as a result of back projection on a plane.

In the example of [i] of FIG. 5, the result of the back projection by the back projection unit 4 with respect to the point cloud of a certain individual target OB1 is shown in the above-mentioned white or black binary image format.

<Extractor 5>
_{In the extraction unit 5, the point cloud data for each individual target o i} on the image corresponding to the virtual viewpoint set by the user obtained by the back projection unit 4 (as in the example of [i] in FIG. 5 on the image). Of the regions formed between the discrete and sparse points, the region determined to correspond to the region occupied by the _{individual target o i} is defined as the region formed between the discrete and sparse points. By filling in, the _{area occupied by the individual target o i} is extracted. When filling in, the information of k points determined to be near each point t by the neighborhood acquisition unit 3, that is, the information of the set {t, pt ₁ , pt ₂ , ..., pt _k } is referred to. Therefore, the set {t, pt ₁ , pt ₂ , ..., pt _k } determined to be in the vicinity of k is also projected onto the two-dimensional image by the back projection unit 4. with assumption that forms a convex hull that is included in the area occupied by the individual target o _i, extracting a region occupied by the individual target o _i by triangulation (process of filling the interior choose three vertices of the triangle) To do.

Specifically, the set (set of pixel positions on the image) obtained by back-projecting the set {t, pt ₁ , pt ₂ ,…, pt _{k } (in 3D space) by the back projection unit 4 is {I t.} , I ₁ , I ₂ ,…, I _k }, and the triangulation by any existing method may be applied to the _{point set {I t} , I ₁ , I ₂ ,…, I _k}. .. Triangulation can be formally expressed by the function triangulation as shown in Eq. (5) below. Here, the output T is a matrix of m rows and 3 columns, and each of the m rows (that is, equivalent to a row vector of size 3) stores an index representing the vertices of the constructed triangle. To do.

[J] of FIG. 5 shows an example of the triangulation in a local region on the image, and [k] shows the result of performing the local triangulation over the entire points related to the individual target OB1. As the sum of the above, an example in which the area of the individual target OB1 (person) is obtained is shown. The individual target OB1 as the person can see the area of the head on the upper side, the torso and one hand in the middle part, and the areas of both feet on the lower side. Here, by performing the triangulation based on the points determined to be in the vicinity as in Eq. (5), for example, the area of both lower feet is appropriate without filling the area located between both feet. Has been obtained in.

<Drawing part 6>
In the drawing unit 6, it is determined from the input multi-viewpoint image that the area (foreground) of each individual target in the image corresponding to the virtual viewpoint set by the user obtained in the extraction unit 5 is close to the virtual viewpoint. By performing texture mapping using one or more images, a free viewpoint image at the virtual viewpoint is generated. Here, with respect to the area (foreground) of each individual object, depth information is added to the input multi-view image in advance, or stereo matching is applied to the individual object in the multi-view image. As a result, it is possible to generate a billboard by giving depth information about the area (foreground) of the individual object obtained by the extraction unit 5, and then generate a free viewpoint image by an arbitrary existing method (bilboard method). At this time, the background information may be given as known in advance. For the generation of the free viewpoint image by the billboard method, for example, the above-mentioned Non-Patent Document 2 or the following may be used as an existing method.
HAYASHI, Kunihiko; SAITO, Hideo. Synthesizing free-viewpoint images from multiple view videos in soccer stadium. In: Computer Graphics, Imaging and Visualization, 2006 International Conference on. IEEE, 2006. p. 220-225.

As described above, according to the present invention, since the area of the individual object for obtaining the billboard for generating the free viewpoint image can be calculated at high speed and with high accuracy, the free viewpoint image is also generated at high speed and with high accuracy. It becomes possible. All or any part of the following can be mentioned as points that enable the high-speed and high-precision calculation.
(1) Instead of generating a visual hull as a complete three-dimensional model, the point cloud generation unit 1 obtains point cloud data that can be calculated at high speed as an approximation thereof.
(2) Since the point cloud data is generated based on the points at the boundary between the foreground and the background in each image of the multi-viewpoint image, the shape is accurately reflected even if it is an approximation of the visual hull. It has become.
(3) Without using the polyhedron model as in Non-Patent Document 3, it is possible to separate a plurality of individual objects being photographed at high speed by clustering in the sorting unit 2.
(4) Even if the individual object has a recess, it is possible to deal with it by using the information on the neighborhood points of each point obtained by the neighborhood acquisition unit 3.
(5) An individual target region can be obtained by a simple process of local triangulation in the extraction unit 5.

The present invention can also be provided as a program that causes a computer to function as an image processing device 10. A well-known hardware configuration such as a CPU (central processing unit), memory, and various I / Fs can be adopted for the computer, and the CPU executes instructions corresponding to the functions of each part of the image processing device 10. It will be. In addition, the computer is further equipped with a GPU (graphic processing unit) capable of performing parallel processing at a higher speed than the CPU, and instead of the CPU, the GPU reads a program for all or any part of the functions of the image processing unit 10. You may want to do it.

10 ... image processing device, 1 ... point group generating unit, 2 ... sorting unit, 3 ... neighborhood acquisition unit, 4 ... back projection unit, 5 ... extraction unit, 6 ... drawing unit

Claims (9)

A point cloud generator that obtains target point cloud data based on the viewpoint position in each image and the points on the boundary between the foreground and the background from the multi-view image.
A sorting unit that clusters the point cloud data into point cloud data for each individual object,
A neighborhood acquisition unit that acquires neighborhood points of each point in the point cloud data for each individual object,
A back-projection unit that back-projects the point cloud data for each individual object onto the image plane at the designated virtual viewpoint, and
By setting the neighborhood points of the acquired points as the neighborhood points to which the points on the image plane are connected, the region of the individual target in the image plane is extracted as the region defined by the connecting relationship. An image processing device including a unit and a unit. The point group generation unit uses one of the multi-viewpoint images as the target image and the other as the reference image, the first point on the foreground and background boundaries of the target image, and the optical center of the target image. The epipolar plane determined by the second point, which is the optical center of the reference image, cuts the line segment from the foreground region of the reference image, and the line of sight in the target image passing through the second and first points. The image processing apparatus according to claim 1, wherein the target point group is obtained from the line segment projected onto the image. The image processing apparatus according to claim 2, wherein the point cloud generation unit obtains the target point cloud, assuming that the points at both ends of the projected line segment are included. The point group generating unit selects the first point on the boundary between the foreground and the background in the target image at regular intervals at a distance on the image from the boundary, and then obtains the target point group. The image processing apparatus according to claim 2 or 3. The sorting unit according to any one of claims 1 to 4, wherein the point cloud data is clustered based on the positional relationship on the predetermined plane when the point cloud data is projected onto the predetermined plane. Characteristic image processing device. The extraction unit is characterized in that a region of an individual object in the image plane is extracted as the sum of triangular regions having a line segment formed between the acquired neighborhood points as a side for each point. The image processing apparatus according to any one of claims 1 to 5. Any of claims 1 to 6, further comprising a drawing unit that generates a free viewpoint image in the designated virtual viewpoint based on the multi-viewpoint image and the extracted individual target area. The image processing device described in Crab. A point cloud generation stage in which target point cloud data is obtained from a multi-viewpoint image based on the viewpoint position in each image and the points on the boundary between the foreground and the background.
A sorting step in which the point cloud data is clustered into point cloud data for each individual object,
In the neighborhood acquisition stage of acquiring the neighborhood points of each point in the point cloud data for each individual object,
A back-projection step in which the point cloud data for each individual object is back-projected onto the image plane at the specified virtual viewpoint, and
By setting the neighborhood points of the acquired points as the neighborhood points to which the points on the image plane are connected, the region of the individual target in the image plane is extracted as the region defined by the connecting relationship. An image processing method comprising: A point cloud generator that obtains target point cloud data based on the viewpoint position in each image and the points on the boundary between the foreground and the background from the multi-view image.
A sorting unit that clusters the point cloud data into point cloud data for each individual object,
A neighborhood acquisition unit that acquires neighborhood points of each point in the point cloud data for each individual object,
A back-projection unit that back-projects the point cloud data for each individual object onto the image plane at the designated virtual viewpoint, and
By setting the neighborhood points of the acquired points as the neighborhood points to which the points on the image plane are connected, the region of the individual target in the image plane is extracted as the region defined by the connecting relationship. An image processing program characterized in that a computer functions as an image processing device including a unit.