JP6201476B2 - Free viewpoint image capturing apparatus and method - Google Patents

Description

  The present invention relates to a free viewpoint image capturing apparatus and method for capturing a free viewpoint image of a subject in three dimensions.

In recent years, with the development of image processing technology and image communication technology, three-dimensional free viewpoint images have attracted attention as next-generation image content.
For this reason, a technique for generating an all-around free viewpoint image using a multi-viewpoint image captured by arranging a video camera around the subject has been developed.
Here, when creating a free viewpoint image based on a photographed image, a method of creating a three-dimensional shape of a subject in a three-dimensional space from a captured image and rendering the created three-dimensional shape is often used.

As a basic idea, a corresponding point is found between two images, and the coordinate position of the corresponding point in a three-dimensional space is measured by triangulation. By this processing, point group data of corresponding points in the three-dimensional space is obtained.
Next, after generating a polygon based on the point cloud data, a texture is generated (texture / object generation) by pasting a captured image to the polygon (performing texture mapping). Finally, based on the three-dimensional computer graphics technique, the texture object obtained by pasting the photographed image on the polygon is rendered to generate an image.

As another method for capturing a free viewpoint image, there is a method based on two-dimensional image processing using morphing. This morphing generally refers to a technique that makes it appear that a state of changing from an image of one subject to an image of another subject is naturally changed by an image processing technique.
By applying this morphing process to a viewpoint change for a three-dimensional object, a natural viewpoint change image can be obtained.

Here, the advantage of morphing is that a photorealistic image can be easily created with respect to a viewpoint change for a three-dimensional object.
That is, as already described, in the method of once creating a three-dimensional shape from a captured image, there is only one normal texture, and the texture does not change even if the observer's viewpoint is changed.
However, in general, the color of an object (subject) changes depending on the incident angle of light applied to the subject and the viewpoint direction of the observer with respect to the subject. For this reason, it is unnatural that the texture does not change when the viewpoint of the observer is changed. On the other hand, morphing produces a photorealistic image because the texture in the image changes as the viewpoint changes.

  One technique for generating a free viewpoint image is view morphing using morphing (see Non-Patent Document 1, for example). In this view morphing technique, when there is a virtual camera (hereinafter referred to as a virtual camera) between two cameras, an image viewed from the viewpoint of the virtual camera is maintained while maintaining the geometrical consistency of the camera. This is a technology that creates images by morphing the captured images of each of the two cameras. However, in the technique of Non-Patent Document 1, since the images that can be generated are limited between the viewpoints of two cameras, the degree of freedom of the viewpoint is low. Also, by using morphing, a photorealistic image can be created with respect to a change in viewpoint, but the quality of the created image depends on the accuracy of the corresponding point search. In general, it is difficult to perform a highly accurate and robust corresponding point search between two captured images.

In the search for corresponding points, it is a favorable condition for searching with high accuracy that the imaging directions of the two cameras are parallel and the imaging distance is short. However, even if a captured image of a virtual camera between two cameras having a parallel imaging direction and a short distance is generated by morphing processing, it is meaningless because the captured image only moves in parallel.
For this reason, in the technique of Non-Patent Document 1, when view morphing is performed, in order to create a captured image of a virtual camera between two cameras, the two cameras have a convergence angle and are separated from each other. The subject is imaged below.

Therefore, in Non-Patent Document 1, it is necessary to perform a difficult corresponding point search under the condition that the two cameras have a convergence angle and are separated.
When the accuracy of the corresponding point search is insufficient, only sparse corresponding points including outliers (incorrect corresponding points not actually corresponding) can be obtained between the two captured images. When view morphing is performed using the captured images of the two cameras in a state where the corresponding points are sparse, there is a problem that a region where no pixel exists in the morphing image is generated or the image is blurred. In addition, even if high-precision and dense corresponding point search succeeds, it is impossible to find corresponding points in the occlusion area. Therefore, in the drawing of the occlusion area, there is a problem that an area where pixels do not exist is generated. End up.

Several methods that have developed view morphing in Non-Patent Document 1 have been proposed. For example, since view morphing in Non-Patent Document 1 has a limitation of two viewpoints, a method of morphing between three viewpoints has been proposed (for example, see Non-Patent Document 2).
As a result, a more free viewpoint image can be generated for Non-Patent Document 1 on a plane formed by three viewpoints. However, even in the method of Non-Patent Document 2, the problem caused by the corresponding point search is not solved.

  If the corresponding point search result between the three viewpoints, that is, the captured images of the three cameras is sparse or too dense, morphing is performed by selecting an appropriate corresponding point and meshing with Delaunay triangulation. There has also been proposed a method of performing processing and drawing a captured image in a virtual camera (see, for example, Non-Patent Document 3). With the morphing process according to the method of Non-Patent Document 3, a morphing image in a virtual camera can be drawn without a hole. However, it does not solve the problem of outliers in correspondence points and the problem of occlusion.

  On the other hand, a technique for drawing an occlusion area has also been proposed (see, for example, Patent Document 4). However, in Patent Document 4, the occlusion area must be manually specified. Although it is possible to draw occlusion, it is not possible to draw fine details of a hidden part that is not captured.

As a method for generating a free viewpoint image using a three-dimensional shape, There is a method by Furukawa et al. (For example, see Non-Patent Document 5). In the method of Non-Patent Document 5, first, a captured image obtained by capturing an object (subject) from a plurality of viewpoints is prepared.
Next, feature amounts are extracted from the captured image using a feature amount extraction process such as SIFT (Scale Invariant Feature Transform). Then, using the extracted feature quantity, SFM (Structure from motion) and MVS (Multi View Stereo) are executed to restore the three-dimensional shape of the imaged subject. Here, SFM is a technique for simultaneously restoring the three-dimensional shape of the subject and the position of the camera from a plurality of captured images obtained by photographing a subject while changing the viewpoint of the camera. MVS is a method for generating an initial 3D model from camera positions and parameters restored by SFM and an input image.

When the above-described SIFT calculation is successful, many feature quantities can be obtained and a highly accurate three-dimensional shape can be restored.
On the other hand, it is difficult to find a feature point with high precision in SIFT, and it is often impossible to obtain a point group of feature points that are close enough to create a free viewpoint image.
Further, since many outliers are generated at points extracted as feature points from a plurality of captured images, it is necessary to remove them.

In addition, even when a dense feature point group cannot be obtained, a technique that enables drawing of a captured image has been proposed (Non-Patent Document 6).
In this non-patent document 6, when SFM and MVS are used, a point group that is obtained with high accuracy is rendered with high priority, and a point group with low accuracy is rendered with blurring to render smooth. The viewpoint is moved.
However, since this method only draws the point cloud by deceiving it when the viewpoint is moved, the area of the point cloud with low accuracy is only drawn in a blurred manner.

  VH (Visual Hull) is known as a method that can stably restore a three-dimensional shape (see, for example, Non-Patent Document 7 and Non-Patent Document 8). This VH captures captured images of a subject that is an object from a plurality of viewpoints, deletes the background in each captured image, and generates a silhouette image. Using the fact that the object to be restored is included in the silhouette, the common area of the silhouette is obtained in the three-dimensional space. Thereby, a region (partial space) where the subject exists in the three-dimensional space is found, and the region is restored as the three-dimensional shape of the subject.

In the methods of Non-Patent Document 7 and Non-Patent Document 8, only a silhouette is calculated without using feature points, so that a three-dimensional shape can be obtained relatively stably. However, the three-dimensional shape restored by VH (Visual Hull) has low accuracy, and even if it tries to apply a photographed image as a texture as it is, it will not work.
In order to solve this problem, a method of improving accuracy by combining VH and stereo matching (for example, see Non-Patent Document 9) and a method of deforming a three-dimensional shape based on a texture have been proposed (for example, Patent Document 1).

There is also a technique for creating a texture from images taken from a plurality of viewpoints by giving priority to the texture (see, for example, Patent Document 2).
However, since these are only textured, it is not possible to create photorealistic images for changing viewpoints like morphing.

JP 2009-94956 A Japanese Patent No. 4464773

S. Seitz, C.M. Dyer. "View morphing", SIGGRAPH 1996, 1996, pp. 21-30. XIAO J, SHAH M, “Tri-view morphing” Comput Vis Image Underst, Volume 96 Issue 3, December 2004 pp. 345-366 David Jelinek and C. J. Taylor “Quasi-Dense Motion Stereo for 3D View Morphing” International Symposium on Virtual and Augmented Architecture (VAA01) pp 219-229, June 2001. G. Chaurasia, O. Sorkine, G. Drettakis "Silhouette-Aware Warping for Image-Based Rendering" Computer Graphics Forum (Proceedings of the Eurographics Symposium on Rendering), Volume 30, Number 4-2011. Y. Furukawa and J. Ponce, “Accurate, Dense, and Robust Multi-View Stereopsis” CVPR 2007. M. Goesele, J.A. Ackermann, S.M. Fuhrmann, C.D. Haubold, R.D. Klowsky, D.C. Steedly, R.D. Szeliski “Ambient Point Clouds for View Interpolation” ACM Transactions on Graphics Proceedings of ACM SIGGRAPH 2010. W. Matusik C. Buehler R. Raskar S. Gortler L. McMillan “Image-Based Visual Hulls” SIGGRAPH 2000. S. Moezzi, L. Tai, and P. Gerard: Virtual View Generation for 3D Digital Video, IEEE Multimedia, pp. 18-26, 1997. N. Hiroyama, M. Katayama, Y. Iwabuchi, H. Imaizumi “Generation of 3D objects from multi-view images using visual volume intersection method and stereo matching method” Video Information Media 58 (6), 797-806, 2004- 06-01.

  The present invention has been made in view of such circumstances, and by morphing a captured image of a virtual camera corresponding to the viewpoint according to the viewpoint of observing the subject based on a plurality of captured images, An object of the present invention is to provide a free viewpoint image capturing apparatus and method for capturing a real free viewpoint image.

The free viewpoint image capturing device of the present invention uses an imaging control unit that acquires a first captured image of a subject from each of a plurality of imaging devices, and a three-dimensional shape of the subject that is reproduced from the first captured image. a corresponding point searching unit for searching the first corresponding point resolution for the first captured image acquired from the imaging device is a corresponding point of pixel between two different second imaging field image, from the plurality of imaging devices A template matching unit that performs template matching of each of the first corresponding points in each of the second captured image and another second captured image using any of the acquired second captured images as a template, and a plurality of the captured images A virtual camera image generation unit configured to generate a third captured image captured by a virtual camera disposed between the devices by morphing the second captured image. The features.

  In the free viewpoint image capturing device of the present invention, a virtual camera image generation unit performs the morphing process by a combination of the second captured images, and the virtual camera captures the combined second captured image. It has an imaging surface on the arranged line or plane.

  The free viewpoint image capturing device of the present invention detects each occlusion region of the image capturing device in the combination from a fourth captured image obtained by projecting the three-dimensional shape onto the image capturing surface of the virtual camera, and the detection result is the It further includes an occlusion area search unit that supplies the virtual camera image generation unit as occlusion information, and the virtual camera image generation unit reflects the occlusion information and performs a morphing process using the captured image of the combination. The third captured image is generated.

  The free viewpoint image capturing device of the present invention is characterized in that the resolution of the second captured image is higher than the resolution of the first captured image.

  The free viewpoint image capturing apparatus of the present invention generates a silhouette image generating unit that generates a silhouette image from the first captured image, and generates a temporary partial 3D shape that is a part of the 3D shape of the subject from the silhouette image. A stereo matching unit that performs stereo matching of each of the first captured images for each combination of the VF processing unit and the imaging device, and extracts a common corresponding point that is a common corresponding point in each of the first captured images; The three-dimensional shape is generated by combining the partial three-dimensional shape with the partial three-dimensional shape generation unit that corrects the temporary partial three-dimensional shape using the common corresponding points and makes the corrected result a partial three-dimensional shape. 5. The free viewpoint image capturing apparatus according to claim 1, further comprising a three-dimensional shape synthesizing unit.

  In the free viewpoint image capturing apparatus according to the present invention, the three-dimensional shape combining unit compares the triangular mesh connecting the common corresponding points with the temporary partial three-dimensional shape, and the direction of the image capturing apparatus is set with respect to the mesh. The temporary part three-dimensional shape region is deleted, and the temporary part three-dimensional shape is corrected.

  In the free viewpoint image capturing device of the present invention, the three-dimensional shape synthesis unit superimposes the partial three-dimensional images obtained from all combinations of the first captured images, and the portion 3 that is equal to or greater than a preset threshold value. The three-dimensional shape is generated from a region where the two-dimensional images overlap.

  The free viewpoint image capturing apparatus according to the present invention includes two image capturing apparatuses, the virtual camera is disposed on a line where the image capturing apparatus is disposed, and the coordinate Ev where the virtual camera is disposed is the image capturing apparatus. When the coordinates Ei of i (1 ≦ i ≦ 2) are used and expressed by the equation Ev = (1−α) E1 + αE2, (0 ≦ α ≦ 1), the pixel of the third captured image of the virtual camera In the coordinate Pv, the first corresponding point of the coordinate Pv is the coordinate Pi, the first corresponding point in the coordinate system of the third captured image is Pi, and Pj of the first captured image as a template in the template matching , Pj difference Dij (i = 1, 2, j = 1, 2, i ≠ j) is expressed by a combination of the imaging device 1 with respect to the coordinate Pv in the coordinate system of the third captured image. The first corresponding point is P1 Arufadi12, first corresponding point of the image pickup apparatus 2 is characterized by comprising a P2 + (1-α) D21.

  The free viewpoint image capturing apparatus according to the present invention includes three image capturing apparatuses, the virtual camera is disposed on a surface where the image capturing apparatus is disposed, and a coordinate Ev where the virtual camera is disposed is an image capturing position. When the coordinates Ei of the device i (1 ≦ i ≦ 3) are used and the virtual camera is expressed by the following equation: Ev = (1−α−β) E1 + αE2 + βE3 (0 ≦ α, 0 ≦ β, α + β ≦ 1) The coordinate Pv of the pixel in the third captured image is the coordinate Pi of the first corresponding point of the coordinate Pv, the Pi is the first corresponding point in the coordinate system of the third captured image, and the template in the template matching As a combination of Pj of the first captured image and the difference Dij of Pj with respect to Pi (i = 1, 2, 3, j = 1, 2, 3, i ≠ j). Paired with the coordinate Pv in the image coordinate system The first corresponding point of the imaging device 1 is P1 + αD12 + βD13, the first corresponding point of the imaging device 2 is P2 + (1−α−β) D21 + βD23, and the first corresponding point of the imaging device 3 is P3 + (1−α−β) D31 + αD32. It is characterized by becoming.

  In the free viewpoint image pickup device of the present invention, the number of the image pickup devices is four, the virtual camera is arranged on the surface on which the image pickup devices are arranged, and the coordinate Ev where the virtual camera is arranged is an image pickup device. Using the coordinates Ei of the device i (1 ≦ i ≦ 4), Ev = (1−α−αβ) E1 + α (1−β) E2−αβE3 + αβE4 (0 ≦ α, 0 ≦ β, α + β ≦ 1) When represented, the coordinate Pv of the pixel in the third captured image of the virtual camera is that the first corresponding point of the coordinate Pv is the coordinate Pi, and the first corresponding point in the coordinate system of the third captured image is Pi, Pj of the first captured image as a template in the template matching, and a difference Dij (i = 1, 2, 3, 4, j = 1, 2, 3, 4, i ≠ j) with respect to Pi When the third captured image is represented by a combination of The corresponding point of the imaging device i with respect to the coordinate Pv in the standard system is (1−α + αβ) P′i1 + α (1−β) P′i2−αβP′i3 + αP′i4 (where P′ii = Pi). And

  In the free viewpoint image pickup device of the present invention, each of the image pickup devices is arranged with the image pickup direction as the subject on the surface of a sphere centered on the subject so that the distance to the subject is the same. It is characterized by.

In the free viewpoint image capturing method of the present invention, the image capturing control unit acquires the first captured image of the subject from each of the plurality of image capturing apparatuses, and the corresponding point search unit is reproduced from the first captured image. using the three-dimensional shape of the object, corresponding to search first corresponding point resolution on the obtained first captured image is a corresponding point of pixel between two different second imaging field image from the imaging device The point search process and the first correspondence in each of the second captured image and the other second captured image, using any one of the second captured images acquired from the plurality of imaging devices as a template by the template matching unit. a template matching process of performing point each template matching, the third captured virtual camera image generation unit, which is captured from a virtual camera disposed between the plurality of the imaging device An image, characterized in that it comprises a virtual camera image generation step of generating a morphing process of the second image.

  According to the present invention, when a free viewpoint image is generated by morphing a plurality of captured images, template matching is performed in the local region in the morphing process, and the first corresponding point between the captured images is corrected. The blur caused by morphing can be reduced, and a captured image of a virtual camera corresponding to the viewpoint can be captured as a photoreal free viewpoint image according to the viewpoint of observing the subject.

It is a figure which shows the structural example of the dome shape imaging device used for the free viewpoint image imaging device by one Embodiment of this invention. FIG. 6 is a conceptual diagram illustrating a configuration example of a free viewpoint image capturing device, and illustrating only a support column 501_3 in a frame 500; It is a figure which shows the example of attachment of the imaging device in each support | pillar which comprises the dome frame. It is a figure which shows the structural example of the free viewpoint image imaging part 10 in FIG. It is a figure explaining the process which produces | generates a silhouette image from the color information of a 1st captured image. It is a figure which shows the production | generation of the three-dimensional shape from the several silhouette image of the to-be-photographed object from which an imaging direction differs. It is a figure explaining the production | generation of the group of the imaging device which consists of 2 or more units | sets. It is a figure which shows the result of the extraction of the common corresponding point which the stereo matching part 105 performed using SAD (Sum of Absolute Difference). FIG. 9 is a diagram showing a triangular mesh created by using the Delaunay triangulation method with the common corresponding point shown in FIG. It is a figure which shows the triangular mesh in the three-dimensional space which the three-dimensional shape production | generation part 106 produced | generated. It is a figure which shows the partial three-dimensional shape produced | generated based on the comparison result of the temporary three-dimensional shape which the three-dimensional shape production | generation part 106 performed, and the three-dimensional shape which a mesh forms. Fourteen image pickup devices provided in the dome-type image pickup device are combined in units of four to generate nine first image pickup device sets, and using the partial three-dimensional shape of each of the first image pickup device sets, It is a figure which shows the production | generation of a three-dimensional shape. It is a figure explaining the process which searches the occlusion area | region which the occlusion area search part 109 performs. It is a figure explaining the process of the template matching between the 1st captured images which the template matching part 110 performs. It is a figure which shows the arrangement | positioning possible range of a virtual camera, when performing the morphing process which produces | generates the virtual captured image which is a captured image of a virtual camera in the imaging device which comprises a 1st imaging device group. FIG. 4 is a diagram for describing selection of an imaging device used for morphing processing corresponding to the arrangement of the imaging device in the dome type imaging device shown in FIG. 3. It is a figure which shows the coordinate Ev of the virtual camera v at the time of using three imaging devices, the imaging device 1, the imaging device 2, and the imaging device 3. FIG. It is a figure explaining the weighting coefficient in (20) Formula which calculates | requires the color Cv of the pixel Pv of the virtual captured image of the virtual camera v. It is a figure which shows the relationship of the corresponding point in the 1st captured image which each imaging device 1, 2, and 3 imaged, and the virtual captured image which the virtual camera v imaged. It is a figure which shows the coordinate Ev of the virtual camera v at the time of using three imaging devices, the imaging device 1, the imaging device 2, and the imaging device 3. FIG. It is a figure which shows a response | compatibility of the 1st common point in each 1st captured image of the imaging devices 1, 2, 3, and 4 in the two-dimensional plane of a virtual captured image. It is a figure explaining the weighting coefficient in (27) Formula which calculates | requires the color Cv of the pixel Pv of the virtual captured image of the virtual camera v. It is a flowchart which shows the operation example which the free viewpoint image imaging device by this embodiment produces | generates a free viewpoint image by carrying out the morphing process of several 1st captured images. It is a figure which shows the 1st captured image imaged with the 1st imaging device group of the dome shape imaging device. It is a figure which shows the free viewpoint image produced | generated using View Morphing (trademark) mounted by OpenCV (trademark) in an open source library. It is a figure which shows the free viewpoint image which produced | generated the mesh by Delaunay triangulation method using SAD as a stereo matching method, and synthesize | combined this mesh. It is a figure which shows the free viewpoint image produced | generated by the morphing process with respect to the result of calculating | requiring the three-dimensional shape by Visual Hull. It is a figure which shows the free viewpoint image produced | generated by the morphing process which calculated | required 3D shape by Visual Hull, performed stereo matching with this 3D shape and each 1st captured image, and calculated | required the common corresponding point. A three-dimensional shape is obtained by Visual Hull, and stereo matching is performed between the three-dimensional shape and each of the first captured images, and the template matching between the first captured images is performed using the three-dimensional shape after obtaining the common corresponding points. It is a figure which shows the free viewpoint image produced | generated by the morphing process using the 1st corresponding point obtained by performing.

In the embodiment according to the present invention, the following three conditions are satisfied in order to solve the problem.
a. Dense corresponding points that do not include outliers are obtained regardless of the result of the corresponding point search between captured images to be morphed.
b. A free viewpoint image can be drawn by detecting an occlusion area of each imaging apparatus used for occlusion imaging.
c. A free viewpoint image as an imaging result of the virtual camera is generated from a captured image captured by an imaging device near the virtual camera by morphing processing.

Embodiments of the present invention will be described below with reference to the drawings. FIG. 1 is a diagram showing a configuration example of a dome type imaging apparatus used in a free viewpoint image imaging apparatus according to an embodiment of the present invention.
A subject 600 to be imaged is arranged in the center of the hemispherical dome frame 500. The dome frame 500 has a structure in which the imaging device 53_1 to the imaging device 59_4 are arranged on a hemispherical surface having the subject 600 as a center point. The dome frame 500 is composed of a plurality of support posts, in this embodiment, 12 support posts 500_1 to 500_12. In addition, a plurality of imaging devices (four imaging devices in the present embodiment) are provided in the columns 500_3, 500_5, 500_7, and 500_9 with predetermined intervals along the vertical direction of the columns. Accordingly, the dome frame 500 is provided with 16 imaging devices. Further, each of the columns 500_1 to 500_n is provided at intervals of 15 degrees in a dome frame 500 on a hemisphere centered on the subject 600.

FIG. 2 is a conceptual diagram showing a configuration example of the free viewpoint image capturing apparatus, and showing only the support column 500_3 in the frame 500. The free viewpoint image capturing apparatus according to the present embodiment includes the dome-shaped image capturing apparatus and the free viewpoint image capturing unit 10 shown in FIG. In FIG. 2, an LED (Light Emitting Diode) light source 800 irradiates a subject 600 with light by a light guide 850_3 provided on a column 500_3.
The blue back 900 is disposed on the dome frame 500 side with the subject interposed therebetween, and is provided so that the background of the subject 600 in the captured image is blue when the subject is imaged by each imaging device.

  Each of the imaging devices 53_1, 53_2, 53_3, and 53_4 is provided on an arc-shaped column 501_3 centered on the subject 600, and the distance between the imaging surface and the subject 600 is the same. Although not shown, each of the imaging devices 53_1, 53_2, 53_3, and 53_4 is attached to the column 500_3 via a free pan head. Therefore, fine adjustment of the optical axis of each imaging device can be easily performed, and processing for easily aligning the optical axis of each imaging device with the subject 600 can be easily performed. In addition, each imaging apparatus has a common world coordinate system in a three-dimensional space because the mounting position and the angle of the imaging direction are calibrated and known in advance. The coordinate values of the three-dimensional shape described below are coordinates in the world coordinate system.

In addition, each of the imaging devices 53_1, 53_2, 53_3, and 53_4 is controlled by the free viewpoint image imaging unit 10 to capture a captured image.
The free viewpoint image capturing unit 10 generates a free viewpoint image from images captured by the image capturing apparatuses provided in the dome-shaped image capturing apparatus of FIG. 1 including each of the image capturing apparatuses 53_1, 53_2, 53_3, and 53_4.
The free viewpoint image capturing apparatus according to the present embodiment includes the dome-shaped image capturing apparatus and the free viewpoint image capturing unit 10 shown in FIG.

  FIG. 3 is a diagram illustrating an example of attaching the imaging device to each support column constituting the dome frame 500. FIG. 3 (a) shows a configuration in which 16 image pickup devices are attached to each of four support columns, four of which are arranged at substantially equal intervals. That is, in FIG. 3A, four imaging devices are arranged on each of the columns 500_3, 500_5, 500_7, and 500_9 shown in FIG. Imaging devices 53_1, 53_2, 53_3, and 53_4 are arranged on the column 503. Imaging devices 55_1, 55_2, 55_3, and 55_4 are arranged on the support column 505. Imaging devices 57_1, 57_2, 57_3, and 57_4 are arranged on the column 507. Imaging devices 59_1, 59_2, 59_3, and 59_4 are arranged on the column 509.

  Further, FIG. 3B shows a configuration in which 16 image pickup devices are attached and arranged in a zigzag pattern for each adjacent support column. That is, in FIG. 3B, two imaging devices are arranged on each of the columns 500_3, 500_4, 500_5, 500_6, 500_7, 500_8, and 500_9 shown in FIG. Imaging devices 53_1 and 53_2 are provided on the column 500_3. Imaging devices 54_1 and 54_2 are provided on the support post 500_4. Imaging devices 55_1 and 55_2 are provided on the column 500_5. Imaging devices 56_1 and 56_2 are provided on the column 500_6. Imaging devices 57_1 and 57_2 are provided on the column 500_7. Imaging devices 58_1 and 58_2 are provided on the column 500_8. Imaging devices 59_1 and 59_2 are provided on the column 500_9. In addition, the interval between the imaging devices provided on one support is the same as the interval between the imaging devices attached to each support. The attached heights of the two imaging devices attached to the column 500_3 are the same as the attached heights of the two imaging devices attached to the columns 500_5, 500_7, and 500_9. In addition, the attached heights of the two imaging devices attached to the column 500_4 are the same as the attached heights of the two imaging devices attached to the columns 500_6 and 500_8.

  FIG. 4 is a diagram illustrating a configuration example of the free viewpoint image capturing unit 10 in FIG. In FIG. 4, a free viewpoint image capturing unit 10 includes an image capturing control unit 101, a silhouette image generating unit 102, a VF processing unit 103, an image capturing device selecting unit 104, a stereo matching unit 105, a 3D shape generating unit 106, and a 3D shape. A synthesis unit 107, a corresponding point search unit 108, an occlusion area search unit 109, a template matching unit 110, a virtual camera setting unit 111, a morphing imaging device selection unit 112, a virtual camera image generation unit 113, and a database 114 are provided.

  The imaging control unit 101 reads a captured image from an imaging device provided on each column of the dome frame 500 and temporarily writes and stores it in the database 114 together with identification information of the imaging device of the corresponding imaging device. Here, the imaging control unit 101 sets a low-resolution first captured image by thinning out pixels of the captured image from each imaging device and a second captured image with the same resolution as the captured image without thinning out pixels. Remember me.

The silhouette image generation unit 102 generates a silhouette image for each of the first captured images from each of the second captured images using a color filter. Here, as described with reference to FIG. 3, the blue background 900 is provided in the background when imaging is performed in the imaging apparatus, and the background of the captured image is captured as blue background blue because of the necessity of clipping the background. ing.
The silhouette image generation unit 102 can easily cut out the background using the color information of the background because the background of the captured image is captured as a blue-blue background when the silhouette image is generated.

  FIG. 5 is a diagram for explaining processing for generating a silhouette image from color information of the first captured image. FIG. 5A shows the first captured image, and shows that the background of the subject 600 is blue. FIG. 5B shows a silhouette image in which only the silhouette of the subject 600 is obtained by cutting out the background by using a color filter based on the blue color information. The silhouette image of FIG. 5B is generated from all the captured images of the imaging devices provided on the respective columns of the dome frame 500, in this embodiment, all the 16 first captured images.

  Returning to FIG. 4, the VF processing unit 103 generates a provisional three-dimensional shape from the 16 silhouette images generated by the silhouette image generation unit 102 using the method of Visual_Hull. This Visual_Hull is a product space of a cone formed by silhouette images generated from captured images captured from a number of directions and the optical axis of each imaging device, so that the subject 600 is assumed to be a rough three-dimensional shape. A three-dimensional shape is generated. In addition, Visual_Hull can stably obtain a provisional three-dimensional shape without being greatly affected by the imaging conditions.

  FIG. 6 is a diagram illustrating generation of a three-dimensional shape from a plurality of silhouette images of subjects having different imaging directions. FIG. 6A shows silhouette images of a plurality of subjects with different imaging directions generated by the silhouette image generation unit 102. FIG. 6B shows a provisional three-dimensional shape generated from the silhouette image of FIG. The provisional three-dimensional shape obtained by Visual_Hull has low feature point accuracy. For this reason, even if the corresponding points between the first captured images subjected to the morphing process are obtained based on the provisional three-dimensional shape, the accuracy required for the morphing process cannot be obtained.

  For this reason, the provisional three-dimensional shape is corrected by a stereo matching method described later. However, in general, the corresponding points found as a result of stereo matching are sparse corresponding points and include outliers, and thus cannot be used for correction of the provisional three-dimensional shape as they are. Therefore, the following stereo matching process is performed.

Returning to FIG. 4, the imaging device selection unit 104 generates a set of a plurality of imaging devices by setting two or more imaging devices provided in the dome frame 500 as a set.
FIG. 7 is a diagram for explaining generation of a first imaging device set that is a set of imaging devices including two or more devices. FIG. 7A shows a case in which a set of four adjacent imaging devices is combined. For this reason, in the case of this embodiment, since there are 16 imaging devices, nine combinations are generated. FIG. 7A corresponds to the arrangement of the imaging device on the support of the dome frame 500 shown in FIG. FIG. 7B shows a case where three adjacent imaging devices are used as one set of first imaging devices. For this reason, in the case of this embodiment, since there are 16 imaging devices, 15 combinations are generated. FIG. 7B corresponds to the arrangement of the imaging device on the support of the dome frame 500 shown in FIG. The combination of the imaging devices can be freely selected, but the captured images used for stereo matching can be obtained with good results when the images captured by the imaging devices located at close positions are possible. FIG. 7A and FIG. As shown in b), adjacent imaging devices are selected as the first imaging device group.

The stereo matching unit 105 performs stereo matching of common corresponding points between the first captured images captured by the imaging devices included in the first imaging device set for each first imaging device set generated by the imaging device selection unit 104. . That is, the stereo matching unit 105 selects any one of the first captured images as a template from the first captured images captured by the imaging devices included in the first imaging device group, and the other first captured image. Matching is performed on the image to detect corresponding points, and common corresponding points are obtained. This stereo matching is performed between the first captured images captured for each first imaging device group in which the imaging devices are combined. In addition, various methods have been proposed for stereo matching, and better results can be obtained by using a highly accurate matching method.
However, in the present embodiment, the corresponding points are sparse and the outliers may be included. Therefore, in order to show the robustness of the present embodiment, SAD (Sum of Absolute Difference), which is one of the most basic matching methods, was used in the experiment using the dome-type imaging device.

SAD cuts out a local area of one captured image as a template, compares the luminance difference between the pixel of this template and the pixel of another captured image in pixel units, and calculates the absolute value of the luminance difference of the pixel in the local area. The local region with the smallest sum is taken as the corresponding point. With this SAD, the stereo matching unit 105 extracts a common corresponding point for all captured images included in this set for each first imaging device set.
For example, when the first imaging device set includes three imaging devices 53_A, 53_B, and 53_C illustrated in FIG. 7B, the stereo matching unit 105 includes the three imaging devices. , Two imaging devices 53_A and 53_B and imaging devices 53_A and 53_C are selected.

  The stereo matching unit 105 searches for common corresponding points on the epipolar line using either of the imaging devices 53_A and 53_B as a template. Here, the stereo matching unit 105 generates a template from the first captured image of the imaging device 53_A, executes SAD with each of the imaging devices 53_B and 53_C, and extracts corresponding points. At this time, if the extracted common corresponding points are on (or existed on) the epipolar lines in the first captured images of the imaging devices 53_B and 53_C, the stereo matching unit 105 commonly handles the extracted corresponding points. Let it be a point.

  In addition, as shown in FIG. 7A, when the first imaging device set is configured by each of the four imaging devices 53_A, 53_B, 53_C, and 53_D, the three imaging devices that have already been described The same processing as in the case of one imaging device group is performed. That is, the stereo matching unit 105 generates a template from the first captured image of the imaging device 53_A, executes SAD with each of the imaging devices 53_B, 53_C, and 53_D using this template, and extracts corresponding points. When the corresponding points extracted from the first captured images of the imaging devices 53_B, 53_C, and 53_D are on the epipolar line, the stereo matching unit 105 sets the extracted corresponding points as common corresponding points.

  FIG. 8 is a diagram illustrating a result of common corresponding point extraction performed by the stereo matching unit 105 using SAD. FIG. 8A shows the first captured images 200A, 200B, 200C, and 200D of the imaging devices 53_A, 53_B, 53_C, and 53_D, respectively. FIG. 8B shows common corresponding points in the first captured image 200A that are extracted from the first captured image 200B, 200C, and 200D by generating a template from the first captured image 200A. In FIG. 8B, the common corresponding points with each of the first captured images 200B, 200C, and 200D on the first captured image 200A are indicated by black circles. From FIG. 8B, it can be seen that the common corresponding points can be extracted only in a sparse state. The common corresponding points shown in FIG. 8B are generated for each first imaging device set.

Returning to FIG. 4, the three-dimensional shape generation unit 106 uses the common corresponding points in FIG. 8B, and corrects the temporary three-dimensional shape in FIG. 6B generated by the VF processing unit 103 using VH (Visual Hull). To generate a partial three-dimensional shape.
The three-dimensional shape generation unit 106 uses a common corresponding point obtained by stereo matching as a vertex, and creates a triangular mesh using Delaunay triangulation.
FIG. 9 is a diagram showing a triangular mesh created using the Delaunay triangulation method with the common corresponding point shown in FIG. 8B as a vertex. In FIG. 9, since the first captured image 200A is a captured image in which a template is generated, a triangular mesh having a vertex corresponding to the common corresponding point on the first captured image 200A is generated.

  Returning to FIG. 4, the three-dimensional shape generation unit 106 reads camera parameters from the camera parameter table of the database 114, and uses the camera parameters (such as the position and focal length of the imaging device) to capture the first captured images of the three imaging devices. The coordinates of the common corresponding point in the three-dimensional space are obtained by triangulation. The camera parameter table is written and stored in advance in the database 114 for each combination of the types of imaging devices in different types of imaging devices and dome type imaging devices.

Then, the three-dimensional shape generation unit 106 associates the triangular mesh in FIG. 9 with the common corresponding point for which the coordinates in the three-dimensional space are obtained, and arranges the triangular mesh in the three-dimensional space.
FIG. 10 is a diagram illustrating a triangular mesh in the three-dimensional space generated by the three-dimensional shape generation unit 106. In FIG. 10, the three-dimensional shape formed by the triangular mesh described above is a shell shape surrounding the temporary three-dimensional shape generated by the VF processing unit 103.

  Returning to FIG. 4, the three-dimensional shape generation unit 106 compares the three-dimensional shape formed by the triangular mesh shown in FIG. 10 with the provisional three-dimensional shape shown in FIG. Here, the three-dimensional shape generation unit 106, when a region having a provisional three-dimensional shape is close to the imaging device in the imaging device direction with respect to the three-dimensional shape of the mesh, the three-dimensional shape of the mesh (formation of a triangular mesh) A region projecting in the direction from the shell-shaped figure) to the imaging device is removed from the provisional three-dimensional shape. Then, the three-dimensional shape generation unit 106 generates a temporary three-dimensional shape from which the protruding region is removed as a partial three-dimensional shape that is a corrected temporary three-dimensional shape. Here, since it is a premise that the imaging devices are all calibrated at the position coordinates in the three-dimensional space, the relative values of the coordinates and orientations of all the imaging devices are known. For this reason, all the imaging devices have a common coordinate system (world coordinate system), and the common corresponding point for each first imaging device set is the world coordinate as the coordinates of the vertex of the three-dimensional shape composed of the common coordinate points. It will be mapped to the system.

FIG. 11 is a diagram illustrating a partial three-dimensional shape generated based on a comparison result between the temporary three-dimensional shape performed by the three-dimensional shape generation unit 106 and the three-dimensional shape formed by the mesh. Each of the partial three-dimensional shapes in FIGS. 11A, 11B, and 11C lacks a part of the shape because the corresponding points include outliers in the result of stereo matching. It becomes a shape.
As described above, the partial three-dimensional shape data shown in FIG. 11 is generated by the number of the first imaging device sets. That is, there are as many pieces of partial three-dimensional shape data as there are first imaging device groups. If stereo matching is performed in each of the n first imaging device groups, n pieces of partial three-dimensional data are generated. The partial three-dimensional shapes shown in FIG. 11A, FIG. 11B, and FIG. 11C are generated by different first imaging device groups, and it can be seen that the shapes differ depending on the groups.

Returning to FIG. 4, the three-dimensional shape synthesis unit 107 synthesizes the partial three-dimensional shape generated by the three-dimensional shape generation unit 106 for each first imaging device set.
The three-dimensional shape synthesis unit 107 superimposes all the partial three-dimensional shapes generated by the three-dimensional shape generation unit 106 on common corresponding points having the same coordinate value in the above-described world coordinate system, and generates the partial three-dimensional shape. Perform the process of overlapping. At this time, the three-dimensional shape synthesizing unit 107 subdivides the space of the superimposed partial three-dimensional image, and detects how many partial three-dimensional shapes overlap each other for each subdivided region.

  Then, the three-dimensional shape synthesis unit 107 determines, for each subdivided region, whether or not the number of partial three-dimensional shapes overlapping in each region is equal to or greater than a preset threshold value. Here, if the number of partial three-dimensional shapes overlapping in the subdivided area is equal to or greater than the threshold value, the three-dimensional shape synthesis unit 107 determines that a three-dimensional shape exists in the area. On the other hand, when the number of partial three-dimensional shapes overlapping in the subdivided area is less than the threshold value, the three-dimensional shape synthesis unit 107 determines that no three-dimensional shape exists in the area. Then, the three-dimensional shape synthesis unit 107 synthesizes the three-dimensional shape based on the pixels in the region where it is determined that the three-dimensional shape exists in the synthesized image obtained by superimposing a plurality of temporary three-dimensional shapes.

  In FIG. 12, 16 image pickup devices provided in the dome-type image pickup device are combined four by four to generate nine first image pickup device sets, and the partial three-dimensional shape of each of the first image pickup device sets is generated. FIG. 3 is a diagram illustrating generation of a three-dimensional shape of a subject used. FIG. 12A shows a partial three-dimensional shape generated by each first imaging device group. FIG. 12B shows a synthesized three-dimensional shape when nine partial three-dimensional images are overlapped and two or more partial three-dimensional images that are threshold values overlap each other in a subdivided region. ing. By superimposing the partial 3D images and combining the regions where the data of the 3D shape exceeding the threshold exists, the 3D shape is generated, so that the temporary 3D shape and the triangular mesh are formed. Many outliers can be removed, including shape comparisons.

Returning to FIG. 4, the corresponding point search unit 108 searches for a first corresponding point that is a corresponding point in the second captured image, using the combined three-dimensional shape shown in FIG. Since the morphing using the second captured image is performed based on the first corresponding point, the search for the first corresponding point is performed on the high-resolution second captured image used for morphing. The second captured image has a higher resolution than the first captured image, and is a captured image in a state (resolution) captured by the imaging device provided in the dome-shaped imaging device. This second captured image is used for a morphing process to be described later.
The corresponding point search unit 108 reads out the camera parameters of each imaging device from the database 114, and extracts corresponding points between the second captured images of the imaging devices by three-dimensional geometric calculation (triangulation calculation). . Here, in the search for the first corresponding point, the corresponding point search unit 108 sets the three-dimensional shape to the two-dimensional plane perpendicular to the imaging direction of each imaging device, that is, the two-dimensional plane corresponding to the imaging surface of each imaging device. This is performed using a virtual first captured image obtained by projecting each pixel having a three-dimensional shape (described later).

Morphing is possible by searching for the first corresponding point described above and using the search result of the first corresponding point. However, since the accuracy of the three-dimensional shape obtained by combining is not high, it is not possible to obtain an exact corresponding point between the three-dimensional shape and the second imaging device. Here, if the morphing process using inaccurate corresponding points is performed, the image generated by the morphing process is blurred or blurred after performing the morphing process due to the error of the corresponding points Occurs.

  In order to solve this problem, the corresponding point search unit 108 performs template matching in the local region between the three-dimensional shape and the second captured image in order to search for the first corresponding point. Here, the accuracy of the synthesized three-dimensional shape is not high because the first captured image is used, but the outlier has been removed by the method already described, so that a large error has occurred. Absent. For this reason, the error between the first corresponding point searched by the corresponding point search unit 108 in the first captured image and the three-dimensional shape is not a large value, and this error is corrected by performing template matching in the local region. Can do.

  Returning to FIG. 4, the corresponding point search unit 108 performs template matching with the three-dimensional shape on the first captured images captured by all the imaging devices included in the first imaging device group illustrated in FIG. 7. Here, the corresponding point search unit 108 projects a three-dimensional shape onto a two-dimensional plane parallel to the imaging direction of each imaging device included in the first imaging device group, and is a virtual first captured image. A virtual first captured image is generated. Thereby, the corresponding point search unit 108 virtually captures the synthesized three-dimensional shape corresponding to the first captured image captured by each of the imaging devices included in the first imaging device group. Generate each of the images. The corresponding point search unit 108 performs local template matching between the first captured image captured by the imaging device and the virtual first captured image corresponding to the imaging device.

  Thus, the corresponding point search unit 108 performs template matching between each of the pixels of the first captured image and each of the pixels of the virtual first captured image, and each of the pixels of the first captured image and the virtual first captured image. The corresponding relationship with each of the pixels is extracted. At this time, since the corresponding point search unit 108 knows the coordinates of each pixel projected on the virtual first captured image in the three-dimensional space, it detects the coordinates of each pixel of the first captured image in the three-dimensional space. be able to. The corresponding point search unit 108 performs the above-described processing on all the first imaging device groups, and extracts the correspondence between each pixel of the three-dimensional image and each pixel of the first captured image.

  The occlusion area search unit 109 searches for an occlusion area when the corresponding point search unit 108 searches for a corresponding point with respect to a surface point of a three-dimensional shape. This occlusion area indicates a case where no corresponding point is extracted due to an obstacle. Therefore, when searching for the first corresponding point between the two second captured images, the occlusion area searching unit 109 blocks between any one of the imaging devices and the corresponding point of the three-dimensional shape (three-dimensional The occlusion area is detected by the presence or absence of the shape itself. At this time, the occlusion area search unit 109 detects where the occlusion occurs by recording which imaging device can capture an image and which imaging device cannot capture an image.

  That is, the occlusion area search unit 109 detects pixels that exist in the three-dimensional shape and do not exist in the virtual first captured image for each virtual first captured image on which the corresponding point search unit 108 projects the three-dimensional shape. Each device is written and stored in the database 114. Thereby, when performing morphing, it is possible to detect pixels in a three-dimensional shape that are not captured in the first captured image captured by the imaging device.

  FIG. 13 is a diagram for explaining the process of searching for an occlusion area performed by the occlusion area searching unit 109. In FIG. 13, when searching for the first corresponding point, the first corresponding point 70 (pixel) is imaged from the imaging device 53_A. On the other hand, with respect to the first corresponding point 70 from the imaging device 53_B, a part of the three-dimensional shape 600 ′ becomes an obstacle (easily detected from the above-described virtual first captured image), and the imaging device 53_B and the first corresponding point 70 are detected. The optical axis of the imaging device on the line connecting the two is blocked, and the first corresponding point 70 is not imaged from the imaging device 53_B.

  Returning to FIG. 4, the template matching unit 110 performs template matching between first captured images captured by all of the imaging devices included in each first imaging device set in each first imaging device set. Here, since the template matching unit 110 does not take into account the geometric consistency of the imaging device, the template matching unit 110 does not perform stereo matching but only performs template matching between the first captured images. As a result, when geometric matching is pursued, the corresponding pixel cannot be detected due to the restriction of the matching, or the color between the pixels (for example, RGB (Red / Green / Blue) gradation degree). ) Can be reduced. Although it is desirable to achieve geometric matching, it is generally difficult to satisfy the geometric matching and detect corresponding points for each pixel between captured images.

For example, when the first imaging device set includes three imaging devices 53_A, 53_B, and 53_C as illustrated in FIG. 5B, the template matching unit 110 performs the first imaging that is captured by each imaging device. Template matching between layers is performed by the following combinations. That is, the template matching unit 110
F1: Template matching for the first captured image of the imaging device 53_B using the first captured image of the imaging device 53_A as a template F2: Template matching for the first captured image of the imaging device 53_A using the first captured image of the imaging device 53_B as a template F3: Template matching for the first captured image of the imaging device 53_A using the first captured image of the imaging device 53_C as a template F4: Template matching for the first captured image of the imaging device 53_A using the first captured image of the imaging device 53_C as a template F5: Template matching for the first captured image of the imaging device 53_C using the first captured image of the imaging device 53_B as a template F6: Imaging device 53_ using the first captured image of the imaging device 53_C as a template Therefore, when the number of image pickup devices constituting the first image pickup device group is 2, the number of image pickup devices is twice. If there are four, twelve template matchings are performed for each first imaging device group.

  Further, the template matching unit 110 sets the template as a predetermined area centered on the corresponding point (the pixel already extracted by the corresponding point searching unit 108), and sets the search range only in the vicinity of the corresponding point. Here, as already shown, since the corresponding point search unit 108 has already extracted the corresponding points between the three-dimensional shape and the first captured image, the pixels of the first detection image are pixels of any three-dimensional shape. It is known whether it corresponds to. For this reason, when performing template matching, the template matching unit 110 uses the coordinates of the corresponding point pixel of the three-dimensional shape detected by the corresponding point search unit 108 as a reference.

  FIG. 14 is a diagram illustrating template matching processing between first captured images performed by the template matching unit 110. In FIG. 14, for example, a first captured image 601 is a first captured image captured by the imaging device 53_A, and a first captured image 601 is a first captured image captured by the imaging device 53_B. A template frame 611 is shown in the first captured image 601, and a search range (local region) is shown in the second captured image 602. The template frame 611 is set as a rectangular frame within a predetermined range with the corresponding point 621 as the center. The search range 612 is set as a rectangular frame having a larger range than the template frame 611 with the corresponding point 622 as the center. Each of the corresponding points 621 and 622 is a corresponding point (pixel) that is the same coordinate extracted in the three-dimensional shape of the first captured images 601 and 602 extracted by the corresponding point search unit 108. Thereby, the template matching part 110 detects the 1st corresponding point which is a common corresponding point in the 1st captured image which each imaging device included in the 1st imaging device group imaged.

Returning to FIG. 4, the virtual camera setting unit 111 selects camera parameters of a plurality of different types of imaging devices from the camera parameter table stored in the database 114. There is no theoretical limit to the camera parameters of this virtual camera.
However, since the morphing process is performed, good results cannot be obtained under conditions that are significantly different from those of the imaging device that captured the first captured image. Therefore, the internal parameters (focal length, optical center, lens distortion, etc.) are set to values close to the imaging device used for shooting. In addition, by setting the external parameters (such as the position in the three-dimensional space) to positions close to a line segment or a plane connecting the imaging devices that have captured the first captured image used for morphing to obtain the captured image of the virtual camera, The result of the morphing process can be improved.

  The morphing imaging device selection unit 112 sets the imaging device used for morphing when the position of the virtual camera is set by the user by an input unit (not shown). That is, when it is set from which imaging direction (viewpoint direction) to project the three-dimensional shape of the subject onto the imaging surface of the virtual camera (that is, the two-dimensional plane), the morphing imaging device selection unit 112 performs morphing. Setting of the imaging device that captured the first captured image to be used is performed. Here, in the imaging device arranged in the dome-type imaging device, the virtual camera is arranged on a line segment connecting the imaging devices or on a surface (two-dimensional plane) formed by the imaging device. In other words, each of the imaging devices used for the morphing process is limited to one on a line segment formed by each, or on a surface formed by a plurality of imaging devices. The camera parameters of the virtual camera depend on the arrangement position of the virtual camera. When the arrangement positions of a certain imaging device and a virtual camera match, the camera parameters of the virtual camera have the same values as those of the imaging device with the same arrangement position.

  FIG. 15 is a diagram illustrating a virtual camera arrangement possible range when performing a morphing process for generating a virtual captured image that is a captured image of a virtual camera in an imaging apparatus that constitutes the first imaging apparatus set. In FIG. 15, for example, when the first imaging device group is configured by three imaging devices 53_A, 53_B, and 53_C, the range in which the virtual camera 55 is arranged is on the triangle formed by the imaging devices 53_A, 53_B, and 53_C. On the two-dimensional plane 760. The direction from this surface to the three-dimensional shape is the imaging direction of the virtual camera 55. On the two-dimensional plane 760, a virtual captured image captured by the virtual camera 55 can be generated by morphing from the first captured images captured by the imaging devices 53_A, 53_B, and 53_C.

FIG. 16 is a diagram for describing selection of an imaging device used for morphing processing corresponding to the arrangement of the imaging device in the dome-type imaging device shown in FIG. FIG. 16A shows an arrangement position of the virtual camera 55 in the configuration of FIG. 3A of the imaging device in each column of the dome type imaging device. When the virtual camera is arranged on the two-dimensional plane 701 formed by the four imaging devices 53_A, 53_B, 53_C, and 53_D of the first imaging device set, the imaging device 53_A, 53_B, 53_C, and 53_D are selected by the morphing imaging device selection unit 112.
FIG. 16B shows an arrangement position of the virtual camera 55 in the configuration of FIG. 3B of the imaging device in each column of the dome-type imaging device. When the virtual camera is arranged on the two-dimensional plane 702 formed by the three imaging devices 53_A, 53_B, and 53_C of the first imaging device set, the imaging devices 53_A, 53_B, and morphing imaging devices are used. 53_C is selected by the morphing imaging device selection unit 112.

Returning to FIG. 4, the virtual camera image generation unit 113 performs a morphing process for generating a virtual captured image captured by the virtual camera using the first captured image captured by the selected imaging device. In the following description of the operation of the virtual camera image generation unit 113, it is assumed that the first imaging device set includes two imaging devices 1 and 2. At this time, the virtual camera v is arranged on a line connecting each of the imaging device 1 and the imaging device 2. Further, when the coordinates of a pixel of a virtual captured image captured by the virtual camera v searched by the corresponding point search unit 108 is Pv, the corresponding point in the first captured image captured by the imaging device i (i = 1, 2). Let Pi be the coordinates of the pixel. In addition, the color (RGB gradation data) at the coordinate x of the pixel of the imaging device i is Ci (x), and the coordinate of the imaging device i is Ei. In this case, the coordinate Ev of the virtual camera v in the world coordinate system can be expressed by the following equation (1) using the parameter α (0 ≦ α ≦ 1). The position and coordinates in the following description are coordinates on a two-dimensional plane of a virtual captured image captured by the virtual camera v. With respect to the two-dimensional coordinates, the coordinate points E1, E2, the coordinate points P1, P2, etc. on the two-dimensional plane of the first captured image are subjected to coordinate conversion to perform mapping.
Ev = (1-α) E1 + αE2 (1)

The coordinates Pv can be obtained geometrically from the coordinates Pi of the corresponding points and the camera parameters of the imaging device i. The camera parameters of the imaging device 1 and the imaging device 2 are constants, and the camera parameter of the virtual camera depends on the position Ev. Since the position Ev depends on the parameter α, the coordinate Pv can be obtained by the following equation (2) using the coordinates P of the corresponding point and the function F depending on α.
Pv = F (P1, P2, α) (2)

Here, as a precondition for the expression (2) to be satisfied, if Ev = E1, the virtual camera v has the same camera parameters as the imaging device 1, and Pv = E1. Similarly, when Ev = E2, the virtual camera v has the same camera parameters as the imaging device 2, and Pv = E2. For this reason, the following formulas (3) and (4) are established.
Pv = F (P1, P2, 0) (3)
Pv = F (P1, P2, 1) (4)

For this reason, the virtual camera image generation unit 113 obtains the color Cv to be morphed in the above-described case based on the following equation (5) assuming that the color Cv depends on the position Ev of the virtual camera.
Cv (Pv) = (1-α) C1 (P1) + αC2 (P2) (5)

  However, since an error has actually occurred in the coordinates P1 and P2 as corresponding points, “Pv, P1, P2” does not have an accurate relationship between the positions of the corresponding points. For this reason, as already described, the template matching unit 110 performs template matching of the first corresponding points between the first captured images used for morphing.

Next, the virtual camera image generation unit 113 captures an image of the virtual camera based on the result of template matching performed by the template matching unit 110 from the first captured images captured by the imaging device 1 and the imaging device 2 by template matching. Processing for obtaining the color Cv of each pixel of the virtual captured image will be described.
When template matching is performed between the first captured image captured by the imaging apparatus i and the template using the first captured image captured by the imaging apparatus j as a template, the corresponding point in the imaging apparatus j is set as a coordinate P′ij. Further, the value of the coordinate difference (distance) between the coordinates P′ij and the coordinates Pi is set to Dij (= P′ij−Pi). For example, when the first captured image captured by the imaging device 1 is used as a template and template matching is performed with the first captured image captured by the imaging device 2, the coordinate P′21 is detected as the corresponding point of the coordinate P1, The distance between the coordinates P2 and the coordinates P′21 is D21.

  As a result, as a result of template matching performed by the template matching unit 110, a set of two corresponding points is generated. That is, when the first captured image captured by the image capturing apparatus 1 is used as a template and template matching is performed with the first captured image captured by the image capturing apparatus 2, the coordinate P′21 is obtained as the corresponding point of the coordinate P1, and the corresponding A point set (P1, P'21) is obtained. On the other hand, when the first captured image captured by the imaging device 2 is used as a template and template matching is performed with the first captured image captured by the imaging device 1, the coordinate P′12 is obtained as the corresponding point of the coordinate P2, and the corresponding A point set (P2, P′12) is obtained.

Each of the corresponding point set (P1, P′21) and the corresponding point set (P2, P′12) has the following formula (2) as shown in the following formulas (6) and (7). Therefore, there is no positional relationship corresponding to the coordinate Pv.
Pv ≠ F (P1, P′21, α) (6)
Pv ≠ F (P2, P′12, α) (7)

As shown in the equations (6) and (7), it can be seen that there is no set of corresponding points corresponding to the coordinates Pv to be obtained. When the corresponding point set corresponding to the coordinate Pv described above is expressed by, for example, (P1 + δ, P2 + δP2), the following equation (8) is established.
Pv = F (P1 + δ, P2 + δP2, α) (8)

In addition, when α = 0 from the preconditions expressed by the equations (3) and (4), that is, when the virtual camera v overlaps the imaging device 1, Pv = P1, while α = 1, that is, the virtual camera v captures an image. When overlapping with the device 2, the following equations (9) and (10) are established.
P1 = F (P1 + δ, P2 + δP2, 0) (9)
P2 = F (P1 + δ, P2 + δP2, 1) (10)

When the virtual camera v and the imaging device 1 overlap, the coordinate Pv of the corresponding point of the virtual camera v and the coordinate P1 of the corresponding point of the imaging device 1 are the same coordinates, so the following equation (11) holds. .
Pv = P1 + δP1 = P1 (11)
According to the above equation (11), when α = 0, δP1 = 0. Similarly, when α = 1, δP2 = 0.

Further, since the coordinate P1 and the coordinate P′21 are in a corresponding point relationship, and the coordinate P2 and the coordinate P′12 are in a corresponding point relationship,
When α = 0, P2 + δP2 = P′21, that is, δP2 = D21
When α = 1, P1 + δP1 = P′12, that is, δP1 = D12
It becomes.

As an expression that satisfies the above conditions, the following expression (12) is used.
δP1 = αD12, δP1 = (1-α) D21 (12)
From this equation (12), the coordinate Pv is obtained by the following equation (13).
Pv = F (P1 + αD12, P2 + (1-α) D21, α) (13)
In this case, the color Cv (Pv) at the coordinate Pv of the virtual camera v is obtained by the following equation (14).
Cv (Pv) = (1-α) C1 (P1 + αD12)
+ ΑC2 (P2 + (1-α) D21) (14)

Further, even when three or more imaging devices are used, the morphing process for generating a virtual captured image captured by the virtual camera v can be performed in the same manner as when two imaging devices are used. That is, when the imaging device used for the morphing process, that is, the first imaging device set is composed of the imaging device 1, the imaging device 2, and the imaging device 3, using the parameters α and β, the following (15 ) To obtain the coordinate Ev of the virtual camera v. Here, α ≧ 0, β ≧, and 0 ≦ α + β ≦ 1.
Ev = (1−α−β) E1 + αE2 + βE3 (15)

  FIG. 17 is a diagram illustrating the coordinates Ev of the virtual camera v in the case where three image capturing apparatuses, the image capturing apparatus 1, the image capturing apparatus 2, and the image capturing apparatus 3, are used. In FIG. 17, the coordinates of the imaging device 1 are E1, the coordinates of the imaging device 2 are E2, the coordinates of the imaging device 3 are E3, and the coordinates of the virtual camera v are Ev. In this way, the virtual camera v is arranged on a triangular two-dimensional plane formed by the imaging device 1, the imaging device 2, and the imaging device 3. In addition, as shown in FIG. 17, the coordinate Ev is obtained by multiplying a distance vector obtained by subtracting the coordinate E1 from the coordinate E2 by α (E2-E1) and a distance vector obtained by subtracting the coordinate E1 from the coordinate E3. The coordinate E1 is obtained by moving in a plane on a two-dimensional plane by β (E3-E1) multiplied by β.

In this case, the coordinate Pv of the pixel of the virtual captured image of the virtual camera v, the coordinate P1 of the pixel of the first captured image of the image capturing apparatus 1 that is the corresponding point of the coordinate Pv, and the pixel of the first captured image of the image capturing apparatus 2 And the coordinate P3 of the pixel of the first captured image of the imaging device 3 are obtained by the following equation (16) from the same concept as in the case of two imaging devices.
Pv =
F (P1 + δP1, P2 + δP2, P3 + δP3, α, β) (16)

In the above equation (16), when α = 0 and β = 0, the coordinates of the virtual camera and the imaging device 1 coincide and Ev = E1, and when α = 1 and β = 0, the virtual camera and the imaging device. 2 coincides with Ev = E2, and when α = 0 and β = 1, the coordinates of the virtual camera and the imaging device 3 coincide with each other, and Ev = E3. From these initial conditions, the following equation (17) holds.
When α = 0 and β = 0, δP1 = 0, δP2 = D21, δP3 = D31
When α = 1 and β = 0, δP1 = D12, δP2 = 0, δP3 = D32
When α = 0 and β = 1, δP1 = D13, δP2 = D23, δP3 = 0
... (17)

As a condition for satisfying the above expression (17), the relationship of the following expression (18) was used.
δP1 = αD12 + βD13
δP2 = (1−α−β) D21 + βD23
δP3 = (1−α−β) D31 + αD32 (18)

From the equation (18), the pixel Pv of the virtual captured image of the virtual camera v is obtained by the following equation (19).
Pv = F (P1 + αD12 + βD13, P2 + (1−α−β) D21 + βD23, P3 + (1−α−β) D31 + αD32, α, β) (19)

When the above equation (19) is used, the color Cv of the coordinate Pv of the pixel of the virtual captured image of the virtual camera v is the first corresponding point corresponding to the coordinate Pv, and each of the pixels in the imaging devices 1, 2, and 3 Using the coordinates P1 + δP1, P2 + δP2, and P3 + δP3 obtained by converting the coordinates into the coordinate system of the virtual captured image, the following equation (20) is obtained.
Cv (Pv) = w1C1 (P1 + αD12 + βD13) + w2C2 (P2 + (1−α−β) D21 + βD23) + w3C3 (P3 + (1−α−β) D31 + αD32)
... (20)
In the above equation (20), wi (i = 1, 2, 3) is a weighting coefficient, and w1 + w2 + w3 = 1.
Each of C1, C2, and C3 represents the respective colors of P1, P2, and P3 that are the first corresponding points in the first captured images captured by the imaging devices 1, 2, and 3, respectively.

FIG. 18 is a diagram for explaining a weighting coefficient in the equation (20) for obtaining the color Cv of the pixel Pv of the virtual captured image of the virtual camera v. In FIG. 18, the coordinates of the imaging device 1 are E1, the coordinates of the imaging device 2 are E2, the coordinates of the imaging device 3 are E3, and the coordinates of the virtual camera v are Ev. Here, each of the weighting factors w1, w2, and w3 is an area obtained by dividing a triangle formed by coordinates E1, E2, and E3 as vertices by a triangle having coordinates E1, E2, and E3 and coordinates Ev as vertices. Is the ratio. For example, the area of a triangle formed with coordinates E1, E2, and E3 as vertices is S, the area of a triangle with coordinates E2, E3, and Ev as vertices is Sw1, and a triangle that has coordinates E1, E3, and Ev as vertices The area of the triangle is Sw2, and the area of the triangle whose apexes are the coordinates E1, E2, and Ev is Sw3. That is, S = Sw1 + Sw2 + Sw3.
In this case, as weighting factors, w1 = Sw1 / S, w2 = Sw2 / S, and w3 = Sw3 / S. Further, the weight coefficient wi is set to a ratio of the area SWi of the triangle located at a position facing the coordinate Ei via the coordinate Ev to the area S. As a result, the weight of the imaging device i at the coordinate Ei closest to the coordinate Ev is the largest, while the weight of the imaging device i at the coordinate Ei farthest from the Ev is the smallest.

Returning to FIG. 4, in the above equation (20), the set of corresponding points between the first captured image captured by each of the image capturing apparatuses 1 and 2 and the virtual captured image captured by the virtual camera is [P1 + αD12 + βD13, P2 + (1- α−β) D21 + βD23, P3 + (1-α−β) D31 + αD32]. With this set, the morphing process of the pixel at the coordinate Pv of the virtual captured image is performed by the virtual camera image generation unit 113.
That is, the color Cv of the pixel Pv of the virtual captured image of the virtual camera v is calculated on the assumption that the local region (for example, a cubic region having the difference Dij as a side) generated by the difference Dij includes corresponding points. .

FIG. 19 is a diagram illustrating a relationship between corresponding points in the first captured image captured by each of the image capturing apparatuses 1, 2, and 3 and the virtual captured image captured by the virtual camera v.
FIG. 19A illustrates how to obtain coordinates P1 + δP1 in which the pixels of the first captured image captured by the imaging device 1 corresponding to the coordinates Pv of the virtual captured image captured by the virtual camera v are coordinate-converted. In FIG. 19A, the coordinate P <b> 1 indicates a coordinate obtained by coordinate-converting the pixel coordinate in the captured image captured by the imaging device 1 into the virtual captured image of the virtual camera v. The coordinate P′12 is obtained by using the first captured image captured by the imaging device 2 as a template, and performing coordinate conversion on the coordinates of the pixel extracted as the first corresponding point by performing template matching on the first captured image captured by the imaging device 1. The coordinates are shown. The coordinate P′13 is obtained by converting the coordinates of the pixel extracted as the first corresponding point by using the first captured image captured by the imaging apparatus 3 as a template and performing template matching on the first captured image captured by the imaging apparatus 1. The coordinates are shown. The distance vector D12 is generated by subtracting the coordinate P1 from the coordinate P′12, and the distance vector D13 is generated by subtracting the coordinate P1 from the coordinate P′13. The coordinate P1 + δP1 is moved on the two-dimensional plane in a two-dimensional plane by αD12 obtained by multiplying the distance vector D12 by α and βD13 obtained by multiplying the distance vector D13 by β based on the above equation (18). It is obtained by doing.

  FIG. 19B shows how to obtain coordinates P2 + δP2 in which the pixels of the first captured image captured by the imaging apparatus 2 corresponding to the coordinates Pv of the virtual captured image captured by the virtual camera v are coordinate-converted. In FIG. 19B, a coordinate P2 indicates a coordinate obtained by coordinate-converting the pixel coordinate in the captured image captured by the imaging device 2 into the virtual captured image of the virtual camera v. The coordinate P′21 is obtained by converting the coordinates of the pixel extracted as the first corresponding point by using the first captured image captured by the image capturing apparatus 1 as a template and performing template matching on the first captured image captured by the image capturing apparatus 2. The coordinates are shown. The coordinate P′23 is obtained by converting the coordinates of the pixel extracted as the first corresponding point by using the first captured image captured by the imaging device 3 as a template and performing the template matching on the first captured image captured by the imaging device 2. The coordinates are shown. The distance vector D21 is generated by subtracting the coordinate P2 from the coordinate P'21, and the distance vector (D23-D21) is generated by subtracting the distance vector D21 from the distance vector D23. The coordinate P2 + δP2 is calculated based on the above equation (18) by using αD21 obtained by multiplying the distance vector D21 by α and β (D23−D21) obtained by multiplying the distance vector D23-D21 by β (D23−D21). It is obtained by moving in a plane on a two-dimensional plane.

  FIG. 19C shows how to obtain coordinates P3 + δP3 in which the pixels of the first captured image captured by the imaging apparatus 3 corresponding to the coordinates Pv of the virtual captured image captured by the virtual camera v are transformed. In FIG. 19C, a coordinate P3 indicates a coordinate obtained by coordinate-converting the pixel coordinate in the captured image captured by the imaging device 3 into the virtual captured image of the virtual camera v. The coordinate P′31 is obtained by converting the coordinates of the pixel extracted as the first corresponding point by using the first captured image captured by the image capturing apparatus 1 as a template and performing the template matching on the first captured image captured by the image capturing apparatus 3. The coordinates are shown. The coordinate P′32 is obtained by using the first captured image captured by the imaging device 2 as a template, and performing coordinate conversion on the coordinates of the pixel extracted as the first corresponding point by performing template matching on the first captured image captured by the imaging device 3. The coordinates are shown. The distance vector D31 is generated by subtracting the coordinate P2 from the coordinate P'21, and the distance vector (D32-D31) is generated by subtracting the distance vector D31 from the distance vector D32. The coordinate P3 + δP3 is based on the above equation (18), and the coordinate P′31 is obtained by βD31 obtained by multiplying the distance vector D31 by β and α (D32−D31) obtained by multiplying the distance vector D32-D31 by α. Is obtained by moving in a plane on a two-dimensional plane.

Further, the morphing process for generating the virtual captured image captured by the virtual camera v can be performed in the same manner as in the case where the above-described three imaging devices are used even when a plurality of imaging devices exceeding three are used. it can. In the present embodiment, when the imaging device used for the morphing process, that is, the first imaging device set is composed of the imaging device 1, the imaging device 2, the imaging device 3, and the imaging device 4, the parameters α and β are set. By using the following equation (21), the coordinate Ev of the virtual camera v can be obtained. Here, 0 ≦ α ≦ 1 and 0 ≦ β ≦ 1.
Ev = (1-α + αβ) E1 + α (1-β) E2
-ΑβE3 + αβE4 (21)

  FIG. 20 is a diagram illustrating the coordinates Ev of the virtual camera v in the case of using four imaging devices, that is, the imaging device 1, the imaging device 2, the imaging device 3, and the imaging device 4. In FIG. 20, the coordinates of the imaging device 1 are E1, the coordinates of the imaging device 2 are E2, the coordinates of the imaging device 3 are E3, the coordinates of the imaging device 4 are E4, and the coordinates of the virtual camera v. Is Ev. Here, the coordinate Ev is a coordinate E12 generated by dividing the coordinate E2 and the coordinate E1 by a ratio of α: 1-α, and a coordinate generated by dividing the coordinate E3 and the coordinate E4 by α: 1-α. 34. A coordinate β (α (E4-E3) −α (E2-E1)) obtained by dividing the coordinate E12 and the coordinate 34 by a ratio of β: 1−β is set as the coordinate Ev.

Further, the corresponding points in the first captured image captured by each of the imaging device 1, the imaging device 2, the imaging device 3, and the imaging device 4 are expressed by the following formula (22) when considered in the same manner as when there are three imaging devices. Can be represented by In the following formula, i = 1, 2, 3, and 4.
Pi + δPi = (1−α + αβ) P′i1
+ Α (1-β) P'i2-αβP'i3 + αβP'i4
... (22)

  FIG. 21 is a diagram illustrating the correspondence between the first corresponding points in the first captured images of the imaging devices 1, 2, 3, and 4 on the two-dimensional plane of the virtual captured image. In FIG. 21, coordinates Pi1 are coordinates obtained by coordinate-converting the first corresponding point of the first captured image captured by the imaging device Pi into the coordinate system of the virtual captured image using the first captured image captured by the imaging device 1 as a template. It is. The coordinate Pi2 is a coordinate obtained by coordinate-converting the first corresponding point of the first captured image captured by the imaging device Pi into the coordinate system of the virtual captured image using the first captured image captured by the imaging device 2 as a template. The coordinate Pi3 is a coordinate obtained by coordinate-converting the first corresponding point of the first captured image captured by the imaging device Pi into the coordinate system of the virtual captured image using the first captured image captured by the imaging device 1 as a template. The coordinate Pi4 is a coordinate obtained by coordinate-converting the first corresponding point of the first captured image captured by the imaging device Pi into the coordinate system of the virtual captured image using the first captured image captured by the imaging device 1 as a template. For example, when i = 1, the first common point in the first captured image captured by the imaging device 1 is P′11 = P1, and the corresponding points corresponding to the coordinates P1 are the respective imaging devices 2, 3, and 4. Are the coordinates P′12, the coordinates P′13, and the coordinates P′14, respectively.

  Further, in FIG. 21, the corresponding point Pi + δPi in the virtual image picked up by the virtual camera v is obtained by changing the coordinates P′i2 and the coordinates P′i1 to α: 1 as in the case of obtaining the coordinates of the virtual camera v in FIG. A coordinate Pa (= α (P′i2−P′i1)) generated by dividing at a ratio of −α and a coordinate P′i4 and a coordinate P′i3 generated at a ratio of α: 1−α. Coordinate Pb (= α (P′i4−P′i3)) is obtained. A coordinate β (α (P′i4−P′i3) −α (P′i2−P′i1)) obtained by dividing the coordinate Pa and the coordinate Pb by a ratio of β: 1−β is set as a coordinate Pi + δPi. Yes. This coordinate Pi + δPi becomes the coordinate Pi of the first corresponding point in the first captured image captured by the imaging device i in the virtual captured image captured by the virtual camera.

However, in the above equation (22), P′ii = Pi. For example, when i = 3, the result of the above expression (22) becomes the following expression (23).
P3 + δP3 = (1−α + αβ) P′31
+ Α (1-β) P′32−αβP′3 + αβP′34
... (23)

In the above equation (23), there is a relationship represented by the following equation (24).
P'31 = P3 + D31
P'32 = P3 + D32
P′34 = P3 + D34 (24)

Therefore, by substituting the equation (24) for the equation (23), δP3 can be expressed as the following equation (25).
δP3 = (1−α + αβ) D31 + α (1-β) D32 + αβD34
... (25)

Therefore, the coordinates Pv of the first corresponding point in the captured image can be expressed by the following equation (26).
Pv = F (P1 + α (1-β) D12-αβD13 + αβD14, P2 + (1-α + αβ) D21-αβD23 + αβD24, P3 + (1-α + αβ) D31 + α (1-β) D32 + αβD34, P4 + (1-α + αβ) D41 + α (1-β) D42−αβD43, α, β) (26)

When the above equation (26) is used, the color Cv of the coordinate Pv of the pixel of the virtual captured image of the virtual camera v is the first corresponding point corresponding to the coordinate Pv, and the pixel in the imaging devices 1, 2, 3, and 4 Using the coordinates P1 + δP1, P2 + δP2, P3 + δP3, and P4 + δP4 obtained by converting each coordinate into the coordinate system of the virtual captured image, the following equation (27) is obtained.
Cv (Pv) = w1C1 (P1 + α (1-β) D12-αβD13 + αβD14) + w2C2 (P2 + (1-α + αβ) D21-αβD23 + αβD24) + w3C3 (P3 + (1-α + αβ) D31 + α (1-β) D32 + αβD34 (w41 + 4) 1 -Α + αβ) D41 + α (1-β) D42-αβD43) (27)
In the above equation (20), wi (i = 1, 2, 3, 4) is a weighting coefficient, and w1 + w2 + w3 + w4 = 1.
In addition, each of C1, C2, C3, and C3 is a pixel corresponding to each of coordinates P1, P2, P3, and P4 that are first corresponding points in the first captured images captured by the imaging devices 1, 2, 3, and 4, respectively. Represents color.

  FIG. 22 is a diagram for explaining a weighting coefficient in the equation (27) for obtaining the color Cv of the pixel Pv of the virtual captured image of the virtual camera v. In FIG. 18, the coordinates of the imaging device 1 are E1, the coordinates of the imaging device 2 are E2, the coordinates of the imaging device 3 are E3, the coordinates of the imaging device 4 are E4, and the coordinates of the virtual camera v. Is Ev. Here, each of the weighting factors w1, w2, w3, and w4 has an area ratio obtained by dividing a quadrangle formed by coordinates E1, E2, E3, and E4 into four parts. That is, a coordinate E12 obtained by dividing the coordinates E2 and E1 by α: 1−α and a coordinate E34 obtained by dividing the coordinates E4 and E3 by α: 1−α are generated. Further, a coordinate E13 obtained by dividing the coordinate E3 and the coordinate E1 by β: 1−β and a coordinate E24 obtained by dividing the coordinate E4 and the coordinate E2 by β: 1−β are generated.

Then, the quadrangle of the area S having the coordinates E1, E2, E3, and E4 as vertices is divided into four by a line segment connecting the coordinates 12 and the coordinates 34 and a line segment connecting the coordinates E13 and the coordinates E24. Here, the area of a quadrangle having coordinates E1, E12, Ev, and E13 as vertices is Sw4, the area of a quadrangle having coordinates E12, E2, E24, and Ev as vertices is Sw3, and coordinates E13, Ev, E34, and E3 are The area of the quadrangle having the vertex is Sw2, and the area of the quadrangle having the coordinates Ev, E24, E4, and E34 as the vertex is Sw1. That is, S = Sw1 + Sw2 + Sw3 + Sw4.
In this case, the weighting factors are w1 = Sw1 / s, w2 = Sw2 / S, w3 = Sw3 / S, and w4 = Sw4 / S. Further, the weight coefficient wi is set to a ratio of the area SWi of the quadrangular area SWi at a position facing the coordinate Ei via the coordinate Ev to the area S. As a result, the weight of the imaging device i at the coordinate Ei closest to the coordinate Ev is the largest, while the weight of the imaging device i at the coordinate Ei farthest from the Ev is the smallest.

The general formula with 0 ≦ i ≦ n in the above equation (27) is the following equation (28).
Cv (Pv) = ΣwiCi (Pi + δPi) (28)
In the above equation (28), wi represents the weighting coefficient of the color of the pixel at the first corresponding point of the first captured image captured by the imaging device i. Ci (Pi + δPi) indicates the color of the pixel at the first corresponding point of the first captured image captured by the imaging apparatus i, which is coordinate-converted to the virtual imaging coordinates Pi + δPi captured by the virtual camera v.

Next, an operation of generating a free viewpoint image by morphing a plurality of first captured images in the free viewpoint image capturing apparatus according to the present embodiment will be described with reference to the drawings. FIG. 23 is a flowchart illustrating an operation example in which the free viewpoint image capturing apparatus according to the present embodiment generates a free viewpoint image by morphing a plurality of first captured images.
Step S1:
The imaging control unit 101 reads a captured image from each of the imaging devices provided on each column of the dome frame 500, and once writes and stores it in the database 114 together with the identification information of the imaging device of the corresponding imaging device. At this time, the imaging control unit 101 obtains a low-resolution first captured image obtained by thinning out pixels of the captured image from each imaging device, and a second captured image having a resolution similar to that of the captured image without thinning out pixels. As a set, it is written and stored in the database 114.

Step S2:
The silhouette image generation unit 102 reads out the second captured image from the database 114, and generates a silhouette image using a color filter from each of the read out second captured images as shown in FIG. At this time, the silhouette image generation unit 102 can easily perform a process of generating a silhouette image because the background of the captured image is captured as blue blue.

Step S3:
The VH processing unit 103 uses the silhouette image generated by the silhouette image generation unit 102 and uses the Visual_Hull method. For example, in the case of the present embodiment, as shown in FIG. Is generated.

Step S4:
The imaging device selection unit 104 sets two or more imaging devices provided in the dome frame 500 as a set, and sets a plurality of imaging devices as three or four as shown in FIG. A first imaging device set that is a set of imaging devices is generated.

Step S5:
The stereo matching unit 105 performs stereo matching of common corresponding points between the first captured images captured by the imaging devices included in the first imaging device set for each first imaging device set generated by the imaging device selection unit 104. .

Step S6:
The three-dimensional shape generation unit 106 uses the common corresponding points between the first captured images of the respective imaging devices, which are corresponding points extracted by the stereo matching unit 105, and the VF processing unit 103 generates the temporary points generated by VH (Visual Hull). The three-dimensional shape is corrected and a partial three-dimensional shape is generated.
That is, the three-dimensional shape generation unit 106 generates a temporary three-dimensional shape from which the protruding region is removed as a partial three-dimensional shape that is a corrected temporary three-dimensional shape. Here, the three-dimensional shape generation unit 106 maps and superimposes common coordinate points of each first imaging device set in the world coordinate system. Here, if the common coordinate points between different first imaging device sets are the same coordinates, they are overlapped.

Step S7:
The three-dimensional shape synthesis unit 107 superimposes all the partial three-dimensional shapes generated by the three-dimensional shape generation unit 106 for each first imaging device set. At this time, the three-dimensional shape synthesis unit 107 performs a process of superimposing the partial three-dimensional shapes by aligning the coordinate positions in the respective pixel units of the partial three-dimensional shape with the common corresponding point as a reference.
In addition, the corresponding point search unit 108 searches for the first corresponding point, which is a corresponding point in the second captured image, using the three-dimensional shape combined by the three-dimensional shape combining unit 107.

Since the corresponding point search unit 108 performs the morphing process using the second captured image based on the first corresponding point, the search for the first corresponding point is performed on the high-resolution second captured image used for morphing. Here, the corresponding point search unit 108 reads the second captured image from the database 114. In the search for the first corresponding point, the corresponding point search unit 108 determines the three-dimensional shape from the two-dimensional plane perpendicular to the imaging direction of each imaging device, that is, the two-dimensional plane corresponding to the imaging surface of each imaging device. This is performed using a virtual first captured image obtained by projecting each pixel of the shape.
Further, the occlusion area search unit 109 detects pixels that exist in the three-dimensional shape and do not exist in the virtual first captured image for each virtual first captured image on which the corresponding point search unit 108 projects the three-dimensional shape. Each device is written and stored in the database 114.

Step S8:
The template matching unit 110 performs template matching between first captured images captured by all the imaging devices included in each first imaging device set in each first imaging device set. Here, since the template matching unit 110 does not take into account the geometric consistency of the imaging device, the template matching unit 110 does not perform stereo matching but only performs template matching between the first captured images.

Step S9:
The virtual camera setting unit 111 selects and reads out camera parameters of the imaging device used in the dome imaging device from the camera parameter table stored in the database 114.

Step S10:
When the position of the virtual camera is set by a user using an input unit (not shown), the morphing imaging device selection unit 112 sets an imaging device used for a morphing process for generating a virtual captured image captured by the virtual camera.
That is, when it is set from which imaging direction (viewpoint direction) to project the three-dimensional shape of the subject onto the imaging surface of the virtual camera (that is, the two-dimensional plane), the morphing imaging device selection unit 112 performs the virtual imaging. The setting of the imaging device that captured the first captured image used for the morphing process for generating an image, that is, the selection of the first imaging device set is performed. At this time, the morphing imaging device selection unit 112 reads out and selects position information from the camera parameters of each imaging device in the database 114, and selects a first imaging device group having a space range including the position of the virtual camera.

Step S11:
The virtual camera image generation unit 113 reads the first captured image of the imaging device selected by the morphing imaging device selection unit 112 from the database 114. In this description, for example, the first imaging device group is composed of three imaging devices.
Next, the virtual camera image generation unit 113 changes the first corresponding points of the imaging devices 1, 2, and 3 that have undergone template matching from the coordinate system of the first captured image thereof to the coordinate system of the virtual captured image. Convert coordinates.
Then, the virtual camera image generation unit 113 calculates the pixel coordinates P1 + δP1, P2 + δP2, and P3 + δP3 of the pixels of the first corresponding point in the captured images of the imaging devices 1, 2, and 3 in the coordinate system of the virtual captured image from these colors. The color of the pixel at the coordinate Pv is obtained by performing the morphing process according to the above equation (20) in the virtual captured image corresponding to the first corresponding point.

The virtual camera image generation unit 113 performs the above-described processing on the coordinates Pv of each pixel in all virtual captured images, and generates a virtual captured image.
At this time, if the virtual camera image generation unit 113 projects a three-dimensional shape onto the coordinate system of the virtual captured image and detects coordinates that cannot be seen due to occlusion, the virtual camera image generation unit 113 does not generate the color of the coordinates.
In addition, when there is an imaging device that does not have the first corresponding point in the first captured image due to occlusion in the first imaging device set that performs the morphing process, the virtual camera image generation unit 113 other than the imaging device in the first imaging device set Morphing processing is performed using the first captured image of the imaging apparatus other than the coordinates.
Here, the weight of the imaging device that does not have the first corresponding point due to occlusion is set to zero. For example, when determining the color of a certain pixel Pv, when using the imaging device 1 to the imaging device 4 and the imaging device 2 is occluded, in the processing described in the paragraph numbers 0114 to 0116, Sw2 Set the value of to 0. As a result, S = Sw1 + Sw3 + Sw4 and the weights are w2 = 0 and w1 + w3 + w4 = 1. Equation (28) is calculated using this weight.

As described above, according to the present embodiment, the first captured image and the second captured image having a higher resolution than the first captured image are generated from the captured images captured by the plurality of imaging devices, and the first captured image is generated. To generate a three-dimensional shape of the subject, perform template matching based on the three-dimensional shape, and generate a virtual captured image captured by the virtual camera, the first corresponding point corresponding to each pixel of the virtual captured image It can be easily and robustly extracted from the second captured image of each imaging device.
In addition, according to the present embodiment, the position of the corresponding point of the pixel of the virtual captured image in the second captured image of each imaging device is coordinate-converted to the coordinates of the virtual captured image, and thus each second captured image in the morphing process. Thus, it is possible to accurately calculate the weighting of the color of each pixel, and to perform highly accurate morphing processing.

Further, according to the present embodiment, since the three-dimensional shape is projected onto the coordinate system of the virtual image image, there are coordinates that cannot be seen due to occlusion on the virtual captured image, or there are second coordinates that cannot be seen by the morphing process. Even when a captured image exists, it is possible to easily deal with the handling of pixels of those coordinates so that holes are not opened or blurred in the image, so the shape of the occlusion area is reflected in the virtual captured image. be able to.
As described above, according to the present embodiment, when generating a free viewpoint image (virtual captured image) by morphing the second captured image captured by the imaging device in response to the viewpoint change, Template matching is performed in the local region, and the first corresponding point between the captured images is corrected in the coordinate system of the virtual captured image. For this reason, according to the present embodiment, blur due to morphing can be reduced, and a captured image of the virtual camera corresponding to the viewpoint of observing the subject can be captured as a photoreal free viewpoint image.

Hereinafter, a comparison of the free viewpoint image generated in each of the free viewpoint image capturing apparatus according to the present embodiment and the conventional one will be shown.
FIG. 24 is a diagram illustrating a first captured image captured by the first imaging device group of the dome-shaped imaging device. In FIG. 24, an example in which the first imaging device group is configured by four imaging devices is illustrated, and thus the number of second captured images is four.

  FIG. 25 is a diagram showing a free viewpoint image generated by using View Morphing (registered trademark) implemented in OpenCV (registered trademark) in an open source library. The free viewpoint image of FIG. 25 was generated by performing morphing processing using View Morphing (registered trademark) using the second captured image of FIG.

  FIG. 26 is a diagram showing a free viewpoint image generated by using SAD as a stereo matching method, creating a mesh by the Delaunay triangulation method, and synthesizing this mesh. In FIG. 26, the corresponding point is not found in the occlusion area 860 or 861, and it can be seen that the outline is not drawn and a part thereof is blurred.

  FIG. 27 is a diagram showing a free viewpoint image generated by morphing the result of obtaining a three-dimensional shape using Visual Hull. In other words, in FIG. 27, the three-dimensional shape obtained by Visual Hull is projected onto a virtual captured image corresponding to the viewpoint (imaging direction) of the virtual camera to generate a free viewpoint image. Since the accuracy of the three-dimensional shape of Visual Hull is low, it can be seen that the corresponding points cannot be acquired in the morphing process, and the image is blurred.

  FIG. 28 is a view showing a free viewpoint image generated by a morphing process in which a three-dimensional shape is obtained by Visual Hull, stereo matching between the three-dimensional shape and each of the first captured images is performed, and a common corresponding point is obtained. It is. That is, FIG. 28 does not perform template matching of each pixel between the second captured images in the present embodiment. FIG. 28A shows the obtained free viewpoint image, and FIG. 28B, FIG. 28C, and FIG. 28D are the regions 851, 852, and 853 in FIG. 28A, respectively. FIG.

  FIG. 29 shows a three-dimensional shape obtained by Visual Hull, stereo matching between the three-dimensional shape and each of the first captured images is performed, and a common corresponding point is obtained. It is a figure which shows the free viewpoint image produced | generated by the morphing process using the 1st corresponding point obtained by performing template matching.

  Further, a program for realizing the function of the free viewpoint image generation 10 in FIG. 4 is recorded on a computer-readable recording medium, and the program recorded on the recording medium is read into a computer system and executed. A viewpoint image (virtual captured image) generation process may be performed. Here, the “computer system” includes an OS and hardware such as peripheral devices.

Further, the “computer system” includes a homepage providing environment (or display environment) if a WWW system is used.
The “computer-readable recording medium” refers to a storage device such as a flexible medium, a magneto-optical disk, a portable medium such as a ROM and a CD-ROM, and a hard disk incorporated in a computer system. Furthermore, the “computer-readable recording medium” dynamically holds a program for a short time like a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line. In this case, a volatile memory in a computer system serving as a server or a client in that case, and a program that holds a program for a certain period of time are also included. The program may be a program for realizing a part of the functions described above, and may be a program capable of realizing the functions described above in combination with a program already recorded in a computer system.

  The embodiment of the present invention has been described in detail with reference to the drawings. However, the specific configuration is not limited to this embodiment, and includes design and the like within a scope not departing from the gist of the present invention.

DESCRIPTION OF SYMBOLS 10 ... Free viewpoint image imaging part 53_1, 53_2, 53_3 ... Imaging device 101 ... Imaging control part 102 ... Silhouette image generation part 103 ... VF processing part 104 ... Imaging apparatus selection part 105 ... Stereo matching part 106 ... Three-dimensional shape generation part 107 ... 3D shape synthesis unit 108 ... corresponding point search unit 109 ... occlusion region search unit 110 ... template matching unit 111 ... virtual camera setting unit 112 ... morphing imaging device selection unit 113 ... virtual camera image generation unit 114 ... database 500_1, 500_2 , 500_3, 500_4, 500_5, 500_6, 500_7, 500_8, 500_9, 500_10, 500_11, 500_12 ... column 600 ... subject 800 ... LED light source 850_3 ... light guide 900 ... blue bag

Claims (12)

An imaging control unit that acquires a first captured image of a subject from each of a plurality of imaging devices;
Using a three-dimensional shape of the object that is reproduced from the first captured image, the corresponding point of the pixel between the second imaging field image resolution two different with respect to the first captured image acquired from the imaging device A corresponding point search unit for searching for a first corresponding point;
A template matching unit that performs template matching of each of the first corresponding points in each of the second captured image and the other second captured image using any one of the second captured images acquired from the plurality of imaging devices as a template. When,
A free viewpoint image, comprising: a virtual camera image generation unit configured to generate a third captured image captured from a virtual camera arranged between a plurality of the imaging devices by morphing the second captured image. Imaging device. The virtual camera image generation unit
Performing the morphing process by a combination of the second captured images;
The virtual camera is
The free viewpoint image capturing apparatus according to claim 1, further comprising an imaging surface on a line or a plane on which the imaging apparatus that captures the combined second captured image is arranged. From the fourth captured image obtained by projecting the three-dimensional shape onto the imaging surface of the virtual camera, each occlusion region of the imaging device in the combination is detected, and the detection result is indicated to the virtual camera image generation unit as occlusion information. An occlusion area search unit to supply as
The virtual camera image generation unit is
The reflecting the occlusion information, free viewpoint image capturing apparatus according to 請 Motomeko 2 you and generates the third captured image by morphing process using the captured image of the combination.   4. The free viewpoint image capturing apparatus according to claim 1, wherein a resolution of the second captured image is higher than a resolution of the first captured image. 5. A silhouette image generator for generating a silhouette image from the first captured image;
A VF processing unit that generates a temporary partial 3D shape that is a part of the 3D shape of the subject from the silhouette image;
A stereo matching unit that performs stereo matching of each of the first captured images for each combination of the imaging devices and extracts a common corresponding point that is a common corresponding point in each of the first captured images;
A partial three-dimensional shape generation unit that corrects the temporary partial three-dimensional shape using the common corresponding points and sets the corrected result as a partial three-dimensional shape;
The free viewpoint image capturing apparatus according to claim 1, further comprising: a three-dimensional shape synthesis unit that synthesizes the partial three-dimensional shape to generate the three-dimensional shape. . The three-dimensional shape synthesis unit
The triangular mesh connecting the common corresponding points and the temporary part three-dimensional shape are compared, the temporary part three-dimensional area close to the imaging device direction with respect to the mesh is deleted, and the temporary part three-dimensional The free viewpoint image capturing apparatus according to claim 5, wherein the shape is corrected. The three-dimensional shape synthesis unit superimposes the partial three-dimensional images obtained from all combinations of the first captured images, and the three-dimensional shape from the region where the partial three-dimensional images equal to or greater than a preset threshold overlap. The free viewpoint image capturing apparatus according to claim 5, wherein a shape is generated. There are two imaging devices, and the virtual camera is arranged on a line where the imaging devices are arranged,
The coordinate Ev where the virtual camera is arranged uses the coordinate Ei of the imaging device i (1 ≦ i ≦ 2),
Ev = (1-α) E1 + αE2, (0 ≦ α ≦ 1)
When expressed by the formula
As for the coordinate Pv of the pixel in the third captured image of the virtual camera, the first corresponding point of the coordinate Pv is the coordinate Pi, the first corresponding point in the coordinate system of the third captured image is Pi, and the template When represented by a combination of Pj of the first captured image as a template in matching and a difference Dij (i = 1, 2, j = 1, 2, i ≠ j) with respect to Pi, the third captured image The first corresponding point of the imaging device 1 with respect to the coordinate Pv in the coordinate system is P1 + αD12, and the first corresponding point of the imaging device 2 is P2 + (1-α) D21. The free viewpoint image capturing device according to any one of the above. The number of the imaging devices is three, and the virtual camera is arranged on the surface where the imaging device is arranged,
The coordinate Ev where the virtual camera is arranged uses the coordinate Ei of the imaging device i (1 ≦ i ≦ 3),
Ev = (1−α−β) E1 + αE2 + βE3 (0 ≦ α, 0 ≦ β, α + β ≦ 1)
When expressed by the formula
As for the coordinate Pv of the pixel in the third captured image of the virtual camera, the first corresponding point of the coordinate Pv is the coordinate Pi, the first corresponding point in the coordinate system of the third captured image is Pi, and the template When expressed by a combination of Pj of the first captured image as a template in matching and a difference Dij (i = 1, 2, 3, j = 1, 2, 3, i ≠ j) with respect to Pi, The first corresponding point of the imaging device 1 with respect to the coordinate Pv in the coordinate system of the third captured image is P1 + αD12 + βD13, the first corresponding point of the imaging device 2 is P2 + (1-α−β) D21 + βD23, and the first corresponding point of the imaging device 3 The free viewpoint image capturing apparatus according to claim 5, wherein P3 + (1−α−β) D31 + αD32. The number of the imaging devices is four, and the virtual camera is arranged on a surface on which the imaging devices are arranged,
The coordinate Ev where the virtual camera is arranged uses the coordinate Ei of the imaging device i (1 ≦ i ≦ 4),
Ev = (1-α-αβ) E1 + α (1-β) E2-αβE3
+ ΑβE4 (0 ≦ α, 0 ≦ β, α + β ≦ 1)
When expressed by the formula
As for the coordinate Pv of the pixel in the third captured image of the virtual camera, the first corresponding point of the coordinate Pv is the coordinate Pi, the first corresponding point in the coordinate system of the third captured image is Pi, and the template It is expressed by a combination of Pj of the first captured image as a template in matching and a difference Dij of Pj with respect to Pi (i = 1, 2, 3, 4, j = 1, 2, 3, 4, i ≠ j). The corresponding point of the imaging device i with respect to the coordinate Pv in the coordinate system of the third captured image is (1−α + αβ) P′i1 + α (1−β) P′i2−αβP′i3 + αP′i4 (where P ′ The free viewpoint image capturing apparatus according to claim 5, wherein ii = Pi).   2. The imaging device according to claim 1, wherein each of the imaging devices is arranged with the imaging direction as the subject on the surface of a sphere centered on the subject so that the distance to the subject is the same. The free viewpoint image capturing device according to any one of 10. An imaging control process in which an imaging control unit acquires a first captured image of a subject from each of a plurality of imaging devices;
Corresponding corresponding point search section, the pixel between the reference three-dimensional shape of the object that is reproduced from the first image, the first image and the resolution is different second imaging field image acquired from the imaging device A corresponding point search process for searching for a first corresponding point that is a point;
A template matching unit uses any one of the second captured images acquired from the plurality of imaging devices as a template, and template matching of each of the first corresponding points in each of the second captured image and the other second captured images. Template matching process, and
A virtual camera image generation unit that generates a third captured image captured from a virtual camera arranged between a plurality of the imaging devices by a morphing process of the second captured image; A free viewpoint image capturing method characterized by the above.