JP2008217243A - Image creation device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image creation device for displaying a smooth intermediate image without any feeling of incompatibility in switching a plurality of multi-aspect images. <P>SOLUTION: This image creation device is provided with an intermediate point determination means for specifying a plurality of positions relating to apparently different same articles on the plan view of one or more objects, and for determining the intermediate points of the plurality of positions; and a floor surface image generation means for converting the intermediate points into positions on a multi-view image, and for generating the image of a floor surface according to the positions on the multi-view images, and configured to generate an intermediate image between a plurality of multi-view images from the images of one or more objects placed on the floor surface in a plurality of camera images and the image of the floor surface. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to an image generation apparatus that generates images useful for sports appreciation and monitoring from multi-viewpoint images captured by a plurality of cameras or the like.

  In a conventional image generation apparatus, when generating an intermediate viewpoint image between an image P1 of a certain region in a three-dimensional space photographed from a certain direction and an image P2 of the same region photographed from another direction, An intermediate viewpoint image is generated by finding points V1 and V2 corresponding to arbitrary points in the three-dimensional space from the images P1 and P2 and linearly interpolating these points V1 and V2 (for example, (Refer nonpatent literature 1).

ただし、Ｖ３は中間点、αは重みを表している。
また、ｖ１，ｖ２，ｖ３は、点Ｖ１，Ｖ２，Ｖ３の位置ベクトルを表している。 For example, the points V1 and V2 are linearly interpolated using the following linear interpolation formula.

However, V3 represents an intermediate point and α represents a weight.
Further, v1, v2, and v3 represent position vectors of the points V1, V2, and V3.

Inamoto et al., “Free viewpoint appreciation system for 3D soccer video based on interpolation of viewpoint position”, Video Information Media Society Vol. 58 No. 4 pp529-539 2004

  Since the conventional image generating apparatus is configured as described above, when generating an intermediate viewpoint image between the image P1 and the image P2, the points V1 and V2 corresponding to arbitrary points in the three-dimensional space are linearly interpolated. To do. However, when the points V1 and V2 are linearly interpolated, the intermediate position of the object existing in the three-dimensional space may be determined as an unnatural position. In this case, the intermediate position determined as the unnatural position is determined. Since the distance between two objects and the floor image are generated based on the position, when switching the display between the images P1 and P2 and the intermediate viewpoint image, the object, the floor, etc. may appear to be shrunk or unnatural. There was a problem that an object sometimes moved to a position.

  The present invention has been made to solve the above-described problems, and an object of the present invention is to provide an image generation apparatus capable of displaying an intermediate image that is smooth and has no sense of incongruity when a plurality of multi-viewpoint images are switched. .

  The image generation apparatus according to the present invention includes a plane projection conversion means for converting a position on the floor surface of one or more objects placed on the floor surface into a position on a floor plan of the floor, and conversion by the plane projection conversion means. Intermediate point determination means for specifying a plurality of positions related to the same article that are apparently different on the plan view from among the positions on the plan view of the one or more objects, and determining intermediate points of the plurality of positions; An intermediate image generation unit configured to convert the intermediate point determined by the intermediate point determination unit into a position on the multi-view image and generate a floor image according to the position on the multi-view image; A plurality of multi-view images generated by the multi-view image generation unit from the images of one or more objects placed on the floor surface in the plurality of camera images and the floor image generated by the floor surface image generation unit Will generate an intermediate image It is obtained by the.

  According to the present invention, the plane projection conversion means for converting the position on the floor surface of one or more objects placed on the floor surface into the position on the floor plan of the floor, and the 1 converted by the plane projection conversion means. Among the positions on the plan view of the object described above, a plurality of positions related to the same article that are apparently different on the plan view are specified, and a midpoint determination means for determining a midpoint between the plurality of positions, and a midpoint determination Means for converting the intermediate point determined by the means to a position on the multi-view image, and generating a floor image according to the position on the multi-view image. An intermediate between a plurality of multi-view images generated by the multi-view image generating means from the image of one or more objects placed on the floor surface in the camera image and the floor image generated by the floor image generating means Configured to generate images In, when switching a plurality of multi-view images, there is an effect that can be displayed smooth, no feeling of strangeness intermediate image.

Embodiment 1 FIG.
1 is a block diagram showing a multi-view image generating means of an image generating apparatus according to Embodiment 1 of the present invention.
In the case of the image generating apparatus of FIG. 1, for a plurality of images obtained by photographing N-dimensional images from different directions using N cameras, an object existing in the region is all images. Each image is enlarged or reduced so as to have the same size, moved so as to be positioned at the same coordinates in the image, and further, distortion in the horizontal direction or the vertical direction is corrected.
In addition, after generating the above images using N images taken by N cameras, when these images are displayed in the order of camera arrangement, an equidistant viewpoint centered on an object It is possible to obtain an image effect such that the images are sequentially moved along the camera. Hereinafter, such an effect is referred to as a multi-view effect or simply a multi-view.
In the following description, it is assumed that an area in which several objects are placed on the floor surface, which is a horizontal plane, is captured by N cameras, and an image having a multi-view effect is generated for the area. To do.

  In the figure, the multi-view image generation means 30 performs multi-view conversion of a plurality of camera images in which the same three-dimensional area is photographed from different directions, and is placed on the floor surface in the plurality of camera images. A process of generating a plurality of multi-view images in which the positions and sizes of the objects to be multi-viewed match is executed.

The cameras 1-1 to 1-N photograph the same three-dimensional area from different directions, and output an original image n (1 ≦ n ≦ N) that is a photographing result of the three-dimensional area.
The image data temporary storage unit 2 is a memory for temporarily storing an original image n (1 ≦ n ≦ N) output from the camera 1-n (1 ≦ n ≦ N).
In the first embodiment, however, it is assumed that N = 8 for the sake of simplicity.

The multi-view position designating unit 3 accepts the selection of an arbitrary original image n ₁ (1 ≦ n ₁ ≦ 8) from the eight original images n stored in the image data temporary storage unit 2, and the original image n _A process of accepting designation of image coordinates indicating the position on the floor surface of the multi-view target object (object that the user desires for multi-view) existing in ₁ is performed.
The image coordinates are read by the multi-view position designating unit 3 while the user views the original image n ₁ and, for example, operates the mouse to designate the position in the image, thereby reading the image coordinates at that position. Alternatively, it may be automatically specified by some method.

The reference position specifying unit 4 accepts the selection of an arbitrary original image n ₂ (1 ≦ n ₂ ≦ 8) from the eight original images n stored in the image data temporary storage unit 2, and the original image n _2. The process of accepting designation of image coordinates indicating the position on the floor surface where the vertical reference bar present in FIG.
The reading of the image coordinates in the reference position specifying unit 4 may be performed by reading the image coordinates of the position by, for example, operating the mouse and specifying the position in the image while the user views the original image n _2. Alternatively, it may be automatically specified by some method.

The floor surface coordinate reading unit 5 is an apex of the marking (see FIG. 2) that exists in common on the floor surface of the original image n (1 ≦ n ≦ 8) stored in the image data temporary storage unit 2. (1) A process of reading the image coordinates of the vertex (2), the vertex (3), and the vertex (4) is performed. Here, the vertices (1) to (4) are quadrangular vertices marked on the floor surface.
For each original image n (1 ≦ n ≦ 8) stored in the image data temporary storage unit 2, the horizontal vanishing point calculation unit 6 uses the horizontal vanishing point of the original image n (for example, the optical axis of the camera 1-n as the floor). The process of calculating the image coordinates of the vanishing point obtained by projecting the infinity point of the straight line on the floor surface projected in the direction perpendicular to the surface onto the original image is performed.
For each original image n (1 ≦ n ≦ 8) stored in the image data temporary storage unit 2, the vertical vanishing point calculation unit 7 calculates a vertical vanishing point of the original image n (a straight line infinity point perpendicular to the floor surface). The process of calculating the image coordinates of the vanishing points projected on the image is performed.

The planar projective transformation matrix calculation unit 8 images the marking vertex (1), vertex (2), vertex (3), and vertex (4) in the original image n ₁ and other original images n (excluding the original image n ₁ ). Using the coordinates, a plane projective transformation matrix H _{n1-1 to} H _n1-8 (1 ≦ 1) of the plane constituting the floor surface between the original image n ₁ and another original image n (excluding the original image n ₁ ). The process of calculating 7 pieces of n ₁ ≦ 8 is performed.
The plane projective transformation matrix calculation unit 8 also performs marking vertex (1), vertex (2), vertex (3), vertex (4) in the original image n ₂ and other original images n (excluding the original image n ₂ ). The plane projection transformation matrices H _{n2-1 to} H _n2-8 of the plane constituting the floor surface between the original image n ₂ and another original image n (excluding the original image n ₂ ) A process of calculating 7 of 1 ≦ n ₂ ≦ 8 is performed.
The calculation method of the planar projective transformation matrix is described in, for example, a non-patent document “Koichiro Deguchi“ Basics of Robot Vision ”p47 (described as a homography matrix in this document)”.

Multi-view position calculating unit 9 using the homography matrix _{H n1-1} ~H n1-8 calculated by planar projective transformation matrix calculation unit 8, present in the original image n (excluding original image n ₁₎ Multi A process of calculating image coordinates indicating the position of the object to be viewed on the floor surface is performed. That is, using the plane projection transformation matrices H _{n1-1 to} H _n1-8 , the image coordinates indicating the position on the floor surface of the object of the multi-view target accepted by the multi-view position designating unit 3 (or the multi-view By converting the image coordinates converted by the view image coordinate conversion unit 22 into the image coordinates of the original image n (excluding the original image n ₁ ), the multi-image existing in the original image n (excluding the original image n ₁ ) is converted. A process of calculating image coordinates indicating the position of the object to be viewed on the floor surface is performed.

The reference position calculation unit 10 uses the plane projection transformation matrices H _{n2-1 to} H _n2-8 calculated by the plane projection transformation matrix calculation unit 8 to use the vertical reference existing in the original image n (excluding the original image n ₂ ). A process of calculating image coordinates indicating the position of the bar on the floor is performed. That is, using the plane projection transformation matrices H _{n2-1 to} H _n2-8 , the image coordinates indicating the floor position of the vertical reference bar in the original image n ₂ that has been designated by the reference position designating unit 4 are converted into the original coordinates. by carrying out the processing of converting the image coordinates of the image n (excluding the original image n _2), image coordinates indicating the position on the floor surface of the vertical reference rod present in the original image n (excluding the original image n ₂₎ The process of calculating is performed.

  The in-image magnification calculation unit 11 is present in the original image n calculated by the multi-view position calculation unit 9 and in the original image n calculated by the reference position calculation unit 10. The object of the multi-view target is the same as the vertical reference bar using the image coordinates indicating the floor position of the vertical reference bar and the horizontal vanishing point image coordinates of the original image n calculated by the horizontal vanishing point calculation unit 6. A process of calculating an in-image magnification (change magnification of how the size of the object of the multi-view target is viewed) when the object is moved to the floor position of the depth is performed.

The camera horizontal axis distortion correction parameter calculation unit 12 calculates the horizontal axis distortion caused by the tilt of the camera 1-n when shooting the original image n from the vertical vanishing point of the original image n calculated by the vertical vanishing point calculation unit 7. A process for calculating a correction parameter is performed.
The perspective projection distortion correction parameter calculation unit 13 is based on perspective projection of a three-dimensional space onto a two-dimensional plane from the vertical vanishing point of the original image n calculated by the vertical vanishing point calculation unit 7 when the camera 1-n is photographed. Processing for calculating a correction parameter for perspective projection distortion of an image is performed.

The primary correction image generation unit 14 uses the horizontal axis distortion correction parameter calculated by the camera horizontal axis distortion correction parameter calculation unit 12 and the perspective projection distortion correction parameter calculated by the perspective projection distortion correction parameter calculation unit 13. Then, the horizontal axis distortion and the perspective projection distortion of the original image n are corrected, and the corrected image is output as a primary corrected image n (1 ≦ n ≦ 8).
The reference length reading unit 15 performs a process of reading the apparent length of the vertical reference bar in the primary correction image n output from the primary correction image generation unit 14.

The inter-image magnification calculator 16 calculates the in-image magnification of the object to be viewed by the multi-view target calculated by the in-image magnification calculator 11 and the apparent vertical reference bar in the primary correction image n read by the reference length reader 15. Using the length, a process of calculating an image-to-image magnification for each primary correction image n in a case where an object to be multiviewed in the primary correction image n (1 ≦ n ≦ 8) is displayed with the same size is performed. .
At this time, the size may be corrected by enlarging or reducing the primary correction image n using the inter-image magnification.

  The secondary correction image generation unit 17 includes a horizontal axis distortion correction parameter calculated by the camera horizontal axis distortion correction parameter calculation unit 12, a perspective projection distortion correction parameter calculated by the perspective projection distortion correction parameter calculation unit 13, and an image. By performing secondary perspective transformation using the inter-image magnification calculated by the inter-image magnification calculation unit 16, the distortion and size of the original image n (1 ≦ n ≦ 8) are corrected, and the corrected image Is output as a secondary corrected image n (1 ≦ n ≦ 8).

The secondary perspective transformation coordinate calculation unit 18 is calculated by the secondary correction image generation unit 17 by the horizontal axis distortion correction parameter calculated by the camera horizontal axis distortion correction parameter calculation unit 12 and the perspective projection distortion correction parameter calculation unit 13. The image coordinates of the multi-view target object when the second perspective transformation is performed using the perspective projection distortion correction parameter and the inter-image magnification calculated by the inter-image magnification calculating unit 16 (the multi-view target object image coordinates). A process of calculating the image coordinates of the movement destination is performed.
The movement parameter calculation unit 19 performs a process of calculating, as a movement parameter, a shift between the image coordinates of the movement destination of the multi-view target object calculated by the secondary perspective transformation coordinate calculation unit 18 and the image center.

The multi-view image generation unit 20 moves the secondary correction image n (1 ≦ n ≦ 8) output from the secondary correction image generation unit 17 in accordance with the movement parameter calculated by the movement parameter calculation unit 19. A process of generating a multi-view image n (1 ≦ n ≦ 8) in which a multi-view target object exists at the center of the image is performed.
The multi-view position designation unit 21 performs a process of receiving designation of a new multi-view position for the multi-view image n (1 ≦ n ≦ 8) generated by the multi-view image generation unit 20.

  The multi-view image coordinate conversion unit 22 is a reverse movement parameter of the movement parameter calculated by the movement parameter calculation unit 19, an inverse conversion of the secondary perspective conversion calculated by the secondary perspective conversion coordinate calculation unit 18, and a planar projection conversion matrix. Using the inverse transformation matrix of the planar projective transformation matrix calculated by the calculation unit 8, the coordinate position on the multi-view image n (1 ≦ n ≦ 8) received by the multi-view position specification unit 21 is converted into the original A process of converting to a coordinate position on the image n (1 ≦ n ≦ 8) is performed.

FIG. 10 is a block diagram showing a part of the image generating apparatus (part excluding the multi-view image generating means) according to Embodiment 1 of the present invention.
However, the cameras 1-1 to 1-N and the image data temporary storage unit 2 described in FIG. 1 are also described in FIG.

The background image data storage unit 41 acquires the background image n captured by the camera 1-n (1 ≦ n ≦ N) from the image data temporary storage unit 2, and stores the background image n for each camera 1-n. It is memory.
Here, the “background image” is an image in which an object moving in the screen or an object expected to move in the future is not photographed.
In the first embodiment, the camera 1-n is not moved after the background image is acquired.
The background image may be selected by the user from a plurality of images, or may be generated by the user. Further, it may be automatically generated by executing a method described later.
In general, a background image is often obtained by photographing a state where there is no moving object.
As a background image automatic selection method or generation method, for example, there is a method of generating a background image from an image of a moving object photographed using a median filter or the like.
The background image stored in the background image data storage unit 41 may be acquired periodically or aperiodically. Further, the background image may be updated when a difference or the like for each background image is obtained and when there is a certain threshold or more that has been found not to affect the extraction of objects such as persons.

The object extraction unit 42 acquires the original image n captured by the camera 1-n (1 ≦ n ≦ N) from the image data temporary storage unit 2 and is captured by the camera 1-n from the background image data storage unit 41. A background image n is acquired, and image processing is performed on the original image n and the background image n to extract an object (an object placed on the floor) in the original image n. That is, a pixel having motion is determined by obtaining a difference between pixels at the same position in the original image n and the background image n, the position of the pixel having motion is represented by a binary image, and the binary image is contracted. An object in the original image n is extracted by performing processing, enlargement processing, labeling processing, and the like.
Shrinkage processing, enlargement processing, labeling processing, and the like are disclosed in, for example, “Practical image processing learned by C language Ohmsha”.

When the object extraction unit 42 extracts an object, the object position coordinate calculation unit 43 uses, for example, the center of gravity of the object region, the plane projection transformation matrix calculated by the plane projection transformation matrix calculation unit 8, and the vertical vanishing point calculation unit 7. A process of calculating the position coordinates of the object in the original image n is performed using the calculated vertical vanishing point.
The floor plan view input unit 44 captures a plan view of the floor surface photographed in common by the cameras 1-1, 1-2,.
The “plan view of the floor surface” is, for example, a square if the quadrangular shape marked on the floor surface is a square, and an a: b rectangle if the aspect ratio is an a: b rectangle. As long as the four vertices can be determined in a similar shape, the size may be appropriate.
Further, it may be a triangle other than a quadrangle, a polygon other than that, or any other shape, and the positional relationship between four or more points on the plane may be a geometric similarity.

The floor coordinate designating unit 45 in the image has the coordinates of the point corresponding to the vertex of the floor surface captured by the floor plan input unit 44 (the position of the floor surface) in the background image n stored in the background image data storage unit 41. (Coordinates) is specified.
The method for specifying the position coordinates of the floor surface may be specified by the user while viewing the image, or may be automatically specified by some method.
In the first embodiment, the floor surface coordinates in the background image are designated, but the floor surface coordinates in the original image other than the background image may be designated.

The input image coordinate acquisition unit 46 calculates the position coordinates of the object in the original image n calculated by the object position coordinate calculation unit 43 or the position of the floor surface in the background image n specified by the in-image floor surface coordinate specification unit 45. A process of selecting one of the coordinates is performed.
The object extraction unit 42, the object position coordinate calculation unit 43, the floor plan input unit 44, the in-image floor coordinate specification unit 45, and the input image coordinate acquisition unit 46 constitute a floor surface acquisition unit.

The multi-view conversion equation generation unit 47 uses the camera horizontal axis distortion correction parameter calculated by the camera horizontal axis distortion correction parameter calculation unit 12 and the perspective projection distortion correction parameter as image conversion parameters when the multi-view image generation unit 30 performs multi-view conversion. The perspective projection distortion correction parameter calculated by the calculation unit 13 and the inter-image magnification calculated by the inter-image magnification calculation unit 16 are collected, and the camera horizontal axis distortion correction parameter, the perspective projection distortion correction parameter, and the inter-image magnification are used. Thus, when the original image is converted into a multi-view image, a multi-view conversion formula (for example, 3) can be calculated as to which coordinate on the multi-view image is converted from a certain position on the original image. A process of generating (× 3 matrix) is performed.
The inverse multiview conversion equation generation unit 48 performs a process of generating an inverse multiview conversion equation that is an inverse conversion equation of the multiview conversion equation generated by the multiview conversion equation generation unit 47. By the inverse multi-view conversion formula, it is possible to calculate which coordinate is in the original image before the coordinate of a certain position on the multi-view image is subjected to the multi-view conversion.

The converted image coordinate calculation unit 49 uses the multi-view conversion formula generated by the multi-view conversion formula generation unit 47 to use the position coordinates (or background image) of the object in the original image n selected by the input image coordinate acquisition unit 46. The processing of converting the position coordinates of the floor surface in n) into the position coordinates on the multi-view image is performed.
The original image coordinate calculation unit 50 uses the inverse multiview conversion equation generated by the inverse multiview conversion equation generation unit 48 to use the position coordinates (each camera 1- 1) on the multiview image converted by the conversion image coordinate calculation unit 49. a process of converting n (position coordinates on the multi-view image converted from the original image n of 1 ≦ n ≦ N) into position coordinates on the original image of a certain camera (for example, camera 1-1). carry out.
Here, the original image of a certain camera may be, for example, the original image currently displayed by the image generation device, or the original image of the camera that captured the multi-view image, It may be an original image of another camera.

The plane projective transformation calculation unit 51 uses the floor surface vertices (coordinate values of four or more points on the plan view) captured by the floor plan view input unit 44 and the background image n specified by the in-image floor surface coordinate specification unit 45. A plane projective transformation matrix (for example, a plane projection transformation matrix used when the floor plan view coordinate calculation unit 53 calculates the floor plan view coordinates from the position coordinates (four or more coordinate values corresponding to the vertices of the floor surface) of (3 × 3 matrix) is calculated for each camera 1-n.
The plane projective transformation calculation unit 52 uses the floor surface vertices (coordinate values of four or more points on the plan view) captured by the floor plan view input unit 44 and the background image n specified by the in-image floor surface coordinate specification unit 45. A plane projective transformation matrix (for example, a plane projection transformation matrix used when the original image coordinate calculation unit 55 calculates the coordinates on the original image from the position coordinates of the floor surface (coordinate values of four or more points corresponding to the vertices of the floor surface). (3 × 3 matrix) is calculated for each camera 1-n.
Note that, in an image shot by a certain camera, the plane projection transformation matrix calculated by the plane projection transformation calculation unit 51 and the plane projection transformation matrix calculated by the plane projection transformation calculation unit 52 are inversely related to each other. In this case, after calculating one of the plane projection transformation matrices, calculate the inverse of the plane projection transformation matrix mathematically, and calculate the remaining one of the plane projection transformation matrices. You may make it ask.

The floor plan coordinate calculation unit 53 uses the plane projection conversion matrix calculated by the plane projection conversion calculation unit 51 to use the position coordinates on the original image (for example, the camera 1-1) calculated by the original image coordinate calculation unit 50. A process of converting the position coordinates on the original image) to the position coordinates on the floor plan of the floor is performed.
Note that the multi-view conversion formula generation unit 47, the inverse multi-view conversion formula generation unit 48, the converted image coordinate calculation unit 49, the original image coordinate calculation unit 50, the plane projection conversion calculation unit 51, and the floor plan view coordinate calculation unit 53 perform planar projection. Conversion means is configured.

The floor plan intermediate coordinate calculation unit 54 selects a plurality of positions related to the same object on the plan view from among the position coordinates on the plan view of one or more objects converted by the floor plan coordinate calculation unit 53. A process of specifying coordinates and determining an intermediate point of a plurality of position coordinates is performed.
That is, the floor plan intermediate coordinate calculation unit 54 lies on a circle or ellipse arc centered at the position on the plan view of the object to be multiviewed when determining the intermediate point of a plurality of position coordinates related to the same object. A point is determined as the midpoint. The floor plan intermediate coordinate calculation unit 54 constitutes an intermediate point determination unit.

The original image coordinate calculation unit 55 uses the plane projection conversion matrix calculated by the plane projection conversion calculation unit 52 to convert the midpoint determined by the floor plan intermediate coordinate calculation unit 54 into position coordinates on the original image. To implement.
At this time, it may be converted into coordinates on one original image, or may be converted into coordinates on several original images with different cameras.
The converted image coordinate calculation unit 56 uses the multi-view conversion formula generated by the multi-view conversion formula generation unit 47 to convert the position coordinate on the original image converted by the original image coordinate calculation unit 55 into the position coordinate on the multi-view image. Perform the process of converting to.

The floor surface intermediate image plane projection conversion calculation unit 57 uses the position coordinates on the multi-view image converted by the conversion image coordinate calculation unit 49 and the position coordinates on the multi-view image converted by the conversion image coordinate calculation unit 56. Then, a process of calculating a plane projection transformation matrix (for example, a 3 × 3 matrix) used when the floor intermediate image generation unit 59 generates an intermediate image of the floor is performed.
The floor area extraction unit 58 performs a process of extracting a floor area from the background image n of the camera 1-n (1 ≦ n ≦ N) stored in the image data temporary storage unit 2.
As a method for extracting a floor area, there is a method in which a user designates a floor area from a background image.
Further, for example, using a plane projection conversion matrix between cameras calculated by the plane projection conversion matrix calculation unit 8, a background image shot by one camera is converted into a background image shot by another camera, and both The floor area may be determined by performing, for example, a contraction process, an enlargement process, or the like on pixels whose difference is equal to or less than a certain threshold.
Further, without using a background image, an image corresponding to the background image may be generated and used from an image in which an object is photographed using a median filter or the like.

The floor intermediate image generation unit 59 uses the plane projection transformation matrix calculated by the floor surface intermediate image plane projection transformation calculation unit 57 to convert the floor area extracted by the floor area extraction unit 58 into the intermediate image of the floor surface. Perform the process of converting to.
From the plane projection conversion calculation unit 52, the original image coordinate calculation unit 55, the converted image coordinate calculation unit 56, the floor surface intermediate image plane projection conversion calculation unit 57, the floor surface area extraction unit 58, and the floor surface intermediate image generation unit 59. Floor surface image generation means is configured.

The intermediate image object generation unit 60 performs processing for generating an intermediate image object from the objects extracted by the object extraction unit 42.
A method for generating an object for an intermediate image is described in, for example, the non-patent document “Satoru Yaguchi et al. SIG 6 (CVIM 2) June 2001 ”, and in particular, an object for an intermediate image is generated using an arbitrary viewpoint image described in“ Section 3.2 (3) (p13) ”.

The object size correction unit 61 generates a multi-view conversion formula on the assumption that the intermediate image object generated by the intermediate image object generation unit 60 exists in the position coordinates on the multi-view image converted by the original image coordinate calculation unit 55. Using the multi-view conversion formula generated by the unit 47, a process for correcting the size of the object for the intermediate image is performed.
The intermediate image generation unit 62 uses the floor surface intermediate image generation unit 59 to convert the intermediate image object whose size has been corrected by the object size correction unit 61 at the position coordinates on the multi-view image converted by the converted image coordinate calculation unit 56. A process of generating an intermediate image is performed by overwriting the generated intermediate image of the floor surface.
The intermediate image generating unit 60, the object size correcting unit 61, and the intermediate image generating unit 62 constitute intermediate image generating means.

Next, the operation will be described.
FIG. 2 is an explanatory diagram showing a situation where images are taken using eight cameras 1-n (1 ≦ n ≦ 8), and FIG. 3 shows cameras 1-1, 1-3, and 1-5. , 1-7 are explanatory diagrams showing images taken by 1-7.

In the first embodiment, as shown in FIG. 2, a floor surface composed of a horizontal plane is considered, and a “stuffed animal” or “vase” is placed on the floor surface, and the surroundings of the “stuffed animal” or “vase” are arranged. It is assumed that eight cameras 1-n (1 ≦ n ≦ 8) fixed on a tripod of substantially the same size are arranged so as to surround.
However, it is assumed that all of the eight cameras 1-n (1 ≦ n ≦ 8) are installed at positions where the “floor surface area within the field of view of all cameras” shown in FIG.
In FIG. 2, the installation form of the eight cameras 1-n and the “floor area that falls within the field of view of all cameras” are indicated by a circular or elliptical dotted line. However, this dotted line does not actually exist. Good.

Further, in the example of FIG. 2, a square marking that fits the frame of “a floor area that falls within the field of view of all cameras” is provided, and all eight cameras 1-n (1 ≦ n ≦ 8) It is assumed that the four corners of the marking can be photographed without being disturbed by “stuffed animals” or “vases”.
In the example of FIG. 2, a case using a quadrangular marking is shown, but it is not limited to a quadrangular marking as long as four or more points can be photographed.
In addition, although eight cameras 1-n are supposed to shoot four points at the same time, it is sufficient that at least four or more of the same points can be shot simultaneously by two different cameras, and they may not be quadrangular four corners.

In FIG. 2, the vertical reference bar is a bar that stands vertically near the center of the “floor area that enters the field of view of all cameras”, and is marked at a position that is half the total length. .
The topmost vertex of the vertical reference bar, the center marking, and the contact point in contact with the floor surface are all the eight cameras 1-n (1 ≦ n ≦ 8) are used as “stuffed animals” and “vases”. It is assumed that the camera can be photographed without interruption.
The vertical reference rod is sufficiently thin and can be regarded as a straight line when photographed by the camera 1-n (1 ≦ n ≦ 8).

In the example of FIG. 2, the vertical reference rod is used for the purpose of reading the size when taken in different directions and calculating the vanishing point obtained by projecting the infinity point of a straight line perpendicular to the plane onto the image. It is what is done.
The vertical reference bar is shaped as described above to improve accuracy, but the vertical reference bar is used only for reading the size, and the vanishing point is calculated by another method as described later. You may make it calculate. In this case, the vertical reference rod does not have to have the form as described above, and may be substituted by, for example, an actual person.

The vertical reference bar in FIG. 2 is marked at a position of a half of the entire length, but may not be a position of a half, and it is sufficient that the ratio is known.
Further, the vertical reference bar in FIG. 2 is set near the center of the “floor surface area that enters the field of view of all cameras”, but it may not be near the center.
In addition, the vertical reference bar in FIG. 2 can be photographed from all the cameras 1-n (1 ≦ n ≦ 8), with the topmost vertex, the contact point in contact with the floor surface, and the central marking. It is not always necessary to photograph the same vertical reference rod. For example, the vertical reference bar A photographed in common by the camera 1-1 and the camera 1-2 and the vertical reference stick B photographed in common by the camera 1-1 and the camera 1-3 may not be the same. . In this case, it is only necessary to know the size of the vertical reference rod A and the size of the vertical reference rod B in the camera 1-1.

In the first embodiment, a vertical reference rod, which is a rod, is used as a reference for the vertical direction and size. In this case, once installed, the shape is not easily lost, and straight lines can be easily measured. There is an advantage that is.
In the first embodiment, a vertical reference rod is used, but a string or the like may be used. In this case, a weight or the like is attached to the tip, and the weight is suspended vertically so that the tip contacts the floor. In this case, there are advantages that the accuracy in the vertical direction is high and the length can be easily adjusted.

In FIG. 2, a horizontal reference bar for the camera 1-7 is drawn.
The horizontal reference bar is, for example, a floor so that the optical axis of the target camera (camera 1-7 in the example of FIG. 3) is parallel to a straight line projected onto a plane along a direction perpendicular to the floor surface. It is installed on the surface.
In addition, marking is provided at a half of the length of the horizontal reference bar. It is assumed that the markings at both ends and the center of the horizontal reference bar are ready to be photographed without being disturbed by the “stuffed toy” or “vase” from the camera 1-7.
The horizontal reference rod is sufficiently thin and can be regarded as a straight line when photographed by the camera 1-7.

In addition, the horizontal reference bar is marked at a position that is a half of the entire length, but it may not be a position that is a half, and it is sufficient that the ratio is known.
In addition, the horizontal reference bar is, for example, “installed on the floor so that the optical axis of the camera is parallel to a straight line projected on the floor along the direction perpendicular to the floor”, but in this case, There is an advantage that accuracy is easily improved.
In addition, the horizontal reference bar may be in either direction, and a vanishing point obtained by projecting a straight infinity point on the floor surface including the horizontal reference bar onto the image may be used. In this case, there is an advantage that it is not necessary to move the horizontal reference rod for each camera.

Furthermore, as shown in FIG. 2, “a square side marked on the floor surface” may be used.
In addition, this quadrilateral is set as a square, a rectangle, or a parallelogram, and the infinity point of the straight line on the floor surface is obtained using the two sides facing each other, and the vanishing point obtained by projecting the infinity point onto the image is used. May be.
In the first embodiment, a bar called a horizontal reference bar is used as a horizontal reference, but it can be replaced with a string or the like as described above in the “example of using a string instead of a vertical reference bar”. May be. In that case, it is necessary to apply tension so that the string becomes a straight line, but the same advantage as the case where the vertical reference bar is replaced with the string can be obtained.

In the first embodiment, the cameras 1-1 to 1-8 release the shutter at the same synchronization timing from the outside, and shoot an image with a frame at the same synchronization timing.
Shooting with such synchronized timing is essential when the subject moves. For example, if the stuffed animal, vase, horizontal reference bar / vertical reference bar, floor marking, etc. do not move, It does not have to be synchronized. In that case, it is possible to sequentially shoot with one camera without preparing eight cameras 1-1 to 1-8.
Although it is indispensable when the subject moves, it does not necessarily have to be synchronized if the frame period to be photographed is sufficiently short and the amount of movement during that period is small enough.

  In the first embodiment, a subject to be multi-viewed such as “stuffed toy” or “vase”, a horizontal reference bar / vertical reference bar for obtaining a parameter necessary for multi-viewing, a floor marking, etc. After taking a picture, the horizontal reference bar / vertical reference stick and the marking on the floor surface may be removed without moving the camera, and a subject to be multi-viewed such as “stuffed toy” may be placed and photographed. In this case, care must be taken not to move the camera, but there is an advantage that the horizontal reference bar / vertical reference bar and the marking on the floor surface can be photographed without being obstructed by the subject to be multiviewed.

Hereinafter, the processing content of the multi-view image generation means 30 in the image generation apparatus will be described in detail.
When the camera 1-n (1 ≦ n ≦ 8) captures the same three-dimensional area from different directions, the image data temporary storage unit 2 is an original image captured by the camera 1-n (1 ≦ n ≦ 8). n (1 ≦ n ≦ 8) is temporarily stored.

When the image data temporary storage unit 2 stores the original image n (1 ≦ n ≦ 8), the multi-view position specifying unit 3 stores an arbitrary original image n ₁ from the original image n (1 ≦ n ≦ 8). A process of accepting selection and accepting designation of image coordinates indicating a position on the floor surface of an object to be viewed in the original image n ₁ (an object for which the user desires multi-view) is performed.
The image coordinates are read by the multi-view position designating unit 3 while the user views the original image n ₁ and, for example, operates the mouse to designate the position in the image, thereby reading the image coordinates at that position. Alternatively, it may be automatically specified by some method.

Further, when the image data temporary storage unit 2 stores the original image n (1 ≦ n ≦ 8), the reference position specifying unit 4 selects an arbitrary original image n ₂ from the original image n (1 ≦ n ≦ 8). And a process of receiving designation of image coordinates indicating the position on the floor surface where the vertical reference bar existing in the original image n ₂ is in contact with the floor surface.
The reading of the image coordinates in the reference position specifying unit 4 may be performed by reading the image coordinates of the position by, for example, operating the mouse and specifying the position in the image while the user views the original image n _2. Alternatively, it may be automatically specified by some method.
When the image data temporary storage unit 2 stores the original image n (1 ≦ n ≦ 8), the floor surface coordinate reading unit 5 exists in common on the floor surface of the original image n (1 ≦ n ≦ 8). Processing for reading the image coordinates of the vertex (1), vertex (2), vertex (3), and vertex (4) of the marking is performed.
The reading of the image coordinates in the floor surface coordinate reading unit 5 may be performed by, for example, operating the mouse to specify the position in the image while the user views the original image n, thereby reading the image coordinates at that position. Alternatively, it may be automatically specified by some method.

In addition, when the image data temporary storage unit 2 stores the original image n (1 ≦ n ≦ 8), the horizontal vanishing point calculation unit 6 performs horizontal processing of the original image n for each original image n (1 ≦ n ≦ 8). The vanishing point, for example, the image coordinates of the vanishing point obtained by projecting the infinity point of a straight line on the floor surface obtained by projecting the optical axis of the camera 1-n in the direction perpendicular to the floor surface is calculated.
As a method of calculating the horizontal vanishing point, for example, it can be obtained by using a horizontal reference bar placed on a straight line that satisfies the above-mentioned conditions of the floor surface.
That is, the horizontal reference bar is marked, and the horizontal vanishing point can be obtained from the actual position of the marking and the apparent position on the image. Since this is the same as the method for obtaining the vanishing point, see the method for obtaining the vertical vanishing point described later).
In addition, it is also possible to set two or more straight lines parallel to the straight line on the floor and use the intersection point when these straight lines are projected on the image as vanishing points.

When the image data temporary storage unit 2 stores the original image n (1 ≦ n ≦ 8), the vertical vanishing point calculation unit 7 calculates the vertical vanishing point of the original image n for each original image n (1 ≦ n ≦ 8). That is, the image coordinates of the vanishing point obtained by projecting a straight line infinity point perpendicular to the floor surface to the image are calculated.
Hereinafter, an example of a method for calculating the vertical vanishing point will be described.
In the first embodiment, a vertical reference bar exists in all original images n (1 ≦ n ≦ 8), and the vertical reference bar has a marking at the center in the length direction as shown in FIG. It has been subjected.
When the vertical reference bar is projected on the image as shown in FIG. 6, the three points X1, X2, and X3 on the vertical reference bar and the infinity point X _∞ on the straight line passing through the three points X1, X2, and X3. but in the image, respectively x1, x2, x3, and those projected to x _∞.

式（１）において、Ｘ_∞は無限に遠い点であるから｜Ｘ_∞−Ｘ₂｜／｜Ｘ_∞−Ｘ₃｜＝１であり、また、垂直基準棒の実距離の比は｜Ｘ₁−Ｘ₂｜／｜Ｘ₁−Ｘ₃｜＝２である。
また、｜ｘ₁−ｘ₂｜／｜ｘ₁−ｘ₃｜は、画像座標から得ることができるので、垂直消失点ｘ_∞の座標（ｘ_vin，ｙ_vin）を算出することができる。
詳細は非特許文献「小島他 “消失点を用いた多視点カメラキャリブレーション”ＦＩＴ２００４」に開示されている。
この他、上記床面に垂直な直線（棒などで代用する）を２つ以上設定して、それらの直線を画像に投影した場合の２つの直線の交点を垂直消失点として利用することもできる。 At this time, since the cross ratio calculated from the four points on the straight line is unchanged before and after the projection, the following equation (1) is established.

In the formula (1), X _∞ infinitely distant point because it _{| X ∞ -X 2 | / |} X ∞ -X 3 | a = 1, also, the ratio of the actual distance of the vertical reference rod | X _{1 -X 2 | / | X 1 -X} 3 | a = 2.
Since | x ₁ −x ₂ | / | x ₁ −x ₃ | can be obtained from the image coordinates, the coordinates (x _vin , y _vin ) of the vertical vanishing point x _∞ can be calculated.
Details are disclosed in the non-patent document “Kojima et al.“ Multi-viewpoint camera calibration using vanishing points ”FIT 2004”.
In addition, it is possible to set two or more straight lines (substitute with a bar or the like) perpendicular to the floor surface and use the intersection of the two straight lines as a vertical vanishing point when these straight lines are projected on an image. .

The planar projective transformation matrix calculation unit 8 is configured such that the floor coordinate reading unit 5 has the marking vertex (1) and vertex (2) that are commonly present on the floor surface of the original image n (1 ≦ n ≦ 8), for example. When the image coordinates of the vertex (3) and the vertex (4) are read, the marking vertex (1), vertex (2), vertex (in the original image n ₁ and other original images n (excluding the original image n ₁ ) 3) Using the image coordinates of the vertex (4), a plane projective transformation matrix H _n1− of the plane constituting the floor surface between the original image n ₁ and another original image n (excluding the original image n ₁ ). _{1 to} H _n1-8 (1 ≦ n ₁ ≦ 7) are calculated.
Further, the plane projective transformation matrix calculation unit 8 performs marking vertex (1), vertex (2), vertex (3), vertex (4) in the original image n ₂ and other original images n (excluding the original image n ₂ ). ), The plane projection transformation matrices H _{n2-1 to} H _n2-8 of the plane constituting the floor surface between the original image n ₂ and another original image n (excluding the original image n ₂ ) are _used. (7 of 1 ≦ n ₂ ≦ 8) is calculated.
The calculation method of the planar projective transformation matrix is described in, for example, the non-patent document “Koichiro Deguchi“ Basics of Robot Vision ”p47 (described as a homography matrix in this document)”.

When the plane projection transformation matrix calculation unit 8 calculates the plane projection transformation matrices H _{n1-1 to} H _n1-8 , the multi-view position calculation unit 9 uses the plane projection transformation matrices H _{n1-1 to} H _n1-8. Then, the image coordinates indicating the position on the floor surface of the object of the multi-view target existing in the original image n (excluding the original image n ₁ ) are calculated.
That is, the multi-view position calculation unit 9 uses the plane projection transformation matrices H _{n1-1 to} H _n1-8 to determine the position on the floor of the object to be multi-viewed that has been designated by the multi-view position designation unit 3. Or the image coordinates converted by the multi-view image coordinate conversion unit 22 is converted into the image coordinates of the original image n (excluding the original image n ₁ ), thereby obtaining the original image n (original image n _1). Image coordinates indicating the position on the floor surface of the object of the multi-view target existing in (1) is calculated.
Note that the image coordinates indicating the position on the floor surface of the object of the multi-view target that has been designated by the multi-view position designating unit 3 are converted, or the image coordinates converted by the multi-view image coordinate converting unit 22 are converted. For conversion, for example, the former may be selected only when the system is activated, or both may be monitored constantly and the one that has changed may be selected.

When the plane projection transformation matrix calculation unit 8 calculates the plane projection transformation matrices H _{n2-1 to} H _n2-8 , the reference position calculation unit 10 uses the plane projection transformation matrices H _{n2-1 to} H _n2-8 . Image coordinates indicating the position of the vertical reference bar on the floor surface existing in the original image n (excluding the original image n ₂ ) are calculated.
That is, the reference position calculation unit 10 uses the plane projection transformation matrices H _{n2-1 to} H _n2-8 to specify the floor position of the vertical reference bar in the original image n ₂ that has been specified by the reference position specification unit 4. Is converted into the image coordinates of the original image n (excluding the original image n ₂ ), so that the position on the floor surface of the vertical reference bar existing in the original image n (excluding the original image n ₂ ) is converted. The image coordinates shown are calculated.

Up to this point, the original image whose selection has been received by the multi-view position specifying unit 3 is the original image n ₁ , and the original image whose selection has been received by the reference position specifying unit 4 is the original image n ₂ . However, the original image n ₁ and the original image n ₂ may be separate images or the same image.
If the original image n ₁ and the original image n ₂ are the same original image (n ₁ = n ₂ ), there are _seven plane projection matrices H _{n1-1 to} H _n1-8 (1 ≦ n ₁ ≦ 8). Only need.
Hereinafter, a case where n ₁ = n ₂ will be described.

The in-image magnification calculation unit 11 calculates image coordinates indicating the floor surface position of the multi-view target existing in the original image n (1 ≦ n ≦ 8) by the multi-view position calculation unit 9, and the reference position calculation unit 10 performs the original position calculation. When the image coordinates indicating the position on the floor surface of the vertical reference bar existing in the image n are calculated, the image coordinates and the image coordinates of the horizontal vanishing point of the original image n calculated by the horizontal vanishing point calculating unit 6 are obtained. Use this to move the multi-view target object to the floor position at the same depth as the vertical reference bar. In-image magnification m _{n of the} multi-view target object (change in the appearance of the multi-view target object Magnification) is calculated.

Hereinafter, a method for calculating the in-image magnification will be described.
FIG. 4 shows an example in which the horizontal vanishing point is at the center of the image, and the camera is installed so that the optical axis of the camera is parallel to the floor and the horizontal axis of the camera image is substantially parallel to the floor. It is explanatory drawing which shows.
FIG. 5 shows a case where the horizontal vanishing point is above the center of the image, and the horizontal axis of the camera image is substantially parallel to the floor surface, but the optical axis of the camera is slightly downward with respect to the horizontal floor surface. It is explanatory drawing which shows the example currently image | photographed.

また、これらの垂直基準棒の位置や大きさ、および水平消失点の位置を図４や図５に示すように利用することで、被写体である物体の大きさや人物の身長などを精度よく推定することができる。また、それらの物体や人物の大きさがわかっている場合、その物体や人物が存在する床面上の位置を容易に知ることができる。 4 and 5, the coordinates of the horizontal vanishing point are (x _hin , y _hin ), the position coordinates of the multi-view target object are (x _mv , y _mv ), and the position coordinates of the same depth as the vertical reference bar are (x Assuming that _b , y _b ), the size of the multi-view target object can be approximated by the following equation (2) when the multi-view target object is moved to the position of the vertical reference bar. Therefore, the in-image magnification m _n can be calculated from the following equation (2).

Further, by using the position and size of these vertical reference bars and the position of the horizontal vanishing point as shown in FIG. 4 and FIG. 5, the size of the object that is the subject, the height of the person, etc. can be accurately estimated. be able to. Further, when the sizes of these objects and persons are known, the position on the floor where the objects and persons are present can be easily known.

When the vertical vanishing point calculating unit 7 calculates the coordinates (x _vin , y _vin ) of the vertical vanishing point x _∞ of the original image n (1 ≦ n ≦ 8), the camera horizontal axis distortion correction parameter calculating unit 12 calculates the original image n. Using the coordinates (x _vin , y _vin ) of the vertical vanishing point x _∞ , a correction parameter for the horizontal axis distortion caused by the tilt of the camera 1-n when the original image n is captured is calculated.
That is, for example, the camera horizontal axis distortion correction parameter calculation unit 12 performs the image center point B (x _size / 2, y _size / 2) and the vertical vanishing point x with respect to the vertical axis of the image as shown in FIG. _Considering that the inclination (angle rz) of the straight line connecting _∞ can be approximated to the camera horizontal axis distortion, the angle rz is calculated.
For example, the horizontal size of the original image is x _size , the vertical size is y _size , the coordinates of the vertical vanishing point x _∞ are (x _vin , y _vin ), and y _vin >> x _size , y _vin >> y _size If there is, the following equation (3) is established.

Here, with respect to the vertical axis of the image, the inclination (angle rz) of the straight line connecting the center point B (x _size / 2, y _size / 2) of the image and the vertical vanishing point x _∞ is approximated to the camera horizontal axis distortion. Assuming that it is possible to calculate the angle rz, the center point of the upper end of the image (x _size / 2, 0) and the center point of the lower end of the image (x _{size are shown).} / 2, y _size ) may be regarded as being able to approximate the camera horizontal axis distortion.
Furthermore, instead of approximation, an accurate angle may be calculated and used.
Also, the other image of the upper end of the middle point (x _size / 2,0), the midpoint of the lower end of the image _{(x size / 2, y size} ) may be used a point on a line connecting the, other A point inside the image may be used. Furthermore, you may utilize the point on the plane containing an image.

The perspective projection distortion correction parameter calculation unit 13 calculates the vertical vanishing point x when the vertical vanishing point calculation unit 7 calculates the coordinates (x _vin , y _vin ) of the vertical vanishing point x _∞ of the original image n (1 ≦ n ≦ 8). _Using the coordinates (x _vin , y _vin ) of _∞ , a correction parameter for the perspective projection distortion of the image resulting from the perspective projection of the three-dimensional space onto the two-dimensional plane at the time of photographing by the camera 1-n is calculated.
The perspective projection distortion is a distortion that is considered to be caused by photographing with a camera, that is, a distortion that is considered to be generated by projective transformation of an object in a three-dimensional space to a plane. This is a phenomenon in which when there are a plurality of straight lines perpendicular to the floor surface and the straight lines are projected on the image, the straight lines are not parallel to each other on the image.

As shown in FIG. 6, the perspective projection distortion correction parameter calculation unit 13 includes an image including the center point B (x _size / 2, y _size / 2) of the image with respect to the image whose camera horizontal axis distortion has been corrected previously. Perspective projection distortion at an angle (angle rx) between the straight line connecting the first pixel (0, y _size / 2) of the horizontal line and the vertical vanishing point and the vertical axis of the image after correcting the camera horizontal axis distortion Is approximated and the angle rx is calculated.
For example, the horizontal size of the original image is x _size , the vertical size is y _size , the coordinates of the vertical vanishing point x _∞ are (x _vin , y _vin ), and y _vin >> x _size , y _vin >> y _size If there is, the following equation (4) is established.

Here, assuming that the horizontal axis distortion correction angle rz is sufficiently small, the perspective projection distortion correction angle rx is defined as “the first horizontal line of the image including the center point B (x _size / 2, y _size / 2) of the image”. It is assumed that the perspective projection distortion can be approximated by the inclination (angle rx) of the vertical axis at one pixel (0, y _size / 2). However, as shown in FIG. 6, the center point of the image perpendicular to the straight line BX _∞ The coordinates of a point on a straight line passing through may be used.
Further, “the perspective projection distortion is approximated by the inclination of the vertical axis at the x _size pixel (x _size , y _size / 2) of the horizontal line of the image including the center point B (x _size / 2, y _size / 2) of the image. Can be used ", a straight line connecting points on the left and right ends of another image may be used, or a straight line connecting points in the other image may be used.
Furthermore, instead of approximation, an accurate angle may be calculated and used.

When the camera horizontal axis distortion correction parameter calculation unit 12 calculates a correction parameter for horizontal axis distortion and the perspective projection distortion correction parameter calculation unit 13 calculates a correction parameter for perspective projection distortion, the primary correction image generation unit 14 calculates the horizontal correction. Using the correction parameter for axial distortion and the correction parameter for perspective projection distortion, the horizontal axis distortion and perspective projection distortion of the original image n are corrected, and the corrected image is defined as a primary corrected image n (1 ≦ n ≦ 8). Output.
That is, as shown in FIG. 7, the primary corrected image generation unit 14 sets the three-dimensional coordinate axis so that the origin of the three-dimensional coordinate is the image center and the image is included in the XY plane, and performs perspective projection. Consider a case where the screen is set on a plane perpendicular to the Z axis and the viewpoint is set on the Z axis.
At this time, the image is rotated by rz around the Z axis, and a perspective projection image on the screen when the image is rotated by rx around the X axis is calculated and output as a primary correction image.
At this time, rz is a horizontal axis distortion correction parameter calculated by the camera horizontal distortion correction parameter calculation unit 12, and rx is a perspective projection distortion correction parameter calculated by the perspective projection distortion correction parameter calculation unit 13.

FIG. 8 is an explanatory view of the three-dimensional coordinates of FIG. 7 viewed from the positive direction of the X axis.
In the example of FIG. 8, the camera horizontal axis distortion correction parameter calculation unit 12 calculates a horizontal axis distortion correction parameter at the image center, and the perspective projection distortion correction parameter calculation unit 13 at a point on the horizontal axis of the image including the image center. The perspective transformation distortion correction parameter is calculated, and the primary correction image generation unit 14 performs the perspective projection transformation with the image center as the origin. The horizontal axis distortion correction parameter and the perspective transformation distortion correction parameter of the point that is not the image center are set. It may be calculated and perspective projection conversion with the point that is not the center of the image as the origin may be performed. Further, the perspective transformation may be performed without matching the two.

When the primary correction image generation unit 14 outputs the primary correction image n (1 ≦ n ≦ 8), the reference length reading unit 15 determines the apparent length L _n of the vertical reference bar in the primary correction image n. read. At this time, the user may read while viewing the primary correction image n, or may automatically read by some method.
Here, the reading of the apparent length L _n of the vertical reference bar in the primary correction image n is shown, but the apparent length L _n of the vertical reference bar in the original image n may be read. .
In this case, since the process becomes simple, there is an effect that the processing speed can be improved.
When the reference length reading unit 15 reads the apparent length L _n of the vertical reference bar in the primary correction image n, the inter-image magnification calculation unit 16 apparent length of the vertical reference bar in the primary correction image n The same multi-view target object in the primary correction image n (1 ≦ n ≦ 8) is used by using L _n and the intra-image magnification m _n of the multi-view target object calculated by the intra-image magnification calculation unit 11. An inter-image magnification ratio m _o1n when displaying in size is calculated for each primary correction image n.

ただし、Ｌ₁はオリジナル画像ｎ₁の１次補正画像ｎ₁から基準長読取部１５により読み取られ垂直基準棒の長さである。
式（５）のＬ₁は、倍率ｍ_olnを適当な範囲に収めるために使用される定数であるため、実際の基準棒の見かけの長さそのものでなくてもよく、全てのオリジナル画像ｎで同じであればよい。

However, L ₁ is the length of the vertical reference rod read by reference length reading unit 15 from the primary corrected image n ₁ original image n _1.
Since L _{1 in} equation (5) is a constant used to keep the magnification _moln within an appropriate range, it does not have to be the apparent length of the actual reference bar itself, and in all original images n If it is the same.

When the inter-image magnification calculation unit 16 calculates the inter-image magnification m _o1n for each primary correction image n, the secondary correction image generation unit 17 uses the inter-image magnification m _o1n and the camera horizontal axis distortion correction parameter calculation unit 12. By performing secondary perspective transformation using the calculated horizontal axis distortion correction parameter and the perspective projection distortion correction parameter calculated by the perspective projection distortion correction parameter calculation unit 13, the original image n (1 ≦ 1) is obtained. n ≦ 8) is corrected, and the corrected image is output as a secondary corrected image n (1 ≦ n ≦ 8).
That is, like the primary correction image generation unit 14, the secondary correction image generation unit 17 sets a three-dimensional coordinate axis, generates a perspective transformation image on the screen, and outputs it as a secondary correction image.
At this time, the primary correction image generation unit 14 has an image magnification (magnification of the image perspective-transformed on the screen) which is a parameter other than the Z-axis rotation rz and the X-axis rotation rx being 1 ×. As shown, the secondary correction image generation unit 17 _sets the magnification on the screen to m _o1n .

The secondary perspective transformation coordinate calculation unit 18 is calculated by the correction parameter for the horizontal axis distortion calculated by the camera horizontal axis distortion correction parameter calculation unit 12 by the secondary correction image generation unit 17 and the perspective projection distortion correction parameter calculation unit 13. The image coordinates of the multi-view target object when the second perspective transformation is performed using the perspective projection distortion correction parameter and the inter-image magnification calculated by the inter-image magnification calculation unit 16 (multi-view target object). Image coordinates of the movement destination).
When the secondary perspective transformation coordinate calculation unit 18 calculates the image coordinates of the movement destination of the multi-view target object, the movement parameter calculation unit 19 and the image coordinates of the movement destination and the image center (for example, 640 × 480 image) are obtained. In this case, the deviation from the center of the image (320, 240) is calculated as a movement parameter.
In this example, the deviation from the center of the image is calculated so that the multi-view target object moves to the center of the image. The deviation from the position may be calculated.

When the movement parameter calculation unit 19 calculates the movement parameter, the multi-view image generation unit 20 outputs the secondary correction image n (1 ≦ n ≦ 8) output from the secondary correction image generation unit 17 according to the movement parameter. By moving the object, a multi-view image n (1 ≦ n ≦ 8) in which the object to be multi-viewed exists at the center of the image is generated.
That is, the multi-view image generation unit 20 moves the secondary correction image n (1 ≦ n ≦ 8) output from the secondary correction image generation unit 17 by the movement amount indicated by the movement parameter, thereby correcting the secondary correction image. A multi-view image n (1 ≦ n ≦ 8) is generated from n (1 ≦ n ≦ 8).

FIG. 9 is an explanatory diagram illustrating an example in which the image of FIG. 3 is converted so that the size of the “stuffed animal” is the same in each image and the “stuffed animal” is positioned at the center of each image.
However, for ease of viewing, the horizontal reference bar and the vertical reference bar that are shown in each image are omitted.

The multi-view position designation unit 21 performs a process of receiving designation of a new multi-view position for the multi-view image n (1 ≦ n ≦ 8) generated by the multi-view image generation unit 20.
The new multi-view position may be specified while the user is viewing the multi-view image, or may be automatically specified by some method.

When the multi-view image coordinate conversion unit 22 receives a new multi-view position from the multi-view position specifying unit 21, the multi-view image coordinate conversion unit 22 and the secondary perspective conversion coordinate calculation unit Using the inverse transformation of the secondary perspective transformation calculated by 18 and the inverse transformation matrix of the plane projection transformation matrix calculated by the plane projection transformation matrix calculation unit 8, the designation is accepted by the multiview position designation unit 21. The coordinate position (new multi-view position) on the multi-view image n (1 ≦ n ≦ 8) is converted to the coordinate position on the original image n (1 ≦ n ≦ 8).
Here, using the inverse transformation matrix of the planar projection transformation matrix calculated by the planar projection transformation matrix calculation unit 8, the coordinate position on the multi-view image n (1 ≦ n ≦ 8) is converted to the original image n (1 ≦ n ≦ 8). 8) Although shown about what is converted to the upper coordinate position, the coordinate position at the stage of being converted to the coordinate position on the original image n (1 ≦ n ≦ 8) by not using the inverse transformation matrix of the planar projective transformation matrix May be substituted for the coordinate position corresponding to the multi-view position for each original image n calculated by the multi-view position calculation unit 9.

Up to this point, the processing content of the multi-view image generation unit 30 in the image generation apparatus has been described.
By configuring the multi-view image generating means 30 as shown in FIG. 1, the following effects can be obtained.
In the multi-view image generation unit 30, the position of the object to be multi-view calculated by the multi-view position calculation unit 9, the floor surface position of the vertical reference bar calculated by the reference position calculation unit 10, and the reference length reading unit 15 Using the size of the vertical reference bar calculated by the above, the positions of the multiview target objects in the original image n (1 ≦ n ≦ 8) are aligned, and the sizes of the multiview target objects are the same. Since the magnification is calculated for each original image n and the original image n is converted in accordance with the image magnification, the size and position of the subject match without performing high-precision camera calibration. There is an effect that it is possible to generate an easy-to-see realistic image.

In other words, according to the first embodiment, an object placed in contact with a certain plane and a multi-view image after conversion of the image A photographed from a certain direction and an image obtained by photographing the object from another direction. In the multi-view image after the conversion of B, the position of the object is the same and the size is the same.
In addition, by specifying the position of another object placed at a different position on the plane while viewing the multi-view image, a new multi-view with the same position and the same size of the other object An image can be easily generated from images A and B.

For example, the position of a vanishing point (horizontal vanishing point), which is a point obtained by projecting an infinite point of a straight line on a plane, which is a straight line obtained by projecting the optical axis of the camera perpendicularly to the floor surface, and an image A And the position and size of the same object S appearing in the image B and the position of the object T, a magnification is obtained so that the size of the object T is the same in the converted image. The magnification for making the size the same can be obtained with high precision anywhere in the plane.
A point obtained by projecting a distortion caused by the horizontal tilt of the camera of images A and B and a distortion caused by the principle of perspective projection at the time of photographing onto an image at a point at a straight line at infinity perpendicular to the plane. Using the perspective projection transformation after rotating the angle formed by the straight line connecting the vanishing point (vertical vanishing point) and the point inside the image with the vertical axis of the image as the rotation angle of the image placed in the three-dimensional space, Since image correction is performed, it is possible to perform image correction with less distortion of an object caused by image correction. In addition, parameters necessary for correction can be easily determined.

  When the coordinates of the position a where the object in the image A is placed are known, the coordinates of the position b where the object in the image B is placed are calculated using the plane projective transformation matrix for the plane. Thus, the movement of the image such that the position a and the position b are at the same position in the image can be accurately and efficiently performed for all points on the plane.

In the first embodiment, the camera is assumed to be an “ideal pinhole camera model”. In this “ideal pinhole camera model”, the camera's optical center, which is an internal parameter of the camera, coincides with the image center of the original image, and the XYZ axes of the camera coordinates of other camera's internal parameters are orthogonal to each other. In addition, it is assumed that the camera coordinates coincide with the image coordinates photographed by the camera.
When these assumptions do not hold, it is necessary to improve the algorithm in consideration of errors in the assumptions. However, in many cases, there is little problem even if the processing content of the first embodiment is performed using an image captured by a normal camera.
The “ideal pinhole camera model” and “internal parameters of the camera” are disclosed in non-patent documents such as Koichiro Deguchi Robot Vision Basic Corona.

In the first embodiment, lens distortion (distortion aberration) is ignored for the original image of the camera.
This “lens distortion” includes distortion called barrel distortion and pincushion distortion, but this “lens distortion” is negligible as long as a good lens is used under good conditions. However, it may not always be ignored depending on the performance of the lens used, the distance between the camera and the subject, and image processing performed by the camera after shooting.
If this “lens distortion” cannot be ignored, the lens distortion is corrected for the original image and then used as the original image of the first embodiment, and the subsequent algorithm does not need to be improved.
The correction of lens aberration distortion is disclosed in a non-patent document “Koichiro Deguchi Robot Vision Basic Corona”.

Next, the processing content of the processing unit in FIG. 10 in the image generation apparatus will be specifically described.
In the example of FIG. 10, N cameras (N = 8) are installed, but only original images taken by the cameras 1-1, 1-3, 1-5, and 1-7 are used. The other original images taken by the cameras 1-2, 1-4, 1-6, and 1-8 are not used. If a large number of cameras are used, the accuracy of the intermediate image of the multi-view image generated by the image generation device is improved.
Here, the intermediate image of the multi-view image is, for example, a multi-view image obtained by multi-view conversion of the original image captured by the camera 1-1 and a multi-view conversion of the original image captured by the camera 1-3. When there is a multi-view image, it is an image corresponding to an image photographed by a virtual camera located at an intermediate position between two multi-view images.

The background image data storage unit 41 acquires an original image n which is a background image captured by the camera 1-n (1 ≦ n ≦ 8) from the image data temporary storage unit 2 and uses the background image n as the camera 1-n. Save every n.
Here, the “background image” is an image in which an object moving in the screen or an object expected to move in the future is not photographed.
The background image may be selected by the user from a plurality of images, or may be generated by the user. Further, it may be automatically generated by executing a method described later.
In general, a background image is often obtained by photographing a state where there is no moving object.
As a background image automatic selection method or generation method, for example, there is a method of generating a background image from an image of a moving object photographed using a median filter or the like.
The background image stored in the background image data storage unit 41 may be acquired periodically or aperiodically. Further, the background image may be updated when a difference or the like for each background image is obtained and when there is a certain threshold or more that has been found not to affect the extraction of objects such as persons.

The object extraction unit 42 acquires the original image n captured by the camera 1-n from the image data temporary storage unit 2, and acquires the background image n captured by the camera 1-n from the background image data storage unit 41. .
Next, the object extraction unit 42 performs image processing on the original image n and the background image n, and extracts objects in the original image n.
That is, the object extraction unit 42 determines a moving pixel by obtaining a difference between pixels at the same position in the original image n and the background image n, and expresses the position of the moving pixel as a binary image. An object in the original image n is extracted by performing contraction processing, enlargement processing, labeling processing, and the like on the value image.
Shrinkage processing, enlargement processing, labeling processing, and the like are disclosed in, for example, “Practical image processing learned by C language Ohmsha”.

Here, the object in the original image n corresponds to, for example, a “stuffed toy” placed after shooting the background image or a “pedestrian” passing through the background image area.
FIG. 28 is an explanatory diagram for explaining an object. (A) is a background image, (b) is an image including the object, (c) is a difference between the image including the object of (b) and the background image of (a). It is an image of the object extracted by obtaining.
FIG. 11 shows a background image (corresponding to the background image of FIG. 28A) taken by the cameras 1-1, 1-3, 1-5, and 1-7, and FIG. 12 shows the camera 1-1. , 1-3, 1-5, and 1-7, an image including an object (corresponding to an image including the object in FIG. 28B) is shown. At this time, the cameras 1-1, 1-3, 1-5, and 1-7 do not move and take the images of FIGS. 11 and 12.

However, in FIGS. 11 and 12, the vanishing points in the vertical direction and the horizontal direction are already known, and the vertical reference bar and the horizontal reference bar for measuring the vanishing points are not photographed.
Further, only the square marking on the floor surface is written in the background image of FIG.
FIG. 14 is an image of the objects of the cameras 1-1, 1-3, 1-5, and 1-7 extracted by the object extracting unit 42.

Here, although the object extraction part 42 shows what extracts the object by calculating | requiring the difference of the background image in which the person etc. were not imaged, and the image in which the person was image | photographed, a person is imaged. The object may be extracted by detecting the motion of each pixel or block of the image being compared for each frame.
Further, by comparing the images from two or more cameras for each pixel or block (for example, a process called stereo matching), the distance to the object may be obtained to extract the object.
As another method, for example, an object may be extracted by performing three-dimensional measurement using a sound wave or a laser beam.

The object position coordinate calculation unit 43 calculates the position coordinates of the object in the original image n when the object extraction unit 42 extracts the object.
That is, the object position coordinate calculation unit 43 determines the center of gravity of the object image extracted by the object extraction unit 42, the point inside the object image, and the point having the maximum distance to the nearest contour. Then, a vertical line drawn in the vanishing point direction in the lower direction of the image or the vanishing point direction in the vertical direction of the image is obtained for each image.
Next, the object position coordinate calculation unit 43 converts the vertical line of one image among the vertical lines for each image onto another image using the plane projection conversion matrix calculated by the plane projection conversion matrix calculation unit 8. Then, the intersection of the converted vertical line and the vertical line of another image is calculated as the object position.

FIG. 15 is an explanatory diagram showing the position of the object calculated by the object position coordinate calculation unit 43.
In FIG. 15, a solid line is drawn from the center of gravity of the object 1 extracted from an original image (for example, the original image 1) taken by a certain camera toward the vanishing point in the vertical direction of the image. It is a vertical line.
A broken line indicates a vanishing point in the vertical direction from the center of gravity of an object in the original image (for example, the original image 2) taken by another camera, for the same object as the object 1 of the original image 1. The drawn straight line is a straight line written on the original image 1 by using the plane projection transformation matrix calculated by the plane projection transformation matrix calculation unit 8.
At this time, when an area where a certain object exists is surrounded by the cameras 1-1, 1-2, 1-3,..., 1-7, the area in each original image captured by a different camera is captured. The position of the object is different.

In this example, the position of the object is calculated in all the original images n. However, the position of the object is calculated in an image photographed by a certain camera, and is calculated by the planar projective transformation matrix calculation unit 8. Using the planar projective transformation matrix, the position of the object may be converted to a position on the original image taken by another camera, and the position may be used as the position of the object in the original image.
In addition, the lowest point of the extracted object image may be simply used as the position coordinates of the object, or the image of the object is divided into several parts in the vertical direction, and the coordinate value of the lowest point of each divided partial image. The average of these may be used as the position coordinates of the object. At this time, the x coordinate value and the y coordinate value of the position coordinate may be determined by different methods.

Further, the position coordinates of the object calculated by the object position coordinate calculation unit 43 may be corrected by predicting in advance the image of the object, for example, a person or the shooting distance to the person.
In these methods, when a plurality of objects are extracted in an image, before the position coordinates of the object are determined by a highly accurate method (for example, the method shown in FIG. 15), It may be used to determine which object in the original image of another camera corresponds to the object.

The floor plan input unit 44 captures a plan view of the floor surface photographed in common by the cameras 1-1, 1-2,.
In the “plan view of the floor surface”, for example, if the quadrangular shape marked on the floor surface is a square, the square in the “plan view of the floor surface” is a square, and the aspect ratio is a: b. Is a rectangle of a: b, and it is sufficient that the four vertices of the quadrangle can be determined by similar shapes, and the size may be appropriate.
In addition, the shape marked on the floor surface may be a triangle other than a quadrangle, other polygons, or other shapes, and the positional relationship between four or more points on the plane is “floor surface”. In the “plan view” of FIG.
In the first embodiment, an example is shown in which four vertices of a square are marked on the floor surface, but there are planes only inside the markings, such as platforms where the four corners and upper part of the floor of the room are flat surfaces. You may do.
In FIG. 3, FIG. 12, and FIG. 16, which is a “plan view of the floor surface”, a quadrangle marked on the floor surface is indicated by a solid line.
In the following description, it is assumed that the square vertex (1) is arranged near the center of the image (see FIG. 16).

The in-image floor coordinate designation unit 45 corresponds to the vertices (1) to (4) of the floor surface captured by the floor plan input unit 44 in the background image n stored in the background image data storage unit 2. Specifies the coordinates of the point (floor surface position coordinates).
The method for specifying the position coordinates of the floor surface may be specified by the user while viewing the image, or may be automatically specified by some method.
In the first embodiment, the floor surface coordinates in the background image are designated, but the floor surface coordinates in the original image other than the background image may be designated.

FIG. 13 shows the position coordinates of the floor surface designated in the original image taken by the cameras 1-1, 1-3, 1-5, and 1-7. The broken lines are shown as auxiliary lines for easy viewing.
In this example, the position on the floor determined by the floor plan input unit 44 indicates which position in the original image is designated by the in-image floor coordinate designating unit 45. On the contrary, the position on the floor surface in the original image designated by the in-image floor surface coordinate designating unit 45 may be inputted by the plan view input unit 44 as a floor plan of the floor surface.
Further, instead of specifying directly on the original image, the original image may be specified on a multi-view image obtained by multi-view conversion, and converted to coordinates on the original image by inverse multi-view conversion described later.
Also, using the plane projection transformation matrix calculated by the plane projection transformation matrix calculator 8, the position on the original image taken by another camera is converted to the coordinates on the original image taken by the required camera. You may make it do.

In the first embodiment, the point corresponding to the point designated by the floor plan input unit 44 is designated by the in-image floor coordinate designating unit 45. After designating a figure that is easy to create a plan view, such as a rectangle or a rectangle, the figure is marked on the floor surface, and the in-image floor surface coordinate designating unit 45 determines the coordinates of the marked position from the image of the appearance. It is specified. This has the effect that it is easy to create a plan view and that it is easy to mark the floor surface.
In addition, the floor plan view input unit 44 may generate a plan view of the floor surface by measuring the position on the floor surface read by the floor coordinate reading unit 5 of FIG. In this case, since the floor surface coordinate reading unit 5 has already read the coordinates of the floor surface, the in-image floor surface coordinate designating unit 45 is unnecessary.

  The input image coordinate acquisition unit 46 calculates the position coordinates of the object in the original image n calculated by the object position coordinate calculation unit 43 or the floor surface in the background image n specified by the in-image floor surface coordinate specification unit 45. Select one of the position coordinates.

The multi-view conversion equation generation unit 47 uses the camera horizontal axis distortion correction parameter calculated by the camera horizontal axis distortion correction parameter calculation unit 12 and the perspective projection distortion correction as image conversion parameters at the time of multi-view conversion by the multi-view image generation unit 30. The perspective projection distortion correction parameter calculated by the parameter calculation unit 13 and the inter-image magnification calculated by the inter-image magnification calculation unit 16 are collected.
Next, the multi-view conversion equation generation unit 47 uses the camera horizontal axis distortion correction parameter, the perspective projection distortion correction parameter, and the inter-image magnification, and the cameras 1-1, 1-3, 1-5, and 1-7. A multi-view conversion formula that can calculate which coordinates on a multi-view image are converted to coordinates of a certain position on the original image when the photographed original image is converted into a multi-view image (for example, 3 × 3 matrix).
Note that the multi-view conversion formula is an image conversion formula between an original image and a multi-view image using the camera horizontal axis distortion correction parameter, the perspective projection distortion correction parameter, and the magnification between images as variables.

When the multi-view conversion expression generation unit 47 generates the multi-view conversion expression, the inverse multi-view conversion expression generation unit 48 uses the coordinates of a certain position on the multi-view image in the original image before the multi-view conversion. Generate an inverse multi-view transformation formula that can calculate whether or not there was.
The inverse multiview conversion equation can be obtained by mathematically calculating the inverse matrix of the multiview conversion equation generated by the multiview conversion equation generation unit 47.
As another method, for example, an inverse multi-view transformation expression may be obtained by executing several transformations constituting the multi-view transformation in reverse.

In the converted image coordinate calculation unit 49, the input image coordinate acquisition unit 46 selects the position coordinate of the object in the original image n or the position coordinate of the floor in the background image n, and the multi-view conversion formula generation unit 47 selects the multi-position conversion formula generation unit 47. When the view conversion formula is generated, the position coordinate of the object in the original image n selected by the input image coordinate acquisition unit 46 (or the position coordinate of the floor in the background image n) is selected using the multi-view conversion formula. Convert to position coordinates on multi-view image.
FIG. 18 shows a multi-view image as shown in FIG. 9 by converting the original image (see the image in FIG. 3 or FIG. 12) taken by the cameras 1-1, 1-3, 1-5, and 1-7. Is generated, the position on the floor surface shown in FIG. 13 indicates the position on the multi-view image converted by the converted image coordinate calculation unit 49.
FIG. 19 shows the position on the multi-view image where the converted image coordinate calculation unit 49 converts the position of the object shown in FIG. 14 when the multi-view image shown in FIG. 9 is generated.
In FIG. 19, the position of the object is indicated by a cross. The broken line in FIG. 18 is an auxiliary line written so that the converted vertex position can be easily imaged, and the broken line in FIG. 19 is an auxiliary line written so that the shape of the converted object can be easily imaged.

When the converted image coordinate calculating unit 49 converts the position coordinates of the object in the original image n (or the position coordinates of the floor surface in the background image n) into the position coordinates on the multi-view image, the original image coordinate calculating unit 49 Using the inverse multi-view transformation expression generated by the inverse multi-view transformation expression generator 48, the position coordinates on the multi-view image (multi-points converted from the original image n of each camera 1-n (1 ≦ n ≦ 8)). The position coordinates on the view image) are converted into the position coordinates on the original image of a certain camera (for example, camera 1-1).
Here, the original image of a certain camera may be, for example, the original image currently displayed by the image generation device, or the original image of the camera that captured the multi-view image, It may be an original image of another camera.
FIG. 21 shows the position of an object (stuffed animal, vase) in a multi-view converted image (see the image of FIG. 20) of the original image taken by the cameras 1-1, 1-3, 1-5, 1-7. An example is shown in which an original image captured by the camera 1-1 is converted into a multi-view converted image.

The plane projective transformation calculation unit 51 uses the floor vertices (coordinate values of four or more points on the plan view) captured by the floor plan input unit 44 and the background image specified by the in-image floor coordinate specification unit 45. The plane projection transformation matrix (when the floor plan view coordinate calculation unit 53 calculates the floor plan view coordinates from the position coordinates (four or more coordinate values corresponding to the vertices of the floor surface) of the floor in n) For example, a 3 × 3 matrix) is calculated for each camera 1-n.
The calculation method of the planar projective transformation matrix is described in, for example, the non-patent document “Koichiro Deguchi“ Basics of Robot Vision ”p47 (described as a homography matrix in this document)”.

The plane projective transformation calculation unit 52 uses the floor vertices (coordinate values of four or more points on the plan view) captured by the floor plan input unit 44 and the background image specified by the in-image floor coordinate specification unit 45. The plane projection transformation matrix (when the original image coordinate calculation unit 55 calculates the coordinates on the original image from the position coordinates (four or more coordinate values corresponding to the vertices of the floor surface) of the floor surface in n. For example, a 3 × 3 matrix) is calculated for each camera 1-n.
Note that, in an image shot by a certain camera, the plane projection transformation matrix calculated by the plane projection transformation calculation unit 51 and the plane projection transformation matrix calculated by the plane projection transformation calculation unit 52 are inversely related to each other. In this case, after calculating one of the plane projection transformation matrices, calculate the inverse of the plane projection transformation matrix mathematically, and calculate the remaining one of the plane projection transformation matrices. You may make it ask.

The floor plan coordinate calculation unit 53 is configured so that the original image coordinate calculation unit 50 converts the position coordinate on the multi-view image (the position on the multi-view image converted from the original image n of each camera 1-n (1 ≦ n ≦ 8)). (Coordinate) is converted into position coordinates on the original image of one camera (for example, camera 1-1), the plane projection transformation matrix calculated by the plane projection transformation calculation unit 51 is used. The position coordinates are converted into the position coordinates on the floor plan.
FIG. 16 shows a plan view of the floor surface, and FIG. 17 shows a position on the plan view of an object (stuffed animal, vase) photographed by the camera 1-1.
FIG. 22 shows objects (stuffed animals, vases) and markings (cameras 1-1, 1-3, 1-5, 1-7) and markings (photographs are taken by the camera 1-1 because the drawing is complicated). (Only marked markings) are shown on the plan view.

In FIG. 22, K1 is a position on the plan view of the vase photographed by the camera 1-1, K3 is a position on the plan view of the vase photographed by the camera 1-3, and K5 is photographed by the camera 1-5. A position on the plan view of the vase, K7, is a position on the plan view of the vase taken by the camera 1-7.
N is a position on the plan view of the stuffed toy photographed by the cameras 1-1, 1-3, 1-5, and 1-7. Since the stuffed animal is a multi-view target article, it is at the same position in the multi-view image that is a converted image of the original image photographed by the cameras 1-1, 1-3, 1-5, and 1-7. Therefore, it is apparently one point on the floor plan.

When the floor plan view coordinate calculation unit 53 converts the position coordinates on the original image into the position coordinates on the floor plan, the floor plan intermediate coordinate calculation unit 54 is a plan view of one or more objects (stuffed animals, vases). From the upper position coordinates, points K1, K3, K5, and K7 that indicate the positions of the apparently different objects (vases) are identified on the plan view, and intermediate points of the points K1, K3, K5, and K7 are determined. To do.
The intermediate point here means, for example, a point on the curve in the middle of moving from the point K1 to the point K3, and does not mean a point located in the middle (center).
Specifically, an intermediate point between the points K1, K3, K5, and K7 is determined as follows.

ただし、ｘは平面図の横方向の座標値、ｙは平面図の縦方向の座標値である。 Here, an example will be described in which three intermediate points K1-1, K1-2, and K1-3 are determined between the points K1 and K3.
Floor plan view intermediate coordinate calculating unit 54, as shown in FIG. 23, the distance between the point N and the point K1 d1, the distance between the point N and the point K3 d3, and the coordinates of the point N and (n _x, n _y) A circle (see the broken line in FIG. 23) passing through the point K1 centered on the point N is prepared.
At this time, the point (x, y) on the circumference is expressed by the following equation (6).

Here, x is a coordinate value in the horizontal direction of the plan view, and y is a coordinate value in the vertical direction of the plan view.

Next, the floor plan intermediate coordinate calculation unit 54 divides the formed angle α ₀ into four with the angle formed by the line segment N-K1 and the line segment N-K3 on the plan view being α ₀ .
Then, the floor plan intermediate coordinate calculation unit 54 determines the points where the line segments corresponding to the angles α ₁ , α ₂ , α ₃ , α ₄ formed after the division intersect with the circumference K1-1p, K1-2p, K1−. 3p. Hereinafter, K1-1p, K1-2p, and K1-3p are expressed as K1-mp. However, m = 1, 2, 3.

ただし、Ｍは点Ｋ１と点Ｋ３を結ぶ中間点の総数、ｍは中間点のうち、点Ｋ１に近い方から順番に数えた番号であり（１≦ｍ≦Ｍ−１）、この例では、Ｍ＝４、ｍ＝１，２，３である。 Next, the floor plan intermediate coordinate calculation unit 54 determines the intermediate point where the distance Lm from the point N is expressed by the following equation (7) on the straight line connecting the point N and the point K1-mp on the circumference. Determine the point K1-m.

However, M is the total number of intermediate points connecting the points K1 and K3, m is a number sequentially counted from the intermediate points closer to the point K1 (1 ≦ m ≦ M−1), and in this example, M = 4 and m = 1, 2, 3.

また、ここでは、点Ｎを中心とする点Ｋ１（または点Ｋ３）を通る円周を決定してから、点Ｋ１と点Ｋ３の中間点Ｋ１-１，Ｋ１-２，Ｋ１-３を決定するようにしているが、点Ｎを中心として、点Ｋ１と点Ｋ３を通る楕円を決定した上で、線分Ｎ−Ｋ１と線分Ｎ−Ｋ３のなす角α₀をＭ分割（この例では、Ｍ＝４）して、中間点Ｋ１-１，Ｋ１-２，Ｋ１-３を求めるようにしてもよい。 Here, the intermediate points K1-1, K1-2, and K1-3 are determined for determining the circumference passing through the point K1 centered on the point N. However, the point K3 centered on the point N is shown. For example, on a straight line connecting the point N and the point K1-mp on the circumference, a point where the distance Lm from the point N is expressed by the following equation (8) is defined as the intermediate point K1- It may be determined to be 1, K1-2, K1-3.

Further, here, after determining the circumference passing through the point K1 (or the point K3) centered on the point N, intermediate points K1-1, K1-2, and K1-3 between the points K1 and K3 are determined. However, after determining the ellipse passing through the points K1 and K3 with the point N as the center, the angle α ₀ formed by the line segment N-K1 and the line segment N-K3 is divided into M (in this example, M = 4), and intermediate points K1-1, K1-2, K1-3 may be obtained.

FIG. 24 shows the same method as the intermediate points K1-1, K1-2, and K1-3 between the points K1 and K3, the intermediate point between the points K3 and K5, the intermediate point between the points K5 and K7, and the point K7 and the point K7. The example which calculated | required the intermediate point of K1 is shown.
In the figure, angles α _{1 to} α ₄ , β _{1 to} β ₄ , γ _{1 to} γ ₄ , and δ _{1 to} δ ₄ are angles α ₀ and line segments formed by line segment N-K 1 and line segment N-K ₃ , respectively. An angle β ₀ formed by N-K3 and line segment N-K5, an angle γ ₀ formed by line segment N-K5 and line segment N-K7, and an angle δ ₀ formed by line segment N-K7 and line segment N-K1 are set to 4 Since the midpoint is determined by equally dividing, the following equations (9) to (12) are established.

Here, χ = 0.25 because it is divided into four equal parts, but χ may be a value other than 0.25 if the number of divisions is different.
The values of χ _α , χ _β , χ _γ , χ _δ vary according to the ratio of dividing the angles α ₀ , β ₀ , γ ₀ , δ _{0 made} with respect to the point N.

In the above example, an intermediate point is obtained using a circle or ellipse passing through the point K1 (or K3, K5, K7) centered on the point N, but the point N (the point where the stuffed animal is placed) An intermediate point between the point K1 and the point K3 (or the point K3 and the point K5, the point K5 and the point K7, and the point K7 and the point K1) may be determined so that the position affects, for example, the point K1 (or Spline curves passing through K3, K5, and K7) may be used.
Further, for example, when determining an intermediate point between the point K1 and the point K3, not only the position of the point N is affected, but either the point K5 or the point K7, or both the point K5 and the point K7 are You may make it affect.
In this example, since only four cameras are used, there are only four actual points K1, K3, K5, and K7. However, when there are many actual points, depending on the positions of these points. The degree of influence may be different (for example, a larger value is given to a point having a closer degree of adjacent positions).

Here, when determining an intermediate point between the points K1 and K3, the angle α ₀ formed by the line segment N-K1 and the line segment N-K3 is equally divided by a numerical value M (M = 4) to obtain three intermediate points. Although what has determined K1-1, K1-2, and K1-3 has been shown, it need not be equally divided. In that case, the division ratio may be changed in consideration of the positions of those points. Further, the closer to the actual point (K1, K3, K5, K7), the finer the division may be.
In this example, the angle is divided, but the intermediate point may be determined by dividing the circle or ellipse by the circumference length or other curve length (spline curve or the like).

FIG. 26 shows points K1 and K3 in which the positions of the vases in the original images taken by the cameras 1-1 and 1-3 are converted into coordinates on the floor plan, as well as the cameras 1-1 and 1. The square vertices in the original image photographed by -3 indicate points converted into coordinates on the floor plan. Point N indicates the position of the stuffed animal.
The vertices of the rectangle photographed by the camera 1-1 are indicated by the vertex (1), the vertex (2), the vertex (3), and the vertex (4), and the vertex of the square photographed by the camera 1-3 is the vertex (1). 3, vertex (2) 3, vertex (3) 3, and vertex (4) 3.

  In FIG. 26, the vertex (1), vertex (2), vertex (3), and vertex (4) of the rectangle photographed by the camera 1-1, and the vertex (1) of the rectangle photographed by the camera 1-3. ) 3, vertex (2) 3, vertex (3) 3, and vertex (4) 3 are intermediate points (the intermediate point here is, for example, a curve in the middle of moving from vertex (1) to vertex (1) 3) Vertex (1) -1, Vertex (2) -1, Vertex (3) -1, Vertex (4), which means the upper point, not just the middle (center) point) -1 determination method is shown.

This determination method is basically the same as the method shown in FIGS.
For example, a line segment (vertex (1) -point N) and a line segment (vertex (1) with respect to an angle A formed by the line segment (vertex (1) -point N) and the line segment (vertex (1) -point N) ) -1−the ratio λ of the angle a formed by the point N) is a line segment with respect to the angle B formed by the line segment (vertex (2) −point N) and the line segment (vertex (2) 3-point N) ( It is equal to the ratio of the angle b formed by the vertex (2) -point N) and the line segment (vertex (2) -1-point N).
This ratio λ is the line segment (vertex (3) −point N) and line segment (angle (vertex (3) −point N)) with respect to the angle C formed by the line segment (vertex (3) −point N) and line segment (vertex (3) −point N). The ratio of the angle c formed by the vertex (3) -1-point N) and the line with respect to the angle D formed by the line segment (vertex (4) -point N) and the line segment (vertex (4) 3-point N) It is also equal to the ratio of the angle d formed by the minute (vertex (4) -point N) and the line segment (vertex (4) -1-point N).
These can be expressed as follows.

When the floor plan intermediate coordinate calculation unit 54 determines intermediate points (for example, intermediate points K1-1, K1-2, K1-3, etc.) as described above, the plane image transformation conversion is performed. Using the planar projective transformation matrix calculated by the calculation unit 52, the intermediate point is converted into position coordinates on the original image.
At this time, it may be converted into coordinates on one original image, or may be converted into coordinates on several original images with different cameras.

When the original image coordinate calculation unit 55 converts the intermediate point to a position coordinate on the original image, the converted image coordinate calculation unit 56 uses the multi-view conversion formula generated by the multi-view conversion formula generation unit 47 to convert the original image. The upper position coordinates are converted into position coordinates on the multi-view image.
FIG. 25A shows an example in which the intermediate point of the object position in FIG. 24 is converted into position coordinates on the multi-view image, and FIG. 25B shows an example of the intermediate viewpoint position determined by the conventional method. ing. FIG. 27 shows an example in which an intermediate point between vertices on the floor surface is converted into position coordinates on the multi-view image.

When the converted image coordinate calculation unit 56 converts the position coordinates on the original image into the position coordinates on the multi-view image, the floor intermediate image plane projection conversion calculation unit 57 converts the position coordinates on the multi-view image and the converted image. A plane projection transformation matrix (for example, a 3 × 3 matrix) used when the floor intermediate image generation unit 59 generates an intermediate image of the floor surface from the position coordinates on the multi-view image converted by the coordinate calculation unit 49. ) Is calculated.
The calculation method of the planar projective transformation matrix is described in, for example, the non-patent document “Koichiro Deguchi“ Basics of Robot Vision ”p47 (described as a homography matrix in this document)”.

The floor area extraction unit 58 extracts a floor area from the background image n of the camera 1-n (1 ≦ n ≦ 8) stored in the image data temporary storage unit 2.
As the extraction of the floor area, for example, the user may specify the floor area from the background image.
Alternatively, using a plane projection transformation matrix between cameras calculated by the plane projection transformation matrix calculation unit 8, a background image shot by one camera is converted into a background image shot by another camera, and both pixels The floor area may be determined by performing a difference for each pixel and performing, for example, a contraction process, an enlargement process, or the like on a pixel whose difference is equal to or less than a certain threshold value.
Further, without using a background image, an image corresponding to the background image may be generated and used from an image in which an object is photographed using a median filter or the like.
For example, FIG. 28D shows a floor area specified by the user, and FIG. 28E shows a floor area extraction result.
FIG. 28D shows a state in which the user designates, as the floor surface, an area other than the area where the floor surface is hidden by a three-dimensional object such as a desk among the floor surface of the background image.

When the floor area extraction unit 58 extracts a floor area from the background image, the floor intermediate image generation unit 59 uses the plane projection transformation matrix calculated by the plane projection conversion calculator 57 for the floor intermediate image, Convert the floor area to an intermediate image of the floor.
In this example, if it is intermediate between the camera 1-1 and the camera 1-3, an intermediate image of the floor surface is generated only from the background image photographed by the camera 1-1 or the camera 1-3. An intermediate image is generated from both the background image captured by the camera 1 and the background image captured by the camera 1-3, and the two intermediate images are weighted according to the degree of the intermediate position, for example, and averaged. Also good.

When the object extraction unit 42 extracts an object from the original image, the intermediate image object generation unit 60 generates an intermediate image object from the object.
A method for generating an object for an intermediate image is described in, for example, the non-patent document “Satoru Yaguchi et al. SIG 6 (CVIM 2) June 2001 ”, and in particular, an object for an intermediate image is generated using an arbitrary viewpoint image described in“ Section 3.2 (3) (p13) ”.
The intermediate points K1-1, K1-2, and K1-3 calculated by the floor plan intermediate coordinate calculation unit 54 divide the angle α ₀ formed by the line segment N-K1 and the line segment N-K3 into four equal parts. In the case of the determined point, the value of ω in “Section 3.2 (3) (p13)” is, for example, 0.25, 0.5, and 0.75.

  Here, an example in which an object for an intermediate image is generated using an arbitrary viewpoint image is shown. For example, an object extracted from an original image of the camera 1-1 or an original image of the camera 1-3 is extracted. The object may be used as an object for an intermediate image as it is, or an object extracted from the original image of the camera 1-1 and an object extracted from the original image of the camera 1-3 are weighted and averaged. The converted object may be used as an intermediate image object.

When the intermediate image object generation unit 60 generates an intermediate image object, the object size correction unit 61 exists at the position coordinates on the multi-view image converted by the original image coordinate calculation unit 55. As an example, the size of the intermediate image object is corrected using the multi-view conversion formula generated by the multi-view conversion formula generation unit 47.
When correcting the size of the object for the intermediate image, for example, the size of the object in the multi-view image of the camera 1-1 and the size of the object in the multi-view image of the camera 1-3 are measured. The size of the object at the intermediate position may be corrected so that changes linearly.

The intermediate image generation unit 62 is converted by the converted image coordinate calculation unit 56 when the floor intermediate image generation unit 59 generates an intermediate image of the floor and the object size correction unit 61 corrects the size of the object for the intermediate image. The intermediate image is generated by overwriting the intermediate image on the floor surface with the intermediate image object whose size has been corrected by the object size correction unit 61 at the object position coordinates on the intermediate image. This is shown in FIG. In FIG.
At this time, the ratio λ for determining the intermediate point of the corresponding object position in FIG. 23 or FIG. 24 and the ratio χ for determining the intermediate point of the corresponding vertex on the floor surface in FIG. By combining the object position of the generated intermediate image and the intermediate image of the floor surface, it is possible to generate an intermediate image with high accuracy and no sense of incongruity. At this time, the following equation is established.

また、図２６において、下記の式（１５）が成立する場合、下記の式（１６）が成立する。

In FIG. 23 or FIG. 24, the following formula (14) is established.

In FIG. 26, when the following equation (15) is established, the following equation (16) is established.

Note that α _n (n = 1, 2, 3, 4, etc.) is an angle by an intermediate point that divides the angle α ₀ with respect to the camera 1-1 to the camera 1-3, but is not necessarily α ₁ = α ₂ = It is not α ₃ = α ₄ .
In other words, for example, when an object is overwritten at an intermediate position derived from the point K1-1, “the object is overwritten at the intermediate point between the camera 1-1 and the camera 1-3, which is the determination method of the point K1-1. When the angle division ratio for determining the curve of the object position of the camera 1-1 → camera 1-3 is determined, the ratio at which the intermediate point of the floor surface used for generating the intermediate image of the floor surface on which the object is written is determined. Is equal to ".

  As apparent from the above, according to the first embodiment, the plane projection conversion means for converting the position on the floor surface of one or more objects placed on the floor surface into the position on the floor plan. From among the positions on the plan view of one or more objects converted by the plane projective conversion means, a plurality of positions related to the same article that are apparently different on the plan view are specified, and intermediate points of the plurality of positions are determined. Intermediate point determination means for converting the intermediate point determined by the intermediate point determination means to a position on the multi-view image, and generating a floor surface image according to the position on the multi-view image; And the intermediate image generation means is generated by the multi-view image generation means from the image of one or more objects placed on the floor in the plurality of camera images and the floor image generated by the floor image generation means. Between multiple multi-view images Since it is configured to generate between images, when switching the plurality of the multi-view image, an effect that can be displayed in an intermediate image not smooth and discomfort.

  In the first embodiment, for example, the intermediate point between the camera 1-1 and the camera 1-3 is determined by three intermediate points obtained by dividing the central angle in the plan view into four equal parts. When the multi-view image, the three intermediate images, and the intermediate image of the camera 1-3 are sequentially displayed, the camera 1-3 is moved from the shooting position of the camera 1-1 around the multi-view target object (stuffed animal). The image effect can be obtained as if the camera were moved to the shooting position at a constant speed three-dimensionally.

Further, in the plan view of the floor surface, since the intermediate position is determined using a circle or an ellipse, the three-dimensional distance from the object to be multiviewed (stuffed toy) to the camera is kept equal while the camera 1- An image effect as if the camera was moved from the position 1 to the position of the camera 1-3 can be obtained.
Further, for example, when an intermediate image that returns from the camera 1-1 to the camera 1-1 via the camera 1-3 and the camera 1-5 is generated, the position and size of the object and the moving direction of the object Without causing a sense of incongruity, it is possible to switch between the original images of the cameras 1-1, 1-3, 1-5, and 1-7, which are actual images, and the intermediate images before and after that. In addition, the original images of the cameras 1-1, 1-3, 1-5, and 1-7, which are actual images, and the front and back of the images can be obtained without causing a sense of incongruity between the position and size of the floor surface and the moving direction of the front and rear. An intermediate image can be switched.

In the first embodiment, for example, between the adjacent cameras 1-1 and 1-3, 1-3 and 1-5, 1-5 and 1-7, and 1-7 and 1-1 are planar. By dividing into four with respect to the central angle in the figure, camera 1-1 → camera 1-3, camera 1-3 → camera 1-5, camera 1-5 → camera 1-7, camera 1-7 → camera At the time of switching to 1-1, an image effect is obtained as if the camera is moving three-dimensionally at a constant speed. If the camera is changed from 1-1 → 1-3 → 1-5 → When switching continuously from 1-7 to 1-1, the angles formed by the cameras 1-1, 1-3, 1-5, and 1-7 with respect to the multi-view target object (stuffed animal) are not equal. As a series of continuous switching operations, there may be an image effect in which the speed changes for each actual camera position. That.
If such an image effect is not preferable, the intermediate position may be determined so that the center angle formed by the first and last camera positions for performing a series of continuous switching is equally divided by a certain numerical value.

In the first embodiment, as shown in FIG. 24, the intermediate position is once determined on the floor plan, and then the intermediate point on the multi-view image is determined as shown in FIG. However, for example, when the converted image coordinate calculation unit 49 converts the position coordinates of the object in the original image n into the position coordinates on the multi-view image, the circle, ellipse, or other curve is directly displayed on the multi-view converted image. The intermediate point may be determined using.
At this time, in the intermediate image, the apparent movement speed may be slower as the position is farther three-dimensionally so that the three-dimensional movement speed and size do not feel strange. Further, the display size may be displayed so that the display size becomes smaller as the position is farther three-dimensionally.

In the first embodiment, an intermediate image of a multi-view image is generated. However, an intermediate image of a converted image other than the multi-view conversion may be generated.
For example, by expressing the position of the camera, the direction of the optical axis, the position of the floor or the object (stuffed animal, vase) on the floor surface in 3D space coordinates, etc. A converted image having the same position and size in the image may be generated, and an intermediate image of the converted image may be generated.

It is a block diagram which shows the multi view image generation means of the image generation apparatus by Embodiment 1 of this invention. It is explanatory drawing which shows a mode that the image is image | photographed using eight cameras 1-n (1 <= n <= 8). It is explanatory drawing which shows the image image | photographed with the camera 1-1, 1-3, 1-5, 1-7. It is explanatory drawing which shows the example installed so that the horizontal vanishing point is in the center of the image, the optical axis of the camera is parallel to the floor surface, and the horizontal axis of the camera image is substantially parallel to the floor surface. is there. It is explanatory drawing which shows the example which is making the horizontal axis of a camera image substantially parallel to a floor surface, and makes the optical axis of a camera face down slightly with respect to a horizontal floor surface. It is explanatory drawing explaining calculation of the correction parameter of the horizontal axis distortion of a camera, and perspective projection distortion. It is explanatory drawing explaining correction | amendment of the horizontal axis distortion and perspective projection distortion of the original image n. It is explanatory drawing which looked at the three-dimensional coordinate of FIG. 7 from the positive direction of the X-axis. It is explanatory drawing which shows the example of a conversion of the image of FIG. It is a block diagram which shows a part (part except a multi view image generation means) of the image generation apparatus by Embodiment 1 of this invention. It is explanatory drawing which shows the background image image | photographed with the camera 1-1, 1-3, 1-5, 1-7. It is explanatory drawing which shows the image containing the object image | photographed with the cameras 1-1, 1-3, 1-5, and 1-7. It is explanatory drawing which shows the position coordinate of the floor surface designated within the original image image | photographed with the camera 1-1, 1-3, 1-5, 1-7. It is explanatory drawing which shows the image of the object of the cameras 1-1, 1-3, 1-5, and 1-7 extracted by the object extraction part 42. FIG. It is explanatory drawing which shows the position of the object calculated by the object position coordinate calculation part 43. FIG. It is explanatory drawing which shows the top view of the marked floor surface. It is explanatory drawing which shows the position on the top view of the object (stuffed animal, vase) image | photographed with the camera 1-1. It is explanatory drawing which shows the position on the multi view image in which the position on the floor surface shown by FIG. 13 is converted by the conversion image coordinate calculation part 49. It is explanatory drawing which shows the position on the multi view image in which the position of the object shown by FIG. 13 is converted by the conversion image coordinate calculation part 49. It is explanatory drawing which shows the position of the object (stuffed animal, vase) in the original image image | photographed with the camera 1-1, 1-3, 1-5, 1-7. An example in which the position of an object (stuffed animal, vase) in an original image photographed by the cameras 1-1, 1-3, 1-5, and 1-7 is converted into position coordinates on the original image of the camera 1-1. It is explanatory drawing which shows. Objects (stuffed animals, vases) photographed by the cameras 1-1, 1-3, 1-5, and 1-7, and markings (only the marking photographed by the camera 1-1 because the drawing is complicated) It is explanatory drawing which shows the position on the top view. It is explanatory drawing which shows the determination method of an intermediate point. It is explanatory drawing which shows the determination method of an intermediate point. It is explanatory drawing which shows the example which has converted the intermediate point in FIG. 24 into the position coordinate on an original image. The positions of the vases in the original images photographed by the cameras 1-1 and 1-3 are converted into the coordinates K1 and K3 on the floor plan and the cameras 1-1 and 1-3. It is explanatory drawing which shows the point by which the square vertex in the image | photographed original image was converted into the coordinate on the floor plan of a floor surface. It is explanatory drawing which shows the example which converted the intermediate point of FIG. 26 into the position on a multi view image. It is explanatory drawing explaining an object, (a) is a background image, (b) is an image containing an object, (c) is the difference of the image containing the object of (b), and the background image of (a). It is an image of the object to be extracted.

Explanation of symbols

  1-1 to 1-N camera, 2 image data temporary storage unit, 3 multi-view position designation unit, 4 reference position designation unit, 5 floor surface coordinate reading unit, 6 horizontal vanishing point calculation unit, 7 vertical vanishing point calculation unit, 8 plane projection transformation matrix calculation unit, 9 multi-view position calculation unit, 10 reference position calculation unit, 11 intra-image magnification calculation unit, 12 camera horizontal axis distortion correction parameter calculation unit, 13 perspective projection distortion correction parameter calculation unit, 14 primary Correction image generation unit, 15 Reference length reading unit, 16 Inter-image magnification calculation unit, 17 Secondary correction image generation unit, 18 Secondary perspective transformation coordinate calculation unit, 19 Movement parameter calculation unit, 20 Multi view image generation unit, 21 Multi View position designation unit, 22 multi-view image coordinate conversion unit, 30 multi-view image generation means, 41 background image data storage unit, 42 object extraction unit (floor surface) Position acquisition unit), 43 object position coordinate calculation unit (floor surface position acquisition unit), 44 floor plan input unit (floor surface position acquisition unit), 45 in-image floor coordinate designation unit (floor surface position acquisition unit), 46 Input image coordinate acquisition unit (floor surface position acquisition unit), 47 Multi-view conversion formula generation unit (plane projection conversion unit), 48 Inverse multi-view conversion formula generation unit (plane projection conversion unit), 49 Conversion image coordinate calculation unit (plane Projection conversion unit), 50 Original image coordinate calculation unit (plane projection conversion unit), 51 Plane projection conversion calculation unit (plane projection conversion unit), 52 Plane projection conversion calculation unit (floor surface image generation unit), 53 Floor plan view coordinates Calculation unit (planar projection conversion unit), 54 floor plan intermediate coordinate calculation unit (intermediate point determination unit), 55 original image coordinate calculation unit (floor surface image generation unit), 56 converted image coordinate calculation unit (floor Image generation means), 57 floor surface intermediate image plane projection conversion calculation section (floor surface image generation means), 58 floor area extraction section (floor surface image generation means), 59 floor surface intermediate image generation section (floor surface image generation) Means), 60 intermediate image object generation unit (intermediate image generation unit), 61 object size correction unit (intermediate image generation unit), 62 intermediate image generation unit (intermediate image generation unit).

Claims (5)

  Multi-view transformation of a plurality of camera images in which the same three-dimensional area is photographed from different directions, and the position and size of the object to be multi-viewed placed on the floor in the plurality of camera images Multi-view image generating means for generating a plurality of multi-view images having the same height, and a floor surface position for acquiring a position on the floor surface of one or more objects placed on the floor surface from the plurality of camera images An acquisition means, a plane projection conversion means for converting a position on the floor surface of one or more objects acquired by the floor surface position acquisition means to a position on a floor plan of the floor surface, and conversion by the plane projection conversion means Intermediate point determining means for identifying a plurality of positions related to the same article that are apparently different on the plan view from among the positions on the plan view of the one or more objects, and determining intermediate points of the plurality of positions; Intermediate point determination means A floor image generation means for converting the determined intermediate point to a position on the multi-view image and generating a floor image according to the position on the multi-view image; and a floor surface in the plurality of camera images An intermediate image between a plurality of multi-view images generated by the multi-view image generating unit is generated from the image of one or more objects placed on the floor and the floor image generated by the floor image generating unit. An image generation apparatus comprising intermediate image generation means.   The plane projection conversion means uses the image conversion parameters at the time of multiview conversion by the multiview image generation means to determine the position on the floor surface of one or more objects acquired by the floor surface position acquisition means. And converting the position on the multi-view image to a position on an arbitrary camera image, and converting the position on the arbitrary camera image to a position on a floor plan of the floor surface The image generation apparatus according to claim 1.   The midpoint determination means determines a midpoint between a plurality of positions related to the same article on a circular or elliptical arc centered on the position on the plan view of the object to be multi-view converted by the plane projection conversion means. The image generating apparatus according to claim 1, wherein a point at a point is determined as an intermediate point.   When determining an intermediate point between two positions related to the same article, the intermediate point determination means prepares a circle or an ellipse that passes through at least one of the two positions, and connects one position to the center point of the circle or the ellipse. The angle formed by the line segment and the line segment connecting the other position and the center point is divided at a predetermined ratio, and the point on the arc of the circle or ellipse corresponding to the angle formed after the division is determined as an intermediate point. The image generating apparatus according to claim 3.   The floor image generation means converts the intermediate point determined by the intermediate point determination means to a position on the multi-view image using the image conversion parameter at the time of multi-view conversion by the multi-view image generation means. The image generation device according to claim 1.

Images