JP4446041B2 - Camera vector computing device, shake component detecting device, image stabilizing device, position and orientation stabilizing device, target object lock-on device, and live-action object attribute calling device provided in this camera vector computing device - Google Patents

Description

The present invention relates to an apparatus that calculates a camera position and a camera direction of a camera that takes a video of a moving image and corrects an image blur based on the camera position information.
In particular, the present invention provides a camera vector calculation device that automatically obtains a camera vector such as a highly accurate camera position and camera rotation angle from a plurality of frame images of a moving image by calculation, and a video based on the obtained camera vector. Shake component detection device and image stabilization device capable of correcting irregular blur caused by camera shake of an image acquired in moving image shooting such as video, and generating a blur-free image from the blurred image , A position and orientation stabilization device, a target object lock-on device, and a live-action object attribute call device.

In general, a camera that shoots a moving image video includes a correction mechanism and a correction unit for correcting image blur due to camera shake, such as shake caused by vibration of an in-vehicle camera and camera shake of a photographer (for example). For example, refer to Patent Document 1-2).
Conventional image correction extracts the blur component of an image from multiple feature points in the image and moves the image in the vertical and horizontal directions so that the shaking component is reduced, so that the image changes smoothly. It was realized by processing the image.
Japanese Patent Laid-Open No. 10-023322 (page 3-7, FIG. 1) JP-A-10-136304 (page 2-3, FIG. 1)

However, in the conventional image correction process, the vertical and horizontal blur of the image can be suppressed to some extent, but the blur and image distortion caused by the irregular rotational motion in the camera viewpoint direction are corrected. I couldn't.
In other words, the conventional video image correction technology can correct the vertical and horizontal movements of the image due to camera shake, but cannot correct the distortion of the shake caused by the three-axis rotational components of the camera. .

  Therefore, as a result of earnest research, the inventor of the present application can calculate a highly accurate camera position, rotation angle, and the like by extracting a sufficient number of feature points from a plurality of frame images of a moving image. By performing image correction based on accurate camera position information, it has been conceived that distortion due to rotational components in the three-axis directions of the camera, which was impossible with the prior art, can also be corrected.

  That is, the present invention has been proposed to solve the above-described problems of the prior art, and by automatically extracting feature points from a plurality of frame images in a moving image, a highly accurate camera position and camera rotation angle are obtained. And a camera vector arithmetic unit capable of automatically calculating a camera vector, and a camera shake of an image acquired in moving image shooting such as a video image based on the obtained high-accuracy camera vector. It is an object of the present invention to provide a shake component detection device, an image stabilization device, a position and orientation stabilization device, a target object lock-on device, and a live-action object attribute call device that can correct an image and generate a blur-free image.

  In order to achieve the above object, a camera vector arithmetic device according to the present invention is characterized in that, as described in claim 1, a feature point extraction unit that automatically extracts a predetermined number of feature points from image data of a moving image, For feature points, a feature point correspondence processing unit that automatically traces within each frame image of a moving image and obtains a correspondence relationship between the frame images, and obtains the three-dimensional position coordinates of the feature point for which the correspondence relationship is obtained. And a camera vector calculation unit that obtains a camera vector composed of the three-dimensional position coordinates and the three-dimensional rotation coordinates of the camera corresponding to each frame image from the original position coordinates.

  According to another aspect of the present invention, the camera vector calculation device performs statistical processing to minimize the distribution of solutions of a plurality of camera vectors obtained by the camera vector calculation unit, and minimizes errors. In this configuration, an error minimizing unit that automatically determines the processed camera vector is provided.

  Further, in the camera vector calculation device according to the present invention, the camera vector calculation unit uses any two frame images Fn and Fn + m (m = frame interval) used for the camera vector calculation as a unit image. The unit calculation for obtaining the three-dimensional position coordinates of the desired feature points and the camera vector is repeated, and for the frame image between the two frame images Fn and Fn + m, the camera vector is obtained by a simplified calculation, and the error is minimized. The conversion unit adjusts and integrates the final camera so that an error of each camera vector obtained by calculating a plurality of times for the same feature point is minimized by continuously progressing n as the image progresses. This is a configuration for determining a vector.

  According to the camera vector calculation device of the present invention, as described in claim 4, the camera vector calculation unit sets the frame interval m from the camera to the feature point according to the distance from the camera to the feature point. The unit calculation is performed by setting m so as to increase as the distance increases.

  Further, in the camera vector calculation device according to the present invention, as described in claim 5, the camera vector calculation unit deletes a feature point having a large error distribution of the obtained camera vector, and if necessary, The camera vector is recalculated based on other feature points to increase the accuracy of the camera vector calculation.

  In the camera vector calculation device according to the present invention, the camera vector calculation unit is configured so that the camera vector calculation unit is based on a minimum number of frame images with desired accuracy and a minimum number of feature points automatically extracted. Performs computation, obtains and displays the approximate value of the camera vector in real time, and increases the number of frames and the number of feature points as the image accumulates as the image progresses. The approximate value of the camera vector is replaced with a highly accurate camera vector value and displayed.

  In the camera vector arithmetic device according to the present invention, the moving picture image in which the feature points are automatically extracted by the feature point extraction unit is composed of 360-degree all-round images.

  According to another aspect of the present invention, there is provided a plurality of camera vector calculation devices that acquire a video in a synchronized manner with a fixed positional relationship as a moving image video in which feature points are automatically extracted by the feature point extraction unit. A wide-angle field-of-view video with a wide field of view and a narrow-angle field-of-view video with a narrower field of view than the wide-angle field-of-view video acquired by the camera are input, the camera vector is calculated based on the image data of the wide-angle field-of-view video, A high-accuracy camera vector is obtained by substituting a camera vector value based on a camera vector calculation based on image data of a narrow-angle visual field image.

  The camera vector calculation device according to the present invention is based on three-dimensional information sequentially obtained as a part of an image with the camera vector obtained by the camera vector calculation unit as an approximate camera vector. Based on the 3D information tracking unit that continuously tracks partial 3D information contained in multiple frame images between adjacent frames and the tracking results of 3D information obtained by the 3D information tracking unit And a high-accuracy camera vector calculation unit that obtains a camera vector with higher accuracy than the approximate camera vector.

  According to a tenth aspect of the present invention, there is provided a camera vector calculation device according to claim 10, wherein a shift component between a camera vector obtained by the camera vector calculation unit and a scheduled camera vector indicating a camera position and a camera posture scheduled in advance. Is generated from a difference between the scheduled camera vector and the camera vector obtained at the present time or at an arbitrary time, and a predetermined positional deviation component signal and a rotational deviation component signal are generated. Further, it is configured to include a shaking component detection device that converts to a suitable coordinate system and outputs as a shaking of a camera or a fixed object to which the camera is fixed.

  According to the camera vector arithmetic device of the present invention, a camera vector obtained from an image in which the camera is shaking is assumed based on the output of the shaking component detection device, as described in claim 11. An image correction signal generation unit that generates a correction signal for correction and conversion so as to match a reference camera vector to be converted, and a case where the camera does not shake the image data due to the correction signal of the image correction signal generation unit A stabilized image conversion unit that performs conversion processing to a stabilized image equivalent to the case where the camera vector is captured as a reference vector, and a stabilized image output unit that outputs the stabilized image converted by the stabilized image conversion unit, And a display unit that displays the stabilized image output from the stabilized image output unit.

  The camera vector calculation device according to the present invention obtains a difference between a planned camera position and rotation and a current camera position and rotation based on an output of the shake component detection device. A position and orientation correction signal generation unit that three-dimensionally controls the position and orientation of a control object to which the camera is fixed and generates a correction signal that corrects the position and orientation to a predetermined position and orientation; and the position and orientation correction signal generation unit A control signal selection unit that selects an arbitrary correction signal from one or more generated correction signals and outputs it as a control signal for controlling the position and orientation of the camera, and a control signal output from the control signal selection unit A position and orientation stabilization device having a control target drive unit that drives a control target and controls the position and orientation of the camera is provided.

  According to another aspect of the present invention, a camera vector calculation device includes a live-action coordinate setting unit that sets an appropriate three-dimensional coordinate system in a live-action image, and an arbitrary object to be locked on in the image. Based on the camera vector obtained by the lock-on object designating unit designated by the camera vector calculation device, the object designated in the image is measured in a live-action coordinate system, and its three-dimensional coordinates are obtained. A lock-on control unit that controls the position and orientation of a fixed object to which an image display or camera is fixed so that the designated object whose coordinates are obtained is always displayed at the center position or a predetermined position of the image frame. A target object lock-on device is provided.

  Furthermore, as described in claim 14, the camera vector calculation device according to the present invention specifies an arbitrary object in an image based on the camera vector obtained by the camera vector calculation unit, or a specified object. It is configured to include a live-action object attribute calling device that locks on an object at a predetermined position in an image and calls and displays predetermined attribute information including the attribute, function, and content of the object.

According to the present invention as described above, first, in the camera vector calculation device, it is possible to automatically obtain a camera vector such as a highly accurate camera position and camera rotation angle from a plurality of frame images of a moving image by calculation. .
Then, based on the obtained camera vector, the shaking component detection device can detect the shaking component of the camera from the preliminarily scheduled reference camera position and orientation and the current camera position and orientation deviation.
Here, the pre-scheduled reference camera vector is, for example, the average position and rotation posture of the travel route, or the rail center line and the rail inclination in a vehicle traveling on a track. Or, it is the scheduled route data that has been input in advance at the latitude and longitude altitudes.

Based on this camera shake component, the image stabilization device first corrects irregular blurring or the like caused by camera shake in an image captured during moving image shooting such as a video image. No image can be generated.
In addition, the position and orientation stabilization device generates a blur-free image by controlling the position and orientation of the camera itself, or the position and orientation of the fixed object that fixes the camera, based on the camera shake component. can do.
Further, in the target object lock-on device, based on the camera vector obtained by the camera vector calculation device, an arbitrary object in the image is fixed at a predetermined position and posture in advance. The position and posture can be controlled.
Furthermore, the live-action target object attribute calling device specifies predetermined target information such as attributes and contents prepared in advance for the target object by specifying the target target object based on the three-dimensional coordinates obtained by the camera vector arithmetic unit. Can be called.

In the present invention, compared with the number of measurement points theoretically necessary for calculation, the number of measurement points is greatly increased, and the corresponding images may theoretically be two images. The camera position is greatly increased, and a highly accurate camera position can be obtained by calculation based on more information.
In addition, the stabilization of the image focuses not on the shaking of the image itself but on the shaking of the camera. Instead of correcting the image by moving it up, down, left, or right on the screen, it detects the camera vector from the image, detects the three-dimensional coordinates of the camera and the rotation in the three-axis direction, and the fluctuation component due to the shaking of the camera viewpoint Is detected, and image conversion processing is performed so that an image in the normal camera direction is obtained, thereby realizing complete stabilization of the image.

Hereinafter, a camera vector calculation device according to the present invention, and a shake component detection device, an image stabilization device, a position and orientation stabilization device, a target object lock-on device, and a live-action object attribute call device provided in the camera vector calculation device are described. A preferred embodiment will be described with reference to the drawings.
Here, the camera vector arithmetic device of the present invention described below is realized by processing, means, and functions executed by a computer according to instructions of a program (software). The program sends commands to each component of the computer, and performs predetermined processing and functions as shown below, such as automatic extraction of feature points, automatic tracking of extracted feature points, calculation of three-dimensional coordinates of feature points, camera Perform vector operations. Thus, each process and means in the camera vector arithmetic device of the present invention are realized by specific means in which the program and the computer cooperate.
Note that all or part of the program is provided by, for example, a magnetic disk, optical disk, semiconductor memory, or any other computer-readable recording medium, and the program read from the recording medium is installed in the computer and executed. The The program can also be loaded and executed directly on a computer through a communication line without using a recording medium.

[Camera vector arithmetic unit]
First, an embodiment of a camera vector arithmetic device according to the present invention will be described with reference to FIG. FIG. 1 is a block diagram showing a schematic configuration of a camera vector arithmetic device according to an embodiment of the present invention.
As shown in the figure, the camera vector calculation storage 10 includes an image acquisition unit 11, an image temporary recording unit 12, a feature point extraction unit 13, a feature correspondence processing unit 14, a camera vector calculation unit 15, and a minimum error. A conversion unit 16, a three-dimensional information tracking unit 17, a high-precision camera vector calculation unit 18, and a shake component detection device 20.

The image acquisition unit 11 acquires an arbitrary moving image shot by a video camera or the like. Here, the video acquired by the image acquisition unit 11 is preferably a wide-angle image such as a 360-degree all-round image, as will be described later. As will be described later, the video acquired by the image acquisition unit 11 includes a video obtained by a wide-angle camera with a wide field of view using a plurality of synchronized cameras, and a narrow-angle field of view that is narrower than the wide-angle camera. You can also use video obtained with a camera.
The image temporary recording unit 12 temporarily records the moving image acquired by the image acquisition unit 11.

The feature point extraction unit 13 automatically extracts a sufficient number of feature points from the image data in which the moving image shot by the video camera is temporarily recorded.
The feature correspondence processing unit 14 automatically obtains the correspondence relationship by automatically tracking the automatically extracted feature points in each frame image between the frames.
The camera vector calculation unit 15 automatically calculates a camera vector corresponding to each frame image from the three-dimensional position coordinates of the feature points for which the correspondence relationship has been determined.
The error minimizing unit 16 performs statistical processing so that the distribution of the solution of each camera vector is minimized by overlapping calculation of a plurality of camera positions, and automatically determines the camera position direction subjected to the error minimizing process. To do.

The three-dimensional information tracking unit 17 positions the camera vector obtained by the camera vector calculation unit 15 as an approximate camera vector, and based on the three-dimensional information sequentially obtained as part of the image in the subsequent process, a plurality of frame images 3D information is automatically tracked along an image of an adjacent frame. Here, three-dimensional information (three-dimensional shape) is mainly three-dimensional distribution information of feature points, that is, a collection of three-dimensional points, and this collection of three-dimensional points constitutes a three-dimensional shape. To do.
The high-precision camera vector calculation unit 18 generates and outputs a higher-precision camera vector than the camera vector obtained by the camera vector calculation unit 15 based on the tracking data obtained by the three-dimensional information tracking unit 17.
Then, the camera vector obtained as described above is input to the shake component detection device 20, and the shake component of the camera is detected.

There are several methods for detecting camera vectors from feature points of a plurality of images (moving images or continuous still images). However, in the camera vector calculation device 10 of the present embodiment shown in FIG. These feature points are automatically extracted and automatically tracked to obtain a three-dimensional vector and a three-axis rotation vector of the camera by epipolar geometry.
By taking a sufficient number of feature points, camera vector information is duplicated, and an error can be minimized from the duplicated information to obtain a more accurate camera vector.

Here, the camera vector refers to a vector of degrees of freedom possessed by the camera.
In general, a stationary three-dimensional object has six degrees of freedom of position coordinates (X, Y, Z) and rotation angles (Φx, Φy, Φz) of the respective coordinate axes. Therefore, the camera vector refers to a vector of six degrees of freedom of the camera position coordinates (X, Y, Z) and the rotation angles (Φx, Φy, Φz) of the respective coordinate axes. When the camera moves, the direction of movement also enters the degree of freedom, which can be derived by differentiation from the above six degrees of freedom.
As described above, the detection of the camera vector by the camera vector computing device 1 of the present embodiment is that the camera takes six degrees of freedom for each frame and determines six different degrees of freedom for each frame. It is.

Hereinafter, a specific method for detecting a camera vector in the camera vector arithmetic apparatus 10 will be described with reference to FIG.
First, the image data acquired by the image acquisition unit 11 is input to the feature point extraction unit 13 via the image temporary recording unit 12 (or directly), and in the frame image appropriately sampled by the feature point extraction unit 13, A point or a small area image to be a feature point is automatically extracted, and the feature correspondence processing unit 14 automatically obtains a correspondence relationship between the feature points among a plurality of frame images.
Specifically, more than a sufficient number of feature points that are used as a reference for detecting a camera vector are obtained. Examples of feature points between images and their corresponding relationships are shown in FIGS. In the figure, “+” is a feature point that is automatically extracted, and the correspondence is automatically tracked between a plurality of frame images (see correspondence points 1 to 4 shown in FIG. 4).
Here, for extracting feature points, as shown in FIG. 5, it is desirable to specify and extract a sufficiently large number of feature points in each image (see circles in FIG. 5). For example, about 100 feature points are extracted. Extract points.

Subsequently, the camera vector calculation unit 15 calculates the three-dimensional coordinates of the extracted feature points, and calculates the camera vector based on the three-dimensional coordinates. Specifically, the camera vector calculation unit 15 includes a sufficient number of feature positions that exist between consecutive frames, a position vector between moving cameras, a three-axis rotation vector of the camera, and each camera position and feature. In this embodiment, the relative values of various three-dimensional vectors, such as vectors connecting points, are calculated continuously. In this embodiment, for example, camera motion (camera position) is solved by solving an epipolar equation from the epipolar geometry of a 360-degree all-round image. And camera rotation).

Images 1 and 2 shown in FIG. 4 are images obtained by Mercator expansion of 360-degree all-round images. When latitude φ and light θ are assumed, points on image 1 are (θ1, φ1) and points on image 2 are ( θ2, φ2). The spatial coordinates of each camera are z1 = (cos φ1 cos θ1, cos φ1 sin θ1, sin φ1), z2 = (cos φ2 cos θ2, cos φ2 sin θ2, sin φ2). When the camera movement vector is t and the camera rotation matrix is R, z1 ^T [t] × Rz2 = 0 is the epipolar equation.
By providing a sufficient number of feature points, t and R can be calculated as a solution by the method of least squares by linear algebra calculation. This calculation is applied to a plurality of corresponding frames.

Here, it is preferable to use a 360-degree all-round image as an image used for the calculation of the camera vector.
The image used for the camera vector calculation may be any image in principle, but a wide-angle image such as a 360-degree all-round image shown in FIG. 4 can easily select many feature points. Therefore, in the present embodiment, the 360-degree all-round image is used for the camera vector calculation, whereby the tracking distance of the feature points can be increased, and a sufficiently large number of feature points can be selected. It is possible to select feature points that are convenient for medium and short distances. In addition, when correcting the rotation vector, the calculation process can be easily performed by adding the polar rotation conversion process. As a result, a calculation result with higher accuracy can be obtained.
Note that FIG. 4 is a Mercator projection that is a 360-degree spherical image obtained by synthesizing images taken by one or a plurality of cameras in order to facilitate understanding of the processing in the camera vector computing device 1. Although a developed image is shown, the actual camera vector calculation device 1 does not necessarily need to be a developed image by the Mercator projection.

Next, the error minimizing unit 16 calculates a plurality of vectors based on each feature point by a plurality of calculation equations based on a plurality of camera positions and the number of feature points corresponding to each frame, Statistical processing is performed so that the distribution of the position of each feature point and the camera position is minimized to obtain a final vector. For example, the optimal solution of the least square method is estimated by the Levenberg-Marquardt method for multiple frame camera positions, camera rotations, and multiple feature points, and errors are converged to determine the camera position, camera rotation matrix, and feature point coordinates. .
Further, feature points having a large error distribution are deleted, and recalculation is performed based on other feature points, thereby improving the accuracy of computation at each feature point and camera position.
In this way, the position of the feature point and the camera vector can be obtained with high accuracy.

6 to 8 show examples of the three-dimensional coordinates of the feature points and camera vectors obtained by the camera vector calculation device 1. 6 to 8 are explanatory diagrams illustrating the vector detection method of the present embodiment, and are diagrams illustrating the relative positional relationship between the camera and the object obtained from a plurality of frame images acquired by the moving camera. .
In FIG. 6, the three-dimensional coordinates of the feature points 1 to 4 shown in the images 1 and 2 in FIG. 4 and the camera vector that moves between the images 1 and 2 are shown.
7 and 8 show a sufficiently large number of feature points, the positions of the feature points obtained from the frame image, and the position of the moving camera. In the figure, a circle mark that continues in a straight line at the center of the graph is the camera position, and a circle mark located around the circle indicates the position and height of the feature point.

Here, the calculation in the camera vector calculation device 1 is performed in accordance with the distance of the feature point from the camera, as shown in FIG. 9, in order to obtain more accurate three-dimensional information of the feature point and the camera position. Set feature points and repeat multiple operations.
Specifically, the camera vector calculation device 1 automatically detects feature points that have image characteristics in an image and uses them for camera vector calculation when obtaining corresponding points of feature points in each frame image. The unit calculation is repeated by paying attention to the two frame images Fn and Fn + m of the nth and n + mth, and the unit calculation with n and m appropriately set is repeated.
m is the frame interval, and the feature points are classified into a plurality of stages according to the distance from the camera to the feature point in the image. The distance from the camera to the feature point is set so that m becomes larger. It is set so that m is smaller as the distance to is shorter. This is because the change in position between images is less as the distance from the camera to the feature point is longer.

Then, while sufficiently overlapping the classification of the feature points by the m value, a plurality of stages of m are set, and as n progresses continuously with the progress of the image, the calculation proceeds continuously. Then, the overlap calculation is performed a plurality of times for the same feature point in each step of n and m.
In this way, by performing unit calculation focusing on the frame images Fn and Fn + m, a precise camera vector is calculated over a long time between each frame sampled every m frames (frames are dropped). However, in m frames (minimum unit frames) between the frame images Fn and Fn + m, a simple calculation that can be performed in a short time can be performed.

If there is no error in the precision camera vector calculation for every m frames, both ends of the camera vector of the m frames overlap with the Fn and Fn + m camera vectors that have been subjected to the high precision calculation. Accordingly, m minimum unit frames between Fn and Fn + m are obtained by a simple calculation, and both ends of the camera vector of the m minimum unit frames obtained by the simple calculation are Fn and Fn + m obtained by high precision calculation. The scale adjustment of m consecutive camera vectors can be made to match the camera vectors.
In this way, as n progresses continuously with the progress of the image, the scale adjustment is performed so that the error of each camera vector obtained by calculating a plurality of times for the same feature point is minimized, and integration is performed. A camera vector can be determined.
Accordingly, it is possible to speed up the arithmetic processing by combining simple arithmetic operations while obtaining a highly accurate camera vector having no error.

Here, there are various simple calculation methods depending on the accuracy. For example, when (1) many feature points of 100 or more are used in high-precision calculation, the minimum number of simple calculation is about ten. (2) Even if the number of the same feature points is the same as the feature points and camera positions, innumerable triangles are established there, and equations for that number are established. By reducing the number of equations, it can be simplified.
In this way, integration is performed by adjusting the scale so that the error of each feature point and camera position is minimized, distance calculation is performed, and feature points with large error distribution are deleted, and other features are added as necessary. By recalculating the points, the calculation accuracy at each feature point and camera position can be improved.

  In addition, by performing high-speed simple calculation in this way, camera vector real-time processing becomes possible. In real-time processing of camera vectors, calculation is performed with the minimum number of frames that can achieve the target accuracy and the minimum number of feature points that are automatically extracted, the approximate value of the camera vector is obtained and displayed in real time, and then the image is accumulated. Accordingly, the number of frames can be increased, the number of feature points can be increased, camera vector calculation with higher accuracy can be performed, and approximate values can be replaced with camera vector values with higher accuracy for display.

Furthermore, in this embodiment, tracking of three-dimensional information (three-dimensional shape) is performed in order to obtain a more accurate camera vector.
Specifically, first, the three-dimensional information tracking unit 17 positions the camera vector obtained through the camera vector calculation unit 15 and the error minimization unit 16 as an approximate camera vector, and the image generated in the subsequent process Based on the three-dimensional information (three-dimensional shape) obtained as a part, partial three-dimensional information included in a plurality of frame images is continuously tracked between adjacent frames to automatically track the three-dimensional shape.
Then, from the tracking result of the three-dimensional information obtained by the three-dimensional information tracking unit 17, a higher-precision camera vector is obtained in the high-precision camera vector calculation unit.

The feature point extraction unit 13 and the feature point correspondence processing unit 14 described above automatically track feature points in a plurality of inter-frame images, but the number of feature point tracking frames is limited due to disappearance of feature points. May come. In addition, since the image is two-dimensional and the shape changes during tracking, there is a certain limit in tracking accuracy.
Therefore, the camera vector obtained by the feature point tracking is regarded as an approximate value, and the three-dimensional information (three-dimensional shape) obtained in the subsequent process is traced on each frame image, and a high-precision camera vector is obtained from the trajectory. Can do.
The tracking of 3D shapes is easy to obtain matching and correlation accuracy, and the 3D shape does not change in size and size depending on the frame image, so it can be tracked over many frames. The accuracy of the camera vector calculation can be improved. This is possible because the approximate camera vector is known by the camera vector calculation unit 15 and the three-dimensional shape is already known.

  When the camera vector is an approximate value, the error of 3D coordinates over a very large number of frames is small because there are few frames related to each frame by feature point tracking. The error of the three-dimensional shape when a part of the image is cut is relatively small, and the influence on the change and size of the shape is considerably small. For this reason, the comparison and tracking in the three-dimensional shape is extremely advantageous over the two-dimensional shape tracking. In tracking, when tracking with 2D shape, tracking changes in shape and size in multiple frames are unavoidable, so there are problems such as large errors and missing corresponding points. However, in tracking with a three-dimensional shape, there is very little change in shape, and in principle there is no change in size, so accurate tracking is possible.

Here, as the three-dimensional shape data to be tracked, there are, for example, a three-dimensional distribution shape of feature points, a polygon surface obtained from the three-dimensional distribution shape of feature points, and the like.
It is also possible to convert the obtained three-dimensional shape from a camera position into a two-dimensional image and track it as a two-dimensional image. Since the approximate value of the camera vector is known, projection conversion can be performed on a two-dimensional image from the camera viewpoint, and it is also possible to follow a change in the shape of the object due to movement of the camera viewpoint.

The camera vector obtained in this way can be displayed in the generated three-dimensional map.
For example, as shown in FIG. 10, the image from the in-vehicle camera is developed in a plane, the corresponding points on the target plane in each frame image are automatically searched, and the corresponding points are combined to match the target plane. A combined image is generated and displayed in the same coordinate system. Then, the camera position and the camera direction can be detected one after another in the common coordinate system, and the position, direction, and locus can be plotted.
The high-accuracy camera vector obtained as described above is input to the shake component detection device 20 and used for shake evaluation of an arbitrary object in the image.

[Swing component detector]
The shake component detection device 20 determines a vehicle position (corresponding to the camera position on a one-to-one basis) and a vehicle from a camera vector (a three-dimensional position coordinate and a three-axis rotation coordinate of the camera) obtained by the camera vector calculation device 10. A deviation component with respect to the scheduled camera vector that is a rotational attitude (corresponding to the camera attitude one-to-one) is extracted.
Then, a positional deviation component signal and a rotational deviation component signal are generated from the difference between the scheduled camera vector and the current camera vector or the difference between the camera vector at the time of evaluation and all or one of these deviation component signals is generated. The value of each part and their selection and combination is converted into a coordinate system that should be evaluated appropriately according to the purpose, and the shaking of the camera (a fixed object such as a vehicle to which the camera is fixed) is evaluated and output, and necessary If there is, it is displayed.

In the shake component detection device 20, for example, δX, δY, which are shake components due to vehicle positions (that is, camera positions) X, Y, and Z on which an in-vehicle camera is mounted and vehicle rotation postures (that is, camera postures) Φx, Φy, and Φz. , ΔZ, δΦx, δΦy, and δΦz are all subject to evaluation.
Here, δX, δY, δZ, δΦx, δΦy, and δΦz do not necessarily mean differential values or difference values, but mean deviations from a predetermined position and a predetermined posture. In many cases, the vibration component can be detected by substituting with a differential value. However, if a predetermined position and a predetermined posture are determined in advance, the difference between them is δX, δY, δZ, δΦx, δΦy, and δΦz.

Specifically, in a train traveling on a track, the planned camera vector is close to the average value measured during traveling, but when navigating in a three-dimensional space like an aircraft, the planned camera vector is It doesn't match on average with the running one.
As the shake component output, a total of 12 parameters of X, Y, Z, Φx, Φy, Φz and δX, δY, δZ, δΦx, δΦy, δΦz can be output.
However, depending on which shake evaluation is intended, the number of parameters can be selectively combined from these, and can correspond to the evaluation object.
That is, when the outputs from the camera vector calculation device 10 and the shake component detection device 20 are combined, X, Y, Z, Φx, Φy, Φz, δX, δY, δZ, δΦx, δΦy, δΦz.
There are twelve parameters, but only three parameters δΦx, δΦy, and δΦz are required for normal image stabilization processing.

On the other hand, when a plurality of cameras are used at the same time, the three-dimensional position of the image can be corrected. Therefore, it is necessary to prepare parameters δX, δY, and δZ.
In general, posture control requires six parameters in total, including δΦx, δΦy, δΦz, and position control, in addition to δΦ, δY, and δZ in the case of rotation control. Further, if the situation judgment is included, X, Y, Z, and Φx, Φy, Φz, which are outputs from the camera vector calculation device 10
Therefore, it can be selectively combined from the 12 obtained parameters and used for image processing and attitude control.
In addition to the twelve variables, other coefficients depending on the shooting conditions used for image stabilization and posture stabilization include a limitation on the swing width of the image frame as a reference posture of the camera.

FIG. 11 shows a specific example of shaking component detection in the shaking component detection device 20.
The example shown in the figure is a case where a camera is attached to a vehicle for traveling, and the shaking component detection device 20 detects shaking from a moving image taken at that time.
In the figure, the bold arrow indicates the traveling direction of the vehicle to which the camera is attached, and the position and orientation of the camera with the camera optical axis as the origin is the camera coordinate system (Xc, Yc, Zc) (shown in the figure). A broken line is a vehicle coordinate system (Xt, Yt, Zt) that is mounted in a semi-fixed state (solid line shown in the figure), and a coordinate system that always changes the coordinate axis in the vehicle traveling direction is a rotating world coordinate system (Xwr , Ywr, Zwr) (two-dot chain line shown in the figure), and a coordinate system representing the static system of the outside world is a world coordinate system (Xw, Yw, Zw) (one-dot chain line shown in the figure). Then, the relationship between the four coordinate systems is obtained and converted into a coordinate system necessary for evaluation to express the vehicle shake.

The camera vector obtained by the camera vector arithmetic unit 10 is the camera coordinate system (Xc, Yc, Zc) itself. Since the camera coordinate system is generally set in an arbitrary direction, the camera coordinate system is temporarily converted into a vehicle coordinate system (Xt, Yt, Zt) in order to detect vehicle shake. This conversion is simply rotational conversion, and is generally semi-fixed. Once set, there is no change until the measurement is finished.
By selecting the vehicle traveling direction as one of the three axes of the vehicle coordinate system (Xt, Yt, Zt), a coordinate system suitable for evaluating shaking can be obtained.
Further, it is appropriate to express the trajectory of the vehicle movement in the world coordinate system (Xw, Yw, Zw) which is a stationary coordinate system. To express the velocity, it can be simply expressed in the rotating world coordinate system (Xwr, Ywr, Zwr), but to express it as a vector, it is appropriate to express it in the world coordinate system (Xw, Yw, Zw).

In shake evaluation, evaluation is performed in a coordinate system suitable for shake evaluation.
The shake signal is detected as a deviation from the planned course, but in the example shown in FIG. 11, the shake is evaluated using the average course of the vehicle as the planned course. Therefore, the movement trajectory of the camera is obtained on the world coordinate system, the average course is obtained, and this is set as the planned course.
According to the camera vector calculation device 10 and the shake component detection device 20 according to the present invention, it is possible to detect a shake component only with a camera that acquires image data without using a gyro or the like as a reference for the posture. In the case of one camera, the obtained camera vector is a relative value, and since there is no calibration device with a world coordinate system such as a gyro, accumulation of errors occurs. For this reason, in order to always evaluate the swing with respect to the vehicle, it is necessary to give an average vertical horizontal direction.

Thus, if one camera coordinate system is set so as to coincide with the horizontal axis with respect to the vehicle at the time of camera installation, the horizontal posture can be easily calibrated later.
Thus, the camera coordinate system (Xc, Yc, Zc) may be converted into the vehicle coordinate system (Xt, Yt, Zt), and the shake may be measured and evaluated.
The shakes to be evaluated include positional deviation components Xt, Yt, Zt, rotational components Φxt, Φyt, Φzt, and positional deviation differences δXt, δYt, δZt, etc. (where Zt and δZt are the traveling direction velocity and its Because it is an acceleration component, the meaning of shaking is different from other components).

In the evaluation of the shaking component as described above, the following variables and display should be evaluated.
-Vehicle position display in the world coordinate system:
(Xw, Yw, Zw)
・ Speed and acceleration display in the rotating world coordinate system rotated in the vehicle traveling direction:
(ΔXwr, δYwr, δZwr) (ΔδXwr, ΔδYwr, ΔδZwr)
-Shake display in the vehicle coordinate system:
(△ Xt, △ Yt, (△ Zt)) (△ Φxt, △ Φyt, △ Φzt)
・ Rotary display of vehicle coordinate system and camera coordinate system (semi-fixed):
(Xc, Yc, Zc) = F (Xt, Yt, Zt)
・ Direction display in world coordinate system:
(Xw, Yw, Zw) = G (Xt, Yt, Zt)
・ Direction display in camera coordinate system:
(Xc, Yc, Zc) = H (Xt, Yt, Zt)
・ Vehicle coordinate system origin movement and rotation posture display with respect to the world coordinate system:
(Xw, Yw, Zw) (δXw, δYw, δZw)

According to the shaking component detection device 20 of the present embodiment as described above, for example, in the case of a camera attached to a train, the shaking component detection device 20 analyzes and analyzes the shaking of the train, and the vehicle or the track is abnormal. It becomes possible to discover. Whereas swings are usually measured using an expensive device such as a mercury acceleration watch, by using the swing component detection device 20 of the present embodiment, swing components can be easily detected and displayed. Can do.
By using such a shaking component detection device 20, as will be described later, an image stabilization device, a position / posture stabilization device, and a target object lock-on device according to the present invention can be realized.

As described above, according to the camera vector computing device 10 according to the present embodiment, a sufficient number of feature points are automatically detected from a plurality of frame images of a moving image, and feature points are automatically tracked between frames. Thus, it is possible to calculate the camera position and the rotation angle with high accuracy by performing overlap calculation on a large number of feature points.
Therefore, even if it is a normal inexpensive camera, a person takes a picture while moving with the camera, or takes a surrounding image from a moving vehicle carrying the camera, etc., and analyzes the video to obtain a highly accurate camera trajectory. It can be obtained in three dimensions.

Then, the camera vector calculation device 10 can automatically obtain a camera vector such as a highly accurate camera position and a camera rotation angle from a plurality of frame images of the moving image by calculation, and based on the obtained camera vector. Since the shake component detection device 20 can detect and evaluate the shake component of the camera, the image stabilization device 30 uses the output of the shake evaluation to capture a camera of an image acquired in moving image shooting such as a video image. It is possible to correct irregular blurring or the like caused by the shaking of the image and generate a blur-free image from the blurred image.
Further, the position and orientation stabilization device can generate a blur-free image by controlling not the image but the position and orientation of the camera itself based on the camera shake component.

In the target object lock-on device, the object specified in the image is fixed at an arbitrary position by correcting the camera shake component based on the obtained camera vector and if necessary. In addition, the position and orientation of the image or camera can be controlled.
Furthermore, the live-action target object attribute calling device specifies predetermined target information such as attributes and contents prepared in advance for the target object by specifying the target target object based on the three-dimensional coordinates obtained by the camera vector arithmetic unit. Can be called.
Hereinafter, embodiments of an image stabilization device, a position / posture stabilization device, a target object lock-on device, and a live-action target object call device provided in the camera vector calculation device of the present invention will be described with reference to FIGS. .

[Image Stabilizer]
As shown in FIG. 12, the image stabilization device 30 of the present embodiment includes an image correction signal generation unit 31, a stabilized image conversion unit 32, a stabilized image output unit 33, and a display unit 34.
The image correction signal generation unit 31 uses a camera position vector and a rotation vector obtained by the shake component detection device 20 of the camera vector calculation device 10 and a shake vector to obtain a camera vector obtained from an image in which the camera is shaken. A correction signal is generated for correction and conversion so as to match a reference camera vector assumed when the camera does not shake.
The stabilized image conversion unit 32 performs conversion processing to a stabilized image equivalent to a case where a camera vector assumed when the camera is not shaken is used as a reference vector.
The stabilized image output unit 33 outputs the stabilized image converted by the stabilized image conversion unit 32.
The display unit 34 displays the output stabilized image.

As described above, a camera generally has X, Y, Z, its rotation vector, and a total of six degrees of freedom, and the camera vector arithmetic unit 10 detects them. Then, among the detected camera positions X, Y, Z and the degrees of freedom of the three-axis rotation vectors Φx, Φy, Φz, the vibration component detection device 20 uses X, Y, Z, Φx, Φy, Φz, δX, δY, δZ, δΦx, δΦy, and δΦz are detected.
The image correction signal generation unit 31 generates a correction signal for stabilizing the image based on the shake component detected by the shake component detection device 20. With this correction signal, a normalized image stabilized in the stabilized image conversion unit 32 is generated.

Here, the image serving as the reference for the stabilized image may be an image at the start of the frame, but an image after conversion such as by aligning the horizontal position may be used as the reference for the stabilized image. Note that the image converted into the stabilized image has the viewpoint direction corrected, and therefore the projected plane is different and does not overlap with the image before correction.
FIG. 13 shows an example of an image that is converted into a stabilized image by a correction signal. For example, an image with fluctuations as shown in FIGS. As shown in (d), it is output and displayed as a corrected stabilized image.
FIG. 14 is a graph showing the trajectory of the corrected camera vector. In FIG. 14, the trajectory of camera movement is arranged in a straight comb shape at the center of the graph, and indicates the position and height of the moving camera. .

According to the image stabilization device 30 as described above, the camera vector calculation device 10 can automatically obtain a camera vector such as a highly accurate camera position and camera rotation angle from a plurality of frame images of a moving image by calculation. Therefore, based on the obtained camera vector and its shake component, irregular blurring caused by camera shake of an image acquired in video shooting such as a video image is corrected, and there is no blurring from a blurry image. An image can be generated.
In particular, the number of measurement points is greatly increased compared to the number of measurement points that are theoretically necessary for calculation, and the corresponding images may theoretically be two images, but the number is greatly increased. The camera position can be calculated by calculation based on more information.
In addition, the stabilization of the image focuses not on the shaking of the image itself but on the shaking of the camera. Instead of moving and correcting the image, it detects the camera vector from the image, detects the camera's three-dimensional coordinates and rotation in the three-axis direction, detects the fluctuation component due to the camera's viewpoint direction fluctuation, By performing image conversion processing so that the image is in the camera direction, complete image stabilization can be realized.

[Position and orientation stabilization device]
As shown in FIG. 12, the position and orientation stabilization device 40 includes a position and orientation correction signal generation unit 41, a control signal selection unit 42, and a control target drive unit 43.
The position / orientation correction signal generation unit 41 obtains a difference between the planned camera position and orientation and the current camera position and orientation based on the output of the shake component detection device 20, and controls the camera position and orientation to schedule A correction signal for correcting the position and orientation is generated.
The control signal selection unit 42 selects an arbitrary correction signal from one or more correction signals generated by the position / orientation correction signal generation unit 41 and outputs the selected correction signal as a control signal for controlling the position and orientation of the camera.
The control target drive unit 43 drives the control target with the control signal output from the control signal selection unit 42 to control the position and orientation of the camera.

Specifically, the position / orientation correction signal generation unit 41 generates a position correction signal for controlling the camera position and an attitude control signal for controlling the camera attitude.
The camera position correction signal is obtained from the three-dimensional position signal of the camera and the three-axis rotation signal X, Y, Z, Φx, Φy, Φz obtained by the camera vector calculation device 10, and the vehicle position (camera position) X, A position to extract Y, Z, and shake components Φx, Φy, Φz due to rotation from the camera posture, and to control and correct the camera position from the difference between the planned vehicle position (camera position) and the current camera position It is generated as a correction signal.
The camera attitude correction signal generates a rotation correction signal from the difference between the normal vehicle attitude (camera attitude) and the current vehicle attitude (camera attitude), and loads the camera or a holding mechanism to which the camera is fixed or a camera. It is generated as a rotation correction signal for controlling and correcting the posture of a moving object such as a moving vehicle.

  Note that X, Y, Z, δX, δY, and δZ output from the camera vector calculation device 10 and the shake component detection device 20 are relative values. For example, not only the rotation attitude control but also the position coordinates in latitude and longitude altitudes. If control is required, the camera vector must be converted to an absolute coordinate system. For that purpose, the scale is calibrated by aligning with the measured coordinates at least three points or the scale is calibrated by GPS or the like. Here, it is necessary to acquire the absolute coordinate system in the case of the position and orientation stabilization device 40 that controls the posture including the position. In the case of the posture control of only the rotational posture and the image stabilization device 30, Processing can be performed with the relative coordinates output from the camera vector arithmetic unit 10 as they are.

In the control signal selection unit 42, an appropriate correction signal is selected from various correction signals of the position correction signal and the rotation correction signal generated by the position / orientation correction signal generation unit 41, and drive control of the camera, the camera holding mechanism, the vehicle, and the like is performed. Output a control signal.
Then, the control target drive unit 43 drives the control target (camera, camera holding mechanism, vehicle, etc.) based on the control signal output from the control signal selection unit 42 so that the position and orientation of the camera are controlled. become.
In this way, the position / orientation stabilization device 40 drives and controls the position and orientation of the camera itself based on the camera vector obtained by the camera vector calculation device 10 and its shake component, and the image stabilization device 30 As with, the image can be stabilized (see FIG. 14).

Here, the position / orientation stabilization apparatus 40 according to the present embodiment controls the camera, the camera holding mechanism, and the like by a correction signal generated in real time simultaneously with the shooting and movement of the camera, and positions the image acquisition camera in real time. By controlling and rotating, the posture of the camera can be stabilized and used as an image stabilization device.
As for the already obtained image, the triaxial rotation signal can be taken out as a correction signal as post-processing from the acquired image, and the image can be rotationally corrected and stabilized, but the position cannot be controlled.
Further, when the camera vector calculation device 10 is applied to an aircraft, a vehicle, or the like, it is an extremely useful device if the position of the vehicle can be controlled by controlling not only the rotation attitude control of the camera but also the vehicle to which the camera is fixed. It becomes.

Therefore, for example, as shown in FIG. 15, the camera vector calculation camera (control signal acquisition camera unit 11 a) and the video acquisition camera (video acquisition camera unit 11 b) are separated and the positional relationship is fixed. For any of the cameras 11a and 11b, the camera holding mechanism (or a loaded vehicle or the like) is directly controlled in posture to stabilize the position and posture of the camera (vehicle), and at the same time, the target image Can be stabilized.
In this case, as shown in FIG. 15, it is possible to further stabilize the image by the image stabilizing device 30 for the camera unit 11b for acquiring images.

In this way, by separating the camera unit 11a for the camera vector arithmetic device and the camera unit 11b for obtaining the video signal, for example, the video of the narrow-angle lens camera is controlled by position and orientation control based on the video of the wide-angle lens camera. It is also possible to control this, further expanding the applications.
In general, since the accuracy of camera vector calculation is reduced in an image of a narrow-angle lens camera, a wide-angle camera is required for camera vector calculation. However, a camera intended for video is not necessarily a wide-angle field-of-view camera.
Therefore, as shown in FIG. 15, if the camera for calculating the camera vector and the camera for acquiring the video signal are separated, the camera for acquiring the video and the camera for calculating the camera vector can be freely selected. This makes it possible to use an ultra-wide-angle lens as the camera vector calculation camera, eliminating the need to examine image quality. For cameras dedicated to video acquisition, it is not necessary to examine the pixel-by-pixel image accuracy. Will be able to use.

Here, the control of the position and orientation of the camera may be orientation control or position control depending on which combination of X, Y, Z, Φx, Φy, and Φz is used to generate a control signal. Yes, it is possible to appropriately select and set the control signal according to the application target and purpose of use of the camera vector arithmetic unit.
For example, according to the rotation correction signal generated from the camera rotation angles Φx, Φy, Φz, the rotation angle can be controlled to be kept constant.
Further, the movement when the helicopter hovers in the sky can be realized by controlling all the components of X, Y, Z, Φx, Φy, and Φz.
Further, in the case of position control, it is possible to detect a deviation from a planned three-dimensional position and feed it back as a position control signal for a vehicle or the like, thereby leading to a planned route. Thereby, it can utilize as a position control by comparing the planned route of a vehicle or an aircraft with X, Y, and Z.

As described above, the camera vector calculation device 10 according to the present invention can be used as a position and orientation control device for a vehicle, an aircraft, or the like.
In addition, when performing the position and orientation control of the moving object in this way, since real-time property is required, the real-time image 12a is directly guided to the camera vector arithmetic unit 10 as shown in FIG. It is preferable to perform real-time calculation of the camera vector in light of the recorded image 12b). Since the high-precision camera vector obtained by the real-time calculation includes the three-dimensional position and the three-axis rotation signal of the camera, it is possible to realize attitude control and position control of the vehicle loaded with the camera.
Further, by combining a camera vector based on a wide-angle visual field image, which will be described later, with a high-precision camera vector arithmetic device (see FIG. 16) by substituting it into a camera vector operation based on a narrow-angle visual field image, a highly accurate position and orientation Control becomes possible.

[Target object lock-on device]
As shown in FIG. 12, the target object lock-on device 50 includes a live-action coordinate setting unit 51, a lock-on target designating unit 52, and a lock-on control unit 53.
The live-action coordinate setting unit 51 sets an appropriate three-dimensional coordinate system in the live-action image.
The lock-on target specifying unit 52 specifies an arbitrary target to be locked on in the image.
Based on the camera vector obtained by the camera vector computing device 10, the lock-on control unit 53 measures the object specified in the image using the live-action coordinate system and obtains its three-dimensional coordinate, and the three-dimensional coordinate is obtained. The position and orientation of the image display or the camera (fixed object to which the camera is fixed) are controlled so that the designated object is always displayed at the center position (or any predetermined position) of the image frame.

Thereby, the target object lock-on device 50 sets a three-dimensional coordinate system that matches the image in advance in the image, designates the target object in the image in the image with a mouse or the like, and The approximate three-dimensional coordinates of the target object are obtained by the vector arithmetic device, and the image display position is controlled so that the three-dimensional coordinates of the target object obtained by the camera vector arithmetic device are always at the designated position in the image. Can do.
Alternatively, the camera can be controlled so that the target object is always displayed at a designated position in the image.
The target image that is controlled to be locked on in this way may be an original image that still contains the shaking component acquired by the image acquisition unit 11 of the camera vector arithmetic unit 10, or the stabilized image of the image stabilization device 30. It may be a stabilized image output from the output unit 33.

If the target object is designated and captured in the image with a mouse or the like, the camera vector calculation device 10 detects feature points in the designated part and its surroundings and tracks them, so that the designated point is immediately detected from the camera vector. Three-dimensional coordinates are obtained.
As a result, the position and orientation of the camera can always be directed to the object, and the object can always be observed at a predetermined position of the camera (usually the center of the camera). This is a very useful means that can lock on an object without relying on image recognition.
2. Description of the Related Art Conventionally, in order to capture an object at the center of an image, a method of performing lock-on by performing image recognition of the object and chasing the recognized object is known. However, with this conventional method, when the viewpoint changes, the shape of the object changes, and it is difficult to track over many frames following the changing image.
According to the target object lock-on device 50 of the present embodiment, since the three-dimensional coordinates of the object are tracked, the object can be tracked regardless of the shape change of the object due to the viewpoint change.

Note that this lock-on control is possible not only for a stabilized image but also for an original image that still contains a shaking component. Stabilizing and controlling the camera will stabilize the position. In that sense, the target object lock-on device 50 can be regarded as a modified embodiment having a part of the functions of the image stabilization device 30 or the position and orientation stabilization device 40.
In addition, the target object lock-on device 50 can be locked on in response to the output of the camera vector arithmetic device 10, or can be locked on in response to the output of the already stabilized image stabilization device 30. Furthermore, the image can be locked on using the component detected by the shaking component detection device 20. That is, in the lock-on process, even if the position of the image can be fixed at a predetermined position such as the center of the image frame, the rotational component of the image remains as it is, so that the shake may remain, and through the shake component detection device 20 If it passes, it is possible to display a lock-on image without rotation with the rotation component corrected.
Then, by specifying a target object, three-dimensional coordinates are acquired, and the coordinates are fixed and displayed at an arbitrary position in the image frame.

[Live Action Object Attribute Calling Device]
The live-action object attribute calling device 60 shown in FIG. 12 designates an arbitrary object in the image based on the camera vector obtained by the camera vector computing unit 15 of the camera vector computing device 10, or designates the designated object. Is locked on at a predetermined position in the image, and predetermined attribute information including the attribute, function, and content of the object is called.
Specifically, the live-action object attribute calling device 60 uses this camera vector because the camera position and orientation of each adjacent image are already known from the camera vector already obtained by the camera vector computing device 10. By specifying the target object in the image with a mouse, touch panel, light pen, etc., the target object is three-dimensionally measured, and the target object is related in advance with its three-dimensional coordinates and its peripheral coordinates. Is specified, and it has functions such as calling an attribute of a target object, calling another function, and calling a content.
Further, the target object can be locked on at a predetermined position in the image by being designated by the target object lock-on device 50.

Since the target object specified by the mouse or the like in the live-action video is specified by the three-dimensional coordinates obtained by the camera vector calculation device 10, the target object is specified by specifying the target object by the mouse or the like in the image. It is possible to call other attributes and contents, and other contents. That is, the target object itself can be used as a three-dimensional icon.
Then, by combining this live-action object attribute calling device 60 with the target object lock-on device 50, clicking the object in the moving image or the object in the paused image with the mouse or the like, the image is displayed. The target object is always displayed at the designated position, and information related to the target object can be easily extracted or transferred to another function.
Note that the objects referred to here include not only live-action images but also CG (computer graphics). In the case of CG, since the three-dimensional coordinates are known, the attribute information can be called by specifying the object without requiring the camera vector arithmetic unit 10.

[High-precision camera vector calculation using wide-angle and narrow-angle images]
Next, in the camera vector calculation device (including a shake component detection device, an image stabilization device, a position and orientation stabilization device, a target object lock-on device, and a live-action object attribute call device) having the above-described configuration, a wide angle A case where high-precision camera vector calculation using a camera image and a narrow-angle camera image is performed will be described with reference to FIGS.
As shown in FIG. 16, in the camera vector calculation device 10, the positional relationship between a wide-angle field-of-view camera with a wide field of view and a narrow-angle field-of-view camera with a narrower field of view than that of the wide-angle camera is fixed and used at the same time. Input a wide-angle field-of-view video from the wide-angle camera and a narrow-angle field-of-view video from the narrow-angle field-of-view camera, and substitute the camera vector value obtained by calculation from the wide-angle field-of-view image when calculating the camera vector from the narrow-angle field of view Thus, a highly accurate camera vector can be obtained.

In general, since a wide-angle lens has a wide field of view, the tracking point can be taken over a long distance, so that the positional accuracy is high. However, since the number of pixels per field angle is small, the rotation accuracy is lower than that of a narrow angle lens.
On the other hand, since the narrow-angle lens has a narrow field of view, the positional accuracy cannot be obtained. However, since the number of pixels per angle of view is large, the angle accuracy is high.
Therefore, in the camera vector calculation device 10 shown in FIG. 16, a high-precision camera is obtained by performing camera vector calculation using both a wide-angle visual field image obtained by a wide-angle lens and a narrow-angle visual field image obtained by a narrow-angle lens. A vector can be obtained.

FIG. 17 shows a more specific example of the camera vector calculation device 10 having both a wide-angle lens camera and a narrow-angle lens camera.
In the figure, a wide-angle visual field camera and a narrow-angle visual field camera are fixedly installed in a vehicle or the like in the image acquisition unit 11 and are used synchronously.
The video output (1) from the wide-angle viewing camera is image-compressed by a method such as AVI, for example, and is temporarily stored in the recording unit (image temporary recording unit 12). Then, a necessary image frame is called, and the camera vector calculation device 10 performs the above-described camera vector calculation.

On the other hand, the camera vector calculation device 10 also performs camera vector calculation based on the acquired image on the narrow-angle field camera side. However, since the three-dimensional position coordinates are inaccurate, the wide-angle field image of the wide-angle field camera is used. The camera vector is calculated by substituting the three-dimensional coordinates of the camera vector values obtained by the above into the three-dimensional coordinates obtained from the narrow-angle field image of the narrow-angle field camera.
The camera vector of the narrow-angle viewing camera obtained in this way has a sufficiently large number of pixels per viewing angle compared to the wide-angle viewing camera, so that a highly accurate coordinate axis rotation angle can be obtained.
If necessary, the position accuracy can be further improved by substituting the high-precision coordinate axis rotation angle obtained here as the coordinate rotation angle on the wide-angle viewing camera side.

The wide-angle lens camera and the narrow-angle lens camera are integrally arranged by being fixed so that the positional relationship does not change. Wide-angle and narrow-angle cameras are driven synchronously.
The video output (video output (1), video output (2)) of each camera is recorded by the recording unit 12. In the case of image stabilization as post-processing, the video output is compressed by AVI or the like, and all frames are recorded in the storage. In addition, when image stabilization is performed as camera attitude control in real time, or for the purpose of position control and attitude control of the camera holding mechanism and the vehicle itself, it is necessary for real-time processing rather than recording images of all frames. It suffices to record only the number of frames. However, if the video itself is required, all necessary images are recorded.

The video outputs of the wide-angle lens camera and the narrow-angle lens camera are each temporarily stored, and the camera vector calculation device 10 calculates the camera vector. Are the two camera vectors basically the same or simply translated? The position is rotated by a constant number, and if one is obtained, it is the other.
Here, since the camera vector obtained by the wide-angle lens camera has a wide field of view, the feature points can be tracked over a long distance, and the three-dimensional position accuracy can be high. On the other hand, the video output of a narrow-angle lens camera has a narrow field of view and the tracking distance of feature points is shortened, so that the three-dimensional position accuracy cannot be obtained, but since the angle of view per pixel is small, the rotation accuracy can be high.

Therefore, by combining the advantages of both cameras, the three-dimensional position coordinates are obtained by calculation on the wide-angle lens side, and the obtained three-dimensional position coordinates are adopted as the three-dimensional position coordinates of the narrow-angle lens camera. By obtaining the camera vector calculation at, a highly accurate three-axis rotation signal can be obtained.
Furthermore, if necessary, it is possible to calculate a more accurate three-dimensional position by substituting the three-axis rotation signal as the three-axis rotation angle on the wide-angle lens camera side. By repeating this, the accuracy can be further improved.

As described above, after a highly accurate three-dimensional position and three-axis rotation are obtained by real-time processing, the three-dimensional position and posture of a camera holding mechanism, a vehicle carrying the camera, etc. are controlled using this as a posture control signal. Can do. However, in the case of real-time processing, there is no need for AVI image conversion for the purpose of camera vector calculation.
In addition, for the actual calculation of image stabilization of the recorded image, the signal from the shake component signal is processed, but the image pasted on the spherical surface is rotated by the shake component signal as in the case of handling 360 ° video. This can be realized by a correction method, or by converting a normal flat image that is not a spherical surface like a normal image into a plane so that it is rotated in accordance with the swing angle and flattened. Actually, either a spherical transformation or a planar transformation is possible for a planar image or a spherical image. FIG. 17 shows two cases of using a spherical transformation equation and using a planar transformation equation.

FIG. 18 shows the relationship between the wide-angle camera and the narrow-angle camera.
As shown in the figure, each camera performs feature point extraction and feature point tracking in each field of view, so the fields of view do not necessarily have to overlap.
It is only necessary that the positional relationship of each camera is fixed.
Since the position coordinates of the camera vector obtained by one camera are relative values, it is necessary to perform some scale calibration later.However, if the same feature point can be tracked by overlapping the field of view of the camera, the parallax can be reduced. Since measurement is possible, the absolute distance can be obtained, which is advantageous for measurement and accuracy is further improved.
FIG. 19 shows a case where the fields of view of the cameras are overlapped.

As shown in FIG. 19, when overlapping the fields of view of the cameras, it is preferable to install a dedicated field-of-view overlapping camera. As shown in the figure, the three-dimensional position accuracy is improved by arranging the narrow-angle field-of-view cameras so that they overlap in the expected traveling direction of a moving body such as a vehicle and extending the tracking path of feature points. Can be made.
Further, the narrow-angle field-of-view camera of the horizontal independent field shown in the figure is set so as to have high sensitivity at the three-axis rotation angle.
When such a field-of-view overlap camera is used for the image stabilization device 30, the image is stabilized by subjecting the image to rotation correction using the final output 3-axis rotation vector of the camera vector calculation device 10 as a correction signal.
Note that the fields of view of the wide-angle lens and the narrow-angle lens do not necessarily overlap.

  Further, when obtaining a highly accurate camera vector, as shown in FIG. 20, a plurality of narrow-angle view cameras and a wide-angle view camera are arranged along the traveling direction of a vehicle or the like having a rough view as shown in FIG. In this case, the field of view is overlapped, and the entire field of view is grasped with a wide-angle field-of-view camera. By tracking feature points in the direction, it is possible to realize high-precision camera vector calculation for position and orientation. The narrow-angle view cameras at the left and right ends shown in FIG. 20 are installed so as to be highly sensitive to the three-axis rotation angle.

In addition, as shown in FIG. 21, a plurality of wide-angle field-of-view cameras that grasp the entire field of view can be provided.
The example shown in FIG. 21 is a case where a plurality of (two) wide-angle field-of-view cameras are installed with overlapping fields of view when the distance from the ground approaches, such as when an aircraft takes off and landing.
At the time of takeoff and landing of the aircraft, the ground speed increases as it approaches the runway, so that it is difficult to detect and track feature points, and a wide-angle visual field camera is installed laterally outward as shown in FIG.
In addition, since the absolute distance can be measured if there is an overlapping field of view as described above, the relative distance obtained by the camera vector calculation can be calibrated and converted into the absolute distance.

As described above, the camera vector arithmetic device of the present invention and the shake component detecting device, the image stabilizing device, the position / orientation stabilizing device, the target object lock-on device, and the live-action object attribute calling device provided in the camera vector arithmetic device, Although the preferred embodiment has been shown and described, it is needless to say that the camera vector calculation device according to the present invention is not limited to the above-described embodiment, and various modifications can be made within the scope of the present invention. .
For example, as shown in FIG. 22, a feature point is replaced with a radio wave source, a camera is replaced with an antenna, and a camera vector is replaced with an antenna vector, thereby detecting a plurality of radio wave sources, specifying their directions, By tracking the direction, the antenna vector can be detected.

Since the feature point is information of only the direction in the image, it can be considered exactly the same as the image by detecting the direction of the radio wave source and tracking it.
In this case, the antenna direction becomes a feature point. If the antenna direction is tracked, it can be treated mathematically as the same feature point tracking as an image. Unlike the feature points on the image, it can be used at night, and broadcast stations and the like scattered on the ground can be used as radio wave sources.
The coordinates of the radio wave source are not necessary, but obtaining the absolute coordinates can be realized by capturing the radio wave source having three or more three-dimensional coordinates known. It can also be realized by using a plurality of antennas.

  The present invention is used, for example, as an image processing apparatus suitable for correcting an image caused by shaking or vibration of an in-vehicle camera that takes a video of a moving image or acquiring and generating a three-dimensional map provided in a car navigation apparatus. Can do.

It is a block diagram which shows the structure of the camera vector calculating apparatus which concerns on one Embodiment of this invention. It is explanatory drawing which shows the specific detection method of the camera vector in the camera vector arithmetic unit which concerns on one Embodiment of this invention. It is explanatory drawing which shows the specific detection method of a camera vector in the camera vector arithmetic unit which concerns on one Embodiment of this invention. It is explanatory drawing which shows the specific detection method of a camera vector in the camera vector arithmetic unit which concerns on one Embodiment of this invention. It is explanatory drawing which shows the designation | designated aspect of the desirable feature point in the detection method of the camera vector by the camera vector calculating apparatus which concerns on one Embodiment of this invention. It is a graph which shows the example of the three-dimensional coordinate of the feature point obtained by the camera vector arithmetic unit which concerns on one Embodiment of this invention, and a camera vector. It is a graph which shows the example of the three-dimensional coordinate of the feature point obtained by the camera vector arithmetic unit which concerns on one Embodiment of this invention, and a camera vector. It is a graph which shows the example of the three-dimensional coordinate of the feature point obtained by the camera vector arithmetic unit which concerns on one Embodiment of this invention, and a camera vector. FIG. 6 is an explanatory diagram illustrating a case where a plurality of feature points are set according to the distance from a camera to a feature point and a plurality of calculations are repeatedly performed in the camera vector calculation device according to the embodiment of the present invention. It is a figure at the time of displaying the locus | trajectory of the camera vector calculated | required with the camera vector calculating apparatus which concerns on one Embodiment of this invention in the produced | generated three-dimensional map. It is explanatory drawing which shows the specific example of the shake component detection in the shake component detection apparatus with which the camera vector calculating apparatus which concerns on one Embodiment of this invention is equipped. 1 is a block diagram showing a schematic configuration of a shake component detection device, an image stabilization device, a position and orientation stabilization device, a target object lock-on device, and a live-action object attribute call device provided in a camera vector calculation device according to an embodiment of the present invention. FIG. It is explanatory drawing which shows an example of the stabilization image correct | amended with the image stabilization apparatus which concerns on one Embodiment of this invention. It is a graph which shows the locus | trajectory of the camera vector correct | amended with the image stabilization apparatus which concerns on one Embodiment of this invention. It is a block diagram which shows the structure in the case of providing the camera for camera vector calculation and the camera for video acquisition independently in the camera vector calculation apparatus which concerns on one Embodiment of this invention. It is a block diagram which shows a structure in case the camera vector calculating apparatus which concerns on one Embodiment of this invention is equipped with several cameras of a wide-angle visual field and a narrow-angle visual field as a camera for camera vector calculations. FIG. 17 is a block diagram illustrating an example of a specific configuration of a camera vector arithmetic device including a plurality of cameras having a wide-angle field of view and a narrow-angle field of view illustrated in FIG. 16. It is explanatory drawing which shows the relationship between a wide-angle visual field camera and a narrow-angle visual field camera. FIG. 17 is an explanatory diagram in the case of installing a plurality of narrow-angle field-of-view cameras with overlapping fields of view along the moving direction of the camera in the camera vector computing device including a plurality of cameras with a wide-angle field of view and a narrow-angle field of view shown in FIG. 16. In the camera vector computing device including a plurality of cameras having a wide-angle field of view and a narrow-angle field of view shown in FIG. 16, the entire field of view is captured by the wide-angle field-of-view camera and the image is obtained by overlapping the field of view with the plurality of narrow-angle fields of view. It is explanatory drawing. FIG. 22 is an explanatory diagram in a case where a plurality of wide-angle field-of-view cameras that capture the entire field of view are installed in the camera vector computing device including a plurality of cameras with a wide-angle field of view and a narrow-angle field of view shown in FIG. 21. It is explanatory drawing which shows the case where the camera vector arithmetic device which concerns on this invention is applied to the antenna vector arithmetic device which detects the some antenna used as a radio wave source.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 Camera vector calculation apparatus 11 Image acquisition part 12 Image temporary recording part 13 Feature point extraction part 14 Feature point corresponding | compatible process part 15 Camera vector calculation part 16 Error minimization part 20 Shake component detection apparatus 30 Image stabilization apparatus 40 Position and orientation stabilization Device 50 Target object lock-on device 60 Live action object attribute calling device

Claims (14)

A feature point extraction unit for automatically extracting a predetermined number of feature points from the image data of the moving image;
About the extracted feature points, a feature point correspondence processing unit that automatically tracks within each frame image of the moving image and obtains a correspondence relationship between the frame images;
A camera vector calculation unit that obtains the three-dimensional position coordinates of the feature point for which the correspondence relationship is obtained, and obtains a camera vector composed of the three-dimensional position coordinates and the three-dimensional rotation coordinates of the camera corresponding to each frame image from the three-dimensional position coordinates. When,
A camera vector computing device comprising:   An error minimizing unit that performs statistical processing so as to minimize the distribution of solutions of a plurality of camera vectors obtained in the camera vector computing unit and automatically determines a camera vector subjected to error minimization processing. Item 2. The camera vector arithmetic device according to Item 1. The camera vector calculation unit
Using two arbitrary frame images Fn and Fn + m (m = frame interval) used for camera vector calculation as a unit image, the unit calculation for obtaining the three-dimensional position coordinates of the desired feature point and the camera vector is repeated,
For a frame image between the two frame images Fn and Fn + m, a camera vector is obtained by a simplified calculation,
The error minimizing unit includes:
As n progresses continuously with the progress of the image, the final camera vector is determined by adjusting the scale so that the error of each camera vector obtained by calculating multiple times for the same feature point is minimized. The camera vector arithmetic device according to claim 2. The camera vector calculation unit
4. The camera vector calculation device according to claim 3, wherein unit calculation is performed by setting the frame interval m such that m increases as the distance from the camera to the feature point increases according to the distance from the camera to the feature point. The camera vector calculation unit
5. A feature point having a large distribution of camera vector errors is deleted, and if necessary, the camera vector is recalculated based on another feature point to increase the accuracy of the camera vector calculation. The camera vector computing device described. The camera vector calculation unit
Perform calculations based on the minimum number of frame images with the desired accuracy and the minimum number of feature points automatically extracted, and obtain and display the approximate value of the camera vector in real time.
As the image accumulates as the image progresses, the number of frames is increased, the number of feature points is increased, more accurate camera vector calculation is performed, and the approximate value of the camera vector is replaced with a more accurate camera vector value. 6. The camera vector calculation device according to claim 1, wherein the camera vector calculation device is displayed.   The camera vector arithmetic device according to claim 1, wherein the moving picture image from which the feature points are automatically extracted by the feature point extraction unit is composed of a 360-degree all-round image. Wide-angle visual field images with a wide field of view acquired by a plurality of cameras whose positional relationships are fixed and acquired synchronously as video images from which feature points are automatically extracted by the feature point extraction unit, Input a narrow-angle visual field image,
While calculating the camera vector based on the image data of the wide-angle visual field image,
8. The camera vector calculation device according to claim 1, wherein a camera vector value based on the wide-angle visual field image is substituted for a camera vector calculation based on image data of the narrow-angle visual field image, thereby obtaining a highly accurate camera vector. Based on the three-dimensional information sequentially obtained as a part of the image, using the camera vector obtained by the camera vector calculation unit as an approximate camera vector, the partial three-dimensional information included in a plurality of frame images is converted between adjacent frames. 3D information tracking unit that continuously tracks with,
The camera according to claim 1, further comprising: a high-precision camera vector calculation unit that obtains a camera vector with higher accuracy than the approximate camera vector based on a tracking result of the three-dimensional information obtained by the three-dimensional information tracking unit. Vector arithmetic unit.   A deviation component between the camera vector obtained by the camera vector calculation unit and the scheduled camera vector indicating the camera position and the camera posture scheduled in advance is extracted, and the scheduled camera vector and the camera vector obtained at the present time or an arbitrary time point are extracted. A predetermined positional deviation component signal and a rotational deviation component signal are generated from the difference, a part or all of the deviation component signal is converted into an appropriate coordinate system according to the purpose, and the camera or a fixed object to which the camera is fixed is shaken. 10. The camera vector calculation device according to claim 1, further comprising a shake component detection device that outputs a vibration component. Based on the output of the shake component detection device, a correction signal for correcting and converting a camera vector obtained from an image of the camera shaking so as to match a reference camera vector assumed when the camera does not shake An image correction signal generation unit to be generated;
A stabilized image conversion unit that converts the image data into a stabilized image equivalent to a case where the image data is taken as a reference vector based on a correction signal of the image correction signal generation unit when the camera is assumed not to shake.
A stabilized image output unit that outputs the stabilized image converted by the stabilized image conversion unit;
A display unit for displaying the stabilized image output from the stabilized image output unit;
The camera vector arithmetic unit according to claim 10, further comprising an image stabilization device having Based on the output of the shaking component detection device, the difference between the planned camera position and posture and the current camera position and posture is obtained, and the position and posture of the control object to which the camera is fixed are controlled in three dimensions. A position / orientation correction signal generating unit that generates a correction signal for correcting the position and orientation to be planned
A control signal selection unit that selects an arbitrary correction signal from one or more correction signals generated by the position / orientation correction signal generation unit and outputs a control signal for controlling the position and orientation of the camera;
A control target drive unit that controls the position and orientation of the camera by driving the control target according to the control signal output from the control signal selection unit;
The camera vector arithmetic device according to claim 10, further comprising a position / orientation stabilization device having A live-action coordinate setting unit for setting an appropriate three-dimensional coordinate system in the live-action image;
A lock-on object designating unit for designating an arbitrary object to be locked on in the image;
Based on the camera vector obtained by the camera vector calculation device, the object specified in the image is measured in the live-action coordinate system to obtain its three-dimensional coordinates, and the specified object for which the three-dimensional coordinates are obtained is always displayed. A lock-on control unit that controls the position and posture of a fixed object to which an image display or camera is fixed, so that it is displayed at the center position or a predetermined position of the frame;
The camera vector arithmetic device according to claim 10, further comprising a target object lock-on device having   Based on the camera vector obtained by the camera vector calculation unit, an arbitrary object is specified in the image, or the specified object is locked on a predetermined position in the image, and the attribute, function, 14. The camera vector arithmetic device according to claim 1, further comprising a real-action object attribute calling device that calls and displays predetermined attribute information including content.