JP2013037006A - Information processor and information processing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a method for more stably measuring a position and posture of an imaging device with a high degree of accuracy by reducing an impact of falsely-detected index or an index with low detection accuracy.SOLUTION: An index detection part 2030 detects shot images of indices P1, P2 and P3 arranged or set on an object. An evaluation amount calculation part 2060 calculates an evaluation amount using two-dimensional geometric features on the images of the indices P1, P2 and P3 and/or a three-dimensional geometric feature showing each relationship between an imaging device 2060 and the indices P1, P2 and P3 in each three-dimensional space of the indices P1, P2 and P3. A reliability calculation part 2070 calculates each reliability of the indices P1, P2 and P3 corresponding to each calculated evaluation amount of the indices P1, P2 and P3. A position-and-posture calculation part 2080 obtains a position and posture of the object or the imaging device 2010 at least using each calculated reliability of the indices P1, P2 and P3 and information on each detected image coordinate of the indices P1, P2 and P3.

Description

  The present invention relates to a technique for improving accuracy and stability of viewpoint position and orientation measurement.

  In recent years, research on mixed reality has been actively conducted for the purpose of seamless fusion of real space and virtual space. The mixed reality image is generated by superimposing and drawing an image of a virtual space generated according to the position and orientation of the imaging device on an image of the real space taken by the imaging device such as a video camera. An image display device used in the mixed reality system is realized by, for example, a video see-through method. Here, the image in the virtual space is, for example, a virtual object or character information drawn by computer graphics.

  In order to realize mixed reality, it is important how to accurately align the real space and the virtual space, and many efforts have been made so far. The problem of alignment in mixed reality is the problem of obtaining the relative position and orientation (hereinafter referred to as the position and orientation of the imaging device) between the target object on which virtual information is to be superimposed and the imaging device. As a result.

  As a method for solving this problem, the following attempts have been made. That is, a plurality of indexes whose arrangement in the target coordinate system is known are set or set in the environment or on the target object. Then, the position and orientation of the imaging device with respect to the target coordinate system is obtained using the three-dimensional coordinates of the target in the target coordinate system, which is known information, and the coordinates of the projected image of the index in the image captured by the imaging device. (See Non-Patent Document 1).

  In addition, an attempt has been made to realize a stable alignment by attaching an inertial sensor to an imaging apparatus and using sensor measurement values, as compared with a case where only image information is used. For example, a method has been proposed in which the position and orientation of an imaging device estimated based on sensor measurement values are used for index detection processing. A method of using the estimation result as an initial value for position and orientation calculation based on an image has also been proposed. Furthermore, a method has also been proposed in which the estimation result is used as a rough position and orientation even in a situation where the index is not visible (see Patent Document 1 and Non-Patent Document 2).

  The conventional alignment method using image information is based on the premise that all detection results of indices are correct. Furthermore, all the index detection results were calculated as equal weights. For this reason, there is a possibility that a situation in which correct position and orientation measurement cannot be performed due to a large influence of erroneous detection or an index with low detection accuracy may occur.

  Therefore, a statistical estimation technique such as M estimation is introduced, and the image coordinates (reprojection coordinates) of the index estimated from the observation coordinates on the image of the detected index (feature point) and the position and orientation of the imaging device and the position of the index. ) (Reprojection error). In recent years, a method has been proposed in which the reliability of a detection index is calculated based on this error, and a false detection index is eliminated or its influence is reduced (see Non-Patent Document 3).

  In recent years, research on mixed reality has been actively conducted for the purpose of seamless fusion of real space and virtual space. An image display device used in a mixed reality system superimposes and draws a virtual space image generated according to the position and orientation of an imaging device on a real space image captured by an imaging device such as a video camera. This is mainly realized by the video see-through method for displaying the video. Here, the image in the virtual space is, for example, a virtual object or character information drawn by computer graphics.

  In order to realize mixed reality, it is important how to accurately align the real space and the virtual space, and many efforts have been made so far. The problem of alignment in mixed reality is the problem of obtaining the relative position and orientation (hereinafter referred to as the position and orientation of the imaging device) between the target object on which virtual information is to be superimposed and the imaging device. As a result.

  As a method for solving this problem, the following attempts have been made. That is, a plurality of indexes whose arrangement in the target coordinate system is known are set or set in the environment or on the target object. Then, the position and orientation of the imaging device with respect to the target coordinate system is obtained using the three-dimensional coordinates of the target in the target coordinate system, which is known information, and the coordinates of the projected image of the index in the image captured by the imaging device. (See Non-Patent Document 1).

  In addition, an inertial sensor is attached to the imaging device, and the position and orientation of the imaging device estimated based on the sensor measurement value are used for index detection processing, or the estimation result is used as an initial value for position and orientation calculation based on the image. In addition, even when the index is not visible, an attempt has been made to realize stable positioning compared to the case of using only image information by using the estimation result as a rough position and orientation (Patent Document) 1, see Non-Patent Document 2).

  The conventional alignment method using image information is based on the premise that all detection results of indices are correct. Furthermore, all the index detection results were calculated as equal weights. For this reason, there is a possibility that a situation in which correct position and orientation measurement cannot be performed due to a large influence of erroneous detection or an index with low detection accuracy may occur.

  Therefore, a statistical estimation technique such as M estimation is introduced, and the image coordinates (reprojection coordinates) of the index estimated from the observation coordinates on the image of the detected index (feature point) and the position and orientation of the imaging device and the position of the index. ) (Reprojection error). In recent years, a method has been proposed in which the reliability of a detection index is calculated based on this error, and a false detection index is eliminated or its influence is reduced (see Non-Patent Document 3).

JP 2005-33319 A

Sato, Tamura: “Alignment Method in Mixed Reality”, Image Recognition and Understanding Symposium (MIRU 2002) Proceedings I, IPSJ Symposium Series, vol. 2002, no. 11, pp. I. 61-I. 68, July 2002 Hirofumi Fujii, Noriyuki Kanbara, Hidehiko Iwasa, Haruo Takemura, Naokazu Yokoya, alignment with stereo camera combined with gyro sensor for augmented reality, IEICE Technical Report PRMU99-192 (Science Technical Report vol.99) No. 574, pp. 1-8) Sato, Kanbara, Yokoya, Takemura: “Restoring camera movement parameters from moving images by tracking markers and natural feature points”, IEICE Transactions, D-III, vol. J86-D-II, no. 10, pp. 1431-1440, 2003.

However, the reliability is calculated based only on the re-projection error statistic, and the weighting of the index eliminates false detection or reduces the influence of the index with low detection accuracy (accuracy of image coordinates to be detected). Is not necessarily effective. This is because the above-mentioned method is effective in correctly detecting a large number of indices and eliminating erroneously detected indices that appear exceptionally. This is because it may be affected by low indicators. In addition, even if the index is taken with the same image, there is a problem that the index with low detection accuracy may be included depending on the marker arrangement and the situation at the time of shooting. The present invention has been made in view of the above problems. That is, the present invention changes the weighting according to the evaluation amount peculiar to an index having a two-dimensional shape (circle marker, polygon marker, etc.), eliminates or reduces the influence of an index with low detection accuracy, and is highly accurate and more stable. The purpose is to measure the position and orientation of a simple imaging device.

  In order to achieve the above object, the present invention has the following configuration.

  According to the first aspect of the present invention, an image input step of inputting a captured image captured by an imaging device, an index detection step of detecting an index placed or set on an object from the captured image, and the captured image An index evaluation amount calculation step for calculating an evaluation amount to be used as a clue for calculating the reliability of each detected index, and the reliability of the index according to the evaluation amount of the index calculated in the index evaluation amount calculation step Using at least the reliability calculation step for calculating the degree, the reliability of the index calculated in the reliability calculation step, and the information on the image coordinates of each index detected in the index detection step, A position / orientation calculation step for obtaining a position / orientation of the imaging apparatus, wherein the evaluation amount is calculated based on a two-dimensional geometric feature on the image of the index and / or a front of the index in a three-dimensional space Characterized in that based on the three-dimensional geometric features representing the relationship between the index and the image pickup device.

  According to the present invention, since the influence of an index with low detection accuracy can be reduced, the position and orientation of the imaging apparatus can be measured with high accuracy and more stably.

It is a figure which shows schematic structure of the position and orientation calculation apparatus which concerns on 1st Embodiment. It is a flowchart which shows the schematic process of the position and orientation calculation method which concerns on 1st Embodiment. It is a figure which shows schematic structure of the parameter | index which concerns on 1st Embodiment. It is a figure which shows the parameter | index which concerns on 1st Embodiment, and the relative attitude | position of an imaging device. 10 is a flowchart showing a schematic process of a position / orientation calculation unit 2080 according to the first embodiment. FIG. 10 is a block diagram showing a hardware configuration of a computer applicable to the position / orientation measuring apparatus 2000.

  Hereinafter, the present invention will be described in detail according to preferred embodiments with reference to the accompanying drawings.

[First Embodiment]
The position / orientation measurement apparatus of the imaging apparatus according to the present embodiment performs weighting by changing the reliability in accordance with the detected evaluation amount of the index having a two-dimensional shape, and reduces the exclusion or influence of the index with low accuracy. Improve the accuracy of the position and orientation measurement results. Hereinafter, a method for measuring the position and orientation of the imaging apparatus by weighting according to the evaluation amount of the index according to the present embodiment will be described.

  FIG. 1 shows a schematic configuration of a position / orientation measurement apparatus 2000 of the imaging apparatus according to the present embodiment. The position / orientation measurement apparatus 2000 includes an image input unit 2020, an index detection unit 2030, an imaging device approximate position / orientation input unit 2040, a data management unit 2050, an evaluation amount calculation unit 2060, a reliability calculation unit 2070, and a position / orientation calculation unit 2080. Has been. The image input unit 2020 is connected to an imaging device 2010 that is a measurement target.

A plurality of indices are arranged in the environment or on the target object. Here, the index arranged on the object is represented by P ^k (k = 1,, Ko). Ko is the number of indexes arranged (K0 = 3 in the example of FIG. 3A). Further, the polygonal index P ^k as shown in FIG. 3B includes the vertex p ^ki (i = 1,, N _k ). N _k represents the total number of vertices constituting the index P ^k (in this embodiment, N _k = 4). Further, as shown in FIG. 3C, the index may be constituted by points p ^ki (i = 1,, N _k ) whose relative positional relationship between the vertices is known.

  In the following, a coordinate system used as a reference for measurement (here, a coordinate system in which one point in the environment is defined as the origin and three axes orthogonal to each other are defined as the X axis, the Y axis, and the Z axis) is referred to as the world coordinate system. It is assumed that the position and orientation of each index in the world coordinate system are known. Further, it is assumed that the position of each vertex constituting each index arranged in the environment is known in the world coordinate system.

The index P ^k is any index that can detect the screen coordinates of the projected image on the photographed image and is an index that can identify each vertex constituting the index. It may be a form. For example, a rectangular index as shown in FIGS. 3A and 3B has a pattern representing an identifier inside and can be uniquely identified. When such an index is detected, a binarization process is performed on the captured image, and then a labeling process is performed, and an area formed by four straight lines is extracted as an index candidate from a region having a certain area or more. . Further, it is determined whether or not the candidate area is an index area by determining whether or not there is a specific pattern in the candidate area. If it is determined as the index region, the index direction and identifier are obtained by reading the internal pattern.

  An image captured by the imaging device 2010 is input to the index detection unit 2030 via the image input unit 2020.

The index detection unit 2030 receives an image from the image input unit 2020 and detects the image coordinates of each vertex p ^kni constituting the index P ^kn captured in the input image.

Furthermore, the index detection unit 2030 identifies (identifies) each detected index P ^kn .

Then, the input position of the index in the world coordinate system posture M _WM and world coordinates x _w ^Pkni of each vertex ^(held in advance as known information about the index) from the data management unit 2050.

The list with identifiers k _n and the position and orientation _{M WM} in the world coordinate system of the detected respective indices, a set of image coordinates ^{u Pkni} and world coordinates _x ^{w Pkni} of each vertex ^{p kni} as elements (hereinafter, the data list Is created and output to the data management unit 2050. Here, n (n = 1,..., N) is an index for each detected index, and N represents the total number of detected indices. Further, the total number of constituent points (vertices) made up of N indices is N _Total . For example, in the case of FIG. 1, N = 3 because a case where a square index with identifiers 1, 2, and 3 is photographed is shown. Also, identifiers k ₁ = 1, k ₂ = 2 and k ₃ = 3, the position and orientation of these indices, and the image coordinates u ^pk1i , u ^pk2i , u ^pk3i , (i = 1, 2, 3, 4) of each vertex. ) And world coordinates x _w ^pk1i , x _w ^pk2i , x _w ^pk3i , (i = 1, 2, 3, 4). N _Total is 12 (= 3 × 4).

Here, t is a three-dimensional vector representing the position of the three-dimensional coordinate system A with respect to a certain three-dimensional coordinate system B, and R is a 3 × 3 rotation matrix representing the posture. In this case, the coordinate x _B (three-dimensional vector) on B of the point whose position on _A is represented by x _A (three-dimensional vector) uses a 4 × 4 matrix M _BA shown in the following Equation 1. The following expression 2 is expressed by the homogeneous coordinate expression.

In the present embodiment, _MBA is used as means for representing the position and orientation of the coordinate system A relative to the coordinate system B.

The imaging apparatus approximate position / orientation input unit 2040 obtains an approximate value M _WC of the position / orientation of the imaging apparatus 2010 in the world coordinate system, and outputs this to the data management unit 2050. As the approximate position and orientation of the imaging apparatus, for example, a 6-DOF position and orientation sensor may be attached to the imaging apparatus 2010 and the output value thereof may be used. In addition, the attitude of the imaging apparatus 2010 is measured by mounting the attitude sensor of 3 degrees of freedom on the imaging apparatus 2010, the data list obtained by the index detection unit 2030 is input from the data management unit 2050, and obtained based on these information. May be. Alternatively, a method is used in which the data list obtained by the index detection unit 2030 is input from the data management unit 2050, and the position and orientation of the imaging apparatus is estimated by solving a linear simultaneous equation obtained from the correspondence between the image coordinates and the world coordinates. You may ask. Any other method may be used to obtain the approximate position and orientation of the imaging apparatus and use them as initial values.

The data management unit 2050 holds and manages the data list and outputs it as necessary. Further, as known information regarding the index, the position and orientation M _WM of the index in the world coordinate system and the world coordinates x _w ^pkni of each vertex are ^stored in advance. Further, the approximate position and orientation M _WC of the imaging apparatus 2010 in the world coordinate system obtained by the imaging apparatus approximate position and orientation input unit 2040 is input and held, and is output as necessary.

The evaluation amount calculation unit 2060 receives, from the data management unit 2050, the approximate position and orientation of the imaging device 2010 in a certain frame and a data list of indices detected in the same frame. Then, an evaluation amount V ^Pkn is calculated for each index that is an element of the data list, and the obtained evaluation amount is stored in the data list as a parameter of each index.

Here, details of evaluation amount calculation in the evaluation amount calculation unit 2060 according to the present embodiment will be described. First, a collinear vector that connects the index and the viewpoint of the imaging apparatus 2010 based on the center of gravity position of the index in the world coordinate system and the approximate position t _WC of the imaging apparatus 2010

Ask for. Here, the barycentric position of the index in the world coordinate system may be stored in advance as a known value related to the index, or may be obtained as an average value of the world coordinates of each vertex of the index.

  Then, based on the world coordinates of each vertex of the index, the normal vector of the index in the world coordinate system

Ask for. This value may be held in advance as a known value related to the index. In addition, the normal vector of this indicator

And collinear vector

^Is calculated based on the relationship of Equation (7) (the geometrical relationship is shown in FIG. 4A).

Finally, the evaluation value V ^Pkn = θ ^Pkn is stored in the data list. The evaluation amount thus obtained is a value representing how much each index is captured from the front of the camera.

The reliability calculation unit 2070 inputs the data list from the data management unit 2050, and for each index in the data list, the index value is calculated based on the evaluation value V ^Pkn of each index obtained by the evaluation value calculation unit 2060. The reliability ω ^Pkn is calculated. Then, the obtained evaluation amount is stored in the data list as a parameter of each index.

Here, details of reliability calculation in the reliability calculation unit 2070 according to the present embodiment will be described. The index reliability ω ^Pkn is a weighting function ^{using the} index evaluation amount V ^Pkn as an argument.

Calculated by The weight function ω (V ^Pkn ) is positive (> 0). Such a function may be a weighting function (a function that creates a probabilistic model and applies a larger weighting to what fits well with observation data) used in M estimation, which is one of robust estimation methods, It may be a function obtained experimentally or empirically.

  here,

When

The angle θ ^Pkn formed by changes in the range from 0 ° to 90 °. If θ ^Pkn = 90 °, the planar shape index cannot be detected, so the reliability is set to zero. Further, if θ ^Pkn is around 90 °, the accuracy of detection of the observation coordinates of each vertex of the index is lowered, so that a small reliability is given. When ^θPkn is around 0 °, the detection accuracy is high, so that high reliability is given. Thus, the reliability ω ^Pkn is defined by the function ω (θ ^Pkn ) of the evaluation value θ ^Pkn that becomes 0 when θ ^Pkn = 90 ° and becomes maximum when θ ^Pkn = 0 °.

  The position / orientation calculation unit 2080 receives the data list and the approximate position / orientation of the imaging apparatus 2010 from the data management unit 2050. Then, the position / orientation calculation processing of the imaging apparatus 2010 is performed based on the initial position / orientation of the imaging apparatus in a certain frame, the identifier and reliability of each index detected in the same frame, and the image coordinates and world coordinates of each vertex. . Then, the position / orientation information of the imaging apparatus 2010 obtained as a result (that is, the position and orientation of the imaging apparatus in the world coordinate system) is output.

  FIG. 2 is a flowchart of processing performed when the position / orientation measurement apparatus 2000 according to the present embodiment obtains position / orientation information.

  In step S6010, the index detection unit 2030 executes index detection processing on the input image, and outputs a data list generated as a detection result to the data management unit 2050.

In step S6020, the imaging apparatus approximate position / orientation calculation unit 2040 calculates the approximate position / orientation M _WC of the imaging apparatus 2010 in the world coordinate system at the same time as the imaging time of the input image, and inputs it to the data management unit 2050.

In step S6030, evaluation quantity calculating unit 2060 inputs the approximate position and orientation _{M WC} of the image capturing apparatus 2010 from the data management unit 2050, calculates an evaluation amount ^{V Pkn} indices of each of the data list.

In step S6040, the reliability calculation unit 2070 calculates the reliability ω ^Pkn based on the evaluation value V ^Pkn for each index in the data list.

In step S6050, the position and orientation calculation unit 2080 inputs the data list of the approximate position and orientation information _{M WC} and detected index of the image pickup device 2010 to the data management unit 2050 holds. Based on this, the position / orientation calculation processing of the imaging apparatus 2010 is performed, and the position / orientation information of the imaging apparatus 2010 obtained as a result is output.

  Finally, in step S6060, it is determined whether or not to end the position / orientation calculation process. When the operator instructs the position / orientation calculation apparatus 2000 to end the position / orientation calculation process, the process ends. When the operator instructs the continuation of the position / orientation calculation process, the process returns to step S6010 again. Processing for the input image of the next frame is executed.

  Next, the details of the processing of the position / orientation calculation unit 2080 (step S6050 in FIG. 2) will be described with reference to the flowchart shown in FIG.

  In step S4010, the position / orientation calculation unit 2080 obtains an initial position / orientation of the imaging apparatus in a certain frame and a data list (identifier of each index, reliability, image coordinates and world coordinates of each vertex detected in the same frame), Input from the data management unit 2050.

In step S4020, the position / orientation calculation unit 2080 determines whether or not the input index information includes sufficient information for position and orientation estimation, and branches the process accordingly. Specifically, if the total number N _Total of the input indices is 3 or more, the process proceeds to step S4030. If it is less than 3, the process proceeds to step S4090. Regarding N _Total , for example, if there is one index composed of four vertices of a square, N _Total is 4, and if there are two indices composed of three vertices of a triangle, N _Total is 6.

The position / orientation calculation unit 2080 treats the position / orientation of the imaging apparatus 2010 in the world coordinate system (or an arbitrary coordinate system) as an unknown parameter to be obtained. Here, a ternary vector a = [ξ ψ ζ] ^T is used as an orientation expression method. “a” is a method of expressing the posture by the rotation axis and rotation angle, and the rotation matrix R is a function of “a” as shown in the following equation.

At this time, the position and orientation of the imaging apparatus to be calculated are expressed by a position t = [x y z] ^T and an orientation a = [ξ ψ ζ] ^T. The unknown parameter to be obtained is described as a six-valued state vector s = [x yz ξ ψ ζ].

In step S4030, the position / orientation calculation unit 2080 calculates an estimated value (reprojection coordinate) u ^{Pkni ′} of the image coordinates for the vertex p ^kni of each index. calculation of u ^{Pkni 'is,} of vertex ^{p kni} of the world coordinates _x ^{w Pkni} and the current state vector s of the indicator function (observation equation)

Based on.

Specifically, the function F _C () is ^obtained by the following equation for ^{obtaining a} position vector x _C ^{Pkn i} in camera coordinates from x _w ^Pkni.

And the following equation for ^obtaining the coordinate u ^{Pkn i ′} on the image from x _C ^Pkn i

It is constituted by. Here, f ^B _x and f ^B _y are the focal lengths of the imaging apparatus 2010 in the x-axis direction and the y-axis direction, respectively, and are held in advance as known values.

In step S4040, the position and orientation calculation unit 2080, the measured values ^{u Pkni} vertices ^{p kni} of each index, the error (re-projection error) Delta] u ^Pkni the computed value ^{u Pkni 'image} coordinates corresponding to it by the following formula calculate.

In step S4050, the position / orientation calculation unit 2080 calculates an image Jacobian J _us ^Pkni (= ∂u ^Pkni / ∂s) related to the state vector s for each index vertex p ^kni . Here, the Jacobian is a Jacobian matrix of 2 rows × 6 columns having a solution obtained by partial differentiation of the function F _c () of Expression 12 with each element of the state vector s.

Specifically, a 2-row × 3-column Jacobian matrix J _ux ^{Pkn i} (= ∂u ^Pkn / ∂x) having a solution obtained by varying and differentiating the right side of Equation 14 with each element of camera coordinates x _C ^Pkni , A Jacobian matrix J _xs ^Pkni (= ∂x / ∂s) of 3 rows × 6 columns having a solution obtained by partial differentiation of the right side of Equation 13 with each element of the vector s is calculated, and J _us ^Pkni is calculated by the following equation: calculate.

In step S4060, the position / orientation calculation unit 2080 determines the state based on the error Δu ^Pkn of each index calculated in steps S4040 and ^S4050 , the image Jacobian J _us ^Pkni , and the reliability ^ωPkn of each index input in step S4010. A correction value Δs of the vector s is calculated. Details of the correction value Δs calculation process will be described below.

First, a 2N _Total- dimensional error vector U in which the reprojection errors at the vertices of each index are arranged vertically is created.

Also, a matrix Φ of (2N _Total ) rows × 6 columns in which the Jacobian matrix J _us ^Pkni at the vertices of each index is arranged vertically is created.

Next, 2N such that the reliability corresponding to each index p ^kni of the index P ^kn (having two elements of x-coordinate and y-coordinate for each vertex) has the reliability ω ^{Pkn of the} index as a diagonal component. Create a diagonal matrix W of _Total rows x 2N _Total columns.

  Considering that Δs is obtained by the least square method with the matrix W as a weight, the following normal equation

Therefore, Δs is calculated from the following equation.

  In this way, by using the matrix W representing the reliability based on the evaluation amount at the time of detection specific to the two-dimensional index as a weight for calculating Δs, the Δs of each index is calculated according to the evaluation amount of the index at the time of detection. It is possible to obtain an effect that the degree of contribution to is changed. The evaluation amount at the time of detection peculiar to the two-dimensional index is here the relative posture between the imaging device and the index. That is, it is possible to obtain an effect of reducing an adverse effect received from an index that is highly likely to be a reliable index by actively using an index that is highly likely to be low in detection accuracy.

Note that a weight corresponding to the reprojection error Δu ^Pkn of the index is obtained by a statistical estimation method such as M estimation, and a product with ω ^Pkn may be obtained and used as the weight.

Here, since Δs is a six-dimensional vector, if 2M _Total is 6 or more, Δs can be obtained. Although Δs can be obtained as shown in Equation 21, it may be obtained by other methods. For example, since Equation 20 is a simultaneous linear equation, Δs may be solved by the Gauss elimination method or any other method.

  In step S4070, the position / orientation calculation unit 2080 corrects s according to the following equation 22 using the correction value Δs calculated in step S4060, and sets the obtained value as new s.

  In step S4080, the position / orientation calculation unit 2080 converges the calculation using some criterion such as whether the error vector U is smaller than a predetermined threshold or whether the correction value Δs is smaller than the predetermined threshold. It is determined whether or not. If not converged, the processing after step S4030 is performed again using the corrected state vector s.

  If it is determined in step S4080 that the calculation has converged, in step S4090, the position / orientation calculation unit 2080 outputs the position and orientation s of the imaging device in the world coordinate system. The form of output at this time may be s itself, or the position component of s may be represented by a ternary vector, and the posture component may be represented by an Euler angle or a 3 × 3 rotation matrix. The coordinate transformation matrix M generated from s may be used.

  With the above processing, the position or position and orientation of the imaging device with respect to the world coordinate system (that is, in the world coordinate system) can be acquired. As described above, according to the position / orientation measurement apparatus according to the present embodiment, the position / orientation estimation is performed in consideration of the characteristic evaluation amount of the index having a two-dimensional shape, thereby reducing the influence of the index having low detection accuracy. The position and orientation of the imaging device can be measured with high accuracy.

[Second Embodiment]
In the first embodiment, the reliability ω ^Pkn is determined based on the evaluation amount based on the relative posture between the imaging device and the index. However, if the reliability is calculated based on the evaluation amount using the characteristic of the geometric shape of the index having a two-dimensional shape, the type of information used as the evaluation amount is not limited to this. That is, the reliability calculation method may be other methods or a method combined with other methods. For example, two-dimensional geometric information (for example, area, aspect ratio, etc.) regarding the index can be obtained from the image, and the reliability can be calculated using this as an evaluation value.

For example, the reliability ω ^Pkn may be calculated using the area S ^Pkn of the detected index on the image as the evaluation value V _S ^Pkn . In this case, the imaging apparatus approximate position and orientation input unit 2040 is not necessary.

The evaluation amount calculation unit 2060 calculates the area S ^Pkn in the image of the index region detected on the image, using the image coordinates p ^kni of each vertex of the index held in the data list, and uses this for each index. The evaluation amount. The area of the index may be calculated by inputting a labeling image obtained in the index detection process of the index detection unit 2030 from the index detection unit 2030 to the evaluation amount calculation unit 2060 and counting the number of pixels in the index region. . Here, the labeling image is an image in which, for example, pixel values of pixels other than the index region are set to 0, and pixel values of pixels corresponding to the index regions are set to the identification numbers of the respective index regions on the image.

The reliability calculation unit 2070 uses a weighting function ω _S (V _S ^Pkn ) that gives high reliability when the area S ^{Pkn of} the index detected on the image is large and low reliability when the area S ^Pkn is small. The reliability of each index is calculated. The determination of the size of the area ^SPkn may be performed using variance or standard deviation, or may be obtained experimentally or empirically.

^Further , the reliability ω _S ^Pkn based on the area of the index obtained in this way is not only used as the reliability ω ^Pkn as it is, but also by obtaining a product or average with the reliability obtained by other methods. A combined reliability may be obtained and used.

<Modification 1>
In the reliability calculation method obtained in the second embodiment, when the index is a square index, the reliability ω ^Pkn may be calculated using the aspect ratio of the side detected on the image as the evaluation value V _ratio ^Pkn. .

The evaluation amount calculation unit 2060 uses the image coordinates p ^kni of each vertex of the index held in the data list, for example, “apparent short side” for two adjacent sides existing for each vertex. The ratio of length is obtained by “/ apparent long side”. Then, an evaluation amount is obtained by selecting the minimum value among them. For example, when the index is a square index as shown in FIG. 3, the reliability calculation unit 2070 gives a high reliability to an index whose evaluation value is close to 1, and gives a low reliability to an index whose evaluation value is close to 0. The reliability of each index is calculated using a simple weight function ω _ratio (V _ratio ^Pkn ).

In addition, the reliability ω _ratio ^Pkn based on the aspect ratio of the detected side thus obtained is used as the reliability ω ^Pkn as it is, and the product or average of the reliability obtained by other methods is obtained. It is also possible to obtain the reliability obtained by combining these and use this.

<Modification 2>
In the reliability calculation method obtained in the second embodiment, when the index is a square index, the angle between two adjacent sides of the index detected on the image is set as the evaluation value V _angle ^Pkn , and the reliability ω ^Pkn. May be calculated.

Since a combination of two adjacent sides exists for each vertex, for example, an angle formed between the two sides is obtained, and the minimum value of these is used as the evaluation value. For example, a function ω _angle (V _angle ^Pkn ) is defined as a weighting function that gives a high reliability when the angle between adjacent sides of the quadrangle index is around 90 °, and a small reliability when it is around 0 °. To do.

^Further , the reliability ω _angle ^Pkn based on the angle between two adjacent sides in the index obtained in this way is not only used as the reliability ω ^Pkn as it is, but also the product of the reliability obtained by other methods, You may obtain the reliability which combined these by calculating | requiring an average, and may use this.

<Modification 3>
In the reliability calculation method obtained in the second embodiment, when a circular index having a circular plane shape is used as an index, the roundness of the index detected on the image is used as the evaluation value V _circle ^Pkn. ω ^Pkn may be calculated. In this case, a labeling image is output from the index detection unit 2030 to the evaluation amount calculation unit 2060.

  Roundness is an area detected as a projected image of a circular index (this area has a shape close to a perfect circle when the circular index is photographed from the front, and an ellipse when the circular index is photographed from an oblique direction. This is the amount of deviation from a perfect circle. The circularity can be defined by, for example, a difference in radius between the circumscribed circle and the inscribed circle in the detection area.

The reliability calculation unit 2070 calculates this r for each index region, and ^sets this as the evaluation value V _circle ^Pkn . The reliability calculation unit 2070 uses a weight function ω _circle (V _circle ^Pkn ) that gives high reliability to an index with a small r and low reliability to an index with a large r. Calculate reliability.

^Further , the reliability ω _circle ^Pkn based on the roundness of the circular marker thus obtained is not only used as the reliability ω ^Pkn as it is, but also the product or average of the reliability obtained by other methods is obtained. It is also possible to obtain the reliability obtained by combining these and use this.

<Modification 4>
In the reliability calculation method obtained in the first embodiment and the second embodiment, when a plurality of types of indicators are used together, the reliability may be defined according to the type of each indicator. Here, the plurality of indices are, for example, a polygonal index having ID information therein, a circular index composed of a monochrome region that cannot be identified from the image information (the above indices are artificial indices), It is a natural feature index defined by natural features such as edges and textures.

  For example, a high degree of reliability is given to a polygonal index having ID information that is less prone to erroneous detection and identification errors than a circular index or a natural feature index. A medium index is given to a circular index that is relatively less likely to be erroneously detected than natural features. The low reliability is given to the natural feature index having the highest possibility of false detection. Then, the position / orientation calculation based on the formula 21 may be performed according to the reliability.

  In addition to using the reliability based on the type of index obtained in this way as it is, it is also possible to obtain the product or average with the reliability using characteristics of the geometric shape of the index obtained by other methods, for example. You may obtain the reliability which combined these and use this.

<Modification 5>
In the reliability calculation method obtained in the first embodiment and the second embodiment, when the detected index has an identifier and error correction information for correcting a recognition error, whether or not the error is corrected is evaluated. The reliability ω ^Pkn may be calculated as V _IDmdf ^Pkn . Since the error correction of the index is performed when information is lost for some reason, a low reliability is given to the index for which error correction has been performed, and conversely a high reliability is given to an index for which error correction is not performed. The function ω _IDmdf (V _IDmdf ^Pkn ) is defined as a weight function.

^Further , the reliability ω _IDmdf ^Pkn based on the presence or absence of error correction obtained in this way is not only used as the reliability ω ^Pkn as it is, but also by obtaining a product or average with the reliability obtained by other methods. This may be used after obtaining the reliability obtained by combining the two.

<Modification 6>
In the reliability calculation method obtained in the first embodiment and the second embodiment, the reliability ω ^Pkn may be calculated using the contrast strength of the index detected on the image as the evaluation value V _C ^Pkn . The contrast C represents the difference between light and dark in the image. The contrast C is defined by the following equation, where I _{min is} the minimum value of the shading level in the image and I _max is the maximum value.

The contrast C is examined in the peripheral area of the index, and the value is set as the evaluation value V _C ^Pkn . A function ω _C (V _C ^Pkn ) is defined as a weighting function that provides high reliability because the detection accuracy is high for an index with high contrast and low reliability for an index with low contrast because the detection accuracy is low. To do.

^Further , the reliability ω _C ^Pkn based on the contrast thus obtained is used not only as the reliability ω ^Pkn as it is, but also by combining these by obtaining a product or average with the reliability obtained by other methods. This may be used with confidence.

<Modification 7>
In the reliability calculation method obtained in the first and second embodiments, the reliability ω ^Pkn may be calculated using the sharpness of the index detected on the image as the evaluation value V _sharp ^Pkn . The method of measuring the sharpness may be a method of measuring the slope of the edge, a method of measuring the spatial frequency component of the image, or any other method. A function ω _sharp (V _sharp ^Pkn ) is defined as a weighting function that gives high reliability because the edge can be detected clearly when the sharpness around the index is high, and conversely when the sharpness is low. To do.

^Further , the reliability ω _sharp ^Pkn based on the sharpness thus obtained may be used as the reliability ω ^Pkn as it is, or these are combined by obtaining a product or an average with the reliability obtained by other methods. This may be used after obtaining a high degree of reliability.

<Modification 8>
In the first embodiment, the imaging device approximate position and orientation input unit 2040 performs the following processing. That is, the approximate value M _WC of the position and orientation of the imaging device 2010 in the world coordinate system is obtained, and the relative orientation between the imaging device 2010 and the index is obtained from the approximate value M _WC and an index whose position or orientation in the world coordinate system is known. . However, when an index having four or more feature points is used, the following may be used. That is, determine the approximate position and orientation M _MC metrics and imaging apparatus 2010 from the image coordinates of the indices detected on the image based on the 2D homography calculation, based on this, from the normal vector and the imaging apparatus 2010 of the index You may calculate the angle | corner with the visual axis to a parameter | index.

  Specifically, based on the obtained index and the approximate relative position of the imaging device 2010, the joint between the index in the coordinate system based on the imaging device (hereinafter referred to as the imaging device coordinate system) and the viewpoint of the imaging device 2010 is shared. Line vector

Ask for. Then, based on the coordinates of the vertices of the index in the imaging device coordinate system (can be calculated from M _MC), the normal vector of the index in the imaging device coordinate system

Ask for. As described above, the normal vector of the indicator

And collinear vector

After that, processing similar to that in the first embodiment may be performed.

^Further , the reliability ω _θ ^Pkn based on the relative attitude between the imaging device 2010 and the index obtained in this way may be used as the reliability ω ^Pkn as it is. Moreover, the reliability which combined these by calculating | requiring the product and average with the reliability calculated | required with the other method may be used, and this may be used.

<Modification 9>
In the second embodiment, the two-dimensional geometric information of the index is obtained from the image, and the reliability is calculated using this as an evaluation amount. However, the estimated coordinates (reprojected coordinates) of each vertex of the index may be obtained based on the approximate position and orientation _MWC of the imaging apparatus, and the reliability may be calculated based on the two-dimensional geometric information of the reprojected coordinates.

If the approximate position and orientation M _WC of the imaging apparatus 2010 is not greatly deviated from the actual position and orientation (true value), the reliability can be calculated using the reprojected coordinates instead of the coordinates detected on the image. It goes without saying that the essence is not impaired.

<Modification 10>
In the first embodiment and the second embodiment, the position and orientation of the imaging apparatus are obtained assuming that the position of the index is known. However, the position or orientation of the index may be obtained together with the position and orientation of the imaging apparatus. .

At this time, the position or position / orientation information to be obtained is the position / orientation of the imaging apparatus 2010 in the world coordinate system and the position or position / orientation of the index in the world coordinate system. The position of the index in the world coordinate system is _{treated as a} ternary vector [x _Pkn y _Pkn z _Pkn ], or the position and orientation is treated as a six-value vector [x _Pkn y _Pkn z _Pkn ξ _Pkn ψ _Pkn ζ _Pkn ] ^T , and this unknown parameter is in the state It is described as a vector s _Pkn .

An appropriate initial value is given to the state vector s _Pkn . As an initial value, an operator may manually input an approximate value in advance. Further, detection coordinates of a plurality of indices input at a certain same time (that is, detected from a captured image) are extracted from the data list, and using this data, the position of the indices in the world coordinate system at the same time or The position and orientation are calculated by a known method. Here, three or more known points in the world coordinate system are required.

In addition, the position and orientation of the imaging apparatus in the world coordinate system is treated as a six-value vector [x _WC y _WC z _WC ξ _WC ψ _WC ζ _WC ] ^T. This unknown parameter is described as a state vector s _WC . A state vector of the position or position / orientation to be obtained is described as s = [s _WC s _Pkn ].

  In addition, as a known method for calculating the position or position and orientation of an index in the world coordinate system, for example, when the indices are arranged on the same plane, calculation of two-dimensional homography using four or more indices A method for obtaining the position or orientation of the index based on the above can be used. It is also possible to use a technique that uses six or more indices that are not on the same plane, or a technique that uses these solutions as initial values and obtains an optimal solution by iterative calculations such as the Newton method.

Here, the position / orientation calculation unit 2080 calculates an estimated value (reprojection coordinate) u ^{Pkni ′} of the image coordinates for the vertex p ^kni of each index. calculation of u ^{Pkni 'is,} of vertex ^{p kni} of the world coordinates _x ^{w Pkni} and the current state vector s of the indicator function (observation equation)

Based on.

  Subsequent processing is calculated in the same manner as in the first embodiment. That is, as in the first embodiment, the reliability is obtained, the matrix W having the reliability as a diagonal component is used as a weight, and calculation is performed by the least-squares method considering the weight, so that the position or orientation of the index and the imaging The position and orientation of the device can be obtained.

  It goes without saying that the same weighting framework can be applied not only to the position and orientation (external parameters) of the imaging apparatus but also to the problem of obtaining internal parameters (focal length, aspect ratio, distortion correction parameter) of the imaging apparatus. Yes.

<Modification 11>
Further, in the above embodiment, an index having a two-dimensional shape having a plurality of vertices as shown in FIG. 3B (an index composed of an outer shape such as a triangle or a quadrangle) is used. However, as in the case of an index having a two-dimensional shape, even when an index having a set of a plurality of coordinates as a constituent element (collectively referred to as a polygonal shape) is used, It is possible to obtain the evaluation amount based on the feature and the three-dimensional geometric feature of the camera and the index. Therefore, the effects of the invention described in this embodiment can be obtained.

For example, as shown in FIG. 3C, the index may be constituted by points p ^ki (i = 1,, N _k ) whose relative positional relationships are known. In addition, any index may be used as long as it has a set of a plurality of coordinates as a constituent element.

<Modification 12>
In the above embodiment, the Newton-Raphson method is used in which a nonlinear equation is Taylor-expanded in an optimization operation, a correction value is obtained by linearizing by linear approximation, and an optimum solution is obtained by repeatedly correcting. ing. However, the correction value need not necessarily be calculated by the Newton-Raphson method. For example, you may obtain | require using LM method (Levenberg-Marquardt method) which is an iterative solution method of a well-known nonlinear equation. Alternatively, the steepest descent method may be used. It goes without saying that the essence of the present invention is not impaired even if any other numerical calculation method is applied.

[Third Embodiment]
In the above-described embodiment, each part constituting the position / orientation measurement apparatus 2000 illustrated in FIG. 1 has been described as being implemented by hardware. However, some or all of the units shown in FIG. 1 may be realized by software, and the rest may be realized by hardware. In this case, for example, the hardware is realized as a function expansion card that can be inserted into a personal computer, and the function expansion card is inserted into the personal computer. The software is stored on a memory included in the personal computer. According to such a configuration, the CPU described in the first embodiment and various modifications can be performed by the CPU of the personal computer executing the software and also controlling the operation of the function expansion card. it can.

  FIG. 6 is a block diagram showing a hardware configuration of a computer applicable to the position / orientation measuring apparatus 2000.

  Reference numeral 601 denotes a CPU which controls the entire computer using programs and data stored in the RAM 602 and ROM 603.

  A RAM 602 has an area for temporarily storing programs and data loaded from the external storage device 606 and programs and data received from the outside via an I / F (interface) 607. Furthermore, it has a work area used when the CPU 601 executes various processes. That is, the RAM 602 can provide various areas as appropriate.

  Reference numeral 603 denotes a ROM which stores setting data and a boot program for the computer.

  An operation unit 604 includes a keyboard and a mouse, and various instructions can be input to the CPU 601 by an operator of the computer.

  A display unit 605 includes a CRT, a liquid crystal screen, and the like, and can display a processing result by the CPU 601 using an image, text, or the like.

  Reference numeral 606 denotes an external storage device, which is a large-capacity information storage device represented by a hard disk or the like. Here, an OS (operating system), programs and data for causing the CPU 601 to execute various processes performed by the computer, and the like. Is saved. The program and data include the above-described software and the operation control program for the function expansion card 608. The external storage device 606 also stores various types of information described as being held in advance in the above description and captured images received from the imaging device 2010 via the I / F 607.

  Various types of information stored in the external storage device 606 are appropriately loaded into the RAM 602 under the control of the CPU 601. When the CPU 601 executes processing using the loaded program and data, the computer can execute the processing described in the first embodiment and various modifications.

  Reference numeral 607 denotes an I / F for connecting the computer to the imaging apparatus 2010.

  Reference numeral 608 denotes a function expansion card.

  Reference numeral 609 denotes a bus connecting the above-described units.

[Other Embodiments]
Needless to say, the object of the present invention can be achieved as follows. That is, a recording medium (or storage medium) that records a program code of software that implements the functions of the above-described embodiments is supplied to a system or apparatus. Then, the computer (or CPU or MPU) of the system or apparatus reads and executes the program code stored in the recording medium. In this case, the program code itself read from the recording medium realizes the functions of the above-described embodiment, and the recording medium on which the program code is recorded constitutes the present invention.

  Further, by executing the program code read by the computer, an operating system (OS) or the like running on the computer performs part or all of the actual processing based on the instruction of the program code. Needless to say, the process includes the case where the functions of the above-described embodiments are realized.

  Furthermore, it is assumed that the program code read from the recording medium is written in a memory provided in a function expansion card inserted into the computer or a function expansion unit connected to the computer. After that, based on the instruction of the program code, the CPU included in the function expansion card or function expansion unit performs part or all of the actual processing, and the function of the above-described embodiment is realized by the processing. Needless to say.

  When the present invention is applied to the recording medium, program code corresponding to the flowchart described above is stored in the recording medium.

The invention according to claim 1 of the present application relates to an imaging device that includes an unknown index, which is an index whose position in the world coordinate system is unknown, and a known index whose position in the world coordinate system is an index, which are arranged in real space. An image input means for inputting a picked-up image picked up using;
Index detecting means for detecting the image coordinates of the unknown index and the image coordinates of the known index from the captured image;
Imaging device position and orientation acquisition means for acquiring an approximate position and orientation of the imaging device;
Index approximate position acquisition means for acquiring the approximate position of the unknown index;
A reliability calculation means for calculating a reliability for each index arranged in the real space;
Using the position of the known index, the information about the image coordinates detected by the index detection means, and the reliability calculated by the reliability calculation means, the approximate position and orientation of the imaging device and the unknown index Position and orientation calculation means for calculating the position and orientation of the imaging device and the position of the unknown index from the approximate position;
It is characterized by providing.

Claims (3)

An image input step of inputting a captured image captured by the imaging device;
An index detection step of detecting an index placed or set on an object from the captured image;
An index evaluation amount calculating step for calculating an evaluation amount for use as a clue to calculate the reliability of each index detected from the captured image;
A reliability calculation step of calculating the reliability of the index according to the evaluation amount of the index calculated in the index evaluation amount calculation step;
Position / orientation calculation for determining the position / orientation of the object or the imaging apparatus using at least the reliability of the index calculated in the reliability calculation step and the information on the image coordinates of each index detected in the index detection step A process,
The evaluation amount is-out based on the three-dimensional geometric features representing the relationship between the imaging device and the indicator in the 3-dimensional space of the two-dimensional geometric features and / or the indicator on the image of the index,
In the position / orientation calculation step, the object or the imaging apparatus uses at least the reliability of the index calculated in the reliability calculation step and information on the image coordinates of each index detected in the index detection step. A position and orientation measurement method characterized in that, in the process of obtaining the position and orientation of the index, the position or position and orientation of the index is also obtained. An image input means for inputting a captured image captured by the imaging device;
Index detecting means for detecting an index placed or set on an object from the captured image;
An evaluation amount to be used as a clue for calculating the reliability of each index detected in the captured image is used as a two-dimensional geometric feature on the index image and / or an imaging apparatus in the three-dimensional space of the index. Index evaluation amount calculating means for calculating using a three-dimensional geometric feature representing the relationship between the index and the index;
Reliability calculation means for calculating the reliability of the index according to the evaluation value of the index acquired by the index evaluation value calculation means;
Position / orientation for obtaining the position / orientation of the object or the imaging apparatus using at least the reliability of the index calculated by the reliability calculating unit and the information on the image coordinates of each index detected by the index detecting unit and a calculation means possess,
The position / orientation calculation means uses at least the reliability of the index calculated by the reliability calculation means and information on the image coordinates of each index detected by the index detection means, and the object or the imaging device A position / orientation measuring apparatus characterized in that, in the process of obtaining the position / orientation, the position or position / orientation of the index is also obtained. A program for causing a computer to execute each step of the position and orientation measurement method according to claim 1.

Images