JP6534411B2 - Relative angle estimation device, relative angle estimation method, and relative angle estimation program - Google Patents

Description

  The present invention relates to a relative angle estimation device, a relative angle estimation method, and a relative angle estimation program for estimating the relative angle between a subject of a flat object captured in an image and a camera that has captured an image.

  If the subject appears in the image, if the subject in the image can be recognized, the position of the subject in the image and the position in the real space of the recognition target constructed in advance By collation, the approximate position of the camera (photographer) in the real space can be identified. In addition, if the relative angle between the recognized subject and the photographer (for example, the azimuth angle shown in FIG. 1) can be estimated, the location in the real space of the photographer from the image captured by the smartphone, tablet terminal, etc. It is possible to realize navigation in accordance with the direction (presentation of a place to go next, information presentation related to the present location, etc.).

  Non-Patent Document 1 and Non-Patent Document 2 disclose a technique for estimating a base matrix and a base matrix from a set of corresponding points between two images. If the basic matrix and the basic matrix can be correctly estimated from the corresponding point sets of the two images, the corresponding points can be backprojected onto the three-dimensional space, so that the three-dimensional positional relationship between the subject and the photographer can be restored.

  However, in the techniques disclosed in Non-Patent Document 1 and Non-Patent Document 2, the known method for estimating the base matrix and the base matrix is that corresponding points between two images exist in the same plane in a three-dimensional space. If it is assumed that the two images capture only one plane, it is not possible to accurately estimate the base matrix and the base matrix. It is considered that a situation in which a photographer approaches and photographs a flat subject and only the flat part of the subject is captured in the image occurs at a high frequency. In such a situation, the inability to estimate the relative angle between the subject and the photographer has a problem that the availability and usability of the system are greatly impaired.

  On the other hand, in Non-Patent Document 3, a projection transformation matrix that geometrically associates two images in which a recognition target is captured and an internal parameter of a camera used for photographing (hereinafter referred to as "camera parameter") A technique is disclosed for estimating relative rotation information, relative translation information between two images, and a normal of an object. By using the technology disclosed in Non-Patent Document 2, it is possible to accurately estimate the positional relationship between a subject and a photographer in a three-dimensional space, even when only one plane is shown in the input image. it can.

  The estimation of the relative angle between the subject and the photographer can be regarded as an example of the estimation of the positional relationship between the subject and the photographer in a three-dimensional space.

RI Hartley, In Defense of the 8-point Algorithm, TPAMI, 1997. Internet <URL: http://www.cs.cmu.edu/afs/andrew/scs/cs/15/463/f07/proj_final/www/ amichals / fundamental.pdf> D. Niester, An Efficient Solution to the Five-Point Relative Pose Problem, TPAMI, 2004. Internet <URL: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.86.8769&rep=rep1&type= pdf> E. Malis et al., Deeper Understanding of the Photography Decomposition for Vision-based Control, INRIA Research Report, 2007. Internet <URL: https://hal.inria.fr/inria-00174036/document>

  In estimating the relative angle between the subject and the photographer, the technique disclosed in Non-Patent Document 3 has the following three problems.

  First, the technique disclosed in Non-Patent Document 3 assumes that the camera parameters of each of the two images captured by the projective transformation matrix are accurately determined. However, in actual operation, it is expensive to prepare internal parameters in advance for all cameras used for shooting. In particular, test images can be taken with any imaging device, and it is difficult to estimate camera parameters in advance for all of them. As a simple method to realize the same as above when the camera parameters are unknown, the rough camera parameters calculated from the size of the input image, the fixed focal length, etc. are applied to the technology disclosed in Non-Patent Document 1 How to do However, using this method alone causes a problem that the accuracy of the solution is reduced or the solution is not uniquely determined due to the imprecision of the camera parameters.

  Further, the estimation of the relative angle between the subject shown in the test image and the photographer is often performed together with the identification of the subject shown in the test image. In order to improve the accuracy of object identification, it is very common for training images to be subjected to preprocessing such as resizing and clipping. However, in this case, since the camera parameters of the image obtained by the pre-processing are different from the camera parameters of the original image, there arises a problem that the camera parameters of the camera that captured the training image can not be applied as it is.

  Furthermore, even with the technique disclosed in Non-Patent Document 3, although the normal line indicating the subject plane can be obtained, the relative angle between the subject and the photographer considering the real space, such as azimuth, can be accurately calculated. There is also the problem that you can not Although it is easy to calculate the angle itself only from the camera orientation and the normal to the subject, the calculated angle is the angle required to realize navigation (for example, the subject and the photographer are in the horizontal direction). Different from the angle (azimuth angle)). That is, assuming that all the test images were taken by the camera installed perpendicular to the ground, the calculated angle matches the azimuth angle, but especially for the photographer of the test image, the flat part of the camera is on the ground Vertical installation leads to significant restriction of the user's imaging freedom.

  To summarize the above, the known techniques related to the estimation of the relative angle between the planar subject and the photographer in the test image have the following problems (1) to (3):

(1) When the camera parameters of the camera used for capturing a test image are unknown, the obtained solution is not necessarily determined uniquely.

(2) When the test image and the training image are subjected to preprocessing such as resizing and cropping, the camera parameters of the camera that captured the original image can not be applied as they are.

(3) Real space information can not be taken into account in calculating the relative angle between the subject and the photographer. For example, the relative angle (azimuth angle) of the subject and the photographer in the horizontal direction can not be estimated without losing the user's imaging freedom.

  The present invention has been made in view of the circumstances as described above, and is a relative angle estimation device and relative angle estimation method capable of accurately estimating the relative angle between a planar subject and a camera shown in an image. And provide a relative angle estimation program.

  In order to achieve the above object, the relative angle estimation device of the present invention comprises a local feature extraction unit for extracting local feature amounts from each of a training image and a test image in which a planar object appears, and the planar object on the training image. An auxiliary information storage unit that stores information on a plurality of reference points of the above, camera parameters of a camera that captured the training image, and a normal of the flat object captured in the training image; local features of the training image and the test A corresponding point determination unit that determines corresponding points in the training image and the test image from local feature amounts of an image, and a set of corresponding points determined by the corresponding point determination unit, the training image as the test image A projection transformation matrix estimation unit for estimating a projection transformation matrix for projective transformation; and the projection transformation matrix estimated by the projection transformation matrix estimation unit from the camera parameters; A projective transformation matrix decomposing unit for calculating a plurality of solution candidates when it is decomposed into rotation and translational motion between an image and the test image and a normal of the planar object shown in the test image; and a predetermined constraint A palmarity determination unit for selecting the physically feasible solution candidate from the plurality of solution candidates calculated by the projective transformation matrix decomposing unit based on The candidate is input, and the projective transformation matrix estimated by the projective transformation matrix estimation unit is used to collate the normal of the solution candidate with the normal of the training image to select one solution candidate The image captured in the test image from a solution validity determination unit, information on the plurality of reference points, the rotation and translational motion of the solution candidate selected by the solution validity determination unit, and the projective transformation matrix Plane object and the test Includes an angle calculation unit that calculates a relative angle between the cameras taking the image, a.

  Note that the angle calculation unit is configured to calculate the three-dimensional space of the plurality of reference points calculated using the rotation and translational motion of the solution candidate selected by the solution validity determination unit and the projective transformation matrix. The relative angle may be calculated based on the position.

  The image processing apparatus further includes a pre-processing unit which inputs a training image and a test image including a flat object and performs pre-processing on each of the training image and the test image, and the auxiliary information storage unit includes the training image and The transformation parameters before and after the pre-processing are further stored for each of the test images, and the projective transformation matrix estimation unit further includes the camera parameters and a set of corresponding points determined by the corresponding point determination unit. From the transformation parameters, a projection transformation matrix for projective transforming the training image to the test image before the preprocessing is performed by the preprocessing unit may be estimated.

  In addition, when a plurality of the training images are input, a candidate image selection unit may be further included which selects the training image most similar to the test image from the plurality of training images.

  The image processing apparatus may further include a subject normal estimating unit for estimating a normal of the planar object shown in the training image, and the auxiliary information storage unit may estimate the plane of the training image estimated by the subject normal estimating unit. The normal of the object may be stored.

  In order to achieve the above object, according to a relative angle estimation method of the present invention, a local feature extraction unit, information of a plurality of reference points on a planar object captured in a training image, camera parameters of a camera that captured the training image, An auxiliary information storage unit that stores the normal of the planar object in the training image, a corresponding point determination unit, a projective transformation matrix estimation unit, a projective transformation matrix decomposition unit, a palmarity determination unit, solution validity determination A relative angle estimation method in a relative angle estimation device having a unit and an angle calculation unit, wherein the local feature extraction unit extracts a local feature amount from each of the training image and the test image, and the correspondence A corresponding point determination unit that determines corresponding points in the training image and the test image from the local feature of the training image and the local feature of the test image; and the projective transformation matrix estimation unit; Previous Estimating a projective transformation matrix for projective transformation of the training image to the test image from the set of corresponding points determined by the corresponding point determination unit; and the projective transformation matrix decomposing unit determines the camera parameters from the camera parameters A plurality of the projective transformation matrix estimated by the projective transformation matrix estimating unit is decomposed into rotation and translational motion between the training image and the test image and normals of the planar object shown in the test image. The solution candidate physically feasible from the plurality of solution candidates calculated by the projective transformation matrix decomposing unit based on the step of calculating the solution candidate and the hand determining unit based on a predetermined constraint Selecting the solution candidate, the solution validity determination unit inputs the solution candidate selected by the hand property determination unit, and the projective transformation matrix estimated by the projective transformation matrix estimation unit Using the normal of the solution candidate and the normal of the training image to select one solution candidate, the angle calculation unit, information on the plurality of reference points, and the solution valid Calculating a relative angle between the planar object captured in the test image and the camera that captured the test image from the rotation and translational motion of the solution candidate selected by the sex determination unit and the projective transformation matrix; ,including.

  In order to achieve the above object, a relative angle estimation program of the present invention is a program for causing a computer to function as each part of the relative angle estimation device.

  According to the present invention, it is possible to accurately estimate the relative angle between a planar subject and an image captured in an image.

It is a schematic diagram which shows the positional relationship of a to-be-photographed object and a camera, and the azimuth in the said positional relationship. It is a block diagram showing functional composition of relative angle estimating device concerning a 1st embodiment. It is a schematic diagram for demonstrating the auxiliary information storage part which concerns on embodiment. It is a schematic diagram for demonstrating each information memorize | stored by the auxiliary information storage part which concerns on embodiment. It is a schematic diagram for demonstrating the calculation method of the relative angle which concerns on embodiment. It is a schematic diagram for demonstrating the calculation method of the relative angle which concerns on embodiment. It is a schematic diagram for demonstrating the calculation method of the relative angle which concerns on embodiment. It is a flow chart which shows a flow of auxiliary information storage processing concerning a 1st embodiment. It is a flowchart which shows the flow of the relative angle estimation process which concerns on 1st Embodiment. It is a block diagram which shows the functional structure of the relative angle estimation apparatus which concerns on 2nd Embodiment. It is a flowchart which shows the flow of the relative angle estimation process which concerns on 2nd Embodiment. It is a block diagram which shows the functional structure of the relative angle estimation apparatus which concerns on 3rd Embodiment. It is a flow chart which shows the flow of the auxiliary information storage processing concerning a 3rd embodiment.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  The relative angle estimation apparatus according to the present embodiment estimates a relative angle between a planar subject (for example, a signboard, an advertisement, etc., hereinafter referred to as a “planar object”) and a camera, which are captured in a captured image. Device.

  First, the relative angle estimation device according to the present embodiment stores in advance information on the normals of the planar object shown in each training image. When the relative angle estimation device calculates the projective transformation matrix as described later, the normal of the planar object obtained from the projective transformation matrix calculated using the normal of the planar object shown in the training image Evaluate the validity of the candidate solution by evaluating its consistency with the candidate solution. The evaluation of validity includes, for example, uniquely identifying a solution candidate when there are a plurality of solution candidates, and rejecting a solution candidate with low reliability.

  In addition, the relative angle estimation device according to the present embodiment performs preprocessing such as resizing and clipping on a plurality of training images, and the image coordinate system of the training image after preprocessing is preprocessed before camera parameters are known. The parameters for conversion to the image coordinate system of are stored in advance. Then, when estimating the projective transformation matrix, the projective transformation matrix in the image coordinate system whose camera parameters are known is estimated using this transformation parameter.

  Furthermore, the relative angle estimation device according to the present embodiment stores, in advance, information of the reference point on the planar object necessary for estimation of the relative angle in consideration of the information of the real space. When calculating the relative angle, the relative angle between the camera and the planar object is calculated from the reference point and the projective transformation matrix between the images obtained by the processing up to that point, the relative rotation information, and the relative translation information. .

  In the first embodiment, in the case of inputting, as a training image, one image in which a flat object (for example, a signboard installed on a wall, a flat object such as an advertisement or the like) whose relative angle is to be estimated Think. In the first embodiment, when a test image including a planar object whose relative angle is to be estimated is input, the azimuth angle of the planar object with respect to the imaging surface of the camera is estimated as a relative angle, and the estimated relative angle is output. . In addition, the azimuth angle here is a relative angle of the plane object in a horizontal surface, and the imaging surface of a camera, as shown in FIG.

  FIG. 2 is a block diagram showing a functional configuration of the relative angle estimation device 10 according to the first embodiment. As shown in FIG. 2, the relative angle estimation device 10 according to the first embodiment includes a training unit 34 and a recognition unit 36.

  The training unit 34 includes a preprocessing unit 12, an auxiliary information storage unit 14, a parameter storage unit 16, and a local feature extraction unit 18. In addition, the recognition unit 36 includes the above-described preprocessing unit 12, local feature extraction unit 20, corresponding point determination unit 22, projective transformation matrix estimation unit 24, projective transformation matrix decomposition unit 26, palmarity determination unit 28, solution validity determination A unit 30 and an angle calculation unit 32 are provided.

  First, the training unit 34 will be described.

  The preprocessing unit 12 receives a training image 40 as input, and performs predetermined image processing on the input training image 40. Note that the image processing performed by the preprocessing unit 12 may be any image processing, for example, resizing to a predetermined size, manual cropping of a specific area, filtering for reducing the influence of image noise, etc. Can be mentioned.

  The auxiliary information storage unit 14 converts the image coordinate system after pre-processing to the image coordinate system before pre-processing when the image coordinate system changes in the training image 40 before and after the image processing in the pre-processing unit 12 Necessary information (hereinafter referred to as “conversion parameter”) is stored in the parameter storage unit 16. By using the transformation parameters, the coordinates of the training image 40 are transformed to the coordinates of the image coordinate system in which the camera parameters are known.

As an example, as shown in FIG. 3, after the input training image 40 is cropped from the upper left point (sx, sy) by the preprocessing unit 12, the training image 40 is multiplied by 1 / k. In this case, (sx, sy) and 1 / k are stored in the auxiliary information storage unit 14 as conversion parameters. Further, the auxiliary information storage unit 14 further includes information of the normal of the planar object shown in the training image 40, information of a reference point for estimating the relative angle between the planar object reflecting the information of the real space and the camera, And camera parameters unique to the camera that captured the training image 40 are stored in advance. For example, as shown in FIG. 3, a three-dimensional vector may be used as the information on the normal of the planar object shown in the training image 40.

It suffices to store the values of the elements that make up

In the present embodiment, it is noted that the training image 40 captures a flat object from the front, the normal vector is a vector having a value only in the depth direction (here, the z axis) (for example, n _x = It is represented by n _y = 0 and n _z = 1). Further, in order to obtain the azimuth angle between the planar object and the camera, it is sufficient to know the coordinates of the three vertices of the rectangular planar object in the training image 40, and in the example shown in FIG. It stores as x1, y1), B (x2, y2), and C (x3, y3).

  Furthermore, in the example illustrated in FIG. 3, the ID of the camera that has captured the training image 40 is stored as the auxiliary information. The parameter storage unit 16 has a database of camera parameters associated with each of the IDs of a plurality of cameras, and searches for camera parameter data from information of the ID of the camera that has captured the training image 40. As camera parameters of the camera, those based on any model can be used.

Hereinafter, in the present embodiment, it is assumed that 3x3 camera parameters shown in the following equation (1) are used:

...... (1)

The parameters f _x , f _y , c _x , and c _y may be calculated from the size of the captured image, the focal length of the camera, etc., or estimated by a known method disclosed in the following reference 1 You may use a thing.

[Reference 1] Z. Zhang, A Flexible New Technique for Camera Calibration, TPAMI, 2000.

  The local feature extraction unit 18 extracts a local feature amount and attribute information corresponding to each local feature from the training image 40 input by the pre-processing unit 12, and sets a set of the extracted local feature amount and attribute information as a parameter storage unit Make it memorize in 16. As a method of extracting a local feature, any known technique can be used. For example, SIFT disclosed in the following reference 2, LIFT disclosed in the following reference 3, others, SURF, BEIEF, BRISK, ORB, FREAK, AKAZE etc. are mentioned.

[Reference 2] D. G. Lowe, Distinctive Image Features from Scale-Invariant Keypoints, IJCV, 2004.

[Reference 3] K. M. Yi et al., LIFT: Learned Invariant Feature Transform, in Proc. ECCV, 2016.

  Next, the recognition unit 36 will be described.

  The preprocessing unit 12 receives the test image 42 obtained by photographing, and performs predetermined image processing on the input test image 42 as in the case of the training unit 34. Note that the pre-processing applied to the test image 42 may be different from the image processing applied to the training image 40. However, as in the case of the training unit 34, when the image coordinate system of the test image 42 is changed by the image processing by the pre-processing unit 12, the conversion parameter is stored in the parameter storage unit 16, and the projective conversion thereafter is performed. The matrix estimation unit 24 uses the conversion parameters stored in the parameter storage unit 16.

  The local feature extraction unit 20 extracts local feature amounts and attribute information corresponding to each local feature from the test image 42 as in the local feature extraction unit 18. In addition, the local feature extraction unit 20 acquires a set of the local feature amount and the attribute information stored in the parameter storage unit 16.

  The corresponding point determination unit 22 associates local feature quantities extracted from both the test image 42 and the training image 40 based on the similarity of the feature quantities. An arbitrary method can be used as a method of correlating local feature quantities extracted from both the test image 42 and the training image 40. For example, each correspondence between images as disclosed in the above-mentioned reference 2 Examples include a method of calculating the degree of similarity for point pairs, and an approximate neighborhood search method based on k-d tree fast, which is disclosed in Reference 4 below. Note that, in order to remove the incorrect correspondence, the similarity of the corresponding point pair is calculated only when the similarity of the most similar corresponding point pair is larger by a certain threshold or more than the similarity of the second similar corresponding point pair. You may carry out a ratio test.

In the following, a set of corresponding points of the training image 40 and the test image 42 obtained by the corresponding point determination unit 22 is

I assume.

Note that

Is the corresponding point coordinates in the training image 40,

Is the corresponding point coordinates in the test image 42.

[Reference 4] M. Muja et al., Fast Approximate Nearest Neighbors with Automatic Algorithm Configuration, in Proc. VISAPP, 2009.

The projective transformation matrix estimation unit 24 applies the processing of the preprocessing unit 12 to the corresponding point set obtained by the corresponding point determination unit 22, the conversion parameter of the training image 40 obtained by the auxiliary information storage unit 14, and the test image 42. The projective transformation matrix in the image coordinate system in which the camera parameters are known or can be estimated is estimated with the transformation parameter obtained at that time as an input. As a method of estimating the projective transformation matrix, for example, a set of corresponding points

Apply transformation parameters to the set of corresponding points obtained

Method for applying DLT disclosed in reference 5 below to corresponding points set

The projection transformation matrix obtained by DLT can be transformed using transformation parameters of the training image 40 and the test image 42, for example.

[Reference 5] R. Hartley et al., Multiple View Geometry in Computer Vision, 2004.

  In order to reduce the influence of erroneous correspondence left up to the correspondence point determination unit 22, RANSAC disclosed in the above-mentioned reference 5 may be used in combination with DLT.

The projective transformation matrix estimated by the projective transformation matrix estimation unit 24 may be a matrix that transforms the coordinates of the training image 40 into the test image 42 or a matrix that transforms the coordinates of the test image 42 into the training image 40 . In the present embodiment, a projection transformation matrix that transforms the test image 42 into the training image 40

The case where is used will be described.

The projective transformation matrix decomposing unit 26 uses the projection transformation matrix obtained by the projective transformation matrix estimating unit 24 as the relative rotation information between images, the relative translation information, and the candidate of the normal of the plane object based on the following equation (2) Break down into solutions. Specifically, the projective transformation matrix

As shown in the following equation (2), based on the following relationship:

, Translational vector

, And normal vectors

Disassemble:

...... (2)

Note that

Is the camera parameter of the camera that captured the test image 42,

Is a camera parameter of the camera that captured the training image 40.

  As a well-known method of calculating the decomposition | disassembly candidate solution which satisfy | fills said Formula (2), the method currently indicated by the said nonpatent literature 1 is mentioned, for example.

  In addition, the method currently disclosed by the said nonpatent literature 1 is calculated | required specifically, using following (3) Formula-(19) Formula. In addition, following (7) Formula-(19) Formula is a formula shown about the case where it is i = 1 in following formula (6).

...... (3)

...... (4)

...... (5)

...... (6)

...... (7)

...... (8)

...... (9)

...... (10)

...... (11)

...... (12)

...... (13)

...... (14)

...... (15)

...... (16)

...... (16)

...... (18)

...... (19)

  In addition to the methods described above, the methods disclosed in the following reference 6 and the methods disclosed in the following reference 7 can be used as well-known methods for calculating decomposition candidate solutions that satisfy the above equation (2). Etc. can be used.

[Reference 6] O. Faugeras et al., Motion and structure from motion in a piecewise planar environment, International Journal of Pattern Recognition and Artificial Intelligence, 1988.

[Reference 7] Z. Zhang et al., 3D Reconstruction based on Homography Mapping, in ARPA, 1996.

The camera parameters obtained by the auxiliary information storage unit 14 may be used.

Also in the case where the camera parameter is known in advance, the camera parameter may be used, and when the camera parameter is unknown, the camera parameter estimated from the size of the test image 42 may be used. Note that

There is a case where it is not possible to uniquely identify the solution only by the hand-capability determination unit 28 when the exact value of is not known.

The following four solution candidates can be obtained using the technology disclosed in the above-mentioned Non-Patent Document 3:

  The handedness determining unit 28 narrows down physically feasible solutions among the candidate solutions obtained by the projective transformation matrix decomposing unit 26. The method for determining whether the candidate solution is a physically feasible solution can be determined, for example, based on whether all the conditions shown in the following (1) to (3) are satisfied.

(1) The condition is that the two images satisfy the constraint that the image is taken from the same side as viewed from the subject (a flat object is not transparent). That is, if the condition of the following equation (20) is not satisfied, it is rejected.

...... (20)
(2) The condition that the corresponding point can be seen from both of the two images is satisfied. Homogeneous coordinates of corresponding points in the image

If the condition of the following equation (21) is not satisfied:

...... (21)
(3) A condition is satisfied that the direction of the normal to the subject when viewed from each camera coordinate system is the same. If the value of an element (for example, the z-axis) corresponding to the depth of the subject normal is not positive, it is rejected.

  If the projective transformation matrix and the camera parameters used in the projective transformation matrix factorization unit 26 are correct, the handness determination unit 28 can uniquely determine a solution. However, if the camera parameters used are roughly estimated, etc., the hand can not be uniquely identified only by the hand determining unit 28.

The solution validity determination unit 30 collates the normal of the solution obtained by the hand nature determination unit 28 with the information of the object normal in the training image 40 obtained by the auxiliary information storage unit 14, and the solution validity Determine As a specific method of determining solution validity, for example, the normal of a flat object in the image coordinate system of the training image 40 obtained from the auxiliary information storage unit 14

And the normal to the subject viewed from the coordinate system of the test image 42 obtained by the hand-capability determination unit 28

Converted to the normal in the image coordinate system of the training image 40

The calculated cosine similarity may be calculated, and the calculated cosine similarity may be the highest and may be output as a solution that is equal to or higher than a preset threshold.

  The angle calculation unit 32 uses a projective transformation matrix obtained by the processing up to the solution validity determination unit 30, a rotation matrix obtained by decomposing the projective transformation matrix, a translation vector, and a normal, and the auxiliary information storage unit 14. Using the information of the obtained reference point, the relative angle between the planar object and the camera considering the real space is calculated. The calculation of the relative angle may be performed, for example, in the procedure shown in FIGS.

That is, as shown in FIG. 5, each reference point obtained by the auxiliary information storage unit 14 using the projective transformation matrix obtained by the projective transformation matrix estimation unit 24 using the following equation (22)

The position of the test image 42 in the image coordinate system

Calculate

...... (22)

Also, as shown in FIG.

as well as

The position of the reference point on the three-dimensional space is restored by calculating the following equation (23) using

...... (23)

However,








I assume.

  Further, as shown in FIG. 7, the azimuth angle φ of the subject with respect to the imaging surface of the camera is calculated based on the following equation (24) from the position of each reference point in the three-dimensional space. At this time, the angle calculation unit 32 outputs an angle φ satisfying the following equation (24) as an azimuth.

...... (24)

Note that


I assume.

  The relative angle estimation device 10 according to the present embodiment is configured by, for example, a computer device provided with a central processing unit (CPU), a random access memory (RAM), and a read only memory (ROM) for storing various programs. . Moreover, the computer which comprises the relative angle estimation apparatus 10 may be equipped with memory | storage parts, such as a hard disk drive and non-volatile memory. In the present embodiment, when the CPU reads and executes the program stored in the storage unit such as the ROM and the hard disk, the above-described hardware resource and the program cooperate with each other to realize the above-described function.

  The flow of the auxiliary information storage process by the relative angle estimation device 10 according to the present embodiment will be described using the flowchart shown in FIG. In the present embodiment, the auxiliary information storage process is started at a timing when predetermined information for starting execution of the auxiliary information storage process is input to the relative angle estimation device 10, but the auxiliary information storage process is started. The timing is not limited to this, and for example, the auxiliary information storage process may be started at the timing when the training image 40 is input.

  In step S101, the training image 40 in which the preprocessing unit 12 captures a flat object whose relative angle is to be estimated, the camera parameters of the camera used for shooting the training image 40, and the flat image in the training image 40 The information on the normal and the information on the reference point on the planar image are input.

  In step S103, the preprocessing unit 12 performs preprocessing on the input training image 40. At this time, when the image coordinate system of the training image 40 changes, the coordinates of the training image 40 are converted from the coordinates of the image coordinate system after preprocessing to the coordinates of the image coordinate system before preprocessing. Generate conversion parameters.

  In step S105, the local feature extraction unit 18 extracts local feature amounts from the training image 40 on which the preprocessing has been performed.

  In step S107, the auxiliary information storage unit 14 stores the input camera parameter, the input normal line information, the input reference point information, and the generated conversion parameter in the parameter storage unit 16 as auxiliary information. The information of the local feature amount extracted by the local feature extraction unit 18 is stored in the parameter storage unit 16, and the execution of the program of the auxiliary information storage process is ended.

  Next, the flow of relative angle estimation processing by the relative angle estimation device 10 according to the present embodiment will be described using the flowchart shown in FIG. In this embodiment, the relative angle estimation process is started at the timing when the predetermined information for starting the execution of the relative angle estimation process is input to the relative angle estimation device 10, but the relative angle estimation process is started. For example, the relative angle estimation process may be started at the timing when the test image 42 is input.

  In step S201, the preprocessing unit 12 inputs a test image 42.

  In step S203, the preprocessing unit 12 performs preprocessing on the input test image 42. In addition, when the image coordinate system of the test image 42 changes at this time, the coordinates of the test image 42 are changed from the coordinates of the image coordinate system after preprocessing to that of the image coordinate system before preprocessing as in step S103. Generate transformation parameters for transformation to coordinates.

  In step S205, the local feature extraction unit 20 extracts a local feature amount from the test image 42 on which the pre-processing has been performed.

  In step S207, the corresponding point determination unit 22 uses the local feature of the training image 40 stored in the parameter storage unit 16 and the local feature extracted from the extracted test image 42 to execute the training image 40 and the test. The corresponding points in the image 42 are determined.

  In step S209, the projective transformation matrix estimation unit 24 uses the camera parameters and transformation parameters stored in the parameter storage unit 16 and the set of corresponding points determined by the corresponding point determination unit 22 as a test image for the test image Estimate a projective transformation matrix for projective transformation to 42.

  In step S211, the projective transformation matrix decomposing unit 26 decomposes the estimated projective transformation matrix into the rotation and translation between the training image 40 and the test image 42 and the normal of the planar object captured in the test image 42. Calculate multiple solution candidates when you

  In step S213, the palmarity determination unit 28 selects a physically feasible solution candidate from the calculated plurality of solution candidates based on the above-described predetermined constraints.

  In step S215, the solution validity determination unit 30 receives the physically feasible solution candidate selected by the hand property determination unit 28, and uses the estimated projective transformation matrix to determine the normal of the solution candidate and The normal of the training image 40 stored in the parameter storage unit 16 is checked to select one solution candidate.

  In step S217, the angle calculation unit 32 uses the information of the reference point stored in the parameter storage unit 16, the rotation and translational motion of one solution candidate selected by the solution validity determination unit, and the projective transformation matrix. Then, the relative angle between the planar object shown in the test image 42 and the camera that has taken the test image 42 is calculated.

  In step S219, the angle calculation unit 32 outputs the calculated relative angle, and the execution of the program of the present relative angle estimation process ends. In the present embodiment, data indicating the calculated relative angle is displayed on a display means such as a display, data indicating the calculated relative angle is transmitted to an external device, and data indicating the calculated relative angle is stored. By storing the data, the data relating to the calculated relative angle is output.

  As described above, in the present embodiment, local feature quantities are extracted from each of the training image 40 and the test image 42 in which the flat object appears, and information on a plurality of reference points on the flat object in the training image 40 The camera parameters of the photographed camera and the normal of the flat object shown in the training image 40 are stored. Also, corresponding points in the training image 40 and the test image 42 are determined from the local feature quantities of the training image 40 and the local feature quantities of the test image 42. Also, a projection transformation matrix for projective transformation of the training image 40 into the test image 42 is estimated from the camera parameters and the set of determined corresponding points. Also, a plurality of solution candidates are calculated when the estimated projection transformation matrix is decomposed into the rotation and translation between the training image 40 and the test image 42 and the normals of the planar object shown in the test image 42. In addition, physically feasible solution candidates are selected from the plurality of calculated solution candidates based on predetermined constraints. Also, the selected solution candidate is input, and the estimated projective transformation matrix is used to collate the normal of the solution candidate with the normal of the training image 40 to select one solution candidate. Then, based on information of a plurality of reference points, the rotation and translation of one selected solution candidate, and the projective transformation matrix, the relative angle between the planar object captured in the test image 42 and the camera that captured the test image 42 calculate.

  In addition, when calculating the relative angle between the planar object shown in the test image 42 and the camera that captured the test image 42, the rotation and translational motion of the solution candidate selected by the solution validity determination unit 30, and the projective transformation matrix The relative angle is calculated based on the position in the three-dimensional space of the plurality of reference points calculated using.

  Also, a training image 40 and a test image 42 showing a flat object are input, pre-processing is performed on each of the training image 40 and the test image 42, and pre-processing is performed on each of the training image 40 and the test image 42. The transformation parameters before and after are further stored. Also, from the camera parameters, the determined set of corresponding points, and the transformation parameters, a projection transformation matrix for projective transformation of the training image 40 into the test image 42 before the preprocessing is performed is estimated.

  Thereby, even if the parameters of the camera used to capture the test image 42 are unknown, the relative angle between the subject and the camera can be accurately estimated, or those with low estimation accuracy can be rejected in advance. It will be possible. Further, even if the training image 40 and the test image 42 used for estimation of the relative angle are preprocessed, the relative angle can be accurately estimated. Furthermore, regardless of the state in which the test image 42 captures a planar object, it is possible to calculate a relative angle in consideration of information in the real space.

Second Embodiment

  Next, a relative angle estimation device according to a second embodiment will be described.

  In the first embodiment described above, the case has been described in which the test image 42 in which one planar object is photographed is input and the one training image 40 in which the planar object is photographed is input. On the other hand, in the second embodiment, the test image 42 in which any one of a plurality of planar objects is captured is input, and for each of the plurality of planar objects, the planar object is photographed one by one from the front. The case of inputting the training image 40 of FIG.

  FIG. 10 shows a functional configuration of the relative angle estimation device according to the second embodiment. The configuration of the relative angle estimation device 10A according to the second embodiment is the same as the relative angle estimation device 10 according to the first embodiment except for the candidate image selection unit 50. Description of the configuration is omitted.

  When the test image 42 is input, the relative angle estimation device 10A according to the present embodiment receives the type of the planar object shown in the test image 42 and the plane shown in the test image 42 with respect to the imaging surface of the camera that shot the test image Output the azimuth of the object as a relative angle.

  Here, in the case where a plurality of training images 40 are input as the training image 40, it is necessary to associate all the training images 40 input with the test image 42, which increases the calculation cost.

  The candidate image selection unit 50 selects one training image 40 most similar to the input test image 42 based on a predetermined standard. In addition, as a selection method of the training image 40 most similar to the test image 42, for example, a method of evaluating image similarity based on a color histogram, a voting base disclosed in reference 8 below, which stores geometric information. Methods etc.

[Reference 8] Japanese Patent Application Laid-Open No. 2016-018444

  Next, the flow of relative angle estimation processing by the relative angle estimation device 10A according to the present embodiment will be described using the flowchart shown in FIG. In the present embodiment, the relative angle estimation process is started at the timing when predetermined information for starting execution of the relative angle estimation process is input to the relative angle estimation device 10A, but the relative angle estimation process is started. For example, the relative angle estimation process may be started at the timing when the test image 42 is input.

  In the present embodiment, it is assumed that a plurality of training images 40 in which planar objects of different types are captured are input, and information on normals of the plurality of training images 40 is stored in the parameter storage unit 16. .

  In step S301, the preprocessing unit 12 inputs a test image 42.

  In step S303, the preprocessing unit 12 performs preprocessing on the input test image 42. In addition, when the image coordinate system of the test image 42 changes at this time, the coordinates of the test image 42 are changed from the coordinates of the image coordinate system after preprocessing to that of the image coordinate system before preprocessing as in step S203. Generate transformation parameters for transformation to coordinates.

  In step S305, the local feature extraction unit 20 extracts local feature amounts from the test image 42 on which the pre-processing has been performed.

  In step S <b> 307, the candidate image selection unit 50 selects a training image 40 most similar to the test image 42 from the plurality of training images 40.

  In step S309, the corresponding point determination unit 22 performs selection using the local feature of the selected training image 40 stored in the parameter storage unit 16 and the local feature extracted from the extracted test image 42. The corresponding points in the training image 40 and the test image 42 are determined.

  In step S311, the projective transformation matrix estimation unit 24 uses the camera parameters and transformation parameters stored in the parameter storage unit 16 and the set of corresponding points determined by the corresponding point determination unit 22 as a test image for the test image Estimate a projective transformation matrix for projective transformation to 42.

  In step S313, the projective transformation matrix decomposition unit 26 decomposes the estimated projective transformation matrix into the rotation and translation between the training image 40 and the test image 42 and the normal of the planar object captured in the test image 42. Calculate multiple solution candidates when you

  In step S315, the palmarity determination unit 28 selects a physically feasible solution candidate from among the calculated plurality of solution candidates based on the above-described predetermined constraints.

  In step S317, the solution validity determining unit 30 receives the physically feasible solution candidate selected by the hand determining unit 28, and uses the estimated projective transformation matrix to determine the normal of the solution candidate and The normal of the training image 40 stored in the parameter storage unit 16 is checked to select one solution candidate.

  In step S319, the angle calculation unit 32 uses the information of the reference point stored in the parameter storage unit 16, the rotation and translational motion of one solution candidate selected by the solution validity determination unit, and the projective transformation matrix. Then, the relative angle between the planar object shown in the test image 42 and the camera that has taken the test image 42 is calculated.

  In step S321, the angle calculation unit 32 outputs the calculated relative angle, and ends the execution of the program of the relative angle estimation process. In the present embodiment, data indicating the calculated relative angle is displayed on a display means such as a display, data indicating the calculated relative angle is transmitted to an external device, and data indicating the calculated relative angle is stored. By storing the data, the data relating to the calculated relative angle is output.

  In this manner, in the present embodiment, when a planar image is captured in the test image 42, the training image 40 most similar to the test image 42 is selected from the plurality of training images 40.

Third Embodiment

  Next, a relative angle estimation device according to a third embodiment will be described.

  In the first embodiment and the second embodiment described above, the case of inputting the information on the normal of the flat object in the training image 40 has been described. On the other hand, in the third embodiment, the case of estimating the normal of the planar object from the input training image 40 will be described.

  The configuration of the relative angle estimation device according to the third embodiment is the same as the relative angle estimation device 10A according to the second embodiment except for the subject normal direction estimation unit 60. Description of each configuration is omitted.

  FIG. 11 shows the configuration of a relative angle estimation device 10B according to the third embodiment. In this embodiment, a plurality of types of subjects to be recognized exist, and a plurality of images captured from various directions for each subject are registered as the training image 40. When the test image 42 is input, the relative angle estimation device 10B outputs, as a relative angle, the type of the subject shown in the test image 42 and the azimuth between the subject and the camera. By registering images captured from various directions for each subject in the database, the accuracy of recognizing the subject shown in the test image 42 is further improved. On the other hand, since the training image 40 is an image of the subject taken from an arbitrary direction, it takes a lot of time to manually annotate the normal.

  Therefore, the subject normal direction estimation unit 60 automatically estimates the normals of subjects appearing in individual images by taking advantage of the fact that a plurality of images in which the same subject appears are taken. As a method of realizing this, for example, as shown in reference 9, a method of estimating the normal line of the subject shown in each image so as to be highly consistent in the image set showing the same subject is used. it can.

  In addition, when the image captured from the front in the image is known, the information of the object normal in the image (the normal may be a vector having a value only in the depth direction) may be used as the auxiliary information.

[Reference 9] Japanese Patent Application No. 2016-244526

  Next, the flow of the auxiliary information storage process by the relative angle estimation device 10B according to the present embodiment will be described using the flowchart shown in FIG. In the present embodiment, the auxiliary information storage process is started at a timing when predetermined information for starting execution of the auxiliary information storage process is input to the relative angle estimation device 10B, but the auxiliary information storage process is started. The timing is not limited to this, and for example, the auxiliary information storage process may be started at the timing when the training image 40 is input.

  In step S401, the training image 40 in which the preprocessing unit 12 captures a flat object whose relative angle is to be estimated, the camera parameters of the camera used for shooting the training image 40, and the reference points on the flat image Enter information.

  In step S403, the preprocessing unit 12 performs preprocessing on the input training image 40. At this time, when the image coordinate system of the training image 40 changes, the coordinates of the training image 40 are converted from the coordinates of the image coordinate system after preprocessing to the coordinates of the image coordinate system before preprocessing. Generate conversion parameters.

  In step S405, the local feature extraction unit 18 extracts local feature amounts from the training image 40 on which the preprocessing has been performed.

  In step S <b> 407, the subject normal direction estimation unit 60 estimates, for each type of planar object, the normals of the planar object shown in each training image 40 using a plurality of training images 40 in which the same planar object appears. .

  In step S409, the auxiliary information storage unit 14 stores the input camera parameter, information of the estimated normal, information of the input reference point, and the generated conversion parameter in the parameter storage unit 16 as auxiliary information. The information of the local feature amount extracted by the local feature extraction unit 18 is stored in the parameter storage unit 16, and the execution of the program of the auxiliary information storage process is ended.

  As described above, in the present embodiment, the normal of the planar object shown in the training image 40 is estimated, and the estimated normal of the planar object shown in the training image 40 is stored in the parameter storage unit 16.

  In this embodiment, the operation of the component of the function shown in FIG. 1 is constructed as a program and installed in a computer used as the relative angle estimation device 10 or 10A and executed, but the present invention is not limited thereto. You may distribute it through.

  Further, the constructed program may be stored in a portable storage medium such as a hard disk or a CD-ROM, and may be installed in a computer or distributed.

10 Relative Angle Estimation Device 12 Preprocessing Unit 14 Auxiliary Information Storage Unit 16 Parameter Storage Unit 18 and 20 Local Feature Extraction Unit 22 Corresponding Point Determination Unit 24 Projective Transformation Matrix Estimation Unit 26 Projective Transformation Matrix Decomposition Unit 28 Palmarity Determination Unit 30 Solution Validity Ability judgment unit 32 Angle calculation unit 34 Training unit 36 Recognition unit 50 Candidate image selection unit 60 Subject normal line estimation unit

Claims (7)

A local feature extraction unit for extracting local feature amounts from each of a training image and a test image in which a planar object is taken;
An auxiliary information storage unit that stores information of a plurality of reference points on the planar object captured in the training image, camera parameters of a camera that captured the training image, and a normal of the planar object captured in the training image;
A corresponding point determination unit that determines corresponding points in the training image and the test image from the local feature of the training image and the local feature of the test image;
A projective transformation matrix estimation unit for estimating a projective transformation matrix for projective transformation of the training image to the test image from the set of corresponding points determined by the corresponding point determination unit;
The projective transformation matrix estimated by the projective transformation matrix estimation unit from the camera parameters is converted into rotation and translation between the training image and the test image, and a normal of the planar object captured in the test image. A projective transformation matrix factorization unit for calculating a plurality of solution candidates when being decomposed;
A handness determination unit which selects the physically feasible solution candidate from the plurality of solution candidates calculated by the projective transformation matrix decomposing unit based on a predetermined constraint;
The solution candidate selected by the hand determining unit is input, and the normal of the solution candidate and the normal of the training image are input using the projective transformation matrix estimated by the projective transformation matrix estimating unit. A solution validity determination unit that collates and selects one solution candidate;
The planar object and the test image reflected in the test image are obtained from information on the plurality of reference points, the rotation and translational motion of the solution candidate selected by the solution validity determination unit, and the projective transformation matrix. An angle calculation unit that calculates a relative angle with a captured camera;
Relative angle estimation device including. The angle calculation unit is configured to calculate the positions of the plurality of reference points in three-dimensional space calculated using the rotation and translational motion of the solution candidate selected by the solution validity determination unit and the projective transformation matrix. The relative angle estimation device according to claim 1, wherein the relative angle is calculated based on the relative angle. The method further includes a pre-processing unit that inputs a training image and a test image in which a planar object is taken, and performs preprocessing on each of the training image and the test image,
The auxiliary information storage unit further stores, for each of the training image and the test image, conversion parameters before and after the preprocessing.
The projective transformation matrix estimation unit performs the training before the preprocessing is performed by the preprocessing unit from the camera parameter, the set of corresponding points determined by the corresponding point determination unit, and the conversion parameter. The relative angle estimation device according to claim 1, wherein a projective transformation matrix for projective transforming an image into the test image is estimated. The candidate image selection part which selects the said training image most similar to the said test image from the said some said training images is further included, when the said some training image is input. Relative angle estimation device. The image processing apparatus further includes a subject normal estimating unit for estimating a normal of the planar object shown in the training image,
The relative angle estimation device according to any one of claims 1 to 4, wherein the auxiliary information storage unit stores the normal of the planar object shown in the training image estimated by the subject normal estimation unit. A local feature extraction unit, information of a plurality of reference points on a planar object captured in a training image, camera parameters of a camera that captured the training image, and auxiliary information storage storing normals of the planar object captured in the training image Relative angle in a relative angle estimation apparatus including a division unit, a corresponding point determination unit, a projective transformation matrix estimation unit, a projective transformation matrix decomposition unit, a handedness determination unit, a solution validity determination unit, and an angle calculation unit The estimation method,
Extracting the local feature amount from each of the training image and the test image by the local feature extraction unit;
A corresponding point determination unit that determines corresponding points in the training image and the test image from the local feature amount of the training image and the local feature amount of the test image;
The projective transformation matrix estimating unit estimating a projective transformation matrix for projective transforming the training image to the test image from the set of corresponding points determined by the corresponding point determining unit;
The projective transformation matrix decomposing unit reflects the projective transformation matrix estimated by the projective transformation matrix estimating unit from the camera parameters in rotation and translation between the training image and the test image and the test image Calculating a plurality of solution candidates when being decomposed into the normal of the planar object;
Selecting the physically feasible solution candidate from the plurality of solution candidates calculated by the projective transformation matrix decomposing unit based on a predetermined constraint;
The solution validity determining unit receives the solution candidate selected by the hand determining unit, and uses the projective transformation matrix estimated by the projective transformation matrix estimating unit to obtain the normal of the solution candidate Matching with the normal of the training image and selecting one of the candidate solutions;
The planar object captured in the test image from the information on the plurality of reference points, the rotation and translational motion of the solution candidate selected by the solution validity determination unit, and the projective transformation matrix. Calculating the relative angle between the camera and the camera that captured the test image;
Relative angle estimation method including.   The relative angle estimation program for functioning a computer as each part of the relative angle estimation apparatus in any one of Claims 1-5.