CN110782492A - Pose tracking method and device - Google Patents

Abstract

A pose tracking method and device are provided. The pose tracking method comprises the following steps: acquiring an image of a tracking object, wherein a mark which flickers at a specific frequency is arranged on the tracking object; acquiring pixels with changed brightness from the acquired image; the 6-degree-of-freedom posture of the tracked object is calculated based on the acquired pixels, so that the dependency of posture tracking on the specific layout of the LED markers is reduced, the delay of posture tracking is reduced, and the precision and the efficiency of posture tracking are improved.

Description

Pose tracking method and device

technical field

The present disclosure relates to the technical field of computer vision. More specifically, the present disclosure relates to a pose tracking method and apparatus.

Background technique

In recent years, a variety of 6-DOF pose estimation methods have been proposed and widely used in the fields of robotic grasping, virtual reality/augmented reality, and human-computer interaction. Virtual reality and augmented reality have high demands on system latency. If the system is slow to respond to head movements, it can cause dizziness and nausea to the user. Valve's system has a latency of 7-15 milliseconds. The lowest latency of current commercial virtual reality (VR) tracking products is 15 milliseconds, which does not allow users to have a perfectly immersive experience.

Many current optical tracking systems are based on complementary metal-oxide-semiconductor (CMOS) cameras. However, the CMOS camera delay of this kind of consumer machine is generally greater than 16.7 milliseconds (60FPS). Due to hardware limitations, these methods cannot display the user's action input on the screen in a timely manner, and cannot meet the low-latency requirements of VR.

Active light-emitting diode (LED) markers are also used in some schemes to recover the 6-DOF pose of the object. However, these methods have some limitations, either the number of LED lights must be four and they are coplanar, or the computational cost ratio exceeds a preset deviation threshold and cannot be applied to real-time systems. Having only 4 LED lights affects the accuracy of pose tracking and the robustness of the system, because if one of the lights is not detected, the pose solution fails. In addition, it is very time-consuming to brute force to solve the correspondence between 2D/3D point sets and cannot be applied to real-time systems where there are many LED lights.

SUMMARY OF THE INVENTION

Exemplary embodiments of the present disclosure are to provide a pose tracking method and apparatus to reduce the dependency of pose tracking on a specific layout of LED markers, reduce the delay of pose tracking, and improve the accuracy and efficiency of pose tracking.

According to an exemplary embodiment of the present disclosure, a pose tracking method is provided, comprising: acquiring an image of a tracking object, wherein the tracking object is provided with a mark that flickers at a specific frequency; acquiring a brightness change from the acquired image pixel; the 6-DOF pose of the tracked object is calculated based on the acquired pixels, thereby reducing the dependency of the pose tracking on the specific layout of the LED markers, reducing the delay of the pose tracking, and improving the accuracy and efficiency.

Optionally, the step of calculating the 6-DOF attitude of the tracking object based on the acquired pixels may include: acquiring IMU data of the tracking object, and estimating the 3-DOF of the tracking object based on the acquired IMU data. posture, wherein the 3-DOF posture is a posture that rotates around the three coordinate axes of x, y, and z in the body coordinate system of the tracking object; the tracking is calculated based on the 3-DOF posture and the acquired pixels The 6-DOF posture of the object, wherein the 6-DOF posture is the posture along the three coordinate axes of x, y, and z and the three rectangular coordinates around x, y, and z in the body coordinate system of the tracking object The pose of the axis rotation, thereby improving the accuracy and efficiency of pose tracking.

Optionally, the step of calculating the 6-DOF posture of the tracking object based on the 3-DOF posture and the acquired pixels may include: solving the marked 2D point set based on the 3-DOF posture and the acquired pixels. The corresponding relationship with the 3D point set, the matching pair of the 2D point set and the 3D point set of the mark is obtained, wherein the 2D point set includes the pixel coordinates of the mark, and the 3D point set includes the mark in the The coordinates in the body coordinate system of the tracking object; based on the matching pair, the 6-DOF pose of the tracking object is calculated, thereby improving the accuracy and efficiency of pose tracking.

Optionally, the step of calculating the 6-DOF pose of the tracked object may include: removing pixels whose reprojection deviation exceeds a preset deviation threshold from the matched pair, and calculating the 6-DOF pose according to the remaining pixels after removal. ; Perform a reprojection error minimization operation on the calculated 6-DOF pose to obtain the 6-DOF pose of the tracked object, thereby improving the accuracy and efficiency of pose tracking.

Optionally, the step of calculating the 6-DOF pose of the tracked object may include: removing pixels whose reprojection deviation exceeds a preset deviation threshold from the matched pair, and calculating the 6-DOF pose according to the remaining pixels after removal. ; Minimize the re-projection error operation on the calculated 6-DOF pose; optimize the 6-DOF pose after the re-projection error minimization operation according to the 3-DOF pose, and obtain the 6-DOF pose of the tracking object pose, thereby improving the accuracy and efficiency of pose tracking.

Optionally, after obtaining the 6-DOF attitude of the tracking object, the pose tracking method may further include: according to the 6-DOF attitude of the tracking object, the reprojection error in the matching pair exceeds a preset deviation. The thresholded pixels are rematched for subsequent pose tracking.

According to an exemplary embodiment of the present disclosure, a pose tracking device is provided, comprising: an image acquisition unit configured to acquire an image of a tracking object, wherein the tracking object is provided with a mark that flickers at a specific frequency; pixel acquisition a unit configured to acquire pixels of varying brightness from the acquired image; and an attitude calculation unit configured to calculate a 6-DOF attitude of the tracked object based on the acquired pixels, thereby reducing the effect of pose tracking on a specific layout of LED markers At the same time, the delay of pose tracking is reduced, and the accuracy and efficiency of pose tracking are improved.

Optionally, the attitude calculation unit may be configured to: acquire inertial measurement unit data of the tracked object, and estimate a 3-DOF attitude of the tracked object based on the acquired inertial measurement unit data, wherein the 3-DOF attitude is The posture of rotating around the three coordinate axes of x, y, and z in the body coordinate system of the tracking object; the 6-DOF posture of the tracking object is calculated based on the 3-DOF posture and the acquired pixels, wherein the The 6-DOF posture is the posture along the three coordinate axes of x, y, and z and the posture of rotating around the three rectangular coordinate axes of x, y, and z under the body coordinate system of the tracking object, thereby improving the posture tracking. accuracy and efficiency.

Optionally, the attitude calculation unit may also be configured to: based on the 3-DOF attitude and the acquired pixels, solve the correspondence between the marked 2D point set and the 3D point set, and obtain the marked 2D point set. a matching pair with a 3D point set, wherein the 2D point set includes the pixel coordinates of the marker, and the 3D point set includes the coordinates of the marker in the body coordinate system of the tracked object; based on the matching pair , the 6-DOF pose of the tracked object is calculated, thereby improving the accuracy and efficiency of pose tracking.

Optionally, the pose calculation unit can also be configured to: remove the pixels whose reprojection deviation exceeds the preset deviation threshold from the matching pair, and calculate the 6-DOF pose according to the remaining pixels after the removal; The DOF pose is performed to minimize the reprojection error to obtain the 6-DOF pose of the tracked object, thereby improving the accuracy and efficiency of pose tracking.

Optionally, the pose calculation unit can also be configured to: remove the pixels whose reprojection deviation exceeds the preset deviation threshold from the matching pair, and calculate the 6-DOF pose according to the remaining pixels after the removal; The DOF pose is operated to minimize the reprojection error; the 3DOF pose is fused with the 6DOF pose after the operation to minimize the reprojection error, and the 6DOF pose of the tracking object is obtained, thereby improving the performance of the tracking object. Accuracy and efficiency of pose tracking.

Optionally, the pose tracking device may further include: a re-matching unit configured to, after obtaining the 6-DOF pose of the tracked object, make the re-projection error exceed a predetermined amount according to the 6-DOF pose of the tracked object. The pixels with the deviation threshold are re-matched for subsequent pose tracking.

According to an exemplary embodiment of the present disclosure, there is provided an electronic device, comprising: a camera for acquiring an image of a tracking object, and acquiring pixels whose brightness changes from the acquired image, wherein the tracking object is provided with a specific A marker that flickers at a frequency; a processor for calculating the 6-DOF pose of the tracked object based on the pixels acquired by the camera, thereby reducing the dependence of pose tracking on the specific layout of the LED marker, and reducing the complexity of pose tracking. delay, and improve the accuracy and efficiency of pose tracking.

According to an exemplary embodiment of the present disclosure, there is provided a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, implements the pose tracking method according to the exemplary embodiment of the present disclosure .

According to an exemplary embodiment of the present disclosure, there is provided a computing device, comprising: a processor; a memory storing a computer program that, when executed by the processor, implements a pose according to an exemplary embodiment of the present disclosure tracking method.

According to the pose tracking method and device according to the exemplary embodiments of the present disclosure, by acquiring an image of the tracking object, acquiring pixels with brightness changes from the acquired image, and calculating the 6-DOF pose of the tracking object based on the acquired pixels, thereby reducing the The dependence of pose tracking on the specific layout of LED markers, while reducing the latency of pose tracking, and improving the accuracy and efficiency of pose tracking.

Additional aspects and/or advantages of the present disclosure will be set forth in part in the description that follows, and in part will be apparent from the description, or may be learned by practice of the present disclosure.

Description of drawings

The above and other objects and features of exemplary embodiments of the present disclosure will become more apparent from the following description taken in conjunction with the accompanying drawings that exemplarily illustrate embodiments, wherein:

1 shows a flowchart of a pose tracking method according to an exemplary embodiment of the present disclosure;

Figure 2 shows the 2D active LED marker tracking results of the tracked object;

3 shows a block diagram of a pose tracking apparatus according to an exemplary embodiment of the present disclosure;

FIG. 4 shows a schematic diagram of an electronic device according to an exemplary embodiment of the present disclosure; and

5 shows a schematic diagram of a computing device according to an exemplary embodiment of the present disclosure.

Detailed ways

Reference will now be made in detail to the exemplary embodiments of the present disclosure, examples of which are illustrated in the accompanying drawings, wherein like reference numerals refer to like parts throughout. The embodiments are described below in order to explain the present disclosure by referring to the figures.

FIG. 1 shows a flowchart of a pose tracking method according to an exemplary embodiment of the present disclosure. The pose tracking method shown in FIG. 1 is suitable for a tracking object provided with a plurality of markers flashing at a specific frequency, wherein the markers flashing at a specific frequency may be, for example, active LEDs. In the following description, the LED lamp is used as an example of a symbol, but it should be understood that the present invention is not limited to this. Those skilled in the art may adopt other forms of marking according to the needs of the embodiment.

Referring to FIG. 1, in step S101, an image of a tracking object is acquired.

In an exemplary embodiment of the present disclosure, an image of a tracked object may be acquired through a DVS camera. The pose tracking method shown in FIG. 1 may be applied to an electronic device having a camera capable of acquiring pixels of brightness change and a host capable of performing calculations, or may be applied to an electronic device having a camera capable of acquiring pixels of brightness change and a computer capable of performing computation A system of hosts.

In step S102, pixels with varying brightness are acquired from the acquired image.

In an exemplary embodiment of the present disclosure, the DVS camera does not directly transmit the image to the host after acquiring the image in step S101, but acquires pixels with brightness changes from the acquired image in step S102, and then transmits the acquired pixels to The host is used for pose tracking, thereby reducing the amount of data transmitted and the amount of data used for calculation, and reducing the delay of pose tracking.

Specifically, when a marker (eg, an active LED light) blinks, the DVS can generate corresponding on and off events. These events can be easily distinguished from other events by the frequency of LED blinking. These filtered events can be divided into different clusters, each cluster representing an LED light. These filtered events can then be processed using a region-growing-based clustering algorithm and a lightweight voting algorithm, identifying the point with the highest density in the cluster as the center of the marker. In addition, multiple markers can be tracked using a global nearest neighbor tracking method and a uniform model-based Kalman filter, thereby reducing the probability of missed and false detections of markers.

In step S103, the 6-DOF pose of the tracked object is calculated based on the acquired pixels. Here, the 6-DOF attitude represents the attitude along the three coordinate axes of x, y, and z and the attitude of rotation around the three rectangular coordinate axes of x, y, and z in the body coordinate system of the tracking object.

In an exemplary embodiment of the present disclosure, only those pixels whose brightness changes due to motion are used to calculate the 6-DOF pose of the tracked object, thereby reducing the latency of pose tracking.

In an exemplary embodiment of the present disclosure, when calculating the 6-DOF pose of the tracked object based on the acquired pixels, inertial measurement unit (IMU) data of the tracked object may be acquired first, and estimation based on the acquired inertial measurement unit (IMU) data may be performed. The 3DOF pose of the tracked object (that is, the pose rotated around the three axes of x, y, and z in the body coordinate system of the tracked object), and then the 6DOF of the tracked object is calculated based on the 3DOF pose and the acquired pixels pose, thereby improving the accuracy and efficiency of pose tracking.

In an exemplary embodiment of the present disclosure, when calculating the 6-DOF pose of the tracked object based on the 3-DOF pose and the acquired pixels, first, based on the 3-DOF pose and the acquired pixels, the marked 2D point set and the 3D The corresponding relationship of the point set is obtained, and the marked matching pair about the 2D point set and the 3D point set is obtained, and then based on the matching pair, the 6-DOF pose of the tracked object is calculated. Here, the 2D point set includes the pixel coordinates of each marker, and the 3D point set includes the coordinates of each marker in the body coordinate system of the tracked object.

Specifically, when calculating the 6-DOF pose of the tracked object based on the 3-DOF pose and the acquired pixels, it can be defined that p _IA and p _IB are the two LED lights (LED light A and LED light) that have been de-distorted on the image, respectively. B) pixel coordinates, p _A and p _B are the coordinates of LED light A and LED light B in the object body coordinate system, respectively. The 3-DOF attitude matrix R=[r ₁ , r ₂ , r ₃ ] of the tracked object can be estimated by the ^heading reference system through the IMU data, r ₁ , r ₂ and r ₃ represent the first row of R Row vector for the second and third row. t=[t _x , _ty , t _z ] ^T is an unknown translation vector, where t _x , _ty , and t _z represent displacements along the three coordinate axes of x, y, and z, respectively. Through the principle of pinhole imaging, the following equation can be obtained:

Among them, x _A and y _A represent the x-coordinate and y-coordinate of the LED light A on the image, respectively. x _B , y _B represent the x-coordinate and y-coordinate of the LED light B on the image, respectively. In the above equations, only t is unknown. There are 4 equations and 3 unknowns. Solving the equations can get:

t=A _z p _IA -Rp _A (3)

in,

Through the DVS camera detection and clustering algorithm, the pixel coordinate point set O of the LED light in the image and the coordinate point set L of the known LED light in the object body coordinate system can be obtained. Two points (o, l) are arbitrarily selected from the point sets O and L for pairing, then the translation t can be calculated by formula (3). Through the above operations, a list T of possible translation vectors can be obtained. Some invalid translation vectors (the elements of the vector are too large or t _z are negative) can be deleted, and some translation vectors that are approximately equal can be merged. For any valid translation vector t _valid in T, the Moller-Trumbore ray intersection algorithm can be used to determine the point set L _v where the corresponding visible LED light is visible: if a certain LED light and the camera's determined ray intersect the object A point is the LED light, then the LED light is visible, otherwise the LED light is invisible in this pose. Determining the set of visible LED light points can reduce computation and mis-matching in the case of multiple LED lights. Then the visible point set L _v is projected onto the image plane P with the corresponding 6-DOF pose and camera intrinsics. In this way, the Kuhn-Munkres algorithm can be used to solve the best matching and matching error of the visible point set L _v and the observation point set O. Traversing the list of possible translation vectors T, the set of matches with the smallest matching error is the correct matching pair about the 2D point set and the 3D point set.

In an exemplary embodiment of the present disclosure, when calculating the 6-DOF pose of the tracked object, pixels whose reprojection deviation exceeds a preset deviation threshold may be first removed from the matching pair, and the 6-DOF is calculated according to the remaining pixels after removal. Then, the calculated 6-DOF pose is performed to minimize the reprojection error to obtain the 6-DOF pose of the tracked object, thereby further improving the accuracy and efficiency of pose tracking.

In an exemplary embodiment of the present disclosure, when calculating the 6-DOF pose of the tracked object, pixels whose reprojection deviation exceeds a preset deviation threshold may be first removed from the matching pair, and the 6-DOF is calculated according to the remaining pixels after removal. pose, and then perform the operation of minimizing the reprojection error on the calculated 6-DOF pose, and then optimize the 6-DOF pose after minimizing the re-projection error operation according to the 3-DOF pose to obtain the 6-DOF of the tracked object. pose, thereby further improving the accuracy and efficiency of pose tracking.

In an exemplary embodiment of the present disclosure, after the 6-DOF pose of the tracked object is obtained, the pixels whose re-projection error exceeds a preset deviation threshold in the matching pair can also be re-matched according to the 6-DOF pose of the tracked object. In addition, the correspondence between 2D point sets and 3D point sets or matching pairs of 2D point sets and 3D point sets for newly observed markers (e.g., active LED lights) can also be updated, thereby further improving the accuracy of pose tracking and efficiency.

Specifically, after obtaining the corresponding relationship between the 2D point set and the 3D point set, the Random Sample Consensus (RANSAC for short) algorithm can be used to remove the points whose reprojection deviation exceeds the preset deviation threshold in the matching pair, and then Use the Efficient Per Efficient Perspective-n-Pointspective-n-Point (EPnP) algorithm to solve the 6-DOF pose. Then, the Bundle Adjustment (BA) algorithm is used to further optimize the rough pose obtained by the EPnP algorithm, so that a more accurate pose can be obtained. This precise pose can be used to re-match the points in the matching pair where the reprojection error exceeds a preset deviation threshold and update the matching relationship of the newly observed LEDs. Finally, the sensor fusion algorithm based on the extended Kalman filter can be used to fuse the 6-DOF pose obtained above with the 3-DOF pose in the IMU to obtain a smoother and more consistent 6-DOF pose (that is, the fused pose 6DOF pose). In addition, the fused 6-DOF pose can also be used to re-match the points whose re-projection error exceeds the preset deviation threshold and update the newly observed LED matching relationship.

The pose tracking method according to the exemplary embodiment of the present disclosure reduces the dependency of the pose tracking on the specific layout of the LED markers, reduces the delay of the pose tracking, and improves the accuracy and efficiency of the pose tracking.

The DVS camera used to obtain the pixels whose brightness changes caused by the movement of the tracked object can be, for example, Samsung's third-generation VGA device, whose resolution is 640*480, and can be connected to the host through USB3.0. DVS cameras can generate events for pixels whose relative light intensity changes. Each event is represented by a tuple <t,x,y,p>, where t is the timestamp of the event (microsecond resolution), x,y are the pixel coordinates corresponding to the event, p∈{0, 1} is the polarity of the event, in which, when the LED light is on, the event of <t,x,y,1> will be generated, and when the LED light is off, the event of <t,x,y,0> will be generated .

The pose tracking method needs to apply some fixed parameters (camera internal parameters) and transfer matrix (IMU to handle body, DVS camera to helmet, etc.). In the exemplary embodiment of the present disclosure, the OptiTrack optical motion tracking system is used to calibrate these fixed parameters and transition matrices, and the OptiTrack optical motion tracking system is also used to provide the true value of the handle pose to evaluate the accuracy of the pose tracking. When the calibration is completed, the OptiTrack optical motion tracking system is not required during the actual use of the user.

是手柄模型坐标系到相机坐标系的旋转矩阵，^CP_M是模型坐标系原点在相机坐标系下的表示。需要提前标定DVS相机的内参和一些固定的转移矩阵 There are multiple coordinate systems in the OptiTrack optical motion tracking system, such as camera (C) coordinate system, helmet (H) coordinate system, world (W) coordinate system, IMU (I) coordinate system, handle body (B) coordinate system, The handle model (M) coordinate system and the OptiTrack (O) coordinate system. In order to simplify the OptiTrack optical motion tracking system, the handle body coordinate system can be aligned with the handle model coordinate system, and the world coordinate system can be aligned with the OptiTrtack coordinate system. So the 6-DOF pose of the motion handle to be solved can be expressed as

is the rotation matrix from the handle model coordinate system to the camera coordinate system, and ^C P _M is the representation of the origin of the model coordinate system in the camera coordinate system. The internal parameters of the DVS camera and some fixed transfer matrices need to be calibrated in advance

Since the optical properties of DVS are the same as those of ordinary CMOS cameras, standard pinhole imaging models can be used to determine camera intrinsics (eg, focal length, projection center, and distortion parameters). The difference between a DVS camera and a normal camera is that a DVS camera can't see things without lighting changes, so a flashing checkerboard can be used to calibrate the DVS.

也可以通过那些特定模式的闪烁LED来确定：

其中，

和

都由OptiTrack光学运动跟踪系统来提供，可以通过用EPnP算法计算得到

(特定模式的点集匹配关系是固定的)。计算得到的两个矩阵例如可如下所示：Some fixed transfer matrices can be calibrated with OptiTrack beads. In the 3D model, the origin of the handle model is the center of the great circle at the top of the handle. The OptiTrack ball can be fixed on the circumference of the large circle, and the directions of the X, Y, and Z axes in the marked model can be correspondingly marked, so that the model coordinate system of the motion handle can be determined in the OptiTrack optical motion tracking system. Fix the OptiTrack ball on the IMU chip, and use the readings of the IMU to determine the directions of the X, Y, and Z axes, so that the IMU coordinate system is determined in the OptiTrack optical motion tracking system. The OptiTrack optical motion tracking system can record the pose of the IMU coordinate system and the model coordinate system in real time, and then the conversion relationship between the model coordinate system of the motion handle and the IMU coordinate system can be calculated. And the helmet to camera transformation matrix

It can also be determined by those specific patterns of blinking LEDs:

in,

and

All are provided by OptiTrack optical motion tracking system, which can be calculated by EPnP algorithm

(The point set matching relationship for a specific pattern is fixed). The calculated two matrices can be, for example, as follows:

The active LED marker blinking interval can be determined by on-off-on, off-on-off events. If the flicker interval of a pixel is in the [800μs, 1200μs] interval, then it can be concluded that the event is caused by LED flickering. A region growing algorithm and voting method are then used to determine the center position of the LEDs. Multiple LED lights can be tracked using global nearest neighbors and Kalman filtering. Figure 2 shows the 2D active LED marker tracking results of the tracked object. In FIG. 2 , each continuous line represents the trajectory of a marker (eg, LED), the small open circle represents the starting point of the trajectory, and the large circle with a solid circle inside represents the end point of the trajectory.

In addition, in order to improve the performance of the BA algorithm, the optimization window of the BA algorithm can be set to 10, and the optimization is performed only every 4 frames. Instead of updating the matching relationship every time, the matching relationship is updated only when the ratio of the number of matched LED lights to the number of all observed LED lights is less than a preset value such as 0.6, thereby improving the performance of the BA algorithm. efficiency. After the startup phase is initialized, the entire processing flow takes only 1.23 milliseconds. So adding the filter time for the LED flashing event (1ms), the total delay is 2.23ms.

The pose tracking method according to the exemplary embodiment of the present disclosure has been described above with reference to FIGS. 1 to 2 . Hereinafter, a pose tracking apparatus and a unit thereof according to an exemplary embodiment of the present disclosure will be described with reference to FIG. 3 .

FIG. 3 shows a block diagram of a pose tracking apparatus according to an exemplary embodiment of the present disclosure.

Referring to FIG. 3 , the posture tracking device includes an image acquisition unit 31 , a pixel acquisition unit 32 and a posture calculation unit 33 .

The image acquisition unit 31 is configured to acquire an image of the tracked object, wherein the tracked object is provided with a marker that blinks at a specific frequency.

The pixel acquisition unit 32 is configured to acquire pixels of varying brightness from the acquired image.

The pose calculation unit 33 is configured to calculate the 6-DOF pose of the tracked object based on the acquired pixels.

In an exemplary embodiment of the present disclosure, the attitude calculation unit 33 may be configured to: acquire inertial measurement unit data of the tracked object, and estimate the 3-DOF pose of the tracked object based on the acquired inertial measurement unit data, here, the 3-DOF pose It is the posture of rotating around the three coordinate axes of x, y, and z in the body coordinate system of the tracking object; based on the 3-DOF posture and the acquired pixels, the 6-DOF posture of the tracking object is calculated. Here, the 6-DOF posture is tracking The posture of the object along the three coordinate axes of x, y, and z in the body coordinate system of the object and the posture of rotating around the three rectangular coordinate axes of x, y, and z.

In an exemplary embodiment of the present disclosure, the pose calculation unit 33 may be further configured to: based on the 3-DOF pose and the acquired pixels, solve the correspondence between the marked 2D point set and the 3D point set, and obtain the marked 2D point The matching pair between the set and the 3D point set. Here, the 2D point set includes the pixel coordinates of each marker, and the 3D point set includes the coordinates of each marker in the body coordinate system of the tracked object; based on the matching pair, calculate the 6-DOF pose of the tracked object .

In an exemplary embodiment of the present disclosure, the pose calculation unit 33 may be further configured to: remove the pixels whose reprojection deviation exceeds a preset deviation threshold from the matching pair, and calculate the 6-DOF pose according to the pixels remaining after the removal; Minimize the reprojection error of the calculated 6-DOF pose to obtain the 6-DOF pose of the tracked object.

In an exemplary embodiment of the present disclosure, the pose calculation unit 33 may be further configured to: remove the pixels whose reprojection deviation exceeds a preset deviation threshold from the matching pair, and calculate the 6-DOF pose according to the pixels remaining after the removal; The calculated 6-DOF pose is performed to minimize the reprojection error; the 6-DOF pose after minimizing the re-projection error is optimized according to the 3-DOF pose, and the 6-DOF pose of the tracked object is obtained.

In an exemplary embodiment of the present disclosure, the pose tracking apparatus may further include: a re-matching unit configured to, after obtaining the 6-DOF pose of the tracked object, reproject the error according to the 6-DOF pose of the tracked object Pixels that exceed a preset deviation threshold are rematched.

FIG. 4 shows a schematic diagram of an electronic device according to an exemplary embodiment of the present disclosure.

Referring to FIG. 4 , the electronic device 4 includes a camera 41 and a processor 42 .

Wherein, the camera 41 is used to acquire an image of the tracking object, and acquires pixels whose brightness changes from the acquired image, wherein the tracking object is provided with a mark that flickers with a specific frequency. The processor 42 is configured to calculate the 6-DOF pose of the tracked object based on the pixels acquired by the camera.

In an exemplary embodiment of the present disclosure, the processor 42 may be configured to acquire inertial measurement unit data of the tracked object, and to estimate the 3-DOF pose of the tracked object based on the acquired inertial measurement unit data, where the 3-DOF pose is the tracking object The posture of rotating around the three coordinate axes of x, y, and z in the body coordinate system of the The attitude of the system along the three coordinate axes of x, y, and z and the attitude of rotation around the three rectangular coordinate axes of x, y, and z.

In an exemplary embodiment of the present disclosure, the processor 42 may be configured to solve the correspondence between the marked 2D point set and the 3D point set based on the 3-DOF pose and the acquired pixels, and obtain the marked relative 2D point set and 3D point set Here, the 2D point set includes the pixel coordinates of each marker, and the 3D point set includes the coordinates of each marker in the body coordinate system of the tracked object; based on the matching pairs, the 6-DOF pose of the tracked object is calculated.

In an exemplary embodiment of the present disclosure, the processor 42 may be configured to remove pixels whose reprojection deviation exceeds a preset deviation threshold from the matching pair, and calculate a 6-DOF pose according to the remaining pixels after removal; The DOF pose is performed to minimize the reprojection error, and the 6 DOF pose of the tracked object is obtained.

In an exemplary embodiment of the present disclosure, the processor 42 may be configured to remove pixels whose reprojection deviation exceeds a preset deviation threshold from the matching pair, and calculate a 6-DOF pose according to the remaining pixels after removal; The DOF pose is operated to minimize the reprojection error; the 6DOF pose after minimizing the reprojection error is optimized according to the 3DOF pose, and the 6DOF pose of the tracked object is obtained.

In an exemplary embodiment of the present disclosure, the processor 42 may be configured to re-match pixels whose reprojection error exceeds a preset deviation threshold according to the 6-DOF pose of the tracked object after obtaining the 6-DOF pose of the tracked object .

In addition, according to an exemplary embodiment of the present disclosure, there is also provided a computer-readable storage medium on which a computer program is stored, and when the computer program is executed, implements the pose tracking according to the exemplary embodiment of the present disclosure method.

As an example, the computer-readable storage medium may carry one or more programs, and when the computer program is executed, the following steps may be implemented: acquiring an image of a tracking object, wherein the tracking object is provided with flashing lights with a specific frequency. mark; obtain pixels with varying brightness from the acquired image; calculate the 6DOF pose of the tracked object based on the acquired pixels.

The computer readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or any combination of the above. More specific examples of computer readable storage media may include, but are not limited to, electrical connections with one or more wires, portable computer disks, hard disks, random access memory (RAM), read only memory (ROM), erasable Programmable read only memory (EPROM or flash memory), fiber optics, portable compact disk read only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing. In embodiments of the present disclosure, a computer-readable storage medium may be any tangible medium that contains or stores a computer program that can be used by or in conjunction with an instruction execution system, apparatus, or device. A computer program embodied on a computer-readable storage medium may be transmitted using any suitable medium including, but not limited to, electrical wire, optical fiber cable, RF (radio frequency), etc., or any suitable combination of the foregoing. The computer-readable storage medium may be included in any apparatus; it may also exist alone without being incorporated into the apparatus.

The pose tracking device and the electronic device according to the exemplary embodiments of the present disclosure have been described above with reference to FIGS. 3 and 4 . Next, a computing device according to an exemplary embodiment of the present disclosure will be described with reference to FIG. 5 .

5 shows a schematic diagram of a computing device according to an exemplary embodiment of the present disclosure.

Referring to FIG. 5 , a computing device 5 according to an exemplary embodiment of the present disclosure includes a memory 51 and a processor 52 , the memory 51 stores a computer program, and when the computer program is executed by the processor 52 , realizes the implementation according to the present disclosure Pose tracking method of an exemplary embodiment of .

As an example, when the computer program is executed by the processor 52, the following steps can be implemented: acquiring an image of the tracking object, wherein the tracking object is provided with a mark that flickers at a specific frequency; acquiring pixels with varying brightness from the acquired image ; Calculate the 6DOF pose of the tracked object based on the acquired pixels.

The computing device shown in FIG. 5 is only an example, and should not impose any limitation on the function and scope of use of the embodiments of the present disclosure.

The pose tracking method and apparatus according to the exemplary embodiments of the present disclosure have been described above with reference to FIGS. 1 to 5 . However, it should be understood that the pose tracking device and its unit shown in FIG. 3 may be configured as software, hardware, firmware or any combination of the above to perform specific functions, respectively, and the computing device shown in FIG. It is not limited to include the components shown above, but some components may be added or deleted as needed, and the above components may also be combined.

According to the pose tracking method and device according to the exemplary embodiments of the present disclosure, by acquiring an image of the tracking object, acquiring pixels with brightness changes from the acquired image, and calculating the 6-DOF pose of the tracking object based on the acquired pixels, thereby reducing the The dependence of pose tracking on the specific layout of LED markers, while reducing the latency of pose tracking, and improving the accuracy and efficiency of pose tracking.

Although the present disclosure has been particularly shown and described with reference to exemplary embodiments thereof, it will be understood by those skilled in the art that form and detail may be made therein without departing from the spirit and scope of the present disclosure as defined by the following claims various changes on.

Claims (10)

1. A pose tracking method, comprising:

acquiring an image of a tracking object, wherein a mark which flickers at a specific frequency is arranged on the tracking object;

acquiring pixels with changed brightness from the acquired image;

a 6 degree of freedom pose of the tracked object is calculated based on the acquired pixels.

2. The pose tracking method of claim 1, wherein the step of calculating a 6 degree of freedom pose of the tracked object based on the acquired pixels comprises:

acquiring inertial measurement unit data of the tracked object, and estimating a 3-degree-of-freedom attitude of the tracked object based on the acquired inertial measurement unit data, wherein the 3-degree-of-freedom attitude is an attitude rotating around three coordinate axes of x, y and z under a body coordinate system of the tracked object;

and calculating a 6-degree-of-freedom attitude of the tracked object based on the 3-degree-of-freedom attitude and the acquired pixels, wherein the 6-degree-of-freedom attitude is an attitude along three coordinate axis directions of x, y and z and an attitude rotating around three rectangular coordinate axes of x, y and z under a body coordinate system of the tracked object.

3. The pose tracking method of claim 2, wherein the step of calculating a 6 degree of freedom pose of the tracked object based on the 3 degree of freedom pose and the acquired pixels comprises:

based on the 3-degree-of-freedom posture and the obtained pixels, solving a corresponding relation between a 2D point set and a 3D point set of the marker to obtain a matching pair of the 2D point set and the 3D point set of the marker, wherein the 2D point set comprises pixel coordinates of the marker, and the 3D point set comprises coordinates of the marker in a body coordinate system of the tracked object;

based on the matching pairs, a 6 degree of freedom pose of the tracked object is calculated.

4. The pose tracking method of claim 3, wherein the step of calculating a 6 degree of freedom pose of the tracked object comprises:

removing pixels with the reprojection deviation exceeding a preset deviation threshold value from the matching pair, and calculating a pose with 6 degrees of freedom according to the remaining pixels after removal;

and carrying out minimum reprojection error operation on the calculated pose with 6 degrees of freedom to obtain the pose with 6 degrees of freedom of the tracked object.

5. The pose tracking method of claim 3, wherein the step of calculating a 6 degree of freedom pose of the tracked object comprises:

removing pixels with the reprojection deviation exceeding a preset deviation threshold value from the matching pair, and calculating a pose with 6 degrees of freedom according to the remaining pixels after removal;

performing minimum reprojection error operation on the pose with 6 degrees of freedom obtained by calculation;

and optimizing the pose of 6 degrees of freedom after the operation of minimizing the reprojection error according to the pose of 3 degrees of freedom to obtain the pose of 6 degrees of freedom of the tracked object.

6. The pose tracking method according to claim 4 or 5, further comprising, after obtaining the 6 degree-of-freedom pose of the tracked object:

and carrying out re-matching on the pixels of which the re-projection errors in the matching pairs exceed a preset deviation threshold according to the 6-degree-of-freedom posture of the tracked object.

7. A pose tracking apparatus, comprising:

an image acquisition unit configured to acquire an image of a tracking object on which a marker that blinks at a specific frequency is provided;

a pixel acquisition unit configured to acquire a pixel of which luminance varies from the acquired image; and

a pose calculation unit configured to calculate a 6-degree-of-freedom pose of the tracking object based on the acquired pixels.

8. An electronic device, comprising:

a camera for acquiring an image of a tracking object on which a marker blinking at a specific frequency is provided, and acquiring a pixel of which luminance varies from the acquired image;

a processor to calculate a 6 degree of freedom pose of the tracked object based on the pixels acquired by the camera.

9. A computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the pose tracking method of any one of claims 1 to 6.

10. A computing device, comprising:

a processor;

a memory storing a computer program that, when executed by the processor, implements the pose tracking method of any one of claims 1 to 6.

Images