JP4656016B2 - Motion tracker device - Google Patents

Description

  The present invention relates to an optical motion tracker device (hereinafter also referred to as an MT device), and more particularly to an optical MT device having a function of detecting a current position and a current angle of an optical marker. The present invention relates to a head motion tracker device (hereinafter, the current head position and head angle) for detecting the current position and current angle of a helmet with a head-mounted display device used in, for example, game machines and vehicles (hereinafter, the current head position and head angle). , Also referred to as an HMT device).

  Here, the optical HMT device is a device in which a helmet equipped with an optical marker such as a reflector or a light emitter is attached to the head, and the position of the optical marker is measured with a camera device capable of stereoscopic viewing. A device that tracks the movement of the head.

  A technique for accurately measuring the current position and current angle of an object that changes from moment to moment is used in various fields. For example, in a game machine, an image is displayed by using a helmet with a head-mounted display device in order to realize virtual reality (VR). At this time, it is necessary to change the image in accordance with the current position and the current angle of the head mounted display-equipped helmet. Therefore, in order to measure the current position and current angle of the helmet with a head-mounted display device, an HMT device is used.

  Also, in the rescue operation by the rescue flying boat, in order not to lose sight of the found rescue target, by locking when the aiming image displayed by the helmet with head mounted display device corresponds to the rescue target, Calculating the position of the locked rescue target has been done. At this time, in order to calculate the position of the rescue target, in addition to the latitude, longitude, altitude, and attitude of the flying object, the head angle and head position of the pilot with respect to the relative coordinate system set for the flying object are measured. ing. At this time, the HMT device is used.

As an HMT device used for a helmet with a head-mounted display device, an apparatus that optically measures the current position and current angle of the helmet with a head-mounted display device is disclosed (for example, see Patent Document 1). ). For example, an optical HMT device is disclosed in which a plurality of reflectors are attached to a helmet with a head-mounted display device, and reflected light when light is emitted from a light source is monitored by a camera device. There is also an optical HMT device that the applicant has applied for earlier (Japanese Patent Application No. 2005-106418). Specifically, on the outer peripheral surface of the helmet with a head-mounted display device, as an optical marker group, LEDs as light emitters are attached at three locations so as to be separated from each other, and the relative of these three optical markers is relative to each other. The positional relationship is stored in advance in the HMT apparatus. Then, these three optical markers can be stereoscopically photographed simultaneously with two cameras that can be viewed in stereo and fixed in place, so that the present three optical markers can be obtained according to the so-called triangulation principle. The relative position of the marker is measured. If the position of the three points (position of three LEDs) fixed to the head-mounted display-equipped helmet can be specified, the position and orientation (angle) of the head-mounted display-equipped helmet can be specified. The movement distance amount and the movement angle amount of the helmet with a head-mounted display device for two cameras are calculated.
JP-T 9-506194

  In order to measure the relative positional relationship of the three optical markers with the optical HMT apparatus as described above, it is necessary to identify each of the three optical markers. Therefore, three optical markers that can be individually identified are attached to the head-mounted helmet with a display device. For example, on the outer peripheral surface of a helmet with a head-mounted display device, LEDs that emit infrared light having different wavelengths are attached as three optical markers so as to be separated from each other, and each of these three LEDs Are previously stored in the HMT apparatus. Then, the current positions of the three current LEDs are measured by identifying these three LEDs by the wavelength difference.

  However, when three optical markers that can be identified are attached, when one optical marker breaks down, etc., one failed optical marker is replaced with one new optical marker, and a new one It is necessary to store the identification information (for example, wavelength information) of one optical marker in the HMT apparatus anew. Therefore, it takes time and effort to store the identification information of the new optical marker in the HMT device every time it is replaced with a new optical marker.

  In addition, it is conceivable to identify the three optical markers having no identification information by lighting them one by one in order, but they must be turned on one by one. It is difficult to monitor the movement of a helmet with a head mounted display device that changes.

  Accordingly, the present applicant identifies the LEDs 57a, 57b, and 57c when identifying the current positions of the three LEDs 57a, 57b, and 57c attached to the object 50 such as the head-mounted display-equipped helmet. We examined a method that can identify the current position of each of the LEDs 57a, 57b, and 57c without giving information or lighting them one by one. For example, as shown in FIG. 7, at time t1, the positions of the LEDs 57a, 57b, and 57c are stored, and a spherical shape having a fixed size centered on the stored positions of the LEDs 57a, 57b, and 57c. By setting a certain expected moving range (D), the LED existing in the expected moving range (D) at time t2 is the same as the LED corresponding to the expected moving range (D) set at time t1. Identified.

  However, as shown in FIG. 8, when the size (l1) of the set expected movement range (D1) is reduced, no LED exists in the expected movement range (D1) when the moving speed of the object 50 is high. There was a thing. On the other hand, as shown in FIG. 9, when the size (l2) of the set expected movement range (D2) is increased, two LEDs may exist simultaneously in the expected movement range (D2). That is, it may be impossible to identify an LED existing in the expected movement range (D) as being the same as the LED corresponding to the expected movement range (D).

  Therefore, the present invention provides the optical marker with identification information when identifying the current position of each of the three or more optical markers such as LEDs attached to an object such as a head-mounted display-equipped helmet. It is another object of the present invention to provide a motion tracker device that can reliably identify the current position of each optical marker without lighting them one by one.

  In order to solve the above problems, a motion tracker device according to the present invention includes three or more optical markers attached to an object, a camera device that detects light rays from the optical markers in stereoscopic view, and detected light rays. An optical marker position information calculation unit that calculates optical marker position information including the current position of each of the three or more optical markers, and based on the optical marker position information, the current position of the object with respect to the camera device A motion tracker device including a relative information calculation unit that calculates relative information including a position and a current angle, and an optical marker storage unit that stores at least two optical marker position information calculated at different times; The estimated movement position of the optical marker is estimated by comparing the two optical marker position information. And an optical marker estimator, said optical marker position information calculating unit, based on the expected movement position of the optical markers, and the three or more optical markers to identify respectively.

  According to the motion tracker device of the present invention, for example, the position of each of the three or more optical markers stored at time t1 is compared with the current position of each of the three or more optical markers stored at time t2. Thus, the expected movement position of the optical marker at time t3 is estimated. Further, for example, by setting an expected movement range centered on the estimated movement position of one estimated optical marker, the optical marker existing in the expected movement range at time t3 is moved to the expected movement set at time t2. It is identified as the same optical marker corresponding to the range. Therefore, even when the moving speed of the object is high, it is possible to prevent the optical marker from being present in the expected moving range.

(Means and effects for solving other problems)
In the above invention, the optical marker position information calculation unit identifies each of the three or more optical markers by setting an expected movement range of the optical marker based on an expected movement position of the optical marker. You may make it do.
In the above invention, the expected movement range of the optical marker may be a sphere centered on the expected movement position of one optical marker.

In the above invention, an expected movement range determination unit may be provided that determines the size of the expected movement range of the optical marker based on the expected movement position of the optical marker and the optical marker position information.
According to the present invention, by comparing the respective positions of the three or more optical markers stored at time t1 with the respective current positions of the three or more optical markers stored at time t2, the time t3 The estimated movement position of the optical marker at is estimated. At this time, the moving speed of the optical marker from time t1 to time t2 is also calculated. As a result, the size of the predicted movement range centered on the predicted movement position of one optical marker is increased, for example, when the movement speed of the optical marker is fast, while the movement speed of the optical marker is slow. Can be small. Therefore, even when the moving speed of the object is fast, it is possible to prevent the optical marker from being present in the expected moving range, and when the moving speed of the object is slow, two or more optical markers are simultaneously present in the expected moving range. It can be prevented from existing.

In the above invention, the optical marker estimator may estimate the predicted movement positions of the three optical markers based on the optical marker position information of the three optical markers.
According to the present invention, the rotational movement of the optical marker is also measured by comparing the positions of the three optical markers stored at time t1 with the current positions of the three optical markers stored at time t2. Therefore, it is possible to further prevent the optical marker from being present in the expected movement range.

  Furthermore, in the above invention, the object may be a helmet mounted on a passenger's head, and the camera device may be attached to a moving body on which the passenger rides.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. Note that the present invention is not limited to the embodiments described below, and it is needless to say that various aspects are included without departing from the spirit of the present invention.

(Embodiment 1)
FIG. 1 is a diagram showing a schematic configuration of an HMT device according to an embodiment of the present invention, and FIG. 2 is a plan view of the helmet with a head-mounted display device shown in FIG. In this embodiment, the current position and the current angle of the helmet with a head-mounted display device worn by the pilot on the flying object are calculated. That is, the HMT device 1 calculates relative information including the head position and head angle of the pilot 3 with respect to the relative coordinate system (XYZ coordinate system) set for the vehicle 30. The relative coordinate system (XYZ coordinate system) is based on a camera device 2 (2a, 2b) described later, and is stored in the relative coordinate storage unit 43.

  The HMT device 1 includes a head-mounted display-equipped helmet 10 attached to the head of the pilot 3, a camera device 2 (2a, 2b) attached to the ceiling of the vehicle 30, and a control constituted by a computer. Part 20.

The mounted body 30 is a cockpit of a flying body on which the pilot 3 is boarded, and includes a seat 30a on which the pilot 3 is seated.
The helmet 10 with a head-mounted display device includes a display (not shown), a combiner 8 that leads to the eyes of the pilot 3 by reflecting image display light emitted from the display, and a position and orientation (that is, LED group 7 that functions as a head marker serving as an index when measuring the head position and head angle. In addition, the pilot 3 wearing the helmet 10 with a head-mounted display device can visually recognize the display image by the display and the front actual thing of the combiner 8.

  In the LED group 7, as shown in FIG. 2, three (or three or more) LEDs 7a, 7b, 7c that emit infrared light or visible light having the same wavelength are separated from each other by a certain distance (d2). It is attached in this way. Therefore, the LEDs 7a, 7b, and 7c emit infrared light (visible light) having the same wavelength, and thus cannot identify each LED.

The camera device 2 (2a, 2b) is composed of two cameras 2a, 2b. The photographing direction is directed to the head-mounted helmet 10 with a display device, and the three-dimensional structure of the helmet 10 with a head-mounted display device. It is installed on the ceiling of the riding body 30 via the fixed shaft 2c so as to be separated by a certain distance (d1) that can be seen.
Therefore, as shown in FIG. 3, the position of the LED 7a with respect to the camera device 2 (2a, 2b) extracts the position of the head marker displayed in the image photographed by the camera device 2 (2a, 2b). Further, the direction angle (α) from the camera 2a and the direction angle (β) from the camera 2b are extracted, and the distance (d1) between the camera 2a and the camera 2b is used to calculate by the triangulation method. You can do that. The positions of the LEDs 7b and 7c, which are other head markers, with respect to the camera device 2 (2a and 2b) are calculated in the same manner.

  A relative coordinate system that is a coordinate system that is fixed to the camera device 2 (2a, 2b) and moves together with the camera device 2 so that the position of each head marker at this time can be expressed in spatial coordinates. (XYZ coordinate system) is used. A specific origin position of the relative coordinate system (XYZ coordinate system) and description of the XYZ axis directions will be described later. The position coordinates of the LEDs 7a, 7b, 7c can be expressed as (X1, Y1, Z1), (X2, Y2, Z2), (X3, Y3, Z3) by the relative coordinate system (XYZ coordinate system).

  As shown in FIG. 1, the control unit 20 is configured by a computer including a CPU 21, a memory 41, and the like, and performs various controls and arithmetic processes. The processing executed by the CPU 21 will be described separately for each functional block. The motion tracker drive unit 28, the relative information calculation unit 22, the optical marker position information calculation unit 24, the optical marker estimation unit 26, and the video display unit 25 will be described. And have.

  The memory 41 has an area for storing various data necessary for the control unit 20 to execute processing, a relative coordinate storage unit 43 that stores a relative coordinate system (XYZ coordinate system), A time storage unit 42, an optical marker storage unit 44 that stores optical marker position information including the positions of the three LEDs 7a, 7b, and 7c, and a predicted movement range that has a spherical shape with a diameter of (d2) / 2. And an expected movement range storage unit 45 for storing the size.

  The relative coordinate system (XYZ coordinate system) can arbitrarily determine the origin and the direction of each coordinate axis. In this embodiment, as shown in FIG. 3, the direction from the camera 2b to the camera 2a is the X-axis direction. And the vertical direction to the X axis direction, the vertical direction to the ceiling and the downward direction as the Z axis direction, the vertical direction to the X axis direction, the horizontal direction to the ceiling, and the right direction as the Y axis direction. Is set in the relative coordinate storage unit 43 so as to be defined as a midpoint. In addition, an angle (Θ) described later is an angle in the roll direction (rotation with respect to the X axis), an angle (Φ) is an angle in the elevation direction (rotation with respect to the Y axis), and the angle (Ψ) is azimuth. It is the angle of the direction (rotation with respect to the Z axis).

  The motion tracker driving unit 28 outputs a command signal for turning on the LED group 7 and controls the camera device 2 (2a, 2b) to detect image data of light rays emitted from the LED group 7. However, since the LEDs 7a, 7b, and 7c emit infrared light having the same wavelength, the LEDs cannot be identified. Therefore, each LED is identified by obtaining optical marker position information from the optical marker position information calculation unit 24 described later. Accordingly, the image data and the optical marker position information are stored and accumulated in the optical marker storage unit 44.

  The optical marker estimation unit 26 performs control for estimating the expected movement positions of the LEDs 7a, 7b, and 7c based on the optical marker position information and the image data. As shown in FIG. 4, for example, the expected movement position 17a of the LED 7a at the time t3 is estimated by comparing the position of the LED 7a stored at the time t1 with the current position of the LED 7a stored at the time t2. . That is, by calculating the movement amount from the position of the LED 7a stored at time t1 to the current position of the LED 7a stored at time t2, and adding the movement amount to the current position of the LED 7a stored at time t2, the time t3 The expected movement position 17a of the LED 7a is determined. Further, the predicted movement positions 17b and 17c of the LEDs 7b and 7c at the time t3 are also determined by adding the calculated movement amount to the current positions of the LEDs 7b and 7c stored at the time t2.

  The optical marker position information calculation unit 24 sets the predicted movement ranges (Da, Db, Dc) of the LEDs 7a, 7b, 7c based on the predicted movement positions 17a, 17b, 17c of the LEDs 7a, 7b, 7c. Thus, control is performed to calculate optical marker position information including the current positions of the LEDs 7a, 7b, and 7c. As shown in FIG. 5, first, at time t2, the predicted movement ranges (Da, Db, Dc) that are spherical around the respective predicted movement positions 17a, 17b, 17c of the LEDs 7a, 7b, 7c are represented in the XYZ coordinate system. Set to. Next, at time t3, the LED existing in the expected movement range (Da, Db, Dc) is the same as the LED corresponding to the expected movement range (Da, Db, Dc) set at time t2. Identify. For example, at time t3, the LED 7a existing in the expected movement range (Da) is identified as the same LED 7a (t2) corresponding to the expected movement range (Da) set at time t2. Similarly, the LED 7b existing in the expected movement range (Db) is the same as the LED 7b (t2), and the LED 7c existing in the expected movement range (Dc) is the same as the LED 7c (t2). Identify. In this way, optical marker position information including the current positions of the LEDs 7a, 7b, and 7c is calculated.

  Based on the optical marker position information and the image data, the relative information calculation unit 22 determines the head position (X, Y, Z) and head angle (Θ, Φ) of the pilot 3 with respect to the camera 2a and the camera 2b (XYZ coordinate system). , Ψ) is controlled to calculate relative information. Therefore, since the current positions of the three LEDs 7a, 7b, and 7c with respect to the camera devices 2a and 2b are specified, the position and angle (orientation) of the helmet 10 with head mounted display device to which the LEDs 7a, 7b, and 7c are fixed. Can be specified using the coordinate system. That is, the camera device 2 (2a, 2b) can be obtained by detecting the current position coordinates of the three LEDs 7a, 7b, 7c by detecting the infrared light emitted from the LED group 7 as the head marker. The current position and orientation (angle) of the helmet 10 with a head-mounted display device can be calculated, and can be expressed by the position coordinate and the angle with respect to the coordinate axis.

  The video display unit 25 performs control for emitting video display light from the display based on the relative information. As a result, the pilot 3 can visually recognize the display image displayed on the display.

  Next, the measurement operation for measuring the head position (X, Y, Z) and the head angle (Θ, Φ, Ψ) of the pilot 3 with respect to the relative coordinate system (XYZ coordinate system) by the HMT device 1 will be described. FIG. 6 is a flowchart for explaining the measurement operation by the HMT apparatus 1.

First, in the process of step S101, 0 is stored in the time storage unit 42 as time (t).
Next, in the process of step S102, the motion tracker drive unit 28 causes the camera device 2 (2a, 2b) to detect the image data of the LED group 7. At this time, the image data of the LED group 7 is stored in the optical marker storage unit 44 as initial image data.
Next, in the process of step S103, the relative head information calculation unit 22 determines the initial position (X, Y, Z) of the pilot 3 with respect to the camera device 2 (2a, 2b) (relative coordinate system) based on the initial image data. ) And initial angles (Θ, Φ, ψ).

  Next, in the process of step S104, at t = 0 (initial state), the respective expected movement ranges (Da, Db, Dc) of the LEDs 7a, 7b, 7c around the respective current positions of the LEDs 7a, 7b, 7c. ) Is set in the relative coordinate system (XYZ coordinate system).

Next, in the process of step S105, t = t + 1 is stored in the time storage unit 42 so as to be updated to +1 at time (t).
Next, in the process of step S106, the motion tracker drive unit 28 causes the camera device 2 (2a, 2b) to detect the image data of the LED group 7. At this time, the image data of the LED group 7 is stored in the optical marker storage unit 44.

  Next, in the process of step S107, the optical marker position information calculation unit 24 converts the LED existing in the expected movement range (Da, Db, Dc) to the expected movement range (Da, Db, Dc) set at time t. The optical marker position information including the current position of each of the LEDs 7a, 7b, and 7c is calculated by identifying that the LED is the same as the corresponding LED. At this time, the optical marker storage unit 44 stores optical marker position information including the current positions of the LEDs 7a, 7b, and 7c.

  Next, in the process of step S108, the relative head information calculation unit 22 determines the head position of the pilot 3 with respect to the camera device 2 (2a, 2b) (relative coordinate system) (based on the optical marker position information and the image data). X, Y, Z) and head angle (Θ, Φ, ψ) are calculated.

  Next, in step S109, it is determined whether the vehicle 30 is flying. When it is determined that the vehicle 30 is flying, this flowchart is terminated. On the other hand, when it is determined that the vehicle 30 is flying, the process proceeds to step S110.

  Next, in the process of step S110, the optical marker estimation unit 26 estimates the expected movement positions 17a, 17b, and 17c of the LEDs 7a, 7b, and 7c based on the optical marker position information and image data of t = t and t + 1. . At this time, by calculating the movement amount from the position of the LED 7a stored at time t to the current position of the LED 7a stored at time t + 1, and adding the movement amount to the current position of the LED 7a stored at time t + 1, The expected movement position 17a of the LED 7a at time t + 2 is estimated. Further, the predicted movement positions 17b and 17c of the LEDs 7b and 7c at the time t + 2 are determined by adding the calculated movement amount to the current positions of the LEDs 7b and 7c stored at the time t + 1.

  Next, in the process of step S111, the optical marker position information calculation unit 24, based on the expected movement positions 17a, 17b, and 17c of the LEDs 7a, 7b, and 7c, the respective expected movement ranges of the LEDs 7a, 7b, and 7c ( Da, Db, Dc) are set in the relative coordinate system (XYZ coordinate system).

  As described above, according to the HMT device 1, the respective positions of the LEDs 7a, 7b, 7c stored at the time t1 are compared with the current positions of the LEDs 7a, 7b, 7c stored at the time t2. Thus, the estimated movement positions 17a, 17b, and 17c of the LEDs 7a, 7b, and 7c at the time t3 are estimated. Furthermore, by setting an expected movement range (Da, Db, Dc) around the estimated movement positions 17a, 17b, 17c of the estimated LEDs 7a, 7b, 7c, the expected movement range (Da, Db, The LED existing in Dc) is identified as the same LED 7a, 7b, 7c corresponding to the expected movement range (Da, Db, Dc) set at time t2. Therefore, even when the movement speed of the helmet 10 with a head-mounted display device is fast, it is possible to prevent the LEDs from being present in the expected movement range (Da, Db, Dc).

(Embodiment 2)
In addition to the above, the HMT device may be configured to include an expected movement range determination unit that determines the size of the expected movement range of the LED based on the expected movement position of the LED and the optical marker position information.
According to the HMT apparatus of the second embodiment, by comparing the respective positions of the three LEDs stored at time t1 with the current positions of the three LEDs stored at time t2, the time t3 The expected movement position of the LED at is estimated. At this time, the expected movement range determination unit calculates the movement speed of the LED from time t1 to time t2, thereby calculating the size of the expected movement range centered on the expected movement position of one LED. When the speed is high, it is increased. On the other hand, when the moving speed of the LED is low, it is decreased. Therefore, even when the movement speed of the helmet 10 with a head-mounted display device is fast, it is possible to prevent the LED from being present in the expected movement range, and when the movement speed of the helmet 10 with a head-mounted display device is slow. Thus, it is possible to prevent two or more LEDs from being simultaneously present in the expected movement range.

(Embodiment 3)
The HMT device may be configured such that the above-described optical marker estimation unit estimates the expected movement position of the three LEDs based on the optical marker position information of the three LEDs.
According to the HMT device of the third embodiment, the rotational movement of the LEDs is also measured by comparing the positions of the three LEDs stored at time t1 with the current positions of the three LEDs stored at time t2. Therefore, it is possible to further prevent the LED from being present in the expected movement range.

  The HMT device of the present invention is used, for example, as a device that detects the current position and the current angle of a helmet with a head-mounted display device used in game machines, vehicles, and the like.

It is a figure which shows schematic structure of the HMT apparatus which is one Embodiment of this invention. It is a top view of the helmet with a head-mounted display device shown in FIG. It is a figure for demonstrating the setting of a relative coordinate system. It is a figure for demonstrating a movement of the helmet with a head mounting | wearing type display apparatus. It is a figure for demonstrating a movement of the helmet with a head mounting | wearing type display apparatus. It is a flowchart for demonstrating the measurement operation | movement by a HMT apparatus. It is a figure for demonstrating the movement of the conventional helmet with a head-mounted display apparatus. It is a figure for demonstrating the movement of the conventional helmet with a head-mounted display apparatus. It is a figure for demonstrating the movement of the conventional helmet with a head-mounted display apparatus.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Head motion tracker apparatus 2 Camera apparatus 3 Pilot 7 LED group 10 Helmet with a head-mounted display apparatus 22 Relative information calculation part 24 Optical marker position information calculation part 26 Optical marker estimation part 44 Optical marker memory | storage part

Claims (6)

Three or more optical markers attached to the object;
A camera device for detecting a light beam from the optical marker in a stereoscopic view;
An optical marker position information calculating unit that calculates optical marker position information including the current positions of the three or more optical markers based on the detected light beam;
A motion tracker device comprising a relative information calculation unit that calculates relative information including a current position and a current angle of an object with respect to the camera device based on the optical marker position information,
An optical marker storage unit for storing at least two optical marker position information calculated at different times;
An optical marker estimation unit that estimates the predicted movement position of the optical marker by comparing the at least two optical marker position information;
The motion tracker device, wherein the optical marker position information calculation unit identifies each of the three or more optical markers based on an expected movement position of the optical marker. The optical marker position information calculation unit identifies each of the three or more optical markers by setting an expected movement range of the optical marker based on an expected movement position of the optical marker. Item 2. The motion tracker device according to Item 1. The motion tracker device according to claim 1, wherein the predicted movement range of the optical marker is a sphere centered on the predicted movement position of one optical marker. The motion according to claim 2, further comprising an expected movement range determination unit that determines a size of an expected movement range of the optical marker based on the expected movement position of the optical marker and optical marker position information. Tracker device. The said optical marker estimation part estimates the estimated movement position of the said three optical marker based on the optical marker position information of three optical markers, The any one of Claims 1-4 characterized by the above-mentioned. The motion tracker device described. The object is a helmet to be worn on a passenger's head; and
The motion tracker device according to claim 1, wherein the camera device is attached to a moving body on which the passenger rides.

Images