JP3376207B2 - Eye gaze detecting apparatus and method - Google Patents

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a line-of-sight detection device and method for detecting a position indicated by a line of sight.

[0002]

2. Description of the Related Art When an operator works while looking at a monitor screen of a computer, there is a work of instructing a specific position on the screen, such as selecting a window displayed by a multi-window system. In a general computer at present, an operator operates a mouse to instruct a position on the screen.

On the other hand, an eye movement analysis device (Japanese Patent Publication No. 4-46570) proposes a device for indicating a position by detecting the line of sight of an operator. In this device, an operator attaches a sensor for detecting the line of sight and a sensor for detecting the posture of the head, and calculates the line of sight of the operator in the three-dimensional space from the outputs of both sensors. For this reason, there is a problem that the operator needs to wear two sensors on the head, which is a burden on the operator. The outputs of the two sensors detect the posture of the head and the direction of the line of sight with respect to the head, respectively, and calculate the direction of the line of sight by summing these results. There was a problem that the accuracy of the two sensors could not be expected due to the accumulated error.

In order to solve the above problems, "Non Intrusi
ve Gaze Tracking Using Artificial Neural Network "
Shumeet Baluja, Dean Pomerleau, Advances in Neural I
nformation Processing Systems 6 Cowan JD, Tesaur
o, G. & Alspector, J. (eds) MOrgan Kaufman Publishers,
In 1994, we proposed a device that estimates the direction of the line of sight by inputting an image of the vicinity of the eye into a neural network.
However, this device has a problem that it is affected by changes in the illumination state of the operating environment and changes in the position of the operator's head because all images near the eyes are used. In addition, there is a problem that it is necessary to train a neural network using a large number of images in advance in order to estimate the line-of-sight direction from information having an individual difference, which is an image in the area near the eyes.

Further, in the above method, since the operator's line-of-sight direction is calculated as the direction with respect to the camera, the positional relationship between the object indicated by the operator's line-of-sight and the camera is fixed and cannot be moved during operation. In order to detect the line of sight by allowing the operator's head to move while the camera is fixed, it is necessary to widen the line-of-sight range of the camera. However, widening the field of view lowers the resolution per pixel. There was a problem that the accuracy was also reduced.

[0006]

As described above, in the conventional method in which a sensor is attached to the operator when detecting the position indicated by the line of sight by the operator,
There was a problem of imposing a load on the operator. Also,
In the method that detects the position indicated by the line of sight from the image captured by the camera, it is used for estimating the direction of the line of sight by detecting features with individual differences, and because the direction of the camera is fixed and the resolution is low, etc. There was a problem that the measurement accuracy of was not sufficient.

In view of the above problems, the present invention provides an eye-gaze detecting device and method for easily and accurately detecting a position indicated by an eye-gaze by an operator on a display device such as a display.

[0008]

According to a first aspect of the present invention, there is provided image input means for capturing an image of a user, and eye detection means for detecting the eye of the user from the image captured by the image input means. A pupil detecting means for obtaining a position of a pupil in the eye detected by the eye detecting means, and a finger designated by the user
The three or more light emitting means, which are arranged on substantially the same plane as the target area and irradiate light toward the user, emit light together with an image taken in a state in which the three or more light emitting means emit light.
Before subtracting the image taken in the state where there is no
The reference point detecting means for detecting the reference points which are the respective reflection positions of the light emitted from the respective light emitting means on the eyeball of the user, and the three or more reference points detected by the reference point detecting means. /> and, based on the position of the pupil detected by the pupil detection unit, line of sight, characterized in that it consists of an instruction position detecting means for detecting an indication position of the user's line of sight in the instruction target area It is a detection device.

The line-of-sight detection device of the present invention photographs the reflection of the light emitted by the light emitting means on the eyeball and compares the position of the pupil in the photographed image with the reflection position of the light emitting means. To calculate the direction of the line of sight.

According to a second aspect of the present invention, the eye detection means compares the image photographed by the image input means with three or more reference points detected by the reference point detection means, and the image is compared. The line-of-sight detection device according to claim 1, wherein a region in the vicinity of the reflection position in is detected as an eye.

In the line-of-sight detection device according to claim 2, the eye detection means compares the image photographed by the image input means with three or more reflection positions detected by the reference point detection means. Since the area near the reflection position in the image is an eye, the position of the user's eye can be accurately and reliably determined.

According to a third aspect of the present invention, the pupil detecting means detects the center of gravity of the pupil region of the eye detected by the eye detecting means .
The position is detected, and the pointing position detecting means uses the positions of three or more reference points detected by the reference point detecting means and the center of gravity of the pupil region detected by the pupil detecting means to indicate the pointing target.
The line-of-sight detection device according to claim 1, wherein a position indicated by the line-of-sight in the area is detected.

In the line-of-sight detection device according to the present invention, the designated position detection means has three or more reference points of the reference point detection means .
From the point and the center of gravity of the pupil area of the pupil detecting means,
The position indicated by the line of sight can be accurately detected in the area. According to a fourth aspect of the invention, an image input step of capturing an image of the user, an eye detection step of detecting the user's eye from the image captured in the image input step, and the eye detection step detected in the eye detection step. A pupil detection step of obtaining the position of the pupil in the eye, and a pointing target area designated by the user
An image taken in a state in which three or more light emitting means, which are arranged on the same surface and emit light toward the user, emit light.
Difference processing by subtracting images taken in the state of not emitting light
And a reference point detection step of detecting a reference point which is each reflection position of the irradiation light of each of the light emitting means on the eyeball of the user, and three or more references detected in the reference point detection step . and the point, based on the position of the pupil detected by the pupil detection step, the line-of-sight detection, characterized in that consist of an instruction position detecting step of detecting the indicated position of the line of sight of the said in the instruction target area user Is the way.

According to a fifth aspect of the present invention, the eye detection step includes the image captured in the image input step,
Compares the three or more reference points detected by the reference point detection step, the line-of-sight detection method of claim 4, wherein the region near the location glare said in the image and detects that the eye Is.

According to a sixth aspect of the invention, the pupil detecting step is performed.
Is the pupil area in the eye detected in the eye detection step.
The position of the center of gravity of the area is detected, and the pointing position detection step includes three or more references detected in the reference point detection step .
The position of the point and the pupil region detected in the pupil detection step
The eye gaze detection method according to claim 4, wherein the position indicated by the eye gaze is detected in the instruction target area by using the center of gravity of .

[0016]

DETAILED DESCRIPTION OF THE INVENTION First Embodiment Hereinafter, a first embodiment relating to the present invention will be described with reference to FIG.

FIG. 1 is a block diagram showing the construction of the visual axis detecting device of the first embodiment. Also, according to this embodiment,
FIG. 2 shows a procedure of acquiring the position indication information of the line of sight.

(Control Unit 1) The control unit 1 receives an instruction information input request from the operator such as the operator pressing a mouse button or an instruction information input request from an operating system, an application program, etc. Output location information.

At this time, after receiving the instruction information input request, a series of control signals are generated to measure the pointed position and the pointed position information is output, or a control signal is generated at regular time intervals. The designated position is constantly measured, and when the designated information input request is received, the latest designated position information is output for the measured designated position information.

(Light Emitting Section 2) The light emitting section 2 is composed of a plurality of light emitting elements which emit light in synchronization with the instruction signal received from the image input section 3.

In the line-of-sight detection device in which the display surface of the monitor display is the designated area, this light-emitting element is shown in FIG.
A plurality of them are arranged at three or more positions on the same plane as the display surface of the monitor display instructed by the operator. In the figure, the shaded portion is the position of the light emitting element.

In the case where the display device is a device using the cathode ray tube on the left side of FIG. 3, the light emitting element is arranged around the cathode ray tube.

When the display device is a large display device in which cathode ray tubes or projection displays called video walls are arranged on a plane, a light emitting element is also arranged at the joint between the individual display devices. The light emitting element is
In order to make the operator dazzling, an LED element that emits infrared light is used, and this LED is arranged at the position of the element in FIG. Alternatively, a strobe device may be used as the light source, and light from the light source may be guided to the position of the light emitting element of FIG. 3 through an optical fiber through a filter that absorbs visible light and transmits infrared light to emit light. Although the display is flat in this example, the display surface may be curved.

The display of the monitor display may be used as the light emitting unit 2. In this case, for example, the pixels in a plurality of small areas of the monitor display are made to have high brightness, or
In the case of multi-window display, a state in which pixels in a known shape region having a predetermined area or more such as the frame of the window being displayed shown in FIG. When the window frame is used as the light emitting element, a plurality of window frames are not necessary.

(Image Input Section 3) The image input section 3 captures the operator's eyes with a TV camera,
It outputs to the reference point detection unit 4 and the pupil detection unit 5.

First, a control signal is sent to the light emitting unit 2 to photograph the operator's eyes while the light emitting unit 2 is emitting light.
When a strobe is used as a light source, the strobe is emitted while shooting an image.

Next, an image is taken while the light emitting section 2 does not emit light. It does not matter which of the two images is taken first. In addition, the shooting times of these two images are set to short intervals, for example, two consecutive images at the video rate.
Let it be a frame.

(Reference Point Detecting Section 4) The reference point detecting section 4 first causes the light emitting section 2 to emit light from the two images captured by the image input section 3 while the light emitting section 2 is emitting light. The difference processing is performed by subtracting the image taken in the state where it is not.

In the difference image obtained as a result of this processing, the value of the pixel in the portion where the light emitted from the light emitting unit 2 is reflected on the operator's eye takes a positive value, and the values of the other pixels are almost zero. Has become.

The image obtained as a result of the difference processing is compared with a threshold value and binarization processing is performed to obtain binary pixels by extracting pixels larger than the threshold value.

The obtained image is labeled for each adjacent pixel and divided into regions.

After that, the area in which the area of each area is within a certain range is extracted, and the barycentric coordinates of the extracted area are used as the coordinates of the reference point.

In the case of continuously measuring the designated position information, once the position of the reference point is detected by the above-mentioned difference processing, it is not necessary to perform the difference processing thereafter. Since it is considered that the position and orientation of the operator's head and the direction of the eyes do not change significantly between images that are continuous in time, tracking processing is performed on the reference point obtained in the previous processing result to detect the coordinates. I can go.

An image photographed with the light emitting section 2 emitting light is input, the image is binarized and labeled to detect a region having a certain area, and the barycentric coordinates of the extracted region are obtained. Then, for each reference point, the barycentric coordinate closest to the reference point coordinate detected in the previous processing by the reference point detection unit 4 is set as the coordinate of the next reference point. If the distance between the reference point coordinates detected in the previous processing and the detected area barycentric coordinates is larger than a predetermined threshold value, the reference point is not used in the subsequent processing. If the average value of all the reference points of the distance between the reference point coordinates detected in the previous processing and the detected area barycentric coordinates is larger than a predetermined constant value, the reference point coordinates are calculated using the above difference processing. calculate.

(Eye Area Detection Unit 5) The eye area detection unit 5 inputs the reference point coordinates from the reference point detection unit 4 and calculates a rectangle circumscribing the set of reference points. With the center of gravity of the rectangle as the center, the vertical and horizontal lengths of the rectangle are multiplied by a predetermined constant value to form an eye region.

(Image Input Control Unit 6) The image input control unit 6 controls the image input unit 3 to keep the photographing conditions of the eyes of the operator optimal. TV of image input unit 3
The camera is mounted on a pan / tilt head capable of rotating in a horizontal direction (pan) and rotating in a vertical direction (tilt). First, a set of reference points is input from the reference point detection unit 4. When the reference point is not detected in the input image, the platform is operated according to the flowchart shown in FIG. 5, and the space photographable by the camera is scanned to detect the reference point.

When the reference point is detected, the center position and area of the eye area are input from the eye area detecting unit 5. The image input unit 3 is T so that the eye region is at the center of the image captured by the image input unit 3 and the area is within a predetermined range.
Controls the orientation of the V-camera and the zoom ratio of the camera lens.
When continuously measuring the designated position information, the orientation of the camera is controlled so that the position of the center of gravity of the pupil measured in the previous process is at the center of the image.

(Pupil Detection Unit 7) The pupil detection unit 7 detects the position of the pupil from the eye region in the image. In this process, first, a convolution process of an image of the eye region and a mask image given in advance is performed. The mask image is shown in Figure 6.
In the image shown in (1), the pixel inside the circle of radius r is represented by -1, and the pixel outside the circle is represented by 1.

Next, the pixel values of the respective pixels of the image obtained by performing the convolution processing are compared, and the pixel having the highest value is taken out.

With the position of the pixel taking the highest value as the center,
Ten masks are arranged on the left and right circumferences of a circle having a radius r shown in FIG. 7, and step edge detection processing for dividing the inside and outside areas of the circle is performed in this mask.

After that, least square estimation is performed using the position coordinates of the edge position information detected in each mask and the equation of the ellipse to find the ellipse closest to the edge positions of the 10 points. The pupil detection unit 7 outputs the barycentric coordinates of the ellipse obtained by the above least square estimation.

(Pointed Position Detecting Section 8) The pointed position detecting section 8 calculates using the center of gravity of the pupil area and the coordinates of the reference point to find the position coordinates indicated by the line of sight in the pointing area.

In this example, the designated area is the display surface of the monitor display. The coordinates of the center of gravity of the pupil are M = (xm, y
m). The position of the i-th (1 ≦ i ≦ n) reference point Ri captured in the image is defined as (xi, yi). Among the combinations that select three reference points from the set of reference points, the combination in which the triangle formed by the three reference points includes M inside is selected. The three points that make up the triangle are Ra,
Rb and Rc.

At this time, the coordinates of the center of gravity of the pupil are

[Equation 1] Can be expressed as The parameters s and t of Equation 1 above
Is

[Equation 2] Can be found at.

At this time, the designated position (Xm, Ym) for the display device by the line of sight is calculated using the calibration data (Xi, Yi) of each reference point Ri described later.

[Equation 3] Decide.

Next, another calculation method will be described. The center of gravity of the coordinates of all the reference points Ri is (xo, yo).

[0047]

[Equation 4] Then, the relationship between x and X is calculated by using a certain 2 × 2 conversion matrix a ′ shown below.

[Equation 5] Have a relationship. This matrix a ′ is a pseudo-inverse matrix,

[Equation 6] Can be found at. Using a ′ , the position indicated by the line of sight from the center of gravity of the pupil is calculated by the following formula.

[0048]

[Equation 7] In the description of the light emitting unit 2 described above, it is stated that the light emitting elements are arranged on the same plane as the display plane, but the display may not be a plane as long as it does not affect the image captured by the camera. The position of the light emitting element does not have to be strictly on the same plane as the display surface.

In the image input section 3, the operator's eyeball is photographed in close-up, so in the above calculation, the projection of the reflection of the light emitting section 2 on the image plane photographed by the camera is the weak perspective projection. It is calculated assuming. This weak perspective projection is an approximation of the central projection, which is the projection of the actual camera, but the "THREE-DIMENSINA MODEL MATCHING FROMAN
UNC ONSTRAINED VIDEPOINT "Thompson, DWMundy, JL
IEEE proceedings of Robotics and Automation 1987
According to pp.208-220, if the size of the object photographed by the camera is 10% or less of the distance from the camera to the object, the approximation by weak perspective projection is effective. Usually, the distance from the operator's eyes to the display is about twice the maximum diameter of the display surface of the display. Since the human eye has a radius of about 15 mm, the width of the display surface of a 15-inch display is about 300 mm, and the distance from the eye to the display is 600 mm, it can be approximated by weak perspective projection. In this state, the displacement of the position of the light emitting element from the display plane is calculated.

In weak perspective projection, the projection axis from the eyeball to the camera image pickup surface can be approximated to be parallel to the camera optical axis (FIG. 8). In FIG. 8, the projection position of the reflection of the light emitting element arranged at the edge of the display on the camera image is γsin
(Θ), the position of the light emitting element is represented by 2 ltan (2θ).

At this time, θ and arctan (0.
5l / 2l) is almost equal. And arctan
(0.5l / 2l) is approximately 14 degrees.

It is assumed that the projection angle on the eyeball deviates by 2dθ as a result of the position of the light emitting element being dl away from the display plane.

[0053]

[Equation 8] Assuming that the camera of the image input unit 3 is capturing a range of the eyeball diameter of 30 mm with a resolution of 500 pixels, the deviation of 1 pixel in the position of the reflection of the light emitting element on the image is θ.
The deviation dθ corresponds to 0.24 degree. By applying this dθ to the above equation, dl = 0.07 is obtained. In other words, the position of the light emitting element has a deviation of 1 in the projected image if the distance from the display plane is within 7% of the display diameter.
Since it is less than pixels, it does not affect the measurement.

Therefore, the position of the light emitting element need not be in the same plane as the display plane. Also, the display surface is not flat and may be curved as long as it is 7% or less of the display diameter.

When the operator wears spectacles, the reference point detection unit 4 detects the reflection of the light emitted from the light emitting unit 2 on the eyeball and the reflection on the lens of the spectacles at the same time. Since there is a possibility that the subsequent processing will be erroneous, the following method will be used.

First, when the number of extracted reference points is larger than the number of light emitting elements, the reference point detection unit 4 finds the distances between the respective reference points and the barycentric coordinates of all the reference points, and the distances in descending order. Delete a fixed number of reference points.

The projection position of the reflection of the light emitting element arranged at the edge of the display on the camera image shown in FIG.
It is represented by sin (θ). Assuming that the lens surface of the spectacles is spherical, the radius γ ′ of this sphere is larger than the radius γ of the eyeball, so the reflection position on the spectacle lens is
It is distributed over a wider range than the position of reflection on the eyeball, and the distance from the barycentric coordinates of all reference points is also large.

Next, in the designated position detection unit 8, after obtaining the matrix a of the equation 6, the calibration data (X'i, Y'i) for each reference point Ri and the following reference point coordinates (x
′ I, y ′ i) Calculate the distance between the coordinates and the matrix a. A reference point whose calculated distance is larger than a predetermined threshold is deleted, and the matrix a is recalculated only with the remaining reference points.

At the reference point obtained from the reflection on the eyeball, the equation 6 is established. Therefore, if there is no measurement error, the result of applying the matrix a to the coordinates of the reference point and the calibration data should match. However, the result of applying the matrix a to the reflection on the spectacle lens does not match the calibration data. Therefore, by performing the above comparison, the reflection on the lens can be eliminated.

(Calibration Unit 9) The calibration unit 9 learns the parameters for pointing position calculation and stores the parameters. FIG. 9 is a flowchart of this calibration.

First, in front of the display, a mirror is placed at a position where the image of the light emitting element is projected on the image input by the image input unit 3.

Predetermined coordinates (Xi,
Display the figure in Yi). For example, the entire surface of the display is darkened and a white filled circle is displayed at the coordinates (Xi, Yi) (step 101).

The position (xi, yi) of the displayed figure in the image is detected from the image taken by the camera (step 102).

For detection, difference processing is performed between the photographed image and the image in which the entire surface of the display is previously darkened, and the difference image is binarized and labeled with a certain threshold value, and an area having a value equal to or greater than the threshold value is obtained. Then, the center of gravity of the area having the maximum area is taken as the position of the displayed figure in the image. The position of the coordinates displayed on the display is changed, and the above processing is repeated until a fixed number (here, k) is reached (step 103).

The position (Xi, Yi) on the display of the figure displayed for the k images and the position (x
i, yi),

[Equation 9] Is established. The matrix a is obtained from the above equation (step 104).

The matrix a is a pseudo-inverse matrix.

[Equation 10] Can be found at.

Next, the position (xri,
yri) is calculated by the reference point detection unit 4 (step 10).
5).

Using the detected coordinates and the matrix a, the calibration data (Xri, Yri) of the reference point is obtained as follows (step 106).

[0069]

[Equation 11] The calibration data obtained by the above process can be determined once when the light emitting unit 2 is arranged. Therefore, the calibration unit 9 is used when the eye gaze detection device according to the present invention is created or when the eye gaze detection device is attached to the display device. To create calibration data.

(Individual adjustment unit 10) Next, the individual adjustment unit 10 will be described.

The human pupil is not an exact circle, and the position of the optical center of the optical axis also varies from person to person due to individual differences. Therefore, the designated position detected by the designated position detection unit is the designated position measured from the image input from the image input unit 3, and the designated position intended by the operator may not match.

The individual adjusting unit 10 is a unit for matching the output of the system with the above-mentioned individual characteristics, and learns the conversion parameter from the output of the pointing position detecting unit 8 to the pointing position intended by the operator, and uses this parameter. The output of the pointed position detection unit is converted by the above to adjust the output of the system.

The calibration unit 9 measures and learns the difference between individual systems such as the arrangement of the light emitting unit 2. The individual adjustment unit 10 learns and adjusts the difference for each operator. In this way, by preparing the calibration unit 9 for learning the system difference and the individual adjustment unit for the influence of the individual difference of humans, each operator can learn the individual difference once, and thereafter, Even if the system is changed, the position data can be indicated by the line of sight using the calibration data of the new system and the individual adjustment parameters of the user.

FIG. 10 shows the configuration of the individual adjustment unit 10.

The individual adjustment parameter learning unit 11 displays the figure to be instructed on the display, and the correspondence between the output of the instructed position detection unit 8 and the display position of the figure when the operator gazes at this figure. Then, the conversion parameter to the designated position intended by each operator is learned from the output of the designated position detection unit 8. Pointed position information conversion unit 13
Uses the parameters learned by the individual adjustment parameter learning unit 11 to convert and output the input designated position information.

FIG. 11 shows the flow of processing in the individual adjustment parameter learning unit 11.

First, a graphic is displayed at a predetermined position (x'i, y'i) on the display. The display position is
They are arranged evenly on the display surface of the display (step 201).

The operator depresses a key on the keyboard, depresses a button on the mouse or the like while gazing at the displayed graphic. The individual adjustment unit 10 waits until the input of the keyboard or the mouse button is detected (step 202).

After the above steps are completed, the designated position detecting unit 8
Then, the designated position (xi, yi) by the line of sight is input (step 203).

The processing from step 201 to step 203 is repeated until the number of designated positions reaches the constant number k (step 204).

Next, the display position (x 'on the display
i, y'i) and the position indicated by the line of sight (xi, yi) are subjected to least squares estimation by applying the model expressed by the following equation. For the number n in the equation, a predetermined value such as 2 is used (step 205).

[0082]

[Equation 12] In the least-squares estimation, the parameters aj and bj representing the model
Among (j = 0..n), aj (j = 0..n) is obtained as a solution of the following equation.

[0083]

[Equation 13] The parameter bj (j = 0..n) can also be calculated as a solution of the same formula as aj.

Next, the residual mean obtained by the least squares estimation in step 205 is compared with a predetermined threshold value (step 206). When the residual mean is smaller than the threshold, the parameters aj, bj (j
= 0. ． n) is output to the individual adjustment unit 10 as a learning parameter.

When the residual mean by the least squares estimation is larger than a predetermined threshold value, the processing from step 201 is performed again.

Next, another example of the configuration of the individual adjustment parameter learning section 11 is shown in the figure. Display position on display (x '
It is assumed that there is a relation of the following formula between i, y'i) and the position (xi, yi) indicated by the line of sight. For the number n in the equation, a predetermined value such as 2 is used.

[0087]

[Equation 14] First, a1 = 1 and b1 = 1 are set as initial estimated values of the parameters aj and bj (j = 0..n), and 0 is set to the other parameters (step 301).

A predetermined position (x 'on the display
A graphic is displayed on i, y'i), and the operator prompts the user to pay attention to the displayed graphic (step 302).

The designated position (xi, yi) by the line of sight is input from the designated position detection unit 8 (step 303).

Positions (xi, yi) and (x'i, y'i)
Parameters aj and bj (j = 0..n) are estimated (step 304). For this parameter estimation, for example, "Digital Signal Processing and Control" by Hideki Kimura, pp.
The Kalman filter technique described in 212 can be used.

The Kalman filter is a model expressed by the following equation, from the observation data y [t] to the state vector x
This is an algorithm for estimating [t].

[0092]

[Equation 15] Position of the figure on the t-th display (x'i, y '
i) and the pointed position (xt, yt) by the line of sight input from the pointed position detection unit 8, the parameters aj, bj (j
= 0. ． n), the state vector x [t]

[Equation 16] Express with. In addition, the transition matrix A, the measurement matrix C, and the observation data y [t] are represented by the following formulas.

[0093]

[Equation 17] The state matrix is estimated using the above matrix. A predetermined constant matrix is given to the error covariance matrices Q [t] and R [t] used for estimation.

Next, the matrix C [t] obtained from the estimated state vector x and the output (xt, yt) of the pointed position detector 8
Using,

[Equation 18] And the graphic display positions of (x't, y't) are changed (step 305).

It is determined whether or not the distance between the previous display position and the newly estimated position is smaller than a predetermined threshold value. If it is smaller than the threshold value, the process ends (step 306).

It is determined whether the newly estimated position is out of the display range of the display. If it is not out of the display range, a graphic is displayed, and the processing from step 302 is repeated. If the estimated position is out of the display range, the display position is returned to the initial value and the processing from step 301 is repeated (step 30).
7).

The above processing is repeated several times while changing the position where the initial value is displayed on the display. For example, the initial position for displaying a graphic on the display is changed in order to five positions shown in FIG. 13, starting from each of the initial positions and ending the learning of the individual adjustment parameter when the processing of the step is completed, The parameters aj and bj (j = 0..n) are output to the individual adjustment unit 10.

[0098] [Second Embodiment] Next, a configuration example of the second embodiment. In this configuration example,
The pointed position detection unit 8, the individual adjustment unit 10, and the individual adjustment parameter learning unit 11 are different from the configuration example of FIG.

(Pointed position detector 8) First, the configuration of the pointed position detector 8 will be described.

The coordinate of the input pupil centroid is M = (xm, y
m). The position of the i-th (1 ≦ i ≦ n) reference point Ri captured in the image is set to (xi, yi), and the calibration data is set to (Xi, Yi).

In equation 10, (xi, yi) is changed to (Xi, Yi)
The matrix a to be converted to is obtained. For an image I having a constant size centered on the center of gravity of the pupil, the brightness of the pixel at the position (x, y) is represented by I (x, y). Mapping is performed on the image obtained by subjecting this image to the affine transformation represented by the matrix a. The image I is projected by the affine transformation onto another image I ′ according to the following equation. Calibration data (X
Since i, Yi) does not change depending on the image, the light emitting unit 2 located at (xi, yi) in the image before projection is always projected at the position of the calibration data (Xi, Yi).

[0102]

[Formula 19] FIG. 14 shows an example of the affine map. In the mapped image, edge extraction is performed on a predetermined straight line to obtain the position of the contour edge of the projected pupil region. FIG. 15 is an example of a straight line that performs edge extraction by the above method. Edge extraction is performed on this straight line. For edge extraction, first, a pixel on a straight line is set to I (x), and a convolution operation of a stepwise edge extraction filter is performed. A local maximum value of the convolution result is extracted, and a value larger than a predetermined threshold value is extracted therein. If there is a value larger than the threshold value, the position where the edge filter takes the maximum value and the position where the edge filter takes the next largest value are taken as the edge positions.

The pointed position detection unit 8 outputs the position of the edge on each straight line to the individual adjustment unit 10.

(Individual Adjustment Unit 10) The individual adjustment unit 10 uses the parameters learned by the individual adjustment parameter learning unit 11 to convert the output of the indicated position detection unit 8 into the indicated position intended by the operator. There are n straight lines from which the pointed position detection unit 8 extracts edges, and the positions of the edges input from the i-th straight line are xli and xri. The designated position (xp, yp) output at this time is

[Equation 20] Calculate with. Parameters ai and bi used in this calculation
(I = 0 to 2n) is obtained by the individual adjustment parameter learning unit.

(Individual Adjustment Parameter Learning Unit 11) The individual adjustment parameter learning unit 11 displays the figure to be instructed on the display and outputs the output of the instructed position detecting section 8 when the operator gazes at the figure. , The correspondence of the display position of the figure is given, and the conversion parameter from the output of the pointing position detecting unit 8 to the pointing position intended by each operator is learned.

The flow of the processing of the individual adjustment parameter learning unit 11 is the same as that of FIG. 12 described above, and therefore will be described using this figure.

First, the parameters aj and bj (j = 0 ..
2n) has an initial estimated value of a1 = 1, b1 = 1, and 0 for other parameters (step 301).

A predetermined position (x 'on the display
A graphic is displayed on i, y'i), and the operator prompts the user to pay attention to the displayed graphic (step 302).

Edge position xl from the pointed position detector 8
Input i, xri (i = 1..n) (step 30)
3).

Edge positions xli, xri (i = 1 ..
n) and the display position (x'i, y'i) of the figure, the parameters aj and bj (j = 0.0.2n) are estimated using the Kalman filter method as in step 304 (step 304).

Position of the figure on the t-th display (x
'T, y't) and the edge positions xli, xri (i = 1..n) input from the pointed position detection unit 8 are state vectors x expressing parameters aj, bj (j = 0.0.2n). [T]

[Equation 21] And Moreover, the transition matrix A, the measurement matrix C, and the observation data y
[T] is represented by the following formula.

[0112]

[Equation 22] The state matrix is estimated using the above matrix. Regarding the error covariance matrices Q [t] and R [t] used for estimation, Q [t] gives a predetermined constant matrix. R
A matrix corresponding to the reliability of the edge extraction result at the corresponding edge position xli or xri is given to [t].

Next, using the estimated state vector x and the matrix C [t] obtained from the outputs xli, xri (i = 1..n) of the designated position detection unit 8, the next figure display position (x ′
t, y't) is calculated (step 305).

[0114]

[Equation 23] It is determined whether or not the distance between the previous display position and the newly estimated position is smaller than a predetermined fixed threshold value, and if smaller than the threshold value, the process ends (step 306).

It is determined whether or not the newly estimated position is out of the display range of the display. If it is not out of the display range, a graphic is displayed and the process from step 302 is repeated. If the estimated position is out of the display range, the display position is returned to the initial value and the processing from step 301 is repeated (step 30).
7).

When the distance is smaller than the fixed threshold value, the position where the initial value is displayed on the display is changed and the above processing is performed. For example, the initial position for displaying a graphic on the display is changed in order to five positions shown in FIG. 13, starting from each of the initial positions and ending the learning of the individual adjustment parameter when the processing of the step is completed, Parameter a
It outputs j and bj (j = 0..2n) to the individual adjustment unit 10.

[0117]

As described above, the line-of-sight detection device according to the first aspect and the line-of-sight detection method according to the fourth aspect of the present invention are arranged such that the user indicates a position on the display device indicated by the line of sight by the user. Since it is possible to measure the image of the user's eyes without wearing it, it is possible to measure the indicated position without imposing a load on the user. In addition, since the measurement is performed using the reflection position of the light emitting unit without utilizing the characteristics such as the inner corner of the eye and the outer corner of the eye, it is possible to accurately measure the indicated position.

[Brief description of drawings]

FIG. 1 is a block diagram of a line-of-sight detection device according to a first embodiment.

FIG. 2 is a flowchart of a procedure for acquiring position indication information.

FIG. 3 is a configuration example of a light emitting unit.

FIG. 4 is a configuration example of a light emitting unit using a window frame.

FIG. 5 is a flowchart of a pan head control.

FIG. 6 is a mask image for detecting a pupil position.

FIG. 7 is a mask position for pupil contour detection.

FIG. 8 is a projection of a reflection of a light emitting element.

FIG. 9 is a flowchart of calibration.

FIG. 10 is a configuration diagram of an individual adjustment unit.

FIG. 11 is a flowchart (1) of an individual adjustment parameter learning unit.

FIG. 12 is a flowchart (2) of an individual adjustment parameter learning unit.

FIG. 13 is an example of a graphic display position.

FIG. 14 is an example of an Affine map.

FIG. 15 is an edge detection position.

[Explanation of symbols]

1 ... Control unit 2 ... Light emitting part 3 ... Image input section 4 ... Reference point detector 5 ... Eye area detector 6 ... Image input control unit 7 ... Eye detector 8 ... Pointed position detector 9 ... Calibration section 10 ... Personal adjustment department 11 ... Individual adjustment parameter learning unit 12 ... Display

─────────────────────────────────────────────────── --Continued from the front page (56) References Japanese Patent Laid-Open No. 7-199045 (JP, A) Japanese Patent Laid-Open No. 5-199996 (JP, A) Actual Development Sho 58-18502 (JP, U) (58) Field (Int.Cl. ⁷ , DB name) G01B 11/00 G06T 1/00 280

Claims (6)

(57) [Claims] 1. An image input unit for capturing an image of a user, an eye detecting unit for detecting the user's eye from the image captured by the image input unit, and an eye detecting unit for detecting the eye of the user. The pupil detection means for determining the position of the pupil and the pupil detection means are arranged on substantially the same plane as the pointing target area designated by the user.
And three or more light emitting means for emitting light toward the user, and an image taken in a state in which the three or more light emitting means emit light
And subtract the image taken in the state where it does not emit light
And a reference point detection unit that detects a reference point that is a respective reflection position on the eyeball of the user of the irradiation light of each light emitting unit , and three or more reference points detected by the reference point detection unit. Reference point and before
Based on the position of the pupil detected by the serial pupil detecting means, wherein
A line-of-sight detection device comprising: a pointed position detection unit that detects a position pointed by the user's line of sight in the pointing target area . 2. The eye detecting means compares the image photographed by the image input means with three or more reference points detected by the reference point detecting means,
The line-of-sight detection apparatus according to claim 1, wherein a region near the reflection position in the image is detected as an eye. 3. The pupil detecting means detects the position of the center of gravity of a pupil region in the eye detected by the eye detecting means, and the pointing position detecting means detects three or more points detected by the reference point detecting means. Using the position of the reference point and the center of gravity of the pupil area detected by the pupil detection means,
The line-of-sight detection device according to claim 1, wherein the position indicated by the line of sight is detected in the pointing target area . 4. An image input step for capturing an image of a user
When, The use from the image taken in the image input step
An eye detection step of detecting a human eye, Position of the pupil in the eye detected in the eye detection step
A pupil detection step for obtainingArranged on substantially the same plane as the target area designated by the user
Is Three or more light sources that illuminate the user
Light meansThe image was taken with the flash on and no flash
The image taken in the state is subtracted and the difference processing is performed.
Luminous meansOf the illuminating light of each on the eyeball of the user.
Reflection positionReference point that isReference point detection step
And 3 or more detected in the reference point detection stepReference point
When,The aboveBased on the position of the pupil detected in the pupil detection step
There, In the target areaInstructing the line of sight of the user
And a designated position detecting step for detecting the
A visual line detection method characterized by: Wherein said eye detecting step, said image taken by said image input step, by comparing the three or more reference points detected by the reference point detection step, the position of reflection the in the image The eye gaze detection method according to claim 4, wherein the vicinity region of is detected as an eye. 6. The pupil detecting step detects a position of a center of gravity of a pupil region in the eye detected in the eye detecting step.
And the indicated position detecting step, the position of three or more reference points to be detected by the reference point detection step, the heavy of the pupil region detected by said pupil detecting step
The eye gaze detection method according to claim 4, wherein a position indicated by the eye gaze in the instruction target area is detected using the mind .