JP2988235B2 - Gaze direction detection device for vehicles - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a gaze direction detecting device for a vehicle, which measures the gaze direction of a vehicle driver in a non-contact manner and uses it for an interface such as a switch operation.

[0002]

2. Description of the Related Art A conventional gaze direction detecting apparatus of this type is disclosed in, for example, Japanese Patent Application Laid-Open No. 2-134130. In this method, a human corneal sphere is imaged by an illumination light source and a camera, the coordinates of the reflected light from the light source and the coordinates of the camera are connected, and an equation of a straight line passing through the center of the corneal sphere is obtained. The two operations are performed by a set of two illumination light sources and a camera, and a straight line connecting the central coordinates of the corneal sphere and the central coordinates of the pupil is detected as a line of sight by the triangulation method.

FIG. 13 is an explanatory diagram of this conventional line-of-sight direction detecting device. In the embodiment described in "Non-contact line-of-sight detecting device" of Japanese Patent Laid-Open No. 2-134130, a portion related to one camera is extracted. It is shown. The camera 102 captures, as information, an image (corneal reflection image) in which light emitted from the light source 101 that emits near-infrared light to the eyeball 103 is regularly reflected on the corneal surface. The specular reflection point P at this time is
It must be between A1-A2 corresponding to the iris portion. This is because when the specular reflection point P is outside of A1-A2, that is, on the white of the eye, the reflection surface is not smooth and diffuses and the reflection image has a spread, resulting in an irregular reflection image and a position of the image. This is because identification becomes impossible. The regular reflection point P is A1-
If it is between A2, the corneal reflection image becomes a clear bright spot, and the position can be specified reliably.

[0004]

However, in such a conventional example, the light source 10 for irradiating the eyeball 103 is used.
1 is one for one camera 102, the specular reflection point P of the light source 101 is between A1 and A2, and the specular reflection point of the second camera light source (not shown) is also A1-A2.
The gaze direction can be calculated only when the distance is between A2,
There is a problem that the measurable range in the line of sight direction is very narrow. SUMMARY OF THE INVENTION Accordingly, it is an object of the present invention to provide a vehicular gaze direction detecting device that expands a gaze direction measurable range by a camera and improves measurement accuracy in view of the above conventional problems.

[0005]

In order to achieve the above object, the present invention according to the first aspect of the present invention is directed to irradiating a driver's eyeball with invisible light, capturing a reflected image from the eyeball with a camera, and driving the driver's eyeball. In a gaze detection device for detecting a gaze direction of a camera, a coaxial illumination in which an illumination direction coincides with an optical axis of the camera;
A plurality of non-coaxial illuminations arranged at point-symmetric positions with respect to the center of the lens, an image memory for storing image data for each illumination, and a calculation of the coaxial illumination by calculation of the image data. A first corneal reflection image extraction unit for extracting a corneal reflection image, a retinal reflection image extraction unit for extracting a retinal reflection image from coaxial illumination image data and calculating a pupil center, and The corneal reflection images of a plurality of non-coaxial illuminations are obtained by calculation, and the coaxial illumination
A second corneal reflection image extracting unit that extracts a corneal reflection image closest to the bright corneal reflection image, and a first corneal reflection image extracting unit.
Corneal reflection image and second corneal reflection of coaxial illumination extracted
Corneal reflection image of non-coaxial illumination extracted by the image extraction unit
From the center of the corneal sphere and the center of the pupil
And a line-of- sight direction calculation unit that calculates the direction connecting the two as the line-of-sight direction of the driver .

According to a third aspect of the present invention, in the above configuration, instead of selecting the corneal reflection image by the second corneal reflection image extraction unit, each of the plurality of corneal reflection images of non-coaxial illumination is used. The line-of-sight direction calculation unit extracts the corneal reflection images of the plurality of non-coaxial illuminations and the corneal reflection images of the coaxial illuminations.
And calculate a plurality of corneal sphere centers from each of the corneal sphere centers and the pupil.
Calculate a plurality of line- of- sight directions connecting the center and
If both are in the same direction on either side, the coaxial illumination
And the gaze direction calculated based on the non-coaxial illumination on the same direction side as the gaze direction of the driver.

[0007]

According to a first aspect of the present invention, image data of a driver's eyeball by coaxial illumination and a plurality of non-coaxial illuminations is stored in an image memory, and the first corneal reflection image extracting unit stores the image data. The corneal reflection image of coaxial illumination is extracted from the calculation of the image data, and the pupil center is calculated by the retinal reflection image extraction unit. In the second corneal reflection image extraction unit, a plurality of corneal reflection images of non- coaxial illumination are extracted from the calculation of the image data, and a coaxial reflection image is extracted from the plurality of extracted corneal reflection images of non- coaxial illumination. Corneal reflection image
Also selects a close corneal reflection image. The gaze direction calculation unit calculates the corneal reflection image of the coaxial illumination and the selected angle of the non-coaxial illumination.
The corneal sphere center is calculated using the corneal reflection image and the corneal sphere center is calculated.
The direction connecting to the center of the pupil is calculated as the line-of-sight direction . Measurement range is widened by the non-coaxial system illumination from a plurality of positions
In addition, the non-coaxial
The measurement accuracy is improved by selecting the corneal reflection image of the axial illumination .

According to the third aspect of the present invention, all the corneal reflection images of non-coaxial illumination are extracted and output without selection by the second corneal reflection image extraction unit, and the plurality of non-coaxial reflection images are extracted by the gaze direction calculation unit. According to the corneal reflected image of the cornea reflection images and coaxial system illumination system illumination
Each gaze direction is calculated from a plurality of corneal sphere centers and a pupil center. And each gaze direction is the same direction to the left or right
In the case of the non-coaxial illumination on the same direction as the coaxial illumination
Is the gaze direction of the driver.

[0009]

1 is a block diagram showing an embodiment of a gaze direction detecting device for a vehicle according to the present invention. A camera 4 using a CCD or the like is installed on an instrument panel so as to image the driver's eyeball 13. A first illumination 1 of a coaxial system including a near-infrared LED or the like mounted on the front surface of a lens of the camera 4 so that an irradiation direction coincides with an optical axis of the camera 4 is provided. Further, a non-coaxial second illumination 2 and a third illumination 3 made of a near-infrared LED or the like whose irradiation direction does not coincide with the optical axis of the camera 4 are provided at symmetric positions of the camera 4. . An image output from the camera 4 is A / D converted by an A / D converter 5 as image data and stored in an image memory 6. A first corneal reflection image extraction unit 7 is connected to the image memory 6, to which a retinal reflection image extraction unit 8, a second corneal reflection image extraction unit 9, and a line-of-sight direction calculation unit 10 are sequentially connected.

The first corneal reflection image extracting unit 7 calculates the difference between the image data stored in the image memory 6 and the illumination 1 by calculating the difference.
Is extracted by calculation. The retinal reflection image extraction unit 8 similarly extracts a retinal reflection image from image data obtained by the illumination 1, and further calculates and extracts the pupil center. The second corneal reflection image extraction unit 9 calculates the difference between the image data stored in the image memory 6 and the corneal reflection images of illumination 2 and illumination 3 centering on the corneal reflection image of illumination 1 described above. And the extracted cornea reflection image and white eye reflection image of the extracted illuminations 2 and 3 are identified.

Then, in the line-of-sight direction calculation unit 10, the corneal reflection image of the illumination 1 extracted by the first corneal reflection image extraction unit 7, the pupil center extracted by the retinal reflection image extraction unit 8,
And the illumination 2 extracted by the second corneal reflection image extraction unit 9
Alternatively, based on the corneal reflection image of No. 3, the driver's line-of-sight direction is calculated and specified as a linear direction connecting the center of the pupil and the center of the cornea sphere. An illumination control unit 11 for controlling the light emission is connected to each of the illuminations 1, 2, and 3, and the illumination control unit 11, A /
An overall control unit 12 including the D converter 5 and controlling the operation of the entire apparatus is provided.

FIG. 2 shows an overall configuration of a vehicle line-of-sight switch system using the above-described line-of-sight direction detecting device, and FIG. 3 shows its on-board layout. In FIG. 2, the gaze direction detecting device shown in FIG. An HUD (head-up display) display control unit 24 is connected to the gaze direction detection device 20 via a gaze stop determination unit 23. The steering switch 22, the HUD display unit 25, and the controller switching unit 27 are connected to the HUD display control unit 24.

Now, when the main switch 21 is pressed, the system is turned on, and the gaze direction calculation unit 10 of the gaze direction detection device 20 outputs the gaze direction of the driver. The gaze stop determining unit 23 determines which area of the gaze switch area 26 that displays each operation target item (area) such as the air conditioner and the wiper is output.

When the area on the line-of-sight switch area 26 where the driver is gazing is determined, the HUD display controller 24
Displays the contents of the determined area on the HUD display unit 25, and controls the air conditioner controller 28, the CD controller 29, the radio controller 30, the constant speed traveling device controller by the controller switching unit 27. Item switching is performed to select 31, the headlight controller 32, or the wiper controller 33. Subsequently, a predetermined control target is controlled based on a user input determination from the steering switch 22.

As shown in FIG. 3, the main switch 2
1 and the steering switch 22 are attached to the steering 42. Light 1, camera 4, light 2, light 3, etc. are installed on the panel inside the instrument board in front of the driver's seat.
An HUD display 25 and a line-of-sight switch area 26 are set on the windshield 41. The line-of-sight switch area 26 further includes, as shown in detail in FIG.
The names of various controllers are HUD-displayed. Also, FIG.
2 are shown.

Next, the operation will be described. 5 and 6
Shows a flow of processing in the gaze direction detecting device shown in FIG. When the system is turned on and the control of the overall control unit 12 is started, first, the illumination 1 is turned on by the illumination control signal from the illumination control unit 11, and the illumination 2 and the illumination 3 are turned off. Then, the camera 4 captures an eyeball image and a corneal reflection image by the coaxial illumination 1 shown in S-50 in FIG.

In general, a corneal reflection image is a corneal reflection image captured by the camera 4 as shown in FIG. An image is observed. When illuminated by illumination 2 or illumination 3 that is not coaxial with the camera, the sphere (center is O
Specular reflection occurs at a point P on the corneal sphere, and a virtual corneal reflection image occurs at a point R. When the camera 4 is arranged coaxially with the camera 4 as in the case of the illumination 1, these points are on the same line as P ′, R ′, and O, and the positions of the corneal reflection image on the camera 4 and The corneal sphere center O exists on a straight line connecting the position of the illumination 1 (the focal position of the camera).

The imaging output including the corneal reflection image of the illumination 1 by the camera 4 is A / D converted by the A / D converter 5 and stored in the image memory 6 as image data I (x, y). Is done. Next, according to the illumination control signal of the illumination control unit 11, the illuminations 2 and 3 are simultaneously turned on instead of the illumination 1. The imaging output of the camera 4 at this time is similarly A / D converted by the A / D converter 5, and the image data shown in S-51 in FIG.
The data is stored in the image memory 6 as data J (x, y).

In the first corneal reflection image extraction unit 7, data stored in the image memory 6 is read, and a difference operation | I (x, y) -J is performed by a synthesis operation as a logical operation process of the image data. (X, y) | As a result of the calculation, the point (x0, y0) having the brightest and largest value is determined by the illumination 1 as shown by S-52 in FIG.
Is extracted as a corneal reflection image. Subsequently, a retinal reflection image extraction unit 8 extracts a retinal reflection image from the image data I (x, y).

FIG. 8 is a diagram showing a modeled general eyeball configuration. In the case of coaxial illumination 1, a light beam passing through the pupil is reflected on the retina and is projected brightly on the camera 4. The pupil region is called a retinal reflection image, and the center of gravity of the retinal reflection image coincides with the pupil center Q. The pupil center Q exists on a straight line connecting the position of the center of gravity on the CCD of the camera 4 and the camera focal point, and a straight line passing through the corneal sphere center O and the pupil center Q is the line of sight. The sphere K is the center of the corneal sphere O
A sphere having a radius of 4.2 mm between O and Q around
The sphere H has a corneal sphere radius of 7.8 m centered on the specular reflection point P.
m represents a sphere.

The extraction of the retinal reflection image in this case is performed by using S in FIG.
A window of m × n pixels is set around the corneal reflection image (x0, y0) of the illumination 1 indicated by −52, and the window is binarized and extracted by “threshold value Th1” in this window. The region is defined as a retinal reflection image as shown in S-53, and the barycentric coordinates (xG, yG) of the region of the extracted retinal reflection image are calculated and extracted as the pupil center α.

Next, in the second corneal reflection image extracting section 9, corneal reflection images of the illuminations 2 and 3 are extracted from the image data J (x, y). This time, a difference operation | J (x, y) -I (x, y) | as shown in S-54 of FIG. 5 is performed. Then, as shown in S-55, a window of m ′ × n ′ pixels is set around the point (x0, y0) for the difference result. In this case, (m'.times.n ') is used to limit the corneal reflection images (x0, y0) of the illumination 1 to the corneal reflection images of the illumination 2 and the illumination 3 to limit the existence candidate areas. The window (m ′ × n ′) is binarized by “threshold value Th2”, and a region having a gradation value higher than Th2 is extracted as a corneal reflection image of illuminations 2 and 3.

The following cases can occur as a result of the corneal reflection image extraction processing in S-55. (1) Extract only corneal reflection image of illumination 2 (2) Extract only corneal reflection image of illumination 3 (3) Extract both corneal reflection images of illumination 2 and illumination 3 (4) Extract corneal reflection image of illumination 2 and illumination 3 Extract white eye reflection image (5) Extract corneal reflection image of illumination 3 and white eye reflection image of illumination 2

The identification processing of each of these cases is performed only in the (m '× n') window which has been previously performed.
Labeling (numbering of each area) is performed on the binarized result by "2", and classification is performed according to the number of areas existing in the window. (A) When there is only one region In this case, as shown in S-56 of FIG. 6, since there is only one region that is imaged through the camera lens and extracted by image processing, -Case (1) or case (2).

The cases (1) and (2) are distinguished by determining the barycentric coordinates (xg, yg) of the area and illuminating the illumination 1 in the window.
From the coordinate values of the corneal reflection image (x0,-) of the above and the corneal reflection images (xg,-) of the left and right illuminations 2 and 3, if xg.gtoreq.x0, the case (1) is judged to be the corneal reflection image of the illumination 2. If xg <x0, it is determined to be the corneal reflection image of the illumination 3 in case (2).

(B) When there are two regions In this case, as shown in S-57, the barycentric coordinates (xg, yg) of each region and the region area S are also obtained. If the area S of either area is smaller than the "threshold area Sth" for image extraction, both of the areas are corneal reflection images.
In step (3), it is determined that the corneal reflection images of the illuminations 2 and 3 are both extracted.

If the area S of one of the areas is smaller than the "threshold area Sth" and the other is larger, and the coordinates of the area having a smaller area S are xg≥x0, case (4) If xg <x0, it is determined that case (5) applies.

The specific processing in case (3) is as follows. First, the coordinates of the center of gravity of the corneal reflection image area by illuminations 2 and 3 are (xg1, yg1) and (xg2, yg2), and the area of both areas is Since S <threshold area Sth, the corneal reflection image of illumination 1 (x0
, Y0), i.e., "i" which gives the "minimum (x0 -xgi)". However,
“I” is either 1 or 2 in this case. X found
If gi is xgi ≧ x0, it is processed as case (1), and if xgi <x0, it is handled as case (2) and the gaze direction is calculated.

If the area S of one of the areas is smaller than the "threshold area Sth" and the other is a larger case (4) or case (5), the coordinates of the smaller area are , Xgi≥x0, it is treated as case (1), and if xgi <x0, it is treated as case (2).

(C) When there are three or more regions In this case, as shown in S-58 of FIG. 6, the barycentric coordinates (xg1, yg1), (xg2, yg2), (xg1) of each region
g3, yg3) and the area S of each area, and the area closest to the previously obtained corneal reflection image (x0, y0) of the illumination 1 is searched for in the area of "area S <threshold Sth". If xg.gtoreq.x0, then the case corresponds to Case (4), and if xg <x0, it corresponds to Case (5).

Further, in this case, the coordinates of the center of gravity of each area are (xg1, yg1), (xg2, yg2), (xg3,
yg3)..., the shortest distance from (x0, y0), that is, "i" that gives "minimum (x0 -xgi)" is determined. “I” is 1, 2, or 3 in this case. If the obtained xgi is xgi ≧ x0, it is handled as case (1) processing, and if xgi <x0, it is handled as case (2) processing.

The extraction process in the second corneal reflection image extraction unit 9 is completed, and all cases are either the extraction of the corneal reflection image of illumination 2 of (1) or the corneal reflection image of illumination 3 of (2). Only the case can be reduced to the extraction case. Subsequently, the process proceeds to a process of calculating the driver's line-of-sight direction by the line-of-sight direction calculation unit 10.

FIG. 9 is an explanatory diagram showing the principle of calculating the line-of-sight direction in the line-of-sight direction calculation unit 10 shown in FIG. When the illumination and the optical axis of the camera coincide, the camera focus F
A straight line connecting 1 (= the illumination position of the light source 1) and the corneal reflection image β on the camera (on the CCD surface) passes through the center O of the corneal sphere. Therefore, the corneal sphere center O is on the straight line F1 -β. The actual pupil center Q is the camera focal point (illumination position) and CC
It is on the straight line F1-α connecting the pupil center α on the D image.

The focal length of the camera 4 is f and the focal position is F1.
(0,0,0), and considering the X-, Y-, and Z-axis world coordinate system F1-XYZ, the light source 2 for non-coaxial illumination is F2.
In (a, b, c), the illumination light is specularly reflected on the corneal surface. The specular reflection point P is a straight line F1 -δ connecting the corneal reflection image δ from the light source 2 on the CCD surface and the camera focal point F1.
It's above. Then, if the temporary regular reflection point P is placed on the straight line F1 -δ and the spherical surface J is drawn with a radius of 7.8 mm, which is the radius of the corneal sphere, the solid position of the center O of the corneal sphere as the intersection of the spherical surface J and the straight line F1 -β Is determined. The distance between the corneal sphere center O and the pupil center Q is about 4.2 mm, and the center of the pupil is determined as the intersection of the determined spherical surface K having a radius of 4.2 mm about the corneal sphere center O and the straight line F1 -α. The three-dimensional position of Q is determined. As a result, a line connecting the points O and Q is determined as the line-of-sight direction.

As described above, the gaze direction of the driver can be calculated by determining the corneal reflection image of coaxial illumination, the pupil center, and the corneal reflection image of non-coaxial illumination. Therefore, the gaze direction calculation unit 10 performs the following processing. (1 ′) When only the corneal reflection image of illumination 2 is extracted from the non-coaxial system, the gaze direction is calculated and determined from the corneal reflection image of illumination 1, the pupil center, and the corneal reflection image of illumination 2. (2 ′) When only the corneal reflection image of illumination 3 is extracted from the non-coaxial system, the gaze direction is calculated and determined from the corneal reflection image of illumination 1, the pupil center, and the corneal reflection image of illumination 3.

(3 ') When both the corneal reflection images of the illuminations 2 and 3 are extracted The corneal reflection image of the illumination 1 or the illumination 3 or the corneal reflection image of the illumination 3 which has a shorter distance is employed. Assuming that the coordinate of the center of gravity is (xg, yg), the processing of (1 ') is performed if xg ≧ x0, and the processing of (2') is performed if xg <x0. As described above, in case (4), the processing is (1 '), and in case (5), the processing is (2'). This completes the process of calculating the driver's line of sight.

In the gaze stop determining unit 23, the gaze direction calculated by the gaze direction calculation unit 10 as described above is:
It is determined which area (control item) of the line-of-sight switch area 26 is being viewed. This determination is made based on whether or not the driver views the same area continuously for a certain number of times in the processing cycle. 400m in one cycle of gaze direction calculation
If sec is required, it is determined that the same area is gazing at a gazing time of about 0.8 sec, for example, two times in succession.

Now, if the gaze stop determination unit 23 determines that the driver is gazing at the air conditioner area (A / C) of the gaze switch area 26 shown in FIG.
Upon receiving the determination result, the HUD display control unit 24 displays the current air conditioner set temperature on the HUD display unit 25 as, for example, 25 ° C., as shown in FIG. When the set temperature of the air conditioner is displayed on the HUD display section 25, FIG.
As shown in the figure, the up / down button of the steering switch 22 functions to adjust the set temperature up and down.

The driver operates the up / down button of the steering switch 22 while watching the HUD display section 25 to set a desired set temperature. If the up / down button is not operated for a certain period of time (for example, 5 seconds), it is determined that the operation has been completed, and all the switch functions are stopped.
The driver needs some operation again, and when the main switch 21 is pressed, the above processing is repeated again.

Considering the measurable range of such an eye gaze detecting device, according to the data of a normal model eye, the angle from the center of the corneal sphere to the boundary between the iris and the white of the eye is 6 right and left.
Each 6 ° gives a total of about 132 °. As shown in FIG. 10, assuming that the distance from the camera 4 to the eyeball is D and the interval between the illumination 1 and the illumination 3 is W, the angle of the line of sight with respect to the camera axis is θ = 66−tan ⁻¹ (W / D). For example, when W = 20 cm and D = 100 cm, θ = 54.7.
°, W = 30 cm, D = 100 cm, θ = 4
9.3 °, indicating that the smaller the interval W of the illumination, the wider the measurable range.

FIG. 11 is a conceptual diagram showing the limit of the measurable range, in which the measurable range of the conventional example and that of the present embodiment are compared and shown. As described above, the corneal reflection image of coaxial illumination and the corneal reflection image of non-coaxial illumination are required. For simplicity, ignoring the angle of reflection of the illumination light source, the limit of the measurable range is corneal. A straight line connecting the center O of the sphere and the boundary of the iris and eye is determined by the position passing through the illumination light source. For this reason, in the conventional example having only one light source as shown in (a) of the figure, the measurable range is narrowed on the opposite side of the light source because the boundary between the iris and the white of the eye is deviated with the limit straight line S1 as a limit. On the other hand, in the present embodiment, as shown in (b), by installing a plurality of illuminations 2 and 3 at symmetrical positions, the limit straight lines S2 and S3 are on both sides with respect to the line connecting the camera and the center of the eyeball. , The measurable range is widened, and the measurement accuracy is improved even though only one camera is used.

Next, FIG. 12 shows a second embodiment of the present invention. In the previous embodiment, the corneal reflection image obtained by the coaxial illumination 1 in the second corneal reflection image extraction unit 9 from the plurality of corneal reflection images obtained by the plurality of non-coaxial illuminations 2 and 3 The corneal reflection image closest to is selected and extracted, and the gaze direction is calculated using this. However, the corneal reflection image extraction unit 9 'of the present embodiment does not perform the above selection. Then, the line-of-sight direction calculation unit 10 'calculates the line-of-sight direction based on each corneal reflection image.

Here, in FIG. 12, when each of the calculated line-of-sight directions is a lower right direction from the eyeball, that is, when the driver is looking to the left, an image formed by the illumination 1 and the illumination 3 on the right side toward the driver is displayed. The gaze direction calculated based on the selected gaze direction is selected and output. In addition, when both of the calculated gaze directions are the lower left direction in FIG. 12, that is, when the driver is looking rightward, the gaze direction calculated based on the image of the illumination 1 and the left illumination 2 toward the driver is selectively output. .

Further, when the gaze direction calculated for each illumination is detected in the lower right direction and the lower left direction,
The average direction of both is calculated and output. Lighting 2, 3
When either one of the corneal reflection images is not observed, the gaze direction is calculated from the observed corneal reflection image and the corneal reflection image of the illumination 1. In the above description, since the corneal reflection images are arranged in the order in which the illuminations are arranged on the image input by the camera, it is easy to identify which reflection image is due to which illumination. Other configurations are the same as those of the first embodiment shown in FIG.

Since this embodiment is configured as described above, by adopting an image obtained by illuminating the gaze side by immediately following the driver's gaze direction, an error factor in the gaze direction measurement is reduced. The measurement accuracy is further improved due to the decrease. In addition,
In each of the above embodiments, the case where the left and right non-coaxial illuminations are two has been described. However, the present invention is not limited to this. By further increasing the number of non-coaxial illuminations, the line-of-sight direction detection resolution can be further increased. Can be raised.

[0046]

As described above, according to the present invention, a plurality of
From corneal reflection image under non-coaxial illumination to corneal reflection image under coaxial illumination
The corneal reflection image closest to the corneal reflection image is selected , and calculated from the selected optimal corneal reflection image and the corneal reflection image of the coaxial illumination.
Since the gaze direction is calculated from the center of the corneal sphere and the center of the pupil, the measurable range in the gaze direction is widened and the measurement accuracy is improved. Also, instead of selecting one from a plurality of corneal reflection images of non-coaxial illumination ,
After calculating each gaze direction, both gaze directions are left and right
Non-coaxial illumination in the same direction if one is in the same direction
Is the gaze direction calculated based on the driver's gaze direction.
Output is also improved by the measurement accuracy.
The effect is obtained .

[Brief description of the drawings]

FIG. 1 is a diagram showing a first embodiment of the present invention.

FIG. 2 is a diagram illustrating an overall configuration of a line-of-sight switch system of a vehicle using the line-of-sight direction detection device.

FIG. 3 is a diagram showing an on-vehicle layout of the eye-gaze switch system.

FIG. 4 is an enlarged view showing details of a main portion of a vehicle-mounted layout.

FIG. 5 is an explanatory diagram showing a flow of a process in a gaze direction detecting device.

FIG. 6 is an explanatory diagram showing a flow of a process in the gaze direction detecting device.

FIG. 7 is an explanatory diagram of corneal reflection image generation.

FIG. 8 is a diagram showing a modeled eyeball configuration.

FIG. 9 is an explanatory diagram illustrating a principle of calculating a line-of-sight direction.

FIG. 10 is an explanatory diagram of a lighting arrangement in the embodiment.

FIG. 11 is an explanatory diagram showing a limit of a measurable range.

FIG. 12 is a diagram showing a second embodiment.

FIG. 13 is a view schematically showing a conventional visual line detection device.

[Explanation of symbols]

 Reference Signs List 1 coaxial illumination 2, 3 non-coaxial illumination 4 camera 5 A / D converter 6 image memory 7 first corneal reflection image extraction unit 8 retinal reflection image extraction unit 9 second corneal reflection image extraction unit 10 line of sight Direction calculation unit 11 Lighting control unit 12 Overall control unit

Claims (4)

(57) [Claims] 1. A gaze detecting apparatus for irradiating an invisible light to a driver's eyeball and capturing a reflected image from the eyeball by a camera to detect a driver's gaze direction, wherein an optical axis of the camera coincides with an irradiation direction. Coaxial illumination, and arranged at a point-symmetric position with respect to the camera lens center.
A plurality of non-coaxial illuminations, an image memory for storing image data for each of the illuminations, and a first cornea for extracting a corneal reflection image of the coaxial illumination by calculating the image data A reflection image extraction unit; a retinal reflection image extraction unit that extracts a retinal reflection image from the image data of the coaxial illumination to calculate a pupil center; and a plurality of non-coaxial systems by calculating the image data. The corneal reflection images of the illumination are obtained , and the corneal reflection images of the coaxial illumination are obtained therefrom.
A second corneal reflection image extraction unit for extracting a corneal reflection image closest to the corneal reflection image, and a coaxial illumination extracted by the first corneal reflection image extraction unit.
Extracted by the bright corneal reflection image and the second corneal reflection image extraction unit
Of corneal sphere center from corneal reflection image of non-coaxial illumination
The driver connects the direction connecting the center of the corneal sphere and the center of the pupil.
A line-of- sight direction calculating unit that calculates a line-of-sight direction. 2. The plurality of non-coaxial illuminations are simultaneously turned on.
Vehicular gaze direction detecting apparatus according to claim 1, characterized in that that. 3. A camera for irradiating the driver's eyeball with invisible light.
Captures the reflected image from the eyeball, and the driver's line of sight
In the eye-gaze detecting device, a coaxial illumination whose illumination direction coincides with the optical axis of the camera is disposed at a point-symmetric position with respect to the lens center of the camera.
A plurality of non-coaxial illuminations, an image memory for storing image data for each illumination, and a corneal reflection image of the coaxial illumination by calculation of the image data.
A first corneal reflection image extraction unit to extract, and a retinal reflection image extracted from the coaxial illumination image data.
A retinal reflection image extraction unit for calculating a pupil center; and a plurality of non-coaxial illumination angles by calculating the image data.
A second corneal reflection image extraction unit for extracting a corneal reflection image, and a coaxial illumination extracted by the first corneal reflection image extraction unit.
Extracted by the bright corneal reflection image and the second corneal reflection image extraction unit
Multiple corneas with each cornea reflection image of non-coaxial illumination
Calculate sphere centers and connect multiple corneal sphere centers to pupil centers
Gaze direction is calculated, and each gaze direction is either left or right
In the case of the same direction, the coaxial illumination and the non-
The gaze direction calculated based on the axis system illumination is the driver's gaze
A gaze direction calculation unit that outputs the direction.
A gaze direction detecting device for a vehicle to be a feature . 4. The eye-gaze direction calculating unit includes:
Each line of sight connecting the heart and the center of the pupil differs from side to side
In that case, calculate the average direction and use it as the driver's line of sight
The gaze direction detecting device for a vehicle according to claim 3 , wherein the gaze direction is outputted .