JP3176147B2 - Eye gaze detection device - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an improved eye-gaze detecting device which picks up an eye image illuminated by an illuminating means by an image sensor, detects a pupil and a corneal reflection image from the eye image, and obtains an eye gaze. Things.

[0002]

2. Description of the Related Art Various devices (for example, eye cameras) for detecting a so-called line of sight (a visual axis) for detecting which position on an observation surface the observer is observing have been proposed.

For example, in Japanese Patent Application Laid-Open No. 1-274736, a parallel light beam from a light source is projected to the anterior segment of an observer's eyeball, and a corneal reflection image formed by light reflected from the cornea and an image forming position of a pupil are used. Looking for the visual axis.

FIG. 10 is a view for explaining the principle of the visual line detection method.

FIG. 1A shows an image sensor 14 shown in FIG.
It is a very normal case of an eyeball image projected on a surface,
Reference numeral 60 in (B) denotes an image signal output on lines (I)-(I) '.

[0006] In the figure, reference numeral 50 denotes a so-called white-eye portion of the eyeball;
1 represents a pupil, and 52a and 52b represent corneal reflection images of an eyeball illumination light source.

FIG. 2 is an example of an eyeball image of a photographer wearing glasses. (52a, 52b) are corneal reflection images, (53a, 52b)
53b) and (54a, 54b) represent ghost images of the light source by the glasses.

Next, a gaze detection method will be described with reference to FIGS. 10, 1A and 1B. The infrared light emitting diodes 13a and 13b are arranged substantially symmetrically in the Z direction with respect to the optical axis A of the light receiving lens 12, and divergently illuminate the eyeball of the photographer.

The infrared light emitted from the infrared light emitting diode 13b illuminates the cornea 16 of the eyeball 15. At this time, a corneal reflection image d due to a part of the infrared light reflected on the surface of the cornea 16
Is condensed by the light receiving lens 12 and the image sensor 14
An image is formed again at the upper position d '.

Similarly, the infrared light emitted from the infrared light emitting diode 13a illuminates the cornea 16 of the eyeball. At this time, a corneal reflection image e due to a part of the infrared light reflected on the surface of the cornea 16 is condensed by the light receiving lens 12 and the image sensor 1
The image is re-imaged at the position e 'on the image 4.

Light beams from the ends a and b of the iris 17 form images of the ends a and b at positions a 'and b' on the image sensor 14 via the light receiving lens 12. When the rotation angle θ of the optical axis a of the eyeball 15 with respect to the optical axis (optical axis a) of the light receiving lens 12 is small, the Z coordinate of the ends a and b of the iris 17 is Z
Assuming a and Zb, the coordinate Zc of the center position c of the pupil 19 is expressed as Zc ≒ (Za + Zb) / 2.

Since the Z coordinate of the midpoint between the corneal reflection images d and e matches the Z coordinate Zo of the center of curvature O of the cornea 16,
The Z coordinates of the corneal reflection image generation positions d and e are Zd and Ze, the standard distance from the center of curvature O of the cornea 16 to the center C of the pupil 19 is L _OC, and a coefficient that takes into account individual differences with respect to the distance L _OC Let Al be the rotation angle θ of the optical axis a of the eyeball substantially satisfies the relational expression of (Al * L _OC ) * sin θ ≒ Zc− (Zd + Ze) / 2 (1) For this reason, by detecting the position of each feature point (corneal reflection images d and e and the ends a and b of the iris) projected on a part of the image sensor as shown in FIG. The rotation angle θ of the optical axis a of the eyeball can be obtained. At this time, equation (1) can be rewritten as β (Al * L _OC ) * sin θ ≒ (Za ′ + Zb ′) / 2− (Zd ′ + Ze ′) / 2 (2) Here, β is a magnification determined by the position of the eyeball with respect to the light receiving lens 12, and is substantially obtained as a function of the interval | Zd'-Ze '| of the corneal reflection image. The rotation angle θ of the eyeball 15 can be rewritten as θ {ARCSIN {(Zc′−Zf ′) / β / (Al * L _OC )} (3) Where Zc ′ ≒ (Za ′ + Zb ′) / 2 Zf ′ ≒ (Zd ′ + Ze ′) / 2. By the way, since the horizontal rotation angle θ of the optical axis A of the photographer is calculated because the optical axis A of the photographer's eyeball does not match, the angle correction δ between the optical axis of the eyeball and the visual axis must be performed. Thus, the horizontal line of sight θH of the photographer is obtained. Assuming that a coefficient considering the individual difference with respect to the correction angle δ between the optical axis A of the eyeball and the visual axis is B1, the horizontal line of sight θH of the photographer is obtained as follows: θH = θ ± (B1 * δ) (4) Can be Here, as for the sign ±, if the rotation angle to the right with respect to the photographer is positive, the sign + is selected when the photographer's eyes, except for the observation device, are the left eye, and the sign-is selected when the photographer's eyes are the right eye.

In FIG. 1, the eyeball of the photographer is ZX.
An example of rotation in a plane (for example, a horizontal plane) is shown,
Even when the photographer's eyeball rotates in an XY plane (for example, a vertical plane), the same detection can be performed. However, since the vertical component of the line of sight of the photographer matches the vertical component θ ′ of the optical axis of the eyeball, the vertical line of sight θV is θV = θ ′. Further, the position (Zn, Y) on the focus plate in the viewfinder field viewed by the photographer from the line-of-sight data θH, θV.
n) is Zn ≒ m * θH ≒ m * [ARCSIN ｛(Zc′-Zf ′) / β / (Al * L _OC )｝ ± (B1 * δ)] (5) Yn ≒ m * θV Desired. Here, m is a constant determined by the finder optical system of the camera.

Here, the values of the coefficients Al and B1 for correcting the individual difference of the eyeball of the photographer are obtained by having the photographer fixate on an index provided at a predetermined position in the viewfinder of the camera, and setting the position of the index. And the position of the fixation point calculated according to the equation (5).

In this embodiment, the calculation for obtaining the line of sight and the gazing point of the photographer is executed by software of the microcomputer of the line-of-sight calculation processing device based on the above equations.

A coefficient for correcting the individual difference of the line of sight is calculated, and the position of the line of sight of the observer who looks into the viewfinder of the camera is calculated using the equation (5), and the line of sight information is used to adjust the focus or exposure of the taking lens. Used for control.

[0017]

As described above, the line-of-sight direction / coordinate of the photographer is determined by the corneal reflection image (“Purkinje image”, hereinafter referred to as “IRED”) of the infrared light emitting diode (hereinafter referred to as “IRED”) for illumination. P image), and can be calculated from the positional relationship of the pupils in the eyeball image. This specific detection method is disclosed in Japanese Patent Application No. 3-121098 filed by the present applicant.

Since the P image is a corneal reflection image of the IRED,
The brightness is high in the whole eyeball image, and the image is IRED
Since this is a reflection image of the chip light emitting surface, the size thereof is also extremely small. Therefore, a P image can be extracted from the eyeball image signal by using the feature. In particular,
When the luminance value of a certain pixel in the eyeball image signal is equal to or greater than a predetermined value and the luminance difference between the pixel and a peripheral pixel is equal to or more than a predetermined value, the pixel (or a peripheral pixel area including the pixel) is regarded as a P image. I try to be a candidate.

Further, in order to detect the distance between the camera and the eyeball, the IREDs are turned on in pairs, and the P images are used in pairs. Is selected.

As shown in FIG. 1, the high luminance portion is indicated by (52) in the figure.
If there is only one set of a, 52b), it can be easily detected that the high-luminance portion is a correct P image under the conditions described above.

However, when there are a plurality of sets of high-luminance portions as shown in FIG. 2, a correct P image is often erroneously selected only under the above conditions, which is very inconvenient for visual line detection.

[0022]

The object of the present invention is to solve the above-mentioned problems.

That is, according to the first aspect of the present invention, there is provided an illuminating means for illuminating an eyeball, an image sensor for obtaining an image of the eyeball illuminated by the illuminating means, and a predetermined method based on the eyeball image obtained by the image sensor. A corneal reflection image candidate detecting means for detecting a portion satisfying the condition as a corneal reflection image candidate; a gaze calculating means for calculating gaze information using the corneal reflection image candidate detected by the corneal reflection image candidate detecting means; And determining means for determining whether or not the line-of-sight information calculated by the calculating means satisfies a certain standard.When the determining means determines that the line-of-sight information does not satisfy the certain standard, The corneal reflection image candidate detecting means detects another corneal reflection image candidate, and the sight line calculating means calculates sight line information again.

Further, according to a fourth aspect of the present invention, there is provided an illuminating means for illuminating the eye, an image sensor for obtaining an image of the eye illuminated by the illuminating means, and a predetermined image from the eyeball image obtained by the image sensor. A corneal reflection image candidate detecting unit that detects a plurality of portions that satisfy the conditions as corneal reflection image candidates, and a line of sight that calculates a plurality of rotation angles of an eyeball using the plurality of corneal reflection image candidates detected by the corneal reflection image candidate detection unit. It is characterized by comprising a calculating means and a selecting means for selecting the smallest rotation angle from the plurality of rotation angles of the eyeball calculated by the visual line calculating means.

[0025]

FIG. 3 is a schematic view of a main part of a first embodiment when the present invention is applied to a single-lens reflex camera.

In each figure, reference numeral 1 denotes a photographing lens for convenience.
Although shown with a single lens, it is actually composed of more lenses. Reference numeral 2 denotes a main mirror which is obliquely connected to or retracted from the photographing optical path according to the state of observation of the subject image by the viewfinder system and the state of photographing of the subject image. Reference numeral 3 denotes a sub-mirror, which reflects a light beam transmitted through the main mirror 2 toward a later-described focus detection device 6 below the camera body.

Reference numeral 4 denotes a shutter and reference numeral 5 denotes a photosensitive member, which comprises a silver halide film, a solid-state image pickup device such as a CCD or MOS type, or an image pickup tube such as a vidicon.

Reference numeral 6 denotes a focus detection device, which is a field lens 6a, reflection mirrors 6b and 6 disposed near the image plane.
c, a secondary imaging lens 6d, an aperture 6e, a line sensor 6f including a plurality of CCDs, and the like.

The focus detecting device 6 in the present embodiment uses a well-known phase difference method. As shown in FIG. 4, a plurality of areas (five places) in an observation screen (in the viewfinder field) are set as distance measuring points. The distance measuring point is configured to enable focus detection.

Reference numeral 7 denotes a focusing plate arranged on a predetermined image forming plane of the photographing lens 1, 8 denotes a pentaprism for changing a finder optical path, 9 and 10 each denote an image forming lens for measuring the luminance of a subject in an observation screen. It is a photometric sensor. The imaging lens 9 conjugately connects the focusing plate 7 and the photometric sensor 10 via a reflection optical path in the penta roof prism 8.

Next, behind the exit surface of the penta roof prism 8, an eyepiece lens 11 having a light splitter 11a is arranged, and is used for observation of the focus plate 7 by a photographer's eye 15. The light splitter 11a is composed of, for example, a dichroic mirror that transmits visible light and reflects infrared light.

Reference numeral 12 denotes a light receiving lens, and reference numeral 14 denotes an image sensor having a two-dimensional array of photoelectric elements such as CCDs, which are arranged so as to be conjugate with the vicinity of the pupil of the photographer's eye 15 at a predetermined position with respect to the light receiving lens 12. ing. 13a-13f
Are infrared light emitting diodes, each of which is an illumination light source.

Reference numeral 21 denotes a high-intensity superimposing LED which can be visually recognized even in a bright subject. The emitted light is reflected by the main mirror 2 via a light projecting prism 22 and provided on a display portion of the focus plate 7. The light is bent in the vertical direction by the micro prism array 7 a and reaches the photographer's eye 15 through the pentaprism 8 and the eyepiece 11.

The micro prism array 7a is located at a plurality of positions (ranging points) corresponding to the focus detection area of the focus plate 7.
Are formed in the shape of a frame, and each of them is provided with five superimposing LEDs 21 (each LED-L1, LED
-L2, LED-C, LED-R1, LED-R2).

Thus, as can be seen from the finder field of view shown in FIG.
Reference numerals 1, 202, 203, and 204 illuminate in the finder visual field to display a focus detection area (distance measurement point) (hereinafter, this is referred to as a superimposed display).

Reference numeral 23 denotes a visual field mask for forming a finder visual field area. Reference numeral 24 denotes an LCD in the finder for displaying photographing information outside the finder field of view.
-LED) 25.

The light transmitted through the LCD 24 is transmitted to the triangular prism 2
The light is guided into the field of view of the finder by the reference numeral 6 and 207 in FIG.
Are displayed outside the finder field of view, as shown by, so that the photographer can know the photographing information.

Reference numeral 31 denotes an aperture provided in the taking lens 1;
A diaphragm driving device including a diaphragm driving circuit 111 described later;
Reference numeral 3 denotes a lens driving motor, reference numeral 34 denotes a lens driving member including a driving gear and the like, and reference numeral 35 denotes a photocoupler which detects rotation of a pulse plate 36 interlocked with the lens driving member 34 and transmits the rotation to a lens focus adjustment circuit 110. Focus adjustment circuit 110
The lens drive motor is driven by a predetermined amount based on this information and the information on the lens drive amount from the camera side, and the photographing lens 1 is moved to the in-focus position. 37
Reference numeral denotes a mounting contact which serves as an interface between a known camera and a lens.

FIG. 5 is an explanatory diagram of an electric circuit diagram built in the camera of the present invention. The same components as those in FIG. 3 are denoted by the same reference numerals.

A central processing unit (hereinafter referred to as a CPU) 100 of a microcomputer built in the camera body includes a line-of-sight detection circuit 101, a photometry circuit 102, and an automatic focus detection circuit 10.
3. Signal input circuit 104, LCD drive circuit 105, LE
A D drive circuit 106, an IRED drive circuit 107, a shutter control circuit 108, and a motor control circuit 109 are connected. Also, a focus adjustment circuit 1 disposed in the taking lens
10. Signals are transmitted to the aperture drive circuit 111 via the mount contacts 37 shown in FIG.

EEPROM 10 attached to CPU 100
Reference numeral 0a has a storage function of a line of sight correction data for correcting an individual difference in line of sight as storage means.

The line-of-sight detection circuit 101 A / D converts the output of the eyeball image from the image sensor 14 (CCD-EYE), and transmits this image information to the CPU 100. CPU1
In step 00, each feature point of the eyeball image necessary for eye gaze detection is extracted according to a predetermined algorithm, and the gaze of the photographer is calculated from the position of each feature point.

The photometric circuit 102 amplifies the output from the photometric sensor 10, performs logarithmic compression and A / D conversion, and sends it to the CPU 100 as luminance information of each sensor. The photometric sensor 10 is located at the left ranging point 2 in the viewfinder visual field shown in FIG.
SPC-L for metering the left area 210 including 00 and 201
SPC-C for metering a central area 211 including a center ranging point 202, an SPC-R for metering a right area 212 including right ranging points 203 and 204, and their peripheral areas 21.
It is composed of SPC-A for photometry 3 and photodiodes for photometry in four regions.

The line sensor 6f shown in FIG. 5 has five sets of line sensors CCD-L2 and CCD-L corresponding to the five ranging points 200 to 204 in the screen as shown in FIG.
1, a known CCD line sensor composed of CCD-C, CCD-R1, and CCD-R2.

The automatic focus detection circuit 103 A / D converts the voltage obtained from the line sensor 6f, and
Send to 0. SW-1 is a switch that is turned on by the first stroke of the release button 41 to start photometry, AF, line-of-sight detection operation, etc., SW-2 is a release switch that is turned on by the second stroke of the release button, and SW-AEL is an AE lock. Button 43
AE lock switch, which is turned on by pressing, SW
-DIAL1 and SW-DIAL2 are signal input circuits 104 by dial switches provided in an electronic dial (not shown).
Is input to the up / down counter of the electronic dial and counts the amount of rotation click of the electronic dial.

Reference numeral 105 denotes a well-known LCD drive circuit for driving the liquid crystal display element LCD to display the aperture value, the shutter time, the set photographing mode, and the like according to the signal from the CPU 100. Both can be displayed at the same time. L
The ED drive circuit 106 is an illumination LED (F-LED) 25
And superimpose LED 21 are turned on and off. The IRED drive circuit 107 includes an infrared light emitting diode (I
RED1 to 6) 13a to 13f are selectively turned on according to the situation.

When energized, the shutter control circuit 108 controls the magnet MG-1 for running the front curtain and the magnet MG-2 for running the rear curtain to expose the photosensitive member with a predetermined amount of light. The motor control circuit 109 controls a motor M1 for winding and rewinding the film and a motor M2 for charging the main mirror 2 and the shutter 4. These shutter control circuits 108,
A series of camera release sequences is operated by the motor control circuit 109.

Next, FIG. 6 shows a flowchart of the operation of the camera having the visual axis detection device, and the following description will be made based on these flowcharts.

When the mode dial (not shown) is rotated to set the camera from a non-operation state to a predetermined photographing mode (this embodiment will be described based on the case where the shutter priority AE is set). (# 100), C
The variables used for the eye gaze detection of the PU 100 are reset.

The camera stands by until the release button 41 is pressed and the switch SW1 is turned on (# 10).
2). Release button 41 is pushed in and switch SW1 is turned O
When the signal input circuit 104 detects that N
U100 checks with the line-of-sight detection circuit 101 (# 10
3).

At this time, if the line-of-sight prohibition mode is set, a specific distance measuring point is selected by the distance measuring point automatic selection subroutine (# 116) without executing line-of-sight detection, that is, without using line-of-sight information. At this distance measuring point, the automatic focus detection circuit 103 performs a focus detection operation (# 107).

As described above, the photographing mode in which the focus detection point is selected without using the line-of-sight information (the line-of-sight prohibition automatic focus photographing mode) and the photographing mode in which the distance measurement point is selected using the line-of-sight information (the line of sight automatic focus photographing mode) Both are provided, and the photographer can arbitrarily select whether or not to set the gaze prohibition mode.

Although several methods can be considered as an algorithm for automatic selection of the ranging points, a near-point priority algorithm in which weights are assigned to the central ranging points is effective, and is not directly related to the present invention. Therefore, the description is omitted.

If the visual axis detection mode is set, visual axis detection is executed (# 104). At this time LED
The drive circuit 106 turns on the illumination LED (F-LED) 25, and the LCD drive circuit 105 controls the LCD in the finder.
The gaze input mark 78 is turned on so that the photographer can confirm that the camera is performing gaze detection outside the viewfinder screen 207.

Here, the line of sight detected by the line of sight detection circuit 101 is converted into coordinates of the gazing point on the focusing screen 7. C
PU 100 selects a ranging point close to the gazing point coordinates,
A signal is transmitted to the LED drive circuit 106 to blink the distance measuring point mark using the superimposing LED 21 (# 105).

When the photographer sees that the distance measuring point selected by the photographer's line of sight is displayed, recognizes that the distance measuring point is incorrect, and releases the release button 41 to release the switch SW.
When 1 is turned off (# 106), the camera
It waits until 1 is turned on (# 102).

As described above, the distance measuring point mark in the viewfinder field of view is blinked and displayed to notify the photographer that the distance measuring point has been selected based on the line of sight information. You can check if it is.

After the photographer has displayed the distance measuring point selected by his / her line of sight, the switch SW1 is subsequently turned on.
If it continues (# 106), the automatic focus detection circuit 103
Performs focus detection at one or more distance measuring points using the detected line-of-sight information (# 107).

Here, it is determined whether or not the selected distance measuring point cannot be measured (# 108). If the distance cannot be measured, the CPU 100 sends a signal to the LCD drive circuit 105 to send the signal in the finder.
The focus mark on the CD24 flashes and the distance measurement is NG (impossible)
Is warned to the photographer (# 118), and the operation is continued until SW1 is released (# 119).

If distance measurement is possible and the focus adjustment state of the distance measurement point selected by a predetermined algorithm is not in focus (#
109), the CPU 100 sends a signal to the lens focus adjustment circuit 110 to drive the photographing lens 1 by a predetermined amount (# 11).
7). After driving the lens, the automatic focus detection circuit 103 performs focus detection again (# 107), and determines whether or not the photographing lens 1 is in focus (# 109).

If the photographing lens 1 is in focus at a predetermined distance measuring point, the CPU 100
, The focus mark on the LCD 24 in the finder is turned on, and a signal is also sent to the LED drive circuit 106 to display the in-focus distance measuring point 201 in focus (# 110).

At this time, the blinking display of the distance measuring point selected by the line of sight is turned off, but the distance measuring point displayed by focusing often coincides with the distance measuring point selected by the line of sight. The focusing distance measuring point is set to a lighting state so that the photographer can recognize that the subject is in focus. The photographer observes that the focused ranging point is displayed in the viewfinder, recognizes that the ranging point is not correct, releases the release button 41, and turns off the switch SW1 (# 111). The camera waits until the switch SW1 is turned on (# 10
2).

Further, if the photographer looks at the focusing point displayed in focus and continues to turn on the switch SW1 (# 1).
11), the CPU 100 transmits a signal to the photometric circuit 102 to perform photometry (# 112). At this time, a weighted exposure value is calculated for the photometric regions 210 to 213 including the focused distance measuring point.

Further, it is determined whether or not the release button 41 is depressed and the switch SW2 is turned on (#
113) If the switch SW2 is OFF, the state of the switch SW1 is confirmed again (# 111). or,
When the switch SW2 is turned on, the CPU 100 transmits a signal to each of the shutter control circuit 108, the motor control circuit 109, and the aperture driving circuit 111.

First, power is supplied to M2 to raise the main mirror 2 and the aperture 31 is stopped down. Then, power is supplied to MG1 to open the front curtain of the shutter 4. The aperture value of the aperture 31 and the shutter speed of the shutter 4 are determined from the exposure value detected by the photometric circuit 102 and the sensitivity of the film 5.
M after a lapse of a predetermined shutter time (for example, 1/250 second)
G2 is energized to close the rear curtain of shutter 4. When the exposure of the film 5 is completed, power is supplied to M2 again, mirror down and shutter charging are performed, and power is also supplied to M1 to advance the film, thereby completing a series of shutter release sequence operations (# 114). . Thereafter, the camera waits until the switch SW1 is turned on again (# 102).

FIG. 7 and FIG. 8 are flow charts of gaze detection according to the present invention.

As described above, the line-of-sight detection circuit 101 is a CPU
When a signal is received from 100, gaze detection is performed (# 1)
04).

First, the CPU 100 executes infrared light emitting diodes (IRED) 13a to 13f for illuminating the photographer's eyes.
, An appropriate combination of IREDs is selected and turned on (# 201). Selection of the IRED is made by a posture switch (not shown) depending on whether the camera is in the horizontal position or the vertical position, or whether the photographer wears glasses.

Next, charge is stored in the image sensor 14 for a predetermined storage time (# 202). When the accumulation is completed, the IRED is also turned off (# 203).

The CPU 100 reads the eyeball image of the photographer from the image sensor 14 for which the accumulation has been completed, and at the same time, sequentially performs the process of extracting the P image and the feature of the pupil (# 20).
4). The specific method is disclosed by the present applicant in Japanese Patent Application No. 3-12 / 1993.
Since it is described in detail in Japanese Patent No. 19098, detailed description here is omitted.

After the reading of the entire eyeball image is completed and the extraction of a plurality of P images and pupil features is completed, a set of P image positions is detected based on the information (# 205).
As described above, since the P image is a corneal reflection image of the IRED for eyeball illumination, it appears in the image signal as a bright spot having a high light intensity.
The positions (xd ', yd') and (xe ', ye') can be obtained. In the case of an eyeball image as shown in FIG. 1, there is only one set of luminescent spots that seem to be a P image, so that its detection is easy. However, as shown in FIG. If so, it can often fail to detect the correct P image pair. P
In order to recognize that the image is an image, it is desirable to add the interval between the P images forming a pair or the size of the bright spot on the sensor in addition to the bright spot condition. This is because ghosts caused by glasses generally have large bright spots.

In any case, one P image pair is (# 205)
Then, the pupil center ( xc ', yc' ) and the pupil diameter ( rc ) are detected (# 206).

If the position of the P image and the pupil can be detected from the eyeball image of the photographer, the direction of the line of sight of the photographer or the coordinates on the finder can be calculated from equation (5) (# 20).
7).

In FIG. 8, next, the CPU 100 determines whether or not the result of the detected line of sight is usable (# 20).
8). Specifically, it is determined whether the detected rotation angle of the eyeball is not abnormally large in both the horizontal and vertical directions. This is because the rotation angle should not be too large as long as the photographer normally looks through the viewfinder. If the rotation angle is calculated to be larger than a predetermined value (for example, about 20 °), the P image may be erroneously detected. I think there is. The criterion may be not only the rotation angle but also whether the gazing point is inside or outside the visual field coordinates.

If the detection result is OK, it is regarded that "the line of sight has been detected successfully"(# 209), and the routine of line of sight detection is returned (# 210).

If the detection result is not normal, it is determined that there is a possibility that the P image pair has been erroneously selected, and it is checked whether another P image pair exists (# 211). In the case of an eyeball image as shown in FIG. 2, pairs of P images are (52a, 52b),
There are three sets (53a, 53b) and (54a, 54b). Of course, the correct P image pair is (52a, 52b)
And the pair of (53a, 53b) and (54a, 54b) is a ghost with glasses. If the P image pair of (52a, 52b) is selected from the beginning, a correct detection result is given. However, if the (53a, 53b) pair is selected first and gaze detection is performed, the detection result becomes abnormal. Therefore, if there are a plurality of P image pair candidates as in the example of FIG. 2, another P image pair is selected (# 212), and the eye gaze detection is attempted again (T1).

If there is only one P image pair in the first place as shown in FIG. 1, even if it is an incorrect P image pair, it is impossible to select another pair again. Regards the "line-of-sight detection failure"(# 213), and returns to the line-of-sight detection routine (# 214).

In the embodiment described above, if the result of the line of sight calculated based on the initially selected P image pair is normal, the processing is terminated there, otherwise, another P image pair is reselected. Then, when the result becomes normal, the processing of gaze detection ends.

However, when there are a plurality of P image pair candidates,
In particular, when a ghost caused by glasses exists very close to the true P image, performing a line-of-sight calculation using the ghost as a P image will give a moderate result, and the processing may be terminated.

Therefore, the second embodiment of the present invention proposes a processing method in which gaze calculation is performed for all P image pair candidates, and the most probable result is adopted from the results. are doing.

FIG. 9 shows a flowchart of the second embodiment.

When the calculation of the line of sight is completed in step # 207 of FIG. 7, the process proceeds to step # 220, and it is determined whether or not the calculation of the line of sight has been completed for all the P image pair candidates. In the example of the eyeball image as shown in FIG. 1, there is only one P image pair, so the calculation is completed once, the result is set as the final result (# 222), and the routine of the eye-gaze calculation is returned (# 222). 223).

In the example of an eyeball image as shown in FIG. 2, there are three pairs of P images. (In fact, a high-brightness area having a large spread is regarded as a ghost and is first rejected. You should not be a candidate for a pair,
Here, for the sake of explanation, it is assumed that there are three P image pair candidates in the example of the eyeball image in FIG. 2). Even if the result of the line of sight of the initially selected P image pair is good, Another P image pair is reset (# 221), and the line-of-sight calculation process is performed again.

When the calculation is completed for the three P image pairs, the most probable gaze result is adopted from the results (# 222). Specifically, a method of employing the result with the smallest angle among the detected eyeball rotation angles in each P image pair is realistic. Then, if the final result of the line of sight is adopted, the line of sight detection routine is returned (# 223).

[0085]

As described above, according to the present invention,
An incorrect gaze detection calculation result using a high-luminance portion other than the corneal reflection image can be eliminated, and more accurate gaze detection can be performed.

[Brief description of the drawings]

FIG. 1 is a diagram showing a normal photographer's eyeball image;

FIG. 2 is a diagram showing an eyeball image of a photographer wearing glasses.

FIG. 3 is a schematic view of a main part of a first embodiment in which the present invention is applied to a single-lens reflex camera.

FIG. 4 is an explanatory diagram in the viewfinder field of FIG. 3;

FIG. 5 is an electric circuit diagram of the optical device according to the present invention.

FIG. 6 is a flowchart of the operation in FIG. 6;

FIG. 7 is a flowchart of gaze detection according to the present invention.

FIG. 8 is a flow chart of gaze detection according to the present invention (first embodiment)
Example)

FIG. 9 is a flow chart of eye gaze detection according to the present invention (second example).
Example)

FIG. 10 is a diagram illustrating the principle of a conventional gaze detection method.

[Explanation of symbols]

 Reference Signs List 1 photographing lens 2 main mirror 6 focus detection device 6f image sensor 7 focus plate 10 photometric sensor 11 eyepiece 13 infrared light emitting diode (IRED) 14 image sensor (CCD-EYE) 15 eyeball 16 cornea 17 iris 21 LED for superimpose 24 LCD in viewfinder 25 LED for illumination 31 Aperture 50 White part of eyeball 51 Pupil of eyeball 100 CPU 101 Eye gaze detection circuit 103 Focus detection circuit 104 Signal input circuit 105 LCD drive circuit 106 LED drive circuit 107 IRED drive circuit 11 Focus adjustment circuit 200 to 204 AF point mark

Claims (4)

(57) [Claims] An illumination unit for illuminating an eyeball, an image sensor for obtaining an image of the eyeball illuminated by the illumination unit, and a part satisfying a predetermined condition from an eyeball image obtained by the image sensor is used as a candidate for a corneal reflection image. A corneal reflection image candidate detection unit that detects the corneal reflection image candidate, a sight line calculation unit that calculates sight line information using the corneal reflection image candidate detected by the corneal reflection image candidate detection unit, and a sight line information calculated by the sight line calculation unit. And determining means for determining whether or not a predetermined standard is satisfied.When the determining means determines that the line-of-sight information does not satisfy the predetermined standard, the corneal reflection image candidate detecting means includes A gaze detection device, wherein another gaze reflection image candidate is detected, and the gaze calculation means calculates gaze information again. 2. The gaze detection apparatus according to claim 1, wherein the gaze information includes a rotation angle of the eyeball, and the predetermined reference is that the rotation angle of the eyeball is within a predetermined angle. 3. The line of sight according to claim 1, wherein the line of sight information includes position information of a gazing point, and the predetermined reference is that the position information of the gazing point is within the field of view coordinates of an observation screen. Detection device. 4. An illuminating unit for illuminating an eyeball, an image sensor for obtaining an image of the eyeball illuminated by the illuminating unit, and a part that satisfies a predetermined condition from an eyeball image obtained by the image sensor is a corneal reflection image candidate. A plurality of corneal reflection image candidate detection means for detecting a plurality of corneal reflection image candidate detection means, and a plurality of eye-gaze calculation means for calculating a plurality of rotation angles of an eyeball using the plurality of corneal reflection image candidates detected by the corneal reflection image candidate detection means. Selecting means for selecting the smallest rotation angle from among the plurality of calculated rotation angles of the eyeballs.