JPH06262959A - Alarm device for distracting driving - Google Patents

Abstract

PURPOSE:To automatically give alarm for distracting driving constantly and accurately by photographing the eyes of a driver two-dimensionally and processing the image, thereby judging distracting driving when the position change in the eyes is not matched with the proceeding direction of a vehicle, and by giving alarm. CONSTITUTION:The face of a driver who drives a vehicle is photographed by a CCD camera 3 under the light of a floodlight 4. On one hand, the eyes in the photographed image are detected by an input/output control CPU 6 and a calculation CPU 8 while the position change of the eyes is detected. On the other hand, the proceeding direction of the vehicle is detected by an angle velocity detector 21 and an angle velocity detection circuit 20. When the position change of the eyes does not match with the proceeding direction of the vehicle, it is judged that the driver is driving distractingly. A buzzer 5 is activated through a buzzer controller 19, and the attention of the driver is thus provoked. Alarm for distracting driving can thus be issued automatically, and constantly correctly.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a vehicle driving warning device, and more particularly to a driver's face photographed by a video camera to detect an eye image on the image by image processing and monitor the eye image in time series. The present invention relates to an alarm device that automatically issues an alarm for a driver's behavior that impairs vehicle safety.

[0002]

2. Description of the Related Art For example, in Japanese Examined Patent Publication No. 3-14656, in order to prevent a driver (driver) from falling asleep, infrared rays are radiated to the driver's eyes to detect reflected light, and the reflected light detection value is set. There is proposed a technique of determining blinking based on the blinking frequency and detecting drowsiness of the driver based on the blinking frequency. The drowsiness detection device is equipped with glasses. Japanese Patent Publication No. 4-284 discloses a technique in which the face of a vehicle driver (driver) is photographed by a camera and the eyes and mouth are recognized on the photographed image. This is because it is not necessary to attach anything special to the driver, the driver's face is photographed with a video camera, the eyes are detected and tracked on the photographed image, the eye opening (open, closed) is calculated, and the eyes are opened. Whether or not the eyes are closed is determined based on the degree. The brightness of the lamp that illuminates the driver's face is optimally adjusted to recognize the eyes and mouth based on the image data. The image data is stored in the frame memory and binarized. The driver's eyes and mouth are detected based on the binary data.
If the eyes are closed for too long, the driver is considered dozing and an alarm is issued.

[0003]

When driving a vehicle, not only the driver's drowsiness but also the inattentiveness may impair the safety of the vehicle traveling. If there is a device for automatically detecting the inattentive and issuing an alarm, it is safe to drive the vehicle. It is expected that the sex will improve further.

An object of the present invention is to automatically give an alarm to a driver looking aside.

[0005]

The inattentive driving warning device of the present invention is a two-dimensional image pickup means (3) for photographing an on-vehicle driver;
Image detecting means (6, 8) for detecting an eye image on the image photographed by the two-dimensional imaging means (3); Calculate a change in the position of the eye image detected by the image detecting means (6, 8) Movement detecting means (6, 8); Direction detecting means (21, 20) for detecting the traveling direction of the vehicle; Change calculated by the movement detecting means (6, 8) and the direction detecting means (2
Inattentive judgment means (6,8) for detecting the match / mismatch of the vehicle traveling directions detected by (1,20); warning means (19,5); and in accordance with the detection of inconsistency by the inattentive judgment means (6,8). An alarm control means (6, 8) for activating the alarm means (19, 5) is provided. In addition, the symbols in parentheses are attached to corresponding elements or corresponding matters in the embodiments shown in the drawings and described later.

[0006]

When the driver on the vehicle looks aside, the direction of the driver's face shifts from the front of the vehicle to the right or left, and the two-dimensional image pickup means.
When the driver is photographed in (3), the position of the eye image shifts to the right or left on the photographed image. That is, the position of the eye image changes. The movement detecting means (6, 8) calculates this deviation from the position of the eye image detected by the image detecting means (6, 8). By the way, when the vehicle is turning, unlike the longitudinal axis of the vehicle and the traveling direction of the vehicle, the driver looks in the turning direction, so the position of the eye image on the photographed image is in the turning direction. Shift to (right or left). That is, in the curve of the road, the position change of the eye image occurs as in the case of looking aside when traveling straight ahead. In the inattentive driving warning device of the present invention, the direction detection means (21,
20) detects the traveling direction of the vehicle, and the inattentiveness judging means (6, 8) detects a change in the position of the eye image and a match / mismatch between the vehicle traveling directions. In the case of coincidence, even if there is a change in the position of the eye image, it is due to the curve running. Then alarm control means
Since (6, 8) activates the alarm means (19, 5) in accordance with the detection of the inconsistency, the alarm is automatically issued when the driver's line of sight is different from the traveling direction of the vehicle.

By the way, the driver is a room mirror (a back mirror inside the vehicle) or a side mirror (a door mirror outside the vehicle).
Alternatively, when looking at the hood mirror, the driver's line of sight deviates from the traveling direction of the vehicle. This rearward confirmation or lateral confirmation is a short time. Therefore, in the preferred embodiment of the present invention, the alarm control means (6, 8) starts the clocking in response to the detection of the inconsistency, and if the detection of the inconsistency continues, the alarm means (19, 5) after the set time (1 second). Shall be activated. According to this, the alarm cannot be issued in the case of looking aside for a short time (less than 1 second) or rearward confirmation or lateral confirmation for a short time (less than 1 second). Even if you check backward or sideward, if it exceeds the set time (1 second), an alarm will be issued.

Further, the base point (reference) of the position change of the eye image is
It is possible to set a fixed value in the device in advance assuming the eye image position when the driver looks at the front of the vehicle in normal vehicle driving, or the driver inputs or adjusts position information using the operation board. In addition, the driver operates the set switch of the device when the driver considers the posture to be the most stationary, and the device stores the eye image position at that time as a reference position, or it can be determined in various modes. In a preferred embodiment of the present invention, the movement detecting means (6, 8) is configured so that the first average position of the eye image position in a relatively short time section (0.5 seconds) and a relatively long time section (5 seconds). ) Is calculated, and the deviation of the first average position from the second average position is taken as the change in the position of the eye image. According to this, the second average position in a relatively long time section (5 seconds) is the eye image position during steady driving looking in the vehicle traveling direction.
(Time series center value or representative value), and the first average position in a relatively short time section (0.5 seconds) is, so to speak, an instantaneous position. It is common for the driver's driving posture to change over time, but this change is gradual. The second average position follows this change. Therefore, the deviation of the first average position from the second average position (that is, the deviation of the instantaneous value from the time-series center value) is an unsteady position change amount, for example, the inattentive or mirror.
This is due to confirmation. According to this embodiment, therefore, the probability of giving a false alarm due to the time-series variation of the driver's driving posture is low. Other objects and features of the present invention will become apparent from the following description of embodiments with reference to the drawings.

[0009]

EXAMPLE FIG. 1 shows the configuration of an example. In this embodiment, the driver's face on the vehicle is photographed to detect the eyes in the captured image, the eyes are tracked in time series to recognize whether the eyes are open or closed, and the eyes are closed for a predetermined time or longer. When it is sleeping (sleeping), it activates the buzzer 5. As shown in FIG. 2, the CCD camera 3 and the illuminating lamp 4 for illuminating at least the face portion of the driver are integrally configured to form an instrument panel 2.
In addition, it is fixed so that the pointing direction can be adjusted vertically and horizontally. A buzzer 5 for generating an alarm and a stop switch SS for stopping the alarm of the buzzer 5 are provided on the side surface of the CCD camera 3.
W is provided. The illuminating lamp 4 is an infrared illuminator composed of a light emitting diode, and the CCD camera 3 has sensitivity in the infrared region. The infrared illumination does not give glare to the driver, unlike the illumination by ordinary light, and thus does not make the driver's front view difficult. Therefore, the illumination lamp 4 may be constantly turned on. In FIG. 2, reference numeral 1 is a steering wheel operated by a driver for controlling the traveling direction of the vehicle.

Referring again to FIG. The buzzer 5 is turned on / off by the buzzer controller 19. As will be described later, the buzzer 5 is urged when the driver's eyes are closed for a predetermined time or longer (assuming that the driver is asleep). The illumination lamp 4 is energized by the infrared lamp controller 18. Buzzer controller 19, infrared lamp controller 18, other outputs (controller that controls radio, air conditioner, wiper, headlight, etc.)
Includes a microprocessor 6 and an interface 13
An on / off instruction signal or the like is given via.

An angular velocity detector 21 is mounted on the vehicle, and the detector 21 is excited and excited by the angular velocity detection circuit 20. When the vehicle is in the right turn, the positive turn is made in the left turn. In the case of, a negative rotation angular velocity signal is generated, and the detection circuit 20 calibrates (waveforms and amplifies) this angular velocity signal and supplies it to the parallel interface 13. The interface 13 discriminates the polarity of the rotational angular velocity signal and gives a signal indicating the polarity to the microprocessor 6.

The CCD camera 3 is provided with a two-dimensional CCD, and the microprocessor 6 gives an ON / OFF signal to the CCD camera 3 via an interface 16. The CCD camera 3 repeats the captured video signal (video signal: analog) to repeat the A / D converter 1
7 and outputs the pixel sync pulse to the A / D converter 17 as an A / D conversion sync pulse, and also applies the frame sync pulse, the line sync pulse and the pixel sync pulse to the microprocessor 6 via the interface 16.

The microprocessor 6 uses the image data 1
When reading a frame, the frame memory (RAM) 12 is instructed to write in synchronization with the frame sync pulse, and the write address of the frame memory 12 is advanced in synchronization with the line sync pulse and the pixel sync pulse. In the frame memory 12, in this example, the CCD camera 3 captures one screen with 512 horizontal pixels and 485 vertical pixels.
An image memory capable of storing 256-gradation image data for 2 × 485 pixels is used.

The A / D converter 17 converts the video signal into 8
It is converted into digital data of bits (256 gradations). That is, the image data written in the frame memory 12 is 8 bits (256 gradations) per pixel (hereinafter, dot).
Is data indicating the brightness. Therefore, the microprocessor 6 can know the brightness at any position on the screen by reading the value in the frame memory 12.

The microprocessor 6 is connected to another microprocessor 8 for performing various image processing and arithmetic operations for determination, and a ROM 9 and a RAM 10 for controlling a normal computer system, a bus controller 15 and the above-mentioned. The frame memory 12 of is connected.

FIG. 3 shows an outline (main routine) of the operation of the microprocessors 6 and 8, and details of the operation (subroutine) are shown in FIGS. The control operation of the microprocessors 6 and 8 based on the set program will be described below with reference to these drawings.

First, the outline of the operation will be described with reference to FIG. When the power is turned on, the microprocessor 6 performs initialization (step 1: hereinafter, the word step is omitted in parentheses). In this, the I / O ports are initialized, the internal registers, flags, etc. are cleared, and the CCD
The camera 3 and the infrared lamp controller 18 are supplied with an energizing signal indicating ON, and the buzzer controller 19 is supplied with a signal indicating OFF. Further, the microprocessor 6 is similar to that shown in FIG.
As shown in FIG. 6, image data to be written in the frame memory 12
25 screens in the center of the screen,
A total of 425 points of interest (cursors), 17 in the vertical direction,
It is arranged at intervals of 15 dots horizontally. This interval varies depending on the size of the human eyes on the screen, but as shown in FIG. 17, the maximum vertical width of one eye on the shooting screen of the CCD camera 3 is 30 dots. ， Horizontal direction is 40-5
Since there are 0 dots, the interval is 15 dots in this embodiment so that at least one target point exists in one eye area. As shown in FIG. 16, when the horizontal direction of the image written in the frame memory 12 is the x-axis and the vertical direction is the y-axis (the upper left corner coordinates (0, 0) of the image, the lower right coordinates (511,
484)) and the upper left corner coordinates of the 425 points of interest are (68,
200), the target point is arranged so that the lower right coordinate is (428, 440), and its coordinate data (X _m , Y _n ) is calculated as shown in FIG.
_Write to address A _nm in the memory table of RAM (memory) 10 (FIG. 1) as shown in FIG.

However, X _m = 68 + 15 (m-1); 1
≦ m ≦ 25, Y _n = 200 + 15 (n−1); 1 ≦ n ≦ 17.

Then, a predetermined time (lighting lamp 4
Of the image data for one frame in synchronization with the frame sync pulse from the CCD camera 3 after waiting for the brightness of the driver to stabilize and the driver's face to stop at the driving position). Writing is started, and image data for one frame (8-bit multi-gradation data per dot) is written in the memory 12 (2).

Next, the microprocessor 6 sets a small area centered on each target point and updates the coordinates of the target point to the darkest point in the area (3). This operation is performed for all 425 points of interest. The contents will be described later with reference to FIG.

Next, the microprocessor 6 erases the isolated points having no other points of interest from the points of interest after the coordinate update, or combines other points of interest in the vicinity of a certain point of interest into one. Etc., and finally, it is determined whether the number N of attention points is the number of eyes of a person, that is, whether there are two attention points (4). The contents will be described later with reference to FIGS. Finally, when the number N of attention points is not 2, the process returns to step 2.

If there are two points of interest, the coordinates of the points of interest are regarded as indicating the positions of both eyes of a person, and the points of interest are tracked each time each frame is written in the memory 12, and the points of interest are tracked in time series. Then, the open / closed state of both eyes is detected to determine whether the driver is dozing or looking aside (5). The contents will be described later with reference to FIGS. 13 and 14. If tracking becomes impossible during the tracking process, the process returns to the initialization of step 1.

Referring to FIG. 4, "dark spot detection in a small area"
The contents of (3) will be described. This is a process of setting a small area within a range of ± 10 dots in the vertical and horizontal directions around the target point, and updating the coordinates of the target point to the coordinates indicating the darkest point in the small area.

First, the microprocessor 6 sets 1 in the registers m and n (31) and reads the coordinate data X _m and Y _n at the address A _nm of the memory shown in FIG. 18 (3
2). Initially, the coordinate data X ₁ and Y ₁ of the address A ₁₁ are read. Next, register x ₁ has X _m −10 and y ₁ has Y _n −1.
Set 0 to set the coordinates of the upper left corner of the small area (3
3), X _m +10 is set in the register x ₂ and Y _n +10 is set in y ₂ to set the coordinates of the lower right corner of the small area (34). Then, the data of the frame memory 12 is read in the area x ₁ ≤x≤x ₂ & y ₁ ≤y≤y ₂ to detect the darkest point of the brightness value, and the coordinates are updated to the memory address A _nm. Yes (35). Next, the register m is incremented by 1 (36), and the processes of steps 32 to 36 are repeated until the value of m becomes larger than 25 in step 37. That is, this allows the coordinates (X ₁ ,
Y ₁ ), (X ₂ , Y ₁ ), ..., (X ₂₅ , Y ₁ ) are updated. When the register m becomes larger than 25 (37), the value of m is set to 1 and the register n is incremented by 1 (3
8), the processes of steps 32 to 38 are repeated until the value of n becomes larger than 17 in step 39. As a result, the coordinates (X ₁ , Y ₂ ) of the point of interest, (X ₂ , Y ₂ ), ...
(X ₂₅ , Y ₂ ), (X ₁ , Y ₃ ), (X ₂ , Y ₃ ), ...
(X ₂₅ , Y ₃ ), ... Are updated, and when the coordinates of a total of 425 points of interest are updated (YES in step 39), the routine returns. As a result, the points of interest arranged at equal intervals during initialization move to dark areas such as the human eyes, eyebrows, and hair.

The control operation of the "determination" (4) will be described with reference to FIG. In the judgment, since a plurality of points of interest are gathered on the black eye, the isolated point is deleted.
1), when there are multiple points of interest around the point of interest, they are integrated into one point of interest, "deletion of other points of interest near the point of interest"
(42), comparing the average gray scales in the vertical and horizontal directions centering on the point of interest and deleting the point of interest with darker vertical gray scale, "deletion by cross gray scale difference" (43), detected in advance “Delete by template matching”, which removes the points of interest with a large comparison error by comparing it with the template image
(44), comparing the average gray scales in the upper right diagonal direction and the upper left diagonal direction centering on the target point to delete the target point having a large comparison error, "deletion due to diagonal direction gradation difference" (45), remaining By moving the point of interest to the darkest point in the small area by the same processing as in step 3, "dark point detection in small area" (4
6), the process of "unifying neighboring attention points" (47), which integrates the attention points that are close to each other, is performed, and whether or not the number N of the finally remaining attention points is 2 for both eyes Then, the “determination of the number of points of interest” (48) is executed.

Referring to FIG. 6, "deletion of isolated points" (4
The control operation 1) will be described in more detail. First, the microprocessor 6 sets ₁ in the registers m ₁ and n ₁ (410
1), the values of the registers m ₁ and n ₁ are set in the registers m and n (4102), and the coordinate data X _m and Y _{n of the} memory address A _nm are set.
Is read (4103). m ₁ and n ₁ are registers indicating points of interest. Initially, the coordinate data X ₁ and Y ₁ of the address A ₁₁ are read. Next, register x ₁ has X _m −10 and y ₁ has Y _n −1.
0 is set to set the coordinates of the upper left corner of the area for detecting the presence or absence of another target point (4104), and X _m + is set in the register x _2.
10. Set y ₂ to Y _n +10 to set the coordinates of the lower right corner of the detection area (4105).

Next, 1 is set in the registers m ₂ and n _2.
(4106). The registers m ₂ and n ₂ are registers indicating other points of interest with respect to the points of interest indicated by m ₁ and n ₁ . Then m ₁ = m ₂ &
It is checked whether n ₁ = n _2, that is, whether the target point is the same as another target point (4107). If they are the same, it is not necessary to delete the isolated points, and therefore the process proceeds to step 4112, which will be described later.
M ₂ and n ₂ are set to (4108), and it is checked whether or not data exists at the memory address A _nm (4109). If there is no data, the process proceeds to step 4112. The data X _m and Y _n are read (4110). Initially, 1 is set in the registers m ₁ and n ₁ and m ₂ and n ₂ , so the flow advances to step 4112 to increment m ₂ by 1 and step 4
Since the process returns to 107, the coordinate data X ₁ and Y _{1 at the} address A ₁₁ are read as the target points and the coordinate data X ₂ and Y _{1 at the} address A ₂₁ are read as the other target points.

Next, it is checked whether or not the coordinate data X _m , Y _n of another target point exists in the detection region of the target point (411).
1) If it exists, the point of interest is not an isolated point and is left as it is, but if it does not exist in the detection area, there is one whether all other points of interest also exist in the detection area. Repeat until it is detected. That is, m ₂ is incremented by 1 until m ₂ becomes larger than 25 (4112,411
3), the n ₂ sets m ₂ to 1 when greater than 25 1 increments and (4114), n ₂ repeats the processing at steps 4107 to 4111 until greater than 17 (4115). Then, when there is no other target point in the detection area centered on the target point (YES in step 4115), m ₁ in the registers m and n,
sets the n ₁ (4116), to erase the address A _nm coordinate data in the memory (4117).

If another point of interest exists in the area where the coordinate data is erased or detected, the register m ₁ indicating the point of interest is set to 1
Increment (4118), return to step 4102, and leave the above processing (YES at step 4109) and repeat for all points of interest (4118-4121-4102-4118 ...). When the determination of whether all the points of interest are isolated points is completed (Y in 4121
ES), return. That is, ± 10 around the point of interest
If there is no other point of interest in the dot area, and if there is no other point of interest, the reference point of interest is the memory 10
Erased from. As a result, human eyes, eyebrows, hair, etc.
The attention points moved to the dark part are left, but the other attention points are erased from the memory table. With reference to FIG. 7, the control operation of “deletion of other attention points near the attention point” (42) will be described in more detail. First, the microprocessor 6
Sets 1 to registers m ₁ and n ₁ (4201), and register m 1
The values of ₁ and n ₁ are set in the registers m and n (4202), and it is checked whether data exists at the memory address A _nm (4203). This is because the data may have been erased from the memory 10 by the "deletion of isolated points" (41) process. It should be noted that m ₁ and n ₁ are registers indicating a reference point of interest. If there is no data, the process proceeds to step 4217, but if there is data, the coordinate data X _m and Y _n are read (4204). Next, X _m -5 is set in the register x ₁ and Y _n -5 is set in y ₁ to set the coordinates of the upper left corner of the area for detecting the presence or absence of another target point (4205), and register x ₂ To X _m +5 and y ₂ to Y _n +
Set 5 to set the coordinates of the lower right corner of the detection area (42
06).

Next, the register m ₂ is set to m ₁ +1 and the register n ₂ is set to n ₁ (4207). As in "Deletion of isolated points" (41), m ₂ , n
_The reason why 1 is not set to ₂ is that the data remaining at the address A _nm of the memory is outside the detection area with respect to the point of interest before that and does not need to be detected in duplicate. Therefore, the registers m ₁ and n ₁ indicating the reference point of interest
On the other hand, the registers m ₂ and n ₂ of the other points of interest, which are the objects of existence in the detection area, may be checked if they are larger than m ₁ and n ₁ .

Next, m ₂ and n ₂ are set in the registers m and n (4208), and it is checked whether or not data exists at the memory address A _nm (4209). If there is no data, step 42
12, the coordinate data X _m and Y _n are read if there is data (4210).

Next, it is checked whether or not the coordinate data X _m , Y _n of another target point exists within the detection region of the target point (421).
1) If it exists, the point is near the target point, and the coordinate data of the address A _nm of the memory is deleted (4216). on the other hand,
If it is outside the detection area, the point is left. These processes are performed for all remaining attention points (4212 to 4215 to 4217 to 4220 to 4202 to 4212, ..., Similar to `` deletion of isolated points '' (41).
・). When the determination as to whether all the points of interest are isolated points is completed (YES in 4220), the process returns. That is, the other points of interest within the ± 5 dot area around the point of interest are the memory 10
And the target point is set to be one in the area. As a result, among the points of interest that have moved to dark areas such as human eyes, eyebrows, and hair, the points of interest in the vicinity are set to one, and the number of remaining points of interest decreases.

With reference to FIG. 8, the control operation of "deletion by cross gradation difference" (43) will be described in more detail. First,
The microprocessor 6 sets 1 in the registers m and n
(4301), it is checked whether or not data exists at the memory address A _nm (4302). If there is no data, step 43
The process proceeds to 10, the coordinate data X _m If there is data, it reads out the Y _n (4303), the X _m -20 to register x ₁ to y ₁ Y _n -
20 is set (4304), and X _m +20 is set to y in the register x _2.
Set Y _n +20 to ₂ (4305).

Next, the area, y = Y _n & x ₁ ≦ x ≦ x ₂ , that is, the brightness value of each point within ± 20 dots in the horizontal direction centering on the target point is detected, and the sum is set in the register T _H. (4306), the area, x = X _m & y ₁ ≦ y ≦ y ₂ , that is, the brightness value of each point in the vertical direction ± 20 dots centering on the target point is detected, and the sum thereof is set in the register T _V. (430
7). Then, T _H and T _V are compared (4308), and only when the vertical gradation T _H is larger than the horizontal gradation T _V , the target point is erased from the memory address A _nm (4309).

This processing is performed by "deleting other attention points near the attention point".
Perform all remaining attention points at the end of (42) (4310-4
313-4302-4310, ...). That is, as shown in FIG. 17, the size of human eyes is longer in the horizontal direction than in the vertical direction, and the horizontal gradation is darker than the vertical gradation. Memory as not in the eye position of the person
Erase from 10. As a result, the number of remaining attention points that have moved to dark areas other than the human eyes and eyebrows, particularly hair, is reduced.

The control operation of "deletion by template matching" (44) will be described in more detail with reference to FIG. First, the microprocessor 6 sets 1 in the registers m and n.
Is set (4401), and it is checked whether data exists at the memory address A _nm (4402). The process proceeds to step 4408 if there is no data, if there is data the coordinate data X _m, reads Y _n (4403), the X _m -30 sets Y _n -20 to y ₁ to the register x ₁ (4404 ), X _{m in} register x ₂
Set +30 to y ₂ and Y _n +20 (4405). Next, the brightness value of each point in the area, x ₁ ≦ x ≦ x ₂ & y ₁ ≦ y ≦ y ₂ , and the area of the template corresponding to the position of each point,
x ₀ −30 ≦ x ≦ x ₀ +30 & y ₀ −20 ≦ y ≦ y ₀ +2
The sum of the absolute values of the differences in the brightness value of each point within 0, is detected,
The register T _nm is set (4406). That is, the tonality of each point (2400 = 60 × 40) in the range of ± 20 dots in the vertical direction and ± 30 dots in the horizontal direction around the point of interest and the center of the human eye (x ₀ , y ₍₀ ) Similarly, each point in the range of ± 20 dots in the vertical direction and ± 30 dots in the horizontal direction (2400 = 60 × 40)
And the gradient of each of the corresponding points, (X _m −30, Y _n −20)
_{(x 0 -30, y 0 -20} ), (X m -29, Y n -20) and (x ₀
−29, y ₀ −20), (X _m −28, Y _n −20) and (x ₀ −2
8, y ₀ -20), ..., (X _m , Y _n ) and (x ₀ , y ₀ ),
_{··, (X m + 30,} Y n +20) and _{(x 0 + 30, y 0} +2
0), respectively. And the set T
Save _nm with address A _nm (4407).

This processing is performed for all remaining attention points at the end of the "deletion by cross tone difference" (43) (4408 to 4411 to 4).
402-4408, ...). Next, the maximum value and the minimum value in the total sum group are detected, the intermediate value T ₀ between them is calculated (4412), and the coordinate data of the address whose total sum is larger than the intermediate value T ₀ is erased from the memory (4413). This reduces the number of remaining attention points that have moved to dark areas such as eyebrows and hairs other than human eyes.

With reference to FIG. 10, the control operation of "deletion by diagonal gradation difference" (45) will be described in more detail. This is a process of performing deletion by gradation difference at a position obtained by rotating the cross in the vertical and horizontal directions centered on the point of interest of the above "deletion by cross gradation difference" (43) by 45 degrees. First, the microprocessor 6 sets 1 in the registers m and n (450
1), it is checked whether or not data exists at the memory address A _nm (4502). Proceeds Without data to step 4512, if there is data the coordinate data X _m, reads _{Y n (4503), Y n} -20 to X _m -20 to y ₁ in the register x ₁
Sets (4504), the X _m +20 sets Y _n +20 to y ₂ into register x ₂ (4505).

Next, the brightness value of each point is detected on a straight line connecting (x ₁ , y ₁ ) and (x ₂ , y ₂ ), that is, in the left diagonally upper direction to the right diagonally lower direction passing through the center of the point of interest. Then, the sum is set in the register T _RD (4506), and (x ₁ , y ₂ ) and (x ₂ ,
y ₁ ) on a straight line, that is, the brightness value of each point in the diagonally right upper direction to the left diagonally lower direction passing through the center of the point of interest is detected,
The sum is set in the register T _RU (4507). Then, T _RD and T _RU are compared (4508), and it is checked whether the difference between the two is within + 20% of the smaller gradation sum (4509, 4510). The point is deleted from the memory address A _nm (4511).

This processing is performed for all remaining points of interest at the end of the “deletion by template matching” (44)
(4512-4515-4502-4512, ...). That is, since there is almost no difference between T _RD and T _{RU for} the point of interest located at the center of the human eye, the point of interest with a large difference is deleted from the memory 10. As a result, the number of remaining attention points that have moved to the edge of the human eye or the edge of the eyebrow decreases.

Thereafter, the remaining points of interest are identified in step 46.
In (Fig. 5), the small area and the point of interest are centered in the vertical and horizontal directions ± 2.
The "detection of dark spots in a small area" (46) of moving the target point to the darkest spot within the range of 0 dots is executed. This process (not shown) is the same as step 3 (FIG. 4), except that the area range is ± 20 dots (± 10 dots in FIG. 4) and that between step 31 and step 32 in FIG. The difference is that a determination process of "data exists at memory address A _nm ?"

Referring to FIG. 11, "integration of neighboring points of interest"
The control operation (47) will be described in more detail. In "Delete other attention points in the vicinity of attention point" (42), other attention points within the range of ± 5 dots in the vertical and horizontal directions centering on the attention point were deleted, but in "Integration of neighborhood attention points" (47) , Horizontal and vertical ± 2
If another point of interest exists within the 0 dot range, the point of interest is moved to an intermediate position between the two points and the other point of interest is deleted. In addition, in the "deletion of other attention points near the attention point" (42), the registers m ₁ and n ₁ indicating the reference attention points are set to the other attention points which are the objects of presence or absence in the detection area. It suffices to check registers m ₂ and n _{2 that} are larger than m ₁ and n ₁ , but in “Integration of neighboring attention points” (47), the reference attention point moves, so Check all remaining attention points.

First, the microprocessor 6 uses the register m.
₁ is set to ₁ and n ₁ (4701), the values of the registers m ₁ and n ₁ are set to the registers m and n (4702), and it is checked whether or not data exists at the memory address A _nm. (4703) If there is no data, the process proceeds to step 4722, but if there is data, the coordinate data X _m1 and Y _n1 at the address A _nm of the memory are read (4704). m ₁ and n ₁ are registers indicating a reference point of interest. Next, register x ₁ has X _m1 -20 and y ₁ has Y _n1 -2
Set 0 to set the coordinates of the upper left corner of the area for detecting the presence / absence of another target point (4705), and set X _{m1 in the} register x _2.
The coordinate of the lower right corner of the detection area is set by setting Y _n1 +20 to y ₂ of +20 (4706).

Next, 1 is set in the registers m ₂ and n _2.
(4707). The registers m ₂ and n ₂ are registers indicating other points of interest with respect to the points of interest indicated by m ₁ and n ₁ . Then m ₁ = m ₂ &
It is checked whether n ₁ = n _2, that is, whether the target point is the same as another target point (4708). If they are the same, it is not necessary to integrate them, and the process proceeds to step 4718 described later. If they are not the same, m ₂ and n ₂ are set in the registers m and n (4709), and the data is stored in the memory address A _nm. It is checked whether or not it exists (4710), and if there is no data, the process proceeds to step 4718, but if there is data, the coordinate data X _m2 , Y _n2 is read (4711).

Next, the coordinate data X _m2 , Y _{n2 of the} other points of interest.
Check whether or not exists in the detection area of the point of interest (4
712) If it does not exist, there is no need to integrate it, and it is left as it is, but if it exists in the detection area, both points of interest are integrated. That is, the intermediate position (X _i , Y _i ) between the reference point of interest and the target point of interest is calculated (X _i = (X _m1 + X _m2 ) / 2,
Y _i = (Y _m1 + Y _m2 ) / 2), the attention point is moved to the intermediate position (X _i , Y _i ) and the coordinate data of the address A _nm of the memory is updated and stored (4713 to 4715) to be the target. The coordinate data of the target point is erased from the memory address A _nm (4716, 4717).

This processing is performed for all remaining points of interest.
(4718-4725-4702-4718, ...). As a result, the point of interest approaches the position indicating the center of the human iris.

Referring to FIG. 12, "determination of the number of points of interest"
The control operation (48) will be described in more detail. First, the microprocessor 6 registers 1 in registers m and n and 0 in N.
Are set (4801), and it is checked whether data exists at the memory address A _nm (4802). If there is no data, the process proceeds to step 4805, but if there is data, the register N is incremented by 1 (4803), and it is checked whether N is 2 or less (4804). If N is 2 or less, it is checked whether or not data exists in all areas of the memory (4805 to 4809 to 4802 to 4805, ...). That is, it is checked whether the number of data existing in the memory 10, that is, the number of attention points is the number of human eyes (= 2). If N becomes greater than 2 during this check (step 4804NO),
Alternatively, if N is not 2 at the end of the check for all areas (step 4810 NO; when N = 0 or 1), it is determined that the positions of both eyes cannot be specified by the points of interest, and the process returns to step 2 of FIG. Is input, and the above-mentioned “dark spot detection in small area” (3) and “determination” (4) are repeated until N becomes 2.

As a result, if N is 2, the positions of both eyes are specified by the point of interest, so the coordinate data (m ₁ , n ₁ ) and (m ₂ , n ₂ ) in the memory are saved (4811). . FIG. 19 shows a state in which the last two remaining attention points specify the positions of both eyes of a person.

FIG. 13 and FIG. 14 show "Additional information" shown in FIG.
The content of "trace" (5) is shown. In this
The server 6 first stores one frame of image data in the frame memory.
Write on 12 (501). And m in the registers m, n₁, N₁
Set, that is, one of the two eyes of a person
Set the value that indicates the address of the point of interest that indicates the eye position
(502). Then, from the memory address
Coordinate data X to show_m, Y_nIs read (503). Next, "President
Area detection (in this case, the point of interest in this case).
(The range of ± 20 dots in the horizontal and
Wachi register x₁To X _m-20 to y₁To Y_nSet -20
Then set the coordinates of the upper left corner of the small area (504) and register x₂
To X_m+20 to y₂To Y_nSet +20 for the detection area
Set the coordinates of the lower right corner (505). And the region, x₁≤
x ≦ x₂& Y₁≤ y ≤ y₂In the frame memory 12
Data is read to detect the darkest point in the brightness value,
The coordinates are the address A of the memory_nmUpdate memory to (50
6). This allows the position of one eye to change over time.
To be detected. Next, the open / closed state of the eyes is checked. sand
That is, the vertical direction centered on the point of interest after moving to the darkest point
Maximum brightness value T within ± 20 dot range_(max)And up
Small value T_(min)Detected, calculate the average value, and register
T_AVSet to (507). And T_AVThe following brightness values
Showing the width W in the vertical (y) direction₁Is detected (508). FIG.
In the schematic diagram shown in, the width in the vertical direction is W₁= | Y'-
y ″ |

The same process is performed for the other eye to detect the width W ₂ in the vertical (y) direction (509 to 515). As a result, the degree of opening / closing of both eyes that changes in time series is detected.

Next, the widths W ₁ and W _{2 of} both eyes are set to a predetermined value W _0.
It is checked whether or not it is the following (516), and if any of the eyes is larger than the predetermined value W _0, it is determined that the driver is normal (not in the dozing state), the “inattentiveness determination” 525 is executed, and the routine proceeds to step 501. Return The above processing (501 to 516 to 501, ...) Is continued. The contents of the “side-looking determination” 525 will be described later with reference to FIG. When the widths W ₁ and W _{2 of both} eyes become equal to or less than the predetermined value W ₀ , it is checked whether the flag F indicating whether the timer is operating is 0 (not operating) (517). If the flag F is 0, The microprocessor 6 turns on the internal timer (518),
The flag F is set to 1 to indicate that the timer is operating (519). Then, it is checked whether the timer is over (520), and if the timer is not over, the process returns to step 501 and repeats the process, but if the timer is over, it is determined that the driver is in the dozing state and the buzzer controller 19 is set. Via the driver, a warning is continued until a stop instruction is issued from the driver (stop switch SSW is turned on by the driver) (521 to 52).
2). It should be noted that while the timer is once turned on and the timer has not expired, the width of one of the eyes of the driver is equal to the predetermined value W _0.
If it becomes larger, the (516) flag will be set to 0.
(524), the alarm is not issued in the state where the eyes are temporarily closed (less than the set time of the timer) such as the driver's natural knocking. If the driver gives a stop instruction while the alarm is on, the process returns to the initialization of step 1 (FIG. 3).

Next, referring to FIG. 15, a "look aside" 52
The contents of 5 will be described. It should be noted that this “inattentive judgment” 525
Determines that at least one of the left and right eyes is open (see 5 in FIG. 14).
16) Please note that it is executed while doing. Here, first, the data of address 49 of table Wa (one area of memory) is set to address 50 and the data of address 48 is set.
Data to address 49, table Wa
Data (address of) is shifted to the address 1 opened by this shift to the latest black eye position coordinates (Xm, Y
n) (at step 506 in FIG. 13, the memory address An1
The position data written in m1) is written (5251).

Since "tracking" 5 including this "looking aside" 525 is repeatedly executed at a cycle of 0.1 sec, when the eyes are continuously open, the address 1 of the table Wa is calculated this time. The W value is 0.1 seconds from address
The previous calculated value is ... And the calculated value 5 seconds before is stored in the address 50.

As described above, the latest iris position coordinates (Xm,
Yn) is written to the address 1, the second average coordinates (Xm5, Yn5) of the coordinates of the addresses 1 to 50 of the table Wa
Calculate (center value between the present and the past 5 seconds) (525
2) Furthermore, the first average coordinates (Xmo, Yno) of the coordinates of addresses 1 to 5 of the table Wa (from the present to the past 0.5se).
The central value between c) is calculated (5253). The first average coordinates (Xmo, Yno) represent, so to speak, the instantaneous position of the black eye, and the second average coordinates (Xm5, Yn5) represent the center position of the time series change of the black eye.

The second average coordinates (Xm5, Yn5)
Of the first average coordinates (Xmo, Yno) with respect to the horizontal direction (X-axis direction) Xdif = Xmo-Xm5 (polarity is the position change direction of the black eye position, absolute value is the amount of change) is calculated.
(5254). Then, it is determined whether the angular velocity signal Wj detected by the angular velocity detector 21 is positive (right turn) or negative (left turn) (5255), and Xdif ≧ A (the line of sight is on the left from the front of the vehicle). If Wj ≧ 0 (the vehicle is straight ahead or right turn), it is determined that the vehicle is on the left side, and Xdif ≦ −A (the line of sight is on the right side of the front of the vehicle) and Wj ≦ 0 (the vehicle is straight ahead or left turn).
It is determined that the user is looking aside to the right (5256, 525).
7). If neither of these is the case, it is determined that the person is not looking away. If it is determined that the driver is looking aside to the right or looking aside to the left, the data in the register Fsa is checked, and if it is 0 (not determined to be looking aside immediately before), the driver starts looking aside. Write 1 to the register Fsa (5260,
5261) Start the 1 second timer (5262).
When it is judged to be looking aside and the data in the register Fsa is checked, it is 1 (it is already judged to be looking aside and the 1 second timer is started.
Check that the 1-second timer is time-over, and if it is time-over, buzzer 5
Is sounded (5263, 5264). After that, when the person is no longer looking aside, the register Fsa is cleared in step 5258 and the buzzer stops ringing (5259). If the one-second timer does not look aside before the time is over (the buzzer is off), the register Fsa is cleared in step 5258, and the buzzer does not ring. This is to prevent the buzzer from ringing with respect to the movement of the eyes that visually recognizes the side mirror (door mirror) and the indoor mirror (back mirror) for a moment. In addition, side mirror (door mirror)
Or, the buzzer sounds when continuously watching the indoor mirror (back mirror) for 1 second or more.

In some cases, eye detection may fail during the "tracking". At that time, the buzzer 5 is energized as a result of the above-mentioned "drowsy state". Therefore, the buzzer 5 sounds when the eye detection fails during the “tracking” and when the driver to be monitored is in the “dozing state or looking aside”.

In the above-mentioned embodiment, the vehicle traveling direction is detected by the angular velocity detector 21, but it is detected by using the steering angle sensor of the steering wheel instead of or in addition to the angular velocity detector 21. The vehicle heading direction may be detected from the steering angle.

In the above-described embodiment, the darkest point is detected in the small area to detect human eyes, and the point of interest is moved to the darkest point. However, the brightest point or the predetermined brightness is set. May be set to the value of, and if the desired image is subjected to template matching, gradation difference deletion processing, etc., the object to be photographed is not limited to human eyes, and other objects or parts thereof, etc. In the above, it may provide a characteristic dark pattern or light pattern.

In the above embodiment, the coordinates of the points of interest dispersed in advance at a predetermined pitch are written in the memory table, but this writing is omitted and instead, for example, the first points of interest (68,
200) and write the coordinate data to the coordinate register, detect the darkest part of the periphery and write the coordinates to the memory table, and then set the x coordinate data of the coordinate register to 1
The coordinate data may be sequentially incremented and the points of interest may be sequentially specified, such that the coordinates of the second points of interest are incremented by 5 dots. In the above embodiment, the memory table is
Although the coordinate data of the target point is held in the blue memory, a frame memory for allocating one frame binary data table to the RAM 10 or a separate memory and writing the address to the multi-gradation data. The point of interest position information is binary-coded by writing "1" indicating the "point of interest" in the binary data table address corresponding to the coordinates of the point of interest in a one-to-one correspondence with the 12 addresses. -When the data is held in the table and the point of interest is confirmed (searched or read), the address is sequentially updated to 2
The data of the value data table is sequentially read and the read data is read.
The read address when the data is "1" (point of interest) may be saved (extracted as the address of the point of interest).

[0060]

As described above, in the present invention, the driver's inattentiveness is automatically detected by detecting the position change of the eye image on the image photographed by the vehicle driver. When the vehicle turn causes the driver's line of sight to deviate in the turn direction with respect to the longitudinal axis of the vehicle, this does not give a false alarm as a look aside. When the driver's line of sight is different from the traveling direction of the vehicle, an alarm is automatically issued.

[Brief description of drawings]

FIG. 1 is a block diagram showing a configuration of an exemplary embodiment of the present invention.

FIG. 2 is a perspective view showing the external appearance of the arrangement of a CCD camera 3, an illumination lamp 4 and a buzzer 5 shown in FIG.

FIG. 3 is a flowchart showing an outline of a control operation of the microprocessor 6 shown in FIG.

FIG. 4 is a flowchart showing a control operation of “dark spot detection in a small area” (3) shown in FIG.

FIG. 5 is a flowchart showing a control operation of “determination” (4) shown in FIG.

FIG. 6 is a flowchart showing a control operation of “deletion of isolated points” (41) shown in FIG.

[FIG. 7] “Delete other attention points near the attention point” shown in FIG.
It is a flowchart which shows the control operation of (42).

FIG. 8 shows “deletion due to cross gradation difference” (4
It is a flow chart which shows control operation of 3).

9 is a flowchart showing a control operation of "deletion by template matching" (44) shown in FIG.

[FIG. 10] “Delete due to gradation difference in diagonal direction” shown in FIG.
It is a flowchart which shows the control operation of (45).

[FIG. 11] “Integration of neighboring points of interest” shown in FIG. 5 (47)
3 is a flowchart showing the control operation of FIG.

12 is a flowchart showing a control operation of "determination of the number of points of interest" (48) shown in FIG.

13 is a flowchart showing a control operation of "tracking" (5) shown in FIG.

FIG. 14 is a flowchart showing a control operation of “tracking” (5) shown in FIG.

FIG. 15 is a flowchart showing the contents of the “inattentiveness judgment” (525) shown in FIG.

16 is a state diagram showing a state in which 425 points of interest are arranged in one screen distribution of image data to be written in the frame memory 12. FIG.

FIG. 17 is a block diagram showing vertical and horizontal sizes of human eyes.

18 is a block diagram showing the contents of a memory table of RAM 10 shown in FIG.

FIG. 19 is a state diagram showing how two points of interest specify the positions of both eyes of a person.

[Explanation of symbols]

3: CCD camera 4: Illumination lamp 5: Buzzer 6, 8: Microprocessor 7: Address bus & control bus 9: ROM 10: RAM 12: Frame memory 17: A / D converter 18: Infrared lamp controller 19: Buzzer controller SSW : Stop switch

Claims (3)

[Claims] 1. A two-dimensional image pickup means for photographing an on-vehicle driver; an image detection means for detecting an eye image on an image photographed by the two-dimensional image pickup means; a change in the position of the eye image detected by the image detection means. Movement detection means for calculating the traveling direction of the vehicle; inattentive judgment means for detecting a match / mismatch between the change calculated by the movement detection means and the traveling direction of the vehicle detected by the direction detection means; And a warning control means for energizing the warning means in accordance with the detection of the disagreement by the inattentiveness judging means. 2. The inattentive driving alarm according to claim 1, wherein the alarm control means starts clocking in response to the detection of the inconsistency of the inattentiveness determining means and activates the alarming means after a set time when the inconsistency detection continues. apparatus. 3. The movement detecting means calculates a first average position of the eye image position in a relatively short time section and a second average position of a relatively long time section, and a first average position with respect to the second average position. The inattentive driving warning device according to claim 1 or 2, wherein the deviation of the average position is a change in the position of the eye image.