JP2000193420A - Eye position detecting device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To accurately detect the position of an eye by detecting the nose region from an image taken from a side diagonally. SOLUTION: A face region coordinate is calculated from a face image stored in an image memory, the calculated face region coordinate is stored in a face region register while a nose line search region coordinate is calculated from the face region coordinate, which is stored in a nose line search region coordinate register (100). A left side edge and a right side edge in the nose line search region are extracted using a different threshold value with the calculated nose line search region coordinate, and a plurality of regions based on the extracted left-side edge and right-side edge are extracted as a candidate for nose line (102). A nose line is selected based on the vertical length of a nose-line candidate among the extracted nose-line candidates (104), for judging whether a nose line is selected or not (106). If it is judged that a nose line is detected, the center point of the selected nose line is detected (110), a region where an eye is present (eye region) is set based on the center point of nose line (112), and the position of eye is detected from the set eye region (114).

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an eye position detecting device, and more particularly to an eye position detecting device capable of stably detecting an eye position without being affected by the angle and direction of a nose. .

[0002]

2. Description of the Related Art Japanese Patent Laying-Open No. 61-77703 discloses a device for recognizing a position of an eye of a vehicle driver. In this device, in order to clearly outline the nose, the driver's face is photographed obliquely from the lateral direction, and a nose search area for searching the nose is fixed to a certain area on the photographed driver's face image, and the nose search is performed. The position of the nose tip is detected by pattern matching of the nose within the region. Then, the nose muscle is detected from the detected position of the nose tip, the inclination of the detected nose muscle is calculated, and the center position of the pupil is calculated from the position of the nose tip and the inclination of the nose muscle.

A vehicle occupant riding position recognition device described in Japanese Patent Application Laid-Open No. 63-311105 detects the front and rear positions and postures of a seat using a driver's seat state detecting means, and the detected position is detected. The position of the driver's face is estimated from the seat position and attitude, and a predetermined area for searching for a nose is set.

[0004] In the above-mentioned conventional apparatus for recognizing the position of the eyes of a vehicle driver, the driver's face is photographed at a considerably shallow angle with respect to the driver's face, that is, an angle close to the side. Therefore, the position of the nose also changes back and forth depending on the position before and after the driver sits. For this reason, when the position of the nose search area is fixed, the nose does not always exist in the nose search area, and it may be difficult to detect the position of the nose tip by pattern matching. In addition, the shape of the nose greatly differs depending on individual differences, and the shape of the nose greatly changes depending on the direction of the face, that is, the angle at which the image is taken. For this reason, in the case of an infrared image photographed obliquely from the side, the left side of the nose may be dark and the edge may be unclear, and the unclear edge may affect the shape of the detected nose. In pattern matching, it is not possible to respond to changes in the shape of the nose, and in order to respond to changes in the shape of the nose, it is necessary to prepare various types of nose patterns according to the shape of the changing nose. The method is not practical.

[0005] Further, the above-mentioned device for recognizing the occupant's boarding position has a problem that the cost is high because a seat state detecting means is required. In addition, the method of sitting on the seat and the posture with respect to the backrest differ depending on the individual difference of the driver, so that when estimating the position of the driver's face from the position of the seat and the posture, high accuracy cannot be obtained. There's a problem.

SUMMARY OF THE INVENTION The present invention has been made to solve the above-mentioned problems, and it is possible to detect the position of the eye by detecting the region where the nose is present even in an infrared image or the like taken obliquely from the side. It is an object to provide a position detecting device.

[0007]

SUMMARY OF THE INVENTION In order to achieve the above object, the present invention provides a method for determining the position of an eye based on an image of a face taken diagonally from the front or an image of a face received light diagonally from the front. An eye position detecting apparatus for detecting, comprising: a face area extracting unit that extracts a face area from an input image; and a left edge and a right edge using different thresholds from the extracted face area. A nose ridge candidate extracting means for extracting a region based on the extracted left and right edges as a nose ridge candidate, a nose ridge extracting means for extracting a nose ridge from the extracted nose ridge candidates, and a position of the extracted nose ridge. An eye area setting unit configured to set an eye existence area based on the eye area detection unit that detects an eye position from the set eye existence area.

In the present invention, the position of the eyes is detected based on an image of a face photographed diagonally from the front or an image of a face photographed diagonally from the front. When shooting a face diagonally from the front, the face may receive light from any direction,
When capturing a face that receives light obliquely from the front, the image may be captured from any direction such as directly in front of the face or obliquely forward. As a result, the photographing is performed such that the left edge and the right edge of the nose line are different. The angle at which the face is photographed obliquely from the front and the angle of the light are 0 ° in front of the face and 90 ° just beside the face.
In the case of °, 20 ° to 70 ° is appropriate. The face area extracting means extracts a face area from the input image. This face area can be extracted by converting the input face image into a binary image using an appropriate luminance threshold, and using the converted binary image.

The nose line candidate extracting means extracts a left edge and a right edge from each of the extracted face regions using different threshold values, and uses the region based on the extracted left edge and right edge as a nose line candidate. Extract.

In this case, it is effective to set a predetermined area as a nose ridge search area in the extracted face area and extract nose ridge candidates. This predetermined area can be set according to the direction in which the face is photographed or the direction in which the face receives light. When the position of the eyes is detected based on the image of the face that has been received, the area on the left half of the face area can be set as the nose muscle search area. In addition, a nose ridge candidate can be detected by detecting a pair of a left edge and a right edge, and extracting a region where the edge pair exists as a nose ridge candidate.

In the present invention, the threshold values for extracting the left edge and the right edge are made different from each other because the face is photographed obliquely from the front or the face photographed obliquely from the front. This is because the left and right edge intensities of the nose muscles are different. When the face is photographed diagonally right front toward the face, or when the face is photographed obliquely right front facing the face, the threshold value for the left edge is smaller than the threshold value for the right edge. .

The nose ridge extracting means extracts the most probable nose ridge from the extracted nose ridge candidates using, for example, a condition relating to the vertical length of the candidate region. If the condition is not satisfied, the nose line cannot be detected. When extracting using the condition regarding the vertical length of the candidate area, a candidate longer than an appropriate threshold is selected, and the longest one of the selected candidates is extracted as a nose line.

The eye area setting means sets an eye existence area based on the position of the extracted nose muscle. The existence area of the eye is
It can be set by assuming the extracted nose muscle as a line segment and estimating the region where the eye exists from the middle point of the extracted nose muscle.

The eye position detecting means detects the position of the eye from the set eye existence region.

In the present invention, when the nose muscle cannot be detected, the position of the eye may be detected in the entire face area.

[0016]

Embodiments of the present invention will be described below with reference to the drawings.

In this embodiment, as shown in FIG. 1, a camera 10 for photographing a driver's face from a diagonally right front toward the driver, a computer for image processing, and a binary image extracted by being connected to the computer. It is configured by a display device 32 configured by a CRT for displaying a structured image and the like.

The computer for image processing is a camera 10
Analog-to-digital (A / D) converter 12 for digitally converting an image signal input from the CPU, a CPU 14, and a ROM 1 storing a program for an eye position detection processing routine.
6, RAM 18, image memory 2 for storing image data
0, an average value register 22 for storing an average luminance value, a histogram memory 24 for storing histograms representing various frequency distributions described below, a face area coordinate register 26 for storing the coordinates of the face area, and coordinates of the nose muscle search area. Nasal muscle search area coordinate register 28 to be stored, output port 30 connected to display device 32, image memory 34 to store the difference between the average brightness values, the sum of the average brightness values and 0, and a control bus connecting these And a bus 36 such as a data bus.

Next, an eye position detection processing routine according to this embodiment will be described with reference to FIG.

The image signal input from the camera 10 is A
The image data is converted into digital image data by the / D converter 12 and stored in the image memory 20.

In step 100, the image memory 20
Of the diagonal of the rectangular face area from the face image stored in
Face area coordinates (x₁ , Y₁ ), (X
_Two , Y _Two ) Is calculated, and the calculated face area coordinates are stored in the face area register.
And a nose line search area based on the face area coordinates.
Area coordinates (x₁ , Y₁ ), (X₀ , Y_Two ) To calculate the nose
It is stored in the search area coordinate register 28. Where x₁ <X
₀ <X_Two , Y₁ <Y_Two , X₀ = (X₁ + X_Two ) / 2
You.

In step 102, the left edge of the nose ridge search area is set using different thresholds in the area indicated by the calculated nose ridge search area coordinates on the face image stored in the image memory 20. A right edge is extracted, and a plurality of regions based on the extracted left edge and right edge are extracted as candidates for a nose line. In step 104, a nose ridge is selected from the extracted nose ridge candidates based on the vertical length of the nose ridge candidate, and it is determined whether or not the nose ridge is selected in step 106, that is, whether or not a nose ridge is detected. .

If it is determined at step 106 that the nose muscle has not been detected, then at step 108
0, a process of detecting the position of the eye in the face area on the face image stored in the image memory 20 detected.
On the other hand, if it is determined in step 106 that the nose muscle has been detected, the midpoint of the nose muscle selected in step 110 is detected, and in step 112 the eye existence region (eye region) is determined based on the nose muscle midpoint. Then, the eye position is detected from the eye region set in step 114.

FIG. 3 shows the details of step 100 in FIG. 2. In step 120, the luminance value of each pixel of the face image (gray image) is compared with a predetermined appropriate threshold value, thereby obtaining the face value. The image is converted into a binary image with labels 1 and 0. In the next step 122, FIG.
As shown in the figure, the label of the binary image is represented by the X axis on the XY plane
And the histogram projected on the Y axis is determined, and step 12
4 is stored in the histogram memory 2
4 is stored. This histogram is obtained by using a value obtained by integrating labels of pixels located on a straight line passing through a certain X coordinate and parallel to the Y axis as a frequency. Note that the same applies to the Y-axis, which is obtained by calculating a value obtained by integrating labels of pixels located on a straight line that passes through a certain Y coordinate and is parallel to the X-axis.

In step 126, as shown in FIG.
The peak value of the frequency of the histogram on the X axis is calculated.
The peak value of the frequency is similarly calculated for the Y axis. In step 128, as shown in FIG. 4, in the process of moving left and right from the coordinates of the peak on the X axis while comparing the threshold with the frequency from the coordinates of the peak, the frequency is calculated from the coordinates crossing the threshold. Large X coordinates x ₁ and x ₂ are extracted as face area coordinates. For the Y axis, face area coordinates y ₁ and y ₂ are obtained in the same manner.
Is extracted. Thereby, the face area coordinates (x ₁ , y
₁ ) and (x ₂ , y ₂ ) are calculated. The calculated face area coordinates are stored in the face area coordinate register 2 in step 130.
6 is stored.

In step 132, the coordinates belonging to the left half area of the face from the face area coordinates are selected as the nose ridge search area coordinates (x ₁ , y ₁ ), (x ₀ , y ₂ ). The obtained nose ridge search area coordinates are stored in the nose ridge search area coordinate register 28.

Next, details of the nose ridge candidate extraction process in step 102 of FIG. 2 will be described. The nose ridge candidate extraction process
The processing is divided into three blocks: a left edge extraction processing routine shown in FIG. 5, a right edge extraction processing routine shown in FIG. 6, and a left and right edge pair extraction processing routine shown in FIG.

In the left edge extraction processing routine of FIG.
As shown in FIG. 8A, the nose ridge search area coordinate register 2
8 is set from the left side to the right side so as to be continuous adjacent to the face in the lateral direction on the nose line search area on the face image stored in the image memory 20 based on the nose line search area coordinates stored in the image memory 20. 1 pixel of rectangular area 1 and 2 of rectangular area 2
A rectangular window 40 having two sets of rectangular areas is used. The window 40 includes a rectangular area 1 and a rectangular area 2
Is shifted leftward by one pixel in the horizontal direction of the face, and scanning is performed by shifting one pixel to the right on the nose line search area by one pixel from top to bottom. The left edge is extracted by scanning while shifting each other.

In step 140 of FIG. 5, the luminance integrated values are calculated by integrating the luminance values of all the pixels in each of the rectangular areas 1 and 2. In step 142, the rectangular area 1
Is divided by the total number of pixels in the rectangular area 1, and the sum of the luminance integrated values in the rectangular area 2 is divided by the rectangular area 2.
The luminance average values P1 and P2 in the rectangular areas 1 and 2 are calculated by dividing by the total number of pixels in the pixel area and stored in the luminance average value register 22 according to the position of the window 40.

In the next step 144, each of the differences (P1-P2) between the average brightness values in the adjacent rectangular areas 1 and 2 is calculated.
It is compared with the threshold value K1 to determine whether or not the condition of P1-P2> K1 is satisfied.

If it is determined in step 144 that the above condition is satisfied, that is, if it is determined that the left edge is a candidate for a nose line because of a large difference between the average brightness values in adjacent rectangular areas, step 146 is performed. , An image memory 34 different from the image memory in which the face image is stored.
Then, the value of the difference (P1−P2) between the average brightness values is written in. The coordinates to be written are the coordinates of the uppermost pixel in the rectangular area 1 in FIG. At the same time, in step 148, the histogram difference (P1-P2) representing the frequency distribution of the difference between the average brightness values stored in the histogram memory 24.
Is increased by one.

If it is determined in step 144 that the above condition is not satisfied, that is, if the difference between the average luminance values of the adjacent rectangular areas 1 and 2 is small, it is estimated that the left edge does not exist and step 150 is performed. , An image memory 3 different from the image memory in which the face image is stored.
Write 0 to 4. The coordinates to be written are the coordinates of the uppermost pixel in the rectangular area 1 in FIG.
At the same time, in step 152, the histogram memory 2
A difference of 4 increases the frequency of 0 by 1.

The value of the difference between the average luminance values (P1−P2) and 0 should be written in the center coordinates of the window in FIG. 8A, but in this example, it is the upper left pixel for convenience.

In the next step 154, it is determined whether or not the processing of steps 140 to 152 has been completed for the entire image of the nose ridge search area. If the processing has not been completed, the window is shifted by one pixel and step 140 is performed. And the above processing is repeated.

The above processing is performed by setting the rectangular window 40 to 1
By performing the process on the entire image of the nose ridge search area, which is the input image, while shifting each pixel, an image in which the difference in the average luminance value is written and a histogram are obtained.

If it is determined in step 154 that the processing has been completed for the entire image of the nose ridge search area, in step 156 the difference between the average brightness values (P1-P) is determined based on the histogram stored in the histogram memory.
Calculate the threshold value K2 for 2). That is, the frequencies are integrated from the higher frequencies in the histogram, and the difference (P1-
The value of P2) is set as the threshold value K2. The predetermined ratio to the total frequency is set in advance by simulation or the like.

In step 158, the difference (P1-P2) between the average luminance values of the pixels is compared with the threshold value K2 (P1
-P2)> It is determined whether or not the condition of K2 is satisfied.

Labeling is performed so that the value output at step 160 is 1 for pixels satisfying the above conditions, and the value output at step 162 is 0 for pixels not satisfying the above conditions. To perform binarization. The above-described binarization processing is continued until it is determined in step 164 that the processing has been completed for the entire image of the nose ridge search area. Then, by performing processing on the entire image of the nose ridge search region, a white / black edge corresponding to the left edge of the nose ridge is extracted. FIG. 10A shows an example of a binary image obtained by the left edge extraction processing.

Next, the right edge extraction processing routine shown in FIG. 6 will be described.

In the right edge extraction processing routine shown in FIG.
As shown in FIG. 8B, the nose ridge search area coordinate register 2
8 is set from the left side to the right side so as to be continuous adjacent to the face in the lateral direction on the nose line search area on the face image stored in the image memory 20 based on the nose line search area coordinates stored in the image memory 20. A rectangular window 50 provided with four sets of two rectangular regions including a rectangular region 3 of one pixel and a rectangular region 4 of four pixels is used. The window 50 is configured by shifting a set of rectangular areas including a rectangular area 3 and a rectangular area 4 one pixel at a time in the horizontal direction of the face to the left.
The right edge is extracted by scanning the nose muscle search area by shifting it one pixel to the right in the right direction and by scanning it one pixel down in the downward direction.

The difference between the rectangular window 50 used for the right edge extraction processing and the rectangular window 40 used for the left edge extraction processing is that the driver's face is photographed obliquely to the front of the driver toward the driver. Because the left edge is weaker than the right edge, it is easy to extract the edge of the left edge by calculating the difference in the average brightness with a small area.

The right edge extraction processing shown in FIG. 6 is the same as the left edge extraction processing described with reference to FIG. 5, and the corresponding parts will be denoted by the same reference numerals. In the right edge extraction processing shown in FIG.
Is divided by the total number of pixels in the rectangular area 3 and the sum of the integrated luminance values in the rectangular area 4 is
, The average luminance values P3 and P4 in the rectangular areas 3 and 4 are calculated, and the difference between the average luminance values (P4−P4) is calculated based on the histogram stored in the histogram memory in step 156. A threshold value K3 for P3) is calculated. That is, the frequencies are integrated from the higher frequencies in the histogram, and the value of the difference (P4-P3) when the integrated value reaches a predetermined ratio with respect to all the frequencies is set as the threshold value K3. The predetermined ratio to the total frequency is set in advance by simulation or the like.

Since the right and left edges of the nose are different in intensity because the face is photographed obliquely from the right, the threshold value K3
Is set to a size different from the threshold value K2 of the left edge. That is, since the face is photographed obliquely from the right direction, the threshold value is determined as (the threshold value K2 for the left edge) <(the threshold value K3 for the right edge).

Then, by performing the same processing as in the case of the left edge on the entire image of the nose line region, a black / white edge corresponding to the right edge of the nose line is extracted. FIG. 10B shows an example of a binary image obtained by the right edge extraction processing.

Next, a left edge extraction result and a left / right edge pair extraction processing routine performed on the left edge extraction result will be described with reference to FIG. In this pair extraction process of the left and right edges, a point where the left and right edges are close to each other is extracted.

In step 170, the right edge point is selected, and in step 172, as shown in FIG. 9, starting from the right edge point, a horizontal search is performed to the left by an appropriate distance, and in step 174, the left edge point is searched. It is determined whether or not an edge point exists. If there is a left edge point, the right edge point and the left edge point corresponding to the right edge point, which are the starting points in step 176, are adopted as an edge pair.
When adopted as an edge pair, the coordinates of the right edge are stored in a memory such as a RAM.

On the other hand, if there is no left edge point, the right edge point selected in step 178 is deleted. Then, in step 180, it is determined whether or not the processing of steps 170 to 178 has been performed for all of the right edge points.
Returning to step 70, the above processing is repeated.

As a result, when the right and left edge pairs exist, the coordinates of the right edge are stored, and the area represented by the coordinates of the right edge is determined as a nose line candidate area in FIG.
It is extracted as shown in the example of (C). Usually, a plurality of nose ridge candidate regions are extracted. FIG. 13 shows the relationship between the face region and the extracted nose ridge candidate regions. In FIG. 13, two nose ridge candidate regions 1 and 2 are extracted.

Next, referring to FIG. 11, a process of detecting a nose ridge from a plurality of nose ridge candidate regions extracted is performed.
04 will be described in detail. In step 200, for each of the plurality of candidate nose ridges extracted in step 102, as shown in FIG.
Find the lower left position coordinates. The details of calculating the position coordinates of the nose ridge candidate region will be described later.

In step 202, the extracted nose ridge line candidate area is assumed to be a line segment, and the vertical length of the nose ridge line candidate area is calculated from the upper right and lower left position coordinates of the nose line candidate area.

In step 204, the length of the candidate nose line region in the vertical direction is compared with a predetermined threshold L. If the length in the vertical direction exceeds the threshold L, step 20 is executed.
In step 6, it is selected as a nose ridge candidate. On the other hand, when the length in the vertical direction of the nose ridge candidate region is equal to or smaller than the threshold value L, the process proceeds to the next step without selecting the nose ridge candidate.

In step 208, it is determined whether or not the processing in steps 200 to 206 has been completed for all the candidate nose ridges.
Returning to 0, the above processing is repeated, and when the processing is completed, the nose ridge candidate having the longest length in the vertical direction among the nose ridge candidates selected in step 210 is selected as the nose ridge. FIG.
In the example of No. 3, the nose ridge candidate region 1 is selected. When no nose ridge candidate is selected, the nose ridge cannot be detected, and FIG.
Is negative in step 106.

In step 110 of FIG. 2, the midpoint coordinates (FIG. 14) of a straight line connecting the upper right and lower left position coordinates of the region indicating the nose muscle selected as described above are calculated.

In step 112 of FIG.
Assuming that the midpoint coordinates calculated at 0 are the center points of the nose muscles, an eye area where an eye is expected to be present is set based on the midpoint coordinates. The eye region is set based on the positional relationship between the nose and the eye determined based on the statistical information of the face.
As shown in FIG. 15, the positional relationship between the nose and the eye is represented by an L1 × L3 rectangle whose center coordinate (the midpoint of the nose muscle) is located at the lower right corner and an L1 × L3 rectangle separated by a distance L2 in the right direction from this rectangle.
It is assumed that an eye exists in each rectangle. Then, the size of the face region is calculated, and the eye region is set by changing the lengths of L1, L2, and L3 according to the size of the face region. For example, the length of L1 is changed in proportion to the vertical length of the face area. Further, the lengths of L2 and L3 are changed in proportion to the horizontal length of the face area. Note that L1, L2, and L1 are proportional to the vertical and horizontal lengths of the face area, respectively.
The length of each of the three may be changed.

Details of step 114 in FIG. 2 will be described with reference to FIG. In step 220 of FIG. 16, the image data corresponding to the set eye region is read from the image memory 20, and a histogram representing the frequency distribution of the luminance values of the eye region set in step 222 is calculated. Step 2
At 24, a frequency peak is detected on the calculated histogram. In step 226, as shown in FIG.
The histogram proceeds to the left from the frequency peak on the histogram, detects a valley of the frequency that appears first, and sets the luminance value of the valley as a threshold value. In step 228, the luminance value of each pixel in the eye region is compared with a threshold value, and the value output at step 230 is set to 1 for a pixel whose luminance exceeds the threshold value. For the following pixels, the value output at step 232 is set to 0, and binarization is performed. The above binarization processing is performed in step 234.
Is continued until it is determined that the processing has been completed for the entire eye region. Then, the image of the eye region is converted into a binary image by performing processing on the entire surface of the eye region. FIG. 18 shows an example of the original image of the eye region and the binary image of the eye region at this time.
(A) and (B) show.

Next, at step 236, the 2
When the minimum value filter processing is performed on the value image, FIG.
As shown in (C), a black area is detected, and among the detected black areas, a black area located substantially at the center of the eye area is selected as an eye area.

Next, the details of the routine for calculating the position coordinates of the nose ridge candidate region in step 200 will be described with reference to FIG. In the position coordinate calculation process of the nose ridge candidate region,
Binary image illustrated in FIG. 10C (nose ridge line candidate region image)
Is scanned from right to left in order from top to bottom, while searching for the coordinates with the label “1”, that is, the coordinates with the luminance value 1 of the binary image. The reason why the binary image is scanned from right to left is that the label is “1” and the top right pixel is found first.

After finding the top-right point whose label is "1", at step 240, the pixel at that coordinate is set as the target pixel and the label of the pixel at the coordinates around the pixel is evaluated.

First, as shown in FIG. 20, the label of the D coordinate located at the rightmost position on the scan line following the scan line including the target pixel (i, j) is evaluated (step 242). If the label of the D coordinate is "1", it moves to the D coordinate and sets the label of the D coordinate to "0" (steps 242, 244).

As a second step, the label of the C coordinate located on the left side of the target pixel as shown in FIG. 20 is evaluated (step 246). If the label of the C coordinate is "1", it moves to the C coordinate and sets the label of the C coordinate to "0" (steps 246, 248).

Third, the label of the B coordinate located below the C coordinate as shown in FIG. 20 is evaluated (step 250). If the label of the B coordinate is "1", it moves to the B coordinate and sets the label of the B coordinate to "0" (step 2).
50, 252).

Fourth, the label of the A coordinate located below the target pixel as shown in FIG. 20 is evaluated (step 254). If the label of the A coordinate is "1", the coordinate is moved to the A coordinate and the label of the A coordinate is set to "0" (steps 254, 256).

Fifth, the label of the E coordinate located on the left of the C coordinate as shown in FIG. 20 is evaluated (step 258). If the label of the E coordinate is "1", the label of the E coordinate is set to "0" (steps 258, 260).

Sixth, the label of the F coordinate located on the left of the E coordinate as shown in FIG. 20 is evaluated (step 262). If the label of the F coordinate is "1", the label of the F coordinate is set to "0" (steps 262 and 264).

At step 266, it is determined whether or not the coordinates have been moved. If the coordinates have been moved, the process returns to step 242 and the above-mentioned steps 242 to 26 are performed using the moved coordinates.
Step 4 is repeated. On the other hand, if it is determined in step 266 that there is no movement, since the labels of all the coordinates of A, B, C, and D are "0", the search for the label ends at the coordinate point, and the coordinates of the upper right and lower left are set to the coordinates. Write to register.

By starting from the target pixel located at the upper right to the lower left with respect to the entire nose line candidate area image,
The position coordinates of the nose ridge candidate region are calculated.

In the above embodiment, when the face area is detected, the process proceeds from the frequency peak to the left and right in the process of FIG.
As shown in (1), when a minimum point exists, there is a possibility that a face area is detected at the boundary of the minimum point. Therefore, hereinafter, a face area detection process capable of accurately detecting a face area even when a minimum point exists will be described with reference to FIG. In the face area detection processing, the calculation of the face area coordinates will be described by assigning the same reference numerals to the same parts as in FIG.

In step 128, as shown in FIG. 21, in the process of moving right and left from the coordinate of the frequency peak on the X-axis, the threshold was first crossed while comparing the frequency with the threshold. The area surrounded by the coordinates x _a and x _b is stored in the face area coordinate register 26 as a temporary face area.

[0069] In the next step 270, compares sequentially the frequency and thresholds for each coordinate within a range of left and right, starting from the minimum of the coordinate x _a and maximum coordinates x _b of the face area coordinates, teeth It is determined whether or not there is a coordinate exceeding the threshold. If all of the frequencies of the respective coordinates within the certain range do not exceed the threshold value, there is no minimum point in the face area as shown in FIG.
Is determined as face area coordinates. If the frequency within a certain range exceeds the threshold value even at one point, the stored face area coordinates are regarded as the face area extracted at the minimum point, and in step 274, the face area in the face area coordinate register is determined. Then, the process returns to step 128 and returns to the left and right again from the coordinates exceeding the threshold value, compares the threshold value with the frequency, and sets the coordinates crossing the threshold value as a provisional face area. Stored in the area coordinate register 26,
Steps 270 and 274 are repeated to search for face area coordinates.

In the example of FIG. 21, each seat within a certain range on the left is
The face on the left side because all the frequencies of the target do not exceed the threshold
Coordinate x as area coordinates_a Is determined, but the right face area
Coordinates x_c Exceeds the threshold value, so the face area
Coordinate x of the face area of the target register _b And delete the coordinate x_c From
Go to the right again and compare the threshold with the frequency
Coordinate x across_d Is extracted. Coordinate x_d One on the right
Not all frequencies of each coordinate within the fixed range exceed the threshold.
Therefore, the coordinate x_d Is determined.

The face area coordinates finally determined are stored in the face area coordinate register. For the Y axis, the face area coordinates are calculated in the same manner and stored in the face area coordinate register.

Next, another example of detecting a nose ridge from a nose ridge candidate region will be described.

In the face area detected in advance, as shown in FIG. 23, the horizontal axis (X axis) is on the left side of the center, and the vertical axis (Y
An area below the center on the (axis) is set as a nose ridge candidate selection area.

In the case shown in FIG. 23, the nose ridge candidate region 2 is a candidate region generated by the bangs. By limiting the nose ridge candidate selection area to the area below the center on the vertical axis,
Candidates with such a possibility of erroneous recognition can be excluded.

As another example of detecting a nose line from a nose line candidate region, an example using the aspect ratio of the nose line candidate region will be described. First, from the coordinates of the upper right and lower left of the nose line candidate area,
The aspect ratio of the nose ridge candidate region is determined. This aspect ratio is calculated using a rectangle having a diagonal line segment connecting the upper right and lower left coordinates. Then, an appropriate threshold value is set for the aspect ratio, and only vertically long candidates are selected.

Next, another example of the eye position detecting process will be described with reference to FIG. In this example, as shown in FIG. 24A, a rectangular area 1, a rectangular area 2, and a rectangular area set from the upper side to the lower side so as to be adjacent and continuous in the vertical direction of the face on the eye area. A window 60 composed of three rectangular areas in the area 3 is used. In this window 60,
The rectangular area 1 and the rectangular area 3 have the same shape and the same area, and the width of the rectangular area 2 is the same as that of the rectangular area 1 and is larger than the area of the rectangular area 1, for example, approximately twice as large.

In step 300 of FIG. 25, the integrated luminance values are calculated by integrating the luminance values of all the pixels in each rectangular area. In step 302, the luminance integrated value in each rectangular area is divided by the total number of pixels in each rectangular area to obtain the average luminance values P1, P2, and P3 in each rectangular area (FIG. 2).
4 (B)) and stores the calculated value in the average brightness value register 22 according to the position of the window.

In the next step 304, the differences (P1-P2) and (P3-P2) between the average luminance values in adjacent rectangular areas are determined.
Is compared with a threshold value K1 to determine whether or not the following condition is satisfied.

(P1-P2> K1) and (P3-P2> K1) When it is determined in step 304 that the above condition is satisfied, that is, since the difference between the average luminance values in adjacent rectangular areas is large, If it is determined that the area is a candidate area, in step 306, the sum of the differences between the average luminance values (P1-P2) + (P3-P2) is stored in an image memory 34 different from the image memory in which the face image is stored. Write the value of).
The coordinates to be written are written to the coordinates of the center point of the rectangular area 2. At the same time, in step 308, the total sum of histograms (P1-P2) + (P3-
The frequency for P2) is increased by one.

If it is determined in step 304 that the above condition is not satisfied, that is, if the difference between the average luminance values of adjacent rectangular areas is small, it is estimated that the area is a face skin area, and in step 310, 0 is written in an image memory 34 different from the image memory in which the face image is stored. The coordinates to be written are written to the coordinates of the center point of the rectangular area 2. At the same time, in step 312, the frequency at which the sum of the histogram memory 24 is 0 is increased by 1.

In the next step 314, it is determined whether or not the processing of steps 300 to 312 has been completed for the entire eye area. If the processing has not been completed, the window is shifted by one pixel and the process returns to step 300. The above process is repeated.

The above processing is performed on the entire eye area, which is the input image, while shifting the rectangular window by one pixel, thereby obtaining an image in which the difference value of the average luminance value is written and a histogram.

If it is determined in step 314 that the processing has been completed for the entire eye region, in step 316, the sum of the differences between the average luminance values (P1-P2) +
A threshold value K2 for (P3-P2) is calculated. That is, the frequencies are integrated from the higher frequencies in the histogram, and the value of the sum (P1-P2) + (P3-P2) when the integrated value reaches a predetermined ratio with respect to all the frequencies is determined by the threshold K
Set as 2. The predetermined ratio to the total frequency is set in advance by simulation or the like.

At step 318, it is determined whether or not the following condition is satisfied by comparing the sum (P1-P2) + (P3-P2) of the differences between the average brightness values of the pixels with the threshold value K2.

(P1-P2) + (P3-P2)> K2 For a pixel satisfying the above condition, the value at the time of output in step 320 is set to 1, and for a pixel not satisfying the above condition, step 322 is performed. The value at the time of output is set to 0 and binarization is performed. The above-described binarization processing is continued until it is determined in step 324 that the processing has been completed for the entire eye region. Then, a binary image is obtained by performing processing on the entire eye region. The area of label 1 in the obtained binary image is the position of the eye.

Next, a description will be given of the eye position detection processing in the face area when the nose line detection in step 108 cannot be performed. The eye candidate area is extracted using the processing routine of FIG. 25 described above. However, in this case, candidate extraction is performed not only for the eye region but also for the entire face region. Since a result as shown in FIG. 26 is obtained as a candidate extraction result, an eye region is finally selected from a plurality of candidates as follows.

That is, in the face area, selection is made under the following conditions. First, a horizontally long eye candidate area is selected.
For the eye candidate area having the horizontally long shape, the aspect ratio described above is obtained for each eye candidate area, and a predetermined threshold value is set using the aspect ratio. Are selected by selecting the horizontally long region as the eye candidate region. When a plurality of horizontally long eye candidate regions are selected, an eye region is selected by the following method.

A pair of two eye candidate areas located at the top of the face area and arranged substantially parallel to each other vertically is selected. Since the upper side of the two eye candidate areas is considered to be the eyebrow area, the lower eye candidate area is selected as the eye area. On the other hand, if no pair of eye candidate regions can be found, the eye candidate region located at the top of the face regions is selected as the eye region.

In the above embodiment, a case has been described in which the camera is used to photograph the driver's face from the front diagonally right toward the driver. Often, the driver's face receiving light from the diagonally right front or diagonally left front toward the driver may be photographed from the front, diagonally left front, or diagonally right front. When photographing the face of the driver receiving light from the diagonally right front toward the driver, similarly to the above, (threshold K2 for the left edge) <(threshold K for the right edge)
3) In the case of photographing the driver from the diagonally left front toward the driver and photographing the driver's face receiving light from the diagonally left front toward the driver, (the threshold value K2 for the left edge)> (The threshold value K3 for the right edge).

[0090]

As described above, according to the present invention,
Since the left and right edges of the nose are extracted independently, and then the pair of left and right edges is detected, the position of the nose can be detected stably without being affected by the angle, direction, etc. of the nose. By detecting the region in advance, even when glasses are present, it is possible to avoid erroneously recognizing a frame of the glasses as a nose line, and even in a complicated case where glasses are present, By detecting the position, the effect that the region where the eye exists can be specified stably can be obtained.

[Brief description of the drawings]

FIG. 1 is a block diagram of an embodiment.

FIG. 2 is a flowchart showing an eye position detection processing routine according to the embodiment.

FIG. 3 is a flowchart showing details of step 100 in FIG. 2;

FIG. 4 is a diagram showing a histogram obtained by projecting a binary image on the X axis.

FIG. 5 is a flowchart showing a left edge extraction processing routine;

FIG. 6 is a flowchart showing a right edge extraction processing routine;

FIG. 7 is a flowchart showing a left / right edge pair extraction processing routine;

8A is a plan view of a window for extracting a left edge, and FIG. 8B is a plan view of a window for extracting a right edge.

FIG. 9 is a flowchart illustrating a pair extraction process of right and left edges.

10A is a plan view of a binary image obtained by a left edge extraction process, FIG. 10B is a plan view of a binary image obtained by a right edge extraction process, and FIG. FIG.

FIG. 11 is a flowchart showing details of step 104 in FIG. 2;

FIG. 12 is a diagram showing upper right and lower left position coordinates of each nose ridge candidate region;

FIG. 13 is a diagram showing a positional relationship between a nose ridge candidate region and a face region.

FIG. 14 is a diagram showing a positional relationship between a nose ridge region and a midpoint.

FIG. 15 is a diagram showing a relationship between a nose muscle midpoint and an eye area.

FIG. 16 is a flowchart showing details of step 114 in FIG. 2;

FIG. 17 is a diagram showing an eye area histogram.

18A is a diagram illustrating an eye region original image, FIG. 18B is a diagram illustrating an eye region binary image, and FIG. 18C is a diagram illustrating an image after execution of a minimum value filter.

FIG. 19 is a flowchart showing details of step 200 in FIG. 11;

FIG. 20 is a diagram illustrating a relationship between a target pixel and peripheral pixels.

FIG. 21 is a diagram showing a histogram when a minimum point exists.

FIG. 22 is a flowchart showing a processing routine for detecting a face area when a minimum point exists.

FIG. 23 is a diagram illustrating a relationship between a nose ridge candidate selection area and a face area.

FIG. 24A is a diagram showing a window configuration for extracting eyes, and FIG. 24B is a diagram showing an average luminance value.

FIG. 25 is a diagram showing a routine for converting an eye region into a binary image.

FIG. 26 is a diagram showing an eye region extraction result.

[Explanation of symbols]

 Reference Signs List 10 camera 12 analog-digital converter 20 image memory 22 average value register 24 histogram memory

Claims (1)

[Claims] An eye position detecting device for detecting an eye position based on an image of a face photographed diagonally from the front or an image of a face photographed diagonally from the front. A face region extracting means for extracting a face region, and extracting a left edge and a right edge from the extracted face region using different threshold values, respectively, and extracting a region based on the extracted left edge and right edge into a nose ridge. A nose ridge candidate extracting means for extracting a nose ridge from the extracted nose ridge candidates; an eye area setting means for setting an eye existence area based on the position of the extracted nose ridge; An eye position detecting device, comprising: an eye position detecting unit configured to detect an eye position from the detected eye existence region.