JP2018101212A - On-vehicle device and method for calculating degree of face directed to front side - Google Patents

Abstract

PROBLEM TO BE SOLVED: To accurately detect whether the face is directed to the front side in a photographed face image.SOLUTION: An on-vehicle device calculates, as the degree of the face directed to the front side D, similarity between first image data Fsmall obtained by reducing the resolution of a face image F and second image data FLsmall obtained by reducing the resolution of an image FL obtained by horizontally flipping the face image, when the face is directed to the front side and the line of sight is directed to the front side, automatically execute calibration of the position of the eyeball center in a three-dimensional eyeball model, processes the image with a reduced resolution to reduce the amount of calculation processing, reduces an influence of physical features, such as moles that are irrelevant to the direction of the face, and thereby can accurately calculate the degree of the face directed to the front side D.SELECTED DRAWING: Figure 5

Description

  The present invention relates to an in-vehicle device and a face front degree calculation method using a face image photographed so as to include a face, and more particularly to a technique that can be used for calibration.

  In a vehicle, it is desired to detect the direction of the driver's line of sight and to use the information of the line of sight for safe driving. As representative techniques for detecting the line of sight, a “corneal reflection method”, a “pupil-corneal reflection method” (for example, Patent Document 1), and a “model-based method” (for example, Patent Document 2) are known. .

  In the case of the “corneal reflection method” and the “pupil-corneal reflection method”, a line of sight is detected based on the positional relationship between the pupils using corneal reflection light (Purkinje image) generated by applying infrared rays to the eyes as a reference point. In the case of the “model-based method”, the line of sight is calculated using a facial feature point group, a three-dimensional face model, and a three-dimensional eyeball model.

  The “corneal reflection method” and the “pupil-corneal reflection method” cannot perform line-of-sight measurement without calibration. On the other hand, the “model-based method” can be recognized without calibration. Moreover, since individual differences can be eliminated by calibration, the detection accuracy of the line of sight can be improved.

  For example, in the technique of Patent Document 3, a target that can be visually recognized by the driver is displayed as a virtual image using a head-up display at the time of calibration, and the line-of-sight direction of the driver who is gazing at the position of the target is detected. Calibration is performed.

  For example, Patent Document 4 shows a calibration technique that does not require any special operation by the driver. For example, when it is considered that a specific state is detected based on the distance between the host vehicle and the other vehicle, the brake depression state, etc., and the driver's line of sight is likely to face the other vehicle, Calibration is automatically performed using the images taken in (including other vehicles). Further, when the ODO meter reset button is operated, calibration is performed using an image photographed by the camera assuming that the driver's line of sight is in a specific direction (see paragraph [0055]).

Japanese Patent Laid-Open No. 2006-49260 JP 2008-194146 A JP 2009-183473 A JP 2012-201232 A

  However, when the above-mentioned “corneal reflection method” or “pupil-corneal reflection method” is used for line-of-sight detection, the scale of the system for measurement increases, making it difficult to apply to the interior space of the vehicle. On the other hand, when the above “model-based method” is adopted, it can be realized with a single camera, which is suitable for application to the interior space of a vehicle.

  However, the “model-based method” has a lower gaze measurement accuracy than the “corneal reflection method”. That is, since the error in parameters used for eye gaze detection, such as the center position of the eyeball, is large, the detection accuracy decreases. One reason for this deterioration in detection accuracy is that the center position of the eyeball is detected while the face is not facing the front in the captured face image.

  Therefore, calibration is necessary. However, the method as disclosed in Patent Document 3 is difficult to apply to a vehicle because the burden on the user when performing calibration is very large. In addition, it is difficult to perform calibration with high accuracy by the method disclosed in Patent Document 4.

  The present invention has been made in view of the above-described circumstances, and an object of the present invention is to provide an on-vehicle device capable of accurately detecting whether or not the face orientation is facing the front in a photographed face image. It is to provide a method for calculating the degree of face front.

In order to achieve the above-described object, the vehicle-mounted device according to the present invention is characterized by the following (1) to (4).
(1) An in-vehicle device including a data processing unit that calculates a face front degree representing a degree that the face is facing the front from a face image photographed so as to include the face,
The data processing unit compares the first image data obtained by reducing the resolution of the face image with the second image data obtained by reducing the resolution of an image obtained by horizontally inverting the face image. The degree is calculated.
(2) The data processing unit
When the face front degree satisfies the condition, an eyeball center coordinate where the center of the three-dimensional eyeball model in which the eyeball included in the face image is modeled is calculated;
The vehicle-mounted device according to (1) above, wherein
(3) The data processing unit
Determining whether the face front degree satisfies a condition;
Determining whether the line of sight based on the position of the black eye included in the face image is facing the front,
The in-vehicle device according to (2), wherein the eyeball center coordinates in the three-dimensional eyeball model are calculated when it is determined that the face and the line of sight are facing the front.
(4) The data processing unit
The first image data having the same total number of pixels and the second image data are compared for each pixel at the same position, and the luminance of the pixel in the first image data and the second image data in the second image data are compared. The difference between the brightness of the pixels is totaled to calculate the face front degree.
The on-vehicle device according to any one of (1) to (3) above,

According to the vehicle-mounted device having the configuration (1) described above, since the images obtained by inverting the left and right faces are compared, the degree to which the face is facing the front can be detected only by a simple calculation process. In addition, since the resolution of the image is reduced before the comparison, the amount of data to be processed is reduced and the amount of calculation is reduced. In addition, by reducing the resolution, it is possible to eliminate the influence of local differences in the body such as moles in the face, so that the detection accuracy is not lowered due to individual differences.
According to the vehicle-mounted device having the configuration (2), it is possible to calculate the eyeball center coordinates only under the condition that the calculation error is small based on the face frontal degree. In other words, if the face of the person to be monitored is facing the front, the eyeball center coordinates can be calculated by a simple calculation process without causing a large calculation error.
According to the vehicle-mounted device having the configuration (3), the eyeball center coordinates can be calculated only in a situation where the face of the person to be monitored faces the front and the line of sight in the face faces the front. . As a result, the occurrence of calculation errors can be reduced, and the calculation process can be simplified.
According to the vehicle-mounted device having the configuration (4), the face front degree can be accurately calculated by a relatively simple calculation process.

In order to achieve the above-described object, the face front degree calculation method according to the present invention is characterized by the following (5).
(5) A face front degree calculation method for calculating a face front degree representing a degree that the face is facing the front from a face image photographed so as to include a face,
The face front degree is calculated by comparing the first image data in which the face image is reduced in resolution and the second image data in which the image obtained by horizontally inverting the face image is reduced in resolution. Features.

  According to the face front degree calculation method having the configuration (5) described above, since the images obtained by inverting the left and right faces are compared, the degree to which the face is facing the front can be detected only by a simple calculation process. In addition, since the resolution of the image is reduced before the comparison, the amount of data to be processed is reduced and the amount of calculation is reduced. In addition, by reducing the resolution, the influence of local differences in the body, such as the mole in the face, can be excluded, so that the detection accuracy is not lowered due to individual differences.

  According to the vehicle-mounted device and the face front degree calculation method of the present invention, it is possible to accurately detect whether or not the face is facing the front in the captured face image.

  The present invention has been briefly described above. Further, the details of the present invention will be further clarified by reading through a mode for carrying out the invention described below (hereinafter referred to as “embodiment”) with reference to the accompanying drawings. .

FIG. 1 is a perspective view illustrating a specific example of an arrangement state of the vehicle-mounted device according to the embodiment. FIG. 2 is a flowchart showing an outline of a processing procedure when a line of sight is detected using a three-dimensional eyeball model. FIG. 3 is a schematic diagram illustrating an example of a relationship between a two-dimensional input image and a three-dimensional eyeball model. FIG. 4 is a flowchart showing an outline of a processing procedure for automatically calibrating the eyeball center position. FIG. 5 is a flowchart showing a processing procedure for calculating the face front degree. FIG. 6 is a state transition diagram showing a change in the state of the face image before and after processing when calculating the face front degree. FIG. 7 is a front view showing a specific example of the relationship between the face center line and the left and right black eye positions in the face image. FIG. 8 is a plan view showing an example of the positional relationship between the eyeball center, the black eye center, and the viewing angle of the three-dimensional eyeball model. FIG. 9 is a front view showing an example of the positional relationship between the center line and the center of the left and right eyes when the center of the face is equal to the center of the image. FIG. 10 is a plan view showing an example of the positional relationship between the viewing angle, distance, and left and right black eye centers when the center of the face is equal to the center of the image. FIG. 11 is a front view showing an example of the positional relationship between the center line and the left and right black eye centers when the center of the face and the center of the image are not the same. FIG. 12 is a plan view illustrating an example of the positional relationship between the viewing angle, the distance, and the left and right black eye centers when the center of the face and the center of the image are not the same. FIG. 13 is a schematic diagram illustrating an example of the relationship between the correction value of the eyeball center position, the three-dimensional eyeball model, and the input image. FIG. 14 is a flowchart illustrating a specific example of processing for reflecting the correction value of the center position of the eyeball in the operation of the vehicle-mounted device.

  Specific embodiments of the vehicle-mounted device and the face front degree calculation method of the present invention will be described below with reference to the drawings.

<Specific examples of environment to which the present invention is applied>
The specific example of the arrangement | positioning state of the onboard equipment in embodiment of this invention is shown in FIG.
In the example shown in FIG. 1, a camera 12 and a processing unit 13 are installed in a vehicle interior 10 of a vehicle. Specifically, the person 11 sitting in the driver's seat, that is, the face including the driver's eye area 11a can be photographed with the camera 12 in the meter unit in front of the driver's seat or in the column cover. A camera 12 is installed.

  The processing unit 13 processes the two-dimensional image data obtained by the camera 12 capturing the face of the person 11, and estimates information indicating the direction of the line of sight 11b mainly from the data of the eye region 11a. The information on the line of sight 11b estimated by the processing unit 13 can be used for the vehicle-mounted device to recognize whether or not the driver driving the vehicle is moving the line of sight for safety confirmation. Of course, such line-of-sight information can be used for various purposes. Therefore, it is assumed that the vehicle-mounted device of the present invention is configured as, for example, the processing unit 13 in FIG. 1 or a part thereof.

  Note that the vehicle-mounted device of the present embodiment has a function of automatically calibrating the center position of the eyeball based on two-dimensional image data captured by the camera 12, as will be described later. Therefore, in order to calibrate the gaze detection function of the vehicle-mounted device, it is not necessary to prepare special calibration equipment other than the configuration shown in FIG. Also, the person 11 does not need to perform a special adjustment operation for calibration.

<Outline of eye-gaze detection algorithm>
FIG. 2 shows an outline of a processing procedure in the case where a gaze is detected based on a face image using a three-dimensional eyeball model. That is, when the computer of the vehicle-mounted device executes a predetermined program, operations according to the line-of-sight detection algorithm shown in FIG. 2 are sequentially performed.

  The camera 12 constantly shoots an image of an area including the face of the person 11 at a constant cycle, and outputs a video signal. The computer of the vehicle-mounted device captures one frame of the video signal from the camera 12 in step S11. In addition, “down-processing” such as conversion of the image data format including gray scale is actually performed in S11. For example, for each pixel position in one frame, 8-bit data representing the luminance with gradations in the range of “0 to 255” is aligned in the vertical direction (y) and horizontal direction ( Image data of a two-dimensional (2D) array arranged in x) is generated.

In step S12, the computer processes the one-dimensional or plural-frame two-dimensional image data input in step S11 to detect the face of the person 11. As a specific example, face detection is performed using the “Viola-Jones method”, and a region including a face is extracted as a rectangular region from one frame of two-dimensional image data. That is, a facial region is extracted using a detector that is characterized by the shadow difference of the face and created by learning using “Boosting”. Of course, the face may be detected by using a uniquely developed method. The technique of “Viola-Jones method” is shown in the following document.
"P.viola and MJJones," Rapid Object Detection Using a Boosted Cascade of Simple Features, "IEEE CVPR (2001)."

  In the next step S13, the data in the rectangular area of the face extracted in S12 is processed to detect each of the left and right black eyes (iris). For example, black eyes can be detected by using template matching which is a known technique. That is, for example, reference data of a template representing a feature of a black eye such as a black circular pattern is prepared in advance, and this template is sequentially scanned on the processing target image to specify a position where the feature matches. Note that before executing black-eye detection, the positions of various feature points or shading differences are detected for the entire face, and the positions of the regions of the left and right eyes are grasped. Thereby, when the left and right black eyes are detected, the range of data to be searched can be limited, so that the processing time can be shortened. For example, the eye region can be detected by using the “Viola-Jones method”.

  In this embodiment, since it is assumed that the line of sight is detected by a model-based method using a three-dimensional eyeball model, it is necessary to specify the eyeball center position as a reference point for the left and right eyeballs. The computer estimates the eyeball center position in step S14. For this processing, it is assumed that a method for determining the center position of the eyeball based on the position of the corner of the eye and the position of the eyeball, or a method for calculating the center position of the eyeball by simultaneously estimating the skeleton from the facial feature points. However, in the embodiment of the present invention, the eyeball center position is specified using a special algorithm as described later.

  In step S15, the computer determines the state (position, angle) of the three-dimensional eyeball model based on the detected black eye position and eyeball center position, and calculates the angle of the line of sight based on the result.

  In step S16, the computer performs processing such as data averaging and noise removal on the line-of-sight angle obtained in step S15 as necessary, and uses other information such as an application program that uses information on the line-of-sight direction. Data representing the line of sight is transmitted to the in-vehicle device.

<Description of 3D eyeball model>
An example of the relationship between the two-dimensional input image and the three-dimensional eyeball model is shown in FIG.
In the three-dimensional eyeball model 30 shown in FIG. 3, the eyeball is a sphere represented by an eyeball center 31 and a radius R. The black eye (iris) 32 is regarded as a disk-shaped region having a radius Di attached on the eyeball surface of the three-dimensional eyeball model 30. The center 32a of the black eye (iris) 32 corresponds to the center of the pupil. Note that the dimensions of the eyeball radius R and the black eye radius Di can be determined based on human body dimension data and various parameters of the imaging system of the camera 12.

  In this three-dimensional eyeball model 30, the direction of the line of sight can be specified as the direction from the eyeball center 31 toward the center of the black eye 32, and the rotation angle yaw with respect to the reference direction in the horizontal plane and the rotation angle with respect to the reference direction in the vertical direction It can be represented by a pitch. Further, the coordinates of the black eye center 32a can be expressed by an eyeball radius R, a yaw, and a pitch when the eyeball center 31 is used as a reference.

  On the other hand, the video imaged by the camera 12 represents a specific two-dimensional plane. Therefore, for example, when the two-dimensional image 35 of the eye region shown in FIG. 3 is input as the input image, it is necessary to perform two-dimensional / three-dimensional mutual conversion in order to use the three-dimensional eyeball model 30. .

  Specifically, the two-dimensional coordinates of the position of the black eye center 32 a and the position of the eyeball center 31 are respectively detected in the two-dimensional image 35 of the eye region, and these two-dimensional coordinates are obtained as the third order of the three-dimensional eyeball model 30. Project onto original coordinates. Thereby, the vector of the line of sight 11b can be calculated. The relationship between the vector of the line of sight 11b and the rotation angles yaw and pitch in each direction is expressed by the following equations.

X = −R × cos (pitch) × sin (yaw) (1)
Y = R × sin (pitch) (2)
X: distance in the x direction from the eyeball center 31 on the two-dimensional image plane Y: distance in the y direction from the eyeball center 31 on the two-dimensional image plane

<Description of characteristic processing>
<Outline of processing>
An outline of a processing procedure for automatically calibrating the position of the eyeball center 31 in the three-dimensional eyeball model 30 is shown in FIG. For example, the computer in the processing unit 13 executes the process shown in FIG. 4 as step S14 shown in FIG. 2, whereby the position of the eyeball center 31 is automatically calibrated. Thereby, since the positional accuracy of the eyeball center 31 is improved, the visual line detection accuracy is also improved. There is no need for a user such as a driver to perform a special operation for calibration.

  In step S101 of FIG. 4, the computer calculates the front degree of the face of the person 11 shown in the input image, that is, “the degree that the face is facing the front”. Then, based on the calculation result of S101, if the state can be regarded as “the face is facing the front”, the process proceeds from step S102 to S103.

  In step S103, the computer calculates the front degree of the line of sight of the person 11 shown in the input image, that is, “the degree that the direction of the line of sight is facing the front”. Then, based on the calculation result of S103, if the state can be regarded as “the direction of the line of sight is facing the front”, the process proceeds from step S104 to S105.

  In step S105, the latest eyeball center position is calculated based on the current characteristics of the person 11 shown in the input image, and a correction value necessary for updating the eyeball center position is obtained. In step S106, the correction value is reflected on the coordinates of the eyeball center 31 of the three-dimensional eyeball model 30 to execute calibration.

<Details of “Face Facing” Calculation>
The “face front degree” represents “the degree to which the face is facing the front” on the input image. A specific processing procedure for calculating the “face front degree” is shown in FIG. This processing procedure is executed in step S101 shown in FIG. In addition, FIG. 6 shows changes in the state of the face image before and after processing when calculating the degree of face front. The “face front degree calculation” shown in FIG. 5 will be described below.

  In step S31 of FIG. 5, the image data F of the face area detected in S12 of FIG. 2 is input as a processing target. The center coordinate of the face in the image data F of the face area is represented by the center position (facecen.x) of the face in the x-axis direction and the center position (facecen.y) in the y-axis direction as shown in FIG. .

  In step S32 of FIG. 5, the computer generates the left-right inverted image data FL obtained by processing the face area image data F input in S31 and inverting the left and right with respect to the center (facecen.x) of the x-axis in the face image. To do.

  In the next step S33, the computer generates reduced image data Fsmall by processing the image data F. This process means lowering the resolution of the image. For example, by thinning out one of two pixels adjacent to each other in the x direction in the image data F and reducing it to one pixel, the resolution in the x direction can be halved and the number of pixels can be halved. Further, when performing thinning, for example, an average value of the luminance of two adjacent pixels may be calculated and reflected in the luminance of the pixels after thinning. Similarly, by thinning out one of two pixels adjacent to each other in the y direction in the image data F and reducing it to one pixel, the resolution in the y direction can be halved and the number of pixels can be halved. When the number of pixels is reduced in this way, the size of the entire image is also reduced.

  Actually, the reduced image data Fsmall is created in step S33 so that the resolution is suitable for the detection of the degree of face front. For example, the physical characteristics of an individual such as Kuroko are excluded, and reduced image data Fsmall having an appropriate resolution is created so that information having a large relationship with the face direction such as the position of the black eye remains correctly. In the example illustrated in FIG. 6, the reduced image data Fsmall has the resolution reduced, so that the number of pixels M in the x direction is “20”, the number of pixels N in the y direction is “20”, and the total number of pixels (M × N ) Is significantly reduced compared to the original image data F.

  In the next step S34, the computer creates reduced image data FLsmall by processing the horizontally reversed image data FL created in S32. This process means lowering the resolution of the image. For example, the resolution in the x direction can be halved and the number of pixels can be halved by thinning out one of two pixels adjacent to each other in the x direction and reducing it to one pixel in the horizontally reversed image data FL. Further, when performing thinning, for example, an average value of the luminance of two adjacent pixels may be calculated and reflected in the luminance of the pixels after thinning. Similarly, in the horizontally reversed image data FL, by thinning out one of two pixels adjacent to each other in the y direction and reducing it to one pixel, the resolution in the y direction can be halved and the number of pixels can be halved. In the example shown in FIG. 6, the reduced image data FLsmall is reduced in resolution so that the number of pixels M in the x direction is “20”, the number of pixels N in the y direction is “20”, and the total number of pixels (M × N ) Is significantly reduced compared to the image data FL before processing.

  In step S35, the computer calculates the similarity between the reduced image data Fsmall of S33 and the reduced image data FLsmall of S34 as the face front degree D. In this example, it is assumed that the two reduced image data Fsmall and FLsmall have the same resolution as shown in FIG. The face front degree D in this case is calculated by the following equation.

D: Face front degree N: Number of pixels representing resolution in the x direction M: Number of pixels representing resolution in the y direction Fsmall (i, j): Luminance value FLsmall of the pixel at coordinates (i, j) in the reduced image data Fsmall ( i, j): luminance value α of pixel at coordinates (i, j) in the reduced image data FLsmall: threshold value for determining the degree of face front

  That is, the first image data (Fsmall) having the same total number of pixels (N × M) and the second image data (FLsmall) are compared for each pixel at the same position, and the pixels in the first image data are compared. The face front degree D is calculated by summing up the difference between the brightness of the pixel and the brightness of the pixel in the second image data.

  Note that an x-coordinate value (facecen.x) and a y-coordinate value (facecen.y) representing the center position of the face are respectively the median value in the horizontal (x) direction and the vertical (y ) Can be calculated as the median of the direction.

  When the face front degree D calculated by the above expression (3) is used, it is possible to identify whether or not the face direction is the front by comparing the face front degree D with the threshold value α in step S102 of FIG. As for the magnitude of the threshold value α, it is assumed that an optimum value is adopted in consideration of a parameter representing the photographing condition of the camera 12 and individual differences.

  In other words, assuming that the facial features are symmetrical with respect to the center of the face (facecen.x), the features on the left side and the features on the right side from the center of the face are equal. Therefore, the face front degree D reflecting the similarity between the image data F and the horizontally reversed image data FL is very small when the face direction is close to the front, and increases when the face direction is deviated from the front.

  In the above expression (3), instead of the image data F and FL, the similarity of the reduced image data Fsmall and FLsmall whose resolution is lower than these is calculated as the face front degree D. As a result, the following advantages (1) and (2) can be obtained.

(1) Since the data amount to be processed, that is, the number of pixels is reduced, the calculation processing amount is reduced. Therefore, the processing can be completed in a short time without mounting a high-performance computer on the vehicle-mounted device.
(2) It becomes easy to eliminate the influence of the left and right local differences on the image that are unrelated to the face orientation. For example, the presence / absence and position of moles, which are physical features of an individual, are not related to the orientation of the face, but are detected as a difference between left and right on the image and affect the value of the face front degree D. However, when the reduced image data Fsmall and FLsmall having a low resolution are used, the influence of small moles is ignored and the value of the face front degree D is not affected.

<Details of “Gaze frontality” calculation>
A specific example of the relationship between the face center line and the left and right black eye positions in the face image is shown in FIG.
The gaze front degree FD is calculated from the degree to which the gaze direction of the person 11 shown in the input image is facing the front, that is, the rotation angle obtained by applying the three-dimensional eyeball model 30 to the left and right eyes. Represents the degree to which the viewing direction θ (yaw, pitch) is 0 degrees.

In the present embodiment, as shown in FIG. 4, the computer calculates the gaze front degree FD according to the following expression on the premise that the condition that “the face direction is the front” is satisfied.
FD: Gaze frontality irisR. x: x coordinate irisL. x: x-coordinate facecen. x: x-coordinate of the center of the face β: threshold for identifying whether the direction of the line of sight is front or not

  In the present embodiment, the positional relationship between the left and right eyes such as both eyes is described as representing the left and right viewed from the camera 12 side and the left and right on the two-dimensional image obtained by photographing, not the left and right with respect to the person 11. .

  That is, the line-of-sight frontality FD represented by the above expression (4) represents the difference between the center position between the left and right black eye centers and the position of the face center line (facecen.x) in the x-axis direction. ing. Therefore, for example, if the distance from the center line position (facecen.x) to the center position of the right black eye and the distance to the center position of the left black eye are equal, the direction of the line of sight faces the front. Can be determined. Further, if it can be determined that the direction of the line of sight is facing the front based on the positional relationship in the x-axis direction, it can be considered that the direction of the line of sight is also facing the front in the y-axis direction at the same time.

  The line-of-sight front degree FD becomes 0 when the direction of the line of sight faces the front, and increases as the line of sight deviates from the front. Therefore, when using the gaze front degree FD calculated by the above expression (4), the gaze front degree FD is compared with the threshold β in step S104 of FIG. 4 to determine whether or not the gaze direction is front. Can be identified. As for the magnitude of the threshold value β, it is assumed that an optimum value is adopted in consideration of parameters representing photographing conditions of the camera 12 and individual differences.

<Calculation of eyeball center position>
The process for calculating the latest eyeball center position in step S105 shown in FIG. 4 will be described below.

<When the center of the face in the x-axis direction is equal to the center of the image>
FIG. 8 shows an example of the positional relationship in the horizontal plane (or plane parallel to the x and z axes) of the eyeball center, black eye center, and viewing angle of the three-dimensional eyeball model. FIG. 9 shows an example of the positional relationship between the center line and the center of the left and right eyes when the center of the face in the x-axis direction is equal to the center of the image. FIG. 10 shows an example of the positional relationship in the horizontal plane between the viewing angle, the distance, and the left and right black eye centers when the center of the face in the x-axis direction is equal to the center of the image.

Since each of the left and right eyes can be calculated by substantially the same method, the following description assumes that only the right eye is processed as a calculation target.
For example, in the state shown in FIG. 8, the relationship of the following expression (5) is established.

Xeyer = irisR. x-Ro · sin θ Ryaw (5)
Xeyer: x coordinate of right eyeball center irisR. x: x coordinate of the right eye center Ro: radius of the eyeball (eg constant)
θRyaw: Eye angle of the right eye (yaw direction)

  That is, if the position “irisR.x” of the black eye center and the line-of-sight angle “θRyaw” are known, the x-coordinate “Xeyer” of the eyeball center can be calculated from the above equation (5).

Also, as shown in FIG. 9, the x-coordinate “facecen.x” of the face center position in the image is equal to the x-coordinate “Icenter.x” of the center of the image taken by the camera, as shown in FIG. Assuming the situation, the line-of-sight angle “θRyaw” can be calculated by the following equation (6).
tan θRyaw = (irisR.x-Icenter.x) / Df (6)
Df: Distance between camera and face

  The distance Df can be treated as known information if it is measured using an appropriate distance sensor, for example. Further, for example, if a technique disclosed in “Japanese Patent Laid-Open No. 2015-206764” is used, a distance sensor becomes unnecessary. Further, since the black eye can be detected from the face image as described above, the position “irisR.x” of the center of the black eye can also be used as known information. Further, for example, a predetermined constant can be used as the x-coordinate “Icenter.x” of the center of the image.

  Therefore, the line-of-sight angle “θRyaw” can be calculated from the equation (6). Further, the x-coordinate Xeyer of the right eyeball center can be calculated by substituting the line-of-sight angle “θRyaw”, the black eye center position “irisR.x”, and the eyeball radius Ro into the expression (5).

Also, the y-coordinate Yeyer at the right eyeball center can be similarly calculated by using the following expressions (7) and (8).
Yeyer = irisR. y-Ro · sinθRpitch (7)
Yyer: y-coordinate of the right eyeball center irisR. y: y coordinate of the right eye center Ro: radius of eyeball (for example, constant)
θRpitch: right eye's line-of-sight angle (pitch direction)

tan θRpitch = (irisR.y-Icenter.y) / Df (8)
Icenter. y: y coordinate of the center of the image Df: distance between the camera and the face

<When the center of the face in the x-axis direction is shifted from the center of the image>
An example of the positional relationship between the center line and the center of the left and right eyes when the center of the face and the center of the image are not the same is shown in FIG. FIG. 12 shows an example of the positional relationship in the horizontal plane (plane parallel to the x and z axes) of the center of the left and right eyes when the face center and the image center are not the same.

  For example, as shown in FIG. 12, even if the x-coordinate “facecen.x” of the center line of the face is deviated from the x-coordinate “Icenter.x” of the center of the image as shown in FIG. Is centered on the camera 12 and exists on the circumference whose radius is represented by the distance Df. Further, the face center position “facecen.x” and the image center “Icenter.x” are shifted by a face rotation angle “facerad”. However, if the face is facing the camera 12 and the eyes of both eyes are facing the camera 12, the line-of-sight angle “θRyaw” of the right eye is as shown in FIG. It can be expressed as the rotation angle of the eye position with respect to the center “Icenter.x”.

  Therefore, even when the center of the face in the x-axis direction is shifted from the center of the image, the line-of-sight angle “θRyaw” can be calculated using the above equation (6). Further, the x-coordinate “Xeyer” of the eyeball center can be calculated based on the line-of-sight angle “θRyaw” and the expression (5).

On the other hand, even when the coordinates of the center of the eyeball are calculated for the left eye, it can be calculated in the same manner as described above by using the following calculation formulas.
Xeyel = irisL. x-Ro · sin θLyaw (9)
Xeyel: x-coordinate of left eyeball center irisL. x: x coordinate of the center of the black eye on the left side Ro: radius of eyeball (for example, constant)
θLyaw: Eye angle of the left eye (yaw direction)
tan θLyaw = (irisL.x-Icenter.x) / Df (10)
Df: Distance between camera and face

Yeyel = irisL. y-Ro · sinθLpitch (11)
Yeel: y-coordinate of left eyeball center irisL. y: y coordinate of the left eye center Ro: radius of eyeball (for example, constant)
θLpitch: Eye angle of the left eye (pitch direction)
tan θLpitch = (irisL.y-Icenter.y) / Df (12)
Icenter. y: y coordinate of the center of the image Df: distance between the camera and the face

<Correction of eyeball center position>
Even when there is an error in the position coordinates of the eyeball center 31 in the three-dimensional eyeball model 30, the correction can be performed so that the error can be reduced by correcting using the latest eyeball center position obtained by the above calculation. it can. The correction value can be calculated by, for example, the following formula.

dXeye = Xeye-Xeyemodel (13)
dYye = Yeye-Yeemodel (14)
dXeye: x coordinate correction value (right or left)
dYeye: y coordinate correction value (right or left)
Xeye: The latest x-coordinate of the calculated center position of the eyeball (right or left)
Yeye: the latest y-coordinate of the calculated center position of the eyeball (right or left)
Xeyemodel: eye center x coordinate (right or left) before correction in the three-dimensional eye model
Eyemodel: y-coordinate of the eyeball center before correction in the three-dimensional eyeball model (right or left)

<Eyeball center position correction 1>
An example of the relationship between the correction value dXeye of the eyeball center position, the three-dimensional eyeball model, and the input image is shown in FIG.

  In step S105 shown in FIG. 4, since the face front degree D is less than or equal to the threshold value α and the line-of-sight front degree FD is less than or equal to the threshold value β, the coordinates of the latest eyeball center position (Xeye, Yeye) are determined. It calculates about each of.

  Then, the correction values dXeye and dYeye are calculated for each of the left and right eyes by the above equations (13) and (14). Further, the position coordinates of the eyeball center 31 in the three-dimensional eyeball model 30 are updated for each of the left and right eyes as shown in FIG. 13 using the correction values dXeye and dYeye so as to reduce the error.

<Eyeball center position correction 2>
A specific example of processing for reflecting the correction value of the center position of the eyeball in the operation of the vehicle-mounted device is shown in FIG.
When the process shown in FIG. 4 is applied to the vehicle-mounted device, step S105 is repeatedly executed each time a state where the person 11 faces the front and the line of sight faces the front is detected. Therefore, the update of the eyeball center position can be repeated using the new correction value after the power is turned on to the vehicle-mounted device and before the operation is completed.

  Therefore, in order to perform a more appropriate update process, for example, the computer executes the process of “eyeball center position correction 2” shown in FIG. In FIG. 14, only the coordinate in the x-axis direction is processed, but the coordinate in the y-axis direction can be similarly processed. With the processing shown in FIG. 14, the coordinates of the eyeball center position can be converged to a certain value.

  In step S41 of FIG. 14, the computer calculates a correction value dXeye for the center position of the eyeball based on the above-described equations (5), (6), and (13). In addition, in order to eliminate an apparent outlier that may occur due to the influence of noise, the correction value dXeye is compared with a predetermined threshold value in the next step S42.

  If the correction value dXeye is not an obvious outlier, 1 is added to the update count value N in step S43 (the initial value of N is 0). In the next step S44, the data of the current correction value dXeye is stored in a memory location associated with the update count value N.

  In step S45, the computer reads data of N correction values dXeye stored in the memory, and outputs a result dXeye_T obtained by averaging the data. As a specific method of averaging, calculation methods such as a simple average value, a weighted average value, and a moving average value can be applied.

  By applying the processing as shown in FIG. 14, it is possible to exclude an apparent outlier due to the influence of noise from the update target. Moreover, the rapid change of the eyeball center position can be avoided by averaging. That is, it is robust against noise. Therefore, in an environment where an operation targeting the same user is performed for a long time, such as an in-vehicle device that monitors the driver's line of sight, high eye-gaze detection accuracy can be expected by updating the effective eyeball center as needed. .

<Advantages of OBE>
In the above-described vehicle-mounted device, by applying the face front degree calculation method corresponding to the process shown in FIG. 5, the degree to which the face of the person 11 faces the front can be accurately detected by a simple calculation process. Further, since the reduced image data Fsmall and FLsmall whose resolution is lower than that of the input image are used, the calculation processing amount can be reduced. Also, calculation errors due to the influence of individual physical features such as Kuroko can be reduced.

  Further, by performing the processing shown in FIG. 4 and correcting the position of the eyeball center 31 only in a state where it is detected that the face of the person 11 is facing the front, errors in calibration can be reduced. Further, as in the process shown in FIG. 4, an error is generated by correcting the position of the eyeball center 31 only when it is detected that the face of the person 11 faces the front and the line of sight faces the front. Can be reduced. Further, it is not necessary for the user to perform a special operation for performing this calibration, and it is not necessary to prepare special calibration equipment. Further, by repeatedly executing the processing shown in FIG. 14, it is possible to eliminate the influence of noise and prevent a sudden change, so that malfunction of the vehicle-mounted device can be avoided.

  Therefore, the above-described technique can be used for driver's line-of-sight detection and side-view detection using an in-vehicle camera. Also, for example, it can be used for visual confirmation of digital signage (electronic signage).

Here, the features of the above-described embodiments of the vehicle-mounted device and the face front degree calculation method according to the present invention are summarized and listed in the following [1] to [5], respectively.
[1] An on-vehicle device including a data processing unit that calculates a face front degree representing a degree of the face facing the front from a face image photographed so as to include the face,
The data processing unit (processing unit 13) includes first image data (Fsmall) obtained by reducing the resolution of the face image, and second image data obtained by reducing the resolution of an image obtained by horizontally inverting the face image. The face front degree (D) is calculated by comparing (FLsmall) (S31 to S35),
In-vehicle device characterized by that.
[2] The data processing unit
When the face front degree satisfies the condition, the eyeball center coordinates (eyeball center 31) at which the center of the three-dimensional eyeball model (30) in which the eyeball included in the face image is modeled are calculated (S105),
The on-vehicle device according to [1] above, wherein
[3] The data processing unit
It is determined whether the face front degree satisfies a condition (S102),
It is determined whether or not the line of sight based on the position of the black eye included in the face image is facing the front (S104),
When it is determined that the face and line of sight are facing the front, the eyeball center coordinates in the three-dimensional eyeball model are calculated (S105).
The on-vehicle device according to [2] above,
[4] The data processing unit
The first image data having the same total number of pixels and the second image data are compared for each pixel at the same position, and the luminance of the pixel in the first image data and the second image data in the second image data are compared. Summing up the difference with the luminance of the pixel to calculate the face front degree (see equation (3)),
The vehicle-mounted device according to any one of [1] to [3], wherein
[5] A face front degree calculation method for calculating a face front degree representing a degree that the face is facing the front from a face image photographed so as to include the face,
By comparing the first image data (Fsmall) in which the face image is reduced in resolution and the second image data (FLsmall) in which the image obtained by horizontally inverting the face image is reduced in resolution, the front of the face Calculating the degree (D),
A method for calculating the degree of face front.

DESCRIPTION OF SYMBOLS 10 Vehicle interior 11 Person 11a Eye area 11b Line of sight 12 Camera 13 Processing part 30 3D eyeball model 31 Eyeball center 32 Black eye (iris)
32a Center of black eyes (pupil)
35 Two-dimensional image of eye area F Image data FL Left-right inverted image data Fsmall, FLsmall Reduced image data α, β Threshold

Claims (5)

An in-vehicle device including a data processing unit that calculates a face front degree representing a degree that the face is facing the front from a face image photographed so as to include a face,
The data processing unit compares the first image data obtained by reducing the resolution of the face image with the second image data obtained by reducing the resolution of an image obtained by horizontally inverting the face image. Calculate the degree,
In-vehicle device characterized by that. The data processing unit
When the face front degree satisfies the condition, an eyeball center coordinate where the center of the three-dimensional eyeball model in which the eyeball included in the face image is modeled is calculated;
The on-vehicle device according to claim 1. The data processing unit
Determining whether the face front degree satisfies a condition;
Determining whether the line of sight based on the position of the black eye included in the face image is facing the front,
The in-vehicle device according to claim 2, wherein the eyeball center coordinates in the three-dimensional eyeball model are calculated when it is determined that the face and the line of sight are facing the front. The data processing unit
The first image data having the same total number of pixels and the second image data are compared for each pixel at the same position, and the luminance of the pixel in the first image data and the second image data in the second image data are compared. The difference between the brightness of the pixels is totaled to calculate the face front degree.
The on-vehicle device according to any one of claims 1 to 3, wherein A face front degree calculation method for calculating a face front degree representing a degree to which the face faces the front from a face image photographed so as to include a face,
The face front degree is calculated by comparing the first image data in which the face image is reduced in resolution and the second image data in which an image obtained by horizontally inverting the face image is reduced in resolution.
A method for calculating the degree of face front.