JP2015211741A - Line-of-sight state determination device, line-of-sight state determination program and line-of-sight state determination method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a line-of-sight state determination device for determining a state of a line of sight of a user with high accuracy.SOLUTION: A gazing point image area estimation part 24 acquires information on a gazing point (gazing point and gazing point image area) respectively gazed by a driver at a time A and a time B. Then, a search window calculation part 26 specifies a prescribed range (search window) including a gazing point of the time A and a gazing point of the time B, and a tracking determination part 28 determines whether the line of sight of the driver tracks a visual recognition object on the basis of whether a common object is picked up within the range in the search window of an image at the time A and within the range in the search window of an image at the time B.

Description

The present invention relates to a gaze state determination device, a gaze state determination program, and a gaze state determination method.

  Conventionally, as a technique for analyzing a line-of-sight state, there is known a method for identifying what a user is looking at using a line-of-sight direction measured by a line-of-sight detection device and an in-vehicle image capturing the line-of-sight direction. (For example, see Patent Documents 1 and 2).

  For example, in Patent Document 1, an in-vehicle camera image capturing a region including the line-of-sight direction is determined around the line-of-sight direction measured by the line-of-sight detection device in order to determine the object to be visually recognized during traveling of the host vehicle and its following vision. Convert to a visual field image. Then, a character near the line-of-sight direction in the visual field image is recognized, and an outdoor object holding the character is specified from outdoor object information narrowed down from the current position.

  Further, in Patent Document 2, in order to determine a visual target and following vision using a vehicle-mounted image of a vehicle-mounted camera and a line-of-sight direction measured by a line-of-sight detection device, an optical of a region corresponding to the line-of-sight direction in the vehicle-mounted image is disclosed. Compare changes in flow and gaze direction.

JP 2007-172378 A JP 2010-39933 A

  However, the techniques disclosed in Patent Documents 1 and 2 and the like may not necessarily determine whether or not the same object is being followed. For example, in Patent Document 1, when the visual recognition target does not have a feature such as a character, it is difficult to determine the target to be visually recognized by the user. Further, in Patent Document 2, it is impossible to cope with a case where a camera position for capturing a video is different from a user's line-of-sight position and the movement of an object on the video and the movement of the line of sight following the object do not match.

  In one aspect, an object of the present invention is to provide a gaze state determination device, a gaze state determination program, and a gaze state determination method that can accurately determine a user's gaze state.

  In one aspect, the line-of-sight state determination device includes: an acquisition unit that acquires information on a gaze point that a user gazes at each of a first time and a second time after a predetermined time has elapsed from the first time; A specifying unit that specifies a predetermined range including a gaze point at one time and a gaze point at the second time, a range corresponding to the predetermined range of the image at the first time, and an image at the second time And a determination unit that determines the state of the user's line of sight based on whether or not an object common to the range corresponding to the predetermined range is imaged.

  In one aspect, the line-of-sight state determination program acquires information on a gaze point that the user gazes at each of a first time and a second time after a predetermined time has elapsed from the first time, and the first time A predetermined range including the gazing point of the second time and the gazing point of the second time, and a range corresponding to the predetermined range of the image of the first time and a predetermined range of the image of the second time This is a line-of-sight state determination program for causing a computer to execute a process of determining the state of the line of sight of a user based on whether or not a common target exists in a range to be performed.

  In one aspect, the gaze state determination method acquires information on a gazing point that the user gazes at each of a first time and a second time after a predetermined time has elapsed from the first time, and the first time A predetermined range including the gazing point of the second time and the gazing point of the second time, and a range corresponding to the predetermined range of the image of the first time and a predetermined range of the image of the second time This is a line-of-sight state determination method in which a computer executes a process of determining a state of a user's line of sight based on whether or not a common target exists in a range to be performed.

  The state of the user's line of sight can be accurately determined.

It is a figure showing roughly composition of an in-vehicle system concerning one embodiment. It is a figure which shows the hardware constitutions of information processing apparatus. It is a functional block diagram of an information processor. FIG. 4A and FIG. 4B are diagrams for explaining a method of calculating the parallax of the camera with respect to the viewing direction. FIGS. 5A and 5B are schematic diagrams illustrating a method of calculating a gazing point image region. It is a figure which shows the example of adjustment of a gaze point image area | region. It is a figure which shows the example of calculation of a search window. FIG. 8A and FIG. 8B are diagrams showing an example of adjusting the search window. FIGS. 9A and 9B are diagrams (part 1) for explaining the flow calculation. FIGS. 10A and 10B are diagrams (part 2) for explaining the flow calculation. FIG. 11A and FIG. 11B are diagrams (part 3) for explaining the flow calculation. FIG. 12A and FIG. 12B are diagrams (part 4) for explaining the flow calculation. FIGS. 13A and 13B are diagrams illustrating an example of the tracking determination process performed by the tracking determination unit. It is a figure which shows the example which determines a follow-up vision using the flow calculation result of twice the time length of FIG.13 (b). It is a figure which shows an example of the time change of a gaze direction. 5 is a flowchart (part 1) illustrating an example of processing by the information processing apparatus. 12 is a flowchart (part 2) illustrating an example of processing by the information processing apparatus. 12 is a flowchart (part 3) illustrating an example of processing by the information processing apparatus. It is a figure which shows the example in case a visual recognition direction moves and the change vector of a gaze direction and the movement vector of the visual recognition object on an image become a reverse direction substantially.

  Hereinafter, an embodiment of an in-vehicle system will be described in detail with reference to FIGS. FIG. 1 schematically shows the configuration of an in-vehicle system 100 according to an embodiment. The in-vehicle system 100 in FIG. 1 is a system that is mounted on a moving body (for example, a car) and determines the state of the line of sight of a user (hereinafter referred to as a driver). As shown in FIG. 1, the in-vehicle system 100 includes a plurality of cameras 10, a line-of-sight detection device 12, a vehicle-mounted sensor group 14, and an information processing device 20 as a line-of-sight state determination device.

  The camera 10 is a camera that photographs a surrounding environment mounted on a car. It is preferable that a plurality of cameras are installed in the vehicle so as to reduce the blind spot due to the camera installation position and angle of view and cover the driver's visual recognition area, but only one wide-angle camera may be installed. . In the present embodiment, it is assumed that a plurality of cameras 10 are mounted on the car. It should be noted that information on which position of each camera 10 is installed and in which direction the camera 10 is imaged is stored in advance in a camera information DB 40 (see FIG. 3) described later.

  The line-of-sight detection device 12 acquires the line-of-sight direction of the driver by a known method. For example, the line-of-sight detection device 12 includes an in-vehicle camera that can capture the driver's face and a sensor that estimates the driver's line-of-sight direction from the driver's pupil reflection using an LED light that emits light toward the driver's eyes. Note that the line-of-sight detection device 12 may estimate the line-of-sight direction from the direction of the face or head of the photographed driver, instead of directly acquiring the line-of-sight direction. Note that the gaze direction may be acquired for each of the left and right eyes, or one gaze direction may be specified from the left and right eyes. In the latter case, as the line-of-sight direction, an arbitrary direction using the line-of-sight directions of the left and right eyes, for example, an average value of each line-of-sight direction may be specified. In addition, basically, the right and left eye gaze directions are acquired, and only one eye gaze direction is obtained when only one eye gaze direction can be obtained due to the influence of external light, vibration, glasses, etc. You may make it change the acquisition number of a gaze direction at any time.

  The in-vehicle sensor group 14 is a sensor group that is mounted on a vehicle and acquires various data during traveling, and includes a speed sensor, a GPS (Global Positioning System) positioning device, a rainfall sensor, an illuminance sensor, a vibration sensor, and the like.

  The information processing apparatus 20 is an apparatus that acquires images and data from the camera 10, the line-of-sight detection device 12, and the in-vehicle sensor group 14, and determines the state of the driver's line of sight based on the acquired images and data.

  FIG. 2 schematically shows the hardware configuration of the information processing apparatus 20. As shown in FIG. 2, the information processing apparatus 20 includes a CPU (Central Processing Unit) 90, a ROM (Read Only Memory) 92, a RAM (Random Access Memory) 94, and a storage unit (here, an HDD (Hard Disk Drive)). 96, an input / output interface 97, a portable storage medium drive 99, and the like. Each component of the information processing apparatus 20 is connected to a bus 98. In the information processing apparatus 20, a program (including a line-of-sight determination program) stored in the ROM 92 or the HDD 96, or a program (including a line-of-sight determination program) read from the portable storage medium 91 by the portable storage medium drive 99. 3 is implemented by the CPU 90.

  FIG. 3 shows a functional block diagram of the information processing apparatus 20. When the CPU 90 of the information processing apparatus 20 executes the program, as shown in FIG. 3, the camera selection unit 22, the gazing point image region estimation unit 24 as the acquisition unit, the search window calculation unit 26 as the identification unit, and the determination unit Functions as the follow-up determination unit 28, the visual recognition time calculation unit 30, the necessary visual recognition time determination unit 32, the visual recognition determination unit 34, and the side-view determination unit 36 are realized. 3 also shows a camera information DB 40, a map DB 42, and a target image feature DB 44 stored in the HDD 96 or the like.

  The camera selection unit 22 selects the camera 10 used for processing from a plurality of cameras. Specifically, the camera selection unit 22 detects the driver's line-of-sight direction detected by the line-of-sight detection device 12 at two times (a first time and a second time) sufficiently close to the time at which the driver's visual state is to be determined. The parallax is obtained from the installation position relationship of each in-vehicle camera, and the camera with the least parallax is selected at both times. In this case, the camera selection unit 22 refers to information stored in the camera information DB 40. It is assumed that the camera information DB 40 stores the position of each of the plurality of cameras 10, the position of the driver's viewpoint, and the like.

  In addition, when the camera selection unit 22 selects two times, for example, if the time at which the visual recognition state is to be determined is the current time, the time slightly before the current time and the current time, or the current time and the time slightly later than the current time, Select as follows. In the former case, strictly, the state before the time to be determined is determined, and the latter is the state after the time to be determined.

  FIGS. 4A and 4B are diagrams illustrating a method for calculating the parallax of the camera with respect to the viewing direction. As shown in FIG. 4A, the camera selection unit 22 is in the direction of the line of sight (direction of the line of sight angle K) detected by the line-of-sight detection device 12 at a certain time, and is a driver viewpoint for an arbitrarily determined distance (R). Assuming a visual target (K1) that exists at a location away from (E), a vector K1-C that connects the visual target (K1) and the center (C) of the camera 10 is calculated. And the camera selection part 22 calculates | requires the angle (alpha) K which the vector K1-C and the gaze direction vector K1-E by the gaze detection apparatus 12 comprise, and calculates as arbitrary values proportional to the angle (alpha) K which comprises parallax. FIG. 4A shows another two line-of-sight directions (directions of the line-of-sight angles L and M), objects to be visually recognized (L1, M1), and angles αL and αM formed for comparison purposes. Also, in FIG. 4A, for the sake of convenience, an arc having a radius R centered on the driver viewpoint E is shown to indicate that the visual recognition target (K1, L1, M1) is equidistant from the viewpoint (radius R). doing. The camera selection unit 22 calculates the parallax at the same time in the plurality of cameras 10 and selects the camera 10 having the smallest parallax as a camera used to determine the state of the driver's line of sight.

  4A shows a calculation method in the case where the driver viewpoint E and the line-of-sight direction have a single value for a certain time. For example, the positions of the left and right eyes and the line-of-sight directions of the left and right eyes are acquired. If possible, the camera selection unit 22 may calculate the parallax based on them. In this case, as shown in FIG. 4B, the intersection of a straight line extending from the left eye position in the left visual line direction and a straight line extending from the right eye position in the right visual line direction can be set as the position of the visual target K1. As shown in FIG. 4A, the angle formed without using the distance R can be calculated. When the method of FIG. 4B is adopted, the position and angle formed can be calculated more accurately than the method of FIG. However, since it cannot be used when the viewpoint position and line-of-sight direction of both left and right eyes cannot be acquired, only when the viewpoint and line-of-sight direction for both left and right eyes can be acquired from both the line-of-sight data at two times, as shown in FIG. The method shown in FIG. 4A may be adopted in other cases.

  In FIG. 4A, it is assumed that a visual target is separated from the driver viewpoint E by a distance R. Instead, a visual target having a distance R centered on the position C of the camera 10 is obtained. May be. In this case, an arc having a radius R centered on the position C of the camera 10 is drawn, and an intersection of a straight line and a circle extending from the viewpoint in the line-of-sight direction is set as a visual recognition target. In this case, the assumed visual target position (K1 to M1) is slightly different from that in FIG. 4A, but the camera with the smallest parallax can be selected as in FIG. 4A. Is possible.

  In order to select the camera with the least parallax at two times, for example, the parallax with the line of sight at each time is obtained for each camera, and the sum, average, maximum, or minimum of the parallax at each time is obtained. It is sufficient to obtain the value of, compare the values of all the cameras, and select the camera with the smallest value.

  The camera selection method described above is an example. That is, for example, when it is detected that the object to be viewed on the left side is viewed with respect to the line of sight, the camera selection unit 22 may simply select a camera installed on the left side of the vehicle. In addition, in the camera information DB 40, a table in which the driver's line-of-sight direction is associated with the camera 10 to be selected is prepared in advance, and the camera selection unit 22 may select the camera 10 based on the table. Good. In this case, if the cameras selected at each time are different, the camera selection unit 22 calculates only the parallax for the two selected cameras, and selects the camera based on the parallax. Also good. Alternatively, the camera selection unit 22 may determine the camera priority order in advance, and may select a camera with a higher priority when the cameras selected at each time are different.

  The camera selection may be performed at every measurement interval of the line-of-sight detection device 12 and every imaging interval of the camera 10, but for example, timing when the line-of-sight direction detected by the line-of-sight detection device 12 becomes a predetermined threshold or more. It is good also as implementing in. Alternatively, when the line of sight moves frequently with the passage of time, the cameras may be selected at predetermined time intervals in order to avoid frequent changes of the selected cameras. In addition, when a plurality of cameras are not provided in the car, the camera selection unit 22 may be omitted.

  Returning to FIG. 3, the gazing point image region estimation unit 24 corresponds to the line of sight detected by the line-of-sight detection device 12 from the positional relationship between the line-of-sight detection device 12 and the selected camera 10 at each of the two times described above. A gazing point on an image (hereinafter, referred to as “scene image”) captured by the selected camera 10 is specified. In addition, the gazing point image area estimation unit 24 sets an image area of an arbitrary size (hereinafter referred to as a “gaze point image area”) around the specified gazing point.

  FIGS. 5A and 5B are schematic diagrams illustrating a method of calculating a gazing point image region. The gazing point image area estimation unit 24 is described geometrically as the imaging range determined based on the selected camera position and the imaging angle of view of the camera, and the viewpoint position (the center of the left and right eye positions in FIG. 5A). ) And the intersection of straight lines extending in the line-of-sight direction is calculated as the point of gaze. Then, as shown in FIG. 5B, the gazing point image area estimation unit 24 determines an area having the error width (R1, R2) around the calculated gazing point as the gazing point on the scene image. The image area. The shape of the gazing point image area is arbitrary, but a simple shape such as a rectangle is preferable in consideration of the ease of calculation.

  In FIG. 5A, the geometric calculation method is shown as the method of calculating the gazing point image region, but the present invention is not limited to this. For example, in addition to the case where the gaze image position is determined geometrically, the correspondence between the pixel positions on the scene image when viewing various gaze directions is acquired in advance as calibration information, and the gaze direction You may identify the pixel position with respect to arbitrary gaze directions, ie, a gaze point position, using the correspondence of a pixel position.

  Here, the size (error width R1, R2) of the gazing point image area can be determined based on, for example, the measurement error of the line-of-sight detection device 12 or the magnitude of the measurement error possibility. Note that the measurement error means the measurement performance of the line-of-sight detection device 12 and the measurement accuracy value output by the line-of-sight detection device 12, and the possibility of measurement error means the presence or absence of a factor that deteriorates the measurement result. Factors that deteriorate the measurement results include, for example, whether or not the eyes of both eyes were measured as the lines of sight, the measurement results (acceleration values, etc.) of the acceleration sensors included in the in-vehicle sensor group 14, and the measurement results (traveling position) of the GPS positioning device ) And the map information stored in the map DB 42, and whether or not it is detected that the environment is close to backlight due to the brightness outside the vehicle being bright from the detection result of the illumination sensor. Please mention. These are situations where the accuracy of the line-of-sight data acquired by the line-of-sight detection device 12 seems to be low. The point-of-gaze image area estimation unit 24 increases the size of the point-of-gaze image area around the point of interest on the scene image so that the larger the measurement error or the possibility of measurement error of the line-of-sight detection device 12 is, the larger the area is. To decide. The map DB 42 stores map information that can roughly estimate the surrounding situation of the current position. As the map information, a general digital road map or the like can be adopted.

  In the present embodiment, the gazing point image area estimation unit 24 may determine the size of the gazing point image area based on the parallax between the line-of-sight detection device 12 and the selected camera 10 described above. . When the parallax between the gaze detection device 12 and the selected camera is large, a situation in which the visual target moves on the scene image and a large divergence due to movement of the gaze are likely to occur. If there is an error in the gazing point position of the line-of-sight detection device 12, a large error is likely to occur in the positioning of the image area. For this reason, as the size of the image area, in addition to the error of the line-of-sight detection device 12, the magnitude of the parallax may be taken into consideration, and the larger the size of the parallax, the larger the gazing point image area may be determined. As a result, it is possible to prevent the error influence from expanding due to parallax.

Further, the gazing point image area estimation unit 24 may adjust the size of the gazing point image area using a change in the gaze direction at two times. For example, the gazing point image region estimation unit 24 compares the gazing directions at two times and enlarges the gazing point image region in the direction in which the sight line changes, so that the gazing point image at a later time of the two times is increased. The area may be adjusted. Similarly, the gazing point image region estimation unit 24 may adjust the gazing point image region at the previous time of the two times so as to enlarge the gazing point image region opposite to the direction in which the line of sight changes. FIG. 6 illustrates this adjustment example. In FIG. 6, the left side of the gazing point image area at the previous time and the right side of the gazing point image area at the later time are adjusted to be larger with respect to the line-of-sight movement to the right. The width of each enlarged size can be calculated from the following equations (1) and (2), for example.
Difference size of previous time (left side) = k1 × | transverse component in moving direction | (1)
Later time difference size (right side) = k2 × | transverse component in moving direction | (2)
Here, k1 and k2 are arbitrary parameters.

  For convenience of explanation, FIG. 6 illustrates the case where the size adjustment is performed only in the left-right direction. However, the vertical component is used to adjust the vertical direction using the vertical component of the diagonally right movement direction instead of the left-right direction. Also good. Further, the adjustment in the vertical direction may be performed together with the adjustment in the horizontal direction. As a result, it is possible to set a gazing point image area in which the size of a portion where an error is likely to occur at each time is set in accordance with the actual line-of-sight movement direction at two times. be able to.

  Note that the gazing point image area estimation unit 24 may estimate the gazing point image area based on a predetermined size.

  Returning to FIG. 3, the search window calculation unit 26 uses the scene image region including the gazing point image region (and the gazing point) estimated by the gazing point image region estimation unit 24 for the flow calculation in the follow-up determination unit 28. Calculate as a window.

  The search window calculation unit 26 sets a range including both the gazing point image areas in each scene image calculated at two times as the search window L. FIG. 7 shows a calculation example of the search window L. For example, if the gazing point image areas at the first time and the second time are the areas shown in FIG. 7, the search window calculation unit 26 sets a sufficiently large area including both the gazing point image areas as the search window L. To do. FIG. 7 shows an example in which the search window L is a circumscribed rectangle of the gazing point image area at 2 hours. However, the present invention is not limited to this. For example, the size of the search window L may be about 1.5 times the circumscribed shape of the gazing point image area at two times. As described above, if the search window L is secured to be large, the flow calculation described later can be performed without much influence of the error.

  Moreover, the search window calculation part 26 may adjust the magnitude | size of the search window L from the gaze change direction of 2 time, for example. For example, as shown in FIGS. 8A and 8B, as in the case where the gazing point image region estimation unit 24 adjusts the gazing point image region at each time from the change in the gaze direction (see FIG. 6). Then, the search window is adjusted, and adjusted search windows L1 and L2 are created. In this case, the search window L1 is adjusted to enlarge the vicinity of the gazing point image area at the previous time (first time) on the opposite side to the moving direction, and the search window L2 is set to the later side on the same side as the moving direction. An enlarged view of the vicinity of the gazing point image area at (second time) is created.

  In addition, as the size of the search window, a size based on the size of the parallax obtained from the positional relationship between the selected camera and the line-of-sight detection device 12 may be employed. Here, if the parallax is large, a large deviation is likely to occur in the situation in which the visual target moves on the scene image at two times and the movement of the line of sight, so that a large error is likely to occur in the positioning of the image area. Therefore, by increasing the size of the search window when the parallax is large, it can be expected to improve the accuracy of flow calculation described later due to an error.

  The follow-up determination unit 28 uses the search window L (or L1, L2) calculated by the search window calculation unit 26 to search for an area that best matches the area corresponding to the gazing point image area in the two-time scene image. The flow (optical flow) that is the movement vector of is calculated.

  9 and 10 show an example of flow calculation by the search window L. FIG. As a premise, in each scene image captured at two times T1 and T2, using the value of the line-of-sight detection device 12 at each time, a gazing point image region indicating where the driver saw and a search window in each Assume that L is calculated. In FIG. 9A, the gazing point image area at time T1 is indicated by a black rectangular frame, and in FIG. 9B, the gazing point image area at time T2 is indicated by a white rectangular frame.

  At this time, the follow-up determination unit 28 in the scene image at time T1 (FIG. 9A and FIG. 10A), the gazing point image area at time T2, which is a part of the image at time T2 (FIG. 9B). ) In the range corresponding to the search window L of the scene image at the time T1 (broken line area in FIG. 10A) is searched for. Then, the movement amount from the position in the image at time T2 is calculated as a flow F1.

  Similarly, in the scene image at time T2 (FIGS. 9B and 10B), the gazing point image area at time T1, which is a part of the image at time T1 (black thick line rectangular area in FIG. 9A). ) Is searched for the most similar part in the range corresponding to the search window L of the scene image at time T2 (broken line area in FIG. 10B). Then, the movement amount from the position in the image at time T1 is calculated as a flow F2. Only one of the flows F1 and F2 may be calculated.

  Note that the flow calculation process specifies whether or not a common object is captured in the range corresponding to the search window L of the image at time T1 and the range corresponding to the search window L of the image at time T2. It can be said that there is. In addition, the process of calculating the flow F2 is performed so that an image similar to an image in a range corresponding to the gazing point at time T1 (gaze point image region) in the image at time T1 corresponds to the search frame L of the image at time T2. The processing for calculating the flow F1 is performed by determining whether an image similar to the image in the range corresponding to the gazing point at the time T2 (the gazing point image region) in the image at the time T2 is the image at the time T1. It can be said that it is specified whether it exists in the range corresponding to the search frame L.

  The follow-up determination unit 28 moves the gazing point image region pixel by pixel within the search window L when searching for a range in which the gazing point image region is most similar. Then, the follow-up determination unit 28 obtains a luminance difference between each pixel in the search window L corresponding to each pixel of the gazing point image region at each position, and the position of the gazing point image region when the sum thereof is minimized. Is the most similar part. However, if the minimum value of the sum of the luminance differences exceeds the specified value, it is considered that the flow cannot be calculated.

  FIG. 11 and FIG. 12 show examples when the flow cannot be calculated. In FIG. 11A, the gazing point image area at time T1 is indicated by a black rectangular frame, and in FIG. 11B, the gazing point image area at time T2 is indicated by a white rectangular frame. In the scene image at time T1 (FIGS. 11A and 12A), the follow-up determination unit 28 determines that the gazing point image region (white thick line rectangular region) at time T2, which is a part of the image at time T2, The most similar place is searched for in the range corresponding to the search window L of the scene image at time T1 (broken line area in FIG. 12A). The follow-up determination unit 28 also includes a gazing point image area (black thick line rectangular area) at time T1, which is a part of the image at time T1, in the scene image at time T2 (FIGS. 11B and 12B). However, the most similar place is searched in the range (broken line area in FIG. 12B) corresponding to the search window L of the scene image at time T2. In the case of FIG. 12B, since there is no portion similar to the gazing point image area at time T1 in the search window L, the flow F2 cannot be calculated.

  In the above example, the case where the search windows at two times are common has been described. However, as shown in FIGS. 8A and 8B, different search windows L1 and L2 are calculated for two times. In this case, search windows L1 and L2 at each time are used. That is, a search window L1 shown in FIG. 8A is used as a search window when searching for the same image area as the gazing point image area at the previous time from the image at the later time, and the same image as the gazing point image area at the later time is used. A search window L2 shown in FIG. 8B may be used as a search window when searching for an area from the image at the previous time.

  The follow-up determination unit 28 further determines the driver's follow-up vision based on the calculated flow information and the like. FIG. 13 is a diagram illustrating an example of tracking determination by the tracking determination unit 28.

  FIG. 13B shows two successive times (time pairs: T1 and T2, T2 and T3,... T6) when an image is taken at times T1, T2,..., T7 as shown in FIG. And T7) are images, calculation targets, and flow calculation results. In FIG. 13B, two types of flow calculation results are obtained for each pair of two times. When both have been able to calculate the flow (for example, T2 & T3, T6 & T7), the follow-up determination unit 28 sets the follow-up determination to follow-up (O). When only one of the flows can be calculated (for example, T1 & T2, T5 & T6), the follow-up determination unit 28 considers that the determination accuracy is slightly low or there is a sign of starting to change the follow-up target, and is following (Δ) Is determined. If neither of the flows can be calculated (for example, T3 & T4, T4 & T5), the follow-up determining unit 28 determines that the driver does not follow (the driver's follow-up target has changed) (×).

  In the above description, the follow-up determination unit 28 has been described using the flow calculation determination result in each time pair to determine the driver's follow-up vision. However, the present invention is not limited to this. The tracking determination unit 28 may determine the driver's tracking vision using the flow calculation results for more times and longer times.

  FIG. 14 shows an example in which tracking vision is determined using a flow calculation result twice as long as that in FIG. For example, two flow calculation results are obtained from a pair of T1 & T2, and two flow calculation results are obtained from a pair of T2 & T3, and a total of four flow calculation results are used to follow the probability based on the flow (3/4). Determine if you were watching. If the probability that the flow can be calculated is 3/4, the follow-up determination value (indicated as a calculation rate in FIG. 14) is 75%. Here, if the threshold value of the tracking determination value when assuming that tracking is in progress is set to 0.25, for example, the tracking determination value in T1 to T3 is 0.75 (> 0.25). It is determined that tracking is in progress (with tracking vision).

  Similarly, since the follow-up determination value is 0.5 at T2 to T4 and the follow-up determination value is 0.75 at T5 to T7, the follow-up determination unit 28 determines that the follow-up is in progress at each time. On the other hand, in T3 to T5 and T4 to T6, the follow-up determination values are 0 and 0.25, respectively, which are equal to or less than the determination threshold value, and therefore the follow-up determination unit 28 does not determine during the follow-up. Thus, the follow-up determination can be performed even if the time length of the flow calculation is increased. In order to round off an unexpected error (such as spike noise) of the line-of-sight detection device 12 or the like, it is better to make the follow-up determination by increasing the flow calculation time length.

  Returning to FIG. 3, the visual recognition time calculation unit 30 uses the follow-up determination result at each time determined by the follow-up determination unit 28 to determine how long it is continuously following, that is, the follow-up continuous time (the time when the visual target is visually recognized). ) Is estimated.

  The visual recognition time calculation unit 30 calculates the follow-up continuous time by regarding, for example, a time point when it is determined that the follow-up determination has not been followed even once, as a follow-up end time. In addition, the visual recognition time calculation unit 30 may regard a time point when it is determined that tracking is not continued for a plurality of times as a tracking end time. The former has an advantage that a finer follow-up break can be grasped. In the latter case, there is an advantage that it is hardly affected by errors of the line-of-sight detection device 12 and the like.

  In FIG. 13B, for example, when it is considered that the follow-up has been completed once and twice or more times, the follow-up continuous time is interrupted in the vicinity of T3 to T6. On the other hand, when it is determined that the follow-up is completed when it is determined that the follow-up is not performed three times or more, the follow-up continuous time is not interrupted from T1 to T7 in FIG.

  Note that, for example, when a follow-up target change (discontinuity) occurs, the visual recognition time calculation unit 30 may not add the entire time during the interruption as the visual recognition time. In this case, in the example of FIG. 13B, since the tracking target is changed in two pairs of T3 & T4 and T4 & T5 and two pairs of T4 & T5 and T5 & T6, T3 to T5 are excluded from the addition of the continuous time of tracking. The continuous time from ~ T2 and the continuous time from T7 are calculated.

  Alternatively, the visual recognition time calculation unit 30 indicates the later of the two times when the follow-up determination is changed from x to ○ or Δ in the follow-up determination (in the case of T5 & T6 in FIG. 13B, T6), It may be the earlier of the two times from Δ to × (T3 in the case of T3 & T4 in FIG. 13B). In this case, since only T4 to T5 are times that are not used to determine that tracking is in progress, T1 to T3 and T6 to T7 are calculated as continuous tracking times.

  In addition, the visual recognition time calculation part 30 calculates continuous visual recognition time also in the time after T7 of FIG.13 (b), and the time before T1.

  The necessary visual recognition time determination unit 32 calculates the time necessary for visually recognizing the current visual recognition target (necessary visual recognition time). This time may be uniform for all the visual targets, or the value may be changed depending on the visual targets.

  For example, it is highly possible that the object to be viewed on the side of the car is a road peripheral object, and it is usually necessary to move the line of sight from the point of view of the front of the car and grasp it. For this reason, when the visual recognition target is close to the side, it may be assumed that the visual recognition takes more time, and the required visual recognition time may be set longer than when the visual recognition target is in front of the vehicle. Whether or not the current visual recognition target is on the side can be determined, for example, based on whether or not the angle formed by the line-of-sight direction of the line-of-sight data and the front of the vehicle is large.

  Furthermore, the necessary visual recognition time determination unit 32 may set the necessary visual recognition time to a longer time, for example, when the magnitude of pixel change in the arbitrary image region of the gazing point image region and its neighboring peripheral images is large. In this case, for example, a pixel luminance dispersion value in the gazing point image area can be used as the pixel change. When there are various objects around the visual recognition target, it is considered that the change of the pixel is large, and it is difficult to grasp the visual recognition target because there are more objects. For this reason, a more reliable visual estimation estimation can be performed by lengthening the time required for visual recognition. In addition, this pixel change may use the pixel change of both time of 2 time, and may use the pixel change of either one time.

  Furthermore, the necessary visual recognition time determination unit 32 varies the necessary visual recognition target time according to the weather at each time acquired from the in-vehicle sensor group 14, the current position of the car, and the surrounding situation that can be estimated from the map DB 42. Also good. For example, it is difficult to see in rainy weather or when the illumination is low, it is difficult to see places where there are many objects such as downtown, and side objects (roadside objects) are difficult to see while the car is moving at high speed. The necessary viewing time may be set in consideration of the above.

  The visual recognition determination unit 34 has been able to reliably visually recognize the target based on the continuous follow-up time calculated for each visual target by the visual recognition time calculation unit 30 and the necessary visual recognition time for each target calculated by the necessary visual recognition time determination unit 32. Determine if. That is, when it is a continuous follow-up time that is longer than a certain time, it is determined that the object has been visually recognized.

  Note that the visual recognition determination unit 34 may determine that the visual recognition is performed even if the continuous follow-up time is slightly shorter in consideration of some time estimation error, instead of strictly comparing the time. The visual recognition determination unit 34 may make a more precise determination using not only the time but also the detailed contents of the visual line data at that time. For example, the ratio of data in which the movement speed of the line-of-sight data group measured between two times occupies a threshold value or more is calculated, and if the ratio is large, the determination time may be longer. As shown in FIG. 15, for example, the line-of-sight data is classified into three lines of sight, saccade, smooth pursuit, and fixation according to the magnitude of the moving speed. Here, the line of sight with the highest movement speed is the line of saccade by eye movement that causes ultra-high-speed movement (jump). Further, the smooth pursuit occurs only when there is a visually recognizing object that moves at a low speed and causes a slow low-speed line-of-sight movement. Furthermore, strictly speaking, the fixation is performed with very high-speed fine vibrations but roughly with a substantially fixed line of sight. When the movement speed of the line-of-sight data is high, there is a high possibility that the saccade is difficult to recognize the object being viewed. Accordingly, the visual recognition determination unit 34 can determine the visual recognition state in consideration of whether the visual line data is actually recognized by taking a longer determination time when there is a large amount of visual line data belonging to the saccade.

  The side-view determination unit 36 uses the presence / absence of visual recognition for each visual target determined by the visual recognition determination unit 34 to determine whether a side-view is being performed. For example, when the visual recognition target existing on the side of the car is visually recognized but the visual recognition target existing in front of the vehicle is not visually recognized for a long time, the side-view determination unit 36 has a high possibility of a side-view occurrence. Judge.

  In addition to this, the image features of the object to be visually recognized are stored in advance in the object image feature DB 44, and the side-view determination unit 36 determines that it is a side-view when a long-term view of a different visual target occurs. Also good. For example, in an environment where a visual target such as an expressway can be limited in advance, a rough visual target such as a sign or a preceding vehicle can be prepared in advance. By doing in this way, it is possible to more accurately determine the side effect.

  Next, an example of processing by the camera selection unit 22, the gazing point image region estimation unit 24, the search window calculation unit 26, the follow-up determination unit 28, and the necessary visual recognition time determination unit 32 will be described with reference to the flowcharts of FIGS. To do. The details of the processing of each unit already described will be omitted.

  In step S10 of FIG. 16, the camera selection unit 22 records the current time as the determination start time in the HDD 96 or the like. Next, in step S12, the camera selection unit 22 determines whether or not (current time−determination start time) is equal to or greater than the prescribed time M of 2 hours. If the determination is negative, the process proceeds to step S14, and the camera selection unit 22 updates the current time, and then returns to step S12. On the other hand, if the determination in step S12 is affirmative, the process proceeds to step S16.

  In step S16, the camera selection unit 22 sets the number of determinations to zero. Next, in step S18, the camera selection unit 22 calculates the required number of viewing times as the specified initial value T and calculates the number of stored calculation results N = T / M at two times.

  Next, in step S20, the camera selection unit 22 determines time A (current time-specified time M) and time B (current time) as two times, and increments the determination count by one. Next, in step S <b> 22, the camera selection unit 22 acquires images captured by the camera 10 at times A and B, line-of-sight data, and data on the in-vehicle sensor group 14.

  Next, in step S24, the camera selection unit 22 determines whether the line-of-sight direction change is equal to or greater than the threshold value or whether the camera is not selected. If the determination here is affirmed, the process proceeds to step S26, and the camera selection unit 22 selects one camera from the plurality of cameras, and proceeds to step S28 in FIG. On the other hand, if the determination in step S24 is negative, the process proceeds to step S28 in FIG. 17 without passing through step S26.

  When the process proceeds to step S28 in FIG. 17, the gazing point image region estimation unit 24 identifies the gazing point from the positional relationship between the line-of-sight detection device 12 and the selected camera 10 (see FIG. 5A). Next, in step S30, the gazing point image area estimation unit 24 determines the size of the gazing point image area. Next, in step S32, the gazing point image area estimation unit 24 calculates a gazing point image area (see FIGS. 5B and 6).

  Next, in step S34, the search window calculation unit 26 calculates a search window (see FIG. 7). Next, in step S36, the search window calculation unit 26 adjusts the search window (see FIGS. 8A and 8B). Next, in step S38, the follow-up determination unit 28 calculates a flow from the image taken at time A (see FIGS. 10A and 12A). Next, in step S40, the follow-up determination unit 28 calculates a flow using an image taken at time B (see FIGS. 10B and 12B). Next, in step S42, the follow-up determination unit 28 stores the flow calculation result in the HDD 96 or the like. Thereafter, the process proceeds to step S44 in FIG.

  When the process proceeds to step S44 in FIG. 18, the necessary visual recognition time determination unit 32 determines whether or not the necessary visual recognition time is the prescribed initial value T. If the determination in step S44 is affirmative, the process proceeds to step S46, and the necessary visual recognition time determination unit 32 determines whether or not the luminance variance in the gazing point image area at time B is equal to or greater than the threshold value. If the determination in step S46 is negative, the process proceeds to step S48, and the required visual recognition time determination unit 32 determines whether high-speed traveling, rainy weather, and downtown driving can be detected at time B. If the determination in step S48 is negative, the process proceeds to step S50, and the necessary visual recognition time determination unit 32 determines whether or not the ratio of the saccade line of sight to the data of the line-of-sight sensor group during time AB is greater than or equal to the threshold value. to decide. If the determination in step S50 is negative, the process proceeds to step S54. On the other hand, if the determination in step S44 is negative, or if the determinations in steps S46, S48, and S50 are affirmative, in step S52, the required visual recognition time determination unit 32 corrects the required visual recognition time to be longer and corrects it. The number of stored calculation results N at the second time is corrected from the necessary necessary viewing time T ′ / specified time M later, and is set to N ′. Thereafter, the process proceeds to step S54.

  In step S54, the follow-up determination unit 28 determines whether or not the current determination count has exceeded the current two-time calculation result storage number N (N ′). If the determination in step S54 is negative, the process returns to step S20 in FIG. 16, but if the determination is positive, the process proceeds to step S56.

  When the process proceeds to step S56, the follow-up determination unit 28 obtains a calculation rate of (number of flow calculations / total number of times) for all two saved times (FIG. 14).

  Next, in step S58, the follow-up determination unit 28 determines whether or not the calculated number exceeds a threshold value. If the determination in step S58 is affirmative, the tracking determination unit 28 proceeds to step S60, determines that tracking vision is continuing between times AB, and returns to step S16 in FIG. On the other hand, if the determination in step S58 is negative, in step S62, the tracking determination unit 28 determines that the tracking vision has ended between times AB. Thereafter, the process returns to step S16 in FIG.

  Note that the visual recognition time calculation unit 30, the visual recognition determination unit 34, and the side-view determination unit 36 perform the above-described processing using information obtained as a result of the processing in FIGS.

  As described above in detail, according to the present embodiment, the gazing point image area estimation unit 24 is information of the gazing point that the driver gazes at each of time A (first time) and time B (second time). (Gaze point and gaze point image area) are acquired. Then, the search window calculation unit 26 specifies a predetermined range (search window L) including the watch point at time A and the watch point at time B, and the follow-up determination unit 28 sets the range within the search window L of the image at time A. Whether or not the driver's line of sight is following the visual recognition target is determined based on whether or not a common target is captured in the search window L of the image at time B. Accordingly, for example, as shown in FIG. 19, there is a parallax between the driver viewpoint and the camera, and the change vector of the line-of-sight direction when the visual target moves and the movement vector of the visual target on the image are almost opposite to each other. Even in such a case, it can be accurately determined whether or not the driver's line of sight follows the target. Further, according to the present embodiment, even when the driver is following an object other than a roadside stationary object, it is possible to accurately determine the presence or absence of the driver's follow-up while considering the movement of the object.

  In this embodiment, the follow-up determination unit 28 searches for an image similar to the image in the range corresponding to the point of interest at time A in the image at time A (first time) at the time B (second time). Based on whether it exists in the range of the window L and whether an image similar to the image in the range corresponding to the gazing point at the time B in the image at the time B exists in the range of the search window L of the image at the time A Then, it is determined whether the driver's line of sight follows the object to be viewed. As a result, it is possible to accurately determine whether the driver's line of sight follows the visual target without being affected by the error of the gazing point image area or the search window L, as compared with the case where the determination is based on either one. Can do.

  In this embodiment, the optimal camera is selected in consideration of the parallax between a plurality of cameras mounted on the car and the driver viewpoint, so that the driver's line of sight also follows the visual target from this point. Can be accurately determined. Further, by selecting an optimal camera, it is possible to suppress a decrease in determination accuracy due to the blind spot of the camera.

  The above processing functions can be realized by a computer. In that case, a program describing the processing contents of the functions that the processing apparatus should have is provided. By executing the program on a computer, the above processing functions are realized on the computer. The program describing the processing contents can be recorded on a computer-readable recording medium (except for a carrier wave).

  When the program is distributed, for example, it is sold in the form of a portable recording medium such as a DVD (Digital Versatile Disc) or a CD-ROM (Compact Disc Read Only Memory) on which the program is recorded. It is also possible to store the program in a storage device of a server computer and transfer the program from the server computer to another computer via a network.

  The computer that executes the program stores, for example, the program recorded on the portable recording medium or the program transferred from the server computer in its own storage device. Then, the computer reads the program from its own storage device and executes processing according to the program. The computer can also read the program directly from the portable recording medium and execute processing according to the program. Further, each time the program is transferred from the server computer, the computer can sequentially execute processing according to the received program.

  The above-described embodiment is an example of a preferred embodiment of the present invention. However, the present invention is not limited to this, and various modifications can be made without departing from the scope of the present invention.

In addition, the following additional remarks are disclosed regarding description of the above embodiment.
(Additional remark 1) The acquisition part which acquires the information of the gaze point which a user gazes in each of 1st time and 2nd time which passed predetermined time from this 1st time,
A specifying unit for specifying a predetermined range including the gazing point at the first time and the gazing point at the second time;
The user's line of sight based on whether or not a common object is captured in a range corresponding to the predetermined range of the image at the first time and a range corresponding to the predetermined range of the image at the second time. A line-of-sight state determination apparatus comprising: a determination unit that determines the state of
(Supplementary Note 2) In the determination unit, an image similar to an image in a range corresponding to the gazing point at the first time in the image at the first time corresponds to the predetermined range of the image at the second time. An image that exists in the range and / or is similar to the image in the range corresponding to the gaze point at the second time in the image at the second time corresponds to the predetermined range of the image at the first time The line-of-sight state determination apparatus according to appendix 1, wherein the line-of-sight state of the user is determined based on whether the line of sight exists within a range to be displayed.
(Additional remark 3) The said determination part determines whether the user's gaze is following the said object, The gaze state determination apparatus of Additional remark 1 or 2 characterized by the above-mentioned.
(Supplementary Note 4) Acquiring information on a gazing point that the user gazes at each of a first time and a second time after a predetermined time has elapsed from the first time,
A predetermined range including the gazing point at the first time and the gazing point at the second time,
The user's line of sight based on whether there is a common target in the range corresponding to the predetermined range of the image at the first time and the range corresponding to the predetermined range of the image at the second time. Determine the status of
A line-of-sight determination program that causes a computer to execute processing.
(Supplementary Note 5) In the determination process, an image similar to an image in a range corresponding to the gazing point at the first time in the image at the first time corresponds to the predetermined range of the image at the second time. And / or an image similar to the image in the range corresponding to the gaze point at the second time in the image at the second time is in the predetermined range of the image at the first time. The visual line state determination program according to appendix 4, wherein the visual line state of the user is determined based on whether the line of sight exists in a corresponding range.
(Additional remark 6) The said determination process determines whether the user's visual line is following the said object, The visual line state determination program of Additional remark 4 or 5 characterized by the above-mentioned.
(Appendix 7)
Obtaining information of a gaze point that the user gazes at each of a first time and a second time after a predetermined time has elapsed from the first time;
A predetermined range including the gazing point at the first time and the gazing point at the second time,
The user's line of sight based on whether there is a common target in the range corresponding to the predetermined range of the image at the first time and the range corresponding to the predetermined range of the image at the second time. Determine the status of
A gaze state determination method in which processing is executed by a computer.
(Supplementary Note 8) In the determination process, an image similar to an image in a range corresponding to the gazing point at the first time in the image at the first time corresponds to the predetermined range of the image at the second time. And / or an image similar to the image in the range corresponding to the gaze point at the second time in the image at the second time is in the predetermined range of the image at the first time. The gaze state determination method according to appendix 7, wherein the gaze state of the user is determined based on whether or not it exists in a corresponding range.
(Additional remark 9) The said determination process determines whether the user's visual line is following the said object, The gaze state determination method of Additional remark 7 or 8 characterized by the above-mentioned.

20 Information processing device (Gaze state determination device)
24 Gaze point image area estimation unit (acquisition unit)
26 Search window calculation part (specific part)
28 Tracking determination unit (determination unit)

Claims (5)

An acquisition unit that acquires information of a gaze point that the user gazes at each of a first time and a second time after a predetermined time has elapsed from the first time;
A specifying unit for specifying a predetermined range including the gazing point at the first time and the gazing point at the second time;
The user's line of sight based on whether or not a common object is captured in a range corresponding to the predetermined range of the image at the first time and a range corresponding to the predetermined range of the image at the second time. A line-of-sight state determination apparatus comprising: a determination unit that determines the state of   The determination unit includes an image similar to an image in a range corresponding to the gazing point at the first time in the image at the first time in a range corresponding to the predetermined range of the image at the second time. And / or an image similar to the image in the range corresponding to the gaze point at the second time in the image at the second time exists in the range corresponding to the predetermined range of the image at the first time The gaze state determination apparatus according to claim 1, wherein the gaze state of the user is determined based on whether or not   The gaze state determination apparatus according to claim 1, wherein the determination unit determines whether a user's line of sight is following the target. Obtaining information of a gaze point that the user gazes at each of a first time and a second time after a predetermined time has elapsed from the first time;
A predetermined range including the gazing point at the first time and the gazing point at the second time,
The user's line of sight based on whether there is a common target in the range corresponding to the predetermined range of the image at the first time and the range corresponding to the predetermined range of the image at the second time. Determine the status of
A line-of-sight determination program that causes a computer to execute processing. Obtaining information of a gaze point that the user gazes at each of a first time and a second time after a predetermined time has elapsed from the first time;
A predetermined range including the gazing point at the first time and the gazing point at the second time,
The user's line of sight based on whether there is a common target in the range corresponding to the predetermined range of the image at the first time and the range corresponding to the predetermined range of the image at the second time. Determine the status of
A gaze state determination method in which processing is executed by a computer.

Images