JP7570164B2 - DISPLAY CONTROL DEVICE, DISPLAY CONTROL SYSTEM, AND DISPLAY CONTROL METHOD - Google Patents

Description

This disclosure relates to a display control device, a display control system, and a display control method.

Patent Document 1 discloses a display control device that selects the display device that the driver is looking at from among multiple display devices installed in a vehicle, and displays information necessary for driving on the screen of the selected display device.

This display control device includes a gaze detection means for detecting the direction in which the driver is looking based on the facial image of the driver captured by the image capture unit, and a selection means for selecting, from among a plurality of display devices, the display device at which the driver is looking based on gaze information indicating the direction in which the gaze is directed, as the display device for displaying information necessary for driving.

With this display control device, the information necessary for driving is displayed on the display device that the driver is looking at, so the driver can check the information necessary for driving while understanding the situation around the vehicle where the driver is looking.

JP 2007-249567 A

However, in the conventional technology disclosed in Patent Document 1, specific display information continues to be displayed on the display device where the driver is looking until a certain period of time has elapsed, which creates the problem that display information with a high degree of urgency necessary for safe driving cannot be displayed until the certain period of time has elapsed.

Non-limiting examples of the present disclosure contribute to providing a display control device, a display control system, and a display control method that display information of high urgency even when specific display information is being displayed on a display device where the driver is looking.

A display control device according to an embodiment of the present disclosure includes a gaze detection unit that detects a direction in which a driver of the vehicle is looking, based on data of an image captured of the driver of the vehicle; a selection unit that selects, based on gaze information indicating the direction in which the driver is looking, from among a plurality of display devices provided in the vehicle, a display device that is in the direction in which the driver is looking, as a first display device that displays display information to be provided to the driver; a determination unit that determines, based on at least one of information regarding the surrounding conditions of the vehicle and information regarding a state of the vehicle, a degree of urgency in providing the display information to the driver; and a determination unit that determines that the degree of urgency in providing the display information to the driver is higher than the degree of urgency in providing the first display information to the driver. and a display control unit that interrupts and displays the determined second display information on the first display device on which the first display information is being displayed, wherein the determination unit further determines a first degree of visibility when the second display information is displayed on the first display device and a second degree of visibility when the second display information is displayed on a second display device other than the first display device, based on information regarding the surrounding conditions of the vehicle, and the display control unit displays the second display information on the first display device if the first degree is higher than the second degree, and displays the second display information on the second display device if the first degree is lower than the second degree, and displays guidance information on the first display device to inform the user that the second display information has been displayed on the second display device .

A display control system according to one embodiment of the present disclosure includes the above-described display control device and a plurality of the display devices.

A display control method according to an embodiment of the present disclosure includes the steps of receiving image data of a vehicle driver, detecting a direction of the driver's gaze based on the image data, selecting a display device in the direction of the driver's gaze from among a plurality of display devices provided in the vehicle as a first display device for displaying display information to be provided to the driver based on gaze information indicating the direction of the driver's gaze, determining a degree of urgency of providing the display information to the driver based on at least one of information regarding the surrounding conditions of the vehicle and information regarding a state of the vehicle, displaying the first display information on the first display device, and determining whether the degree of urgency of providing the display information to the driver is greater than the degree of urgency of providing the first display information to the driver. and a step of interrupting and displaying second display information determined to have a degree of urgency higher than the degree of urgency on the first display device on which the first display information is being displayed , wherein the determining step further determines a first degree of visibility when the second display information is displayed on the first display device and a second degree of visibility when the second display information is displayed on a second display device other than the first display device, based on information regarding the surrounding conditions of the vehicle, and the step of interrupting and displaying on the first display device includes displaying the second display information on the first display device if the first degree is higher than the second degree, and displaying the second display information on the second display device if the first degree is lower than the second degree, and displaying guidance information on the first display device informing users that the second display information has been displayed on the second display device .

According to one embodiment of the present disclosure, it is possible to provide a display control device, a display control system, and a display control method that display information of high urgency even when specific display information is being displayed on a display device to which a driver is directing his or her gaze.

Further advantages and benefits of an embodiment of the present disclosure will become apparent from the specification and drawings. Such advantages and/or benefits may be provided by some of the embodiments and features described in the specification and drawings, respectively, but not necessarily all of them need be provided to obtain one or more identical features.

FIG. 1 is a diagram showing an example of the configuration of a display control system 100 according to an embodiment of the present disclosure. 1 is a flowchart for explaining an operation of the display control system 100 according to an embodiment of the present disclosure. Flowchart for explaining display control processing FIG. 1 is a diagram showing an example of a hardware configuration of a display control system 100 according to an embodiment of the present disclosure.

A preferred embodiment of the present disclosure will be described in detail below with reference to the attached drawings. In this specification and drawings, components having substantially the same functions are designated by the same reference numerals to avoid redundant description.

(Embodiment)
First, the background that led to the creation of the embodiments of the present disclosure will be described.

For example, in the known technology described in Patent Document 1, when gaze information indicating the direction in which the driver is looking, detected based on an image of the driver's face captured by a camera unit, is output, the information necessary for driving is displayed on the display device toward which the driver is looking out of multiple display devices.

In such known technologies, specific display information continues to be displayed on the display device where the driver is looking until a certain period of time has elapsed, so display information with a high degree of urgency necessary for safe driving cannot be displayed until the certain period of time has elapsed.

For this reason, even if specific display information is being displayed on the display device to which the driver is looking, it is desirable to display information with a high degree of urgency. Below, an embodiment of the present disclosure will be described.

Figure 1 is a diagram showing an example configuration of a display control system 100 according to an embodiment of the present disclosure.

The vehicle 200 shown in FIG. 1 includes a display control system 100, a gaze detection device 10 that sequentially detects the gaze of the driver of the vehicle 200, a surroundings monitoring device 20 that monitors objects present around the vehicle, and an information collection device 60.

Vehicle 200 may be a passenger car, a freight vehicle, a passenger van, etc.

(Display control system 100)
The display control system 100 includes a plurality of display devices 30A to 30C and a display control device 50. Hereinafter, when the plurality of display devices 30A to 30C are not distinguished from one another, they will be referred to as "display device 30."

(Gaze Detection Device 10)
The gaze detection device 10 is a device that detects the gaze direction of the driver of the vehicle 200 and outputs gaze information that indicates the direction in which the driver's gaze is directed.

The gaze detection device 10 includes an image capture unit 11, an image processing unit 12, a gaze detection unit 13, and a gaze information output unit 14.

(Image capturing unit 11)
The image capturing unit 11 is a camera that captures images of the inside of the vehicle, etc. The image capturing unit 11 includes an image capturing element such as a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS).

The image capturing unit 11 is provided in a position where it can capture an image of the driver's face, captures an image including the driver's face, and sequentially outputs data of the captured image (hereinafter referred to as image data) to the image processing unit 12.

Positions where the driver's face can be photographed include, for example, the dashboard of the vehicle 200, the instrument panel in the driver's seat, the ceiling of the vehicle 200, etc.

The number of image capture units 11 is not limited to one, and multiple units may be installed in the vehicle.

(Image processing unit 12)
When the image data is input, the image processing unit 12 obtains a binary image by binarizing the image captured by the image capturing unit 11. The image processing unit 12 inputs the binary image data (hereinafter referred to as binary image data) to the gaze detection unit 13.

(Gaze detection unit 13)
The gaze detection unit 13 detects the gaze based on the binary image data. For example, the gaze detection unit 13 identifies the eyeball area of the driver by analyzing the binary image data, and obtains the center coordinates of the eyeball from the curvature of the eyeball shape.

In addition, the gaze detection unit 13 identifies the pupil area from the eyeball area to obtain the central coordinates of the iris, and determines the direction from the central coordinates of the eyeball to the central coordinates of the iris as the gaze direction.

Furthermore, the gaze detection unit 13 identifies a line extending from the starting point, which is the center position of the eyeball, in the gaze direction as the gaze, and transmits gaze information indicating the identified gaze to the gaze information output unit 14.

(Gaze Information Output Unit 14)
The line-of-sight information output unit 14 outputs the line-of-sight information transmitted from the line-of-sight detection unit 13 to an information input unit 51 of the display control device 50. The configuration of the information input unit 51 will be described in detail later.

(Periphery monitoring device 20)
The surroundings monitoring device 20 is a device for detecting objects existing around the vehicle.

The surroundings monitoring device 20 includes a surroundings image capturing unit 21, an image processing unit 22, an object detection sensor 23, an object detection unit 25, and a detection information output unit 26.

(Peripheral image capturing unit 21)
The surrounding image capturing unit 21 is an omnidirectional camera, a panoramic camera, or the like that captures images of the surroundings of the vehicle. Hereinafter, the images of the surroundings of the vehicle will be referred to as "surrounding images." The surrounding image capturing unit 21, like the image capturing unit 11 of the gaze detection device 10, includes an image sensor such as a CCD or a CMOS.

The surrounding images are, for example, images of the scenery in front of the vehicle 200, the scenery to the side of the vehicle 200, the scenery behind the vehicle 200, etc. The scenery includes, for example, objects present on the roadway on which the vehicle 200 is traveling. The objects present on the roadway include, for example, automobiles, buildings, structures (advertisements, road signs, traffic lights, telegraph poles, etc.), people, animals, etc.

The surrounding image capturing unit 21 is disposed in a position where it can capture, for example, the scenery outside the vehicle. Such positions include the front grill, side mirrors, ceiling, rear bumper, etc. The surrounding image capturing unit 21 inputs data of the captured surrounding images to the image processing unit 22. Hereinafter, the data of the surrounding images is referred to as "surrounding image data."

(Image processing unit 22)
The image processing unit 22 performs image processing based on the surrounding image data to detect a preset object to be detected from among objects present around the vehicle 200. Hereinafter, the preset object to be detected will be referred to as a "target object."

The object may be, for example, a car, a motorbike, a bus, a taxi, a bicycle, a pedestrian, an animal, etc. The image processing may be, for example, edge detection processing. The image processing unit 22 inputs the image of the object to the object detection unit 25.

(Object detection sensor 23)
The object detection sensor 23 is a sensor for detecting the position of an object present around the vehicle, and is, for example, a sonar.

For example, a sonar emits a pulsed sound wave of a certain frequency. When the pulsed sound wave hits an object around the vehicle, it is reflected, and some of the pulsed sound wave returns to the sonar. The longer the distance from the vehicle to the object, the longer it takes for the pulsed sound wave to return to the sonar.

Using this relationship, the sonar estimates the distance to an object by measuring the time it takes for the echo to return to the sonar after being reflected by the object after emitting a pulsed sound wave, and then detects the object's position relative to the vehicle based on the estimated distance.

In addition, the object detection sensor 23 inputs object position information, which is information indicating the position of the detected object, to the target object detection unit 25.

(Object detection unit 25)
The object detection unit 25 performs, for example, the following determination based on the image of the object and the object position information.

(1) The object detection unit 25 determines that there is no risk of collision, for example, when the rear end of the preceding vehicle to be detected is located in front of the vehicle 200 and the distance from the rear end to the front end of the vehicle 200 is equal to or greater than a predetermined value (e.g., several meters to several tens of meters) that changes depending on the relative speed of the preceding vehicle with respect to the vehicle 200.

(2) The object detection unit 25 determines that there is a risk of collision, for example, when the rear end of the preceding vehicle to be detected is located in front of the vehicle 200 and the distance from the rear end to the front end of the vehicle 200 is less than the above-mentioned predetermined value.

In this way, the object detection unit 25 detects an object that may collide with the vehicle, generates object detection information that indicates that an object that may collide with the vehicle has been detected, and transmits this information to the detection information output unit 26.

(Detection information output unit 26)
The detection information output unit 26 outputs the object detection information transmitted from the object detection unit 25 to an information input unit 54 of the display control device 50. The configuration of the information input unit 54 will be described in detail later.

(Information collection device 60)
The information collecting device 60 collects information about the surrounding conditions of the vehicle 200 and information about the state of the vehicle 200 , and transmits the information to the information input unit 54 and the determination unit 53 .

The information collection device 60 includes, for example, a surrounding monitoring unit, a surrounding illuminance measurement unit, a fault detection unit, an operation status detection unit, an in-vehicle device status detection unit, and a driving-related information receiving unit.

(Periphery Surveillance Division)
The periphery monitoring unit is a device for monitoring the periphery of the vehicle 200 and detecting objects around the vehicle 200. The periphery monitoring unit is, for example, a camera, a 3D-LiDAR (Light Detection And Ranging), a millimeter wave radar, an ultrasonic sensor, or the like.

When a camera is used as the periphery monitoring unit, the periphery monitoring unit transmits images of objects present around the vehicle to the determination unit 53 as information on the surrounding conditions of the vehicle 200. The images of objects present around the vehicle include images of the lane to which the vehicle 200 will change lanes when it changes lanes.

When 3D-LiDAR, millimeter wave radar, ultrasonic sensors, etc. are used as the perimeter monitoring unit, the perimeter monitoring unit transmits the positions of objects present around the vehicle to the determination unit 53 as information about the surrounding conditions of the vehicle 200.

(Ambient illuminance measurement unit)
The surrounding illuminance measuring unit is a device for measuring the illuminance around the vehicle. The surrounding illuminance measuring unit is, for example, an illuminance sensor. The surrounding illuminance measuring unit transmits the measured illuminance to the determination unit 53 as information on the surrounding situation of the vehicle 200.

(Fault detection section)
The failure detection unit is a device for detecting failures of devices mounted on the vehicle, and includes an accelerator failure determination device, a brake failure determination device, a drive shaft failure determination device, an engine failure determination device, a fuel system failure determination device, a steering failure determination device, an air pressure detection device, etc.

The accelerator failure determination device diagnoses failures in the mechanism that transmits the amount of accelerator pedal operation, and transmits the diagnosis results to the notification control device 15 as information regarding the state of the vehicle 200.

The brake failure determination device diagnoses failures in the mechanism that transmits the amount of brake pedal operation, and transmits the diagnosis results to the notification control device 15 as information regarding the state of the vehicle 200.

The drive shaft failure determination device diagnoses the drive shaft failure and transmits the diagnosis result to the notification control device 15 as information about the condition of the vehicle 200.

The engine failure determination device diagnoses engine failures and transmits the diagnosis results to the notification control device 15 as information about the condition of the vehicle 200.

The fuel system failure determination device diagnoses failures in the fuel supply system and transmits the diagnosis results to the notification control device 15 as information about the state of the vehicle 200.

The steering failure determination device diagnoses failures in the mechanism that transmits steering force, and transmits the diagnosis results to the notification control device 15 as information about the state of the vehicle 200.

The air pressure detection device detects the air pressure in the tires and transmits the detected air pressure to the notification control device 15 as information about the state of the vehicle 200.

(Operation state detection unit)
The operation state detection unit is a device for detecting an operation state of the driver of the vehicle 200. The operation state detection unit includes, for example, a steering angle sensor, an accelerator pedal sensor, a brake pedal sensor, a shift sensor, and the like.

The steering angle sensor detects the steering angle of the steering wheel of the vehicle 200 and transmits the detected steering angle to the determination unit 53 as information regarding the state of the vehicle 200.

The accelerator pedal sensor detects the amount of depression of the accelerator pedal and transmits the detected amount of depression of the accelerator pedal to the determination unit 53 as information about the state of the vehicle 200.

The brake pedal sensor detects the amount of depression of the brake pedal and transmits the detected amount of depression of the brake pedal to the determination unit 53 as information about the state of the vehicle 200.

The shift sensor detects the operating position of the shift lever and transmits the detected operating position to the determination unit 53 as information about the state of the vehicle 200.

(In-vehicle device status detection unit)
The vehicle-mounted device status detection unit is a device for detecting the status of the vehicle-mounted device, and includes, for example, a sensor for detecting that an ETC (Electronic Toll Collection System) card is not inserted in an ETC device, and a sensor for detecting that the ETC vehicle-mounted device is in an unusable state due to a malfunction or the like.

(Driving-related information receiving unit)
The driving-related information receiving unit is a device for receiving information related to driving and providing the received information, etc., to the driver. Examples of the driving-related information include the following.
(1) Guidance information indicating that the vehicle 200 has passed through an ETC gate.
(2) Warning information defined as downlink information in ETC 2.0 (registered trademark).
(3) Priority information transmitted from an optical beacon.
(4) Emergency information contained in FM (Frequency Modulation) multiplex broadcasting.
(5) Information regarding traffic accidents or disasters.
(6) Information alerting the driver that a vehicle traveling in the wrong direction is present on the road on which the vehicle 200 is traveling.
(7) Fuel remaining information indicating that the remaining amount of fuel in the vehicle 200 is less than a predetermined value.
(8) Status information indicating that the driving condition of the vehicle 200 is less safe than a reference driving condition.
(9) Information alerting the driver to avoid collision of the vehicle 200 with objects present around the vehicle 200.
(10) Information indicating that another vehicle around vehicle 200 has cut in to the lane in which vehicle 200 is traveling.

(Display control device 50)
The display control device 50 selects, from among the multiple display devices 30 provided on the vehicle 200, a display device 30 present in the direction of the line of sight as a first display device that displays display information to be provided to the driver, based on the line of sight information, the object detection information, information on the surrounding conditions of the vehicle 200, and information on the state of the vehicle 200. The display control device 50 causes the selected first display device to display the display information.

The display information includes first display information and second display information.

(Example of first display information)
The first display information is display information that needs to be provided to the driver with a lower degree of urgency than the second display information.

Examples of the first display information include the following information:
(1) The map displayed on the navigation screen.
(2) Images of operating audio equipment.
(3) Image of a speedometer.
(4) Image of oil pressure gauge.
(5) Image of fuel level gauge.
(6) Image of a tachometer.
(7) An image showing the vehicle's direction of travel.

The first display information can be reproduced on the display device 30, for example, by the first display information creation unit 58 referring to the correspondence information stored in the storage unit 52. Details of the first display information creation unit 58 will be described later.

(Example of second display information)
The second display information is display information that needs to be provided to the driver with a higher degree of urgency than the first display information.

Examples of the second display information include the following information:
(1) Guidance indicating that the ETC card is not inserted into the ETC onboard unit.
(2) Guidance indicating that the ETC onboard unit is unavailable.
(3) Guidance indicating that the ETC card has been inserted.
(4) Guidance on fees when passing through an ETC gate.
(5) Warning information defined as downlink information in ETC 2.0.
(6) Priority information transmitted from an optical beacon.
(7) Emergency information contained in FM multiplex broadcasting.
(8) Information regarding traffic accidents or disasters.
(9) Information alerting the driver that a vehicle traveling in the wrong direction is present on the road on which the vehicle 200 is traveling.
(10) State information indicating that the driving state of the vehicle 200 is a state in which safety is lower than a reference driving state. Note that specific examples of states in which safety is lower than the reference driving state will be described later.
(11) Information alerting the driver to avoid collision of the vehicle 200 with objects present around the vehicle 200.
(12) An image of the lane to which the vehicle 200 is changing lanes.
(13) Information indicating that another vehicle around vehicle 200 has cut in to the lane in which vehicle 200 is traveling.
(14) Fuel level information indicating that the amount of fuel remaining is less than a predetermined value.

The second display information can be reproduced on the display device 30, for example, by the second display information creation unit 55 referring to the correspondence information stored in the storage unit 52. Details of the second display information creation unit 55 will be described later.

The display control device 50 includes an information input unit 51, a memory unit 52, a determination unit 53, an information input unit 54, a second display information creation unit 55, a display control unit 56, a selection unit 57, and a first display information creation unit 58.

(Information input unit 51)
The information input unit 51 receives the line-of-sight information transmitted from the line-of-sight information output unit 14 and transfers it to a selection unit 57 .

(Memory unit 52)
The storage unit 52 stores, for example, coordinate data, an identification ID (Identifier) of the display device 30, the above-mentioned correspondence information, etc. The coordinate data is data that represents the position of the display surface of each of the multiple display devices 30 within the vehicle.

(Selection unit 57)
For example, when gaze information is output from the gaze information output unit 14 at a predetermined period, the selection unit 57 selects, from among the multiple display devices 30, the display device 30 that is located in the direction of the gaze as the first display device for displaying the above-mentioned first display information or second display information.

Specifically, the selection unit 57 reads out the coordinate data stored in the storage unit 52, and identifies the display device 30 that is in the driver's line of sight, with the front of the vehicle meter panel as the reference position.

Then, the selection unit 57 selects the identified display device 30 as the display device on which to display the first display information or the second display information.

Then, the selection unit 57 reads out the identification ID corresponding to the selected display device 30, for example from the storage unit 52, and transmits it to the display control unit 56. The configuration of the display control unit 56 will be described in detail later.

(Information input unit 54)
The information input unit 54 inputs the object detection information output by the detection information output unit 26, information regarding the surrounding conditions of the vehicle 200 transmitted from the information collection device 60, and information regarding the condition of the vehicle 200 transmitted from the information collection device 60, and transfers them to the first display information creation unit 58 and the second display information creation unit 55.

(First display information generating unit 58)
When the first display information creation unit 58 receives information regarding the status of the vehicle 200 transmitted from the information input unit 54, it reads out data of the first display information corresponding to the status of the vehicle 200 by referring to the correspondence information stored in the memory unit 52, and transmits the data to the display control unit 56.

(Second display information generating unit 55)
When the second display information creation unit 55 receives the object detection information, the information on the surrounding conditions of the vehicle 200, and the information on the state of the vehicle 200 transmitted from the information input unit 54, the second display information creation unit 55 refers to the correspondence information stored in the storage unit 52. Then, the second display information creation unit 55 reads out data of the second display information corresponding to these pieces of information and transmits it to the display control unit 56.

(Determination unit 53)
The determination unit 53 determines the degree of urgency of providing the display information to the driver based on at least one of information related to the surrounding conditions of the vehicle 200 and information related to the state of the vehicle 200.

When the determination unit 53 determines that the degree of urgency of providing the second display information to the driver is higher than the degree of urgency of providing the first display information to the driver, the determination unit 53 transmits an interrupt request signal to the display control unit 56. For example, when a situation occurs in which the first display information should be displayed and a situation in which the second display information should be displayed simultaneously, the determination unit 53 makes the above determination and transmits an interrupt request signal to the display control unit 56.

The interrupt request signal is a signal that requests the first display device, on which the first display information is being displayed, to interrupt and display the second display information.

(Example 1 of Determination by Determining Unit 53)
When the ETC card is not inserted in the ETC vehicle-mounted unit, the determination unit 53 determines that the degree of urgency is high and transmits an interrupt request signal. In this case, the display control unit 56 interrupts and displays guidance indicating that the ETC card is not inserted in the ETC vehicle-mounted unit as the second display information. The guidance in this case is, for example, an image message such as "ETC card is not inserted."

(Example 2 of Determination by Determining Unit 53)
When the ETC on-board unit is unavailable, the determination unit 53 determines that the degree of urgency is high and transmits an interrupt request signal. In this case, the display control unit 56 interrupts and displays guidance indicating that the ETC on-board unit is unavailable as the second display information. The guidance in this case is, for example, an image message such as "ETC is unavailable."

(Example 3 of Determination by Determining Unit 53)
When an ETC card is inserted into the ETC vehicle-mounted unit, the determination unit 53 determines that the degree of urgency is high and transmits an interrupt request signal. In this case, the display control unit 56 interrupts and displays guidance indicating that an ETC card has been inserted as the second display information. The guidance in this case is, for example, an image message such as "ETC card has been inserted."

(Determination Example 4 by Determination Unit 53)
When the vehicle 200 passes through the ETC gate normally, the determination unit 53 determines that the degree of urgency is high and transmits an interrupt request signal. In this case, the display control unit 56 interrupts and displays guidance indicating that the vehicle 200 has passed through the ETC gate normally as the second display information. The guidance in this case is, for example, an image message such as "Passed through the ETC gate. The toll is 500 yen."

(Determination example 5 by the determination unit 53)
When the vehicle 200 receives warning information defined as downlink information in ETC 2.0, the determination unit 53 determines that the degree of urgency is high and transmits an interrupt request signal. Warning information includes, for example, J-Alert (a nationwide instantaneous warning system), emergency earthquake alerts, weather warnings, evacuation information, and information indicating the presence of a wrong-way vehicle. In this case, the display control unit 56 interrupts and displays an image message such as "There is a vehicle ahead that is going the wrong way" as the second display information.

(Example 6 of Determination by Determining Unit 53)
When the determination unit 53 receives priority information transmitted from an optical beacon, the determination unit 53 determines that the degree of urgency is high and transmits an interrupt request signal. The priority information includes construction information, safe driving support information, etc. In this case, the display control unit 56 interrupts and displays an image message such as "There is a construction restricted section 1 km ahead" as the second display information.

(Determination Example 7 by Determination Unit 53)
When the determination unit 53 receives emergency information included in the FM multiplex broadcast, it determines that the degree of urgency is high and transmits an interrupt request signal. Emergency information includes traffic accident information, disaster information, emergency earthquake alerts, etc. In this case, the display control unit 56 interrupts and displays an image message such as "A large earthquake has occurred. Please stop your vehicle in a safe place" as the second display information.

(Example 8 of Determination by Determining Unit 53)
When the determination unit 53 receives driving condition information indicating that the driving condition of the vehicle 200 is in a state where safety is lower than a reference driving condition, the determination unit 53 determines that the degree of urgency is high and transmits an interrupt request signal.

The following conditions can be exemplified as conditions in which safety is lower than the reference driving condition.
(1) The vehicle 200 is traveling at a speed exceeding 60 km/h in a section where the speed limit is 60 km/h.
(2) When the driver is drowsy at the wheel and continues to erratically drive.
(3) When the vehicle 200 deviates from its lane due to inattentive driving or the like.
(4) When the distance between vehicle 200 and a vehicle ahead of vehicle 200 becomes less than a predetermined distance.

In this case, the display control unit 56 interrupts and displays image messages such as "Exceeding the speed limit" or "Please operate the steering wheel" as the second display information.

(Example 9 of Determination by Determining Unit 53)
When the determination unit 53 receives information calling for attention to avoid a collision of the vehicle 200 with an object present around the vehicle 200, the determination unit 53 determines that the degree of urgency is high and transmits an interrupt request signal. In this case, the display control unit 56 interrupts and displays an image message such as "An accident has occurred ahead. Please avoid a collision" as the second display information.

(Determination example 10 by the determination unit 53)
When the determination unit 53 receives an image of a lane to which the vehicle 200 is to change lanes, the determination unit 53 determines that the degree of urgency is high and transmits an interrupt request signal. In this case, the display control unit 56 interrupts and displays, as the second display information, an image captured of a certain range of the lane to which the vehicle 200 is to change lanes.

(Determination example 11 by the determination unit 53)
When the determination unit 53 receives information indicating that another vehicle 200 existing around the vehicle 200 has cut in to the lane in which the vehicle 200 is traveling, the determination unit 53 determines that the degree of urgency is high and transmits a cut-in request signal. In this case, the display control unit 56 cuts in and displays an image message such as "There is a vehicle cutting in. Please be careful" as the second display information.

(Determination example 12 by the determination unit 53)
When the determination unit 53 receives remaining fuel information indicating that the remaining amount of fuel in the vehicle 200 is less than a predetermined value, the determination unit 53 determines that the degree of urgency is high and transmits an interrupt request signal. In this case, the display control unit 56 interrupts and displays an image message such as "Remaining driving distance is 10 km. Please refuel" as the second display information.

In addition, the determination unit 53 may determine the degree of urgency of providing the display information to the driver using either the information about the surrounding conditions of the vehicle 200 or the information about the state of the vehicle 200, or may determine the degree of urgency of providing the display information to the driver using both of these pieces of information.

(Display control unit 56)
When the interrupt request signal is not transmitted from the determination unit 53 , the display control unit 56 transmits the first display information to the display device 30 corresponding to the identification ID transmitted from the selection unit 57 .

When an interrupt request signal is transmitted from the determination unit 53, the display control unit 56 interrupts and displays the second display information by transmitting the second display information to the display device 30 corresponding to the identification ID transmitted from the selection unit 57.

(Display Device 30)
The display device 30A includes an information input unit 31 and an image display unit 32. The display device 30A is, for example, a center display provided in the vehicle interior.

Display device 30B is configured in the same manner as display device 30A. Display device 30B is, for example, an in-vehicle meter panel provided in front of the driver's seat.

Display device 30C is configured similarly to display device 30A. Display device 30C is, for example, a head-up display.

The type of display device 30 provided in the vehicle 200 is not limited to these, and may include, for example, an electronic mirror that combines an on-board camera and a display (e.g., an electronic rearview mirror or electronic side mirror).

Furthermore, the number of display devices 30 provided on the vehicle 200 is not limited to three, but may be two, four or more.

(Information input unit 31)
The information input unit 31 inputs the data of the first display information or the data of the second display information transmitted from the display control unit 56 , and transfers it to the image display unit 32 .

(Image display unit 32)
The image display unit 32 generates an image relating to the display information based on the data of the first display information or the data of the second display information, and displays the image on the screen of the image display unit 32 .

Next, the operation of the display control system 100 will be described with reference to Figures 2 and 3.

Figure 2 is a flowchart for explaining the operation of the display control system 100 according to an embodiment of the present disclosure.

In step S11, the image capture unit 11 starts capturing an image of the driver's face. The captured image data is sent to the gaze detection unit 13 via the image processing unit 12. Then, the process of step S12 is executed.

In step S12, the gaze detection unit 13 detects the direction in which the driver's gaze is directed based on the received image data.

The gaze information, which is the result of gaze detection, is sent to the selection unit 57 via the gaze information output unit 14 and the information input unit 51. The processing of the selection unit 57 will be described later.

In parallel with the processing of step S12, the information input unit 54 receives the object detection information transmitted by the detection information output unit 26, the information on the surrounding conditions of the vehicle 200 transmitted by the information collection device 60, and the information on the state of the vehicle 200, and transfers this information to the first display information creation unit 58 and the second display information creation unit 55 (step S13). Then, the processing of step S14 is executed.

In step S14, the first display information creation unit 58 creates the first display information, and the second display information creation unit 55 creates the second display information. The first display information and the second display information are transmitted to the display control unit 56. Then, the display control process shown in step S15 is executed.

The display control process shown in step S15 will be described with reference to FIG. 3. FIG. 3 is a flowchart for explaining the display control process.

In step S1, the selection unit 57 selects the display device 30 to which the gaze is directed. Then, the process of step S2 is executed.

In step S2, the display control unit 56 causes the display device 30 selected in step S1 to display the first display information. Then, the process of step S3 is executed.

In step S3, the determination unit 53 determines the degree of urgency of providing the display information to the driver based on at least one of information related to the surrounding conditions of the vehicle 200 and information related to the state of the vehicle 200.

If the urgency is high (step S3, YES), the determination unit 53 transmits an interrupt request signal to the display control unit 56 to interrupt and display the second display information on the first display device on which the first display information is being displayed. The process of step S4 is executed.

In step S4, the display control unit 56 receives the interrupt request signal and causes the display device 30, which is currently displaying the first display information, to interrupt and display the second display information. Then, the process of step S5 is executed.

In step S5, the display control unit 56 determines whether a predetermined time has elapsed since the interrupt display of the second display information was executed.

If the specified time has not elapsed (step S5, NO), the process proceeds to step S6.

In step S6, if the gaze direction detected by the gaze detection unit 13 has not changed (step S6, NO), the processing from step S5 onwards is repeated.

In step S6, if the gaze direction detected by the gaze detection unit 13 has changed (step S6, YES), the process of step S16 shown in FIG. 2 is executed. The process of step S16 will be described later.

Returning to step S5, if the predetermined time has elapsed (step S5, YES), the process proceeds to step S7.

In step S7, the display control unit 56 performs a process to erase the second display information. After that, the processes from step S2 onwards are repeated.

Returning to step S3, if the urgency is low (step S3, NO), the process proceeds to step S8.

In step S8, the display control unit 56 determines whether a predetermined time has elapsed since the interrupt display of the first display information was executed.

If the specified time has not elapsed (step S8, NO), the process proceeds to step S9.

In step S9, if the gaze direction detected by the gaze detection unit 13 has not changed (step S9, NO), the processing from step S2 onwards is repeated.

If the gaze direction detected by the gaze detection unit 13 has changed (step S9, YES), the process of step S16 shown in FIG. 2 is executed. The process of step S16 will be described later.

Returning to step S8, if the predetermined time has elapsed (step S8, YES), the process of step S10 is executed.

In step S10, the display control unit 56 performs a process to erase the first display information. Then, the process of step S16 shown in FIG. 2 is executed.

In step S16 shown in FIG. 2, the display control device 50 determines whether the processing up to step S15 has been completed.

If the processing up to step S15 has not been completed (step S16, NO), the display control device 50 repeats the processing from step S13 onwards.

If the processing up to step S15 is completed (step S16, YES), the display control device 50 ends the series of processing.

(Modification of the Determination Unit 53)
The determination unit 53 may be configured as follows.

When the determination unit 53 receives information about the surrounding conditions of the vehicle 200 transmitted from the information collection device 60, the determination unit 53 may be further configured to determine, based on the information about the surrounding conditions of the vehicle 200, whether or not a first degree of visibility when the second display information is displayed on the first display device is higher than a second degree of visibility when the second display information is displayed on a second display device other than the first display device.

The determination unit 53 determines whether the first degree is higher or lower than the second degree, for example, by referring to the correspondence information stored in the memory unit 52.

The corresponding information corresponds to, for example, the brightness outside the vehicle, the weather, the vehicle speed, the type of display device 30, the screen size of the display device 30, the screen position of the display device 30, the content of the displayed information, and the degree of visibility.

If the first degree of visibility is higher than the second degree of visibility, the determination unit 53 transmits a first display request signal to the display control unit 56 to cause the first display device to preferentially display the second display information.

When the first degree of visibility is lower than the second degree of visibility, the determination unit 53 transmits a second display request signal to the display control unit 56 to move the display position of the second display information from the first display device to the second display device.

In addition, when the first degree of visibility is lower than the second degree of visibility, the determination unit 53 may display the second display information on the second display device instead of the first display device.

When the display control unit 56 receives the first display request signal, it causes the first display device to display the second display information.

When the display control unit 56 receives the second display request signal, it switches the destination of the second display information from the first display device to the second display device.

For example, when the vehicle 200 is traveling at night, the driver's field of vision narrows due to the darkness outside the vehicle. In this case, the image displayed on the display device 30, which has a screen with a small display area, such as a head-up display or electronic mirror, may become difficult to see.

In this case, the display control unit 56 determines the illuminance outside the vehicle, the time of day, etc., and moves the second display information displayed on the first display device toward which the line of sight is directed to a second display device having a screen with a large display area, such as a center display or an in-vehicle meter panel.

This allows the driver to quickly recognize the second display information. Also, even in situations where visibility is limited, the driver can accurately recognize the content displayed on a large display area such as the center display.

It is preferable that the display control unit 56 is configured to display, on the first display device, guidance information informing the user that the display position of the second display information has moved from the first display device to the second display device, or guidance information informing the user that the second display information has been displayed on the second display device.

In this case, the display control unit 56 causes the first display device to display image messages such as "See center display" and "Emergency information has been moved to center display" as guidance information.

With this configuration, the driver can immediately understand that the display position of the second display information that he or she was checking has moved from the first display device to the second display device.

When the display position of the second display information is moved from the first display device to the second display device, the display control unit 56 may be configured to guide the line of sight to the second display device, for example, by blinking the second display information that has been moved to the second display device, or by enlarging the character size of the second display information that has been moved to the second display device.

Next, an example of the hardware configuration of the display control system 100 will be described with reference to FIG. 4. FIG. 4 is a diagram showing an example of the hardware configuration of the display control system 100 according to an embodiment of the present disclosure.

The vehicle 200 is equipped with an onboard device 210, which includes a gaze detection device 10, a surroundings monitoring device 20, a display control device 50, a display device 30A, a display device 30B, a display device 30C, an information collection device 60, and an ECU 220.

The ECU 220 includes an auxiliary storage device 230A, a memory device 230B, a CPU 230C, and an interface device 230D. These constitute the information processing device 230, and are connected to each other via a bus line 230E.

The auxiliary storage device 230A is a hard disk drive (HDD) or flash memory that stores files and data necessary for processing by the ECU 220. When an instruction to start a program is received, the memory device 230B reads out the program from the auxiliary storage device 230A and stores it.

The CPU 230C executes the programs stored in the memory device 230B and realizes the various functions of the ECU 220 according to the programs.

The interface device 230D is, for example, an interface that connects the CPU 230C to the gaze detection device 10, the surroundings monitoring device 20, etc.

The second display information mentioned above is not limited to the above display information, and may be, for example, door open/close status information, engine water temperature information, etc.

It is understood that the following aspects, for example, also fall within the technical scope of this disclosure:

(1) The display control device includes a gaze detection unit that detects the direction of the gaze of the driver of the vehicle based on image data of the driver of the vehicle; a selection unit that selects, based on gaze information indicating the direction of the driver's gaze, from among a plurality of display devices provided in the vehicle, a display device that is in the direction of the gaze as a first display device that displays display information to be provided to the driver; a determination unit that determines the degree of urgency of providing the display information to the driver based on at least one of information regarding the surrounding conditions of the vehicle and information regarding the state of the vehicle; and a display control unit that interrupts and displays, on the first display device on which the first display information is being displayed, second display information that has been determined to have a higher degree of urgency for providing to the driver than the degree of urgency for providing to the driver the first display information.

(2) The determination unit further determines a first degree of visibility when the second display information is displayed on the first display device and a second degree of visibility when the second display information is displayed on a second display device other than the first display device, based on information regarding the surrounding conditions of the vehicle, and the display control unit displays the second display information on the first display device if the first degree is higher than the second degree, and displays the second display information on the second display device if the first degree is lower than the second degree.

(3) The display control unit displays guidance information on the first display device to inform the user that the second display information has been displayed on the second display device.

(4) When an ETC card is not inserted in the ETC vehicle unit, the determination unit determines that the degree of urgency is high, and the display control unit interrupts and displays guidance indicating that an ETC card is not inserted in the ETC vehicle unit as the second display information.

(5) When the ETC on-board unit is unavailable, the determination unit determines that the degree of urgency is high, and the display control unit interrupts and displays guidance indicating that the ETC on-board unit is unavailable as the second display information.

(6) When an ETC card is inserted into the ETC vehicle unit, the determination unit determines that the degree of urgency is high, and the display control unit interrupts and displays guidance indicating that an ETC card has been inserted as the second display information.

(7) When the vehicle passes through an ETC gate, the determination unit determines that the degree of urgency is high, and the display control unit displays the toll when passing through the ETC gate as the second display information.

(8) When the vehicle receives warning information defined as downlink information in ETC 2.0, the determination unit determines that the degree of urgency is high, and the display control unit displays the warning information as the second display information.

(9) When the determination unit receives priority information transmitted from an optical beacon, the determination unit determines that the degree of urgency is high, and the display control unit displays the priority information as the second display information.

(10) When emergency information included in FM multiplex broadcasting is received, the determination unit determines that the degree of urgency is high, and the display control unit displays the emergency information as the second display information in an interrupt manner.

(11) When the determination unit receives information about a traffic accident or disaster, the determination unit determines that the degree of urgency is high, and the display control unit displays the information about the traffic accident or disaster as the second display information.

(12) When the determination unit receives information alerting the driver that a vehicle is traveling in the wrong direction on the roadway on which the vehicle is traveling, the determination unit determines that the degree of urgency is high, and the display control unit displays the information as the second display information.

(13) When the determination unit receives status information indicating that the driving state of the vehicle is less safe than a reference driving state, the determination unit determines that the degree of urgency is high, and the display control unit displays the status information as the second display information by interrupting the display.

(14) When the determination unit receives information alerting the driver to avoid a collision of the vehicle with an object present around the vehicle, the determination unit determines that the degree of urgency is high, and the display control unit displays the information as the second display information.

(15) When the determination unit receives an image of the lane to which the vehicle is to change lanes, the determination unit determines that the degree of urgency is high, and the display control unit displays the image as the second display information.

(16) When the determination unit receives information indicating that another vehicle around the vehicle has cut in to the lane in which the vehicle is traveling, the determination unit determines that the degree of urgency is high, and the display control unit cuts in and displays the information as the second display information.

(17) When the determination unit receives remaining fuel information indicating that the remaining amount of fuel in the vehicle is less than a predetermined value, the determination unit determines that the degree of urgency is high, and the display control unit displays the remaining fuel information as the second display information by interrupting the display.

(18) The display control system includes the display control device and a plurality of the display devices.

(19) The display control method includes the steps of receiving image data of a vehicle driver, detecting the direction of the driver's gaze based on the image data, selecting a display device in the direction of the driver's gaze from among a plurality of display devices provided in the vehicle based on gaze information indicating the direction of the driver's gaze as a first display device for displaying display information to be provided to the driver, determining the degree of urgency of providing the display information to the driver based on at least one of information related to the surrounding conditions of the vehicle and information related to the state of the vehicle, displaying the first display information on the first display device, and interrupting and displaying second display information, which is determined to have a higher degree of urgency of providing the first display information to the driver, on the first display device on which the first display information is being displayed.

Although various embodiments have been described above with reference to the drawings, it goes without saying that the present disclosure is not limited to such examples. It is clear that a person skilled in the art could come up with various modified or revised examples within the scope described in this disclosure, and it is understood that these also naturally fall within the technical scope of the present disclosure. Furthermore, the components in the above embodiments may be combined in any manner as long as it does not deviate from the spirit of the present disclosure.

Although specific examples of the present disclosure have been described in detail above, these are merely examples and do not limit the scope of the claims. The technology described in the claims may include various modifications and variations of the specific examples exemplified in this disclosure.

An embodiment of the present disclosure is suitable for a display control device, a display control system, and a display control method.

LIST OF SYMBOLS 10 Gaze detection device 11 Image capture unit 12 Image processing unit 13 Gaze detection unit 14 Gaze information output unit 15 Notification control device 20 Periphery monitoring device 21 Periphery image capture unit 22 Image processing unit 23 Object detection sensor 25 Object detection unit 26 Detection information output unit 30 Display device 30A Display device 30B Display device 30C Display device 31 Information input unit 32 Image display unit 50 Display control device 51 Information input unit 52 Storage unit 53 Determination unit 54 Information input unit 55 Second display information creation unit 56 Display control unit 57 Selection unit 58 First display information creation unit 60 Information collection device 100 Display control system 200 Vehicle 210 Vehicle-mounted device 230 Information processing device 230A Auxiliary storage device 230B Memory device 230C CPU
230D Interface device 230E Bus line

Claims (17)

a gaze detection unit that detects a direction in which the driver of the vehicle is looking based on image data of the driver of the vehicle;
a selection unit that selects, based on line-of-sight information indicating a direction in which the driver's line of sight is directed, from among a plurality of display devices provided in the vehicle, a display device that is in the direction in which the driver's line of sight is directed as a first display device that displays display information to be provided to the driver;
a determination unit that determines a degree of urgency of providing the display information to the driver based on at least one of information regarding a surrounding situation of the vehicle and information regarding a state of the vehicle;
a display control unit that displays, in an interrupt manner, second display information, the degree of urgency of which is determined to be higher than ...
Equipped with
The determination unit further determines, based on information regarding a surrounding situation of the vehicle, a first degree of visibility when the second display information is displayed on the first display device, and a second degree of visibility when the second display information is displayed on a second display device other than the first display device,
The display control unit is
When the first degree is higher than the second degree, the second display information is displayed on the first display device;
When the first degree is lower than the second degree, the second display information is displayed on the second display device;
A display control device that displays, on the first display device, guidance information informing the user that the second display information has been displayed on the second display device . The determination unit determines that the degree of urgency is high when an ETC card is not inserted in the ETC vehicle-mounted unit,
The display control device according to claim 1 , wherein the display control unit interrupts and displays, as the second display information, guidance indicating that an ETC card is not inserted in the ETC vehicle unit. The determination unit determines that the degree of urgency is high when the ETC in-vehicle unit is unavailable,
The display control device according to claim 1 , wherein the display control unit interrupts and displays, as the second display information, guidance indicating that an ETC vehicle-mounted unit is in an unavailable state. The determination unit determines that the degree of urgency is high when an ETC card is inserted into an ETC vehicle-mounted unit,
The display control device according to claim 1 , wherein the display control unit interrupts and displays, as the second display information, guidance indicating that an ETC card has been inserted. the determination unit determines that the degree of urgency is high when the vehicle has passed through an ETC gate;
The display control device according to claim 1 , wherein the display control unit displays, in an interrupt manner, a toll to be charged when passing through the ETC gate as the second display information. The determination unit determines that the degree of urgency is high when the vehicle receives warning information defined as downlink information in ETC 2.0,
The display control device according to claim 1 , wherein the display control unit displays the alarm information as the second display information in an interrupt manner. the determination unit determines that the degree of urgency is high when priority information transmitted from an optical beacon is received,
The display control device according to claim 1 , wherein the display control unit displays the priority information as the second display information in an interrupt manner. the determination unit determines that the degree of urgency is high when emergency information included in FM multiplex broadcasting is received,
The display control device according to claim 1 , wherein the display control unit is configured to interrupt-display the emergency information as the second display information. The determination unit determines that the degree of urgency is high when information regarding a traffic accident or a disaster is received,
The display control device according to claim 1 , wherein the display control unit interrupts and displays the information related to the traffic accident or disaster as the second display information. The determination unit determines that the degree of urgency is high when receiving information alerting the driver that a wrong-way vehicle is present on a roadway on which the vehicle is traveling,
The display control device according to claim 1 , wherein the display control unit displays the information as the second display information in an interrupt manner. When receiving state information indicating that the driving state of the vehicle is in a state where safety is lower than a reference driving state, the determination unit determines that the degree of urgency is high,
The display control device according to claim 1 , wherein the display control unit displays the state information as the second display information in an interrupt manner. The determination unit determines that the degree of urgency is high when information calling for attention to avoiding a collision of the vehicle with an object existing around the vehicle is received, and
The display control device according to claim 1 , wherein the display control unit displays the information as the second display information in an interrupt manner. The determination unit determines that the degree of urgency is high when an image of a lane to which the vehicle is to change lanes is received, and
The display control device according to claim 1 , wherein the display control unit displays the image as the second display information in an interrupt manner. The determination unit determines that the degree of urgency is high when information indicating that another vehicle existing around the vehicle has cut in to the lane in which the vehicle is traveling is received,
The display control device according to claim 1 , wherein the display control unit displays the information as the second display information in an interrupt manner. the determination unit determines that the degree of urgency is high when fuel remaining amount information indicating that the remaining amount of fuel loaded on the vehicle is less than a predetermined value is received,
The display control device according to claim 1 , wherein the display control unit displays the remaining fuel amount information as the second display information in an interrupt manner. A display control system comprising the display control device according to claim 1 and a plurality of the display devices. receiving image data of a driver of the vehicle;
Detecting a direction in which a driver of the vehicle is looking based on the image data;
selecting, based on line-of-sight information indicating a direction in which the driver's line of sight is directed, from among a plurality of display devices provided in the vehicle, a display device that is in the direction in which the driver's line of sight is directed as a first display device that displays display information to be provided to the driver;
determining a degree of urgency of providing the display information to the driver based on at least one of information regarding a surrounding situation of the vehicle and information regarding a state of the vehicle;
displaying first display information on the first display device;
a step of interrupt-displaying, on the first display device on which the first display information is currently being displayed, second display information, the degree of urgency of which is determined to be higher than ...
Including,
The determining step further includes determining a first degree of visibility when the second display information is displayed on the first display device and a second degree of visibility when the second display information is displayed on a second display device other than the first display device, based on information regarding a surrounding situation of the vehicle;
The step of interrupt displaying on the first display device includes:
When the first degree is higher than the second degree, the second display information is displayed on the first display device;
When the first degree is lower than the second degree, the second display information is displayed on the second display device;
displaying, on the first display device, guidance information informing the user that the second display information has been displayed on the second display device;
Display control method.

Images