WO2018163535A1

WO2018163535A1 - Alarm device, alarm system, alarm method and alarm control program

Info

Publication number: WO2018163535A1
Application number: PCT/JP2017/042559
Authority: WO
Inventors: 初美青位; 匡史日向; 芽衣上谷
Original assignee: オムロン株式会社
Priority date: 2017-03-10
Filing date: 2017-11-28
Publication date: 2018-09-13
Also published as: JP2018151763A

Abstract

This alarm device (6) comprises a first acquisition unit (613) which acquires first sensing data indicating the state of a driver, from a first monitoring sensor (7), a second acquisition unit (612) which acquires second sensing data which indicates the state of the surroundings of the vehicle, from a second monitoring sensor (11), a determination unit (614) which determines the nature of information prompting continued operation on the basis of first sensing data and second sensing data; and a control unit (615) which performs control to output, from alarm units (8, 9) provided to the vehicle, information, the nature of which has been determined.

Description

Notification device, notification system, notification method, and notification control program

The present invention relates to a notifying device, a notifying system, a notifying method, and a notifying control program for notifying a driver of a takeover operation required when switching the driving mode of a vehicle from an automatic driving mode to a manual driving mode.

In recent years, as a driving mode of a vehicle, in addition to a manual driving mode in which the vehicle is driven based on a driving operation of the driver, an automatic driving mode in which the vehicle is driven along a predetermined route without depending on the driving operation of the driver. Development is underway. The automatic driving mode includes, for example, navigation system information using GPS (Global Positioning System), traffic information acquired by road-to-vehicle communication, and information on a surrounding monitoring system that monitors the position and movement of surrounding people and vehicles. Basically, the vehicle can be automatically driven by controlling a power unit, a steering device, a brake, and the like (see, for example, JP-A-2015-141053).

Such an automatic driving mode is expected to reduce the burden on the driver's driving operations and reduce traffic congestion.

For example, highway is equipped with road-to-vehicle communication, no signal, no pedestrians, lanes are surely provided on one side, and most driving operations are relatively monotonous. It can be considered that the section is suitable for driving in the automatic driving mode. However, it is still difficult to drive in the automatic driving mode on a general road where road-to-vehicle communication is not established, there is only one lane on both sides, or the lane is disappearing.

Therefore, even if you are driving in the automatic driving mode on the expressway, if you exit the expressway, you need to switch from the automatic driving mode to the manual driving mode before reaching the general road.

For example, if the vehicle is traveling on an expressway in the automatic driving mode according to the travel plan, the vehicle can always grasp the distance from the current position to the exit of the expressway and the general road. The time required to leave the highway and reach a general road can also be estimated. And, for example, when the time to reach the general road is 60 seconds, “I will enter the general road soon. After 60 seconds, switch to manual operation mode. Please let me know "is output. In addition, countdown displays such as “60 seconds remaining”, “59 seconds remaining”,... Are displayed on the screen of the car navigation device, the tablet screen, the meter screen, and the like.

On the other hand, even when driving on the highway in the automatic driving mode, it may be impossible to continue the automatic driving mode due to bad weather such as fog or heavy rain, because the external sensor of the vehicle does not operate normally It is possible. In such a case, it is determined by the vehicle that the automatic operation mode cannot be continued. “Since the automatic operation mode cannot be continued due to bad weather, the operation mode is switched to the manual operation mode after 60 seconds. Please complete the voice announcement "is output. In this case as well, countdown displays such as “60 seconds remaining”, “59 seconds remaining”,...

Note that the time for preparation for switching from the automatic operation mode to the manual operation mode is appropriately set individually by an automobile manufacturer or the like, and the above-described 60 seconds is an example and is not limited. .

The driver takes the driving posture, holds the steering wheel (handover of the steering wheel), and adjusts the pedals such as the accelerator and the brake until the countdown ends, that is, in the state where the automatic driving is performed (handing over the pedal). ), Confirming front and rear, left and right, and turning the line of sight forward (handing over the line of sight) must be completed a series of hand over necessary for manual operation.

In the automatic operation mode, steering of the steering wheel and depression of a pedal such as an accelerator or a brake are automatically performed. On the other hand, since the steering wheel is not automatically steered when the mode is switched to the manual driving mode, the driver needs to grasp the steering wheel himself and prepare for manual steering. Also, when the automatic operation mode is switched to the manual operation mode, the pedal depressing position also returns to the initial position.Therefore, if the brake is depressed in the automatic operation mode immediately before entering the manual operation mode, the brake pedal Return to the initial position and suddenly no longer decelerates. Similarly, when the accelerator is depressed immediately before in the automatic operation mode, the accelerator pedal returns to the initial position and is not suddenly accelerated. As a result, the operation becomes unstable as soon as the manual operation mode is switched. Therefore, in order to prevent this from happening, the driver completes the takeover of the pedal by putting his / her foot on the pedal depression position that has been made in the automatic operation mode before entering the manual operation mode. It is necessary to keep.

Thus, handing over of the steering wheel and handing over of the pedal can be performed by following the state of automatic driving. On the other hand, taking over of the line of sight is not performed simply by following automatic driving.

As described above, taking over of the line of sight is performed by confirming the front, back, left and right, and then turning the line of sight forward. That is, the left and right side mirrors are viewed to confirm the left and right, the rear view mirror is viewed to confirm the rear, and then the line of sight is directed forward.

Therefore, when it is time to prepare for switching from the automatic operation mode to the manual operation mode, as described above, "... Switch to the manual operation mode after 60 seconds. “Please let me see”, “Look at the left and right side mirrors and check the left and right.” “Look at the rear view mirror, and check the back.” “Look forward. Please make an additional voice announcement "for the sake of reliable execution of eye transfer."

However, even if there is no such additional voice announcement, these additional voice announcements are very troublesome for a driver who has already performed some confirmation work spontaneously.

For example, for drivers who have already confirmed left and right side mirrors by looking at the left and right side mirrors, The voice announcement "."

Such an extra voice announcement is not preferable because it gives the driver extra frustration and may disturb the driver's state of mind. For drivers who have already confirmed the left and right side mirrors by looking at the left and right side mirrors, “Look at the rear view mirror and check the back.”, “Look forward.” Only a voice announcement may be added, and for drivers who have also confirmed the rear view by looking at the rearview mirror, only a voice announcement “Please look forward” is added. Just do it.

On the other hand, even if the front / rear / right / left confirmation has already been completed, it may be necessary to confirm again according to changes in the surrounding situation. For example, when a vehicle overtaking appears from the right or left lane, the driver needs to reconfirm left and right by looking at the left and right side mirrors. Alternatively, when the following vehicle approaches abnormally, the driver needs to check the distance from the following vehicle by looking at the rearview mirror.

In such a case, even for drivers who have already completed the front / rear / left / right confirmation, “There is a vehicle approaching from the right lane. Check the right side mirror.” Additional voice announcements such as “There are vehicles approaching from behind, please check the rearview mirror” should be made accordingly.

As described above, the notification for encouraging the confirmation operation of the takeover by looking at the eyes should be appropriately performed with appropriate contents in consideration of not only the driver's confirmation results but also the surrounding circumstances at that time.

The present invention has been made in view of such circumstances, and among the takeover operations necessary for switching from the automatic operation mode to the manual operation mode, in particular, the confirmation operation of takeover by the line of sight is the current situation. It aims at providing the alerting | reporting apparatus, alerting | reporting system, alerting | reporting method, and alerting | reporting control program for alert | reporting to a driver | operator by the appropriate content according to it.

In order to achieve the above object, the present invention takes the following measures.

That is, the first aspect of the present invention is an apparatus for informing information for prompting a driver to take over when the vehicle operation mode is switched from the automatic operation mode to the manual operation mode. A first acquisition unit that acquires first sensing data representing the state of the driver from a first monitoring sensor that monitors the state of the driver of the vehicle, and a second monitoring sensor that monitors the surroundings of the vehicle From the second acquisition unit that acquires the second sensing data representing the surrounding state of the vehicle, the first sensing data acquired by the first acquisition unit, and the second acquisition unit Based on the acquired second sensing data, a determination unit that determines the content of the information that prompts the takeover operation, and a notification that is provided in the vehicle with the information determined by the determination unit And a control unit for controlling so as to output from.

Further, according to a second aspect of the present invention, in the notification device according to the first aspect, the determination unit determines whether the driver has seen a predetermined mirror of the vehicle based on the first sensing data. A first determination is made, a second determination is made from the second sensing data to determine whether there is a confirmation object within a predetermined distance from the vehicle, and the first determination When it is determined that the driver is not looking at the predetermined mirror of the vehicle, and the second determination determines that the object to be confirmed is present, the driver is prompted to perform the predetermined mirror And the content of the notification is determined so as to include prompting attention to the confirmation object, and the driver does not look at the predetermined mirror of the vehicle by the first determination. And the second If it is determined that the object to be confirmed does not exist, the content of the notification is determined so as to include only prompting to see the predetermined mirror, and the operation is performed according to the first determination. Only when it is determined that the person has seen the predetermined mirror of the vehicle, and the second determination determines that the confirmation target is present, only to call attention to the confirmation target The content of the notification is determined as described above, and it is determined by the first determination that the driver has seen the predetermined mirror of the vehicle, and the confirmation target does not exist by the second determination. If it is determined, it is decided not to make the notification.

Further, according to a third aspect of the present invention, in the notification device according to the first or second aspect, the determination unit further includes the content of the notification determined by the determination unit from the time when the notification unit is output from the notification unit. Based on the first sensing data acquired by the first acquisition unit and the second sensing data acquired by the second acquisition unit during a period until the operation mode is switched to the manual mode. The content of the notification that prompts the takeover operation is redetermined.

Furthermore, a fourth aspect of the present invention is a notification control program that causes a computer to execute each unit included in the notification device according to any one of the first to third aspects.

According to a fifth aspect of the present invention, there is provided a system for informing information for prompting a driver to take over when the vehicle operation mode is switched from the automatic operation mode to the manual operation mode. A first monitoring sensor that monitors the state of the driver, a first acquisition unit that acquires first sensing data representing the state of the driver from the first monitoring sensor, and monitors the surroundings of the vehicle The second monitoring sensor that acquires the second sensing data representing the state of the surroundings of the vehicle from the second monitoring sensor, and the second acquisition sensor that is acquired by the first acquisition unit. Based on the first sensing data and the second sensing data acquired by the second acquisition unit, a determination unit that determines the content of the information that prompts the takeover operation, and the content by the determination unit And a control unit which performs control for outputting a constant information under the control of the control unit, and a notification unit for outputting the determined information.

According to a sixth aspect of the present invention, there is provided a method for informing a takeover operation required when the vehicle operation mode is switched from the automatic operation mode to the manual operation mode. Monitoring is performed by a monitoring sensor, and a notification device acquires first sensing data representing the state of the driver from the first monitoring sensor, monitors the surroundings of the vehicle by a second monitoring sensor, and performs the notification. A device acquires second sensing data representing a state around the vehicle from the second monitoring sensor, and the notification device is based on the first sensing data and the second sensing data. The notification content that prompts the handover operation is determined, and the notification device controls the notification unit provided in the vehicle so as to notify the driver of the determined notification content. It is the law.

Therefore, according to the first, fifth, and sixth aspects of the present invention, the takeover operation is urged based on the first sensing data representing the driver's state and the second sensing data representing the surrounding state of the vehicle. The notification contents can be notified with appropriate contents according to the current situation.

According to the second aspect of the present invention, it is possible to perform only necessary notification without giving excessive frustration to the driver by finely determining the notification content.

According to the third aspect of the present invention, it is possible to reliably follow the execution of the takeover operation by the driver while taking into consideration the surrounding conditions that change from moment to moment.

According to the fourth aspect of the present invention, it is possible to cause a computer to execute each unit included in the notification device according to any one of the first to third aspects.

FIG. 1 is a diagram illustrating an overall configuration example of an automatic operation control system including a takeover operation notification system according to the present embodiment. FIG. 2 is a block diagram illustrating a functional configuration example of the takeover operation notification system according to the present embodiment. FIG. 3 is a flowchart (1/2) showing an operation example of the handover operation notification system according to the present embodiment. FIG. 4 is a flowchart (2/2) illustrating an operation example of the handover operation notification system according to the present embodiment.

(Constitution)
FIG. 1 is a diagram illustrating an example of the overall configuration of an automatic operation control system including a takeover operation notification system 10 according to the present embodiment. This automatic driving control system is mounted on a vehicle 1 such as a passenger car.

The takeover operation notification system 10 is an example of a notification system in the present invention, and realizes the notification method in the present invention. The takeover operation notification system 10 includes a takeover operation notification device 6. The takeover operation notification device 6 is an example of a notification device in the present invention.

The vehicle 1 includes, as basic equipment, a power unit 2 including a power source and a transmission, and a steering device 3 equipped with a steering wheel 4, and further includes a manual operation mode and an automatic operation mode as operation modes. . An engine and / or a motor is used as the power source.

The manual driving mode is a mode in which the vehicle 1 is driven mainly by a driving operation manually performed by a driver (hereinafter also referred to as “driver”). The manual operation mode includes, for example, an operation mode for driving the vehicle based only on the driver's driving operation, and an operation mode for performing driving operation support control for supporting the driving operation of the driver while mainly driving the driver's driving operation. Is included.

The driving operation support control assists the steering torque so that the driver's steering becomes an appropriate steering amount based on the curvature of the curve when the vehicle 1 is traveling on the curve, for example. The driving operation support control includes control for assisting a driver's accelerator operation (for example, operation of an accelerator pedal) or brake operation (for example, operation of a brake pedal), manual steering (manual operation of steering), and manual speed adjustment (speed). Adjustment manual operation) is also included. In manual steering, the vehicle 1 is steered mainly by the driver's operation of the steering wheel 4. In manual speed adjustment, the speed of the vehicle is adjusted mainly by the driver's accelerator operation or brake operation.

It should be noted that the driving operation support control does not include control for forcibly intervening in the driving operation of the driver to automatically drive the vehicle 1. In other words, in the manual driving mode, the driving operation of the driver is reflected in the driving of the vehicle within a preset allowable range, but forcibly intervenes in the driving of the vehicle under certain conditions (for example, deviation from the lane of the vehicle). Control to do is not included.

On the other hand, the automatic operation mode is a mode for realizing an operation state in which the vehicle 1 is automatically driven along the road on which the vehicle 1 is traveling, for example. The automatic driving mode includes, for example, a driving state in which the vehicle 1 is automatically driven toward a preset destination without driving by the driver. In the automatic driving mode, it is not always necessary to automatically control all of the vehicle 1, and the driving state in which the driving operation of the driver is reflected in the traveling of the vehicle 1 within a preset allowable range is also included in the automatic driving mode. That is, the automatic driving mode includes a control in which the driving operation of the driver is reflected in the traveling of the vehicle 1 within a preset allowable range, but forcibly intervenes in the traveling of the vehicle under a certain condition.

In FIG. 1, reference numeral 5 denotes an automatic operation control device for executing operation control in the automatic operation mode. The automatic driving control device 5 acquires sensing data from the accelerator pedal sensor 12, the brake pedal sensor 13, the GPS receiver 14, the gyro sensor 15, the vehicle speed sensor 16, and the steering sensor 17, respectively. And these sensing data, route information generated by a navigation system (not shown), traffic information acquired by road-to-vehicle communication, information obtained by a surrounding monitoring system that monitors the position and movement of surrounding people and vehicles Based on this, the traveling of the vehicle 1 is automatically controlled.

Automatic control includes, for example, automatic steering (automatic steering operation) and automatic speed adjustment (automatic driving of speed). Automatic steering is an operating state in which the steering device 3 is automatically controlled. Automatic steering includes LKA (Lane Keeping Assist). For example, the LKA automatically controls the steering device 3 so that the vehicle 1 does not deviate from the traveling lane even when the driver does not perform the steering operation. Even when the LKA is being executed, the driver's steering operation may be reflected in the steering of the vehicle within a range (allowable range) in which the vehicle 1 does not deviate from the traveling lane. Note that automatic steering is not limited to LKA.

Automatic speed adjustment is an operating state in which the speed of the vehicle 1 is automatically controlled. Automatic speed adjustment includes ACC (Adaptive Cruise Control). For example, when there is no preceding vehicle ahead of the vehicle 1, ACC performs constant speed control that causes the vehicle 1 to travel at a constant speed at a preset speed, and when the preceding vehicle exists ahead of the vehicle 1. Is a follow-up control that adjusts the vehicle speed of the vehicle 1 in accordance with the inter-vehicle distance from the preceding vehicle. The automatic operation control device 5 decelerates the vehicle 1 according to the driver's brake operation (for example, operation of the brake pedal) even when ACC is being executed. In addition, the automatic operation control device 5 can perform the driver's accelerator operation (for example, accelerator) up to a preset maximum allowable speed (for example, the maximum speed legally determined on the traveling road) even during execution of ACC. The vehicle 1 can be accelerated according to the pedal operation. The automatic speed adjustment is not limited to ACC but also includes CC (Cruise Control).

The takeover operation notification system 10 is a system for notifying the driver of the takeover operation necessary for switching the operation mode of the vehicle 1 traveling in the automatic operation mode to the manual operation mode, in particular, the takeover operation of the line of sight. .

The takeover operation notification system 10 includes a takeover operation notification device 6, a driver monitoring sensor 7, which is a driver camera as an example, a speaker 8, a display screen 9, and an external monitoring sensor 11. The external monitoring sensors 11 are, for example, four cameras provided in the vehicle 1 for imaging the front, rear, right side, and left side of the vehicle 1, and in FIG. Only the sensor 11 (# 1) and the external monitoring sensor 11 (# 2) provided behind the vehicle 1 are illustrated. The external monitoring sensors 11 on the right and left sides of the vehicle 1 are not shown in FIG.

The four cameras that are the external monitoring sensors 11 respectively image the front, rear, right side, and left side of the vehicle 1 and output video signals C (# 1 to # 4) that are the imaging results to the takeover operation notification device 6. To do. The ambient monitoring system described above may also serve as the external monitoring sensor 11.

In this embodiment, the driver monitoring sensor 7 that is a driver camera is installed at a position in front of the driver, for example, on a dashboard, images the driver, and sends the video signal E to the takeover operation notification device 6. Output.

The speaker 8 outputs a voice announcement for prompting a takeover operation necessary when switching from the automatic operation mode to the manual operation mode based on the audio signal I output from the takeover operation notification device 6. For example, “Please check the left and right side mirrors to check the left and right.” “Look at the rear view mirror and check the back.”, “Please look forward.” “There are vehicles approaching from the right lane. Check the right side mirror.” “There are vehicles approaching from the back. Check the rear view mirror.” However, it is not limited to these.

The display screen 9 displays characters, images, and the like based on the display signal K output from the takeover operation notification device 6. The display screen 9 may correspond to, for example, a meter of the vehicle 1 or a display screen of a car navigation system mounted on the vehicle 1, but is not necessarily limited thereto, and is a monitor screen such as a driving monitor mounted on the vehicle 1. Or a head-up display may be used. The displayed characters and images correspond to the contents of the voice announcement output from the speaker 8, for example, “Please check the left and right side mirrors to check the left and right.” You may use characters such as "Please turn your gaze forward." Also, even if it is not necessarily the same content as the voice announcement, for example, “side mirror”, “room mirror”, and “forward direction”, only necessary keywords are displayed, or the side mirror is displayed. An arrow, an arrow indicating a rearview mirror, or an arrow indicating the forward direction may be used.

FIG. 2 is a block diagram illustrating a functional configuration example of the takeover operation notification system 10 according to the present embodiment.

The takeover operation notification device 6 is a device that comprehensively controls switching of operation modes, and includes a control unit 61, an input / output interface unit 62, and a storage unit 63.

The input / output interface unit 62 receives the manual operation switching request signal A output from the automatic operation control device 5 and outputs it to the control unit 61. When the driving mode of the vehicle 1 is switched from the automatic driving mode to the manual driving mode, the automatic driving control device 5 requests the manual driving switching at the timing of starting preparation for switching (for example, 60 seconds before switching to the manual driving mode). Signal A is output.

A specific example of the timing at which the automatic driving control device 5 outputs the manual driving switching request signal A will be described until the vehicle 1 traveling on the highway in the automatic driving mode in accordance with a preset driving plan enters the general road. In the meantime, the case of switching to the manual operation mode will be described as an example.

The automatic operation control device 5 always knows the position of the vehicle 1 by the GPS receiver 14 and the speed of the vehicle 1 by the vehicle speed sensor 16. Therefore, it is necessary to drive in the manual operation mode by taking into account a preset travel plan, the position of the vehicle 1 grasped by the GPS receiver 14, and the speed of the vehicle 1 grasped by the vehicle speed sensor 16. The time required to reach a general road (for example, another 300 seconds) is calculated. Further, the time required for preparation for switching (for example, 300-60 = 240) is subtracted from the required time (for example, 60 seconds) required for preparation for switching from the automatic operation mode to the manual operation mode. Seconds). At this timing, a manual operation switching request signal A is output.

It should be noted that the time required for preparation for switching from the automatic operation mode to the manual operation mode (hereinafter also referred to as “preparation time”) is not limited to 60 seconds, and may be another time. In addition, for example, automatic driving control such as 60 seconds when switching to manual driving mode by exiting from a highway, 30 seconds when switching due to bad weather, and 4 seconds when switching due to an emergency. The apparatus 5 may set different preparation times according to each case.

The input / output interface unit 62 receives the manual operation switching request signal A and outputs it to the control unit 61.

The input / output interface unit 62 also receives the video signal C output from the external monitoring sensor 11, which is a camera, converts it into video digital data D, and outputs the video digital data D to the control unit 61. Furthermore, the video signal E output from the driver monitoring sensor 7 which is a driver camera is received and converted into video digital data F, and the video digital data F is output to the control unit 61.

The input / output interface unit 62 further converts the audio announcement data H output from the control unit 61 into an audio signal I, and outputs the audio signal I to the speaker 8.

The input / output interface unit 62 further converts the display data J output from the control unit 61 into a display signal K, and outputs the display signal K to the display screen 9.

The storage unit 63 uses, as a storage medium, a non-volatile memory that can be written and read at any time, such as SSD (Solid State Drive) or HDD (Hard Disk Drive), and is used to realize this embodiment. As storage areas, an external monitoring data storage unit 631, a driver monitoring data storage unit 632, a voice announcement information storage unit 633, and a display information storage unit 634 are provided.

The control unit 61 has a CPU (Central Processing Unit) and a program memory that constitute a computer, and as control functions necessary for realizing the present embodiment, an operation mode switching signal receiving unit 611, an external monitoring data acquiring unit 612 are provided. A driver monitoring data acquisition unit 613, a notification content determination unit 614, and an output control unit 615 are provided. All of these control functions are realized by causing the CPU to execute a program stored in the program memory.

When receiving the manual operation switching request signal A output from the input / output interface unit 62, the operation mode switching signal receiving unit 611 generates an activation signal B for starting the operation of the notification content determining unit 614. Then, activation signal B is output to notification content determination unit 614.

The external monitoring data acquisition unit 612 receives the video digital data D output from the input / output interface unit 62, outputs it to the external monitoring data storage unit 631, and stores it in the external monitoring data storage unit 631.

The driver monitoring data acquisition unit 613 receives the video digital data F output from the input / output interface unit 62, outputs it to the driver monitoring data storage unit 632, and stores it in the driver monitoring data storage unit 632.

When the notification content determination unit 614 receives the activation signal B output from the operation mode switching signal reception unit 611, the notification content determination unit 614 acquires the video digital data D from the external monitoring data storage unit 631 and the video digital data from the driver monitoring data storage unit 632. Data F is acquired. Then, based on the video digital data D and the video digital data F, the content of notification to the driver is determined, and the determination result G is output to the output control unit 615. As an example, the following four patterns of the determination result G will be described, but the determination result G is not limited to these.

That is, the notification content determination unit 614 performs a first determination from the video digital data F to determine whether or not the driver has seen the side mirror and the rearview mirror of the vehicle 1. Further, a second determination for determining whether or not a confirmation object (for example, another vehicle, a two-wheeled vehicle, a person, etc.) exists within a predetermined distance (for example, within 2 m) from the vehicle 1 from the video digital data D. Do. If it exists, it is further determined in which direction the confirmation object is present. Further, for example, based on the feature amount acquired from the video digital data F, it may be determined whether the object to be confirmed is an ordinary car, a large car, a two-wheeled vehicle, or a person. . Note that it is known to perform type determination based on feature amounts from video digital data D captured by a camera.

The first determination determines that the driver has not seen all of the side mirrors and the rear view mirrors of the vehicle 1 (there are mirrors that have not been seen). When it is determined that the confirmation target exists within a predetermined distance, it is determined to notify the side mirror and the room mirror, to pay attention to the confirmation target, and the direction of the confirmation target. (First decision). In addition, when the type determination of this confirmation target object is performed, you may determine so that a type may be alert | reported further. The notification content determination unit 614 outputs such a first determination result G ₁ to the output control unit 615.

In addition, it is determined by the first determination that the driver has not seen all of the side mirrors and the rear view mirror of the vehicle 1 (there is a mirror that has not been seen), and from the vehicle 1 by the second determination. When it is determined that there is no object to be confirmed within the predetermined distance, it is determined to notify that the side mirror and the room mirror are to be viewed (second determination). Notification content determination unit 614 outputs such second determination result G ₂ to the output control section 615.

Furthermore, it is determined by the first determination that the driver has seen all of the side mirrors and the rear view mirror of the vehicle 1, and it is determined by the second determination that the confirmation target exists within a predetermined distance from the vehicle 1. If it is determined that the object to be confirmed is to be noted and the direction of the object to be confirmed is determined (third determination). In addition, when the type determination of this confirmation target object is performed, you may determine so that a type may be alert | reported further. Notification content determination unit 614 outputs such third decision result G ₃ to the output control section 615.

Furthermore, it is determined by the first determination that the driver has seen all of the side mirrors and the rear view mirror of the vehicle 1, and it is determined by the second determination that there is no confirmation object within a predetermined distance from the vehicle 1. If so, it is determined not to notify (fourth determination). Notification content determination unit 614 outputs such fourth determination result G ₄ to the output control section 615.

The output control unit 615 acquires appropriate voice announcement data H from the voice announcement information storage unit 633 according to the determination result G output from the notification content determination unit 614, and outputs it to the input / output interface unit 62. Appropriate display data J is acquired from the display information storage unit 634 and output to the input / output interface unit 62.

The input / output interface unit 62 converts the voice announcement data H into a voice signal I and outputs it to the speaker 8. As a result, a corresponding voice announcement is output from the speaker 8.

The input / output interface unit 62 also converts the display data J into a display signal K and outputs it to the display screen 9. As a result, the corresponding character or image is displayed from the display screen 9. In response to this, the driver confirms the mirror or confirms the confirmation object to be approached.

The voice announcement information storage unit 633 stores various voice announcement data H. Various voice message information is defined by adding new voice announcement data H to the voice announcement information storage unit 633 or updating the existing voice announcement data H stored in the voice announcement information storage unit 633. It is possible.

The display information storage unit 634 stores various display data J. Various display data can be defined by adding new display data J to the display information storage unit 634 or updating existing display data J stored in the display information storage unit 634. ing.

For example, if the determination result G ₁ from the notification content determination section 614 is output, voice announcement data H is "Check the left looking at the left and right side mirrors." As an example, "in particular, large right lane Check that the car is approaching. Check the rear view of the room mirror. Check the rear of the car. Turn the line of sight ahead. The text information may be left as it is, for example, an arrow indicating a side mirror or a rearview mirror, or an arrow indicating the right side mirror in an emphasis, especially if there is an approaching vehicle on the right. May be.

Also, if the determination result G ₂ from the notification content determination section 614 is output, voice announcement data H is "Check the left looking at the left and right side mirrors." As an example, rear looking at "room mirror The display data J may be the text information as it is, for example, an arrow indicating a side mirror or a room mirror. good.

In addition, if the decision result G ₃ from the notification content determination section 614 is output, voice announcement data H is, "because we are approaching large vehicles from the right lane, make sure." As an example will be something like that The display data J may be such character information as it is, for example, an arrow display indicating the right side mirror, or an arrow display pointing out the right side mirror.

Furthermore, when the determination result G ₄ is output from the notification content determination unit 614, the output control unit 615 does not acquire any data from the voice announcement information storage unit 633 and the display information storage unit 634. Therefore, no data is output to the input / output interface unit 62. In this way, the output control unit 615 performs control so that nothing is output from the speaker 8 and the display screen 9.

Further, the determination results G ₁ to G ₄ described above are merely examples, and the determination results are defined more finely. In accordance therewith, the sound announcement data H and the display data J are also prepared in detail, and the sound announcement information storage unit 633 and By storing the information in the display information storage unit 634, more detailed notification can be performed.

For example, if not also seen in any one of the side mirrors and a room mirror, the first determination result G ₁ and the second determination result G ₂ is applied to the aforementioned. However, in the determination results G ₁ and G ₂ , when neither one of the side mirror and the rearview mirror is seen, when both the side mirror and the rearview mirror are not seen, If you are looking, but you are not looking at the rear view mirror, you are looking at the rear view mirror, but you are not looking at the side mirror, and you are looking at the right side mirror but not the left side of the side mirror Etc. are not distinguished.

Accordingly, the determination result G is further reclassified according to which mirror is being viewed and which mirror is not being viewed, and accordingly, the detailed speech announcement data H and display data J are stored in the speech announcement information storage unit. 633 and the display information storage unit 634. Then, the output control unit 615 obtains appropriate message information H and display data J from the voice announcement information storage unit 633 and the display information storage unit 634, for example, “confirm left and right by looking at the left and right side mirrors”. It is possible to provide more detailed information such as “Please check the right side mirror and check the right.” Instead of “Please do”.

The notification content determination unit 614 repeats the determination process as described above for each predetermined interval until the switching from the automatic operation mode to the manual operation mode is completed, and outputs the determination result G to the output control unit 615. To do. Accordingly, the output control unit 615 appropriately acquires the message information H and the display data J corresponding to the progress of the confirmation action by the driver from the voice announcement information storage unit 633 and the display information storage unit 634, and the input / output interface unit 62 It becomes possible to output to the speaker 8 and the display screen 9 via the.

For example, in the last time, it is determined by the first determination that the driver has not seen all of the side mirrors and the rearview mirrors of the vehicle 1 (there are mirrors that are not seen). Please check the left and right side of the side mirror. ”And“ Check the back of the room mirror. ”A voice announcement will be made, and the display screen 9 will give a related display. Suppose the driver sees the left and right side mirrors and the room mirror.

This state is captured by the driver monitoring sensor 7 which is a driver camera, and the video digital data F is stored in the driver monitoring data storage unit 632. Based on the video digital data F, in the first determination in the notification content determination unit 614, it is determined that the driver has seen all of the side mirrors and the rearview mirror of the vehicle 1. Therefore, this time, the voice announcement and display that were made last time will not be made.

If it is not determined that the driver has seen all of the side mirrors and the rearview mirror of the vehicle 1 regardless of the voice announcements and displays made last time, the same voice announcements and displays as before are also made this time. . In this way, the handover operation notification system 10 reliably follows the confirmation action by the driver.

On the other hand, if it is determined in the second determination in this determination process that the object to be confirmed exists within a predetermined distance, which is not determined in the previous determination process, for example, “vehicle approaching from the right lane” An audio announcement such as “Please check the right side mirror.” Is output from the speaker 8, and an arrow indicating the right side mirror is displayed from the display screen 9.

(Operation)
Next, the operation of the takeover operation notification system 10 according to the present embodiment will be described.

3 and 4 are flowcharts showing an operation example of the handover operation notification system 10 according to the present embodiment.

In the takeover operation notification system 10, the surroundings of the vehicle 1 are always imaged by the external monitoring sensors 11 that are four cameras provided on the front side, the rear side, the right side, and the left side of the vehicle 1, respectively. C (# 1 to # 4) is output to the takeover operation notification device 6. These video signals C (# 1 to # 4) are converted into video digital data D (# 1 to # 4) by the input / output interface unit 62, and stored in the external monitoring data storage unit 631 by the external monitoring data acquisition unit 612. ing.

Further, for example, the driver's driving state is always imaged by the driver monitoring sensor 7 which is a driver camera installed at a position in front of the driver as on the dashboard, and an image signal E as an imaging result is obtained. It is output to the takeover operation notification device 6. The video signal E is converted into video digital data F by the input / output interface unit 62, output to the driver monitoring data acquisition unit 613, and stored in the driver monitoring data storage unit 632 by the driver monitoring data acquisition unit 613.

Then, at the timing when the preparation for switching to the manual operation mode is started from the state in which the vehicle 1 is traveling in the automatic operation mode, the manual operation switching request signal is sent from the automatic operation control device 5 to the takeover operation notification system 10. A is output. The manual operation switching request signal A is received by the input / output interface unit 62 and output to the operation mode switching signal receiving unit 611 (S1).

In the operation mode switching signal receiving unit 611, when the manual operation switching request signal A is received, the activation signal B for starting the operation of the notification content determination unit 614 is generated, and the activation signal B is converted into the notification content determination unit 614. Is output.

Then, the notification content determination unit 614 that has received the activation signal B acquires the video digital data D from the external monitoring data storage unit 631 and the video digital data F from the driver monitoring data storage unit 632 (S2).

Then, the notification content determination unit 614 determines the notification content to the driver based on the video digital data D and the video digital data F, and outputs the determination result G to the output control unit 615.

As the determination result G, the following four patterns will be described as an example, but are not limited thereto.

That is, the notification content determination unit 614 performs a first determination from the video digital data F to determine whether the driver has seen the side mirror and the rearview mirror of the vehicle 1. Further, a second determination for determining from the video digital data D whether or not a confirmation object (for example, another vehicle, a two-wheeled vehicle, a person, etc.) exists within a predetermined distance (for example, within 2 m) from the vehicle 1. Done. If it exists, it is further determined in which direction the confirmation object is present.

For example, it is determined by the first determination that the driver has not seen all of the side mirrors and the rearview mirror of the vehicle 1 (there is a mirror that has not been viewed) (S3: No), and the second determination When it is determined that there is a confirmation object within a predetermined distance from the vehicle 1 (S4: Yes), the fact that the side mirror and the rearview mirror of the vehicle 1 are viewed, the attention to the confirmation object, and the confirmation object A determination is made to notify the direction of the object (first determination) (S5). The first determination result G ₁ is output from the notification content determination unit 614 to the output control unit 615.

Moreover, it is determined by the first determination that the driver has not seen all of the side mirrors and the rearview mirror of the vehicle 1 (there is a mirror that has not been seen) (S3: No), and the second determination. Therefore, when it is determined that there is no object to be confirmed within a predetermined distance from the vehicle 1 (S4: No), it is determined to notify that the side mirror and the rearview mirror of the vehicle 1 are to be viewed. (Second determination) (S6). Such a second determination result G ₂ is output from the notification content determination unit 614 to the output control unit 615.

Furthermore, it is determined by the first determination that the driver has seen all of the side mirror and the rear view mirror of the vehicle 1 (S3: Yes), and the object to be confirmed within a predetermined distance from the vehicle 1 by the second determination. Is determined to be present (S7: Yes), a determination is made to notify the object to be confirmed and the direction of the object to be confirmed (third determination) (S8). ). The third determination result G ₃ is output from the notification content determination unit 614 to the output control unit 615.

Furthermore, it is determined by the first determination that the driver has seen all of the side mirror and the rear view mirror of the vehicle 1 (S3: Yes), and the object to be confirmed within a predetermined distance from the vehicle 1 by the second determination. Is determined not to exist (S7: No), it is determined not to notify (fourth determination) (S9). Such fourth determination result G ₄ is output from the notification content determination unit 614 to the output control unit 615.

In the output control unit 615, appropriate audio announcement data H is acquired from the audio announcement information storage unit 633 according to the determination result G output from the notification content determination unit 614 and output to the input / output interface unit 62. Appropriate display data J is acquired from the display information storage unit 634 and output to the input / output interface unit 62 (S10). In the case of the fourth decision result G _4, required takeover operation is considered to be complete, the output control section 615, audio announcement data H from the voice announcement information storage unit 633 is not acquired, the display information memory No display data J is acquired from the unit 634, no data is output from the output control unit 615 to the input / output interface unit 62, and nothing is output from the speaker 8 or the display screen 9.

In the input / output interface unit 62, the voice announcement data H is converted into a voice signal I and output to the speaker 8. As a result, a voice announcement corresponding to the voice announcement data H is output from the speaker 8 (S11).

In the input / output interface unit 62, the display data J is converted into the display signal K and output to the display screen 9. Thereby, the display corresponding to the display data J is made from the display screen 9 with characters, images, etc. (S12).

In FIG. 4, step S12 is shown after step S11, but step S11 and step S12 are performed in parallel.

For example, if the determination result G ₁ from the notification content determination unit 614 is outputted (S5), voice announcement data H is "Check the left looking at the left and right side mirrors." As an example, "in particular, the right A large car is approaching from the lane, please check. "Please check the rear view by looking at the rear view mirror", "Please look forward." The display data J may be the text information as it is, for example, an arrow display indicating a side mirror or a rearview mirror, and an arrow pointing out the right side mirror with emphasis, especially if there is an approaching vehicle on the right side. It may be a display. In response to this, the driver confirms the mirror or confirms the confirmation object to be approached.

The processes in steps S2 to S12 described above are repeated for each predetermined interval until the switching from the automatic operation mode to the manual operation mode is completed (S13). As a result, the output control unit 615 appropriately acquires the voice announcement data H and the display data J corresponding to the progress of the confirmation action by the driver from the voice announcement information storage unit 633 and the display information storage unit 634, and the input / output interface The unit 62 converts the sound signal I and the display signal J to the speaker 8 and the display screen 9, respectively. As a result, voice announcement and display are made from the speaker 8 and the display screen 9.

For example, the previous, the second determination result G ₂ is output by the notification content determining section 614, from the speaker 8, "Check the left looking at the left and right side mirrors." Look at "room mirror behind ”(S11), and a display related to that is made from the display screen 9 (S12). In response, the driver sees the left and right side mirrors and the room mirror. .

This state is imaged by the camera which is the driver monitoring sensor 7, and the video digital data F is stored in the driver monitoring data storage unit 632. Therefore, in the first determination in step S3, it is determined by the notification content determination unit 614 that the driver has seen all of the side mirrors and the rearview mirror of the vehicle 1 (S3: Yes).

Then, if it is determined by the second determination that there is no object to be confirmed within a predetermined distance from the vehicle 1 (fourth determination result G ₄ ) (S9), it is considered that the necessary takeover operation has been completed. As a result, nothing is output from the speaker 8 or the display screen 9.

On the other hand, if the driver has not yet seen either the side mirror or the rearview mirror of the vehicle 1 regardless of the voice announcement or display made last time, it is determined from the video digital data F. Therefore, this time, a voice announcement and a display for prompting to see a mirror that has not been seen are made.

Alternatively, after the output from the speaker 8 and a display screen 9 is no longer made, i.e., even after the fourth decision result G ₄ is obtained, the external monitoring sensor 11 before takeover to the manual operation mode is completed However, when a confirmation object (for example, another vehicle, a two-wheeled vehicle, a person, etc.) existing within a predetermined distance (for example, within 2 m) from the vehicle 1 is detected, a fourth determination result G ₃ (S9) is obtained. The direction of the object is notified, and the fact that attention is paid to this confirmation object is also notified.

Thus, an appropriate voice announcement and display are made from the speaker 8 and the display screen 9 according to the progress of the confirmation action by the driver, and the confirmation action by the driver is surely followed.

As described above, according to the takeover operation notification system 10 according to the present embodiment, the surroundings of the vehicle 1 are imaged by the external monitoring sensor 11 that is a camera, and the other is within a predetermined distance from the vehicle 1 based on the imaging result. It can be determined whether or not a confirmation target exists. Moreover, the driver | operator monitoring sensor 7 which is a driver camera can image a driver | operator, and can determine a driver | operator's state based on an imaging result.

As a result, among the takeover operations required for switching from the automatic operation mode to the manual operation mode, in particular, the takeover operation necessary for taking over the line of sight is given to the driver with an appropriate content according to the current situation. It is possible to notify.

Furthermore, the determination result made in the notification content determination unit 614 is finely defined, and the sound announcement data H and the display data J are prepared in detail accordingly, and are stored in the sound announcement information storage unit 633 and the display information storage unit 634. This makes it possible to provide more detailed notification information.

As described above, since the driver is not informed of unnecessary information but only necessary information is provided, the driver can surely concentrate on the takeover operation of the line of sight without feeling excessive frustration. It becomes like this.

Further, the series of processes including the acquisition, determination, and notification of the video as described above is repeated until the switching to the manual operation mode is completed, so that the driver can take into account the surrounding situation that changes from moment to moment. It is also possible to reliably follow the performance of the takeover action.

(Modification 1)
In the above embodiment, the case where a camera is applied as the external monitoring sensor 11 has been described. However, a radar may be used as the external monitoring sensor 11 instead of the camera.

In this case, radar transceivers are provided at the front, rear, right and left sides of the vehicle 1 in place of the cameras. Then, the front radar transceiver emits a radar in front of the vehicle 1 and receives a reflected wave of this radar. On the other hand, a rear radar transceiver emits a radar behind the vehicle 1 and receives a reflected wave of the radar. Similarly, the right radar transceiver emits a radar on the right side of the vehicle 1 and receives a reflected wave of the radar, and the left radar transceiver emits a radar on the left side of the vehicle 1 and reflects the radar. Receive waves.

Each radar transceiver outputs a received signal to the takeover operation notification device 6. In the takeover operation notification device 6, the input / output interface unit 62 receives the received signal and outputs it as external monitoring data to the external monitoring data acquisition unit 612.

The external monitoring data acquisition unit 612 stores the external monitoring data in the external monitoring data storage unit 631.

Then, the notification content determination unit 614 acquires the external monitoring data from the external monitoring data storage unit 631, and determines whether or not a confirmation target exists within a predetermined distance from the vehicle 1 based on the acquired external monitoring data. To do. For example, if the received signal strength is higher than a predetermined value, it is determined that the confirmation target exists within a predetermined distance from the vehicle 1. Furthermore, when the confirmation target exists, the size of the confirmation target can also be determined according to the received signal strength. This makes it possible to distinguish between a small car and a large car.

In this way, it is possible to use a radar as the external monitoring sensor 11 instead of the camera.

(Modification 2)
Instead of the display screen 9 in the above embodiment, a light source provided on a cover of a side mirror or a room mirror can be used.

For example, if the determination result G by the notification content determination unit 614 is notification that the right side mirror is to be viewed, the output control unit 615 turns on the light source provided on the cover of the right side mirror. Also, if it is a notification that the left side mirror is to be viewed, the output control unit 615 turns on the light source provided on the cover of the left side mirror, and if it is a notification that the rear view mirror is to be viewed, the output is output. The control unit 615 turns on the light source provided on the cover of the room mirror. When there are a plurality of mirrors to be viewed, the light sources provided on the covers of all the corresponding mirrors are turned on.

By turning on the light source in this way, the driver can easily grasp which mirror should be confirmed.

The best mode for carrying out the present invention has been described above with reference to the accompanying drawings, but the present invention is not limited to such a configuration. Within the scope of the invented technical idea of the scope of claims, a person skilled in the art can conceive of various changes and modifications. The technical scope of the present invention is also applicable to these changes and modifications. It is understood that it belongs to.

[Appendix 1]
An apparatus for informing information for prompting a driver to take over when switching the driving mode of the vehicle from the automatic driving mode to the manual driving mode, comprising a processor and a memory,
The memory is
A first storage unit that stores first sensing data representing the state of the driver;
A second storage unit that stores second sensing data representing a surrounding state of the vehicle,
The processor is
Obtaining the first sensing data from the first monitoring sensor and storing it in the first storage unit,
Obtaining the second sensing data from a second monitoring sensor and storing it in the second storage unit;
Based on the first sensing data and the second sensing data, determine the content of the information that prompts the takeover operation,
A notification device that controls to output the information whose content is determined from a notification unit provided in the vehicle.

[Appendix 2]
A system for notifying a driver of information necessary for a takeover operation when a vehicle driving mode is switched from an automatic driving mode to a manual driving mode, the processor, a memory, a first monitoring sensor, A second monitoring sensor and a notification unit,
The memory is
A first storage unit that stores first sensing data representing the state of the driver;
A second storage unit that stores second sensing data representing a surrounding state of the vehicle,
The first monitoring sensor monitors a state of a driver of the vehicle;
The second monitoring sensor monitors the surroundings of the vehicle,
The processor is
Obtaining the first sensing data from the first monitoring sensor and storing it in the first storage unit;
Obtaining the second sensing data from the second monitoring sensor and storing it in the second storage unit;
Based on the first sensing data and the second sensing data, determine the content of the information that prompts the takeover operation,
Performing control to output the information whose contents are determined;
The notification unit is configured to output the determined notification content under the control.

[Appendix 3]
A method for notifying a takeover operation required when switching a driving mode of a vehicle from an automatic driving mode to a manual driving mode,
Monitoring the condition of the driver of the vehicle by a first monitoring sensor;
Monitoring the surroundings of the vehicle by a second monitoring sensor;
The processor obtains first sensing data representing the state of the driver from the first monitoring sensor, and obtains second sensing data representing the surrounding state of the vehicle from the second monitoring sensor. Then, based on the first sensing data and the second sensing data, a notification content that prompts the takeover operation is determined, and the vehicle is notified to the driver in order to notify the determined content to the driver. A notification method for controlling a provided notification unit.

Claims

When switching the vehicle driving mode from the automatic driving mode to the manual driving mode, it is a device for informing the driver of information that prompts a necessary takeover operation,
A first acquisition unit that acquires first sensing data representing the state of the driver from a first monitoring sensor that monitors the state of the driver of the vehicle;
A second acquisition unit that acquires second sensing data representing a state of the surroundings of the vehicle from a second monitoring sensor that monitors the surroundings of the vehicle;
Based on the first sensing data acquired by the first acquisition unit and the second sensing data acquired by the second acquisition unit, the content of the information that prompts the takeover operation is determined. A decision unit;
A control unit that controls the information determined by the determination unit to be output from a notification unit provided in the vehicle;
A notification device comprising:
The determination unit is
From the first sensing data, perform a first determination to determine whether the driver has seen a predetermined mirror of the vehicle,
From the second sensing data, a second determination is made to determine whether or not a confirmation object exists within a predetermined distance from the vehicle,
If it is determined by the first determination that the driver is not looking at the predetermined mirror of the vehicle, and the second determination determines that the confirmation target is present, Determining the content of the notification so as to include prompting to look at a predetermined mirror and prompting attention to the confirmation target;
When the first determination determines that the driver is not looking at the predetermined mirror of the vehicle, and the second determination determines that the confirmation target does not exist, The content of the notification is determined so as to include only that the user is prompted to view the predetermined mirror,
If it is determined by the first determination that the driver has seen the predetermined mirror of the vehicle, and the second determination determines that the confirmation target exists, the confirmation target The content of the notification is determined so as to include only a notice to call attention to an object,
If it is determined by the first determination that the driver has seen the predetermined mirror of the vehicle, and the second determination determines that the confirmation target does not exist, the notification The notification device according to claim 1, wherein it is determined not to perform the operation.
The determination unit is further acquired by the first acquisition unit during a period from when the notification content determined by the determination unit is output from the notification unit until the operation mode is switched to the manual mode. The notification according to claim 1 or 2, wherein the notification content that prompts the takeover operation is re-determined based on the first sensing data and the second sensing data acquired by the second acquisition unit. apparatus.
A notification control program for causing a computer to execute each unit included in the notification device according to any one of claims 1 to 3.
A system for informing information that prompts a driver to take over when the vehicle driving mode is switched from the automatic driving mode to the manual driving mode,
A first monitoring sensor for monitoring a state of a driver of the vehicle;
A first acquisition unit that acquires first sensing data representing the state of the driver from the first monitoring sensor;
A second monitoring sensor for monitoring the surroundings of the vehicle;
A second acquisition unit for acquiring second sensing data representing a state around the vehicle from the second monitoring sensor;
Based on the first sensing data acquired by the first acquisition unit and the second sensing data acquired by the second acquisition unit, the content of the information that prompts the takeover operation is determined. A decision unit;
A control unit that performs control for outputting information whose contents are determined by the determination unit;
Under the control of the control unit, a notification unit that outputs the determined notification content;
A notification system comprising:
A method for notifying a takeover operation required when switching a driving mode of a vehicle from an automatic driving mode to a manual driving mode,
Monitoring the state of the driver of the vehicle by a first monitoring sensor;
The notification device acquires first sensing data representing the state of the driver from the first monitoring sensor,
Monitoring the surroundings of the vehicle by a second monitoring sensor;
The notification device acquires second sensing data representing a state around the vehicle from the second monitoring sensor;
Based on the first sensing data and the second sensing data, the notification device determines notification contents for prompting the takeover operation,
A notification method in which the notification device controls a notification unit provided in the vehicle so as to notify the driver of the determined notification content.