JP2024057142A - Information processing device, control method, program, and storage medium - Google Patents

Abstract

To provide an information processing device capable of suitably mediating the communication between a driver of a mobile body and a user existing outside the mobile body, a method, a program, and a storage medium.SOLUTION: In a driving speech system, an on-vehicle machine 1A or a server device 5, which is an information processing device used in a system mediating the communication between a driver of a target vehicle being a mobile body and an out-of-vehicle terminal 2 existing outside the target vehicle, comprises acquisition means, detection means, and display control means. The acquisition means acquires a peripheral photographing image obtained by shooting the periphery of a target vehicle from the target vehicle. The detection means detects a state or an operation of the driver of the target vehicle. The display control means causes the out-of-vehicle terminal 2 to display a prescribed range in the peripheral photographing image. In this case, the display control means determines the prescribed range in the peripheral photographing image displayed by the out-of-vehicle terminal 2 on the basis of the state or operation of the driver.SELECTED DRAWING: Figure 11

Description

The present invention relates to a system that mediates communication between a driver of a mobile object and a user outside the mobile object.

Technologies that enable conversation between a vehicle driver and a person outside the vehicle have been known for some time. For example, Patent Document 1 discloses a conversation providing system that enables a conversation between the vehicle driver and a conversation partner by communicating with an in-vehicle system via a communication network.

JP 2017-138277 A

In a system that allows communication between the driver and a person outside the vehicle, the person outside the vehicle may provide advice on assisting with safety checks while checking captured images of the vehicle's surroundings. In this case, for the person outside the vehicle to provide appropriate assistance with safety checks, it is necessary to present the person outside the vehicle with images suitable for assisting with safety checks.

In view of the above-mentioned problems, one of the objectives of the present invention is to provide an information processing device that can effectively mediate communication between a driver of a moving body and a user outside the moving body.

The invention described in claim 1 is
An information processing device used in a system that mediates communication between a driver of a moving body and a user outside the moving body, comprising:
An acquisition means for acquiring a peripheral image of the moving body by capturing an image of the surroundings of the moving body from the moving body;
A detection means for detecting a state or operation of a driver of the vehicle;
a display control means for displaying a predetermined range in the surrounding photographed image on a terminal device of the user;
The display control means is an information processing device that determines the predetermined range to be displayed on the terminal device based on a state or operation of the driver.

The invention described in claim 8 is as follows:
A control method executed by an information processing device used in a system that mediates communication between a driver of a moving body and a user present outside the moving body, comprising:
an acquisition step of acquiring a peripheral image of the moving body from the moving body;
A detection step of detecting a state or operation of a driver of the moving body;
A display control step of displaying a predetermined range in the peripheral photographed image on a terminal device of the user,
The display control step is a control method for determining the predetermined range to be displayed on the terminal device based on a state or operation of the driver.

The invention described in claim 9 is as follows:
A program executed by a computer of an information processing device used in a system that mediates communication between a driver of a moving body and a user outside the moving body, comprising:
An acquisition means for acquiring a peripheral image of the moving body by capturing an image of the surroundings of the moving body from the moving body;
A detection means for detecting a state or operation of a driver of the vehicle;
causing the computer to function as a display control means for displaying a predetermined range in the peripheral photographed image on a terminal device of the user;
The display control means is a program that determines the predetermined range to be displayed on the terminal device based on a state or operation of the driver.

1 shows an example of the configuration of a driving call system. 2 shows an example of a schematic configuration of an in-vehicle device. 2 shows an example of a schematic configuration of an external terminal. 1A is an overhead view of a target vehicle, showing an exterior camera installed on the target vehicle and the imaging range, and FIG. 1B is an overhead view of the target vehicle, showing the display range of a peripheral image captured in a normal angle of view mode and a wide angle of view mode. 1A is an overhead view of a target vehicle, showing three exterior cameras installed on the target vehicle and the shooting range of each exterior camera, and FIG. 1B is an overhead view of the target vehicle, showing the range displayed as a peripheral shot image in both a normal angle of view mode and a wide angle of view mode. 4 shows a first display example of the external terminal in the first embodiment. 13 shows a second display example of the external terminal in the first embodiment. 5 is a flowchart illustrating an example of a procedure of a process executed by an in-vehicle device in the first embodiment. 13 shows a display example of an external terminal according to a modified example of the first embodiment. 13 is an example of a flowchart according to a modified example of the first embodiment. 1 shows a configuration example of a driving call system according to a modified example of the first embodiment. 2 illustrates an example of a schematic configuration of a server device. 1A is an overhead view of a target vehicle, showing an exterior camera installed on the target vehicle and a range of whether or not a display is present in the shooting range when the low visibility frequency direction is the rightward direction, and FIG. 1B is an overhead view of a target vehicle, showing an exterior camera installed on the target vehicle and a range of whether or not a display is present in the shooting range when the low visibility frequency direction is the leftward direction. (A) is an overhead view of a target vehicle, clearly indicating three exterior cameras installed on the target vehicle, their respective shooting ranges, and whether or not each range is displayed when the low visibility frequency direction is the right direction. (B) is an overhead view of a target vehicle, clearly indicating three exterior cameras installed on the target vehicle, their respective shooting ranges, and whether or not each range is displayed when the low visibility frequency direction is the left direction. 13 shows a first display example of the external terminal in the second embodiment. 13 shows a second display example of the external terminal in the second embodiment. 13 is a flowchart illustrating an example of a procedure of a process executed by an in-vehicle device in a second embodiment. 10 is a flowchart illustrating an example of a procedure of a process executed by an in-vehicle device in a modified example of the second embodiment.

In one preferred embodiment of the present invention, an information processing device used in a system that mediates communication between a driver of a moving body and a user outside the moving body includes an acquisition means for acquiring a peripheral image of the moving body from the moving body, a detection means for detecting the state or operation of the driver of the moving body, and a display control means for displaying a predetermined range of the peripheral image on the user's terminal device, and the display control means determines the predetermined range to be displayed on the terminal device based on the state or operation of the driver. With this aspect, the information processing device can present a peripheral image of the range determined based on the state or operation of the driver to a user outside the moving body.

In one aspect of the information processing device, the display control means determines the predetermined range based on the degree of driving load of the driver. In a preferred example, the display control means may expand the predetermined range as the degree of driving load increases. According to these aspects, the information processing device can present to the user of the terminal device a surrounding captured image of a range corresponding to the driving load of the driver.

In another aspect of the information processing device, the display control means expands the predetermined range based on a predetermined utterance by the driver as an operation by the driver. This allows the information processing device to suitably control the display range of the surrounding captured image in response to a request from the driver.

In another aspect of the information processing device, the display control means causes the terminal device to display a line representing the trajectory of the driver's line of sight on the image obtained by enlarging the predetermined range. In this case, the information processing device can present the image to the user of the terminal device in a manner that allows the user to grasp the areas that the driver is viewing and the areas that he or she is not viewing.

In another aspect of the information processing device, the predetermined range includes at least the forward direction of the moving object, and the display control means changes the width of the predetermined range based on the forward direction based on the state or operation of the driver. With this aspect, the information processing device can suitably determine the display range of the surrounding captured image based on the state or operation of the driver while including at least the forward direction of the moving object.

In another aspect of the information processing device, the surrounding captured image is a composite image obtained by combining images captured by multiple cameras provided on the moving body, and the display control means causes the terminal device to display the predetermined range in the composite image. In this way, even in an aspect in which the surroundings of the moving body are captured by multiple cameras, the information processing device can suitably present to the user of the terminal device an image of the surroundings of the moving body within a range determined based on the driver's state or operation.

In another preferred embodiment of the present invention, a control method is executed by an information processing device used in a system that mediates communication between a driver of a moving body and a user outside the moving body, the control method comprising an acquisition step of acquiring a peripheral image of the moving body from the moving body, a detection step of detecting a state or operation of the driver of the moving body, and a display control step of displaying a predetermined range of the peripheral image on the user's terminal device, the display control step determining the predetermined range to be displayed on the terminal device based on the state or operation of the driver. By executing this control method, the information processing device can present a peripheral image of the range determined based on the state or operation of the driver to a user outside the moving body.

In another preferred embodiment of the present invention, a program executed by a computer of an information processing device used in a system that mediates communication between a driver of a moving body and a user outside the moving body causes the computer to function as an acquisition means for acquiring a peripheral image of the moving body taken from the moving body, a detection means for detecting the state or operation of the driver of the moving body, and a display control means for displaying a predetermined range of the peripheral image on a terminal device of the user, and the display control means determines the predetermined range to be displayed on the terminal device based on the state or operation of the driver. By executing this program, the computer of the information processing device can present a peripheral image of the range determined based on the state or operation of the driver to a user outside the moving body. Preferably, the program is stored in a storage medium.

Below, preferred embodiments of the present invention will be described with reference to the drawings.

First Example
(1-1) System configuration
1 shows a configuration example of a driving call system according to a first embodiment. The driving call system has an in-vehicle device 1 installed in a vehicle and an external terminal 2 used by a person outside the vehicle. In the driving call system, the in-vehicle device 1 and the external terminal 2 communicate with each other to mediate a voice call between the driver of the vehicle and the user of the external terminal 2, and the user of the external terminal 2 talks to the driver of the vehicle while checking various information obtained from the vehicle through the external terminal 2.

The vehicle-mounted device 1 moves with the vehicle and performs processing to realize a call between the driver of the vehicle and the user of the external terminal 2. Hereinafter, the vehicle equipped with the vehicle-mounted device 1 is also called the "target vehicle". The vehicle-mounted device 1 performs data communication with the external terminal 2 via a communication network 3 such as the Internet or a dedicated communication network. Data exchanged between the vehicle-mounted device 1 and the external terminal 2 includes voice data generated in a call between the driver of the target vehicle and the user of the external terminal 2, and display instruction data for displaying information about the target vehicle or the driver. In this embodiment, the vehicle-mounted device 1 transmits display instruction data based on an image (also called "peripheral image") captured from the target vehicle to the external terminal 2. As described later, the vehicle-mounted device 1 determines the angle of view of the peripheral image (i.e., the display range of the peripheral image) to be displayed on the external terminal 2 based on the driver's driving load (workload).

The vehicle-mounted device 1 may be a navigation device that is installed in the target vehicle and provides route guidance to a set destination, or may be a mobile terminal such as a smartphone. The vehicle-mounted device 1 may also be incorporated in the target vehicle. The vehicle-mounted device 1 is an example of an "information processing device." The target vehicle is an example of a "mobile body."

The external terminal 2 is a terminal operated by a person outside the target vehicle, and performs data communication with the in-vehicle device 1 via the communication network 3. The external terminal 2 is, for example, a mobile terminal such as a smartphone. For example, the external terminal 2 establishes communication with the in-vehicle device 1 while the target vehicle is being driven, and exchanges voice data with the in-vehicle device 1 for a call between the driver and the user of the external terminal 2. The external terminal 2 also receives display instruction data from the in-vehicle device 1 during the call, and displays surrounding captured images, etc. based on the display instruction data. The external terminal 2 is an example of a "terminal device."

The communication between the in-vehicle unit 1 and the external terminal 2 may be realized by being relayed by a server device (not shown). Even if the in-vehicle unit 1 and the external terminal 2 establish direct communication, the in-vehicle unit 1 and/or the external terminal 2 may exchange information (e.g., communication address information, etc.) necessary for the in-vehicle unit 1 and the external terminal 2 to establish communication with the server device. In these cases, the server device performs the processes necessary for the in-vehicle unit 1 and the external terminal 2 to perform data communication (including authentication processes for the in-vehicle unit 1 and the external terminal 2).

(1-2) Device configuration
2 shows an example of a schematic configuration of the vehicle-mounted device 1. The vehicle-mounted device 1 mainly includes a communication unit 11, a storage unit 12, an input unit 13, a control unit 14, a sensor group 15, a display unit 16, and a sound output unit 17. The elements in the vehicle-mounted device 1 are connected to each other via a bus line 10.

The communication unit 11 performs data communication with other terminals under the control of the control unit 14. For example, the communication unit 11 may receive map data for updating the map DB (Database) 4 from a map management server (not shown).

The storage unit 12 is composed of various types of memory such as RAM (Random Access Memory), ROM (Read Only Memory), and non-volatile memory (including hard disk drive, flash memory, etc.). The storage unit 12 stores programs for the in-vehicle device 1 to execute predetermined processes. The above-mentioned programs may include application programs for making calls, and application programs for causing the external terminal 2 to display maps and images captured in the target vehicle. The storage unit 12 is also used as a working memory for the control unit 14. The programs executed by the in-vehicle device 1 may be stored in a storage medium other than the storage unit 12.

The memory unit 12 also stores a map DB (Data Base) 4. Various data necessary for route guidance is recorded in the map DB 4. The map DB 4 is data necessary for map display based on a specified position such as the current position of the target vehicle. The map DB 4 is a database that includes, for example, road data that represents a road network by a combination of nodes and links, and facility data that indicates facilities that are candidates for a destination, a stop-off point, or a landmark. The map DB 4 may be updated based on information received by the communication unit 11 from a map management server under the control of the control unit 14.

The input unit 13 is a button, a touch panel, a remote controller, a voice input device, etc. that the user operates. The display unit 16 is a display or the like that displays information based on the control of the control unit 14. The sound output unit 17 is a speaker or the like that outputs sound based on the control of the control unit 14.

The sensor group 15 includes various sensors that sense the state of the target vehicle or the environment outside the vehicle. The sensor group 15 includes an exterior camera 51, a driver camera 52, a vehicle behavior detector 53, and a biosensor 54.

The exterior camera 51 is one or more cameras that capture images of the outside of the target vehicle, such as the area in front of the target vehicle, and generates surrounding images captured at a specified time interval. Hereinafter, the images output by the exterior camera 51 (including a composite of images output by multiple exterior cameras 51) will be referred to as "surrounding images," and an image obtained by cutting out a specified range (display range) to be displayed on the exterior terminal 2 from the surrounding images will also be referred to as "surrounding images for display."

The driver camera 52 is a camera that is installed so that the driver's face is included in the shooting range, and generates images (also called "driver images") captured at a predetermined time interval.

The vehicle behavior detector 53 generates detection signals indicating the behavior of the target vehicle, such as the current position, vehicle speed, acceleration, steering angle, etc. The vehicle behavior detector 53 includes, for example, a Global Navigation Satellite System (GNSS) receiver, a gyro sensor, an Inertial Measurement Unit (IMU), a vehicle speed sensor, an acceleration sensor, a steering angle sensor, etc.

The biosensor 54 is one or more sensors that generate biosignals that indicate the driver's biophenomena, such as heart rate and sweat rate.

The sensor group 15 may include various external sensors (including cameras, lidars, radars, ultrasonic sensors, infrared sensors, sonar, etc.) and internal sensors in addition to the exterior camera 51, the driver camera 52, the vehicle behavior detector 53, and the biosensor 54.

The control unit 14 includes a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), etc., and controls the entire in-vehicle device 1. The control unit 14 functions as an "acquisition means," a "detection means," a "display control means," and a computer that executes programs, etc.

The processing executed by the control unit 14 is not limited to being realized by software using a program, but may be realized by any combination of hardware, firmware, and software. The processing executed by the control unit 14 may also be realized using a user-programmable integrated circuit, such as an FPGA (Field-Programmable Gate Array) or a microcomputer. In this case, the program executed by the control unit 14 in this embodiment may be realized using this integrated circuit.

The configuration of the vehicle-mounted device 1 shown in FIG. 2 is an example, and various modifications may be made to the configuration shown in FIG. 2. For example, instead of the map DB 4 being stored in the memory unit 12, the control unit 14 may receive map information from a map management server (not shown) via the communication unit 11. In another example, at least one of the input unit 13, the display unit 16, and the sound output unit 17 may be provided in the target vehicle as an external device of the vehicle-mounted device 1, and may supply the generated signal to the vehicle-mounted device 1. In addition, at least some of the sensors in the sensor group 15 may be sensors provided in the target vehicle. In this case, the vehicle-mounted device 1 may obtain information output by the sensors provided in the target vehicle from the target vehicle based on a communication protocol such as CAN (Controller Area Network).

Figure 3 shows an example of the schematic configuration of the external terminal 2. The external terminal 2 mainly has a communication unit 21, a memory unit 22, an input unit 23, a control unit 24, a sensor group 25, a display unit 26, and a sound output unit 27. The elements in the external terminal 2 are connected to each other via a bus line 20.

The communication unit 21 performs data communication with other terminals under the control of the control unit 24. The storage unit 22 is composed of various types of memory such as RAM, ROM, and non-volatile memory. The storage unit 22 stores programs for the external terminal 2 to execute predetermined processes. The above-mentioned programs may include application programs for calling the driver, displaying information related to the driving of the target vehicle of the in-vehicle device 1 (including displaying maps and various captured images), and displaying information related to the driver's workload when communication with the in-vehicle device 1 is established. The storage unit 22 is also used as a working memory for the control unit 24. The programs executed by the external terminal 2 may be stored in a storage medium other than the storage unit 22.

The input unit 23 is a button, a touch panel, a remote controller, a voice input device, etc. that the user operates. The display unit 26 is a display or the like that displays information based on the control of the control unit 24. The sound output unit 27 is a speaker or the like that outputs sound based on the control of the control unit 24. The sensor group 25 includes an internal sensor that senses the state of the external terminal 2, and an external sensor that senses the state of the external world of the external terminal 2.

The control unit 24 includes a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), etc., and controls the entire external terminal 2. Note that the configuration of the external terminal 2 shown in FIG. 3 is an example, and various modifications may be made to the configuration shown in FIG. 3.

(1-3) Display of surrounding images
Next, the display of the peripheral captured image displayed by the in-vehicle device 1 on the external terminal 2 will be described. In summary, when the driver's workload is greater than a predetermined level during a call between the driver and the user of the external terminal 2, the in-vehicle device 1 determines the display range of the peripheral captured image so that the angle of view is wider than when the workload is equal to or less than the predetermined level. In this way, when the driver's workload is high, the in-vehicle device 1 allows the user of the external terminal 2 to view the peripheral captured image corresponding to the wider angle of view, thereby promoting the user of the external terminal 2 to assist in confirming the driver's safety. The driver's workload is an example of the "driver's state."

(1-3-1) Calculation of Workload Value First, a method for calculating the workload value, which is a value (score) indicating the degree of the workload, will be described. Hereinafter, for the sake of convenience, the workload value indicates that the higher the workload value, the higher the degree of the driver's workload.

The vehicle-mounted device 1 calculates a workload value based on a score (also called an "element-specific score") that evaluates the current state for each element related to the workload (also called "workload-related element"). The workload-related element is an element that represents the state related to the target vehicle, the driver, or the driving environment. Examples of the workload-related element include elements related to the environment of the road (driving road) on which the target vehicle is traveling (also called "road environment element"), elements related to driving behavior (driving operation) that affects the workload (also called "driving behavior element"), and elements related to the driver's physical condition (also called "driver physical condition element"). In other words, the workload-related element is an element that is a candidate for the cause of an increase in the workload. Note that the workload-related element may be an element obtained by further subdividing at least one of the road environment element, the driving behavior element, and the driver's physical condition element.

Therefore, for example, while communication for a call is established with the external terminal 2, the vehicle-mounted device 1 calculates a score (also called an "element-specific score") that evaluates the current state of the target vehicle, the driver, or the driving environment for each workload-related element. The vehicle-mounted device 1 then determines the total value, average value, or other representative value of the calculated element-specific scores as the workload value. Note that the workload-related elements for which the element-specific scores are calculated may be all of the road environment elements, driving behavior elements, and driver physical condition elements, or any one of these, or any two of them. In addition, the element-specific scores may be preferably weighted according to the degree of influence of each workload-related element on the workload. In this case, for example, the weighting coefficient multiplied by the element-specific score of an element with a high degree of influence on the workload is a larger value, and information indicating these weighting coefficients is stored in advance in the storage unit 12, etc.

Here, when the vehicle-mounted device 1 calculates the element-specific score of the road environment element, for example, the vehicle-mounted device 1 calculates the element-specific score by taking into account at least one of the congestion degree, width, speed limit, and visibility of the road to be traveled. In this case, for example, the vehicle-mounted device 1 refers to the road information in the map DB 4 or the road traffic information acquired by the communication unit 11, and identifies the degree of congestion degree, width, speed limit, visibility, etc. of the road to be traveled, and converts each identified degree into an element-specific score by referring to a predetermined formula or table. The predetermined formula or table is stored in the storage unit 12 or the like in advance. Note that the road environment element may be subdivided into multiple workload-related elements. In this case, for example, the vehicle-mounted device 1 may regard each of the congestion degree, width, speed limit, and visibility of the road to be traveled as a workload-related element, and calculate the element-specific score for each of these.

In addition, when the vehicle-mounted device 1 calculates the element-specific score of the driving operation element, for example, the vehicle-mounted device 1 determines whether the current driving operation corresponds to a driving operation such as a right turn, a left turn, a merge, or a temporary stop, and calculates the element-specific score according to the determined driving operation. In this case, for example, the vehicle-mounted device 1 determines the element-specific score of the driving operation element by referring to a table or the like that associates the corresponding driving operation with the element-specific score of the driving operation element. Here, the vehicle-mounted device 1 may determine the current driving operation based on information obtained by the route guidance process (for example, the current position of the target vehicle and the route to the set destination), or may determine the current driving operation based on a signal indicating the state of the turn signal or the steering state obtained from the target vehicle. In addition, the vehicle-mounted device 1 may determine the element-specific score of the driving operation element based on the presence or absence of auto-cruise driving (or the presence or absence of other automatic driving functions). The driving operation element may be subdivided into multiple workload-related elements. For example, whether or not to turn right, whether or not to turn left, whether or not to merge, whether or not to stop, and whether or not to drive on auto-cruise are each set as workload-related elements, and the vehicle-mounted device 1 determines the element-specific score for each of these workload-related elements by referring to a predetermined table or the like, depending on the presence or absence of the corresponding driving action.

Furthermore, when calculating the element-specific score of the driver's physical condition element, the vehicle-mounted device 1 calculates the element-specific score by taking into account at least one of the driver's continuous driving time, drowsiness level (alertness level), and other bioindicators that affect driving. In this case, the vehicle-mounted device 1, for example, identifies the drowsiness level or other bioindicators that affect driving based on the signal output by the biosensor 54. The vehicle-mounted device 1 then refers to a predetermined formula or table and converts the identified continuous driving time and one or more bioindicators into an element-specific score of the driver's physical condition element. Note that the driver's physical condition element may be subdivided into multiple workload-related elements. In this case, for example, the vehicle-mounted device 1 may regard each of the drowsiness level and the continuous driving time as a workload-related element and calculate the element-specific score for each of these.

By doing the above, the vehicle-mounted device 1 can determine the workload value by taking into account various factors that affect the workload.

(1-3-2) Determining the display range of peripheral captured images Next, a method for determining the display range of peripheral captured images to be displayed on the external terminal 2 by the in-vehicle device 1 will be described. When the workload value is equal to or less than a predetermined threshold (also referred to as the "workload threshold"), the in-vehicle device 1 causes the external terminal 2 to display peripheral captured images corresponding to a portion of the angle of view (also referred to as the "normal angle of view") relative to the overall angle of view captured. On the other hand, when the workload value is greater than the workload threshold, the in-vehicle device 1 causes the external terminal 2 to display peripheral captured images corresponding to an angle of view (also referred to as the "wide angle of view") wider than the normal angle of view. The workload threshold is stored in advance in, for example, the storage unit 12.

Below, a specific description will be given of a method for determining the display peripheral image when one exterior camera 51 is provided and when multiple exterior cameras 51 are provided. Hereinafter, the display mode that displays the peripheral image corresponding to the normal angle of view will be referred to as the "normal angle of view mode," and the display mode that displays the peripheral image corresponding to the wide angle of view will be referred to as the "wide angle of view mode." In both the normal angle of view mode and the wide angle of view mode, the display range of the peripheral image includes at least the forward direction of the target vehicle, and the vehicle-mounted device 1 changes the width of the display range of the peripheral image based on the forward direction of the target vehicle depending on whether the normal angle of view mode or the wide angle of view mode is selected.

Figure 4 (A) is an overhead view of the target vehicle, clearly showing the exterior camera 51 installed on the target vehicle and the imaging range "R" of the exterior camera 51. Figure 4 (B) is an overhead view of the target vehicle, clearly showing the display range of the peripheral image in both the normal angle of view mode and the wide angle of view mode. The imaging range R is the range represented in the original image (i.e., the image before the angle of view is adjusted) output by the exterior camera 51.

In the example shown in FIG. 4(A) and FIG. 4(B), in the normal angle of view mode, the vehicle-mounted device 1 determines the central range "R1" of the shooting range R, which corresponds to the angle of view of the center part, as the display range of the peripheral captured image. In this case, for example, the vehicle-mounted device 1 cuts out an image corresponding to the central range R1 from the original image corresponding to the shooting range R, and transmits display instruction data to the external terminal 2, specifying the cut-out image as the peripheral captured image for display. In other words, the vehicle-mounted device 1 determines a partial image based on the front of the original image output by the external camera 51 as the peripheral captured image for display. Note that the vehicle-mounted device 1 may include information specifying the peripheral captured image and its display range (i.e., the range to be cut out) in the display instruction data, instead of including the peripheral captured image for display cut out from the peripheral captured image, which is the original image, in the display instruction data.

On the other hand, in the wide-angle mode, the vehicle-mounted device 1 determines the peripheral image, which is the original image corresponding to the shooting range R, as the peripheral image for display. In this case, the peripheral image for display is an image corresponding to the range obtained by adding the left range "R2" to the left of the central range R1 and the right range "R3" to the right of the central range R1 (i.e., the entire range of the shooting range R).

Figure 5 (A) is an overhead view of the target vehicle, clearly showing the three exterior cameras 51 (51F, 51L, 51R) installed on the target vehicle and the shooting range R (RF, RL, RR) of each exterior camera 51. Figure 5 (B) is an overhead view of the target vehicle, clearly showing the range displayed as the surrounding shot image in both the normal angle of view mode and the wide angle of view mode.

In the example shown in FIG. 5(A) and FIG. 5(B), in the normal angle of view mode, the vehicle-mounted device 1 determines the image corresponding to the shooting range "RF" of the exterior camera 51F, which captures the area ahead of the target vehicle, as the surrounding captured image for display. In this case, the vehicle-mounted device 1 determines the original image output by the exterior camera 51F as the surrounding captured image for display.

On the other hand, in the wide angle mode, the vehicle-mounted device 1 determines the images corresponding to the shooting range RF of the outside-vehicle camera 51F, the shooting range "RL" of the outside-vehicle camera 51L, and the shooting range "RR" of the outside-vehicle camera 51R as the surrounding captured images to be displayed. In this case, the vehicle-mounted device 1 generates one panoramic image by synthesizing (splicing together) the images output by the outside-vehicle camera 51F, the images output by the outside-vehicle camera 51L, and the images output by the outside-vehicle camera 51R using any image synthesis technology. Note that, if information indicating the installation position, orientation, and other parameters (including internal parameters and external parameters) of each outside-vehicle camera 51 relative to the target vehicle is stored in the storage unit 12, the vehicle-mounted device 1 may generate the panoramic image using this information. The above panoramic image is an example of a surrounding captured image and a synthesized image.

The display peripheral image in the wide angle of view mode does not need to be the entire range of the panoramic image, but may be a partial range corresponding to a wider angle of view than the display peripheral image in the normal angle of view mode. Similarly, the display peripheral image in the normal angle of view mode is not limited to the original image output by the exterior camera 51F, but may be a partial image obtained by cutting out a portion of the panoramic image corresponding to the forward direction of the target vehicle. Furthermore, the number and installation positions of the exterior cameras 51 are not limited to those shown in Figures 4(A) and 4(B), and may be three or more, and may be installed at any position on the target vehicle.

(1-4) Display example
6 shows a first display example of the external terminal 2 during execution of the normal angle of view mode. The external terminal 2 has established communication with the in-vehicle device 1, and displays the display screen shown in FIG. 6 on the display unit 26 based on display instruction data received from the in-vehicle device 1.

Here, the display screen shown in FIG. 6 has a surrounding captured image display area 60, a supplementary information display area 61, and a map display area 62.

The external terminal 2 displays the latest surrounding captured image for display based on the display instruction data on the surrounding captured image display area 60. In this example, the vehicle-mounted device 1 determines that the workload value is equal to or less than the workload threshold, and executes the normal angle of view mode. Therefore, the vehicle-mounted device 1 transmits display instruction data including the latest surrounding captured image for display having the normal angle of view to the external terminal 2. The vehicle-mounted device 1 may transmit to the external terminal 2 the surrounding captured image output by the external camera 51 and display instruction data specifying the display range.

In addition, the external terminal 2 displays a workload display window 71, an indicator 72, and a field of view mode display window 73 in the supplemental information display area 61 based on the display instruction data transmitted by the vehicle-mounted device 1.

The workload display window 71 displays the level of the workload value calculated by the vehicle-mounted device 1. Here, since the workload value calculated by the vehicle-mounted device 1 is level 2 out of levels 1 to 5, the workload display window 71 displays content corresponding to level 2. The indicator 72 indicates the degree of workload, and the greater the workload value calculated by the vehicle-mounted device 1, the more extended the gauge is displayed to the right. The color of the gauge may change depending on the workload value. For example, the higher the workload value, the more noticeable the gauge may be displayed in a color. The angle of view mode display window 73 indicates the current display mode of the surrounding captured image for display (here, normal angle of view mode).

The external terminal 2 also displays a map of the vicinity of the current position of the target vehicle on the map display area 62. The in-vehicle device 1 generates display instruction data for the external terminal 2 to display the above-mentioned map based on, for example, the state of the target vehicle estimated based on the vehicle behavior detector 53, the map DB 4, and information related to route guidance to the destination, and transmits the display instruction data to the external terminal 2. Note that here, in the map display area 62, a current position mark 77 indicating the current position of the target vehicle and a route line 78 indicating the guided route along which the in-vehicle device 1 will guide the driver of the target vehicle are superimposed on the map.

In this way, in the first display example, when the workload value is equal to or less than the workload threshold, the in-vehicle device 1 executes the normal angle of view mode and presents a map of the area around the current location of the target vehicle, information about the workload value, and the like, together with a captured image of the surrounding area for display, to the user of the external terminal 2. This allows the user of the external terminal 2 to conveniently check the current state of the target vehicle and the driver.

Figure 7 shows a second display example of the external terminal 2 when the wide angle mode is being executed. The external terminal 2 has established communication with the in-vehicle device 1, and displays the display screen shown in Figure 7 on the display unit 26 based on the display instruction data received from the in-vehicle device 1. The external terminal 2 has a surrounding captured image display area 60A and a supplemental information display area 61 on the display screen.

The external terminal 2 displays the latest surrounding captured image for display on the surrounding captured image display area 60A based on the display instruction data received from the vehicle-mounted device 1. In this example, the vehicle-mounted device 1 determines that the workload value is greater than the workload threshold and executes the wide angle mode. Therefore, the vehicle-mounted device 1 transmits display instruction data including the surrounding captured image for display having a wide angle of view to the external terminal 2. The vehicle-mounted device 1 may transmit to the external terminal 2 the surrounding captured image output by the external camera 51 and display instruction data specifying the display range.

In this case, the image size of the peripheral captured image for display in the wide angle of view mode is larger (i.e., the range displayed is wider) than the peripheral captured image for display in the normal angle of view mode. Therefore, here, the external terminal 2 provides a field combining the peripheral captured image display field 60 and the map display field 62 in the first display example of FIG. 6 when the normal angle of view mode is being executed as the peripheral captured image display field 60A, and displays the peripheral captured image for display with the wide angle of view in the peripheral captured image display field 60A.

The external terminal 2 also displays a workload display window 71 and an indicator 72 indicating the workload value or the level of the workload value (here, level 4) on the supplemental information display field 61 based on the display instruction data received from the vehicle-mounted device 1. Furthermore, the angle of view mode display window 73 indicates the current display mode of the peripheral captured image for display (here, wide angle of view mode).

In this way, in the second display example, when the workload value is greater than the workload threshold, the in-vehicle device 1 executes the wide angle of view mode and presents the user of the external terminal 2 with a surrounding captured image with an expanded angle of view. This allows the user of the external terminal 2 to check the surroundings of the target vehicle with a wide field of view that includes the direction ahead of the target vehicle, and can ideally assist the driver in checking for safety. In addition, the driver can feel a sense of security as if a passenger is watching over him or her, and can feel a sense of unity as if the user of the external terminal 2 is actually riding with him or her.

The display layouts in the first and second display examples are merely examples, and various modifications may be applied. For example, the exterior terminal 2 may display an image captured by the driver instead of a map, based on an operation on the exterior terminal 2 by the user of the exterior terminal 2. In another example, the exterior terminal 2 may not display the supplemental information display field 61 while the wide angle mode (and the normal angle mode) is being executed. In yet another example, the exterior terminal 2 may display the map display field 62 while the wide angle mode is being executed, in a smaller size than during the normal angle mode.

(1-5) Processing flow
Fig. 8 is an example of a flowchart showing a procedure of a process executed by the vehicle-mounted device 1. The vehicle-mounted device 1 executes the process of the flowchart shown in Fig. 8 when communication with the external terminal 2 is established and a call is started.

First, the vehicle-mounted device 1 calculates the score of each workload-related element related to the workload of the driver of the target vehicle (i.e., element-specific score) (step S101). In this case, the vehicle-mounted device 1 calculates element-specific scores for road environment elements, driving behavior elements, driver physical condition elements, or elements that are further subdivided from these, based on the map DB 4 and data output by the sensor group 15.

Next, the vehicle-mounted device 1 calculates a workload value based on the scores of each workload-related element (element-specific scores) (step S102). In this case, the vehicle-mounted device 1 may calculate a workload value by adding up or averaging (including using a weighting factor) the element-specific scores, or may calculate the workload value by substituting each element-specific score into a formula for calculating the workload value.

Next, the vehicle-mounted device 1 determines whether the workload value is greater than the workload threshold (step S103). If the workload value is greater than the workload threshold (step S103; Yes), the vehicle-mounted device 1 determines that the wide-angle mode should be executed, and supplies display instruction data to the external terminal 2 that instructs the external terminal 2 to display a wide-angle peripheral image for display (step S104).

On the other hand, if the workload value is equal to or less than the workload threshold (step S103; No), the in-vehicle device 1 determines that the normal angle of view mode should be executed, and supplies the external terminal 2 with display instruction data instructing the external terminal 2 to display the surrounding captured image for display at the normal angle of view (step S105). As a result, the external terminal 2 displays the surrounding captured image for display at the normal angle of view based on the display instruction data. Note that in step S104 or step S105, if the display instruction data includes at least one of information related to the workload, map display information for displaying a map around the current position, or an image captured by the driver, the external terminal 2 may perform display based on at least one of these pieces of information.

Then, the vehicle-mounted device 1 determines whether the call has ended (step S106). If the vehicle-mounted device 1 determines that the call has ended (step S106; Yes), it ends the processing of the flowchart. On the other hand, if the vehicle-mounted device 1 determines that the call has not ended (step S106; No), it returns the processing to step S101.

(1-6) Modified Example
Next, preferred modifications of the above-mentioned embodiment will be described. The following modifications may be combined and applied to the above-mentioned embodiment.

(Variation 1-1)
When the in-vehicle device 1 displays the peripheral captured image on the external terminal 2, the in-vehicle device 1 may superimpose a line (gaze tracking line) representing the recognized trajectory of the driver's gaze on the peripheral captured image.

Figure 9 shows a third display example of the external terminal 2 relating to variant example 1-1 in the wide angle of view mode. The external terminal 2 has established communication with the vehicle-mounted device 1, and displays the display screen shown in Figure 9 on the display unit 26 based on the display instruction data received from the vehicle-mounted device 1. Here, as an example, the external terminal 2 provides a peripheral captured image display area 60A that displays peripheral captured images with a wider angle of view than normal, and a supplementary information display area 61 on the display screen based on the display instruction data received from the vehicle-mounted device 1, similar to the second display example.

Here, in the peripheral image display area 60A, an eye tracking line 66 indicating the trajectory of the eye gaze direction of the driver of the target vehicle within the most recent specified period is superimposed on the latest peripheral image in correspondence with the eye gaze direction in the peripheral image. In this case, the vehicle-mounted device 1 detects the eye gaze direction on the peripheral image captured at the same time as the driver's image based on any eye gaze detection technology, and generates eye gaze direction data indicating the detected eye gaze direction. Then, the vehicle-mounted device 1 superimposes the eye gaze tracking line 66 indicating the eye gaze direction of the driver within the most recent specified period indicated by the eye gaze direction data on the peripheral image included in the display instruction data to be transmitted to the external terminal 2. The length of the above-mentioned specified period is stored in advance in the storage unit 12, for example. Then, the external terminal 2 that has received the display instruction data displays the peripheral image with the eye gaze tracking line 66 superimposed on it in the peripheral image display area 60.

The vehicle-mounted device 1 may, for example, store in advance face-gaze correspondence data that associates each position on the peripheral photographed image with a face image model when the driver is looking at that position, and generate gaze direction data based on the face-gaze correspondence data. In this case, the vehicle-mounted device 1 identifies the face image model that most closely matches the driver's face image extracted from the driver's photographed image, and generates gaze direction data that indicates the position on the peripheral photographed image that corresponds to the identified face image model.

In this way, according to this modified example, the user of the external terminal 2 can easily grasp the direction in which the driver is not paying much attention (the right direction in the example of FIG. 8), and the user of the external terminal 2 can assist the driver in checking for safety in the direction in which the driver is not paying much attention by talking to the driver. In addition, the driver can feel a sense of security as if a passenger is watching over him or her, and can feel a sense of unity as if the user of the external terminal 2 is actually riding with the driver.

In a preferred example, the vehicle-mounted device 1 may change the display mode (display color, display on/off, etc.) of the gaze tracking line according to the workload value. For example, the vehicle-mounted device 1 superimposes the gaze tracking line on the peripheral captured image when the workload value is equal to or greater than a predetermined threshold, and hides the gaze tracking line when the workload value is less than the threshold. The above-mentioned predetermined threshold may be the same as the workload threshold, or may be different. In this way, the vehicle-mounted device 1 can display the gaze tracking line only when there is a high probability that the user of the external terminal 2 needs assistance in safety confirmation. On the other hand, during normal times when the user of the external terminal 2 does not need assistance in safety confirmation, the user of the external terminal 2 can enjoy the scenery from the peripheral captured image. In another example, the vehicle-mounted device 1 may display the gaze tracking line in a color according to the workload value. In this case, the vehicle-mounted device 1 displays the gaze tracking line in a color that is more noticeable the higher the workload value, for example.

In addition, when displaying the gaze tracking line, the vehicle-mounted device 1 may highlight an object (also called a "gaze target") to which the driver should direct his/her gaze in the surrounding captured image and display it on the external terminal 2. The gaze target includes, for example, a traffic light, a road sign, an obstacle, a pedestrian, etc. In this case, the vehicle-mounted device 1 uses any object recognition technology (including those using deep learning models such as instance segmentation) to extract objects corresponding to a type determined in advance as a gaze target from the surrounding captured image. Then, the vehicle-mounted device 1 processes the surrounding captured image, for example, to highlight the area of the target on the extracted surrounding captured image by edging, and to superimpose the gaze tracking line, and transmits the processed surrounding captured image to the external terminal 2 together with the display instruction data. The external terminal 2 that has received the display instruction data displays the surrounding captured image with the gaze tracking line superimposed and the gaze target highlighted. This allows the user of the external terminal 2 to recognize the presence of an object that the driver is not looking at from the surrounding captured image, and to advise the driver to pay attention to the object.

(Variation 1-2)
The vehicle-mounted device 1 may switch between the normal angle-of-view mode and the wide angle-of-view mode when a predetermined operation by the driver is detected, instead of switching between the normal angle-of-view mode and the wide angle-of-view mode based on the workload value.

For example, the vehicle-mounted device 1 has a voice input device as the input unit 13, and when the voice input device detects an utterance containing a predetermined keyword from the driver, the vehicle-mounted device 1 switches from the normal angle of view mode to the wide angle of view mode. For example, when the vehicle-mounted device 1 detects an utterance such as "Please help me check safety" or a similar utterance, the vehicle-mounted device 1 switches from the normal angle of view mode to the wide angle of view mode. Note that when the vehicle-mounted device 1 detects an utterance containing a predetermined keyword (such as "That's enough," "Thank you") that signals the end of the safety check, the vehicle-mounted device 1 may switch from the wide angle of view mode back to the normal angle of view mode.

In addition, when a specific operation is detected on the input unit 13, which is a type of interface (e.g., a button, etc.), not limited to speech, the vehicle-mounted device 1 may switch between the normal angle of view mode and the wide angle of view mode.

Figure 10 is an example of a flowchart related to this modified example. The vehicle-mounted device 1 executes the process of the flowchart shown in Figure 10 when communication with the external terminal 2 is established and a call is started.

First, the vehicle-mounted device 1 determines whether or not a specific operation by the driver requesting driving assistance has been detected (step S111). The vehicle-mounted device 1 determines whether or not the specific operation has been performed based on an input signal generated by the input unit 13.

When the vehicle-mounted device 1 detects a predetermined operation by the driver requesting driving assistance (step S111; Yes), it determines that the wide-angle mode should be executed, and supplies the vehicle-mounted device 2 with display instruction data that instructs the vehicle-mounted device 2 to display a surrounding image for display with a wider angle of view than usual (step S112). In this case, the vehicle-mounted device 1 may continue to execute step S112 until it detects a predetermined operation to cancel the request for driving assistance.

On the other hand, if the in-vehicle device 1 does not detect a specific operation by the driver requesting driving assistance (step S111; No), it determines that the normal angle of view mode should be executed, and supplies display instruction data to the external terminal 2 instructing the external terminal 2 to display the surrounding captured image for display at the normal angle of view (step S113).

Then, the vehicle-mounted device 1 determines whether the call has ended (step S114). If the vehicle-mounted device 1 determines that the call has ended (step S114; Yes), it ends the processing of the flowchart. On the other hand, if the vehicle-mounted device 1 determines that the call has not ended (step S114; No), it returns the processing to step S111.

In this manner, in this modified example, the in-vehicle device 1 executes the wide-angle mode in response to a request from the driver, and presents the user of the external terminal 2 with a captured image of the surroundings with an expanded angle of view. This allows the user of the external terminal 2 to favorably assist the driver in checking for safety with a wide field of view. The driver can also feel a sense of security as if a passenger is watching over them, and can experience a sense of unity as if the user of the external terminal 2 is actually riding with them.

(Modification 1-3)
Instead of changing the angle of view of the peripheral captured image for display depending on whether the workload value is greater than the workload threshold, the vehicle-mounted device 1 may change the angle of view of the peripheral captured image for display continuously or stepwise (in three or more steps) depending on the workload value.

Even in this case, the vehicle-mounted device 1 causes the external terminal 2 to display the display peripheral captured image having a wider angle of view as the workload value increases. In other words, the vehicle-mounted device 1 expands the display range of the peripheral captured image in the left-right direction (i.e., the horizontal direction of the image) relative to the target vehicle as the workload value increases. Note that information indicating the angle of view according to the workload value may be stored in advance in, for example, the storage unit 12. Even in this manner, the vehicle-mounted device 1 can accurately determine the display range of the peripheral captured image according to the workload value.

(Modification 1-4)
At least a part of the processing executed by the vehicle-mounted device 1 may be executed by a server device that performs data communication with the vehicle-mounted device 1 and the external terminal 2 .

Figure 11 shows an example of the configuration of a driving call system according to variant 1-3. The driving call system has an in-vehicle device 1A, an external terminal 2, and a server device 5. The in-vehicle device 1A and the server device 5, and the external terminal 2 and the server device 5 each perform data communication via the communication network 3.

The vehicle-mounted device 1A has the same configuration as the vehicle-mounted device 1 described in the first embodiment above (see FIG. 2). Note that, if the server device 5 performs processing based on the map DB 4, the vehicle-mounted device 1A does not need to have the map DB 4. The vehicle-mounted device 1A then transmits to the server device 5 an upload signal including information output by the sensor group 15 and input information input by the input unit 13.

The server device 5 relays data necessary for a voice call between the in-vehicle device 1A and the external terminal 2. During a voice call, the server device 5 generates display instruction data based on an upload signal or the like received from the in-vehicle device 1A, and transmits the generated display instruction data to the external terminal 2. Specifically, during a voice call, the server device 5 executes the process of the flowchart shown in FIG. 8 or FIG. 10 based on an upload signal or the like received from the in-vehicle device 1A.

Figure 12 shows an example of the schematic configuration of the server device 5. The server device 5 mainly has a communication unit 41, a storage unit 42, and a control unit 44. The elements in the server device 5 are connected to each other via a bus line 40.

The communication unit 41 performs data communication with external devices such as the in-vehicle unit 1A and the external terminal 2 under the control of the control unit 44. The storage unit 42 is composed of various types of memory such as RAM, ROM, and non-volatile memory (including a hard disk drive, flash memory, etc.). The storage unit 42 stores programs for the server device 5 to execute predetermined processes. The storage unit 42 also includes a map DB 4. The control unit 44 includes a CPU, a GPU, etc., and controls the entire server device 5. The control unit 44 also executes the programs stored in the storage unit 42 to perform processes required to display the surrounding captured images for display on the external terminal 2.

In this way, even when the server device 5 executes the processing required to display the display peripheral captured image on the external terminal 2, the driving call system allows the user of the external terminal 2 to appropriately recognize the state of the driver's workload, and enables the user to choose a convenient time to talk to the driver. In this modified example, the server device 5 is an example of an "information processing device."

As described above, the in-vehicle device 1 or the server device 5 is an information processing device used in a system that mediates communication between the driver of a target vehicle, which is a moving body, and the user of the external terminal 2 located outside the target vehicle, and has an acquisition means, a detection means, and a display control means. The acquisition means acquires a surrounding image captured from the target vehicle. The detection means detects the state or operation of the driver of the target vehicle. The display control means causes the external terminal 2 to display a predetermined range of the surrounding image. In this case, the display control means determines the predetermined range of the surrounding image to be displayed on the external terminal 2 based on the state or operation of the driver. In this way, the in-vehicle device 1 determines the display range of the surrounding image according to the situation, and can cause the user of the external terminal 2 to perform assistance with safety confirmation, etc. as necessary.

Second Example
The vehicle-mounted device 1 in the second embodiment differs from the vehicle-mounted device 1 in the first embodiment in that the display surrounding image is determined based on the driver's line of sight. Hereinafter, the same components as those in the first embodiment are appropriately designated by the same reference numerals, and the description thereof will be omitted. In the following description, the driving call system according to the second embodiment is assumed to have the configuration shown in FIG. 1, and the vehicle-mounted device 1 and the external terminal 2 according to the second embodiment are assumed to have the configurations shown in FIG. 2 and FIG. 3, respectively.

(2-1) Display of surrounding images
In the second embodiment, when the workload value is higher than the workload threshold, the vehicle-mounted device 1 identifies a direction in which the driver has a low viewing frequency (also referred to as a "low viewing frequency direction"), and determines the range of the surrounding captured images corresponding to the identified direction as the display range of the surrounding captured images. The low viewing frequency direction is a direction (i.e., a bearing) on a horizontal plane based on the target vehicle, and is an example of the "first direction."

In this case, the vehicle-mounted device 1 determines the driver's gaze direction within the most recent specified period based on the method for detecting the driver's gaze direction described in the section "(Variation 1-1)". In this case, for example, the vehicle-mounted device 1 predetermines candidate directions (e.g., three directions, forward, left, and right, based on the target vehicle) that are candidates for the driver's gaze direction, and counts which candidate direction corresponds to the gaze direction samples detected at predetermined time intervals within the most recent specified period. Note that information on each candidate direction (e.g., the range of directions based on the target vehicle, etc.) is prestored in the storage unit 12, for example. Then, the vehicle-mounted device 1 determines the candidate direction with the fewest samples of the corresponding gaze direction as the low viewing frequency direction among all the candidate directions. Note that the vehicle-mounted device 1 may determine multiple low viewing frequency directions and display multiple corresponding surrounding captured images on the external terminal 2 by screen division. For example, if the vehicle-mounted device 1 determines that the driver is most frequently viewing the forward direction, it identifies the right and left directions as directions with a low viewing frequency, and displays the captured images of the surroundings corresponding to each direction on the external terminal 2 by splitting the screen into two.

Next, a method for determining the surrounding captured image to be displayed based on the identified low viewing frequency direction will be described. Below, a specific method for determining the surrounding captured image to be displayed when one exterior camera 51 is installed and when multiple exterior cameras 51 are installed will be described.

Figure 13 (A) is an overhead view of a target vehicle that clearly shows the exterior camera 51 installed on the target vehicle and the range of presence or absence of display in the shooting range R of the exterior camera 51 when the low visibility frequency direction is the right direction. Figure 13 (B) is an overhead view of a target vehicle that clearly shows the exterior camera 51 installed on the target vehicle and the range of presence or absence of display in the shooting range R of the exterior camera 51 when the low visibility frequency direction is the left direction.

13(A), the vehicle-mounted device 1 identifies the low visual frequency direction as the right direction (specifically, the right front of the target vehicle) and determines the range "R11" corresponding to the right direction within the shooting range R of the peripheral captured image generated by the exterior camera 51 as the display range of the peripheral captured image. In this case, for example, the vehicle-mounted device 1 cuts out an image corresponding to range R11 from the peripheral captured image corresponding to the shooting range R, and transmits display instruction data specifying the cut-out image as the peripheral captured image for display to the exterior terminal 2. Note that instead of including the peripheral captured image for display cut out from the peripheral captured image in the display instruction data, the vehicle-mounted device 1 may include information specifying the peripheral captured image and its display range (i.e., the range to be cut out) in the display instruction data.

13B, the vehicle-mounted device 1 determines that the low visibility direction is the left direction (specifically, the left front of the target vehicle) and determines the range "R12" corresponding to the left direction within the shooting range R of the peripheral captured image generated by the exterior camera 51 as the display range of the peripheral captured image. Note that when the workload value is equal to or less than the workload threshold, the vehicle-mounted device 1 determines, for example, a range including the forward direction within the shooting range R of the peripheral captured image (which may not overlap with the ranges R11 and R12, or may partially overlap with them) as the display range of the peripheral captured image.

In this way, when there is one exterior camera 51, the vehicle-mounted device 1 can suitably determine a partial image corresponding to a low-frequency viewing direction from the surrounding captured image output by the exterior camera 51 as the surrounding captured image to be displayed.

Figure 14 (A) is an overhead view of a target vehicle that clearly shows three exterior cameras 51F, 51L, and 51R installed on the target vehicle, the shooting ranges RF, RL, and RR of each exterior camera 51, and whether or not each range is displayed when the low visibility direction is to the right. Figure 14 (B) is an overhead view of a target vehicle that clearly shows three exterior cameras 51F, 51L, and 51R, the shooting ranges RF, RL, and RR of each exterior camera 51, and whether or not each range is displayed when the low visibility direction is to the left.

In the example shown in FIG. 14(A), the vehicle-mounted device 1 identifies the low visibility direction as the right direction, and determines the peripheral image generated by the exterior camera 51R that captures the right side as the display peripheral image. On the other hand, in the example shown in FIG. 14(B), the vehicle-mounted device 1 identifies the low visibility direction as the left direction, and determines the peripheral image generated by the exterior camera 51L that captures the left side as the display peripheral image. In addition, when the workload value is equal to or less than the workload threshold, the vehicle-mounted device 1 determines the peripheral image generated by the exterior camera 51F that captures the front as the display peripheral image.

The vehicle-mounted device 1 may generate a single panoramic image by synthesizing the images output by the exterior camera 51F, the exterior camera 51L, and the exterior camera 51R using any image synthesis technology, and determine the display range from the panoramic image. The above panoramic image is an example of a peripheral image and a synthetic image. In this case, the vehicle-mounted device 1 determines the range corresponding to the low viewing frequency direction in the generated panoramic image as the display range. Furthermore, even if the workload value is equal to or less than the workload threshold, the vehicle-mounted device 1 determines a partial range of the panoramic image (for example, a range corresponding to the forward direction of the target vehicle) as the display range. In this case, the display peripheral image corresponds to a partial image of the panoramic image.

In this way, even when there are multiple exterior camera 51, the vehicle-mounted device 1 can suitably determine, from the surrounding captured images output by each exterior camera 51, an image corresponding to a direction with a low viewing frequency as the surrounding captured image to be displayed.

(2-2) Display examples
Fig. 15 shows a first display example of the external terminal 2 in the second embodiment. The external terminal 2 has established communication with the vehicle-mounted device 1, and displays the display screen shown in Fig. 15 on the display unit 26 based on display instruction data received from the vehicle-mounted device 1.

Here, the display screen shown in FIG. 15 has a surrounding captured image display area 60, a supplementary information display area 61, and a map display area 62.

The external terminal 2 displays the latest display peripheral image based on the display instruction data on the peripheral image display field 60. In this example, the vehicle-mounted device 1 determines that the workload value is equal to or less than the workload threshold, does not identify the low viewing frequency direction, and determines the range of the peripheral image output by the external camera 51 that corresponds at least to the forward direction of the target vehicle as the display peripheral image. Note that, here, as shown in FIG. 14(A) and FIG. 14(B), the target vehicle is provided with the external camera 51F, the external camera 51R, and the external camera 51L, and the vehicle-mounted device 1 determines the peripheral image output by the external camera 51F as the display peripheral image. The vehicle-mounted device 1 then transmits display instruction data including the latest display peripheral image to the external terminal 2.

In addition, based on the display instruction data transmitted by the vehicle-mounted device 1, the external terminal 2 displays a workload display window 71 and an indicator 72 that display information about the driver's workload, and a camera usage display window 74 in the supplemental information display area 61.

The camera in use display window 74 displays information identifying the camera that is outputting the display peripheral image displayed in the peripheral image display field 60. Here, the peripheral image output by the exterior camera 51F, which captures the area in front of the target vehicle, is used as the display peripheral image, so the camera in use display window 74 displays "Front camera display," indicating that the exterior camera 51F is being used.

When the external terminal 2 displays a map near the current position of the target vehicle on the map display field 62, the external terminal 2 clearly indicates a display angle range 79 indicating an area on the map corresponding to the display peripheral image (i.e., a direction with low visibility frequency) together with the current position mark 77 and the route line 78. In this case, for example, the vehicle-mounted device 1 specifies the shooting angle of the external camera 51F that captured the display peripheral image, and determines the display angle range 79 on the map based on the current position and traveling direction of the target vehicle specified by the vehicle behavior detector 53 based on the specified information. Note that the storage unit 12 may store information indicating the shooting angle of each external camera 51. In this way, the vehicle-mounted device 1 displays a map indicating the display angle range 79 corresponding to the direction with low visibility frequency together with the display peripheral image. As a result, even if the shooting direction of the display peripheral image to be displayed in the peripheral image display field 60 is changed as in the display example described later, the user of the external terminal 2 can appropriately recognize the display range after the change.

Figure 16 shows a second display example of the external terminal 2 in the second embodiment. The external terminal 2 has established communication with the vehicle-mounted device 1, and displays the display screen shown in Figure 16 on the display unit 26 based on the display instruction data received from the vehicle-mounted device 1.

The external terminal 2 displays the latest surrounding captured image for display based on the display instruction data on the surrounding captured image display field 60. In this example, the vehicle-mounted device 1 determines that the workload value is greater than the workload threshold value, and identifies the right direction as the direction of low viewing frequency. Therefore, the vehicle-mounted device 1 determines the range of the surrounding captured images output by the external camera 51 (51F, 51R, 51L) that corresponds to the right direction of the target vehicle as the surrounding captured image for display. Here, the vehicle-mounted device 1 determines the surrounding captured image output by the external camera 51R that captures the right direction of the target vehicle as the surrounding captured image for display. Then, the vehicle-mounted device 1 transmits display instruction data including the latest surrounding captured image for display to the external terminal 2.

In addition, based on the display instruction data transmitted by the vehicle-mounted device 1, the external terminal 2 displays, on the supplemental information display field 61, a workload display window 71 and an indicator 72 that display information related to the driver's workload, and a camera in use display window 74. Here, since the peripheral captured image output by the external camera 51R that captures the right direction of the target vehicle is used as the peripheral captured image for display, the camera in use display window 74 displays "Right camera display," indicating that the external camera 51R is being used.

When the external terminal 2 displays a map near the current position of the target vehicle on the map display field 62, it indicates a display angle range 79 indicating the area on the map corresponding to the display peripheral image together with the current position mark 77 and the route line 78. In this case, for example, the vehicle-mounted device 1 identifies the shooting angle of view of the external camera 51R that captured the display peripheral image, and based on the identified information, determines the display angle range 79 on the map based on the current position and traveling direction of the target vehicle identified by the vehicle behavior detector 53. Note that the memory unit 12 may store information indicating the shooting angle of view of each external camera 51. In this way, the vehicle-mounted device 1 displays a map indicating the current position of the target vehicle and the low visibility direction together with the display peripheral image.

As described above, in the second display example, when the driver's load is high, the in-vehicle device 1 causes the external terminal 2 to display peripheral images corresponding to a direction with a low viewing frequency. This allows the user of the external terminal 2 to assist the driver in checking for safety, and the driver can feel a sense of security as if a passenger is watching over him or her, and can experience a sense of unity as if the user of the external terminal 2 is actually riding with the driver. In addition, the in-vehicle device 1 clearly displays the area on the map that corresponds to the peripheral images for display as the display angle range 79, allowing the user of the external terminal 2 to easily recognize the shooting direction of the currently displayed peripheral images for display, etc.

(2-3) Processing flow
Fig. 17 is an example of a flowchart showing the procedure of a process executed by the vehicle-mounted device 1 in Example 2. The vehicle-mounted device 1 executes the process of the flowchart shown in Fig. 17 when communication with the external terminal 2 is established and a call is started.

First, the vehicle-mounted device 1 calculates the score of each workload-related element related to the workload of the driver of the target vehicle (i.e., element-specific score) (step S201). In this case, the vehicle-mounted device 1 calculates element-specific scores for road environment elements, driving behavior elements, driver physical condition elements, or elements that are further subdivided from these, based on the map DB 4 and data output by the sensor group 15.

Next, the vehicle-mounted device 1 calculates a workload value based on the scores of each workload-related element (element-specific scores) (step S202). In this case, the vehicle-mounted device 1 may calculate a workload value by adding up or averaging (including using a weighting factor) the element-specific scores, or may calculate the workload value by substituting each element-specific score into a formula for calculating the workload value.

Next, the vehicle-mounted device 1 determines whether the workload value is greater than the workload threshold (step S203). If the workload value is greater than the workload threshold (step S203; Yes), the vehicle-mounted device 1 counts the frequency of the driver's line of sight direction within a predetermined period and determines the low viewing frequency direction (step S204). The vehicle-mounted device 1 then supplies the external terminal 2 with display instruction data that instructs the external terminal 2 to display the display surrounding image corresponding to the low viewing frequency direction (step S204). As a result, the external terminal 2 displays the display surrounding image corresponding to the low viewing frequency direction based on the display instruction data. Note that if the display instruction data includes at least one of information related to the workload, map display information for displaying a map around the current position, or a driver-captured image, the external terminal 2 may perform display based on at least one of these pieces of information.

On the other hand, if the workload value is equal to or less than the workload threshold (step S203; No), the in-vehicle device 1 supplies the external terminal 2 with display instruction data instructing the external terminal 2 to display a display peripheral image corresponding to a predetermined direction (e.g., the forward direction of the target vehicle) (step S205). As a result, the external terminal 2 displays a display peripheral image in the predetermined direction based on the display instruction data. Note that if the display instruction data includes at least one of information related to the workload, map display information for displaying a map around the current position, and an image captured by the driver, the external terminal 2 may perform a display based on at least one of these pieces of information.

Then, the vehicle-mounted device 1 determines whether the call has ended (step S206). If the vehicle-mounted device 1 determines that the call has ended (step S206; Yes), it ends the processing of the flowchart. On the other hand, if the vehicle-mounted device 1 determines that the call has not ended (step S206; No), it returns the processing to step S201.

(2-4) Modifications
Next, preferred modifications of the above-mentioned embodiment will be described. The following modifications may be combined and applied to the above-mentioned embodiment.

(Variation 2-1)
Although the vehicle-mounted device 1 determines whether or not to display the peripheral captured image corresponding to the low viewing frequency direction based on the workload value, the vehicle-mounted device 1 may determine whether or not to display the peripheral captured image corresponding to the low viewing frequency direction regardless of the workload value. For example, the vehicle-mounted device 1 may determine whether or not to display the peripheral captured image corresponding to the low viewing frequency direction based on a user's operation, or may always display the peripheral captured image corresponding to the low viewing frequency direction.

(Variation 2-2)
Instead of displaying the surrounding captured image corresponding to the low-visibility direction, when a predetermined operation of the driver specifying a direction in which driving assistance is required is detected, the vehicle-mounted device 1 may display the surrounding captured image corresponding to the specified direction. The specified direction is a direction based on the target vehicle and is an example of the "first direction."

For example, the vehicle-mounted device 1 has a voice input device as the input unit 13, and when the vehicle-mounted device detects an utterance that includes a specific keyword from the driver and specifies a direction, the vehicle-mounted device 1 displays a captured image of the surroundings according to the specified direction. For example, when the vehicle-mounted device 1 detects an utterance such as "Please help me check for safety to the right" or a similar utterance, the vehicle-mounted device 1 displays a captured image of the surroundings according to the right direction.

The predetermined operation by the driver to specify the direction in which driving assistance is required is not limited to speech, but may be an input operation to the input unit 13, which serves as various interfaces (e.g., buttons, etc.).

Figure 18 is an example of a flowchart related to this modified example. The vehicle-mounted device 1 executes the process of the flowchart shown in Figure 18 when communication with the external terminal 2 is established and a call is started.

First, the vehicle-mounted device 1 determines whether or not a predetermined operation by the driver specifying a direction in which driving assistance is required has been detected (step S211). The vehicle-mounted device 1 determines whether or not the predetermined operation has been performed based on an input signal generated by the input unit 13.

When the vehicle-mounted device 1 detects a predetermined operation by the driver specifying a direction in which driving assistance is required (step S211; Yes), the vehicle-mounted device 1 supplies the vehicle-mounted terminal 2 with display instruction data that instructs the vehicle-mounted terminal 2 to display a display-use surrounding captured image corresponding to the specified direction (step S212). In this case, the vehicle-mounted device 1 may continue to execute step S212 until it detects a predetermined operation to cancel the request for driving assistance or until a preset time has elapsed.

On the other hand, if the in-vehicle device 1 does not detect a predetermined operation by the driver specifying a direction in which driving assistance is required (step S211; No), it supplies display instruction data to the external terminal 2 instructing the external terminal 2 to display a surrounding captured image for display corresponding to the default direction (step S213).

Then, the vehicle-mounted device 1 determines whether the call has ended (step S214). If the vehicle-mounted device 1 determines that the call has ended (step S214; Yes), it ends the processing of the flowchart. On the other hand, if the vehicle-mounted device 1 determines that the call has not ended (step S214; No), it returns the processing to step S211.

In this manner, in this modified example, the in-vehicle device 1 presents the user of the external terminal 2 with surrounding captured images corresponding to the direction specified by the driver. This allows the user of the external terminal 2 to appropriately assist the driver in checking for safety according to the driver's request. The driver can also feel a sense of security as if a passenger is watching over them, and can experience a sense of unity as if the user of the external terminal 2 is actually riding with them.

(Variation 2-3)
At least a part of the processing executed by the vehicle-mounted device 1 may be executed by a server device that performs data communication with the vehicle-mounted device 1 and the external terminal 2 .

In this case, the driving call system according to this modified example has a configuration shown in FIG. 11, for example. The in-vehicle device 1 transmits an upload signal including information output by the sensor group 15 and input information input by the input unit 13 to the server device 5. The server device 5 relays data required for a voice call between the in-vehicle device 1 and the external terminal 2. During a voice call, the server device 5 generates display instruction data based on the upload signal received from the in-vehicle device 1, and transmits the generated display instruction data to the external terminal 2. Specifically, during a voice call, the server device 5 executes the process of the flowchart shown in FIG. 17 or 18 based on the upload signal received from the in-vehicle device 1A.

In this way, even if the server device executes the processing required to display the display peripheral captured image on the external terminal 2, the driving call system allows the user of the external terminal 2 to appropriately recognize the driver's workload state, and enables the user to choose a convenient time to talk to the driver. In this modified example, the server device 5 is an example of an "information processing device."

As described above, the in-vehicle device 1 or the server device 5 is an information processing device used in a system that mediates communication between the driver of a target vehicle, which is a moving body, and the user of the external terminal 2 located outside the target vehicle, and has an acquisition means, a detection means, and a display control means. The acquisition means acquires a surrounding image captured from the target vehicle. The detection means detects the state or operation of the driver of the target vehicle. In this case, the display control means displays an image based on the surrounding captured image on the external terminal 2 according to the state or operation of the driver. In this case, the display control means specifies a first direction based on the target vehicle based on the state or operation of the driver, and determines the range of the surrounding captured image in which the first direction is captured as the image to be displayed on the external terminal 2. This allows the in-vehicle device 1 to preferably perform assistance for safety confirmation, etc. for the user of the external terminal 2.

In each of the above-mentioned embodiments, the program can be stored using various types of non-transitory computer readable media and supplied to a control unit, which is a computer. Non-transitory computer readable media include various types of tangible storage media. Examples of non-transitory computer readable media include magnetic storage media (e.g., flexible disks, magnetic tapes, hard disk drives), magneto-optical storage media (e.g., magneto-optical disks), CD-ROMs (Read Only Memory), CD-Rs, CD-R/Ws, and semiconductor memories (e.g., mask ROMs, PROMs (Programmable ROMs), EPROMs (Erasable PROMs), flash ROMs, and RAMs (Random Access Memory).

Although the present invention has been described above with reference to the embodiments, the present invention is not limited to the above-mentioned embodiments. Various modifications that a person skilled in the art can understand can be made to the configuration and details of the present invention within the scope of the present invention. In other words, the present invention naturally includes various modifications and amendments that a person skilled in the art could make in accordance with the entire disclosure, including the scope of the claims, and the technical ideas. In addition, the disclosures of the above cited patent documents and the like are incorporated into this document by reference.

1, 1A Vehicle-mounted device 2 External terminal 3 Communication network 4 Map DB
5 Server device 11, 21, 41 Communication unit 12, 22, 42 Storage unit 13 Input unit 14, 24, 44 Control unit 15, 25 Sensor group 16, 26 Display unit 17, 27 Sound output unit

Claims (10)

An information processing device used in a system that mediates communication between a driver of a moving body and a user outside the moving body, comprising:
An acquisition means for acquiring a peripheral image of the moving body by capturing an image of the surroundings of the moving body from the moving body;
A detection means for detecting a state or operation of a driver of the vehicle;
a display control means for displaying a predetermined range in the peripheral photographed image on a terminal device of the user;
The display control means is an information processing device that determines the predetermined range to be displayed on the terminal device based on a state or operation of the driver. The information processing device according to claim 1, wherein the display control means determines the predetermined range based on the degree of driving load of the driver. The information processing device according to claim 2, wherein the display control means expands the predetermined range as the degree of the driving load increases. The information processing device according to claim 1, wherein the display control means expands the predetermined range based on a predetermined utterance of the driver as an operation of the driver. The information processing device according to claim 3 or 4, wherein the display control means causes the terminal device to display a line representing the trajectory of the driver's line of sight on the image obtained by enlarging the predetermined range. the predetermined range is a range including at least a forward direction of the moving body,
5. The information processing device according to claim 1, wherein the display control means changes a width of the predetermined range based on the forward direction, based on a state or operation of the driver. the surrounding captured image is a composite image obtained by combining images captured by a plurality of cameras provided on the moving object,
5. The information processing device according to claim 1, wherein the display control means causes the terminal device to display the predetermined range in the composite image. A control method executed by an information processing device used in a system that mediates communication between a driver of a moving body and a user present outside the moving body, comprising:
an acquisition step of acquiring a peripheral image of the moving body from the moving body;
A detection step of detecting a state or operation of a driver of the moving body;
A display control step of displaying a predetermined range in the peripheral photographed image on a terminal device of the user,
The display control step is a control method for determining the predetermined range to be displayed on the terminal device based on a state or operation of the driver. A program executed by a computer of an information processing device used in a system that mediates communication between a driver of a moving body and a user outside the moving body, comprising:
An acquisition means for acquiring a peripheral image of the moving body by capturing an image of the surroundings of the moving body from the moving body;
A detection means for detecting a state or operation of a driver of the vehicle;
causing the computer to function as a display control means for displaying a predetermined range in the peripheral photographed image on a terminal device of the user;
The display control means is a program that determines the predetermined range to be displayed on the terminal device based on the state or operation of the driver. A storage medium storing the program according to claim 9.