JP2022071798A - Information processing method, information processing system, and program - Google Patents

Abstract

To provide an information processing method and the like capable of providing a content which a user probably wants to view.SOLUTION: A computer reproduces a content with a content reproduction part provided in a vehicle. The computer also detects a reaction of a user viewing the reproduced content. Then, the computer performs content control which includes continuing or finishing the reproduction of the content being reproduced or specifying a content to be reproduced next, according to the detected reaction.SELECTED DRAWING: Figure 1

Description

The present disclosure relates to information processing methods, information processing systems and programs.

Route guidance is provided using a navigation device mounted on the vehicle. Patent Document 1 proposes a system for improving entertainment in the vehicle interior space by providing contents according to the traveling state and traveling history of the vehicle, the hobbies and tastes of the passengers, the family structure, and the like.

Japanese Unexamined Patent Publication No. 2020-56715

However, in the technique disclosed in Patent Document 1, it is difficult to provide the content that the user wants to watch because the provided content does not consider the state of the user when the vehicle is running.

The present disclosure has been made in view of such circumstances, and an object of the present disclosure is to provide an information processing method or the like capable of providing contents that a user has a high possibility of viewing.

In the information processing method according to the present disclosure, the content is reproduced by the content reproduction unit provided in the vehicle, the reaction of the user who views the content is detected, and the reproduction of the content is continued or reproduced according to the detected reaction. It includes a process of performing content control including identification of content to be terminated or to be played next.

The present disclosure can provide content that the user is likely to want to watch.

It is a schematic diagram which shows the configuration example of an information processing system. It is a block diagram which shows the configuration example of a server. It is a schematic diagram which shows the configuration example of DB stored in a server. It is a block diagram which shows the configuration example of an in-vehicle device. It is a schematic diagram which shows the structural example of the inside of a vehicle. It is a schematic diagram which shows the configuration example of the event DB. It is a flowchart which shows an example of the content provision processing procedure. It is a flowchart which shows an example of the content provision processing procedure. It is a flowchart which shows an example of the acquisition processing procedure of the information outside the vehicle. It is a flowchart which shows an example of the acquisition processing procedure of the vehicle interior information. It is a flowchart which shows the other example of the content provision processing procedure. It is a schematic diagram which shows the structural example of the content DB of Embodiment 2. It is a flowchart which shows an example of the display processing procedure of an image content. It is a schematic diagram which shows the display example of the image content. It is a block diagram which shows the structural example of the vehicle-mounted device of Embodiment 3. It is a flowchart which shows an example of the display processing procedure of the image content of Embodiment 3. It is a schematic diagram which shows the display example of the image content.

The information processing method, information processing system, and program of the present disclosure will be described in detail below with reference to drawings showing embodiments thereof.

(Embodiment 1)
An information processing system that reproduces contents according to the situation inside the vehicle and the situation outside the vehicle (around the vehicle) will be described. In the present embodiment, the content may be content by any one of image, sound (music), light (illumination light), vibration, fragrance, and air conditioning (wind), and a plurality of these may be combined and configured as one. Content may be used. Further, the content of the present embodiment may be any content such as entertainment content such as a movie, drama, animation, live music, or advertising content for promoting a product or service, but is viewed (experienced). It is desirable that the content is such that the user's emotions change. For example, content that causes emotional changes, such as content that excites (excites) the user who has watched (experienced), content that excites (excites the chest), and content that calms anger (calms down), is desirable. .. The vehicle in the present embodiment is a moving body such as an automobile or a bus, and may be a vehicle that travels according to an operation by a driver, or may be an automatically traveling vehicle configured to be able to automatically travel. FIG. 1 is a schematic diagram showing a configuration example of an information processing system. The information processing system of the present embodiment includes a server 10 and an in-vehicle device 20 mounted on a vehicle, and the server 10 and the in-vehicle device 20 can be connected to a network N such as the Internet, and information can be transmitted / received via the network N. I do.

The server 10 is an information processing device capable of various information processing and transmission / reception of information, and is configured by using, for example, a server computer or a personal computer. A plurality of servers 10 may be provided and configured to perform distributed processing, may be realized by a plurality of virtual machines provided in one server, and may be realized by using a cloud server. You may. The in-vehicle device 20 is a device mounted on a vehicle on which a user who uses the content distribution service provided by the server 10 rides, and is provided in each of a plurality of vehicles. The in-vehicle device 20 is an information processing device capable of transmitting and receiving various information processing and information, and is configured by using, for example, a personal computer, a tablet terminal, a smartphone, or a car navigation device. The in-vehicle device 20 may be configured by a dedicated device.

In the information processing system of the present embodiment, for each of the plurality of events, conditions conforming to the events (event occurrence conditions) and a plurality of contents are registered in association with each other. The in-vehicle device 20 determines whether or not the occurrence condition of any event is satisfied based on the situation inside the vehicle (inside the vehicle) and the situation outside the vehicle (outside the vehicle), and determines whether or not the occurrence condition of any event is satisfied. If it is satisfied, the server 10 is notified of the occurrence of the event. The server 10 transmits (delivers) the content corresponding to the event to the vehicle (vehicle-mounted device 20) in which any of the events has occurred, and the vehicle-mounted device 20 receives the content transmitted from the server 10. Reproduce. Further, the in-vehicle device 20 detects the reaction of the user (passenger of the vehicle) who views the content and notifies the server 10, and the server 10 appropriately switches the content to be transmitted according to the reaction of the user in the vehicle. The situation inside the vehicle includes, for example, information such as the number of passengers, the attributes and seating positions of the passengers, the movements and emotions of the passengers, and the like. The conditions outside the vehicle include, for example, the date and time (date, season, day, time zone, etc.), current location, environment around the vehicle (surrounding map, scenery, sound), weather (temperature, humidity, precipitation, sunshine hours, etc.). ), Includes information such as running speed / direction, starting point / destination, etc.

FIG. 2 is a block diagram showing a configuration example of the server 10. The server 10 includes a control unit 11, a storage unit 12, a communication unit 13, an input unit 14, a display unit 15, a reading unit 16, and the like, and each of these units is connected to each other via a bus. The control unit 11 includes one or a plurality of processors such as a CPU (Central Processing Unit), an MPU (Micro-Processing Unit), and a GPU (Graphics Processing Unit). The control unit 11 appropriately executes the control program 12P stored in the storage unit 12 to execute various information processing and control processing to be performed by the server 10.

The storage unit 12 includes a RAM (Random Access Memory), a flash memory, a hard disk, an SSD (Solid State Drive), and the like. The storage unit 12 stores in advance various data and the like necessary for executing the control program 12P and the control program 12P executed by the control unit 11. Further, the storage unit 12 temporarily stores data or the like generated when the control unit 11 executes the control program 12P. Further, the storage unit 12 stores the user DB (database) 12a and the content DB 12b, which will be described later. The user DB 12a and the content DB 12b may be stored in another storage device connected to the server 10, or may be stored in another storage device with which the server 10 can communicate.

The communication unit 13 is an interface for connecting to the network N by wired communication or wireless communication, and transmits / receives information to / from another device via the network N. The input unit 14, including, for example, a mouse and a keyboard, receives an operation input by a user who uses the server 10, and sends a control signal corresponding to the operation content to the control unit 11. The display unit 15 is a liquid crystal display, an organic EL display, or the like, and displays various information according to instructions from the control unit 11. The input unit 14 and the display unit 15 may be a touch panel configured as one.

The reading unit 16 stores information stored in a portable storage medium 1a including a CD (Compact Disc) -ROM, a DVD (Digital Versatile Disc) -ROM, a USB (Universal Serial Bus) memory, an SD (Secure Digital) card, and the like. read. The control program 12P and various data stored in the storage unit 12 may be read by the control unit 11 from the portable storage medium 1a via the reading unit 16 and stored in the storage unit 12. Further, the control program 12P and various data stored in the storage unit 12 may be downloaded from another device by the control unit 11 via the communication unit 13 and stored in the storage unit 12.

FIG. 3 is a schematic diagram showing a configuration example of DBs 12a to 12b stored in the server 10. FIG. 3A shows the user DB 12a, and FIG. 3B shows the content DB 12b. The user DB 12a stores information about a user who has registered as a user in order to use the content distribution service provided by the server 10. The user DB 12a shown in FIG. 3A includes a user ID column, a user information column, an event occurrence history column, and the like, and stores information about the user in association with the user ID. The user ID column stores identification information (user ID) pre-assigned to each user. The user information column stores attribute information such as the user's gender and age, personal information such as family structure and hobbies and preferences, destination information for communicating with the in-vehicle device 20 used by the user, and the like. The event occurrence history column stores historical information about events that occurred while the user was in the vehicle. The history information includes the date and time when the event occurred, the event ID of the event that occurred, the content ID of the content transmitted to the user's vehicle (vehicle-mounted device 20), the reaction of the user when viewing each content, and the like. The date and time when the event occurs may be the date and time when the event occurrence condition is satisfied, or may be the time zone from the occurrence of the event to the end or the occurrence of the next event. The user's reaction includes the user's actions (gestures, postures or actions) and emotions (emotion information) when the content is viewed (viewing). The history information may further include the place (position) where the event occurred, the atmosphere in the vehicle at the time of the event (emotion of the occupant, etc.), the playback date and time of each content, and the like. The user ID stored in the user DB 12a is issued and stored by the control unit 11 when a new user is registered. The user information stored in the user DB 12a is added or changed by the control unit 11 when the control unit 11 acquires an instruction for addition or change via the communication unit 13 or the input unit 14. The event occurrence history stored in the user DB 12a is when the control unit 11 is notified of the occurrence of the event via the communication unit 13, or when the content corresponding to the event is transmitted to the in-vehicle device 20 via the communication unit 13. When the user's reaction is notified via the communication unit 13, it is stored by the control unit 11. The storage content of the user DB 12a is not limited to the example shown in FIG. 3A, and for example, information about the user's vehicle may be stored.

The content DB 12b stores the content provided to the in-vehicle device 20 in response to the event generated in the vehicle. The content DB 12b shown in FIG. 3B includes an event ID column, a content ID column, a content data column, a corresponding reaction column, and the like, and stores information about the content provided when each event occurs in association with the event ID. The event ID column stores identification information (event ID) pre-assigned to each event. The content ID column stores identification information (content ID) pre-assigned to the content associated with each event. As the content ID, for example, an ID according to the playback order set by default for a plurality of contents associated with each event is used. The content DB 12b may be provided with a reproduction order sequence to store the reproduction order of each content. In this case, any ID can be used as the content ID. The content data column stores the content data corresponding to the content ID. The content data includes image content, audio content, light (illumination light) content, vibration content, fragrance content, air conditioning (wind) content, or a combination of these. For example, it may be video content including image content and audio content, or may be content in which content by light, vibration, fragrance, or air conditioning according to the content of the video content is further added to the video content. The content by light is defined by the color, intensity, blinking speed, etc. of the illumination light, the content by vibration is defined by the vibration pattern, vibration intensity, duration, etc., and the content by fragrance is defined by the type and intensity of fragrance. , Or the duration, etc. Further, the content by air conditioning is defined by the temperature or humidity of the air (wind) blown from the air conditioning equipment provided in the vehicle, the amount of air volume, the wind direction, the duration, and the like. In addition to being stored in the content DB 12b, the content data may be stored in a predetermined area of the storage unit 12 or another storage device. In this case, the content data string is information for reading the content data (for example, a storage location). (File name indicating) is memorized. The corresponding reaction sequence stores information about the user's reaction suitable for delivering each content, and the user's reaction includes the user's action (gesture, posture or action) and emotion (atmosphere). The event ID stored in the content DB 12b is issued and stored by the control unit 11 when a new event is registered. The content ID, content data, and corresponding reaction stored in the content DB 12b are added or changed by the control unit 11 when the control unit 11 obtains an addition or change instruction via the communication unit 13 or the input unit 14. .. The stored content of the content DB 12b is not limited to the example shown in FIG. 3B, and for example, the occurrence condition of each event may be stored. Further, the content DB 12b may be set for each user. In this case, the content DB 12b may be provided with a user ID column, and the content of each event may be stored in association with the user ID. Further, when the content DB 12b is set for each user, the content DB 12b may be stored in the in-vehicle device 20.

FIG. 4 is a block diagram showing a configuration example of the in-vehicle device 20, and FIG. 5 is a schematic diagram showing a configuration example inside the vehicle. FIG. 5 simply shows the state of the inside of the vehicle when the front window is viewed from the rear seat side, and the dashboard, instrument panel, door, etc. are not shown. The vehicle of the present embodiment is provided with side windows (left side window, right side window) at the left and right ends of the front window via A pillars, respectively, so as to connect the front windows and the upper ends of the two side windows. There is a roof. Further, the vehicle shown in FIG. 5 is provided with two front seats so that the passenger can sit facing the front window, and a display unit 25 is provided at a position where the passenger seated in the front seat can visually recognize the vehicle. ing. The seats provided in the vehicle may be installed so as to be movable in position, and may be configured so that, for example, one front seat can be arranged directly in front of the front window. Further, the seat on the left side in the traveling direction may be arranged toward the left side window, and the seat on the right side may be arranged toward the right side window. Further, each seat may be configured to be foldable, in which case unnecessary seats can be folded. The display unit 25 may be provided at a plurality of locations as long as it is provided at a position that can be visually recognized by the passengers seated in the front seat and the rear seat. Further, in the vehicle shown in FIG. 5, the camera 27 is located at an appropriate position at the central portion of the boundary portion between the front window and the roof, at an appropriate position at the boundary portion between the left side window and the roof, and at an appropriate position at the boundary portion between the right side window and the roof, respectively. It is provided. The camera 27 may be provided at a position where the passengers seated in the front seat and the rear seat can be photographed, and may be provided at a position other than the position shown in FIG. Further, in the vehicle shown in FIG. 5, the lighting unit 30 is provided at an appropriate position at the boundary portion between the front window and the roof. The lighting unit 30 may be provided at a position where the space inside the vehicle can be colored with a light emitting color, and may be provided at an appropriate position on the roof or at a position close to the feet of the occupant, for example.

Although not shown in FIG. 5, the vehicle of the present embodiment includes a microphone 28, a speaker 29, a vibration generating unit 31 that generates vibration, an fragrance generating unit 32 that generates fragrance, and an air conditioning control unit 33 that generates wind. It is provided at an appropriate position. Further, in the vehicle of the present embodiment, in addition to the camera 27 being provided at a position where the passenger inside the vehicle can be photographed, the camera 27 may be provided at a position where the scenery outside the vehicle can be photographed. Further, the microphone 28 may be provided at a position where the voice of the passenger inside the vehicle can be collected, or may be provided at a position where the sound outside the vehicle can be collected. Further, the vehicle of the present embodiment may be provided with a device for collecting the scent (smell) around the vehicle at an appropriate position on the outer surface of the vehicle. The device for collecting the scent may be built in the in-vehicle device 20 or may be externally attached to the in-vehicle device 20.

As shown in FIG. 4, the in-vehicle device 20 includes a control unit 21, a storage unit 22, a communication unit 23, an input unit 24, a display unit 25, a positioning unit 26, a camera 27, a microphone 28, a speaker 29, a lighting unit 30, and vibration generation. A unit 31, an aroma generation unit 32, an air conditioning control unit 33, a vital data acquisition unit 34, and the like are included, and each of these units is connected to each other via a bus. The control unit 21, the storage unit 22, the communication unit 23, the input unit 24, and the display unit 25 of the in-vehicle device 20 are the control unit 11, the storage unit 12, the communication unit 13, the input unit 14, and the display unit 15 of the server 10, respectively. Since it has the same configuration, detailed description will be omitted. The storage unit 22 of the in-vehicle device 20 stores the map data 22D and the event DB 22a described later in addition to the control program 22P executed by the control unit 21. The map data 22D may be map data of a place where the vehicle may travel, and may be map data that is sequentially downloaded as the vehicle travels from another server that stores the map data. Map data includes information on topography, roads, intersections, tourist spots, various stores, agricultural product production areas, marine product production areas (landed fishing ports), and the like.

The input unit 24 of the in-vehicle device 20 may be configured to accept operations by the user by voice input via the microphone 28. The display unit 25 (content reproduction unit) of the in-vehicle device 20 may be a head-up display or a projector that projects (projects) information on glass, in addition to a liquid crystal display or an organic EL display. When the display unit 25 is a head-up display, information may be projected on a part or all of the front window, the rear window, and the side window (vehicle window), and the roof is made of glass or a transparent synthetic resin. If it is composed of a board, information may be projected on a part or the whole area of the roof.

The positioning unit 26 detects the current location of the vehicle-mounted device 20 (or vehicle), acquires current location information (for example, coordinate values of longitude and latitude) indicating the current location, and sends the acquired current location information to the control unit 21. The positioning unit 26 receives, for example, a radio wave transmitted from a GPS (Global Positioning System) satellite via an in-vehicle device 20 or a GPS antenna (not shown) provided in the vehicle, and determines the current location based on the received radio wave. To detect. The method of detecting the current location is not limited to the method based on the radio waves from the GPS satellites. Based on the detected current location and the map data 22D, the positioning unit 26 can identify the detected current location on the map, the road on which the vehicle is traveling, and the intersection. Further, the positioning unit 26 has a gyro sensor and the like, and can specify the traveling direction of the vehicle (for example, the direction indicated by north, south, east, and west).

The camera 27 is an image pickup device that shoots images according to instructions from the control unit 21, and sends the acquired captured images (image data) to the storage unit 22 for storage. The camera 27 may be built in the vehicle-mounted device 20 or may be externally attached to the vehicle-mounted device 20. When externally attached, the in-vehicle device 20 includes a connection unit to which an external camera can be connected or a wireless communication unit capable of wireless communication with the external camera, and the image data taken by the external camera is connected to the connection unit or wireless communication. Get through the department.

The microphone 28 is a sound collecting device that collects sound according to an instruction from the control unit 21, and sends the acquired voice data to the storage unit 22 for storage. The microphone 28 has a microphone provided inside the vehicle and collecting the sound inside the vehicle, and a microphone provided on the outer surface of the vehicle for collecting the sound outside the vehicle, for example. The voice in the vehicle includes, for example, the voice spoken by the passenger in the vehicle, the voice output from the speaker 29 in the vehicle, and the like. The sound outside the vehicle includes, for example, noise around the vehicle, human voice, rain sound, siren sound, and the like. The microphone 28 may be built in the vehicle-mounted device 20 or may be externally attached to the vehicle-mounted device 20. When externally attached, the in-vehicle device 20 includes a connection unit capable of connecting an external microphone or a wireless communication unit capable of wireless communication with the external microphone, and the voice data collected by the external microphone is connected to the connection unit or wireless communication. Get through the department.

The speaker 29 (content reproduction unit) is an audio output unit that outputs audio according to an instruction from the control unit 21, and outputs audio or music instructed by the control unit 21. The speaker 29 may be built in the vehicle-mounted device 20 or may be externally attached to the vehicle-mounted device 20. When externally attached, the in-vehicle device 20 includes a connection unit capable of connecting an external speaker or a wireless communication unit capable of wireless communication with the external speaker, and the connection unit or the wireless communication unit is from the control unit 21. The voice signal (music signal) according to the instruction is sent to the external speaker, and the voice or music is output to the external speaker.

The illumination unit 30 has a light that emits light of any one color such as white, red, blue, and green, or a plurality of colors, and emits light of a color according to an instruction from the control unit 21. The lighting unit 30 may be built in the vehicle-mounted device 20 or may be externally attached to the vehicle-mounted device 20. When externally attached, the in-vehicle device 20 includes a connection unit capable of connecting an external light or a wireless communication unit capable of wireless communication with the external light, and the connection unit or the wireless communication unit is from the control unit 21. A light emission instruction is sent to an external light to make the external light emit light.

The vibration generating unit 31 has, for example, a vibrator capable of generating a plurality of patterns of vibration, and generates a pattern of vibration according to an instruction from the control unit 21 by the vibrator. The vibration generating unit 31 is provided in each seat, for example, and may be configured to generate the same pattern of vibration for each seat, or may be configured to generate different patterns of vibration. May be good. The vibration generating unit 31 may be built in the vehicle-mounted device 20 or may be externally attached to the vehicle-mounted device 20. When externally attached, the in-vehicle device 20 includes a connection unit capable of connecting an external vibrator or a wireless communication unit capable of wireless communication with an external vibrator, and the connection unit or the wireless communication unit is from the control unit 21. The vibration instruction is sent to the external vibrator to vibrate the external vibrator.

The fragrance generation unit 32 has, for example, a storage tank for storing a plurality of types of fragrances or aroma oils, and an fragrance generation device having a discharge unit for releasing (spraying) the fragrances or aroma oils in the storage tank, and controls the fragrance generation unit 32. The aroma (scent) according to the instruction from the part 21 is generated by the aroma generating device. The fragrance generating unit 32 may be built in the vehicle-mounted device 20 or may be externally attached to the vehicle-mounted device 20. When externally attached, the in-vehicle device 20 includes a connection unit capable of connecting an external fragrance generating device or a wireless communication unit capable of wireless communication with an external fragrance generating device, and the connection unit or the wireless communication unit is provided. , The fragrance generation instruction from the control unit 21 is sent to the external fragrance generation device, and the fragrance is generated by the external fragrance generation device.

The air conditioning control unit 33 is an air conditioning control device that controls the state of the air discharged from the air conditioning device (air conditioner) that adjusts the temperature and humidity of the air in the vehicle, and the air according to the instruction from the control unit 21. (Wind) is released to the air conditioner. The air conditioning control unit 33 controls air conditioning conditions such as temperature, humidity, air volume, and wind direction of air discharged from the air conditioning equipment. Further, the air conditioning equipment has, for example, an air blast function for injecting compressed air compressed by a compressor, and the air conditioning control unit 33 can control the air volume (wind power) of the air injected from the air conditioning equipment. The vital data acquisition unit 34 acquires the occupant's vital data according to the instruction from the control unit 21, and sends the acquired vital data to the storage unit 22 for storage. Vital data includes, for example, body temperature, heart rate (pulse), electrocardiogram, respiratory status, and the like. The vital data acquisition unit 34 has, for example, a measuring instrument (not shown) for measuring the vital data of the passenger, and acquires the vital data measured by the measuring instrument. The measuring instrument is provided, for example, on the steering wheel or armrest (armrest) of the vehicle, and the vital data is collected from the hand of the occupant (driver) who holds the steering wheel, or the hand or arm of the occupant placed on the armrest. It may be configured to measure. The measuring instrument may be built in the vehicle-mounted device 20 or may be externally attached to the vehicle-mounted device 20. Further, in addition to the configuration having the measuring instrument, the vital data acquisition unit 34 may have a communication device capable of communicating with the measuring instrument and may acquire the vital data measured by the measuring instrument by the communication device. In this case, the measuring instrument may be, for example, a wristwatch-type wearable device, and is used by being worn on the wrist, arm, or the like of a occupant. The vital data of the occupant may be acquired by analyzing, for example, an image of the occupant, or may include the facial expression of the occupant obtained by analyzing the facial image of the occupant.

FIG. 6 is a schematic diagram showing a configuration example of the event DB 22a. The event DB 22a stores the event occurrence condition for each of the plurality of events. The event DB 22a shown in FIG. 6 includes an event ID column, an occurrence condition column, and the like. The event ID column stores identification information (event ID) of each event. The occurrence condition column stores information about the occurrence condition when each event occurs. Occurrence conditions include conditions related to the inside of the vehicle (conditions inside the vehicle) and conditions related to the outside of the vehicle (conditions outside the vehicle). The conditions inside the vehicle are, for example, the seating position of the passenger in the vehicle, the attributes (gender and age, etc.) of the passenger at each seating position, the relationship (intimacy), movement and emotion, the relative relationship of the seating position of each passenger, etc. Includes conditions regarding the situation inside the car. For example, two men and women in their twenties are in the front seats, children (for example, under 10 years old) are in the vehicle, and each passenger is happy, sad, scared, relaxed, etc. Being emotional can be used as an in-vehicle condition. The conditions outside the vehicle are, for example, the current date and time (season, day, time zone, etc.), weather, current location of the vehicle, traveling speed and direction, departure point and destination, scenery around the vehicle, sound and scent (smell), etc. Includes conditions for. For example, the distance to a predetermined store is within a predetermined distance, the driving place (current location) is in a predetermined town, the average traveling speed is equal to or less than the predetermined speed, and the predetermined time zone of the predetermined season. Alternatively, it can be used as an outside condition such as having a predetermined weather. The event ID stored in the event DB 22a is issued and stored by the control unit 21 when a new event occurrence condition is registered. The generation condition stored in the event DB 22a is added or changed by the control unit 21 when the control unit 21 acquires an instruction for addition or change via the communication unit 23 or the input unit 24. The occurrence condition of each event may be, for example, once registered in the content DB 12b of the server 10, transmitted from the server 10 to the in-vehicle device 20 in association with the event ID, and registered in the event DB 22a of the in-vehicle device 20. .. The stored content of the event DB 22a is not limited to the example shown in FIG. For example, the content data corresponding to each event and the user's reaction (corresponding reaction) suitable for distributing (reproducing) each content may be downloaded in advance and stored in the event DB 22a. In addition to the event occurrence condition, information on the interruption condition that the occurring event is interrupted, information on the completion condition that the occurring event is completed, information on the switching condition for switching to another event, etc. May be stored in the event DB 22a.

In the present embodiment, the occurrence history of the event generated in each vehicle is stored in the user DB 12a of the server 10 in association with the user ID. As a result, even when the same user gets on a different vehicle, it is possible to provide the content considering the event occurrence history registered in the server 10 to the vehicle on which the user gets on. In addition, the event DB 22a or the storage unit 22 of the vehicle-mounted device 20 may store the event occurrence history of each vehicle. In this case, the in-vehicle device 20 can refer to the past event occurrence history when determining the presence / absence of the event occurrence.

Hereinafter, in the information processing system having the above-described configuration, the process of providing the content corresponding to the event generated in the vehicle to the in-vehicle device 20 by the server 10 will be described. 7 and 8 are a flowchart showing an example of a content provision processing procedure, FIG. 9 is a flowchart showing an example of an outside vehicle information acquisition processing procedure, and FIG. 10 is a flowchart showing an example of an in-vehicle information acquisition processing procedure. 7 and 8 show the processing performed by the in-vehicle device 20 on the left side and the processing performed by the server 10 on the right side. The process of acquiring the information outside the vehicle shown in FIG. 9 is the process of step S12 in FIG. 7, and the process of acquiring the information inside the vehicle shown in FIG. 10 is the process of step S13 in FIG. The following processing is realized by the control unit 21 according to the control program 22P stored in the storage unit 22 of the vehicle-mounted device 20, and is realized by the control unit 11 according to the control program 12P stored in the storage unit 12 of the server 10. .. A part of the following processing may be realized by a dedicated hardware circuit.

In the information processing system of the present embodiment, when the in-vehicle device 20 starts operation, it periodically collects information (state information) regarding the state inside and outside the vehicle, and based on the collected state information, any event Judge whether or not the occurrence condition is satisfied. The state information includes information on the state outside the vehicle (outside the vehicle) (information outside the vehicle) and information on the state inside the vehicle (inside the vehicle) (information inside the vehicle). When it is determined that the occurrence condition of any of the events is satisfied, the in-vehicle device 20 notifies the server 10 of the occurrence event, and the server 10 provides the in-vehicle device 20 with the content corresponding to the occurrence event. The case where the in-vehicle device 20 starts the operation may be the case where the vehicle starts running, and the vehicle may be stopped or parked. Further, when the vehicle is an automatic traveling vehicle configured to be able to automatically travel, it may be a case where the vehicle starts automatic traveling.

The control unit 21 of the in-vehicle device 20 determines whether or not the timing for acquiring the information outside the vehicle and the information inside the vehicle has arrived (S11). The control unit 21 periodically acquires information outside the vehicle and information inside the vehicle, and determines whether or not the execution timing has arrived, depending on whether or not a predetermined time has elapsed from the execution of the latest acquisition process. When it is determined that the acquisition timing has not arrived (S11: NO), the control unit 21 stands by while performing other processing. When it is determined that the acquisition timing has arrived (S11: YES), the control unit 21 performs the acquisition process of the information outside the vehicle (S12).

In the vehicle outside information acquisition process shown in FIG. 9, the control unit 21 acquires the date and time information at this point (S31). For example, the vehicle-mounted device 20 has a clock, and the control unit 21 acquires date and time information (year / month / day, hour, minute, second, day of the week) indicated by the clock. Next, the control unit 21 acquires the position information (current location information) of the vehicle (vehicle-mounted device 20) at this time by the positioning unit 26 (S32), and maps the surrounding map of the vehicle based on the vehicle's current location information. Obtained from 22D (S33). For example, the control unit 21 acquires map data within a predetermined range (for example, within 10 km) around the vehicle. Further, the control unit 21 acquires weather information (temperature, humidity, precipitation, sunshine duration, etc.) of the current location of the vehicle (S34). For example, the control unit 21 may acquire the weather information of the current location from a web server (not shown) that publishes the weather information (weather information) via the network N, and the temperature, humidity, precipitation, and sunshine of the vehicle may be acquired. If a sensor for detecting time or the like is provided, the detection result may be acquired from each sensor. The control unit 21 may acquire the current weather information, or may acquire the weather information from the present to a predetermined time (for example, one hour later).

Further, the control unit 21 acquires the traveling speed and the traveling direction of the vehicle (S35). For example, the control unit 21 acquires the speed (traveling speed) detected by the speed sensor (not shown) provided in the vehicle, and acquires the direction in the traveling direction detected by the positioning unit 26. Further, the control unit 21 acquires the departure point and the destination of the vehicle (S36). For example, the control unit 21 acquires the position information detected by the positioning unit 26 at the time when the vehicle starts traveling as the information of the departure place, and sets the destination set in the navigation device provided in the vehicle as the destination of the vehicle. get. The departure point information may be acquired once when the vehicle starts traveling, and the destination information may be acquired when the destination is set in the navigation device or when the destination is changed. do it. Further, the control unit 21 acquires a photographed image around the vehicle by a camera 27 provided at a position where the scenery outside the vehicle can be photographed (S37), and a microphone 28 provided at a position where the sound outside the vehicle can be collected makes a sound around the vehicle. (S38). Then, the control unit 21 acquires the environmental information around the vehicle based on the image and the sound around the vehicle (S39). For example, the control unit 21 has a cityscape and a mountain road, such as whether the environment around the vehicle is a downtown area with many commercial facilities, a quiet residential area, a rural landscape, a seaside, or a mountain road. Detects the state of people. When the vehicle is provided with a device for collecting scents (smell), the control unit 21 may acquire the scent (smell) around the vehicle and determine the surrounding environment in consideration of the acquired scent. .. The control unit 21 stores the information outside the vehicle including the above-mentioned information in the storage unit 22 each time it is acquired. In addition to the above-mentioned information, the information outside the vehicle may include information regarding the situation outside the vehicle, information that can be acquired from outside the vehicle, and the like.

The control unit 21 returns to the process shown in FIG. 7 and performs an in-vehicle information acquisition process (S13). In the vehicle interior information acquisition process shown in FIG. 10, the control unit 21 detects the seating position of the occupant (S51). For example, when each seat in the vehicle is provided with a seating sensor, the control unit 21 determines whether or not a occupant is seated in each seat based on the detection result of each seating sensor. Further, the control unit 21 may photograph the inside of the vehicle with the camera 27 and determine whether or not the passenger is seated in each seat based on the obtained captured image. In this case, the process of step S51 may be performed after the process of step S52.

Next, the control unit 21 photographs the passengers seated in each seat with the camera 27 and acquires the photographed image (S52). Further, the control unit 21 acquires the utterance voice of the passenger seated in each seat with the microphone 28 (S53). Further, the control unit 21 acquires the vital data of the passengers seated in each seat by the vital data acquisition unit 34 (S54). When a camera 27, a microphone 28, and a vital data acquisition unit 34 (measuring instrument) are provided in association with each seat, the control unit 21 attaches the camera 27, the microphone 28, and the vital data acquisition unit 34 to each seat by means of the respective cameras 27, the microphone 28, and the vital data acquisition unit 34. It is possible to acquire captured images, spoken voices, and vital data of seated passengers. The control unit 21 detects the attributes of each passenger based on the captured image and / or the spoken voice of each passenger (S55). For example, the control unit 21 detects the attributes of the passenger in the captured image by the template matching technique. Specifically, for each attribute (gender and age group), a template based on an image of a general person's face or upper body is registered in advance in the storage unit 22, and the control unit 21 uses any of the captured images. When an area matching the template is detected, the attribute corresponding to the detected template is specified as the attribute of the passenger in the captured image. Further, for each attribute (gender and age group), a template based on the voice signal of a general person's utterance voice is registered in advance in the storage unit 22, and the control unit 21 obtains the utterance voice via the microphone 28. When a voice signal matching any of the templates is detected from, the attribute corresponding to the detected template is specified as the attribute of the passenger who collected the uttered voice. The control unit 21 may specify the final occupant attribute in consideration of the occupant attribute specified based on the captured image and the occupant attribute specified based on the utterance voice. ..

The control unit 21 may detect the attributes of each passenger based on the captured image and / or the spoken voice of each passenger by using the neural network constructed by machine learning. For example, a learning model composed of a CNN (Convolution Neural Network) model and trained to output the gender and age group of the subject (passenger) in the captured image when the captured image is input may be used. .. In this case, the control unit 21 inputs the captured image acquired in step S52 into the trained learning model, and identifies the gender and age group of the passenger in the captured image based on the output information from the learning model. Can be done. In addition, using a learning model that is composed of a CNN model, RNN (Recurrent Neural Network) model, etc. and is trained to output the gender and age group of the speaker (passenger) when the spoken voice is input. May be good. In this case, the control unit 21 inputs the utterance voice acquired in step S53 into the learned learning model, and identifies the gender and age group of the speaker related to the utterance voice based on the output information from the learning model. Can be done. Again, the control unit 21 may specify the final occupant attribute in consideration of the occupant attribute identified based on the captured image and the occupant attribute identified based on the utterance voice. good. The control unit 21 may detect the attributes of each passenger in consideration of the vital data of each passenger acquired in step S54.

Next, the control unit 21 detects the movement of each passenger based on the captured image of each passenger (S56). For example, the control unit 21 detects each part of the subject (passenger)'s head, hand, arm, torso, etc. in the captured image, and detects the movement locus of each part of the passenger in the time-series captured image. Further, the movement locus of each part of the occupant is registered in advance in the storage unit 22 in association with each movement that the occupant may perform, and the control unit 21 uses the detected movement locus of each part of the occupant as the movement locus of the occupant. Identify matching behaviors. For example, the control unit 21 detects whether or not the occupant has performed a nodding motion, a waving motion, an arm turning motion, or the like. Here, too, the control unit 21 may detect the movement of the occupant in the captured image by using the neural network. For example, a learning model composed of a CNN model or an RNN model and trained to output information on the movement performed by the subject (passenger) in the captured image when the captured image is input may be used. In this case, the control unit 21 can input the captured image acquired in step S52 into the trained learning model and identify the movement of the occupant in the captured image based on the output information from the learning model.

Next, the control unit 21 detects the emotions of each passenger based on the captured image and / or the spoken voice of each passenger (S57). For example, the control unit 21 detects the emotion of the occupant in the captured image by using an application program that detects the emotion of the subject from the captured image, such as the Emotion API manufactured by Microsoft. Further, the control unit 21 extracts information such as the tone and strength of the passenger's voice and the speaking speed from the spoken voice (conversation voice) of the passenger, and detects the emotion of the passenger based on the extracted information. The passenger's emotions include, for example, joy, sadness, anger, disgust, surprise, relaxation, etc., and each emotion is represented by a numerical value of, for example, a maximum of 10 points (10 points is the highest and 0 points is the lowest). The control unit 21 may detect the occupant's emotions by analyzing the occupant's vital data in addition to the occupant's photographed image and spoken voice. For example, the control unit 21 may extract a heart rate, a respiratory state, or the like from the vital data of the passenger, and detect the emotion of the passenger based on the extracted information. Again, the control unit 21 may detect the emotions of the occupant using a neural network. For example, a learning model composed of a CNN model and trained to output information on the occupant's emotions when a photographed image and / or utterance voice of the occupant is input may be used. In this case, the control unit 21 inputs the captured image acquired in step S52 and / or the utterance voice acquired in step S53 into the learned learning model, and based on the output information from the learning model, the emotions of the occupant are input. Can be estimated. Further, a learning model trained to output information on the occupant's emotions when vital data is input in addition to the occupant's photographed image and / or utterance voice may be used. Since it is highly likely that the passenger does not move the seating position while the vehicle is running, the seating position of each passenger and the attributes of the passengers seated in each seat are acquired once, for example, when the vehicle starts running. do it. The seating position and attributes of each passenger may also be acquired periodically. In this case, even if the passenger moves the seating position while the in-vehicle device 20 is operating (for example, while the vehicle is running). , The attributes of the passengers seated in each seat can be updated sequentially. Through the above-mentioned processing, the control unit 21 acquires in-vehicle information including the presence / absence of seating of the occupant, the attributes, movements, emotions, etc. of the seated occupant for each seat in the vehicle, and acquires the above-mentioned in-vehicle information. Each time it is done, it is stored in the storage unit 22. In addition to the above-mentioned information, the in-vehicle information may include information on the situation inside the vehicle, information that can be acquired in the vehicle, and the like.

The control unit 21 returns to the process shown in FIG. 7, and determines whether or not the occurrence condition of any of the events is satisfied based on the out-of-vehicle information acquired in step S12 and the in-vehicle information acquired in step S13 (). S14). Specifically, the control unit 21 determines whether or not the acquired out-of-vehicle information and in-vehicle information match the occurrence conditions (outside-vehicle conditions or in-vehicle conditions) of each event stored in the event DB 22a. If the occurrence condition of any of the events is met, it is judged that the occurrence condition of this event is satisfied. As a result, for example, outside information such as the vehicle's current location (position information), season, time zone, weather, traveling speed, departure point and destination, scenery and sound around the vehicle, or the number of passengers and each passenger (user). Based on the in-vehicle information such as the seating position, attributes, movements, and emotions of the vehicle, it is possible to determine whether or not the occurrence condition of any of the events is satisfied. When the control unit 21 determines that the event occurrence conditions are not met (event generation conditions are not satisfied) (S14: NO), the control unit 21 returns to the process of step S11. The control unit 21 performs the processes of steps S12 to S14 each time the acquisition timing arrives, acquires the information outside the vehicle and the information inside the vehicle, and repeats the process of determining whether or not the occurrence condition of any of the events is satisfied.

When it is determined that the occurrence condition of any of the events is satisfied (S14: YES), the control unit 21 notifies the server 10 of the occurrence of the event (S15). For example, the control unit 21 transmits the user ID of the user in the vehicle, the destination information of the in-vehicle device 20, and the event ID of the generated event to the server 10 in association with each other. The user ID and the destination information of the vehicle-mounted device 20 may be stored in the storage unit 22 in advance, or may be input via the input unit 24. When the in-vehicle device 20 notifies the occurrence of an event, the control unit 11 of the server 10 associates the user ID received from the in-vehicle device 20 with the date and time and the received event ID as the event occurrence history in the user DB 12a. Remember in.

The control unit 11 reads the content notified from the vehicle-mounted device 20, that is, the content corresponding to the event generated in the vehicle on which the vehicle-mounted device 20 is mounted, from the content DB 12b (S16). For example, the control unit 11 reads out the content having the smallest (first) content ID among the contents corresponding to the event. Then, the control unit 11 (reproduction processing unit) transmits the read content to the in-vehicle device 20 (S17), and causes the in-vehicle device 20 to reproduce the content. The control unit 11 stores the content ID of the content transmitted to the vehicle-mounted device 20 in the user DB 12a in association with the user ID and the event ID. As a result, the transmission history of the content transmitted by the server 10 to the in-vehicle device 20 can be stored in the user DB 12a. The control unit 11 may store the transmission date and time of the content in the user DB 12a in association with the content ID.

The control unit 21 of the in-vehicle device 20 reproduces the content received from the server 10 (S18). For example, the control unit 21 causes the display unit 25 to display the image content when the content includes the image content, and outputs the audio content from the speaker 29 when the content includes the audio content. Further, when the content includes the content due to light, the control unit 21 turns on or blinks the lighting unit 30 according to the content due to light, and when the content includes the content due to vibration, the control unit 21 vibrates according to the content due to vibration. To generate. Further, when the content includes the content due to the fragrance, the control unit 21 generates the fragrance in the fragrance generation unit 32 according to the content due to the fragrance. Further, when the content includes the content by air conditioning (air, wind), the control unit 21 generates wind by the air conditioning control unit 33 according to the content by air conditioning.

The control unit 21 of the in-vehicle device 20 periodically repeats the processes of steps S12 and S13 even during the reproduction of the content received from the server 10, and collects the information outside the vehicle and the information inside the vehicle. Therefore, the control unit 21 (detection unit) can sequentially detect the movements and emotions (reactions) of the passenger who views (experiences) the reproduced content. Then, the control unit 21 determines whether or not the user's reaction has changed based on the passenger's reaction (motion and emotion) that is sequentially acquired during the reproduction of the content (S19). For example, the control unit 21 determines whether or not the emotions of each passenger that are periodically detected have changed, and if so, determines that the reaction of the user (passenger) has changed. Further, the control unit 21 determines whether or not the movement of each passenger has changed, and if so, determines that the reaction of the passenger has changed. When the control unit 21 determines that the reaction of the user (passenger) has changed (S19: YES), the control unit 21 transmits reaction information indicating the user's reaction to the server 10 (S20). For example, the control unit 21 transmits the user ID of the user, the content ID of the content being played, and the reaction information indicating the reaction of the user to the server 10 in association with each other. The reaction information may be information indicating the reaction after the change (for example, the emotion or action of the user), or may be information indicating the reaction before and after the change. Here, a detection target user to detect a change in reaction may be set in each content. In this case, the control unit 21 identifies the detection target user from the passengers in the vehicle. It may be determined whether or not the reaction of the identified user has changed. In this case, the information of the user to be detected and the reaction of this user are included in the information of the corresponding reaction (the reaction of the user suitable for distribution) of each content stored in the content DB 12b. The information of the user to be detected may include, for example, attribute information including age or gender, a seating position, a destination (destination), and the like. Thereby, among the passengers who watch the content, it is possible to detect the reaction of the user according to the content, for example, a child (a passenger of a predetermined age or younger) or a user sitting in the passenger seat. A plurality of users may be registered as users to be detected, and a priority may be set for each user. For example, if a passenger seat user is set in priority 1 and a driver seat user is set in priority 2, and there is a user seated in the passenger seat, the reaction of this user is detected. If there is no user seated in the passenger seat, the reaction of the user (driver) seated in the driver's seat may be detected.

When the control unit 11 of the server 10 receives the reaction information of the user from the in-vehicle device 20, the control unit 11 associates the received reaction information with the user ID and the content ID received from the in-vehicle device 20 and uses the received reaction information as the user reaction in the event occurrence history. Store in DB12a. Then, the control unit 11 determines whether or not to stop the transmission of the content currently being transmitted to the vehicle-mounted device 20 based on the reaction information of the user received from the vehicle-mounted device 20 (S21). For example, the control unit 11 reads a user's reaction (corresponding reaction) suitable for delivering the content being transmitted to the in-vehicle device 20 from the content DB 12b. Then, the control unit 11 determines whether or not the reaction of the user received from the in-vehicle device 20 matches the corresponding reaction read from the content DB 12b, and if not, determines that the transmission of the content should be stopped. .. If the user's reaction received from the in-vehicle device 20 matches the corresponding reaction read from the content DB 12b, the control unit 11 may determine that it is not necessary to stop the transmission of the content.

When it is determined that it is not necessary to stop the transmission of the content (S21: NO), the control unit 11 returns to the process of step S17 and transmits the content corresponding to the event occurring in the vehicle to the in-vehicle device 20. continue. As a result, the in-vehicle device 20 continues to reproduce the content provided by the server 10. The playback order (delivery order) is set by default for the plurality of contents associated with each event, and when the control unit 11 completes the transmission of one content, the content of the next playback order is set. Is read from the content DB 12b and transmission to the in-vehicle device 20 is started. As a result, when the user's reaction matches the corresponding reaction of each content, that is, when the user's reaction is a reaction suitable for content distribution, the server 10 displays each content in the preset reproduction order in the vehicle-mounted device 20. Send to.

When it is determined that the transmission of the content should be stopped (S21: YES), the control unit 11 reads the content corresponding to the same event, which is suitable for the reaction of the current user, from the content DB 12b (S22). ), Sent to the in-vehicle device 20 (S17). For example, the control unit 11 identifies a correspondence reaction that matches the user's reaction received from the in-vehicle device 20 from the correspondence reaction of each content stored in the content DB 12b, and selects the content corresponding to the specified correspondence reaction by this user. It is read out as content to be provided to the in-vehicle device 20 and provided to the in-vehicle device 20. Further, the control unit 11 may specify the content suitable for the reaction of the current user in consideration of the reaction of the past user to each content registered in the user DB 12a. Therefore, the control unit 11 controls the continuation or end of playback of the content being played according to the reaction of the passenger who views (experiences) the content, and when the playback ends, the control unit 11 specifies the content to be played next. Operates as a content control unit that performs processing (content control). As a result, the server 10 can provide the user with the content according to the reaction of the user via the in-vehicle device 20. When there are a plurality of contents corresponding to the corresponding reactions that match the user's reactions received from the in-vehicle device 20, for example, the contents may be read out in the order of the content IDs. After starting the transmission of the content to the in-vehicle device 20, the control unit 11 stores the content ID of the content for which the transmission has started in the user DB 12a in association with the user ID and the event ID. Further, the control unit 11 may store the transmission date and time of the content in the user DB 12a in association with the content ID.

In step S22, the control unit 11 may specify the content suitable for the user's reaction by using the neural network. For example, it is composed of a CNN model and is suitable for providing information about the content to be provided to the passenger (for example, content ID, content type, content provision) when information such as the passenger's emotion, current date and time, and current location is input. A learning model trained to output the user's reaction) may be used. In this case, the control unit 11 acquires the occupant's emotion acquired in step S57, the date and time information acquired in step S31, the current location information acquired in step S32, and the like, and the in-vehicle device 20 has learned these information. It can be input to the model and the content to be provided to the passenger can be specified based on the output information from the learning model. In-vehicle information including the seating position, attributes, emotions, movements, etc. of each passenger and outside information including the current date and time, current location, weather, traveling speed, driving direction, destination, environment around the vehicle, etc. are input. In that case, a learning model trained to output information about the content to be provided to the passenger may be used. In this case, the control unit 11 acquires the out-of-vehicle information acquired in step S12 and the in-vehicle information acquired in step S13, inputs the out-of-vehicle information and the in-vehicle information into the learned learning model, and uses the learning model. Based on the output information of, the content to be provided to the passenger can be specified.

The control unit 21 of the in-vehicle device 20 performs the processes of steps S18 to S20 for the content transmitted from the server 10. That is, the control unit 21 reproduces the content received from the server 10 (S18), determines whether or not the user's reaction at this time has changed (S19), and if the change occurs (S19: YES). ), The user's reaction is transmitted to the server 10 (S20). As a result, the server 10 can provide the plurality of contents associated with one event to the in-vehicle device 20 in an order according to the reaction of the user, and can be reproduced by the in-vehicle device 20.

When the control unit 21 of the in-vehicle device 20 determines that the reaction of the user has not changed during the reproduction of the content (S19: NO), the process of step S20 is skipped and one transmitted from the server 10. It is determined whether or not the content is finished (S23). For example, each content is given information indicating the beginning or the end, respectively, and the control unit 21 determines that the content being played has ended when the information indicating the end of the content is received from the server 10. .. If it is determined that the content being played is not finished (S23: NO), the control unit 21 returns to the process of step S18 and continues playing the content transmitted from the server 10.

When the control unit 21 determines that the content being played has ended (S23: YES), the control unit 21 detects the user's reaction at this point (S24), and transmits the reaction information indicating the detected reaction to the server 10 (S25). ). Here, the control unit 21 detects the user's reaction (motion and emotion) at this time based on the in-vehicle information collected periodically, and the user's user ID, the content ID of the content whose reproduction has been completed, and the user's The reaction information is associated and transmitted to the server 10. When the control unit 11 of the server 10 receives the reaction information of the user from the in-vehicle device 20, the control unit 11 stores the received reaction information in the user DB 12a as a user reaction of the event occurrence history in association with the received user ID and content ID. Keep it (S26).

The control unit 21 of the in-vehicle device 20 has determined whether or not the series of processes described above should be terminated (S27), and if it is determined that the process should not be terminated (S27: NO), the process returns to the process of step S18. Then, the control unit 21 receives and reproduces the content sequentially transmitted from the server 10 (S18), and repeats the above-mentioned processing. As a result, the server 10 is content corresponding to the event generated in the vehicle, can provide the content according to the reaction of the user to the vehicle-mounted device 20, and can be provided to the user via the vehicle-mounted device 20. When the control unit 21 determines that the above-mentioned series of processes should be completed (S27: YES), the control unit 21 ends the processes.

When it is determined by the above-mentioned processing that any event has occurred in the vehicle based on the information outside the vehicle and the information inside the vehicle periodically acquired by the vehicle-mounted device 20, the content corresponding to the event generated in the vehicle is the content corresponding to the vehicle-mounted device. Played at 20. Therefore, the user (passenger) in the vehicle can view (experience) the content corresponding to the event generated in the vehicle on the in-vehicle device 20. For example, by registering the content related to the predetermined address in association with the event occurrence condition that the vehicle has entered the predetermined address, when the vehicle enters the predetermined address, it is associated with the predetermined address. The existing content can be reproduced by the in-vehicle device 20. Specifically, by registering video content including images and sounds (music) related to a predetermined place in association with a predetermined place, when the vehicle travels in the predetermined place, the image of the place and the image of the place and the video content are registered. Music is played. The content may include not only the image and the sound but also the light (illumination light), vibration, fragrance, or air-conditioning content corresponding to the image or the sound. As a result, an image corresponding to the traveling place (current location) of the vehicle can be displayed, a sound can be output, and light, vibration, fragrance, and wind can be generated. In addition, for example, by registering the content of lowering the temperature of the air (wind) blown from the air conditioner by a predetermined temperature in association with the event occurrence condition that the passenger feels angry, the passenger can be informed. When feelings of anger occur, the temperature of the air blown from the air conditioner can be lowered by a predetermined temperature, and as a result, it can be expected that the anger of the passenger will be calmed down. In addition, by registering the amount of air blown from the air conditioner according to the traveling speed of the vehicle (event generation condition), it is possible to control the air volume according to the traveling speed of the vehicle, and the air received by the outer surface of the vehicle. It is possible to feel (experience) resistance in the car.

Further, the content corresponding to the predetermined place may be registered in association with the event occurrence condition that the vehicle has entered the predetermined place such as an intersection, a bridge, a tunnel, or a highway. In this case, while the vehicle is traveling in the predetermined place, the content associated with the predetermined place can be reproduced by the in-vehicle device 20. Further, the content related to the advertisement of this store may be registered in association with the event occurrence condition that the distance to the predetermined store is equal to or less than the predetermined distance. In this case, when the distance to the predetermined store becomes less than or equal to the predetermined distance, the content related to the advertisement of the store can be reproduced by the in-vehicle device 20. In addition, in correspondence with the event occurrence condition that the destination is a beach and the distance to the destination is less than a predetermined distance, advertisements for stores selling floats or parasols used for swimming, or floats, parasols, etc. You may register the content related to the advertisement that encourages the purchase of. In this case, when the distance to the destination beach is less than or equal to a predetermined distance, the in-vehicle device 20 can reproduce the content related to the advertisement of a nearby store or the advertisement for urging the purchase of a float, a parasol, or the like. .. Further, the content related to the predetermined place may be registered in association with the event occurrence condition that the predetermined place is set as the destination. In this case, when a predetermined place is set as the destination, the in-vehicle device 20 can reproduce the content related to the predetermined place until the predetermined place is reached. In addition, content for brightening the passenger's feelings may be registered in association with the event occurrence condition that the passenger has a negative emotion (for example, sadness, anger, disgust, etc.). In this case, when the occupant has a negative emotion, the in-vehicle device 20 can reproduce the content for brightening the occupant's feelings. Further, different contents may be registered in correspondence with the event occurrence condition that the vehicle is running or stopped. In this case, the contents according to the state of the vehicle (running or stopped) are mounted on the vehicle. It can be regenerated by the device 20.

In addition, even if the content related to the coupon that can be used at this store is registered in association with the event occurrence condition that the passenger feels joy when the distance to the predetermined store is less than the predetermined distance. good. In this case, if the passenger feels joy when the distance to the predetermined store is less than or equal to the predetermined distance, the content related to the coupon of this store can be reproduced on the in-vehicle device 20 and the coupon can be provided. can. In this way, a condition that combines the position information of the vehicle and the emotion of the occupant may be set as the event occurrence condition. Furthermore, even if the content that generates the scent of cosmos (fragrance) is registered in association with the event occurrence condition that there is a cosmos field around the vehicle or the event occurrence condition that the scent of cosmos is scented around the vehicle. good. In this case, when the vehicle travels near the cosmos field, the scent (fragrance) of cosmos can be generated by the fragrance generating unit 32 of the in-vehicle device 20. In this case, even when the windows and doors of the vehicle are closed, the user inside the vehicle can experience the same fragrance as outside the vehicle.

Further, romantic content may be registered in association with the event occurrence condition that a man and a woman in their twenties are seated in the front seats and the two people are feeling joy. In this case, when a man or woman in his twenties sitting in the front seat feels joy, romantic content can be reproduced on the in-vehicle device 20. By registering different contents depending on the event occurrence condition that men and women in their 20s are seated in the front seats and the event occurrence conditions that men and women in their 60s are seated in the front seats, It is possible to provide content according to the age group. For example, music that was popular when men and women in their 60s were in their 20s may be registered in association with the event occurrence condition that men and women in their 60s are seated in the front seats. In addition, in correspondence with the event occurrence condition that the siren sound of the emergency vehicle is generated around the vehicle, the content for notifying that the emergency vehicle is approaching or notifying the emergency vehicle that the road is to be handed over. You may register the contents for the purpose. In this case, when the emergency vehicle is approaching, the in-vehicle device 20 can reproduce the corresponding content, so that the passenger can be notified of the approach of the emergency vehicle.

In addition, content corresponding to the attributes of the passengers seated in each seat may be registered, and in this case, different contents can be reproduced according to the seated position of each passenger. For example, the image content displayed on the display unit 25 visible from the front seat may be different depending on whether an adult is seated in the front seat (passenger seat) or a child is seated. It is possible. Further, when a plurality of display units 25 are provided, the image content provided to the passenger is provided when one passenger is seated in the front seat and when the passenger is seated in the rear seat. It is possible to display the display on the display unit 25 that can be visually recognized by the passenger.

As described above, in the present embodiment, it is determined which event is occurring in the vehicle based on the generation condition set by combining the in-vehicle information and the out-of-vehicle information, and the content corresponding to the event generated in the vehicle is determined. Can be reproduced by the in-vehicle device 20. Therefore, by setting the event occurrence conditions based on the vehicle's travel location (current location), season, time zone, weather, number of passengers, attributes of each passenger, seating position, emotions, etc., the event that occurred in the vehicle can be determined. Different contents can be played depending on the situation. The content corresponding to each event may be registered for each user, and in this case, even if a similar event occurs, it is possible to play different content for each user. Further, the occurrence condition of each event may be registered for each user, and in this case, it is possible to determine whether or not the event has occurred under different conditions for each user.

In the present embodiment, the in-vehicle device 20 detects the reaction of the user during the reproduction of the content provided by the server 10, and the content to be reproduced can be appropriately switched according to the reaction of the user. For example, when the emotion of the user viewing the content changes from relaxed to disgusted, the server 10 stops the transmission of the content being provided to the in-vehicle device 20 and starts the transmission of different content. Further, when the user who views the content feels fear, the server 10 can switch to the content that does not make the passenger feel fear by starting the transmission of different content. The user's reaction suitable for transmitting each content is registered in the server 10 (content DB 12b), and the server 10 is based on the registered content of the content DB 12b and the user's current reaction. You can properly select the content to be provided to the server. Thereby, it is possible to provide the content according to the reaction (emotion and behavior) of the user (vehicle occupant) who views the content. The user's reaction may be a physical reaction (gesture or action) or a mental reaction (emotion).

In the present embodiment, the vehicle-mounted device 20 outputs (reproduces) content such as image, sound, light (illumination light), vibration, fragrance, or air conditioning to entertain the passengers in the vehicle and to entertain the passengers in the vehicle. It can be expected to promote the normalization of negative emotions such as excitement. That is, the content can change the emotions of the passengers in the vehicle. In the present embodiment, the user's reaction (emotion, action, etc.) is registered in the server 10 (user DB 12a) for each user during or at the end of playback of each content. This makes it possible to provide content that takes into account the user's past reactions when the user uses the vehicle from the next time onward.

In the present embodiment, the content providing process according to the event generated in the vehicle is not limited to the process shown in FIGS. 7 and 8. For example, the content may be provided according to the vehicle type of the vehicle on which the in-vehicle device 20 is mounted. In this case, the storage unit 12 of the server 10 stores the content DB 12 for each vehicle type. The content DB 12b shown in FIG. 3B may be provided with a vehicle type information (vehicle type ID) column, and the content data of each event may be stored in association with the vehicle type information (vehicle type ID). Further, the vehicle type information (vehicle type ID) of the vehicle on which the vehicle-mounted device 20 is mounted is registered (stored) in the storage unit 22 of the vehicle-mounted device 20. FIG. 11 is a flowchart showing another example of the content provision processing procedure. The process shown in FIG. 11 is the process shown in FIGS. 7 and 8 in which step S161 is added between YES in step S14 and step S15. In FIG. 11, the illustration of each step in FIG. 8 is omitted. In the process shown in FIG. 11, when the control unit 21 of the in-vehicle device 20 determines that the occurrence condition of any of the events is satisfied (S14: YES), the vehicle type information stored in the storage unit 22 is read out (S161). .. Then, the control unit 21 transmits the user ID of the passenger (user), the destination information of the in-vehicle device 20, the event ID of the event that has occurred, and the vehicle type information to the server 10 to generate the event. Is notified to the server 10 (S15). The control unit 11 of the server 10 reads out the content corresponding to the event notified from the vehicle-mounted device 20 from the content DB 12b corresponding to the vehicle model information received from the vehicle-mounted device 20 (S16), and transmits the content to the vehicle-mounted device 20 (S17). .. According to the process shown in FIG. 11, the server 10 can provide the content according to the vehicle type of the vehicle. Therefore, even if the same event occurs, it is possible to provide different contents depending on the vehicle type. The vehicle type information is not limited to the configuration stored in the storage unit 22 in advance, and may be acquired by a user operation via the input unit 24, for example, or may be acquired from another in-vehicle device mounted on the vehicle. May be good.

In the process of acquiring in-vehicle information by the in-vehicle device 20 of the present embodiment, a process of detecting the attributes, actions, and emotions of the occupant based on the photographed image, utterance voice, or vital data of the occupant is performed by, for example, another server. May be. In this case, the in-vehicle device 20 transmits to another server each time the photographed image, spoken voice or vital data of the passenger is acquired, and the other server gets on the vehicle based on the photographed image, spoken voice or vital data of the passenger. Detects a person's attributes, actions and emotions. Then, the in-vehicle device 20 acquires the attributes, actions, and emotions of the passenger detected by the other server. With such a configuration, the processing load on the in-vehicle device 20 can be reduced.

In the present embodiment, the content corresponding to the event generated in the vehicle is provided from the server 10 to the in-vehicle device 20, but the present embodiment is not limited to this configuration. For example, instead of classifying the contents for each event, the server 10 may provide the contents according to the reaction of the passengers of the vehicle from the plurality of contents to the in-vehicle device 20. For example, when providing each content, the content is prepared for each of the optimum emotions of the passenger, and the server 10 notifies from the in-vehicle device 20 without determining which event has occurred in the vehicle. The content according to the reaction of the passenger is provided to the in-vehicle device 20. Even in this case, the content according to the reaction of the passenger can be appropriately switched and provided to the passenger.

(Embodiment 2)
An information processing system for reproducing image contents on a display unit 25 (display) provided in a vehicle window will be described. In the information processing system of the present embodiment, the display unit 25 of the in-vehicle device 20 is a head-up display that projects content onto a transparent window, or a display that is integrally formed with the glass of the window. Therefore, in the in-vehicle device 20 of the present embodiment, the passenger in the vehicle can see the state in which the image content displayed on the display unit 25 is superimposed on the scenery outside the vehicle that can be visually recognized through the window. The window provided with the display unit 25 may be a front window, a rear window, or a side window, and when the roof is made of glass or a transparent synthetic resin plate, the display unit is also provided on the roof. 25 may be provided.

Since the information processing system of the present embodiment can be realized by the same apparatus as the information processing system of the first embodiment, the description of the configuration will be omitted. The content DB 12b stored in the storage unit 12 of the server 10 of the present embodiment is slightly different from the configuration of the content DB 12b of the first embodiment. FIG. 12 is a schematic diagram showing a configuration example of the content DB 12b of the second embodiment. The content DB 12b shown in FIG. 12 includes a superposition condition sequence in addition to the configuration of the content DB 12b of the first embodiment shown in FIG. 3B. The superimposition condition column stores information on conditions for displaying each content on the display unit 25 and superimposing it on the scenery outside the vehicle in association with the content ID. For example, a store or a building in which the content should be superimposed and displayed. Information, landscape information (information on cityscapes such as mountain roads, beaches, buildings, etc.), addresses, etc. are memorized. For contents that do not need to be superimposed on the scenery outside the vehicle, the overlay condition may not be registered. The superposition condition is added or changed by the control unit 11 when the control unit 11 acquires an instruction for addition or change via the communication unit 13 or the input unit 14. The stored content of the content DB 12b of the present embodiment is not limited to the example shown in FIG.

In the information processing system of the present embodiment, the server 10 and the in-vehicle device 20 execute the same processing as those shown in FIGS. 7 to 10. As a result, the in-vehicle device 20 detects the presence or absence of an event based on the information outside the vehicle and the information inside the vehicle, and when any of the events occurs, the content corresponding to the generated event is provided from the server 10 to the in-vehicle device 20. , Reproduced by the in-vehicle device 20. In the information processing system of the present embodiment, the image content is displayed so as to be superimposed on the scenery seen from the outside of the vehicle from the window. Therefore, when the in-vehicle device 20 displays the image content on the display unit 25, the image content is displayed. It is necessary to specify the position.

FIG. 13 is a flowchart showing an example of an image content display processing procedure, and FIG. 14 is a schematic diagram showing an image content display example. The process shown in FIG. 13 is executed when the control unit 21 of the in-vehicle device 20 causes the display unit 25 to display the image content received from the server 10 in step S18 in FIG. 8, for example. In the information processing system of the present embodiment, the control unit 21 of the in-vehicle device 20 determines whether or not the image content has been received from the server 10 (S61), and when it is determined that the image content has not been received (S61: NO). ), Wait until it is received. When the control unit 21 receives content other than image content, for example, audio content, light, vibration, fragrance, or air-conditioned content from the server 10, the speaker 29, the lighting unit 30, the vibration generation unit 31, and the fragrance generation unit 32. , The air conditioning control unit 33 reproduces the content, respectively.

When the control unit 21 determines that the image content has been received (S61: YES), the control unit 21 determines whether or not the superposition condition is set for the received image content (S62). In the present embodiment, when the control unit 11 of the server 10 transmits the content corresponding to the event or the content corresponding to the user's reaction to the in-vehicle device 20, the superposition condition set for the content is set together with the content. Send. Therefore, the control unit 21 can determine whether or not the superposition condition is set for the received image content according to whether or not the superposition condition is received together with the image content. When it is determined that the superposition condition is not set (S62: NO), the control unit 21 shifts the process to step S68 and displays the received image content on the display unit 25 (S68). At this time, the control unit 21 may display the image content in the display area set by default for the display unit 25.

When it is determined that the superposition condition is set (S62: YES), the control unit 21 reads the map data around the current location of the vehicle from the map data 22D (S63). For example, the control unit 21 reads out map data within a range of several tens of kilometers in front of the vehicle. Then, the control unit 21 determines whether or not there is an object that matches the superposition condition among the objects such as buildings and stores included in the map data (S64). For example, when a predetermined store (for example, a predetermined convenience store) is set as a superposition condition, the control unit 21 determines whether or not the map data includes the predetermined store (predetermined convenience store). Further, when a predetermined landscape (landscape) such as a mountain road or a seaside is set as a superposition condition, the control unit 21 determines whether or not the map data includes the predetermined landscape. When it is determined that there is no object that matches the superposition condition (S64: NO), the control unit 21 returns to the process of step S61, and processes the image contents sequentially received from the server 10 in steps S62 to S64. I do. In the present embodiment, it is determined whether or not there is an object that matches the superposition condition among the objects in the map data, but the present embodiment is not limited to this configuration. For example, the vehicle-mounted device 20 may determine whether or not there is an object that matches the superposition condition in the landscape image based on the landscape image obtained by capturing the landscape outside the vehicle with the camera 27. Specifically, when the object that matches the superposition condition is a predetermined convenience store, the in-vehicle device 20 detects whether or not the signboard of the predetermined convenience store is included in the landscape image taken by the camera 27. If it can be detected, it may be determined that there is an object that matches the overlay condition in the landscape image. In this case, it is possible to identify an object that matches the superposition condition even for an object that is not included in the map data.

When it is determined that there is an object that matches the superposition condition (S64: YES), the control unit 21 captures the scenery outside the vehicle with the camera 27 (S65). Here, the scenery equivalent to the scenery when the passenger in the car looks at the outside of the car is taken through the window provided with the display unit 25 for displaying the image contents. Specifically, the view is the same as when a passenger seated in the front seat (driver's seat or passenger's seat) looks out of the vehicle through the front window, and the passenger seated in the rear seat has a side window. Take a picture of the same scenery as when looking outside the car through the window. The camera 27 is provided at a position where a passenger in each seat in the vehicle can take a picture equivalent to the case where the passenger looks at the outside of the vehicle through each window.

The control unit 21 identifies an object that matches the superposition condition from the objects included in the captured landscape image around the vehicle (S66). At this time, the control unit 21 identifies an object that matches the superposition condition in the landscape image by associating each object in the map data with each object in the landscape image. When associating each object in the map data with each object in the landscape image, the control unit 21 may, for example, include information indicating each object in the map data (for example, a signboard of a predetermined store) in the landscape image. Is detected, and each object in the map data is associated with the detected information. The control unit 21 specifies a position where the image content should be displayed with respect to the display area of the display unit 25 based on the position in the landscape image of the object that matches the specified superposition condition (S67). Here, the control unit 21 specifies a position where the image content should be superimposed and displayed on a landscape image equivalent to that when the outside of the vehicle is viewed through each window. Then, the control unit 21 displays the image content at the specified display position with respect to the display unit 25 provided in the window in which the object matching the superposition condition is included in the landscape image (S68).

FIG. 14A shows a view when the outside of the vehicle is viewed from the front seat through the front window, and FIG. 14B shows a state in which three image contents C are superimposed on the view of FIG. 14A. The example shown in FIG. 14B is a display example in which a predetermined store is set as a superposition condition for the image content C indicating a product (buoy or beach sandals) sold at the predetermined store. In this case, the image content C showing the product (buoy or beach sandals) is displayed at a position overlapping the predetermined store in the view with respect to the view when the outside of the vehicle is viewed from the front seat through the front window. .. Further, the example shown in FIG. 14B is a display example in which the superposition condition is not set for the image content C for displaying the message "Would you like to buy sea bathing goods?". The image content of the message is displayed at an appropriate position. Although it is displayed at the upper left position of the display area in FIG. 14B, the display position is not limited to this. By superimposing the image content C on the scenery that can be visually recognized through the window in this way, it is possible to provide the passenger with information about each object in the scenery by the image content C.

Each time the control unit 21 receives the image content from the server 10, the control unit 21 performs the above-mentioned processing and displays the image content at an appropriate position of the display unit 25. By the process described above, the in-vehicle device 20 can display the image content on the display unit 25 provided in the window of the vehicle, and at that time, the image content is displayed at an appropriate position by superimposing the image content on the scenery seen through the window. Can be made to. Therefore, the passenger can check the scenery through the window and receive information about each object in the scenery as image contents. Further, in the present embodiment, when the content related to the predetermined place is registered in association with the event occurrence condition that the predetermined place is set in the destination, the purpose is when the predetermined place is set in the destination. Until it arrives at the ground, the in-vehicle device 20 can reproduce the content related to the predetermined place. At this time, the vehicle-mounted device 20 may display the image content at a position corresponding to the direction connecting the predetermined place from the vehicle in the scenery seen through the window. In this case, the passenger can enjoy traveling by vehicle while recognizing the direction of the destination. Also in this embodiment, the content corresponding to the event generated in the vehicle is provided from the server 10 to the in-vehicle device 20 and reproduced by the in-vehicle device 20. Further, since the content provided from the server 10 is appropriately changed according to the reaction of the passenger to the content reproduced in the vehicle, it is possible to provide the content suitable for the reaction of the passenger. When displaying the image content on the display unit 25 provided in the window of the vehicle, it is not necessary to superimpose the image content on the scenery that can be visually recognized through the window. For example, the image content may be displayed in an area that does not interfere with the driving of the vehicle, and an area for displaying the image content may be set in advance.

Also in this embodiment, the same effect as that of the first embodiment can be obtained. Further, in the present embodiment, when displaying the image content on the display unit 25 provided in the window of the vehicle, it is possible to display the image content in association with each object in the scenery visually recognized through the window. Therefore, the passenger can visually recognize the scenery through the window and obtain information about each object in the scenery from the image contents. Further, also in the information processing system of the present embodiment, the modification described as appropriate in the first embodiment can be applied. In the present embodiment, the in-vehicle device 20 is configured to increase the transparency (transmittance) of the image content when displaying the image content on the display unit 25 provided in the window of the vehicle. It may be configured to display the image content intermittently. With such a configuration, even when the image content is displayed on the display unit 25 provided on the front window, the image content can be displayed so as not to interfere with the driving of the vehicle. For example, the in-vehicle device 20 may be configured to set the transmittance of the image content to a predetermined value (for example, 50%) or more and display the image content every several tens of seconds for only a few seconds. You may. Further, the display area in which the image content is displayed may be fixed to the display unit 25 provided in each window of the vehicle, and the ratio of the display area of the image content to the window is limited to a predetermined value or less. You may be.

(Embodiment 3)
An information processing system in which a display unit 25 (display) is provided on a front window, a rear window, left and right side windows, and a roof of a vehicle will be described. Also in the present embodiment, the passenger in the vehicle sees a state in which the image content displayed on the display unit 25 is superimposed on the scenery outside the vehicle that can be visually recognized through the window. Since the information processing system of the present embodiment can be realized by the same apparatus as the information processing system of the second embodiment, detailed description of the configuration will be omitted.

FIG. 15 is a block diagram showing a configuration example of the in-vehicle device 20 of the third embodiment. The vehicle-mounted device 20 of the present embodiment has the same configuration as the vehicle-mounted device 20 of the first embodiment shown in FIG. In the in-vehicle device 20 of the present embodiment, the display unit 25 is provided on the front display 25a provided on the front window, the rear display 25b provided on the rear window, the left side display 25c provided on the left side window, and the right side window. The right side display 25d provided and the roof display 25e provided on the roof are included.

Further, in the in-vehicle device 20 of the present embodiment, the camera 27 includes an in-vehicle camera 27a for photographing a passenger in the vehicle and an external camera 27b for photographing the scenery around the vehicle (outside the vehicle). .. The in-vehicle camera 27a may be provided at a position where the passengers seated in the respective seats can be photographed, and a plurality of passengers seated in the plurality of seats may be photographed by one camera. It may be configured. The external camera 27b is a camera capable of photographing all directions of the front and rear, left and right, and above of the vehicle. For example, a front camera for photographing the front of the vehicle, a rear camera for photographing the rear of the vehicle, and the left side of the vehicle. It may have a left-hand camera for taking a picture, a right-hand camera for taking a picture of the right side of the vehicle, and an upper camera for taking a picture of the upper part of the vehicle. Further, the external camera 27b may be composed of an omnidirectional camera (360 degree camera), and the omnidirectional camera may be provided on the roof of the vehicle, for example.

In the information processing system of the present embodiment, the server 10 and the in-vehicle device 20 execute the same processing as the processing shown in FIGS. 7 to 10. As a result, the in-vehicle device 20 detects the presence or absence of an event based on the information outside the vehicle and the information inside the vehicle, and when any of the events occurs, the content corresponding to the generated event is provided from the server 10 to the in-vehicle device 20. , Reproduced by the in-vehicle device 20. In the information processing system of the present embodiment, the image contents are displayed on one or more of the front display 25a, the rear display 25b, the side displays 25c, 25d, and the roof display 25e, and the scenery seen from the outside of the vehicle from each window. The image content is displayed superimposed on. In the present embodiment, the display processing of the image content is different from the second embodiment shown in FIG. 13, so only the different parts will be described.

FIG. 16 is a flowchart showing an example of the image content display processing procedure of the third embodiment, and FIG. 17 is a schematic diagram showing a display example of the image content. The process shown in FIG. 16 is the process shown in FIG. 13 in which steps S71 to S75 are added instead of steps S65 to S67. Further, the process shown in FIG. 16 is also executed when the control unit 21 of the in-vehicle device 20 reproduces the image content received from the server 10 in step S18 in FIG. 8 on the display unit 25, for example.

In the information processing system of the present embodiment, the control unit 21 of the in-vehicle device 20 performs the processes of steps S61 to S64 shown in FIG. As a result, when the image content is received from the server 10, the control unit 21 displays the image content in a predetermined display area if the overlay condition is not set for the image content (S68). The predetermined display area may be, for example, an area on the passenger side of the front display 25a, or may be an appropriate area of the other displays 25b to 25e. On the other hand, if the superposition condition is set for the image content, the control unit 21 determines whether or not there is an object that matches the superposition condition in the map data around the vehicle.

When it is determined that there is an object that matches the superposition condition in the map data (S64: YES), the control unit 21 captures the scenery around the vehicle with the camera 27 (S71). Here, the control unit 21 captures the front-back and left-right omnidirectional views of the vehicle. Then, the control unit 21 identifies an object (matching object) that matches the superposition condition from the objects included in the captured omnidirectional landscape image around the vehicle (S72). At this time, the control unit 21 identifies the matching object in the landscape image by associating each object in the map data with each object in the landscape image. When associating each object in the map data with each object in the landscape image, the control unit 21 may, for example, include information indicating each object in the map data (for example, a signboard of a predetermined store) in the landscape image. Is detected, and each object in the map data is associated with the detected information. Next, the control unit 21 identifies the displays 25a to 25e on which the image content should be displayed based on the position of the specified matching object in the landscape image (S73). For example, when the specified matching object is included in the front image of the front of the vehicle, the control unit 21 specifies the front display 25a as a display to display the image content. When the camera 27 is an omnidirectional camera that captures all directions of the vehicle, the control unit 21 intervenes when, for example, a predetermined position in the vehicle (for example, the driver's seat) and a matching object in the landscape image are connected. The window is specified, and the display provided in the specified window is specified as the display to be displayed.

The control unit 21 specifies a position where the image content should be displayed with respect to the display area of the specified display (S74). Here, the control unit 21 specifies a position in the display area of the specified display on which the image content should be superimposed and displayed on a landscape image equivalent to that when the outside of the vehicle is viewed through a window in which the matching object can be visually recognized. Then, the control unit 21 displays the image content at the specified display position on the specified display (display unit 25) (S75). The upper side of FIG. 17A shows the state of looking at the outside of the vehicle from the inside of the vehicle through the front display 25a, and the lower side shows the state of looking at the outside of the vehicle from the inside of the vehicle through the left side display 25c. In the example shown in FIG. 17A, there is an object to be displayed in which the image content is superimposed in front of the vehicle, and the image content C is displayed so as to overlap the two objects (building, store). On the other hand, since there is no object on the left side of the vehicle on which the image content should be superimposed and displayed, the image content is not displayed on the left side display 25c.

After the processing of step S75, the control unit 21 returns to the processing of step S71 and repeats the processing of steps S71 to S75. Specifically, the control unit 21 captures the scenery around the vehicle, identifies an object that matches the superposition condition in the scenery image, and displays a display (window) capable of superimposing the image content on the specified object. The display position is specified, and the image content is displayed on the specified display and display position. The upper side of FIG. 17B shows the state of looking at the outside of the vehicle from the inside of the vehicle through the front display 25a, and the lower side shows the state of looking at the outside of the vehicle from the inside of the vehicle through the left side display 25c. The example shown in FIG. 17B shows a state in which the surrounding landscape changes as the vehicle moves forward from the state shown in FIG. 17A. In the example shown in FIG. 17B, since there is no object to be displayed on top of the image content in front of the vehicle, the image content is not displayed on the front display 25a. On the other hand, on the left side of the vehicle, there is an object to be displayed with the image content superimposed, and the image content C is displayed so as to overlap the two objects (building, store).

In step S72, the control unit 21 repeats the above-described processing until it becomes impossible to identify an object that matches the superposition condition in the landscape image. As a result, the image content received from the server 10 can be superimposed on the scenery that can be visually recognized through the display (window), and information about each object in the scenery can be provided to the passenger by the image content C. .. Each time the control unit 21 receives the image content from the server 10, the control unit 21 performs the above-mentioned processing and displays the image content at an appropriate position on any of the displays 25a to 25e.

Also in this embodiment, the same effect as that of each of the above-described embodiments can be obtained. Further, in the present embodiment, the image content received from the server 10 is displayed on the display (window) and the position corresponding to the object (building, store, etc.) to be displayed by superimposing the image content. Therefore, the passenger can receive information about the object in the scenery that can be visually recognized through the display (window) in the image content. Also in this embodiment, the content corresponding to the event generated in the vehicle is provided from the server 10 to the in-vehicle device 20 and reproduced by the in-vehicle device 20. Further, since the content provided from the server 10 is appropriately changed according to the reaction of the passenger to the content reproduced in the vehicle, it is possible to provide the content suitable for the reaction of the passenger. Further, also in the information processing system of the present embodiment, it is possible to apply the modification described as appropriate in each of the above-described embodiments.

In the configuration in which the display unit 25 (display) is provided in a plurality of windows of the vehicle as in the above-described embodiments 2 and 3, the running state of the vehicle such as running, stopped or parked, the running speed of the vehicle, and automatic running In the case of a vehicle capable of the above, the display for displaying the image content may be switched depending on whether or not the vehicle is in automatic driving (automatic driving) by the automatic driving function. For example, when the vehicle is running, only the display provided in the side window is targeted for displaying the image content, and when the vehicle is parked, the displays provided in all windows are targeted for display. , The display to be displayed may be switched. With such a configuration, the image content can be displayed without disturbing the driver's driving when the vehicle is running, and the image content can be reproduced using all the displays when parking.

The embodiments disclosed this time should be considered to be exemplary in all respects and not restrictive. The scope of the present invention is indicated by the scope of claims, not the above-mentioned meaning, and is intended to include all modifications within the meaning and scope equivalent to the scope of claims.

10 Server 11 Control unit 12 Storage unit 13 Communication unit 14 Input unit 15 Display unit 20 In-vehicle device 21 Control unit 22 Storage unit 23 Communication unit 25 Display unit 27 Camera 28 Microphone 29 Speaker 30 Lighting unit 31 Vibration generation unit 32 Fragrance generation unit 33 Air conditioning control unit 34 Vital data acquisition unit

Claims (12)

The content is played by the content playback unit provided in the vehicle, and the content is played back.
Detecting the reaction of the user who views the content,
An information processing method including a process of performing content control including specifying the content to be reproduced next, or continuing or ending the reproduction of the content according to the detected reaction. The information processing method according to claim 1, wherein the content reproduction unit includes a display provided in the window of the vehicle. Acquires a photographed image of a user who views the content, and obtains a photographed image.
The information processing method according to claim 1 or 2, which comprises a process of detecting an action or an emotion showing a reaction of the user based on the acquired captured image. Acquire the utterance voice of the user who views the content,
The information processing method according to any one of claims 1 to 3, which includes a process of detecting the reaction of the user based on the acquired spoken voice. Information of the user who detects the reaction is set for each of the contents.
From the users who view the content, identify the user who detects the reaction, and
Detects the reaction of the identified user and
The information processing method according to any one of claims 1 to 4, which includes a process of controlling the content according to the detected reaction. For each of the plurality of events, the event occurrence condition and the plurality of contents corresponding to the event are stored in association with each other.
Judge whether or not the occurrence condition of any event is satisfied,
Any of claims 1 to 5, which includes a process of playing a plurality of contents corresponding to the event in an order according to the reaction of a user who views the contents when it is determined that the occurrence condition of any of the events is satisfied. The information processing method described in one. Acquire the location information of the vehicle and
The information processing method according to claim 6, further comprising a process of determining whether or not the occurrence condition of any event is satisfied based on the acquired position information. Acquire the user's actions or emotions and
The information processing method according to claim 6 or 7, further comprising a process of determining whether or not the occurrence condition of any event is satisfied based on the acquired action or emotion of the user. The content and the reaction of the user who is viewing the content are stored in association with each other.
The information processing method according to any one of claims 1 to 8, further comprising a process of controlling the content according to the reaction of the user who views the content based on the stored content and the reaction of the user. The vehicle is provided with displays on the front window, side window, rear window, and roof, and has an fragrance generating device that generates fragrance or an air conditioning control device that controls the state of the emitted air.
The content includes image content.
The process of displaying the image content on the display and generating fragrance according to the content of the image content by the fragrance generating device or discharging air according to the content of the image content by the air conditioning control device is included. The information processing method according to any one of claims 1 to 9. The content playback unit provided in the vehicle and
A playback processing unit that reproduces content by the content reproduction unit, and
A detector that detects the reaction of the user who views the content,
An information processing system including a content control unit that controls content including the continuation or end of reproduction of the content or the identification of the content to be reproduced next according to the detected reaction. The content is played by the content playback unit provided in the vehicle, and the content is played back.
Detecting the reaction of the user who views the content,
A program that causes a computer to execute a process of controlling content including specifying the content to be reproduced next, or continuing or ending the reproduction of the content according to the detected reaction.

Images