JP7122408B2 - Information processing device and information processing method - Google Patents

Description

The present invention relates to an information processing device and an information processing method.
Regarding.

There is a display technology of augmented reality (AR) that superimposes an image on a scenery visually recognized by a person riding in a vehicle. For example, Patent Literature 1 discloses an image processing device that corrects an object according to the movement direction of the vehicle in a head-up display system that projects the object onto the windshield of the vehicle.

JP 2019-202698 A

However, the invention according to Patent Document 1 only projects an object for driving assistance onto the windshield, and does not present content other than driving assistance to the passenger.

An object of one aspect is to provide an information processing apparatus or the like capable of suitably displaying AR content.

An information processing apparatus according to one aspect includes a display unit that displays an image using a window of a vehicle as a display area, a first acquisition unit that acquires data indicating the state of the vehicle, and an object to be displayed on the window. a second acquisition unit, a determination unit that determines a placement position of the object in the real space according to the situation of the vehicle, and a position on the window so that the object can be seen at the placement position through the window. a display control unit that controls a display position of the object , wherein the first acquisition unit acquires a destination of the vehicle; the second acquisition unit acquires an object according to the destination; The determining unit refers to a table that specifies a relative position of each object from the vehicle to place the object, and determines the relative position of the object from the vehicle. Acquiring a detection result of detecting the position and shape of a peripheral object existing in the real space, and displaying the object based on the position and shape of the peripheral object existing at the relative position in the real space. to control .

In one aspect, AR content can be displayed favorably.

1 is an explanatory diagram showing a configuration example of an object display system; FIG. It is a block diagram which shows the structural example of a server. It is a block diagram which shows the structural example of an in-vehicle apparatus. FIG. 4 is an explanatory diagram showing an example of record layouts of a content DB, a vehicle DB, and a display table; FIG. 2 is a schematic diagram showing the inside of the vehicle; FIG. 4 is an explanatory diagram showing a display example of objects according to destinations; FIG. 5 is an explanatory diagram showing an example of display of objects according to surrounding structures; FIG. 11 is an explanatory diagram showing an example of display of objects according to peripheral moving bodies; FIG. 5 is an explanatory diagram showing an example of display of objects according to vehicle speed; FIG. 5 is an explanatory diagram showing a display example of a message from another vehicle; 4 is a flowchart showing an example of a processing procedure executed by an in-vehicle device; 4 is a flowchart showing an example of a processing procedure executed by an in-vehicle device; FIG. 10 is an explanatory diagram showing an overview of Embodiment 2; 9 is a flowchart showing an example of a processing procedure executed by an in-vehicle device according to Embodiment 2; FIG. 11 is an explanatory diagram showing an overview of Embodiment 3;

Hereinafter, the present invention will be described in detail based on the drawings showing its embodiments.
(Embodiment 1)
FIG. 1 is an explanatory diagram showing a configuration example of an object display system. In the present embodiment, an object display system that displays an object related to predetermined AR content on a window of the vehicle 4 (hereinafter referred to as "car window") will be described. The object display system includes a server 1 and an in-vehicle device 2 (information processing device) mounted on a vehicle 4 . Each device is connected via a network N for communication.

The server 1 is a server computer capable of various types of information processing and transmission/reception of information. A device corresponding to the server 1 may be a personal computer or the like. The server 1 is a server computer that manages AR content to be distributed to the in-vehicle device 2 , and distributes the AR content to the in-vehicle device 2 via the network N.

A vehicle 4 is a vehicle used by a user of this system. The vehicle 4 is, for example, an automatic vehicle, but may be a manually operated vehicle that does not have an automatic driving function. An in-vehicle device 2 is mounted in a vehicle 4 and presents AR content to the user. Although only one vehicle 4 is shown in FIG. 1, the server 1 is connected to a plurality of vehicles 4 so as to be communicable.

The in-vehicle device 2 is a device that controls display of objects related to AR content distributed from the server 1 . The in-vehicle device 2 may be a device built into the vehicle 4 like an ECU (Electronic Control Unit), or may be a device externally attached to the vehicle 4 . The in-vehicle device 2 includes display means having a window of the vehicle 4 as a display area, and displays an image on the window of the vehicle. As will be described later in detail, the in-vehicle device 2 displays an object on any one of a plurality of car windows such as the windshield, front door glass, rear door glass, and roof glass of the vehicle 4 as display areas (see FIG. 5). ).

In the present embodiment, the in-vehicle device 2 controls display of objects in the car window according to the situation of the vehicle 4 . As will be described later in detail, the in-vehicle device 2 is configured to detect the destination of the vehicle 4, peripheral objects existing around the vehicle 4, the emotions of a user (person) riding in the vehicle 4, the traveling speed of the vehicle 4 (hereinafter referred to as (described as "vehicle speed"), etc., controls the car window where the object is displayed, the display position of the object in the car window, and the like.

FIG. 2 is a block diagram showing a configuration example of the server 1. As shown in FIG. The server 1 includes a control section 11 , a main storage section 12 , a communication section 13 and an auxiliary storage section 14 .
The control unit 11 has a processor such as one or more CPU (Central Processing Unit), MPU (Micro-Processing Unit), GPU (Graphics Processing Unit), etc., and reads out the program P1 stored in the auxiliary storage unit 14. By executing it, various information processing is performed. The main storage unit 12 is a temporary storage area such as SRAM (Static Random Access Memory), DRAM (Dynamic Random Access Memory), flash memory, etc., and temporarily stores data necessary for the control unit 11 to perform arithmetic processing. Remember. The communication unit 13 is a communication module for performing processing related to communication, and transmits and receives information to and from the outside.

The auxiliary storage unit 14 is a non-volatile storage area such as a large-capacity memory or hard disk, and stores a program P1 and other data necessary for the control unit 11 to execute processing. The auxiliary storage unit 14 also stores a content DB 141 and a vehicle DB 142 . The content DB 141 is a database that stores content to be distributed to the in-vehicle device 2 . The vehicle DB 142 is a database that stores information on vehicles 4 to which content is to be distributed.

Incidentally, the auxiliary storage unit 14 may be an external storage device connected to the server 1 . Moreover, the server 1 may be a multicomputer consisting of a plurality of computers, or may be a virtual machine virtually constructed by software.

Moreover, in the present embodiment, the server 1 is not limited to the configuration described above, and may include, for example, an input unit for receiving operation inputs, a display unit for displaying images, and the like. The server 1 may also include a reader for reading a non-transitory computer-readable storage medium 1a (Non-Transitory Computer-Readable Storage Medium), and read the program P1 from the recording medium 1a.

FIG. 3 is a block diagram showing a configuration example of the in-vehicle device 2. As shown in FIG. The in-vehicle device 2 includes a control unit 21 , a main storage unit 22 , a communication unit 23 , a display unit 24 , a positioning unit 25 , a peripheral imaging unit 26 , an in-vehicle imaging unit 27 , an audio input unit 28 , an audio output unit 29 , and a vehicle speed detection unit 30 . , and an auxiliary storage unit 31 .
The control unit 21 has one or more processors such as CPUs, and performs various information processing by reading and executing the program P2 stored in the auxiliary storage unit 31 . The main storage unit 22 is a temporary storage area such as a RAM, and temporarily stores data necessary for the control unit 21 to execute arithmetic processing. The communication unit 23 is a communication module for performing processing related to communication, and transmits and receives information to and from the outside.

The display unit 24 is display means for displaying an image (object) using the car window as a display area. Specifically, as will be described later, the display unit 24 is a projection device that projects an image using the car window as a screen, or a transparent display that uses the car window itself as a display screen, or the like. The positioning unit 25 is a module that acquires position information indicating the current position of the vehicle 4, and is a module that receives GPS (Global Positioning System) coordinate values, for example.

The periphery imaging unit 26 is a camera that captures an image with the exterior of the vehicle as an imaging range, and captures an image of the periphery of the vehicle 4 . The in-vehicle imaging unit 27 is a camera that captures an image with the interior of the vehicle as an imaging range, and captures the interior of the vehicle. Preferably, the peripheral imaging unit 26 and/or the in-vehicle imaging unit 27 is a camera with a distance sensor using laser light or the like, or a compound eye camera, and is configured to be able to acquire the distance to an object captured in the image. .

The voice input unit 28 is a microphone that collects voice, and acquires the voice inside the vehicle. The audio output unit 29 is a speaker that outputs audio, and outputs audio to the inside of the vehicle. The vehicle speed detector 30 is a sensor that detects the vehicle speed of the vehicle 4 .

The auxiliary storage unit 31 is a nonvolatile storage area such as a hard disk or a large-capacity memory, and stores a program P2 and other data necessary for the control unit 21 to execute processing. Further, the auxiliary storage unit 31 stores map data M and a display table 311 . The map data M is map data to which position information (GPS coordinate values) at each position on the map is added. The display table 311 is a table that stores display rules for objects related to content distributed from the server 1 .

The in-vehicle device 2 may include a reading unit that reads a non-transitory computer-readable storage medium 2a (Non-Transitory Computer-Readable Storage Medium), and may read the program P2 from the recording medium 2a.

FIG. 4 is an explanatory diagram showing an example of record layouts of the content DB 141, the vehicle DB 142, and the display table 311. As shown in FIG.
The content DB 141 includes a content ID column, an object ID column, and an object column. The content ID column stores content IDs for identifying each content. The object ID column stores an object ID for identifying each object in the content in association with the content ID. The object column stores object display data (2D image, 3D image, etc.) in association with the object ID.

The vehicle DB 142 includes a vehicle ID column, a user name column, a user attribute column, an avatar column, and a viewing history column. The vehicle ID column stores vehicle IDs for identifying each vehicle 4 to which content is to be distributed. A user name column, a user attribute column, an avatar column, and a viewing history column are associated with a vehicle ID, respectively, and the name of the user who gets on the vehicle 4, user attribute information (sex, age, etc.), and transmission/reception of messages to be described later. It stores the avatar image used at times and the content viewing history by the user.

The display table 311 includes a content ID column, an object ID column, a display condition column, a position column, an emotion column, and an action column. The content ID column stores content IDs for identifying each content. The object ID column, the display condition column, and the position column are associated with the content ID, respectively, to identify the object ID for identifying each object in the content, the display condition (appearance condition) of the object, and the virtual space of the object in the real space. A typical placement position (default value) is stored. Each of the emotion column and the action column stores patterns of emotions (reactions) of the user who viewed the object, and the display operation and display time of the object according to the emotion of each pattern, in association with the object ID.

For example, when the object with the object ID “Elephant 101” satisfies a predetermined display condition, the distance from the vehicle 4 is 300 m, and the straight-ahead direction of the vehicle 4 is “−30°” (left side 30°). °) direction. If the emotion of the user viewing the object is "excitement", the display operation of "roar cry" continues for "30 seconds". The display control of objects based on the display table 311 will be described later in detail.

FIG. 5 is a schematic diagram showing the interior of the vehicle 4. As shown in FIG. FIG. 5 schematically shows the interior of the vehicle 4 as seen from the rear seats. A vehicle 4 according to the present embodiment will be described with reference to FIG.

The vehicle 4 according to the present embodiment is configured such that the body area around the car window is minimized and the area of the car window is maximized. Specifically, each pillar of the vehicle 4 (A pillar, B pillar, C pillar, roof pillar, etc.; shown in thick lines in FIG. 5) is formed to be long and narrow, and is configured to narrow the visible range blocked by the pillars. there is Each vehicle window (a windshield, a front door glass, a rear door glass, etc.) is provided between each pillar in the vehicle 4 . Further, the entire ceiling of the vehicle 4 according to the present embodiment is made of roof glass.

As described above, the vehicle 4 is provided with display means having a display area of the vehicle window, and displays an image on the vehicle window. Specifically, the vehicle 4 displays images on each of a plurality of vehicle windows such as the windshield, front door glass, rear door glass, roof glass, and rear glass. The display means is, for example, a projector using the car window as a screen, but the car window itself may be configured as a transparent display, and the specific display means is not particularly limited. Further, for example, the roof portion may be shielded instead of being used as a car window (glass), and only an image may be displayed (projected) on the roof.

In addition, the vehicle 4 is equipped with a camera that captures images of the inside and outside of the vehicle. For example, as shown in FIG. 5, a roof pillar of the vehicle 4 is provided with three cameras (in-vehicle imaging units 27) for capturing images inside the vehicle. Although not shown, a plurality of cameras (periphery imaging units 26) for capturing images outside the vehicle are also installed in the same manner, and capture images of the vehicle periphery (for example, the front, sides, rear and/or above the vehicle).

In the following description, for the sake of convenience, an image captured by the vehicle interior imaging unit 27 will be referred to as a "vehicle interior image", and an image captured by the peripheral imaging unit 26 will be referred to as a "surrounding image".

In addition, the vehicle 4 is equipped with a microphone (audio input unit 28) for collecting voices inside the vehicle and a speaker (audio output unit 29) for outputting voices inside the vehicle (both not shown).

As described above, the vehicle 4 is configured so that the view that can be visually recognized by the user is as large as possible. The in-vehicle device 2 superimposes the object on the scenery visually recognized by the user by displaying the object related to the AR content on each car window. In this way, the present system provides the interior of the vehicle 4 as a space for experiencing AR content by superimposing objects on the scenery that can be seen through the car window with the maximum visible range.

Note that the configuration of the vehicle 4 shown in FIG. 5 is an example, and the present embodiment is not limited to this. For example, an in-vehicle imaging camera may be installed at a location other than the pillar, and the vehicle window for displaying the object may be only the windshield.

A specific object display method will be described below, taking as an example a case where the vehicle 4 is heading to a destination desired by the user (for example, a zoo).

FIG. 6 is an explanatory diagram showing a display example of objects according to destinations. For example, the in-vehicle device 2 first receives a destination setting input from the user when starting to travel. The destination is set by voice input from the user, for example. Alternatively, the in-vehicle device 2 may be separately provided with manual operation input means (for example, a touch panel display) so as to receive destination setting input by manual operation.

Note that the operation input means described above is merely an example, and the present embodiment is not limited to this. For example, the in-vehicle device 2 may recognize an operation (gesture) of the user's hand from an in-vehicle image to receive an operation input based on the user's hand operation. Specifically, the in-vehicle device 2 configures the in-vehicle imaging unit 27 as a camera with a distance sensor, acquires a distance image in which each pixel is associated with a measured value of distance as an in-vehicle image, and obtains a distance image from the distance image. Recognizing the user's hand movements. Alternatively, the in-vehicle device 2 may configure the in-vehicle imaging unit 27 as a compound eye camera, and recognize the user's hand motion from a plurality of in-vehicle images. For example, the in-vehicle device 2 displays an operation screen for setting a destination on the car window, and receives a destination setting input on the screen by pointing or the like by recognizing the motion of the user's hand. In this way, the vehicle 4 may accept the destination setting input by recognizing the motion of the user's hand from the in-vehicle image. The same applies to subsequent operation inputs.

When the destination setting input is received, the in-vehicle device 2 acquires from the server 1 content for the user heading to the destination. For example, when the destination is a zoo, the in-vehicle device 2 acquires display objects such as an image of the zoo (for example, a photograph of the front entrance) and 3D images of various animals as content.

In this embodiment, it is assumed that content (object) is prepared according to the destination, but the content may be irrelevant to the destination.

When the destination setting input is received, the in-vehicle device 2 displays an object representing the destination (for example, a photograph of the front entrance of a zoo) on the car window. Specifically, the in-vehicle device 2 displays the object so that it is always visible at the position where the destination is actually located when viewed from the user in the vehicle 4 .

For example, the in-vehicle device 2 identifies the location of the set destination (for example, GPS coordinate values) from the map data M. FIG. Then, the in-vehicle device 2 determines the location of the identified destination as the placement position where the object is virtually placed in the real space visible from the car window. Next, the in-vehicle device 2 acquires position information of the vehicle 4 by the positioning unit 25 . Based on the current position of the vehicle 4 indicated by the acquired position information, the in-vehicle device 2 controls the display position of the object in the car window so that the object can be visually recognized at the arrangement position (located position) determined above. That is, the in-vehicle device 2 controls so that the object is displayed in the direction in which the destination is located. The in-vehicle device 2 displays the object at the above display position.

The in-vehicle device 2 sequentially acquires the position information of the vehicle 4 while the vehicle 4 is running. Based on the current position of the vehicle 4 indicated by the sequentially acquired position information, the in-vehicle device 2 sequentially changes the display position of the object so that the object is always visible in the direction of the destination. This allows the user to visually recognize the location direction of the destination in real time.

Although the object is always displayed until the destination is reached, the illustration of the object is omitted for the sake of convenience in the following drawings (display examples).

In the example of FIG. 6, the object is displayed on the windshield. indicate. For example, the in-vehicle device 2 identifies a car window through which the user in the vehicle can see the destination (arrangement position) based on the arrangement position of the object in the real space and the position and orientation (straight direction) of the vehicle 4. Display objects in car windows. In this way, in the present embodiment, by enabling the display of objects across a plurality of car windows, all directions of the vehicle 4 can be presented as an AR space. The same applies to the following description of displaying an object on a car window other than the windshield according to the arrangement position of the object.

FIG. 7 is an explanatory diagram showing a display example of objects according to surrounding structures. After the vehicle 4 departs, the in-vehicle device 2 makes various objects appear (display) in the car window according to the situation of the vehicle 4, and allows the user to experience AR content.

For example, the in-vehicle device 2 displays the object when the display conditions for each object defined in the display table 311 are cleared. The display conditions include, for example, the entry of the vehicle 4 into a specific area, the elapsed time after departure, the time period, the weather, the view visible from the vehicle window, the emotion, speech, motion, and attributes of the user riding in the vehicle 4. Accordingly, it is specified for each object. The in-vehicle device 2 sequentially determines whether or not the display conditions for each object have been cleared according to these pieces of information.

Note that the display conditions described above are merely examples, and the present embodiment is not limited to these. For example, the in-vehicle device 2 may select an object to be displayed from the user's past content viewing history, and display the object on the car window. For example, the in-vehicle device 2 preferentially selects an object (content) that induces a positive emotion by storing, in the user DB 142, the user's emotion estimated from an in-vehicle image as will be described later, in association with the object. Select and make it visible. In this way, the in-vehicle device 2 may use the content viewing history or the like as a reference for the display condition, and the determination method is not particularly limited.

When it is determined that the display condition is cleared, the in-vehicle device 2 displays the object for which the display condition is cleared on the car window. As an example, FIG. 7 illustrates how a 3D image of an animal is displayed. In this case, the in-vehicle device 2 controls the display position of the object according to the peripheral objects of the vehicle 4 .

For example, the in-vehicle device 2 sequentially detects (recognizes) peripheral objects existing around the vehicle 4 from the peripheral image captured by the peripheral imaging unit 26 . The surrounding objects are, for example, objects such as structures and natural objects (mountains, seas, trees, etc.) existing around the vehicle 4, or mobile objects existing around the vehicle 4, and the like. For example, the in-vehicle device 2 detects the position and shape of each peripheral object from the peripheral image. Specifically, the in-vehicle device 2 configures the peripheral imaging unit 26 as a camera with a distance sensor, similar to the in-vehicle imaging unit 27, and acquires the distance image as the peripheral image. The in-vehicle device 2 recognizes the surrounding objects from the image and simultaneously detects the position (distance, direction) and shape of the recognized surrounding objects.

FIG. 7 shows a case where a structure existing around the vehicle 4 is detected as a peripheral object. A structure is an object that exists at a fixed position in real space, and refers to an artificial object such as a building, a guardrail, or a traffic light. The example of FIG. 7 illustrates how objects are displayed behind buildings and on guardrails. The in-vehicle device 2 controls the display position of the object according to the position and shape of the structure detected from the peripheral image.

For example, the display table 311 predefines a relative position with respect to the vehicle 4, which is a default value of the placement position when the object is placed in the real space. Specifically, as shown in the row of the object ID "Elephant 101" in FIG. 4, the display table 311 defines the distance and direction from the vehicle 4 where the object is arranged.

The in-vehicle device 2 compares the position of the structure detected from the peripheral image with the relative position of the object defined in the display table 311 to determine whether or not the structure exists at the relative position. and determine whether or not the structures overlap. Specifically, the in-vehicle device 2 determines whether or not the distance and direction of the structure detected from the peripheral image are the same or similar to the distance and direction of the object defined in the display table 311 . When it is determined that the structure does not exist at the relative position defined by the display table 311, the in-vehicle device 2 determines the relative position as the arrangement position of the object so that the object can be visually recognized at the arrangement position. Controls the display position of the object in the car window.

When it is determined that a structure exists, the in-vehicle device 2 changes the arrangement position of the object from the specified relative position and adjusts it so that it does not overlap with the structure. For example, the in-vehicle device 2 is placed on the structure, and the arrangement position is determined to be above the structure. In the example of Fig. 7, the object is displayed on the guardrail because the guardrail exists at the prescribed relative position.

The in-vehicle device 2 controls the display position of the object on the car window so that the object can be visually recognized at the arrangement position determined above. Specifically, as described above, the in-vehicle device 2 displays the object on the structure when the arrangement position is determined to be on the structure.

In addition, if the direction in which the object is arranged is substantially the same as the direction of the structure, but the distance of the object is longer than the distance of the structure, the in-vehicle device 2 can arrange the part of the object according to the shape of the structure. Or hide it so that it is completely covered by the structure. In the example of FIG. 7, the object (animal) is displayed behind the building such that part of the object (animal) is hidden behind the building. In this way, the in-vehicle device 2 defines the anteroposterior relationship between the object and the structure based on the distance between them, and depending on the anteroposterior relationship, either displays the entire object or hides part or all of the object. .

The display method described above is merely an example, and the present embodiment is not limited to this. For example, the in-vehicle device 2 may hide the object when the object and the structure overlap. Also, in the above description, the height at which the object is placed is changed by placing it on the structure, but the distance or direction at which the object is placed may be changed. For example, since it is unnatural for an object to be displayed on top of a high-rise building (building), the in-vehicle device 2 places the object on top of the structure ( Alternatively, the object may be hidden without being placed), and the object may be displayed on top of the structure only when the height is below a certain level. Further, for example, when the entire object is hidden behind a structure, the in-vehicle device 2 finely adjusts (changes) the direction in which the object is displayed from the default value so that at least part of the object can be seen. good. Thus, the display method described above is merely an example, and various modifications are conceivable.

In the above description, a structure is mentioned as an object to be detected from the peripheral image, but for example, a natural object such as a mountain, the sea, or a tree may be detected. That is, the in-vehicle device 2 only needs to be able to detect an object existing at a certain position, and the target object is not limited to a structure.

Moreover, although the structure is detected from the peripheral image in the above description, the present embodiment is not limited to this. For example, the map data M may be a three-dimensional map, and the positions and shapes of structures existing around the vehicle 4 may be acquired (detected) from the map data M based on the current position information of the vehicle 4 . In this way, the in-vehicle device 2 only needs to be able to detect structures existing around the vehicle 4, and its detection means is not limited to image recognition.

As described above, the in-vehicle device 2 displays objects related to AR content. Furthermore, the in-vehicle device 2 controls the display of the object according to the reaction of the user viewing the object.

Specifically, the in-vehicle device 2 estimates the emotion of the user viewing the object from the in-vehicle image captured by the in-vehicle imaging unit 27 and/or the in-vehicle audio acquired by the audio input unit 28 . For example, the in-vehicle device 2 estimates emotions from user's facial expressions and actions recognizable from in-vehicle images and/or user utterances recognizable from in-vehicle sounds (utterances of individual users or conversations between users). Note that the user's emotion may be estimated by pattern matching, or may be estimated using a machine learning model.

In addition, although the reaction of the user is estimated from the in-vehicle image and/or voice in the above description, the present embodiment is not limited to this. For example, the in-vehicle device 2 may include a biosensor that measures the user's biometric information (body temperature, respiration, pulse, electrocardiogram, etc.), and estimate the user's reaction from the biometric information.

The in-vehicle device 2 refers to the display table 311 and controls the display of the object according to the estimated user's emotion. Specifically, the in-vehicle device 2 controls the display operation of the object according to the user's emotion, and controls the display time of the object. In the example of FIG. 7, when the user's emotion is "excited", the vehicle-mounted device 2 continues the effect display of the animal roaring for "30 seconds". On the other hand, when the user's emotion is "frightened", the in-vehicle device 2 performs the effect display of the animal leaving in "10 seconds". This allows the user to experience the AR content favorably.

FIG. 8 is an explanatory diagram showing a display example of an object according to a peripheral moving body. FIG. 8 illustrates a case in which moving objects existing in the vicinity of the vehicle 4 are detected as peripheral objects and displayed as objects.

In the present embodiment, the in-vehicle device 2 displays an object specified by the user for a mobile body specified by the user (for example, another vehicle running parallel to the own vehicle). Although there is no particular limitation on the method of specifying the moving body and the object, for example, the in-vehicle device 2 receives a specification input of the moving body and the object based on the motion of the user's hand. The in-vehicle device 2 recognizes from the in-vehicle image the motion of the user pointing to a nearby moving object, and identifies the moving object pointed to by the pointing motion from the surrounding image. Then, the in-vehicle device 2 displays a plurality of objects prepared in advance as AR content on the car window, and recognizes a pointing action of selecting one of the plurality of objects.

The in-vehicle device 2 determines the arrangement position of the object in the real space based on the position of the moving object specified above. For example, as shown in FIG. 8, the in-vehicle device 2 determines the upper portion of the moving body (for example, the roof or trunk of the vehicle) as the placement position of the object. The in-vehicle device 2 displays the object on the car window so that the object can be visually recognized at the determined arrangement position.

After that, the in-vehicle device 2 tracks the movement of the moving object from the surrounding images by a method such as optical flow. Then, the in-vehicle device 2 sequentially changes the display position of the object according to the movement of the moving body so that the relative position of the object with respect to the moving body does not change. As a result, the in-vehicle device 2 displays the object so as to follow the moving object. Note that, for example, the in-vehicle device 2 controls the display of the object according to the emotion of the user viewing the object, as in FIG.

Note that the display method described above is an example, and any method may be used as long as the object can be placed at a position corresponding to the position of the moving body. For example, the in-vehicle device 2 may display the object (animal) so that it moves (runs) together with the moving body by arranging the object at a predetermined distance from the moving body. Further, for example, the in-vehicle device 2 may display an object superimposed on the entire moving object so that the moving object itself can be visually recognized as an object (animal).

Further, in the present embodiment, an object corresponding to a moving object is displayed when specified by the user. An object may be displayed at a position corresponding to the moving object.

In the above description, a vehicle running side by side is used as a moving object, but other moving objects such as a bicycle traveling on a sidewalk or an aircraft appearing in a peripheral image may be used. That is, the in-vehicle device 2 only needs to be able to display an object on a moving object that exists in a real space that is visible from the car window, and the moving object is not limited to a vehicle.

FIG. 9 is an explanatory diagram showing a display example of objects according to vehicle speed. FIG. 9 illustrates how an object appears in the car window when the car stops at a pedestrian crossing.

As shown in the row of the object ID "Otter 101" in FIG. 4, the display table 311 defines the vehicle speed as a display condition for each object. In addition to the vehicle speed, an area, a time zone, etc., in which an object appears may be defined as display conditions. The in-vehicle device 2 sequentially detects the vehicle speed using the vehicle speed detection unit 30 and determines whether or not the vehicle speed has become equal to or less than a specific speed defined in the display table 311 . Specifically, the in-vehicle device 2 determines whether or not the vehicle 4 has stopped or has slowed down sufficiently to approximate a stopped state.

When it is determined that the speed has decreased below a specific speed, the in-vehicle device 2 determines the relative position (distance and direction from the vehicle 4) defined in the display table 311 as the object arrangement position, and displays the object on the car window. . In the example of FIG. 9 , an object is displayed obliquely forward left of the vehicle 4 .

In this case, the in-vehicle device 2 fixes the placement position of the object, and controls the display position in the car window after the vehicle 4 starts so that the object can be visually recognized fixed at that position. In the example of FIG. 9, the vehicle 4 stops at the pedestrian crossing, so the vehicle 4 moves forward. In this case, the in-vehicle device 2 moves the object from the left side of the windshield to the left front door glass and left rear door glass. In this manner, the in-vehicle device 2 causes an object to appear at a predetermined position in the real space when the speed is below a specific speed, and changes the display position of the object over a plurality of car windows according to the subsequent movement of the vehicle 4. do. In this way, the in-vehicle device 2 may display AR content according to the driving conditions (vehicle speed) of the vehicle 4 .

FIG. 10 is an explanatory diagram showing a display example of a message from another vehicle 4. As shown in FIG. The in-vehicle device 2 according to the present embodiment has a message sharing function with another vehicle 4 connected to the present system, in addition to the AR content display function described above. FIG. 10 illustrates how a message received from another vehicle 4 is displayed on the car window.

For example, the in-vehicle device 2 of another vehicle 4 accepts input of a message by voice input or the like, and transmits the message to the vehicle 4 to be transmitted via the server 1 . The in-vehicle device 2 of the transmission target vehicle 4 receives the message. Note that the message may include not only the body (text) but also an image or the like.

When receiving the message, the in-vehicle device 2 first notifies the user that the message has been received. For example, the in-vehicle device 2 may display a predetermined icon on the car window, or output a voice indicating that the message has been received.

The in-vehicle device 2 receives a selection input from the user as to whether or not to display the notified message. For example, the in-vehicle device 2 may recognize a specific action of the user (for example, pointing to an icon) from the in-vehicle image, or may receive an input of a specific utterance for displaying a message. .

When receiving a selection input to display a message, the in-vehicle device 2 displays the received message on the car window. Specifically, as shown in FIG. 10, the in-vehicle device 2 displays the avatar of the other user, who is the sender of the message, and also displays the message in the form of a balloon. This allows the message to be shared, for example, when there is another vehicle 4 accompanying the destination.

Although the message is displayed in the above description, the message (text) may be converted into voice and reproduced.

Further, for example, the in-vehicle device 2 may not only display the message, but also perform navigation according to the content of the message. In the example of FIG. 10, the in-vehicle device 2 recognizes a specific place name (“xx service area”) from the message, specifies a position corresponding to the place name from the map data M, and moves to an intermediate destination. It may be set so that the object of the destination is displayed on the train window in the same manner as in FIG. Thereby, a user can be suitably supported.

As described above, according to the first embodiment, the in-vehicle device 2 determines the placement position of the object according to the situation of the vehicle 4 (destination of the vehicle 4, surrounding objects, vehicle speed, etc.), and determines the placement position through the car window. To control the display of objects in a plurality of car windows so that the objects are visually recognized at their positions. As a result, the inside of the vehicle 4 can be used as an AR content experience space, and the user can preferably experience the AR content.

11 and 12 are flowcharts showing an example of the processing procedure executed by the in-vehicle device 2. FIG. Based on FIG.11 and FIG.12, the content of the process which the vehicle-mounted apparatus 2 performs is demonstrated.
The control unit 21 of the in-vehicle device 2 receives a destination setting input (step S11). The control unit 21 acquires the content corresponding to the destination from the server 1 (step S12). It should be noted that, as already mentioned, the content may be content unrelated to the destination.

The control unit 21 identifies the location of the set destination from the map data M (step S13). The control unit 21 determines the location of the destination as the placement position where the object corresponding to the destination is placed in the real space (step S14). The control unit 21 acquires the position information of the vehicle 4 by the positioning unit 25, and displays the object so that the object can be visually recognized at the arrangement position determined in step S14 based on the current position of the vehicle 4 indicated by the position information ( step S15). In subsequent processing, the control unit 21 sequentially acquires the position information of the vehicle 4 and sequentially changes the display position of the object on the car window.

The control unit 21 captures an image of the periphery of the vehicle 4 using the periphery imaging unit 26 (step S16). The control unit 21 detects surrounding objects from the image of the surroundings of the vehicle 4 (step S17). Specifically, the control unit 21 recognizes structures, moving objects, and the like existing around the vehicle 4 from a surrounding image (distance image) captured by a camera with a distance sensor or the like, and recognizes each object from the distance measured by the sensor. Detect the position and shape of surrounding objects.

The control unit 21 determines whether or not a predetermined display condition has been cleared (step S18). If it is determined that the display conditions have been cleared (S18: YES), the control unit 21 displays the position of the object (distance and direction from the vehicle 4) predefined in the display table 311 and the structure detected in step S17. , and the position of the object in the real space is determined (step S19). Specifically, the control unit 21 refers to the display table 311 that defines the relative position of the object from the vehicle 4, and determines whether or not there is a structure at that relative position. When determining that the structure does not exist, the control unit 21 determines the relative position defined in the display table 311 as the object placement position. When determining that a structure exists, the control unit 21 changes the arrangement position of the object from the default relative position. For example, the control unit 21 determines the top of the structure as the arrangement position according to the shape of the structure.

The control unit 21 displays the object on the car window so that the object can be seen at the placement position determined in step S19 (step S20). Specifically, when the control unit 21 determines that the arrangement position is on the structure, the control unit 21 displays the object on the structure. In addition, when the distance of the object from the vehicle 4 is longer than the distance of the structure from the vehicle 4, the control unit 21 controls the object so that part or all of the object is hidden behind the structure according to the shape of the structure. display.

The control unit 21 estimates the emotion of the user viewing the object (step S21). For example, the control unit 21 estimates the emotion of the user from the in-vehicle image captured by the in-vehicle imaging unit 27 and/or the in-vehicle voice acquired by the voice input unit 28 . Note that the control unit 21 may estimate the emotion from the biological information of the user in addition to or instead of the in-vehicle image and the in-vehicle voice. The control unit 21 refers to the display table 311 and controls the display of the object according to the estimated user's emotion (step S22). Specifically, the control unit 21 controls the action of the object according to the user's emotion, and changes the display time of the object.

After executing the process of step S22, or in the case of NO in step S18, the control unit 21 determines whether or not a designation input designating an object and a moving object to be displayed for the object has been received from the user ( step S23). If it is determined that the designation input of the object and moving object has been received (S23: YES), the control unit 21 determines the upper portion of the designated moving object, etc., as the layout position of the designated object (step S24). The control unit 21 displays the object so that the object can be visually recognized at the determined placement position (step S25). In subsequent processing, the control unit 21 tracks the movement of the moving object from the peripheral image, and sequentially changes the display position of the object according to the movement of the moving object so that the relative position of the object with respect to the moving object does not change.

The control unit 21 estimates the reaction of the user viewing the object (step S26). The control unit 21 controls the display of the object according to the estimated user reaction (step S27).

After executing the process of step S27, or in the case of NO in step S23, the control unit 21 detects the vehicle speed by the vehicle speed detection unit 30 (step S28). The control unit 21 determines whether or not the vehicle speed has become equal to or less than a specific speed (step S29).

If it is determined that the speed is below the specific speed (S29: YES), the control unit 21 refers to the display table 311 and determines the placement position of the object in the real space (step S30). The control unit 21 displays the object on the car window so that the object can be seen at the determined placement position (step S31). The control unit 21 fixes the placement position of the object to the position determined in step S30, and sequentially changes the display position of the object in the car window so that the object can be visually recognized at the corresponding position in the real space even after the vehicle 4 starts moving. .

After executing the process of step S31, or in the case of NO in step S29, the control unit 21 determines whether or not a message has been received from another vehicle 4 (step S32). If it is determined that the message has been received (S32: YES), the control section 21 notifies the user that the message has been received (step S33). For example, the control unit 21 may display a predetermined icon representing the message on the car window, or output a voice indicating that the message has been received.

The control unit 21 receives from the user a selection input as to whether or not to display the message (step S34). For example, the control unit 21 may recognize a specific action of the user from an in-vehicle image, or may receive an input of a specific uttered voice that causes a message to be displayed. If the selection input to display the message is received, the control unit 21 displays the message from the other vehicle (step S35). For example, the control unit 21 displays an avatar corresponding to the user of the other vehicle 4 and a message (text).

After executing the process of step S35, or in the case of NO in step S32, the control unit 21 determines whether or not the vehicle 4 has arrived at the destination based on the position information of the vehicle 4 (step S36). If it is determined that the vehicle has not arrived at the destination (S36: NO), the control unit 21 returns the process to step S16. If it is determined that the destination has been reached (S36: YES), the control unit 21 terminates the series of processes.

As described above, according to the first embodiment, the arrangement position of the object is determined according to the situation of the vehicle 4 such as the destination, the surrounding objects (structures, mobile bodies), and the vehicle speed, and the object is visually recognized at the arrangement position. By controlling the display of the object in the car window so that the AR content is preferably experienced by the user.

(Embodiment 2)
In the present embodiment, a mode will be described in which the display position of an object on the car window is controlled according to the position of the seat of the vehicle 4 on which the user sits. In addition, the same code|symbol is attached|subjected about the content which overlaps with Embodiment 1, and description is abbreviate|omitted.

FIG. 13 is an explanatory diagram showing an overview of the second embodiment. FIG. 13A shows a bird's-eye view of the vehicle 4 conceptually showing the positional relationship between the seats of the vehicle 4 and the arrangement positions of the objects. FIG. 13B conceptually illustrates how the display position of the object on the car window (windshield) is changed according to the seat position. Based on FIG. 13, the outline of this embodiment will be described.

In FIG. 13A , the thick line represents the car window, and the rectangular area surrounded by the thick line represents the interior of the vehicle 4 . White circles located within the rectangular area represent seats (driver's seat and front passenger's seat), and hatched circles located outside the rectangular area represent the positions of objects.

As shown in FIG. 13A, when the same position in real space is viewed from each seat, the position on the train window corresponding to the position is different. For example, as shown in FIG. 13A, the position in front of the driver's seat is located obliquely in front of the right when viewed from the passenger's seat.

Here, for example, when presenting AR content to a user sitting in the front passenger seat, if the display position of the object in the car window is determined with the driver's seat as the reference (viewpoint position), the object will be displayed from the position where it should actually be displayed. will deviate. For example, when presenting AR content for children, if the display position of the object is determined based on the driver's seat even though the child is sitting in the front passenger seat, the object will be displayed in a position that is not the position that is originally intended to be superimposed on the scenery. will be displayed.

Therefore, in the present embodiment, the in-vehicle device 2 controls the display position of the content in the car window according to the position of the seat of the user who is the display target of the content (object). For example, the in-vehicle device 2 receives a designation input that designates the user's seat to be displayed before starting to display the content. Note that, for example, the in-vehicle device 2 prescribes the attributes (age, gender, etc.) of the user whose content is to be displayed, recognizes the user corresponding to the user attribute to be displayed from the in-vehicle image, and selects the seat of the target user. may be automatically identified. Further, for example, the seats where each user (family member, etc.) usually sits may be registered in the user DB 142 in advance, and the user's seat corresponding to the user attribute to be displayed in the content may be automatically specified.

The in-vehicle device 2 determines the arrangement position of each object in the real space, as in the first embodiment. In the present embodiment, the in-vehicle device 2 determines the display position of the object in the car window based on the determined arrangement position and the position of the target user's seat specified (specified) above. Specifically, as shown in FIG. 13A, the in-vehicle device 2 determines the display position of the object as a position on the car window that intersects a straight line connecting the position of the object and the position of the target user's seat. The in-vehicle device 2 displays the object at the determined display position. Thereby, as shown in FIG. 13B, the display position of the object is preferably controlled according to the position of the seat on which the user sits.

FIG. 14 is a flow chart showing an example of a processing procedure executed by the in-vehicle device 2 according to the second embodiment. After acquiring the content from the server 1 (step S12), the in-vehicle device 2 executes the following processes.
The control unit 21 of the in-vehicle device 2 identifies the user's seat for which content (object) is to be displayed (step S201). For example, the control unit 21 receives a designation input that designates the user's seat to be displayed. When displaying an object (steps S15, S20, S25, and/or S31), the control unit 21 determines the display position of the object on the train window based on the position of the object and the position of the target user's seat.

As described above, according to the second embodiment, by controlling the display position of the object in the train window according to the seat where the user who views the content sits, the AR content can be presented more preferably.

(Embodiment 3)
In the present embodiment, an embodiment will be described in which the display of objects in the car window is restricted according to the driving conditions of the vehicle 4. FIG.

FIG. 15 is an explanatory diagram showing an overview of the third embodiment. FIG. 15 conceptually illustrates how a car window (windshield) is divided into a displayable area A1 in which an object can be displayed and a display prohibited area A2 in which display of an object is prohibited. Based on FIG. 15, the outline of this embodiment will be described.

In the present embodiment, the in-vehicle device 2 limits the display of objects in the car window according to the driving conditions of the vehicle 4 . Specifically, the in-vehicle device 2 sets a display prohibited area A2 in which display of the object is prohibited. For example, the in-vehicle device 2 determines whether the vehicle 4 is driving (moving) or stopped (not moving). When the vehicle 4 is stopped, the in-vehicle device 2 does not set the display prohibited area A2 and allows the object to be displayed in the entire area of the vehicle window. On the other hand, when it is determined that the vehicle 4 is driving, the in-vehicle device 2 sets a display prohibited area A2 and prohibits display of objects in this area in order to secure a certain field of view.

Note that "while driving" refers to both automatic driving and manual driving, but the display prohibited area A2 may be set only during manual driving so that the object can be displayed in the entire area of the car window during automatic driving. . Further, the criteria for judging the above-mentioned driving conditions are only an example. It may be determined whether or not to set the display prohibited area A2 according to other driving conditions such as whether the vehicle is Further, the in-vehicle device 2 may change the area, position, etc. of the display prohibited area A2 in the car window according to the vehicle speed, driving direction, etc., for example, increasing the display prohibited area A2 as the vehicle speed increases.

For example, as shown in FIG. 15, the in-vehicle device 2 sets a partial area along the periphery of the car window as the displayable area A1. Then, the in-vehicle device 2 sets the area in the center of the car window other than the displayable area A1 as the display prohibited area A2. Note that the display prohibited area A2 shown in FIG. 15 is an example, and the position and shape of the display prohibited area A2 are not particularly limited. For example, the in-vehicle device 2 may set a certain range upward from the lower edge of the car window as the display prohibited area A2.

Also, although FIG. 15 shows an example of the windshield, the display prohibited area A2 may be similarly set for the front door glass, the rear glass, and the like. In this case, the in-vehicle device 2 may select the car window for setting the display prohibited area A2 according to the type of the car window. For example, even if the display prohibited area A2 is set for the windshield, front door glass, etc. in order to secure the field of view, the display prohibited area A2 is not set for the rear door glass, roof glass, etc., for which there is little need to secure the field of view. good. In addition, the in-vehicle device 2 may change the area, position, etc. of the display prohibited area A2 occupied by each car window according to the type of car window.

Although the display of the object is restricted by setting the display prohibited area A2 in the above description, the present embodiment is not limited to this. For example, in addition to setting the display prohibited area A2, or instead of setting the display prohibited area A2, the in-vehicle device 2 may change the object displayed on the car window to a translucent object. For example, the in-vehicle device 2 increases the transmittance of the object when the vehicle 4 is driving. Thereby, a user's visual field can be secured. As described above, the in-vehicle device 2 only needs to be able to restrict the display of objects according to the driving conditions of the vehicle 4, and the means for restricting the display of objects is not limited to setting the display prohibited area A2.

Since the present embodiment is the same as the first embodiment except that the display of objects is restricted according to the driving conditions of the vehicle 4, detailed description of the flowchart and the like will be omitted in the present embodiment.

As described above, according to the third embodiment, it is possible to secure the user's field of view while displaying the AR content.

The embodiments disclosed this time are illustrative in all respects and should not be considered restrictive. The scope of the present invention is indicated by the scope of the claims rather than the above-described meaning, and is intended to include all modifications within the scope and meaning equivalent to the scope of the claims.

1 Server 11 Control Unit 12 Main Storage Unit 13 Communication Unit 14 Auxiliary Storage Unit P1 Program 2 In-Vehicle Device (Information Processing Device)
21 control unit 22 main storage unit 23 communication unit 24 display unit 25 positioning unit 26 peripheral imaging unit 27 in-vehicle imaging unit 28 voice input unit 29 voice output unit 30 vehicle speed detection unit 31 auxiliary storage unit P2 program M map data 311 display table

Claims (12)

a display unit that displays an image using a vehicle window as a display area;
a first acquisition unit that acquires data indicating the status of the vehicle;
a second acquisition unit that acquires an object to be displayed in the window;
a determination unit that determines the placement position of the object in real space according to the situation of the vehicle;
a display control unit that controls the display position of the object in the window so that the object can be seen at the arrangement position through the window ,
The first acquisition unit acquires a destination of the vehicle,
The second acquisition unit acquires an object corresponding to the destination,
The determination unit refers to a table that defines a relative position of each object from the vehicle to place the object, and determines the relative position from the vehicle to place the object;
The first acquisition unit acquires a detection result of detecting the position and shape of a peripheral object existing in the real space,
The display control unit controls display of the object based on the position and shape of the peripheral object existing at the relative position in the real space.
Information processing equipment. The information processing apparatus according to claim 1, wherein the determination unit determines the placement position of the object according to the position of the peripheral object. The surrounding object includes an object existing at a fixed position,
referring to the table to determine whether the object exists at the relative position;
When determining that the object does not exist, the determination unit determines the relative position to be the placement position of the object;
The information processing apparatus according to claim 2, wherein when determining that the object exists, the determination unit changes the arrangement position of the object from the relative position. The display control unit hides part or all of the object according to the shape of the object when the distance of the arrangement position from the vehicle is longer than the distance of the object from the vehicle. The information processing apparatus according to claim 3. The peripheral objects include moving bodies existing around the vehicle,
A tracking unit that tracks the detected movement of the moving body,
The information processing apparatus according to any one of claims 2 to 4, wherein the display control section sequentially changes the display position of the object according to the movement of the moving body. A position acquisition unit that sequentially acquires position information of the vehicle,
The determination unit determines the location of the destination as the location of the object,
The display control unit displays the object in the window so that the object can be viewed in a direction in which the destination is located, based on the arrangement position and the current position of the vehicle indicated by the position information that is sequentially acquired. The information processing apparatus according to any one of claims 1 to 5, wherein the display position of the is sequentially changed. The first acquisition unit acquires the traveling speed of the vehicle,
A speed determination unit that determines whether the running speed has become a specific speed or less,
If it is determined that the speed is equal to or less than the specific speed, the determination unit determines the placement position of the object,
The information processing according to any one of claims 1 to 6, wherein the display control unit controls the display position of the object in the window so that the object can be viewed fixedly at the determined placement position. Device. An estimating unit that estimates the emotion of a person riding in the vehicle,
The information processing apparatus according to any one of claims 1 to 7, wherein the display control section controls display of the object according to the emotion of the person after the object is displayed. The first acquisition unit acquires data indicating a driving situation of the vehicle,
9. The display controller according to any one of claims 1 to 8, wherein the display control unit sets a display prohibited area for prohibiting display of the object in the window, or changes the object to a translucent object, according to the driving situation. The information processing device described. a specifying unit that specifies a seat in the vehicle on which the person who is the display target of the object is seated;
The information processing apparatus according to any one of claims 1 to 9, wherein the display control unit controls the display position of the object in the window based on the specified position of the seat and the arrangement position of the object. . The display unit displays images on the plurality of windows of the vehicle,
The information processing apparatus according to any one of claims 1 to 10, wherein the display control unit displays the object in one of the windows according to the position of arrangement of the object. Acquiring data indicating the vehicle status,
Acquiring an object to be displayed on a display unit having a window of the vehicle as a display area,
determining the placement position of the object in real space according to the situation of the vehicle;
A program for causing a computer to execute a process of controlling the display position of the object in the window so that the object can be seen at the arrangement position through the window,
obtaining a destination of the vehicle;
obtaining an object corresponding to the destination;
determining the relative position from the vehicle where the object is placed by referring to a table that defines the relative position from the vehicle where the object is placed for each object;
Acquiring a detection result of detecting the position and shape of a peripheral object existing in the real space;
controlling the display of the object based on the position and shape of the peripheral object existing at the relative position in the real space;
A program that causes a computer to carry out a process .

Images