JP2021176033A - Information processing device, information processing method, and program - Google Patents

Abstract

To present information by which an overview of an object space is easily and intuitively grasped to a user.SOLUTION: An information processing device according to one aspect of the present disclosure comprises means for: specifying an entity to be associated with a reference point of time, which is at least one of a sensor arranged in an object space at at least one reference point, and a detection object of the sensor at the reference point of time; generating an entity image corresponding to the entity; generating a space image corresponding to the object space; and generating a synthetic image by synthesizing the entity image with the space image so that the entity image is arranged at a position corresponding to a position of the entity at the reference point of time in the space image.SELECTED DRAWING: Figure 3

Description

The present disclosure relates to information processing devices, information processing methods, and programs.

For example, a project manager needs to manage the progress of work.

In Patent Document 1, a display member on which an identification code including location information of a construction site and information related to construction is displayed is arranged at the construction site, and the construction site is photographed after reading the identification code with a mobile terminal. Disclosed is a technique for displaying images showing the progress of construction by arranging them for each construction site and in order of process.

Japanese Unexamined Patent Publication No. 2020-030519

The work is done in a three-dimensional space and changes with time. Therefore, it is necessary to consider time in the three-dimensional space for progress management.

According to the technique described in Patent Document 1, images showing the progress of construction can be displayed by arranging them for each construction site and in order of process. However, while the object of construction is a three-dimensional space (for example, a building), the captured image is two-dimensional information, so the manager can see the progress of the work just by looking at the image showing the progress of the construction. It is not easy to grasp spatially. In addition, since images showing the progress of construction are arranged for each construction site, it is not easy for the manager to grasp an overview of the progress of work at a plurality of construction sites at a certain point in time.

That is, in Patent Document 1, since time is not considered in the three-dimensional space, it is difficult to intuitively grasp the overview of the target space.

An object of the present disclosure is to present to the user information that makes it easy to intuitively grasp the appearance of the target space.

The information processing apparatus according to one aspect of the present disclosure includes a sensor arranged in the target space at at least one reference time point, and an entity associated with a reference time point which is at least one of the detection targets of the sensor at the reference time point. A means for specifying, a means for generating an entity image corresponding to an entity, a means for generating a spatial image corresponding to a target space, and a position in the spatial image in which the entity image corresponds to the position of the entity at a reference time. It is provided with a means for generating a composite image by synthesizing the entity image and the spatial image so as to be arranged.

According to the present disclosure, it is possible to present to the user information that makes it easy to intuitively grasp the appearance of the target space.

It is a block diagram which shows the structure of the information processing system of this embodiment. It is a block diagram which shows the structural example of the information of this embodiment. It is explanatory drawing of the outline of this embodiment. It is a figure which shows the data structure of the person database of this embodiment. It is a figure which shows the data structure of the object database of this embodiment. It is a figure which shows the data structure of the sensor database of this embodiment. It is a figure which shows the data structure of the sensing result database of this embodiment. It is a figure which shows the data structure of the 3D model database of this embodiment. It is a flowchart of the database update process of this embodiment. It is a flowchart of the display process of this embodiment. It is a figure which shows the area structure of the screen displayed in the process of FIG. It is a figure which shows the example of the time slider object displayed in the timeline viewer area among the screens displayed in the process of FIG. It is a figure which shows the example of the parameter image displayed in the parameter viewer area of the screen displayed in the process of FIG. It is a figure which shows the example of the composite image displayed in the space viewer area in the screen displayed in the process of FIG. It is a flowchart of the display processing of a modification. It is a figure which shows the example of the time slider object displayed in the timeline viewer area in the screen displayed in the process of FIG. It is a figure which shows the example of the composite image displayed in the space viewer area in the screen displayed in the process of FIG. It is a figure which shows the example of the composite image displayed in the space viewer area in the user interface screen of another modification. It is a figure which shows the example of the composite image displayed in the space viewer area in the user interface screen of another modification.

Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. In addition, in the drawing for demonstrating the embodiment, the same components are in principle the same reference numerals, and the repeated description thereof will be omitted.

(1) Configuration of information processing system The configuration of the information processing system will be described. FIG. 1 is a block diagram showing a configuration of an information processing system of the present embodiment.

As shown in FIG. 1, the information processing system 1 includes an information processing device 10 and a sensor 30. The number of sensors 30 is arbitrary.
The information processing device 10 and the sensor 30 are connected via a network (for example, the Internet or an intranet) NW.

The information processing device 10 is, for example, a smartphone, a tablet terminal, a personal computer, or a server (for example, a Web server, an application server, a database server, or a combination thereof).
The information processing device 10 manages a sensing result database based on the sensing data transmitted from the sensor 30.
The information processing device 10 refers to the sensing result database, generates a composite image described later, and presents the composite image to the user.

The sensor 30 is installed in the target space. The sensor 30 performs sensing and transmits the sensing data to the information processing device 10. The sensor 30 is an arbitrary type of sensor device. The sensor 30 may be any of the following.
・ Camera ・ Depth camera ・ LIDAR
・ ToF camera ・ Barometric pressure sensor ・ Temperature sensor ・ Seismic intensity sensor

(1-1) Configuration of Information Processing Device The configuration of the information processing device will be described. FIG. 2 is a block diagram showing a configuration example of information of the present embodiment.

As shown in FIG. 2, the information processing device 10 includes a storage device 11, a processor 12, an input / output interface 13, and a communication interface 14.

The storage device 11 is configured to store programs and data. The storage device 11 is, for example, a combination of a ROM (Read Only Memory), a RAM (Random Access Memory), and a storage (for example, a flash memory or a hard disk).

The program includes, for example, the following program.
・ OS (Operating System) program ・ Application (for example, web browser) program that executes information processing

The data includes, for example, the following data.
-Database referenced in information processing-Data obtained by executing information processing (that is, the execution result of information processing)

The processor 12 is configured to realize the function of the information processing device 10 by activating the program stored in the storage device 11. The processor 12 is an example of a computer.

The input / output interface 13 is configured to acquire a user's instruction from an input device connected to the information processing device 10 and output information to an output device connected to the information processing device 10.
The input device is, for example, a keyboard, a pointing device, a touch panel, a sensor (eg, a camera, a microphone, an acceleration sensor), or a combination thereof.
The output device is, for example, a display, a speaker, or a combination thereof.

The communication interface 14 is configured to control communication between the information processing device 10 and an external device (for example, a sensor 30).

(2) Outline of the Embodiment The outline of the present embodiment will be described. FIG. 3 is an explanatory diagram of an outline of the present embodiment.

As shown in FIG. 3, at a certain reference time point, the target space SP (for example, a building that has already been constructed or a building that is under construction, repair, or demolition) has an element associated with time (referred to as “entity”). ) Exists. The entity includes a sensor SN and a DT to be detected by the sensor SN (for example, at least one of a person, an object, and an organism).
The information processing device 10 refers to, for example, the sensing result of the sensor SN, and identifies the sensor SN and the detection target DT as an entity associated with the reference time point.

The information processing device 10 generates, for example, the entity image EI1 corresponding to the sensor SN and the entity image EI2 corresponding to the detection target DT by referring to the sensing result of the sensor SN.
The information processing device 10 refers to, for example, the three-dimensional model information MD of the target space SP, and generates a spatial image SI corresponding to the target space SP.
In the spatial image SI, the information processing device 10 is arranged so that the entity image EI1 is arranged at a position corresponding to the sensor SN at the reference time point and the entity image EI2 is arranged at a position corresponding to the detection target DT at the reference time point. , The composite image CI is generated by synthesizing the entity image EI1, the entity image EI2, and the spatial image SI.

According to the composite image CI, the user confirms the positions of the entity images EI1 and EI2 corresponding to the entities (that is, the sensor SN and the detection target DT) at the reference time in the target space SP of each entity at the reference time. can do. That is, the information processing device 10 can present the user with a composite image CI that makes it easy to intuitively grasp the positional relationship between the target space and the entity at the reference time.

(3) Database The database of the present embodiment will be described. The following database is stored in the storage device 11.

(3-1) Person database The person database of the present embodiment will be described. FIG. 4 is a diagram showing a data structure of the person database of the present embodiment.

As shown in FIG. 4, the person database includes an "ID" field, a "name" field, and a "person attribute" field. Each field is associated with each other.
Person information is stored in the person database. Person information is information about a person.

Person identification information is stored in the "ID" field. The person identification information is information for identifying a person.

Person name information is stored in the "name" field. The person name information is information related to the name of the person.

Person attribute information is stored in the "person attribute" field. The person attribute information is information about the attribute of the person. The "Personal Attributes" field includes a "Gender" field and an "Age" field.

Gender information is stored in the "Gender" field. Gender information is information about the gender of a person.

Age information is stored in the "Age" field. Age information is information about a person's age or age group.

(3-2) Object database The object database of the present embodiment will be described. FIG. 5 is a diagram showing a data structure of the object database of the present embodiment.

As shown in FIG. 5, the object database includes an "ID" field, a "name" field, and a "type" field. Each field is associated with each other.
Object information is stored in the object database. Object information is information about an object.

Object identification information is stored in the "ID" field. The object identification information is information for identifying an object.

Object name information is stored in the "Name" field. The object name information is information related to the name of the object.

Object type information is stored in the "Type" field. The object type information is information related to the type of the object.

(3-3) Sensor database The sensor database of the present embodiment will be described. FIG. 6 is a diagram showing a data structure of the sensor database of the present embodiment.

As shown in FIG. 6, the sensor database includes an "ID" field, a "name" field, a "sensor type" field, an "output format" field, an "installation position" field, and an "installation direction" field. , Includes "fixed" fields. Each field is associated with each other.
Sensor information is stored in the sensor database. The sensor information is information about the sensor 30.

Sensor identification information is stored in the "ID" field. The sensor identification information is information that identifies the sensor 30.

Sensor name information is stored in the "Name" field. The sensor name information is information about the sensor 30.

Sensor type information is stored in the "sensor type" field. The sensor type information is information regarding the type of the sensor 30.

Output format information is stored in the Output Format field. The output format information is information regarding the output format (for example, the type of sensing data) of the sensor 30.

The installation position information is stored in the "installation position" field. The installation position information is information regarding the installation position of the sensor 30 (the position of the spatial coordinate system having a predetermined position in the target space SP as the origin).

Installation direction information is stored in the installation direction field. The installation direction information is information regarding the installation direction of the sensor 30.

Fixed information is stored in the "fixed" field. The fixed information is information indicating whether or not the sensor 30 is fixed (that is, mobile). The fixed sensor 30 performs sensing in a state of being fixed at the installation position. The mobile sensor 30 senses at a plurality of different positions.

(3-4) Sensing result database The sensing result database of the present embodiment will be described. FIG. 7 is a diagram showing a data structure of the sensing result database of the present embodiment.

As shown in FIG. 7, the sensing result database includes a table (“sensor table”) for each sensor 30. The sensor table includes a "time" field, a "sensing location" field, a "sensing direction" field, a "raw data" field, and an "analysis result" field. Each field is associated with each other.
The sensing result database is associated with the sensor identification information of the sensor 30 that executed the sensing.
Sensing result information is stored in the sensing result database. The sensing result information is information related to the sensing result of the sensor 30.

Sensing time information is stored in the "Time" field. The sensing time information is information related to the time when the sensor 30 senses.

Sensing execution position information is stored in the "sensing execution position" field. The sensing execution position information is information regarding the position of the sensor 30 (hereinafter referred to as “sensing execution position”) when the sensor 30 performs sensing.
For example, the sensing execution position information includes at least one of the following.
-Information on the relative position of the sensing execution position with respect to the installation position of the sensor 30 (for example, the amount of movement from the installation position to the sensing execution position) -Coordinates of the sensing execution position in the spatial coordinate system

Sensing direction information is stored in the "sensing direction" field. The sensing direction information is information regarding the direction in which the sensor 30 senses.

Raw data information is stored in the "raw data" field. The raw data information is information about the sensing data (raw data) transmitted by the sensor 30.
The raw data information may include information about the position (eg, information about the position of the point to be sensed). The raw data information may include information regarding the position of the sensor coordinate system with the position of the sensor 30 (sensing execution position) as the origin. The position of the sensor coordinate system can be converted to the position of the spatial coordinate system by referring to at least one of the installation position information and the sensing execution position information.

The analysis result information is stored in the "Analysis result" field. The analysis result information is information regarding the analysis result of the sensing data transmitted by the sensor 30. The "Analysis Result" field includes an "Analysis Position" field and an "Entity" field.

The analysis position information is stored in the "analysis position" field. The analysis position information is information regarding the position of the spatial coordinate system of the analysis result (for example, the position of the spatial coordinate system of the structure analyzed from the image data).

Entity information is stored in the "Entity" field. Entity information is information about the entity associated with the analysis result. As an example, the entity data is the person identification information, the person name information, or the person attribute information registered in the person database (FIG. 4), or the object identification information, the object name information, or the object name information registered in the object database (FIG. 5). It may be object type information.

(3-5) Three-dimensional model database The three-dimensional model database of the present embodiment will be described. FIG. 8 is a diagram showing a data structure of the three-dimensional model database of the present embodiment.

As shown in FIG. 8, the 3D model database includes a "time" field and a "model file" field. Each field is associated with each other.
The 3D model database stores 3D model information. The three-dimensional model information is information about the three-dimensional model of the target space SP.

Model time information is stored in the Time field. The model time information is time information related to the three-dimensional model information.

Model file information is stored in the "Model File" field. The model file information is the information of the model file related to the three-dimensional model information. The model file is, for example, BIM (Building Information Modeling) data or three-dimensional CAD (Computer-Aided Design) data.

(4) Information processing The information processing of the present embodiment will be described.

(4-1) Database update process The database update process of this embodiment will be described. FIG. 9 is a flowchart of the database update process of the present embodiment.

As shown in FIG. 9, the information processing apparatus 10 executes the acquisition of sensing data (S110).
Specifically, the processor 12 acquires the sensing data transmitted from the sensor 30.
The first example of sensing data is transmitted from the fixed sensor 30. The first example of sensing data includes sensor identification information and raw data.
A second example of sensing data is transmitted from the mobile sensor 30. The second example of the sensing data includes sensor identification information, raw data, and sensing execution position information.

After step S110, the information processing apparatus 10 executes the analysis of sensing data (S111).
Specifically, the processor 12 analyzes the sensing data acquired in step S110. As an example, the processor 12 realizes at least one of the following by analyzing the sensing data.
-Determining the existence or nonexistence of an entity in the target space SP-Estimating the position of the entity in the target space SP-The state of the entity in the target space SP (for example, physical condition, motion state, part (for example, head, hand, or foot)) Estimating position, appearance characteristics, or a combination thereof) ・ Estimating the attributes of an entity in the target space SP (for example, age, gender, type, or a combination thereof) ・ Identifying an entity in the target space SP ・ Identifying the target space SP Calculation of the finished shape associated with at least one of the components of the 3D model (The finished shape can be calculated based on, for example, 3D model information and point group data).

After step S111, the information processing apparatus 10 executes the update of the sensing result database (S112).
Specifically, the processor 12 updates the sensing result database (FIG. 7) with reference to the sensing data acquired in step S110 and the analysis result obtained in step S111. As an example, the processor 12 may store data indicating the acquisition time of the sensing data in step S110 in the “time” field, or if the sensing data includes time data, the time data is stored in the “time” field. It may be stored in. When the sensing data in step S110 includes sensing execution position information, the processor 12 stores the sensing execution position information in the “sensing execution position” field. The processor 12 stores the raw data included in the sensing data in step S110 in the “raw data” field, and stores the analysis result in step S111 in the “analysis result” field.

(4-2) Display processing The display processing of the present embodiment will be described. FIG. 10 is a flowchart of the display process of the present embodiment. FIG. 11 is a diagram showing an area configuration of a screen displayed in the process of FIG. FIG. 12 is a diagram showing an example of a time slider object displayed in the timeline viewer area among the screens displayed in the process of FIG. FIG. 13 is a diagram showing an example of a parameter image displayed in the parameter viewer area of the screen displayed in the process of FIG. FIG. 14 is a diagram showing an example of a composite image displayed in the spatial viewer area of the screen displayed in the process of FIG.

As shown in FIG. 10, the information processing apparatus 10 executes the determination at the reference time point (S120).
Specifically, the processor 12 displays a screen (FIG. 11) on the display.

The screen of FIG. 11 includes a spatial viewer area SVR, a timeline viewer area TVR, and a parameter viewer area PVR.

As shown in FIG. 12, a time slider object TS is arranged in the timeline viewer area TVR. The user operates the input device to select any time TP1 on the time slider object TS. The processor 12 determines the time TP1 selected by the user as a reference time point.

The processor 12 determines at least one reference time point in response to a user instruction received by the input device. The user's instructions are, for example, at least one of tap, click, key input, voice input, and gesture input.

After step S120, the information processing apparatus 10 executes the identification of the entity (S121).
Specifically, the processor 12 identifies an entity associated with the reference time determined in step S120. The entity associated with the reference time is at least one of the sensors arranged in the target space SP at the reference time and the detection target of the sensor at the reference time. As an example, the processor 12 refers to the sensing result database (FIG. 7) and identifies a sensor corresponding to the sensing result associated with the reference time point and a detection target included in the analysis result included in the sensing result.

After step S121, the information processing apparatus 10 executes the identification of the display target (S122).
Specifically, the processor 12 selects a display target from the entities specified in step S121.

In the first example of step S122, the processor 12 selects all the entities specified in step S121 as display targets (hereinafter referred to as "reference entities").

In the second example of step S122, the processor 12 selects the entity specified by the user instruction from the entities specified in step S121 as the display target. As an example, the processor 12 refers to the sensing result database (FIG. 7), generates a parameter image showing information about the reference entity specified in step S121 (hereinafter referred to as “reference entity information”), and generates the parameter image. Is displayed on the output device (display) of the user interface screen arranged in the parameter viewer area PVR.
Information about the entity (hereinafter referred to as "entity information") includes the type of sensor 30, the type of sensing data, the content of sensing data (for example, raw data), the identification information of the detection target (for example, person identification information, object identification information), and the like. The attribute of the detection target, the state of the detection target, or a combination thereof.

As shown in FIG. 13, in the parameter image PI, the reference entity information EI is arranged according to the category to which the reference entity belongs. The categories are "sensor", "person", and "object".
The user can specify whether or not to display the entity corresponding to the entity information EI by performing the operation of selecting the check box object ECB of each entity information EI.
The user can specify whether or not to display all the entities belonging to the category by performing the operation of selecting the check box object CCB of each category.

After step S122, the information processing apparatus 10 executes the generation of the entity image (S123).
Specifically, the processor 12 generates the entity image to be displayed selected in step S122.

In the first example of step S123, the processor 12 generates an entity image by referring to raw data (for example, image data) included in the sensing data transmitted from the sensor 30. For example, the processor 12 may select raw data associated with a reference time among the raw data, refer to the selected raw data, and generate an entity image according to the reference time.

In the second example of step S123, the processor 12 refers to a predetermined image associated with the entity (eg, an icon, an avatar, or a pre-prepared photo) to generate the entity image.
A given image associated with an entity may be further associated with time. In this case, the processor 12 may select an image associated with the reference time among predetermined images and refer to the selected image to generate an entity image according to the reference time.

After step S120, the information processing apparatus 10 executes the generation of the spatial image (S124).
Specifically, the processor 12 generates a spatial image corresponding to the target space SP. As an example, the processor 12 refers to a three-dimensional model database (FIG. 8) and identifies a model file associated with the reference time point determined in step S120. The processor 12 refers to the specified model file and generates a spatial image (that is, a spatial image according to a reference time point) in which the three-dimensional model of the target space SP is viewed from a predetermined viewpoint.

After step S123 and step S124, the information processing apparatus 10 executes the generation of the composite image (S125).
Specifically, the processor 12 synthesizes the entity image generated in step S123 and the spatial image generated in step S124. As an example, the processor 12 synthesizes the entity image and the spatial image so that the entity image of the entity is arranged at a position corresponding to the position of the entity at the reference time determined in step S120 in the spatial image.
As shown in FIG. 14, in the composite image CI, the entity image of the person HM1, the entity image of the object OB1, the entity image of the sensor SN1, and the entity image of the sensor SN2 are at appropriate positions in the spatial image SI1. That is, it is arranged at the position of each entity at the reference time point).

After step S125, the information processing apparatus 10 executes the display of the composite image (S126).
Specifically, the processor 12 causes the output device (display) to display the user interface screen in which the composite image generated in step S125 is arranged in the spatial viewer area SVR.

As described above, the information processing apparatus 10 identifies an entity associated with a reference time point and generates an entity image. The information processing device 10 synthesizes the entity image and the spatial image so that the entity image of the entity is arranged at a position corresponding to the position of the entity at the reference time in the spatial image. Therefore, according to the information processing apparatus 10, it is possible to present to the user a composite image CI that makes it easy to intuitively grasp the positional relationship between the target space SP and each entity (display target) and the positional relationship between the entities.

(5) Modification Example A modification of the present embodiment will be described. The modification is an example in which a plurality of reference time points can be selected.

The display processing of the modified example will be described. FIG. 15 is a flowchart of a modification display process. FIG. 16 is a diagram showing an example of a time slider object displayed in the timeline viewer area among the screens displayed in the process of FIG. FIG. 17 is a diagram showing an example of a composite image displayed in the spatial viewer area of the screen displayed in the process of FIG.

As shown in FIG. 15, the information processing apparatus 10 executes the determination at the reference time point (S220).
Specifically, the processor 12 displays a screen (FIG. 11) on the display.
As shown in FIG. 16, a time slider object TS is arranged in the timeline viewer area TVR. The user operates the input device to select any two time TP2a and TP2b on the time slider object TS. The processor 12 determines the user-selected times TP2a and TP2b as the first reference time point and the second reference time point, respectively.
The first reference time point and the second reference time point are determined according to the instruction of the user who received the input device. User instructions are, for example, tap, click, keystroke, voice input, gesture input, swipe, pinch-in, pinch-out, drag, or a combination thereof.

After step S220, the information processing apparatus 10 executes the identification of the entity (S221).
Specifically, the processor 12 identifies an entity associated with at least one of the first reference time point and the second reference time point determined in step S220.

After step S221, the information processing apparatus 10 executes the identification of the display target (S222).
Specifically, the processor 12 selects a display target from the entities specified in step S221.

After step S222, the information processing apparatus 10 executes entity difference detection (S227).
Specifically, the processor 12 has information about the first reference entity at the first reference time point (hereinafter referred to as “first reference entity information”) and the second reference point time point for each display target selected in step S222. The difference (hereinafter referred to as "entity difference") from the information regarding the second reference entity (hereinafter referred to as "second reference entity information") is detected. For example, the processor 12 detects at least one of the following:
-Difference between the position of the first reference entity and the position of the second reference entity-Difference between the state of the first reference entity and the state of the second reference entity

After step S227, the information processing apparatus 10 executes the generation of the entity image (S223).
Specifically, the processor 12 determines the display form of the display target selected in step S222 according to the magnitude of the entity difference detected in step S227. The display form is a color, a texture, a size, a video effect, or a combination thereof. As an example, the processor 12 sets the entity image to the first color when the entity difference is greater than or equal to the threshold value, and sets the entity image to a second color different from the first color when the entity difference is less than the threshold value. Set to.

The processor 12 generates an entity image according to the determined display form.

For example, the processor 12 generates an entity image by referring to the raw data (for example, image data) included in the sensing data transmitted from the sensor 30 and the display form. For example, the processor 12 may select raw data associated with a reference time among the raw data, refer to the selected raw data and the display form, and generate an entity image according to the reference time.

For example, the processor 12 generates an entity image by referring to a predetermined image (for example, an icon, an avatar, or a photo prepared in advance) associated with the entity and a display form.
A given image associated with an entity may be further associated with time. In this case, the processor 12 may select an image associated with the reference time among predetermined images, refer to the selected image and the display form, and generate an entity image according to the reference time.

After step S220, the information processing apparatus 10 executes index difference detection (S228).
Specifically, the processor 12 has a value at the first reference time of the element index for each component (for example, wall, ceiling, floor, pillar, staircase, or a combination thereof) of the three-dimensional model of the target space SP. The difference between (“first index value”) and the value at the second reference time point (“second index value”) (hereinafter referred to as “index difference”) is detected. As an example, the processor 12 refers to a three-dimensional model database (FIG. 8) and identifies a model file associated with at least one of the first reference time point and the second reference time point determined in step S220. The processor 12 extracts information about the components of the three-dimensional model from the specified model file, and calculates the element index with reference to the extracted information. The element index is, for example, the shape of the component, the number of points associated with the component among a plurality of points related to the point cloud data, or a combination thereof.

After step S228, the information processing apparatus 10 executes spatial image generation (S224).
Specifically, the processor 12 determines the display form of the component according to the magnitude of the difference in the element index with respect to the component of the model detected in step S228. The display form is a color, a texture, a size, a video effect, or a combination thereof. As an example, the processor 12 sets the target component portion of the spatial image to the first color when the difference is equal to or greater than the threshold value, and becomes the target of the spatial image when the difference is less than the threshold value. The component part is set to a second color different from the first color.
The processor 12 generates a spatial image corresponding to the target space SP according to the determined display form. As an example, the processor 12 refers to a three-dimensional model database (FIG. 8) and identifies a model file associated with at least one of the first reference time point and the second reference time point determined in step S220. The processor 12 extracts information about each component of the three-dimensional model from the specified model file, refers to the extracted information and the display form of the component, and views the three-dimensional model of the target space SP from a predetermined viewpoint. Generates the viewed spatial image (that is, the spatial image according to the reference time).

After step S223 and step S224, the information processing apparatus 10 executes the generation of the composite image (S225).
Specifically, the processor 12 synthesizes the entity image generated in step S223 and the spatial image generated in step S224. As an example, the processor 12 synthesizes the entity image and the spatial image so that the entity image of the entity is arranged at a position corresponding to the position of the entity at the reference time determined in step S220 in the spatial image.
As shown in FIG. 17, in the composite image CIa, the entity image of the person HM1 in which the difference between the position at the first reference time and the position at the second reference time is equal to or greater than the threshold value is different from other entity images. Displayed in color. Of the spatial image SI1 of the composite image CIa, the component whose difference between the finished shape at the first reference time and the finished shape at the second reference time is equal to or greater than the threshold value is displayed in a different color from the other components. NS. In this way, the information processing apparatus 10 determines the positional relationship between the target space SP and each entity (display target), the positional relationship between the entities, the time change of the entity, and the local time change of the target space SP. A composite image CIa that is easy to grasp intuitively can be presented to the user.

After step S225, the information processing apparatus 10 executes the display of the composite image (S226).
Specifically, the processor 12 causes the output device (display) to display the user interface screen in which the composite image generated in step S225 is arranged in the spatial viewer area SVR.

As described above, the information processing apparatus 10 determines the display form of the entity image according to the difference between the first entity information and the second entity information. As a result, it is possible to present the user with a composite image CIa that makes it easy to intuitively grasp the time change of the entity.
Further, the information processing apparatus 10 is a portion of the spatial image corresponding to a component of the three-dimensional model of the target space SP according to the difference between the element index at the first reference time point and the element index at the second time point. Determine the display form. As a result, it is possible to present to the user a composite image CIa that makes it easy to intuitively grasp the local time change of the target space SP.

(6) Other Modification Examples Other modification examples will be described. FIG. 18 is a diagram showing an example of a composite image displayed in the spatial viewer area among the user interface screens of other modified examples. FIG. 19 is a diagram showing an example of a composite image displayed in the spatial viewer area among the user interface screens of other modified examples.

The storage device 11 may be connected to the information processing device 10 via the network NW.

The functions of the information processing device 10 can also be shared by a plurality of devices (for example, a client device and a server device).

The data associated with the reference time may include not only the data associated with the same time as the reference time but also the data associated with the closest time in the past than the reference time.

The user can perform an operation of designating a reference time point at an arbitrary time point (for example, an operation on the timeline viewer area TVR). The information processing apparatus 10 re-executes the display process of FIG. 10 when the reference time is changed by the user's operation during or after the execution of the display process of FIG. The information processing apparatus 10 re-executes the display process of FIG. 15 when the reference time is changed by the user's operation during or after the execution of the display process of FIG. As a result, the user interface screen including the composite image can be updated.
As an example, the processor 12 realizes the update of the entity to be displayed by reidentifying the entity associated with the reference time point in response to the operation of selecting the reference time point (step S121, step S221). .. The processor 12 realizes the update of the entity image by generating the entity image again in response to the update of the entity (step S123, step S223). The processor 12 updates the composite image by regenerating the composite image in response to the update of the entity image (steps S225 and S226).

The user can perform an operation of designating a display target at an arbitrary time point (for example, an operation on the parameter viewer area PVR). When the display target is changed by the user's operation during or after the execution of the display process of FIG. 10, the information processing apparatus 10 has at least steps S122 to S123 and step S125 of the display process of FIG. ~ Step S126 is re-executed. When the reference time is changed by the user's operation during or after the execution of the display process of FIG. 15, the information processing apparatus 10 has at least steps S222 to S223 and step S225 of the display process of FIG. ~ S226 is re-executed. As a result, the user interface screen including the composite image can be updated.

The user can perform an operation of designating a viewpoint at an arbitrary time point (for example, an operation on the spatial viewer area SVR). When the viewpoint is changed by the user's operation during or after the execution of the display process of FIG. 10, the information processing apparatus 10 re-executes at least steps S124 to S126 of the display process of FIG. When the reference time is changed by the user's operation during or after the execution of the display process of FIG. 15, the information processing apparatus 10 re-executes at least steps S224 to S226 of the display process of FIG. As a result, the user interface screen including the composite image can be updated.

According to FIGS. 10 and 15, the information processing apparatus 10 generates a spatial image (S124, S224) after generating an entity image (S123, S223). However, the information processing apparatus 10 may generate the spatial image (S124, S224) in parallel with the generation of the entity image (S123, S223), or before the generation of the entity image (S123, S223). Spatial image generation (S124, S224) may be performed.

In the embodiment, it has been described that the model file is associated with the time. However, the model file does not have to be associated with the time. That is, the information processing apparatus 10 may generate a spatial image by referring to the same model file regardless of the reference time point.

In the modified example, an example of determining the first reference time point and the second reference time point is shown.
In the information processing apparatus 10, the entity is associated with both the first reference time point and the second reference time point, and the entity is associated with only one of the first reference time point and the second reference point point time. The display form of the entity image of the entity may be different depending on the case.
The information processing apparatus 10 associates the model file with both the first reference time point and the second reference time point, and the model file with only one of the first reference time point and the second reference time point. The display form of the spatial image of the model file may be different depending on the case where the model file is displayed.
As an example, as shown in FIG. 18, the information processing apparatus 10 may generate a composite image CIb. In the composite image CIb, the object OB2 is associated with the first reference time point, but not with the second reference time point, so that the entity image of the object OB2 has a different color from the entity image of the object OB1. Is displayed. In the composite image CIb, the model file corresponding to the spatial image SI2 is associated with the second reference time point but not with the first reference time point, so that the spatial image SI2 has a different color from the spatial image SI1. Is displayed. In this way, the information processing device 10 can present to the user a composite image CIc that makes it easy to intuitively grasp the time changes of the entity and the target space SP.

In the modified example, an example of determining the first reference time point and the second reference time point is shown. When there is one or more intermediate points (time) between the first reference time point and the second reference time point, the information processing apparatus 10 refers to the entity information at each intermediate point and generates an entity image. You may.
As an example, as shown in FIG. 19, the information processing apparatus 10 may generate a composite image CIc. In the composite image CIc, the entity image of the person HM1 is a locus from the first reference time point to the second reference time point (that is, at each of the first reference time point, one or more intermediate points, and at the second reference time point. An image a (a polygonal line connecting the positions of the person HM1 in chronological order) is added. In this way, the information processing device 10 can present the user with a composite image CIc that makes it easy to intuitively grasp the details of the time change of the entity.

In the modified example, the first reference time point and the second reference time point have been described. The first reference time may be past the second reference time, or may be in the future than the second reference time.

In the embodiment, the processor 12 has been described as referring to a model file to generate a spatial image. However, the processor 12 may generate a spatial image by referring to sensing data (for example, point cloud data) instead of the model file, or may generate a spatial image by referring to the sensing data in addition to the model file. May be good.
Specifically, the processor 12 may generate a spatial image by referring to the finished form of each component of the three-dimensional model of the target space SP calculated based on the point cloud data. As an example, the processor 12 may generate a spatial image in which it is easy to grasp the progress of construction at a reference time point by selecting and drawing a component whose finished shape is equal to or larger than a threshold value among the components.

Although the embodiments of the present invention have been described in detail above, the scope of the present invention is not limited to the above embodiments. Further, the above-described embodiment can be improved or modified in various ways without departing from the spirit of the present invention. Moreover, the above-described embodiment and modification can be combined.

(7) Addendum The matters described in the embodiment are added below.

(Appendix 1)
A means for identifying a sensor (30) arranged in the target space (SP) at at least one reference time point, and an entity associated with a reference time point that is at least one of the detection targets of the sensor at the reference time point (S121). , S221) and
Means (S123, S223) for generating an entity image corresponding to an entity, and
Means (S124, S224) for generating a spatial image corresponding to the target space, and
In the spatial image, the means (S125, S225) for generating the composite image by synthesizing the entity image and the spatial image so that the entity image is arranged at the position corresponding to the position of the entity at the reference time. An information processing device (10) provided.

According to (Appendix 1), it is possible to generate a composite image in which it is easy to intuitively grasp the positional relationship between the target space and each entity and the positional relationship between the entities.

(Appendix 2)
A means for displaying a composite image (S126, S226) is further provided.
The information processing device according to Appendix 1.

According to (Appendix 2), it is possible to present to the user a composite image in which it is easy to intuitively grasp the positional relationship between the target space and each entity and the positional relationship between the entities.

(Appendix 3)
The means for displaying is the information processing device according to Appendix 2, which displays entity information about an entity.

According to (Appendix 3), the entity information can be further presented to the user.

(Appendix 4)
The information processing apparatus according to Appendix 3, wherein the entity information includes at least one of a sensor type, a sensor sensing data type, and a sensing data content.

According to (Appendix 4), at least one of the type of sensor, the type of sensor sensing data, and the content of the sensing data can be further presented to the user.

(Appendix 5)
The information processing device according to any one of Appendix 3 or Appendix 4, wherein the entity information displays at least one of the identification information, the attribute, and the state to be detected.

According to (Appendix 5), at least one of the identification information, the attribute, and the state of the detection target can be further presented to the user.

(Appendix 6)
The means for generating the spatial image refers to the three-dimensional model information of the target space to generate the spatial image.
The information processing device according to any one of Supplementary Note 1 to Supplementary Note 5.

According to (Appendix 6), it is possible to generate a composite image including a spatial image corresponding to the three-dimensional model information of the target space.

(Appendix 7)
The 3D model information of the target space is associated with the time and
At least one of the means for generating the spatial image and the means for generating the entity image generates at least one of the spatial image according to the reference time point and the entity image according to the reference time point, respectively.
The information processing device according to any one of Supplementary note 1 to Supplementary note 6.

According to (Appendix 7), it is possible to generate a composite image including at least one of a spatial image corresponding to a reference time point and an entity image corresponding to a reference time point.

(Appendix 8)
The means for generating the spatial image refers to the sensing result of the sensor and generates the spatial image.
The information processing device according to any one of Supplementary note 1 to Supplementary note 7.

According to (Appendix 8), it is possible to generate a composite image including a spatial image according to the sensing result of the sensor.

(Appendix 9)
Sensing results include point cloud data
The means for generating the spatial image is to generate the spatial image by referring to the finished shape of each component of the three-dimensional model of the target space calculated based on the point cloud data.
The information processing device according to Appendix 8.

According to (Appendix 9), it is possible to generate a composite image including a spatial image according to the finished shape.

(Appendix 10)
The information processing device according to any one of Supplementary note 1 to Supplementary note 9, wherein the entity includes at least one of a person, an object, and an organism.

According to (Appendix 10), it is possible to generate a composite image that makes it easy to intuitively grasp the positional relationship between the target space and each entity (at least one of a person, an object, and a living thing) and the positional relationship between the entities. can.

(Appendix 11)
A means (S122, S222) for acquiring a user instruction for designating a display target for each entity image is further provided.
The means for generating the entity image is to generate the entity image corresponding to the display target specified by the user instruction.
The information processing device according to any one of Supplementary Note 1 to Supplementary Note 10.

According to (Appendix 11), it is possible to generate a composite image in which it is easy to intuitively grasp the positional relationship between the target space and each entity (display target specified by the user instruction) and the positional relationship between the entities.

(Appendix 12)
The means for generating the entity image is the difference between the first entity information about the entity at the first reference time point and the second entity information about the entity at the second reference time point different from the first reference time point. Refer to and generate the entity image corresponding to the entity.
The information processing device according to any one of Supplementary Note 1 to Supplementary Note 11.

According to (Appendix 12), it is possible to generate a composite image in which it is easy to intuitively grasp the time change of an entity.

(Appendix 13)
The means for generating the entity image refers to the locus of the position of the entity from the first reference time point to the second reference time point different from the first reference time point, and generates the entity image corresponding to the entity. do,
The information processing device according to any one of Supplementary note 1 to Supplementary note 12.

According to (Appendix 13), it is possible to generate a composite image in which the details of the time change of the entity can be easily grasped intuitively.

(Appendix 14)
The means for generating the spatial image is a first index value at the first reference time of the index associated with the component of the three-dimensional model information of the target space and a second reference different from the first reference time of the index. With reference to the difference from the second index value at the time point, the part of the spatial image corresponding to the component is generated.
The information processing device according to any one of Supplementary Note 1 to Supplementary Note 13.

According to (Appendix 14), it is possible to generate a composite image in which it is easy to intuitively grasp the local time change of the target space SP.

(Appendix 15)
The index is at least one of the number of points associated with the components of the 3D model information and the finished shape associated with the components of the 3D model information among the plurality of points related to the point cloud data.
The information processing device according to Appendix 14.

According to (Appendix 15), it is possible to generate a composite image in which it is easy to intuitively grasp the local time change of the target space SP.

(Appendix 16)
The means of identifying an entity is to update the entity associated with the reference point in response to the user selecting a reference point in time.
The means of generating the entity image updates the entity image in response to the entity update,
The means for generating the composite image updates the composite image in response to the update of the entity image.
The information processing device according to any one of Supplementary note 1 to Supplementary note 15.

According to (Appendix 16), the composite image can be updated.

(Appendix 17)
The computer (10)
A step (S121) of identifying a sensor (30) arranged in the target space (SP) at at least one reference time point, and an entity associated with the reference time point which is at least one of the detection targets of the sensor at the reference time point. , S221) and
Steps (S123, S223) to generate an entity image corresponding to an entity, and
Steps (S124, S224) to generate a spatial image corresponding to the target space, and
In the spatial image, the steps (S125, S225) of generating the composite image by synthesizing the entity image and the spatial image so that the entity image is arranged at the position corresponding to the position of the entity at the reference time. Information processing method to be provided.

According to (Appendix 17), it is possible to generate a composite image in which it is easy to intuitively grasp the positional relationship between the target space and each entity and the positional relationship between the entities.

(Appendix 18)
A program for realizing each means according to any one of Supplementary note 1 to 16 on a computer.

According to (Appendix 18), it is possible to generate a composite image in which it is easy to intuitively grasp the positional relationship between the target space and each entity and the positional relationship between the entities.

1: Information processing system 10: Information processing device 11: Storage device 12: Processor 13: Input / output interface 14: Communication interface 30: Sensor

Claims (18)

A means for identifying a sensor that was placed in the target space at at least one reference time point, and an entity associated with the reference time point that is at least one of the detection targets of the sensor at the reference time point.
A means for generating an entity image corresponding to the entity,
A means for generating a spatial image corresponding to the target space, and
A means for generating a composite image by synthesizing the entity image and the spatial image so that the entity image is arranged at a position corresponding to the position of the entity at the reference time in the spatial image. An information processing device to be equipped. Further provided with means for displaying the composite image.
The information processing device according to claim 1. The information processing device according to claim 2, wherein the display means displays entity information relating to the entity. The information processing apparatus according to claim 3, wherein the entity information includes at least one of the sensor type, the sensing data type of the sensor, and the content of the sensing data. The information processing device according to claim 3 or 4, wherein the entity information displays at least one of the identification information, the attribute, and the state of the detection target. The means for generating the spatial image refers to the three-dimensional model information of the target space to generate the spatial image.
The information processing device according to any one of claims 1 to 5. The three-dimensional model information of the target space is associated with the time and is associated with the time.
At least one of the means for generating the spatial image and the means for generating the entity image generates at least one of the spatial image according to the reference time point and the entity image according to the reference time point, respectively.
The information processing device according to any one of claims 1 to 6. The means for generating the spatial image refers to the sensing result of the sensor and generates the spatial image.
The information processing device according to any one of claims 1 to 7. The sensing result includes point cloud data and includes point cloud data.
The means for generating the spatial image refers to the finished shape of each component of the three-dimensional model of the target space calculated based on the point cloud data, and generates the spatial image.
The information processing device according to claim 8. The information processing device according to any one of claims 1 to 9, wherein the entity includes at least one of a person, an object, and an organism. Further provided with means for obtaining a user instruction to specify a display target for each entity image.
The means for generating the entity image generates an entity image corresponding to the display target specified by the user instruction.
The information processing device according to any one of claims 1 to 10. The means for generating the entity image is the first entity information about the entity at the first reference time point and the second entity information about the entity at the second reference time point different from the first reference time point. Generate an entity image corresponding to the entity by referring to the difference between them.
The information processing device according to any one of claims 1 to 11. The means for generating the entity image refers to the locus of the position of the entity from the first reference time point to the second reference time point different from the first reference time point, and refers to the entity corresponding to the entity. Generate an image,
The information processing device according to any one of claims 1 to 12. The means for generating the spatial image is different from the first index value at the first reference time of the index associated with the component of the three-dimensional model information of the target space and the first reference time of the index. With reference to the difference between the second index value and the second index value at the reference time of 2, the portion of the spatial image corresponding to the component is generated.
The information processing device according to any one of claims 1 to 13. The index is at least one of a plurality of points related to the point cloud data, the number of points associated with the component of the three-dimensional model information, and the finished shape associated with the component of the three-dimensional model information.
The information processing device according to claim 14. The means for identifying the entity updates the entity associated with the reference time point in response to the user selecting the reference time point.
The means for generating the entity image updates the entity image in response to the update of the entity.
The means for generating the composite image updates the composite image in response to the update of the entity image.
The information processing device according to any one of claims 1 to 15. The computer
A step of identifying a sensor that was placed in the target space at at least one reference time point, and an entity associated with the reference time point that is at least one of the detection targets of the sensor at the reference time point.
Steps to generate an entity image corresponding to the entity,
A step of generating a spatial image corresponding to the target space, and
In the spatial image, a step of generating a composite image by synthesizing the entity image and the spatial image so that the entity image is arranged at a position corresponding to the position of the entity at the reference time point. Information processing method to be provided. A program for causing a computer to realize each of the means according to any one of claims 1 to 16.

Images