JP2020052719A - Data generation method, device, and program - Google Patents

Abstract

To provide a method, a device, and a program for generating three-dimensional data of another subject capable of accurately reproducing a positional relationship with a subject whose three-dimensional data has been acquired.SOLUTION: A processing unit of a data generation device acquires a plurality of captured images captured from a plurality of positions and first three-dimensional data indicating a three-dimensional position of each point in a first subject included in the plurality of captured images. The processing unit generates, based on the plurality of acquired captured images, second three-dimensional data indicating a three-dimensional position of each point in a second subject whose distance from the first subject is a predetermined threshold or less, in the same coordinate space as the three-dimensional position indicated by the first three-dimensional data.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a data generation method, an apparatus, and a program.

  In recent years, virtual reality (VR) technology has been applied to various fields. For example, by converting a real space into a three-dimensional model using various sensors and displaying an image of a virtual space obtained by virtualizing the real space using the three-dimensional model, an environment of a certain place can be three-dimensionally viewed in a remote place. Can be reproduced. A remote user wears a head mounted display (HMD) equipped with a sensor that can detect his or her position and orientation, and walks while displaying an image of a virtual space. You can experience as if walking. Such technology is being developed not only in the entertainment field but also as a technology for supporting various tasks and tasks.

  Also, there is texture mapping as a technique for displaying a three-dimensional image closer to the real space using a three-dimensional model. Regarding this technology, a three-dimensional image generation device that selects, from a plurality of captured images having different viewpoints, an image having the highest resolution for each patch of the three-dimensional image as a texture image of the patch has been proposed. The following image generation apparatus has been proposed for texture mapping for billboard technology applied to objects serving as background elements. This image generation device stores in advance a texture image of an object viewed from a plurality of directions, and generates a texture image corresponding to the plate polygon based on the relationship between the reference direction of the plate polygon and the direction from the plate polygon to the viewpoint. Identify.

JP 2002-24850 A JP 2004-287504 A

  By the way, when the technology of reproducing the environment of a work site in a virtual space using a three-dimensional model is used for training and education of work, not only the main work object but also the shape of the objects existing around it are virtual space. In some cases it is preferable to be reproduced. Thus, the viewer of the reproduced image can accurately grasp the positional relationship between the work target and the objects existing around the work target.

  For example, in the case of operating a crane to lift and lower a load (work target), in order to prevent contact with objects other than the load within the operating range of the crane, the position and shape of the load can only be recognized. Is not enough. In this operation, it is preferable that the situation around the place where the baggage is lifted and lowered can be recognized as three-dimensionally as possible.

  Here, the three-dimensional data indicating the shape of the object can be acquired with relatively high accuracy using, for example, a three-dimensional scanner provided with a distance sensor (depth sensor). However, if such a three-dimensional scanner is used to acquire not only the work target but also three-dimensional data indicating the shape of the object existing therearound, it is possible to measure the three-dimensional shape of the object at a remote position A high-performance and expensive three-dimensional scanner is required. For this reason, it is realistic to acquire only three-dimensional data indicating the shape of the work target by using an inexpensive three-dimensional scanner for cost reduction.

  Further, when only three-dimensional data indicating the shape of the work target is acquired, a method of simply reproducing the environment around the work target using a captured image is considered. For example, a method using the aforementioned billboard technology or a method of pasting an image of the surrounding environment on the inner surface of a sphere centering on the work target can be considered. However, in this method, since the shape of the surrounding object is not three-dimensionally drawn in the virtual space, there is a problem that the positional relationship between the work target and the surrounding object is not accurately reproduced.

  In one aspect, an object of the present invention is to provide a data generation method, apparatus, and program capable of generating three-dimensional data of another subject that can accurately reproduce a positional relationship with a subject for which three-dimensional data has been acquired. And

  In one case, a computer acquires a plurality of captured images captured from a plurality of positions and first three-dimensional data indicating a three-dimensional position of each point in a first subject included in the plurality of captured images. Then, based on the plurality of captured images, a cubic coordinate in the same coordinate space as the three-dimensional position indicated by the first three-dimensional data of each point in the second subject whose distance from the first subject is equal to or less than a predetermined threshold value A data generation method is provided for generating second three-dimensional data indicating an original position.

  In one proposal, the computer acquires a plurality of captured images captured from a plurality of positions and position information of each part on any of the subjects included in the plurality of captured images, and stores the acquired position information in the acquired position information. A first display model corresponding to the three-dimensional shape of any of the subjects is generated based on the acquired plurality of captured images, and a distance from any one of the subjects included in the plurality of captured images is equal to or less than a threshold. A second display model corresponding to the three-dimensional shape of another object is generated, and the generated first display model, the generated second display model, and the first display model and the second display model And a data generation method for generating display data including positional relationship information indicating the positional relationship of the data.

Further, in one proposal, a data generation device that performs processing similar to the above-described data generation method is provided.
Further, in one proposal, a data generation program for causing a computer to execute the same processing as the above data generation method is provided.

  In one aspect, it is possible to generate three-dimensional data of another subject that can accurately reproduce a positional relationship with a subject for which three-dimensional data has been acquired.

FIG. 2 is a diagram illustrating a configuration example and a processing example of a data generation device according to the first embodiment. FIG. 9 is a diagram illustrating a configuration example of an information processing system according to a second embodiment. FIG. 3 is a diagram illustrating an example of a hardware configuration of a server. FIG. 2 is a block diagram illustrating a configuration example of a processing function included in each device of the information processing system. It is a perspective view which shows the example of a work site. It is a top view which shows the example of a work site. It is a figure for explaining a measuring operation by a measurer. FIG. 4 is a diagram for describing generation of a texture image corresponding to a polygon. It is an example (the 1) of the flowchart which shows the production | generation processing procedure of object data. It is an example (the 2) of the flowchart which shows the production | generation processing procedure of object data. It is a figure for explaining narrowing down of a corresponding point. It is a figure for explaining calculation of the distance to a corresponding point. It is an example of the flowchart which shows the production | generation processing procedure of intermediate | middle background data. It is a figure showing the example of the three-dimensional shape of the perimeter. It is an example of the flowchart which shows the generation | occurrence | production process procedure of a periphery data. FIG. 4 is a diagram illustrating an example of an image in which a virtual space is drawn. FIG. 9 is a diagram illustrating an example of a conversion graph used in a position correction process for an intermediate background object. It is an example of the flowchart which shows the processing procedure of position correction of an intermediate background object. FIG. 4 is a diagram for describing a complementary region of a three-dimensional shape in a virtual space. It is a figure showing the 1st example about the complementation processing of a three-dimensional shape. FIG. 11 is a diagram illustrating a second example of a three-dimensional shape complementing process. It is an example of the flowchart which shows the procedure of the complementation process of a three-dimensional shape.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.
[First Embodiment]
FIG. 1 is a diagram illustrating a configuration example and a processing example of the data generation device according to the first embodiment.

  The data generation device 1 shown in FIG. 1 has a storage unit 1a and a processing unit 1b. The storage unit 1a is realized, for example, as a storage device (not shown) included in the data generation device 1. The processing unit 1b is realized, for example, as a processor (not shown) included in the data generation device 1.

  The processing unit 1b calculates the three-dimensional position of each point in the first subject 3a included in the captured images 2a, 2b,... And the captured images 2a, 2b,. The first three-dimensional data 4a shown is acquired (step S1). The processing unit 1b stores the acquired captured images 2a, 2b,... And the first three-dimensional data 4a in the storage unit 1a.

  In the real space, the first subject 3a and the second subject 3b whose distance from the first subject 3a is equal to or less than a predetermined threshold are subjects that can be captured in the captured images 2a, 2b,. Exists. The first subject 3a is a main subject to be imaged, and basically appears in all of the captured images 2a, 2b,... For example, the captured images 2a, 2b,... Are obtained by, for example, capturing the first subject 3a a plurality of times with the camera 5 while orbiting around the first subject 3a.

  For example, when a predetermined work is performed in the real space, the first subject 3a corresponds to an object to be worked. The second subject 3b corresponds to an object existing around the first subject 3a.

  The first three-dimensional data 4a is acquired, for example, based on a measurement result obtained by measuring a distance from the position of the camera 5 to the first subject 3a a plurality of times using a distance sensor (depth sensor). . By using the first three-dimensional data 4a, the shape of the first subject 3a can be three-dimensionally drawn in the virtual space.

  Next, the processing unit 1b generates second three-dimensional data 4b indicating the three-dimensional position of each point on the second subject 3b based on the acquired captured images 2a, 2b,. 1a (step S2). The second three-dimensional data 4b uses, for example, a plurality of captured images of the second captured image 2a, 2b,... It is calculated based on the position of the feature point of the subject 3b. By using the second three-dimensional data 4b, the shape of the second subject 3b can be three-dimensionally drawn in the virtual space.

  The three-dimensional position of each point indicated by the second three-dimensional data 4b indicates a three-dimensional position in the same coordinate space as the three-dimensional position indicated by the first three-dimensional data 4a. For this reason, by using the first three-dimensional data 4a and the second three-dimensional data 4b, the positional relationship between the three-dimensional shape of the first subject 3a and the three-dimensional shape of the second subject 3b can be described. It can be accurately reproduced in the coordinate space (virtual space).

  According to the data generation device 1 described above, even in a situation where only the first three-dimensional data 4a corresponding to the first subject 3a has been obtained, the second subject 3b can be obtained using the captured images 2a, 2b,. Can be generated as the second three-dimensional data 4b. For example, there may be a case where the second subject 3b is in a range where its three-dimensional data cannot be acquired using a distance sensor or the like, but is located at a position relatively short from the first subject 3a. In such a case, the data generation device 1 can generate the second three-dimensional data 4b corresponding to the second subject 3b using the captured images 2a, 2b,. Then, the data generation device 1 uses the acquired first three-dimensional data 4a and the generated second three-dimensional data 4b to generate the three-dimensional shape of the first subject 3a and the second subject 3b. An image that accurately reproduces the positional relationship with the three-dimensional shape can be generated and displayed.

  For example, when the first subject 3a is a work target, in addition to the three-dimensional shape of the first subject 3a, the three-dimensional shape of the second subject 3b existing around the first subject 3a is It is reproduced as an image of a virtual space at a place different from the work site. At this time, the viewer of the image can accurately grasp the positional relationship between the first subject 3a to be worked and the second subject 3b surrounding the first subject 3a at a place different from the work site. become. Therefore, for example, useful three-dimensional data that can be used for work training and education can be generated.

[Second embodiment]
Next, as a second embodiment, an information processing system including the data generation device 1 shown in FIG. 1 will be exemplified.

FIG. 2 is a diagram illustrating a configuration example of an information processing system according to the second embodiment.
The information processing system shown in FIG. 2 converts the environment of the work site 100 into a three-dimensional model and uses the three-dimensional model to reproduce an image showing a virtual space obtained by virtualizing the environment of the work site 100 in a remote place. System. In FIG. 2, an office 200 is illustrated as a remote place. This information processing system includes a server 300. The server 300 is an example of the data generation device 1 shown in FIG.

  At the work site 100, a predetermined work is performed on a work target. For example, in accordance with an operation of an operator, a load is raised and lowered by a construction machine such as a crane or a forklift and transported. In this case, the luggage is the work object. Hereinafter, the work performed by the worker on the work target at the work site 100 will be referred to as “field work”.

  Further, at the work site 100, a measurement operation for measuring the three-dimensional shape of the work object and the surrounding environment is performed by a measurer. As shown in FIG. 2, a space recognition sensor 101, a camera 102, and a position / direction sensor 103 are used as devices for measurement work.

  The space recognition sensor 101 is a sensor that can three-dimensionally recognize the space of the work site 100. As such a sensor, for example, there is a depth sensor that measures a depth (distance) between the sensor and an object. The depth sensor includes, for example, an infrared irradiator and an infrared receiver, and measures the depth of each pixel by a TOF (Time Of Flight) method or the like.

  The camera 102 is a device that can capture an RGB (Red Green Blue) image. The camera 102 is arranged such that its position and direction match the space recognition sensor 101 as much as possible. The image captured by the camera 102 is mainly used to make a three-dimensional modeled object based on the measurement result of the space recognition sensor 101 look more realistic in a virtual space.

  The position / direction sensor 103 is a sensor capable of recognizing the position and direction (posture) of the space recognition sensor 101 and the camera 102. As such a sensor, for example, an acceleration sensor and an angular velocity sensor (gyro) can be used in combination. Further, a direction sensor may be further combined with the acceleration sensor and the angular velocity sensor.

  At the work site 100, a measurer performs measurement work using the above-described sensors, for example, while orbiting around the work target. At this time, sensor data from each sensor is captured at a plurality of locations. In this measurement work, as described above, the work is performed so that the position and the direction of the space recognition sensor 101 and the camera 102 match as much as possible. For example, the measurement operation is performed in a state where the measurer wears the head mount unit on which the space recognition sensor 101, the camera 102, and the position / direction sensor 103 are mounted on the head.

  The sensor data thus obtained is transmitted to the server 300. The server 300 generates three-dimensional data 301 indicating a three-dimensional shape of the space of the work site 100 based on the received sensor data. The three-dimensional data 301 includes point cloud data indicating the three-dimensional coordinates of each feature point detected from the space of the work site 100. This point group data may be data indicating the positions of the vertices of a polygon generated by each feature point. Note that the point cloud data may be calculated by an information processing device arranged in the work site 100 and transmitted to the server 300. Further, the three-dimensional data 301 includes data of a texture image attached to each polygon based on the point cloud data. The texture image is generated based on sensor data (that is, image data) from the camera 102.

  On the other hand, in the office 200, the viewer wears the HMD 201. The HMD 201 is equipped with a device for detecting the position and direction of the viewer in addition to the display, and the position and direction of the viewer are transmitted to the server 300. The server 300 draws an image indicating a virtual space obtained by virtualizing the space (real space) of the work site 100 based on the three-dimensional data 301 and the position and direction of the viewer, and causes the HMD 201 to display the image.

  In this image display, the position and direction of the viewer are associated with the coordinates of the virtual space, and the situation of the virtual space viewed from the viewer's viewpoint is reproduced on the image based on the detection result of the position and direction of the viewer. You. For example, a three-dimensional modeled object is arranged in a virtual space at a position and orientation corresponding to the position and direction of a viewer. Further, when the position or the line of sight of the viewer changes, the position or orientation of the object in the virtual space also changes according to the change. Thus, the viewer can experience as if the viewer actually exists in the work site 100 and looks at the environment while moving in the work site 100.

  With such an information processing system, the viewer can grasp the situation of the work site 100 in detail without actually going to the work site 100. This allows the viewer to receive education and training on the work at the work site 100 in a place other than the work site 100 such as the office 200.

  Further, the audio device 104 may be arranged in the work site 100, and the HMD 201 may have an audio input / output function. In this case, while the viewer of the office 200 is visually recognizing the virtual space of the work site 100, it is possible to communicate by voice with the worker of the work site 100.

  In the office 200, a plurality of viewers each wearing the HMD 201 may visually recognize an image in the virtual space of the work site 100. In this case, by the processing of the server 300, an image corresponding to the position and direction of each viewer is displayed on the HMD 201 of each viewer. Further, by the processing of the server 300, an avatar indicating this viewer may be displayed on the image of the virtual space in accordance with the result of detecting the position and direction of the other viewer.

  Further, a plurality of offices 200 may be installed, and a viewer wearing the HMD 201 in each office 200 may visually recognize an image in the virtual space of the work site 100. Even in this case, an image corresponding to the position and direction of each viewer is displayed on the HMD 201 of each viewer. Further, an avatar indicating this viewer may be displayed on the image of the virtual space in accordance with the detection result of the position and direction of the other viewer. Further, the viewers may be able to communicate by voice.

  As described above, the information processing system according to the present embodiment virtually reproduces the three-dimensional shape of the work site 100 in a remote place such as the office 200. Thus, for example, as compared with a case where a photographer or a moving image showing the situation of the work site 100 is viewed by a remote viewer, the viewer can grasp the situation of the work site 100 accurately. For this reason, the usefulness of the education and training can be enhanced by using this information processing system for education and training of field work in a remote place.

  FIG. 3 is a diagram illustrating an example of a hardware configuration of a server. The server 300 is realized, for example, as a computer as shown in FIG. The server 300 illustrated in FIG. 3 includes a processor 311, a random access memory (RAM) 312, a hard disk drive (HDD) 313, a graphic processing device 314, an input interface (I / F) 315, a reading device 316, and a communication interface (I / F) having 317;

  The processor 311 controls the server 300 as a whole. The processor 311 is, for example, a central processing unit (CPU), a micro processing unit (MPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), or a programmable logic device (PLD). Further, the processor 311 may be a combination of two or more elements among a CPU, an MPU, a DSP, an ASIC, and a PLD.

  The RAM 312 is used as a main storage device of the server 300. The RAM 312 temporarily stores at least a part of an OS (Operating System) program and an application program to be executed by the processor 311. The RAM 312 stores various data required for processing by the processor 311.

  The HDD 313 is used as an auxiliary storage device of the server 300. The HDD 313 stores an OS program, an application program, and various data. It should be noted that another type of non-volatile storage device such as a solid state drive (SSD) can be used as the auxiliary storage device.

  A display device 314a is connected to the graphic processing device 314. The graphic processing device 314 causes the display device 314a to display an image according to an instruction from the processor 311. Examples of the display device include a liquid crystal display and an organic EL (Electroluminescence) display.

  An input device 315a is connected to the input interface 315. The input interface 315 transmits a signal output from the input device 315a to the processor 311. Examples of the input device 315a include a keyboard and a pointing device. Examples of the pointing device include a mouse, a touch panel, a tablet, a touch pad, and a trackball.

  The portable recording medium 316a is attached to and detached from the reading device 316. The reading device 316 reads data recorded on the portable recording medium 316a and transmits the data to the processor 311. Examples of the portable recording medium 316a include an optical disk, a magneto-optical disk, and a semiconductor memory.

The communication interface 317 transmits and receives data to and from another device via the network 317a.
With the hardware configuration described above, the processing functions of the server 300 can be realized.

FIG. 4 is a block diagram illustrating a configuration example of a processing function included in each device of the information processing system.
First, a specific example of a measurement sensor used in the work site 100 will be described with reference to FIG. In the work site 100, a three-dimensional scanner 110 and a spherical camera 120 are arranged as such sensors.

  The three-dimensional scanner 110 includes a depth sensor 111, an inertial measurement unit (IMU) 112, and a three-dimensional data calculation unit 113. The depth sensor 111 is a device that realizes the function of the space recognition sensor 101 in FIG. 2, and detects the depth (distance) from the measurement target object for each pixel. The inertial measurement unit 112 is a device that realizes the function of the position / direction sensor 103 in FIG. 2 and includes an acceleration sensor, an angular velocity sensor, and a direction sensor, and the position and direction of the three-dimensional scanner 110 based on these sensor data. Is detected.

  The three-dimensional data calculation unit 113 is realized as a processor such as a CPU and a DSP. The three-dimensional data calculation unit 113 calculates point cloud data indicating a three-dimensional model of the space to be measured based on the detection results of the depth sensor 111 and the inertial measurement unit 112. Further, the three-dimensional data calculation unit 113 further generates a polygon based on each feature point included in the point cloud data, and outputs polygon data indicating the generated polygon to the server 300. The polygon data includes three-dimensional coordinates of vertices of each polygon. Further, each time the depth sensor 111 measures depth, the three-dimensional data calculation unit 113 outputs the position and direction detection results of the inertial measurement unit 112 to the server 300.

  The omnidirectional camera 120 is a device that realizes the function of the camera 102 in FIG. 2. The omnidirectional camera 120 captures an image in a 360 ° direction around a predetermined reference direction, and stores the obtained spherical image data in the server 300. Output to The omnidirectional camera 120 is arranged so that its reference direction matches the measurement direction of the depth sensor 111. Also, these sensors are placed in close proximity so that they are considered to be at the same position. The imaging operation by the spherical camera 120 is performed in synchronization with the depth measurement operation by the depth sensor 111. Accordingly, the position and direction are detected by the inertial measurement unit 112 at the timing when the spherical image is captured, and the detected position and direction are defined as the camera position and camera direction when capturing the spherical image. Each can be used.

  The three-dimensional scanner 110 and the spherical camera 120 are mounted on, for example, a head mount unit. Then, the measurement operation is performed by a measurer who has mounted the head mount unit on the head. The main purpose of this measurement work is to obtain three-dimensional data of a work target that is a target of the work on site among objects existing in the work site 100. For this reason, the measurer performs the depth measurement and the imaging of the omnidirectional image a plurality of times while directing the three-dimensional scanner 110 and the omnidirectional camera 120 around the operation object toward the operation object. As a specific example, the measurer performs depth measurement and captures a spherical image a plurality of times while moving around the work target. Thereby, the data of the spherical image at each imaging timing, the data of the position and direction at the time of the imaging, and the polygon data are transmitted to the server 300.

  Note that at least a part of the processing of the three-dimensional data calculation unit 113 may be executed in the server 300. For example, in the processing of the three-dimensional data calculation unit 113, the calculation of the point cloud data may be executed by the three-dimensional data calculation unit 113, and the generation of polygon data based on the point cloud data may be executed by the server 300. Alternatively, the detection results of the depth sensor 111 and the inertial measurement unit 112 may be transmitted to the server 300, and the processing of the three-dimensional data calculation unit 113 based on these detection results may be executed by the server 300.

  By the way, in the measurement by the three-dimensional scanner 110, mainly from the performance of the depth sensor 111, there is an upper limit value of a distance from a measurement target capable of measuring a three-dimensional shape with a certain accuracy or more. The measurement work by the measurer is performed such that the work object is surely included in the range within the upper limit value of the distance. For this reason, the polygon data transmitted to the server includes data indicating the three-dimensional shape of the work target, but does not include data indicating the three-dimensional shape of the object located at a position farther away.

  Therefore, the server 300 performs a tertiary-based search on an object existing around the work target and within a certain distance from the work target based on the data of the spherical image and the position and direction data at the time of imaging. Measure the original shape. Hereinafter, such an object is referred to as an “intermediate background object”. The server 300 generates three-dimensional data indicating the three-dimensional shape of the intermediate background object.

  Further, the server 300 generates a spherical three-dimensional model centered on the work object at a position farther than the intermediate background object as viewed from the work object in the virtual space. Then, based on the data of the spherical image, the server 300 pastes, as a texture image, an image indicating the situation of the area farther than the intermediate background object in the real space on the inner surface of the spherical three-dimensional model. As a result, three-dimensional data indicating a three-dimensional shape of almost the entire real space of the work site 100 is obtained. In the following description, the inner surface of the spherical three-dimensional model as described above is referred to as “outer periphery”.

Hereinafter, the processing functions of the server 300 will be described with reference to FIG.
The server 300 includes a polygon data acquisition unit 321, an image data acquisition unit 322, an object data generation unit 323, an intermediate background object data generation unit 324, an outer circumference data generation unit 325, an intermediate background object position correction unit 326, a complement processing unit 327, It has a three-dimensional data storage unit 330 and an image data storage unit 340.

  The processing of the polygon data acquisition unit 321, the image data acquisition unit 322, the target object data generation unit 323, the intermediate background object data generation unit 324, the outer circumference data generation unit 325, the intermediate background object position correction unit 326, and the complement processing unit 327 is performed, for example. Are realized by the processor 311 executing a predetermined program. The three-dimensional data storage unit 330 and the image data storage unit 340 are realized by, for example, a storage area of a storage device included in the server 300, such as the RAM 312 or the HDD 313.

  The polygon data acquisition unit 321 receives the polygon data as a result of the three-dimensional scan from the three-dimensional data calculation unit 113 of the three-dimensional scanner 110. As described above, the polygon data includes data indicating the three-dimensional shape of the work target.

  The image data acquisition unit 322 receives the data of the spherical image from the spherical camera 120 and stores the data in the image data storage unit 340. Thereby, the omnidirectional image 341 captured by the omnidirectional camera 120 is stored in the image data storage unit 340. Further, the image data acquisition unit 322 receives, from the three-dimensional data calculation unit 113 of the three-dimensional scanner 110, data indicating the position and direction at each imaging timing of the omnidirectional image 341. The image data obtaining unit 322 adds the received position and direction data to the omnidirectional image 341 as data indicating the camera position and camera direction at the time of capturing the omnidirectional image 341 and stores the image data. Stored in the section 340.

  The object data generation unit 323 generates a texture image to be attached to each polygon of the polygon data acquired by the polygon data acquisition unit 321 using the omnidirectional image 341 stored in the image data storage unit 340. The object data generating unit 323 stores the polygon data with the texture image added to the polygon data in the three-dimensional data storage unit 330 as the object data 331. Note that the target object data 331 includes at least each data of a polygon indicating the three-dimensional shape of the work target and a texture image.

  The intermediate background object data generation unit 324 generates an intermediate background object in each spherical image 341 based on the spherical image 341 stored in the image data storage unit 340 and the camera position and camera direction added thereto. Calculate the distance from the camera position. The intermediate background object data generation unit 324 calculates point cloud data indicating a three-dimensional model of the intermediate background object based on the calculated distance, the camera position, and the camera direction. The intermediate background object data generation unit 324 generates a polygon based on each feature point included in the calculated point group data, and generates a texture image to be attached to the polygon using the omnidirectional image 341. The intermediate background object data generation unit 324 stores the polygon data with the texture image for the intermediate background object thus generated in the three-dimensional data storage unit 330 as the intermediate background object data 332.

  The outer circumference data generating unit 325 generates a polygon of the outer circumference (the inner surface of a spherical three-dimensional model having a predetermined radius around the work target) and uses the celestial sphere image 341 stored in the image data storage unit 340 to generate A texture image corresponding to the outer polygon is generated. The outer periphery data generation unit 325 stores the polygon data with the texture image for the outer periphery thus generated in the three-dimensional data storage unit 330 as the outer periphery data 333.

The intermediate background object position correction unit 326 corrects the position of the intermediate background object in the virtual space according to the distance between the work object and the intermediate background object in the real space.
The complementing processing unit 327 complements a three-dimensional shape that is not represented by any of the object data 331, the intermediate background object data 332, and the outer periphery data 333 among the three-dimensional shapes in the real space, and indicates the complemented three-dimensional shape. Output data. By this complementing process, for example, data indicating the three-dimensional shape of the region between the work object and the intermediate background object and the region between the intermediate background object and the outer periphery are easily generated. The complement processing unit 327 stores the polygon data with the texture image for these areas in the three-dimensional data storage unit 330 as the complement data 334.

  Thus, the data for reproducing the environment of the work site 100 in the virtual space is stored in the three-dimensional data storage unit 330. These data stored in the three-dimensional data storage unit 330 correspond to the three-dimensional data 301 in FIG.

Next, an example of the work site 100 will be described with reference to FIGS.
FIG. 5 is a perspective view showing an example of a work site. FIG. 6 is a plan view showing an example of a work site. As shown in FIGS. 5 and 6, objects 131 to 136 exist in the work site 100. Of these, the object 131 is a work target that is a target of the field work. The objects 132 and 133 are located relatively close to the object 131. On the other hand, the objects 134 to 136 are located farther from the object 131 than the objects 132 and 133.

FIG. 7 is a diagram illustrating a measurement operation performed by a measurer.
As shown on the left side of FIG. 7, at the work site 100, a measurer wearing a head-mounted unit equipped with a three-dimensional scanner 110 and a spherical camera 120 on his / her head moves toward the object 131, which is the work target. While circling around the object 131 as shown by the arrow 141. The measurer performs the depth measurement and the imaging with the omnidirectional camera 120 a plurality of times at substantially constant intervals during the orbit. Further, the measurer may perform the above measurement by making a plurality of rounds around the object 131 while changing the height and angle of the head mount unit, for example.

  The three-dimensional data calculation unit 113 of the three-dimensional scanner 110 calculates, based on the depth, the position, and the direction measured at each measurement position, point group data of the three-dimensional model 331a in the real space in a range where the depth can be measured. . The three-dimensional data calculation unit 113 generates a polygon based on each feature point included in the calculated point group data, and outputs polygon data indicating the generated polygon to the server 300. In the upper right of FIG. 7, a three-dimensional model 331a as a set of generated polygons is shown. The shape of the polygon (patch) does not have to be a quadrangle as in the example of FIG. 7, but may be a polygon of another shape such as a triangle.

  The three-dimensional model 331a based on the polygons generated by the three-dimensional scanner 110 in this way includes the three-dimensional shape of the object 131 that is the work target. Therefore, in the following description, a polygon generated by the three-dimensional scanner 110 is simply referred to as a “polygon of a work target”.

  On the other hand, the omnidirectional camera 120 outputs the omnidirectional images 341a, 341b,... Taken at the respective measurement positions to the server 300, as shown in the lower right of FIG. As described above, in the server 300, the camera position at the time of imaging and the camera direction are associated with each of the omnidirectional images 341a, 341b,.

Next, a generation process of the target object data 331 will be described with reference to FIGS.
FIG. 8 is a diagram for explaining generation of a texture image corresponding to a polygon. As described above, the object data generation unit 323 uses the omnidirectional image 341 stored in the image data storage unit 340 to generate a texture image to be attached to each polygon of the polygon data acquired by the polygon data acquisition unit 321. Generate.

  Specifically, the object data generating unit 323 selects one polygon and calculates a normal vector of the surface (polygon surface) of the polygon based on the three-dimensional coordinates of each vertex of the polygon. In FIG. 8, as an example, the polygon 151 is selected from the polygons of the three-dimensional model 331a, and the normal vector V of the polygon surface of the polygon 151 is calculated.

  Further, the object data generating unit 323 selects the omnidirectional image 341 including the polygon surface of the polygon 151 from the omnidirectional image 341 stored in the image data storage unit 340. The object data generation unit 323 calculates a vector for each camera position of the selected omnidirectional image 341 from the center of the polygon surface. In the example of FIG. 8, a vector Va for the camera position 152a, a vector Vb for the camera position 152b, and a vector Vc for the camera position 152c are calculated.

  The object data generation unit 323 selects a vector having the smallest difference in direction from the normal vector V from the calculated vectors Va, Vb, and Vc. The target object data generation unit 323 specifies the omnidirectional image 341 corresponding to the selected vector, and sets the image of the area of the polygon 151 in the specified omnidirectional image 341 as a texture image corresponding to the polygon 151. .

  By executing such processing for all polygons, a texture image is associated with each polygon. As shown in the lower part of FIG. 8, when displaying an image in the virtual space, a texture image corresponding to each polygon is drawn, so that an image 331b of a realistic work object close to the appearance in the real space is displayed. Is displayed.

FIG. 9 and FIG. 10 are examples of flowcharts showing the procedure for generating target object data.
[Step S11] The polygon data acquisition unit 321 receives polygon data of a work target from the three-dimensional scanner 110. Further, the image data acquisition unit 322 receives the data of the omnidirectional image from the omnidirectional camera 120 and also receives the position and direction (ie, the camera position and the camera direction) at the time of capturing each omnidirectional image. . The image data acquisition unit 322 stores the data of each celestial sphere image in the image data storage unit 340, and adds the camera position and the camera direction to the corresponding celestial sphere image data, and stores the data in the image data storage unit 340. Store.

[Step S12] The object data generation unit 323 selects one polygon from the polygon data of the work object.
[Step S13] The target object data generation unit 323 selects one of the celestial sphere images stored in the image data storage unit 340.

  [Step S14] The object data generating unit 323 determines whether the surface of the polygon (polygon surface) selected in step S12 is included in the spherical image selected in step S13. This determination is made based on the coordinates of each vertex of the polygon selected in step S12 and the camera position and camera direction of the omnidirectional image selected in step S13. The target object data generation unit 323 executes the process of step S15 when the polygon surface portion is included in the omnidirectional image, and executes the process of step S17 when it is not included.

  [Step S15] The object data generation unit 323 temporarily stores the image of the portion corresponding to the polygon surface in the omnidirectional image selected in Step S13 in the RAM 312 as a texture image candidate corresponding to the polygon selected in Step S12. To save. At this time, the object data generation unit 323 adds the camera position of the corresponding omnidirectional image to the texture image candidate.

  [Step S16] The target object data generation unit 323 deletes the image of the portion specified as the texture image candidate in step S15 from the omnidirectional image selected in step S13. Thereby, the image data of the deleted portion is deleted from the data of the spherical image.

  [Step S17] The target object data generation unit 323 determines whether or not the image data storage unit 340 has an omnidirectional image that has not been selected in step S13. If there is an unselected celestial sphere image, the object data generating unit 323 advances the process to step S13, and selects one from the unselected celestial sphere images. On the other hand, when all the omnidirectional images have been selected, the object data generating unit 323 executes the process of step S18.

  [Step S18] The object data generating unit 323 determines whether there is any polygon not selected in step S12 in the polygon data. If there is an unselected polygon, the object data generating unit 323 advances the process to step S12, and selects one from the unselected polygons. In this case, the selection status of the omnidirectional image is reset so that all the omnidirectional images stored in the image data storage section 340 are not selected in step S13. On the other hand, when all the polygons have been selected, the target object data generation unit 323 executes the process of step S21 in FIG.

Hereinafter, the description will be continued with reference to FIG.
[Step S21] The object data generation unit 323 selects one polygon from the polygon data of the work object.

[Step S22] The object data generating unit 323 calculates a normal vector v of the polygon surface f based on the coordinates of each vertex of the selected polygon.
[Step S23] The object data generating unit 323 calculates a vector for the camera position from the center of the polygon surface f for each of the texture image candidates corresponding to the selected polygon stored in step S15 of FIG. The camera position is information added to the corresponding texture image candidate in step S15 in FIG. 9, and indicates the camera position of the omnidirectional image from which the texture image candidate has been extracted.

  [Step S24] The object data generating unit 323 specifies a vector having the smallest difference in direction from the normal vector v calculated in step S22 from the vectors calculated in step S23. The object data generation unit 323 determines a texture image candidate corresponding to the specified vector as a texture image corresponding to the polygon selected in step S21.

  At the first execution of step S24, the object data generating unit 323 newly generates the object data 331 and stores it in the three-dimensional data storage unit 330. Then, the object data generation unit 323 includes the data indicating the selected polygon and the data indicating the corresponding texture image in the object data 331. The data indicating the polygon includes coordinate data of each vertex of the polygon. On the other hand, in the second and subsequent executions of step S24, the object data generation unit 323 includes the data indicating the selected polygon and the data indicating the corresponding texture image in the existing object data 331.

  [Step S25] The object data generating unit 323 determines whether there is any polygon not selected in step S21 in the polygon data. If there is an unselected polygon, the object data generating unit 323 advances the process to step S21, and selects one from the unselected polygons. On the other hand, when all the polygons have been selected, the object data generation unit 323 ends the generation processing of the object data 331.

  When the above processing is completed, the object data 331 including data indicating all polygons and data indicating a texture image corresponding to each polygon is stored in the three-dimensional data storage unit 330. In addition, with respect to the omnidirectional image stored in the image data storage unit 340, a part in which a three-dimensionally modeled object (mainly a work object) is extracted as the object data 331 is extracted. I have.

Next, the generation processing of the intermediate background object data 332 by the intermediate background object data generation unit 324 will be described with reference to FIGS.
First, the intermediate background object data generation unit 324 aligns the directions of the spherical images stored in the image data storage unit 340. In this process, each omnidirectional image is converted so that the imaging position (camera position) of the camera that captured each omnidirectional image is the same as the imaging direction (azimuth). This conversion processing is performed based on the camera position and camera direction added to each celestial sphere image, or by matching between celestial sphere images. Next, the intermediate background object data generation unit 324 extracts a feature point from each omnidirectional image after conversion, and extracts a corresponding point indicating the same characteristic point between the omnidirectional images.

  In each celestial sphere image, the subject corresponding to the intermediate background object appears on the back side of the subject corresponding to the work target. Therefore, the intermediate background object is located at a distance from the camera position of the spherical camera 120 to some extent. Therefore, the intermediate background object data generation unit 324 first extracts corresponding points by matching feature points between omnidirectional images. After that, the intermediate background object data generation unit 324 excludes the extracted corresponding points that are estimated to be close to the camera position from the extracted corresponding points by the processing shown in FIG. Narrow down to only the corresponding points.

  FIG. 11 is a diagram for describing narrowing down of corresponding points. In FIG. 11, it is assumed that feature points 163 and 164 on the subject are captured in both the image 161a captured from the camera position 161 and the image 162a captured from the camera position 162. That is, the feature points 163 and 164 are all extracted as corresponding points between the image 162a and the image 163a. It is also assumed that the distance from the camera positions 161 and 162 to the feature point 164 is longer than the distance from the camera positions 161 and 162 to the feature point 163.

  Here, usually, when the camera position changes, the position of the same feature point (corresponding point) in the captured image moves. In the case where the amount of change in the camera position is the same, the longer the distance from the camera position to the feature point, the smaller the movement amount (position shift amount) of the feature point in the image. In the case of FIG. 11, the amount of movement of the position of the feature point 163 in each of the images 161a and 162a is larger than the amount of movement of the position of the feature point 164 in each of the images 161a and 162a.

  Therefore, when the moving amount (distance based on the two-dimensional coordinates) of the corresponding point between the omnidirectional images is equal to or less than a predetermined amount, the intermediate background object data generating unit 324 leaves the corresponding point as it is, while keeping the moving amount If the amount exceeds the fixed amount, it is determined that the corresponding point is too close to the camera position, and the corresponding point is excluded. As a result, the corresponding points can be narrowed down to only those located at a certain distance from the camera position. In the celestial sphere image, although the image portion of the work object or the object close to the work object has already been extracted, if a part of such an image portion remains, such an image portion is narrowed down by the above-described narrowing down. The corresponding points extracted from can be excluded. In the example of FIG. 11, the corresponding point corresponding to the feature point 164 is left as it is, and the corresponding point corresponding to the feature point 163 is excluded.

  After that, the intermediate background object data generation unit 324 calculates the distance from the camera position to the corresponding point based on the position of the narrowed corresponding point in the omnidirectional image and the camera position and camera direction of the omnidirectional image. calculate. However, the accuracy of calculating the distance decreases as the distance from the camera position to the corresponding point increases. Therefore, the intermediate background object data generation unit 324 further narrows down the corresponding points narrowed down by the above method to only those corresponding points whose distance from the camera position is equal to or less than a certain value. The feature points of the subject corresponding to the corresponding points excluded at this time are drawn on the outer periphery.

  FIG. 12 is a diagram for explaining calculation of the distance to the corresponding point. In FIG. 12, it is assumed that the corresponding point 172 is captured in both the image captured from the camera position 171a and the image captured from the camera position 171b. Further, the distance between the camera position 171a and the camera position 171b is L. Further, it is assumed that a straight line connecting the camera position 171a and the camera position 171b and a straight line connecting the camera position 171a and the corresponding point 172 form an angle θa. In addition, a straight line connecting the camera position 171a and the camera position 171b and a straight line connecting the camera position 171b and the corresponding point 172 form an angle θb.

  Note that the distance L is calculated based on the camera position added to the omnidirectional image captured from each of the camera positions 171a and 171b. The angles θa and θb are calculated based on the camera position and the camera direction added to the omnidirectional image captured from each of the camera positions 171a and 171b.

  Using the distance L and the angles θa and θb, the intermediate background object data generation unit 324 calculates a distance D from the camera position to the corresponding point 172 according to the following equation (1). The intermediate background object data generation unit 324 calculates a distance from each of the camera positions 171a and 171b to the corresponding point 172 based on the calculated distance D.

D = {(sin θa * sin θb) / sin (θa + θb)} * L (1)
The intermediate background object data generation unit 324 calculates the distance between the camera position of the omnidirectional image in which the corresponding point is captured and the corresponding point by the above procedure. The intermediate background object data generation unit 324 calculates the three-dimensional position of the corresponding point based on the camera position and the camera direction added to the omnidirectional image and the distance between the corresponding point. As a result, point cloud data of the intermediate background object is obtained. The intermediate background object data generation unit 324 further generates a texture image corresponding to each polygon based on the point cloud data using the omnidirectional image. As a result, polygon data with a texture image for the intermediate background object is generated and stored in the three-dimensional data storage unit 330 as the intermediate background object data 332.

FIG. 13 is an example of a flowchart showing the procedure for generating intermediate background object data.
[Step S31] The intermediate background object data generation unit 324 aligns the directions of the spherical images stored in the image data storage unit 340. In this process, each omnidirectional image is converted so that the imaging position (camera position) of the camera that captured each omnidirectional image is the same as the imaging direction (azimuth).

  [Step S32] The intermediate background object data generation unit 324 extracts feature points from each celestial sphere image converted in step S31, and extracts corresponding points indicating the same feature point between the celestial sphere images. In this process, corresponding points are extracted between two or more omnidirectional images for each of a large number of feature points. In addition, the image portion corresponding to the work target is deleted from the omnidirectional image by the process of step S16 in FIG. Therefore, in step S32, there is a high possibility that only the feature points of the subject other than the work target are extracted.

[Step S33] The intermediate background object data generation unit 324 selects one extracted corresponding point (that is, the same feature point).
[Step S34] The intermediate background object data generation unit 324 calculates the movement amount (distance based on two-dimensional coordinates) of the corresponding point between the spherical images. In this process, the movement amount of the corresponding point is calculated for each combination of the omnidirectional images in which the selected corresponding point is captured. Note that, for example, as the other celestial sphere image combined with a certain celestial sphere image for calculating the moving amount, only the one with the shortest distance between camera positions may be selected.

  [Step S35] The intermediate background object data generation unit 324 determines whether the calculated movement amount is equal to or smaller than the predetermined threshold TH1, and executes the process of Step S36 if the calculated movement amount is equal to or smaller than the threshold TH1, and if not, the process proceeds to Step S36. The processing of S38 is executed.

  In this determination, for example, the average value of the movement amount calculated for each combination of the celestial sphere images in which the corresponding points are captured is calculated, and when the average value is equal to or less than the threshold value TH1, “the movement amount is equal to or less than the threshold TH1” Is determined. Alternatively, when the longest value of the movement amounts calculated for each combination of the celestial sphere images is equal to or smaller than the threshold value TH1, it may be determined that “the movement amount is equal to or smaller than the threshold value TH1”.

  [Step S36] The intermediate background object data generation unit 324 calculates the distance between the camera position of the omnidirectional image showing the corresponding point and the corresponding point using the above equation (1). The intermediate background object data generation unit 324 determines whether or not the distance of the corresponding point from the camera position is equal to or less than the threshold TH2. If the distance is equal to or less than the threshold TH2, the process of step S37 is performed. If the distance exceeds the threshold TH2, the process of step S38 is performed. Execute

  In this determination, for example, the average value of the distance to the corresponding point calculated for each celestial sphere image in which the corresponding point is captured is calculated, and when the average value is equal to or less than the threshold value TH2, "the distance is equal to or less than the threshold value TH2" Is determined. Alternatively, when the longest value of the distances calculated for each celestial sphere image is equal to or less than the threshold value TH2, it may be determined that “the distance is equal to or less than the threshold value TH2”.

  [Step S37] The intermediate background object data generating unit 324 calculates the distance calculated for each celestial sphere image including the corresponding point, and the camera position and the camera direction added to each celestial sphere image. Calculate the three-dimensional coordinates of the corresponding point. The intermediate background object data generation unit 324 temporarily stores the three-dimensional coordinates of the corresponding point in the RAM 312.

  [Step S38] The intermediate background object data generating unit 324 determines whether there is a corresponding point not selected in step S33. If there is an unselected corresponding point, the intermediate background object data generating unit 324 advances the process to step S33, and selects one of the unselected corresponding points. On the other hand, when all the corresponding points have been selected, the intermediate background object data generation unit 324 executes the process of step S39.

  As a result of the processing in steps S33 to S38, the extracted corresponding point is located at a position that is at least substantially constant distance d1 from the camera position and within a distance d2 longer than the distance d1 (corresponding to threshold value TH2). Existing corresponding points are identified. Therefore, the threshold values TH1 and TH2 are set such that a corresponding point whose distance from the camera position satisfies the above condition is specified.

  The distance d1 can be set to an upper limit of a distance at which the three-dimensional scanner 110 can measure a three-dimensional shape with a certain accuracy or higher. This upper limit value can be said to be the upper limit value of the distance at which the depth sensor 111 can measure the depth (distance) with a certain accuracy or higher. The distance d1 and the conditions such as how far away from the work object the measurer goes when going around the work object for measurement, at what intervals depth measurement and imaging with a spherical camera are performed. , The threshold value TH1 may be determined.

  On the other hand, the threshold value TH2 may be determined based on how far an object from the position of the work target (for example, the center of gravity) is detected as the work target. This depends on, for example, what distance an object located from the work target affects the work when the work is performed on the work target.

  In consideration of such points, in steps S35 and S36, it is quasi-determined whether or not the distance of the corresponding point from the position of the work target is d1 or more and d2 or less (however, d1 <d2). Can be considered.

  [Step S39] The intermediate background object data generation unit 324 generates an intermediate background object polygon based on the three-dimensional coordinates of each corresponding point stored in the RAM 312 in step S37, that is, based on the point group data of the intermediate background object.

  [Step S40] The intermediate background object data generation unit 324 uses the omnidirectional image stored in the image data storage unit 340 to determine a texture image to be attached to each generated polygon. This determination processing can be executed by executing the processing of steps S21 to S25 in FIG. 10 for each polygon. The intermediate background object data generation unit 324 stores the polygon data with the texture image obtained by adding the data of the texture image to the data indicating each polygon in the three-dimensional data storage unit 330 as the intermediate background object data 332.

Next, the generation processing of the outer circumference data 333 by the outer circumference data generation unit 325 will be described with reference to FIGS.
FIG. 14 is a diagram illustrating an example of a three-dimensional shape of the outer periphery. As shown in FIG. 14, the outer circumference data generation unit 325 generates a spherical three-dimensional model having a predetermined radius R centered on a predetermined reference point 331d on the three-dimensional model 331c of the work target in the virtual space. Then, the inner surface is defined as an outer periphery 333a. The reference point 331d is, for example, the center of gravity of the three-dimensional model 331c of the work target (more precisely, a three-dimensional model based on the target data 331), or the region where the three-dimensional model 331c is in contact with the ground surface. Set as center.

  The radius R is set so that the outer circumference 333a is separated from the outer edge of the three-dimensional model 331c of the work target by a predetermined distance (for example, 50 m). The radius R is set so as to be larger by a predetermined value than the upper limit value of the distance at which the intermediate background object is extracted from the reference point 331d (that is, the distance d2 described with reference to FIG. 13).

  The outer circumference data generation unit 325 generates a polygon based on the three-dimensional model of the outer circumference 333a, and generates a texture image to be attached to each polygon using the omnidirectional image stored in the image data storage unit 340. The outer periphery data generation unit 325 stores the polygon data with the texture image for the outer periphery thus generated in the three-dimensional data storage unit 330 as the outer periphery data 333.

FIG. 15 is an example of a flowchart showing the procedure for generating the outer circumference data.
[Step S51] The outer circumference data generation unit 325 generates a spherical three-dimensional model having a predetermined radius centered on a reference point on the three-dimensional model of the work target in the virtual space. The outer circumference data generation unit 325 generates a spherical inner surface polygon (that is, an outer circumference polygon) based on the generated point group data of the three-dimensional model.

  [Step S52] The outer circumference data generation unit 325 aligns the directions of the spherical images stored in the image data storage unit 340 in the same procedure as in step S31 of FIG. In this process, each omnidirectional image is converted so that the imaging position (camera position) of the camera that captured each omnidirectional image is the same as the imaging direction (azimuth).

[Step S53] The outer circumference data generation unit 325 determines a texture image to be pasted on each polygon on the outer circumference using the omnidirectional image in which the directions are aligned.
In this process, the outer periphery data generation unit 325 specifies the camera position and the camera direction of the omnidirectional image (the omnidirectional image whose direction has not been changed) as a reference when the directions are aligned in step S52. The camera directions represent the camera directions of the omnidirectional image after the directions are aligned. The outer circumference data generation unit 325 specifies an image area corresponding to each polygon in the omnidirectional image for each omnidirectional image based on the specified camera position and camera direction. The outer periphery data generation unit 325 calculates, for each polygon, a logical sum (OR) of the data of the image area specified from each celestial sphere image, and determines the calculated image as a texture image corresponding to the polygon. I do.

  By the processing of step S16 in FIG. 9 and step S40 in FIG. 13, the image portions corresponding to the work object and the intermediate background object have been deleted from the omnidirectional image. Therefore, in step S53, the texture image can be easily formed using only the image in which the object other than these is captured, without extracting and deleting the portion in which the work object and the intermediate background object are captured from the celestial sphere image. Can decide.

The outer periphery data generation unit 325 stores the polygon data with the texture image for the outer periphery thus generated in the three-dimensional data storage unit 330 as the outer periphery data 333.
Here, an example of an image of a virtual space drawn using the target object data 331, the intermediate background object data 332, and the outer periphery data 333 generated by the above processing is shown in FIG.

  FIG. 16 is a diagram illustrating an example of an image in which a virtual space is drawn. As shown in FIG. 16, a three-dimensional model 131 a indicating the object 131 as the work target is drawn based on the target data 331 in the virtual space. Further, based on the intermediate background object data 332, three-dimensional models 132a and 133a representing the objects 132 and 133, which are intermediate background objects, respectively, are drawn. The three-dimensional models 131a, 132a, and 133a are displayed with a texture image. On the other hand, the images 134a to 136a respectively representing the objects 134 to 136 far from the work target are drawn in a state where they are pasted as images on the outer periphery 333b (that is, the inner surface of the sphere).

  Since the three-dimensional models 131a, 132a, and 133a are rendered based on the three-dimensional data, the three-dimensional models 131a, 132a, and 133a are displayed with the position and direction of the viewpoint, that is, the position and angle according to the position and direction of the viewer. Therefore, the viewer can relatively accurately grasp the positional relationship between the three-dimensional model 131a and the three-dimensional models 132a and 133a, that is, the positional relationship between the work object and the intermediate background object in the virtual space. The object drawn on the outer periphery 333b is also displayed at a position and an angle corresponding to the position and direction of the viewpoint. However, the accuracy of the positional relationship between the work object and the image of the object on the outer periphery 333b is lower than the positional relationship between the work object and the intermediate background object. Therefore, the drawing of the outer periphery 333b is performed for the purpose of reproducing the atmosphere of the work site by simply drawing the situation far from the work target by a light load process without calculating the distance.

  Here, when the information processing system according to the present embodiment is used for training or education of on-site work, not only the work target to be directly worked but also the position and shape of the surrounding objects are grasped. It may be preferable to be able to do so. For example, in the case of operating a crane to lift and lower a load (work target), in order to prevent contact with objects other than the load within the operating range of the crane, the position and shape of the load can only be recognized. Is not enough. In this operation, it is preferable that the situation around the place where the baggage is lifted and lowered can be recognized as three-dimensionally as possible.

  As a method for achieving such an object, a method is conceivable in which a spatial recognition sensor 101 such as the three-dimensional scanner 110 having a large upper limit value of a distance at which a three-dimensional shape can be measured with a certain accuracy or more is used. However, this method requires a high-performance and expensive device for measurement, and increases the cost. On the other hand, in the method of drawing all the surrounding conditions on the outer periphery 333b, the reproducibility of the positional relationship between the work target and the surrounding objects is low when the viewer moves, and the usefulness in the training and education of the field work is low. Will drop.

  In order to solve such a problem, according to the above-described processing of the server 300, the three-dimensional image of the intermediate background existing around the work target is obtained by using the image captured using the camera 102 such as the spherical camera 120. The original shape is measured. Accordingly, a relatively low-cost device capable of measuring a three-dimensional shape only in a range including at least the work target as the space recognition sensor 101 can three-dimensionally recognize the situation around the work target. The measurement accuracy of the intermediate background object may be lower than the measurement accuracy of the work target measured using the dedicated space recognition sensor 101. However, since the intermediate background object is not an object existing in the virtual space in the immediate vicinity of the viewer corresponding to the worker, the three-dimensional shape and position need not be reproduced as accurately as the work object. That is, by the processing of the server 300, the viewer can three-dimensionally recognize the positional relationship between the work target and the surrounding objects within a range in which usefulness for work training and education is sufficiently obtained. Become.

  On the other hand, an object farther from the work target than the intermediate background object is simply drawn on the outer periphery 333b. This makes it possible for the viewer to recognize a situation far from the work site while reducing the processing load when calculating the three-dimensional data and drawing the three-dimensional model, thereby enhancing the reality of the image in the virtual space.

  Note that the object drawn using the target object data 331 may include not only an object to be worked but also equipment used for the work. For example, in the case of work using a crane, the three-dimensional shape of the operation panel and operation lever of the crane and the operation room where these are installed are converted into data as object data 331, and drawn using the object data 331. May be done.

Next, the position correction processing of the intermediate background object by the intermediate background object position correction unit 326 will be described with reference to FIGS.
FIG. 17 is a diagram illustrating an example of a conversion graph used in the position correction processing of the intermediate background object. The radius of the outer circumference can be set shorter than the upper limit distance at which the intermediate background object is detected from the position of the work target. In this case, the intermediate background object position correction unit 326 converts the distance from the work target in the virtual space to each feature point of the intermediate background object within the radius of the outer periphery.

  FIG. 17 is an example of a conversion graph used in such conversion. According to the graph in FIG. 17, the distance from 0 m to 1000 m in the real space from the reference point of the work target (the position corresponding to the reference point 331 d in FIG. 14) to each feature point of the intermediate background object is 0 m in the virtual space. Converted to a distance of ~ 50m. At this time, instead of linearly converting the distance in the real space to the distance in the virtual space, the longer the distance in the real space, the lower the conversion magnification as in the example of FIG. You can look natural.

FIG. 18 is an example of a flowchart showing the processing procedure for correcting the position of the intermediate background object.
[Step S61] When a plurality of intermediate background objects are detected, the intermediate background object position correction unit 326 selects one of the intermediate background objects.

[Step S62] The intermediate background object position correction unit 326 selects one vertex (feature point) of a polygon in the selected intermediate background object based on the intermediate background object data 332.
[Step S63] The intermediate background object position correction unit 326 calculates the distance between the selected vertex and the reference point of the work target. The reference point 331d shown in FIG. 14 can be used as the reference point.

  [Step S64] The intermediate background object position correction unit 326 converts the calculated distance according to the graph shown in FIG. The intermediate background object position correction unit 326 converts the coordinates of the selected vertex in the virtual space according to the converted distance. That is, in this conversion, the coordinates of the selected vertex are converted so that the distance between the selected vertex and the reference point of the work target becomes the converted distance.

  [Step S65] The intermediate background object position correction unit 326 determines whether there is an unselected vertex in step S62 among the vertices of the polygon in the selected intermediate background object. If there is an unselected vertex, the intermediate background object position correction unit 326 advances the process to step S62, and selects one unselected vertex. On the other hand, when all the vertices have been selected, the intermediate background object position correction unit 326 executes the process of step S66.

  [Step S66] The intermediate background object position correction unit 326 determines whether there is an unselected intermediate background object in step S61. If there is an unselected intermediate background object, the intermediate background object position correction unit 326 advances the process to step S61, and selects one unselected intermediate background object. On the other hand, when all the intermediate background objects have been selected, the intermediate background object position correction unit 326 ends the process.

Next, a three-dimensional shape complementing process by the complementing processing unit 327 will be described with reference to FIGS.
FIG. 19 is a diagram for explaining a complementary region of a three-dimensional shape in the virtual space. FIG. 19A shows a plan view of the virtual space, and FIG. 19B shows a side view of the virtual space.

  By the processing so far, the target object data 331, the intermediate background object data 332, and the outer periphery data 333 have been generated. Thus, the work object, the intermediate background object, and the outer periphery can be drawn in the virtual space. In the example of FIG. 19, the three-dimensional model 131a of the work target, the three-dimensional models 132a and 133a of the intermediate background, and the outer periphery 333b are drawn.

  However, in this state, there is an area (data insufficient area) between the work object and the intermediate background object or between the intermediate background object and the outer periphery where nothing is drawn because the 3D model is not generated. obtain. In the example of FIG. 19, the data deficient area 334a is shaded, and nothing is drawn in the data deficient area 334a. Therefore, the complementing processing unit 327 simply generates the above-described three-dimensional model of the data-deficient region 334a using the object data 331, the intermediate background object data 332, and the outer periphery data 333 which have been generated at present.

  FIG. 20 is a diagram illustrating a first example of the three-dimensional shape complementing process. FIG. 20 shows an example of a side view of the virtual space. A data deficient area 334b exists between the three-dimensional model 131a of the work object and the three-dimensional model 132a of the intermediate background object, and a data deficient area 334c exists between the three-dimensional model 132a of the intermediate background object and the outer periphery 333b. doing.

  The complementing processing unit 327 draws a perpendicular L1 passing through the outer end 131b of the three-dimensional model 131a of the work target (outside the reference point of the three-dimensional model 131a), and 45 ° from the end 131b with respect to the perpendicular L1. A straight line L2 inclined outward is drawn. Further, the complementing processing unit 327 draws a perpendicular L3 passing through the inner end 132c in the three-dimensional model 132a of the intermediate background object, and draws a straight line L4 inclined 45 ° inward from the end 132c with respect to the perpendicular L3.

  Then, the complementing processing unit 327 generates a curve L5 that smoothly connects the end 131b and the end 132c so as to curve toward the intersection P1 in a region sandwiched between the straight line L2 and the straight line L4. The complement processing unit 327 generates a three-dimensional model indicating a curved surface passing through the curve L5. Thereby, the three-dimensional shape of the data-deficient area 334b is complemented.

  The same supplementary processing is performed on the data-deficient area 334c. The complementing processing unit 327 draws a perpendicular L6 passing through the outer end 132d in the three-dimensional model 132a of the intermediate background object, and draws a straight line L7 inclined 45 ° outward with respect to the perpendicular L6 from the end 132d. Further, the complementing processing unit 327 draws a perpendicular line L8 passing through the outermost end 333c (the intersection of the plane passing through the reference point and the outer periphery) of the outer periphery 333b, and extends 45 ° inward from the end 333c with respect to the perpendicular L8. Draw an inclined straight line L9.

  Then, the complementing processing unit 327 generates a curve L10 that smoothly connects the end 132d and the end 333c so as to be curved toward the intersection P2 in an area sandwiched between the straight lines L7 and L9. The complement processing unit 327 generates a three-dimensional model indicating a curved surface passing through the curve L10. Thereby, the three-dimensional shape of the data-deficient area 334c is complemented.

  The complement processing unit 327 further generates a texture image to be pasted on a three-dimensional model of a curved surface passing through the curves L5 and L10. The texture image is generated, for example, as a gradation image that gradually changes from the former color to the latter color from one end to the other end using the colors at both ends of the curve. For example, a gradation image that gradually changes from the color of the end 131b to the color of the end 132c is attached to the three-dimensional model of the curved surface passing through the curve L5 from the end 131b to the end 132c. . Also, a gradation image that gradually changes from the color of the end 132d to the color of the end 333c is attached to the three-dimensional model of the curved surface passing through the curve L10 from the end 132d to the end 333c. .

The polygon data with the texture image indicating the three-dimensional model of the data-deficient area generated by the above processing is stored in the three-dimensional data storage unit 330 as the supplementary data 334.
FIG. 21 is a diagram illustrating a second example of the three-dimensional shape complementing process. FIG. 21 illustrates an example of a side view of the virtual space. As in FIG. 20, there is a data deficient area 334b between the three-dimensional model 131a of the work object and the three-dimensional model 132a of the intermediate background object, and between the three-dimensional model 132a of the intermediate background object and the outer periphery 333b. A data shortage area 334c exists.

  For an area including a work site, there may be map data including latitude / longitude data, elevation data, and aerial photographs. In this case, the three-dimensional shape of the data-deficient area can be complemented using the map data, and a texture image of the complemented area can be generated.

  The complement processing unit 327 generates a polygon of the data-deficient area 334b using the latitude / longitude data and the elevation data corresponding to the data-deficient area 334b. Thereby, the three-dimensional shape of the data-deficient area 334c is complemented. At this time, the complementing processing unit 327 corrects the distance from the reference point of the work target to the position of each part on the map by a method using the conversion graph shown in FIG. A curve L11 shown in FIG. 21 shows a cross section of the polygon thus generated. Further, the complement processing unit 327 determines an image of a portion corresponding to the polygon in the aerial photograph as a texture image to be pasted on the generated polygon.

  For the data deficient area 334c, a polygon and a corresponding texture image are generated in the same procedure as described above. A curve L12 shown in FIG. 21 shows a cross section of the generated polygon. The complementing processing unit 327 stores the polygon data with the texture image indicating the three-dimensional model of the data-deficient area generated in this manner in the three-dimensional data storage unit 330 as the supplementary data 334.

FIG. 22 is an example of a flowchart showing the procedure of a three-dimensional shape complementing process.
[Step S71] The complementing processing unit 327 determines whether there is map data for an area including the work site. When there is no map data, the complement processing unit 327 executes the process of step S72, and when there is map data, it executes the process of step S73.

  [Step S72] The complementing processing unit 327 supplements the three-dimensional shape of the data-deficient region using the existing polygon data with texture image, that is, the object data 331, the intermediate background object data 332, and the outer periphery data 333. Further, the complement processing unit 327 generates a texture image corresponding to the complemented three-dimensional polygon by the gradation process. The complement processing unit 327 stores the generated polygon data with the texture image indicating the three-dimensional model of the data-deficient area in the three-dimensional data storage unit 330 as the supplement data 334.

  [Step S73] The complementing processing unit 327 complements the three-dimensional shape of the data-deficient area using the latitude / longitude data and the elevation data included in the map data. Further, the complement processing unit 327 generates a texture image corresponding to the complemented three-dimensional polygon using the aerial photograph included in the map data. The complement processing unit 327 stores the generated polygon data with the texture image indicating the three-dimensional model of the data-deficient area in the three-dimensional data storage unit 330 as the supplement data 334.

  By using the object data 331, the intermediate background object data 332, the outer circumference data 333, and the supplementary data 334 generated by the processing of the server 300 described above, the environment of the entire space that can be visually recognized by the viewer in the office 200 becomes a virtual space. Reproduced above. With the object data 331 and the intermediate background object data 332, the work object and the intermediate background object are three-dimensionally drawn, and their positional relationship is accurately drawn. Further, using the outer periphery data 333, the situation of the distant view is drawn by simple processing, and the reality of the image is enhanced. Further, the image of the area other than the work object, the intermediate background object, and the outer periphery is simply drawn by the complementary data 334, and an unnatural area where nothing is drawn can be eliminated.

  Therefore, when such drawing of the virtual space is used for training or education of on-site work, by drawing the entire space, the discomfort of the viewer can be reduced. In addition, only objects important to the viewer are drawn in a three-dimensional shape, and other objects and scenery are simply drawn. Thus, it is possible to allow the viewer to browse images useful for training and education on site work while preventing the processing load for drawing from increasing.

  Note that the processing functions of the devices (for example, the data generation device 1 and the server 300) described in each of the above embodiments can be realized by a computer. In this case, a program describing the processing content of the function that each device should have is provided, and the processing function is realized on the computer by executing the program on the computer. The program describing the processing content can be recorded on a computer-readable recording medium. Computer-readable recording media include magnetic storage devices, optical disks, magneto-optical recording media, and semiconductor memories. The magnetic storage device includes a hard disk device (HDD), a magnetic tape, and the like. Optical disks include CDs (Compact Discs), DVDs (Digital Versatile Discs), and Blu-ray Discs (Blu-ray Discs: BD, registered trademark). Examples of the magneto-optical recording medium include an MO (Magneto-Optical disk).

  When distributing the program, for example, portable recording media such as DVDs and CDs on which the program is recorded are sold. Alternatively, the program may be stored in a storage device of a server computer, and the program may be transferred from the server computer to another computer via a network.

  The computer that executes the program stores, for example, the program recorded on the portable recording medium or the program transferred from the server computer in its own storage device. Then, the computer reads the program from its own storage device and executes processing according to the program. Note that the computer can also read the program directly from the portable recording medium and execute processing according to the program. Further, the computer can also execute processing according to the received program each time the program is transferred from a server computer connected via a network.

Reference Signs List 1 data generation device 1a storage unit 1b processing unit 2a, 2b captured image 3a first subject 3b second subject 4a first three-dimensional data 4b second three-dimensional data 5 camera S1, S2 step

Claims (13)

Computer
Acquiring a plurality of captured images captured from a plurality of positions and first three-dimensional data indicating a three-dimensional position of each point in a first subject included in the plurality of captured images;
Based on the plurality of captured images, each point in the second subject whose distance from the first subject is equal to or less than a predetermined threshold value in the same coordinate space as the three-dimensional position indicated by the first three-dimensional data. Generating second three-dimensional data indicating a three-dimensional position;
Data generation method. The first three-dimensional data is calculated based on a measurement result obtained by measuring a distance from a camera that has captured the plurality of captured images using a distance sensor,
The data generation method according to claim 1. In the generation of the second three-dimensional data, a plurality of first corresponding points indicating the same feature point are detected among two or more of the plurality of captured images, and the detected first corresponding point is detected. Among the corresponding points, the second three-dimensional data is generated using a second corresponding point in which a difference between the positions of the first corresponding points in each of the two or more captured images is equal to or less than a predetermined value.
The data generation method according to claim 1. In the generation of the second three-dimensional data, among the second corresponding points, a third corresponding point whose distance from a camera that has captured the plurality of captured images is equal to or less than a predetermined threshold distance is defined as the first corresponding point. Generating the second three-dimensional data by using the distance from the subject as a point that is equal to or less than the threshold.
The data generation method according to claim 3. Said computer,
A first region corresponding to the first subject is extracted from each of the plurality of captured images, and a first three-dimensional model represented by the first three-dimensional data is extracted using the image of the first region. Generate a texture image,
A second region corresponding to the second subject is extracted from each of the captured images in which the image of the first region has been deleted from each of the plurality of captured images, and an image of the second region is used. Generating a texture image of a second three-dimensional model indicated by the second three-dimensional data.
The data generation method according to claim 1, further comprising a process. Said computer,
In the coordinate space, generating a third three-dimensional model showing a sphere centered on the position of the first three-dimensional model,
A texture image to be pasted on the inner surface of the third three-dimensional model is generated using a captured image in which the image of the first region and the image of the second region have been deleted from each of the plurality of captured images. Do
The data generation method according to claim 5. Said computer,
In the coordinate space, generating a second three-dimensional model indicating a sphere centered on the position of the first three-dimensional model indicated by the first three-dimensional data,
Based on the plurality of captured images, generate a texture image to be pasted on the inner surface of the second three-dimensional model,
The data generation method according to claim 1, further comprising a process. Said computer,
In the coordinate space, the shape of a space other than the space in which the first three-dimensional model, the second three-dimensional model, and the third three-dimensional model indicated by the second three-dimensional data exist is defined. Complementing based on the first three-dimensional data, the second three-dimensional data, and third three-dimensional data indicating the second three-dimensional model,
The data generation method according to claim 7, further comprising a process. Computer
Obtain a plurality of captured images captured from a plurality of positions, and position information of each part on any subject included in the plurality of captured images,
Based on the acquired position information, generate a first display model according to the three-dimensional shape of any of the objects,
Based on the acquired plurality of captured images, a second display model corresponding to a three-dimensional shape of another subject included in the plurality of captured images and having a distance from any one of the subjects equal to or less than a threshold is generated. ,
Generate display data including the generated first display model, the generated second display model, and positional relationship information indicating a positional relationship between the first display model and the second display model. ,
Data generation method. A storage unit,
A plurality of captured images captured from a plurality of positions, and first three-dimensional data indicating a three-dimensional position of each point in a first subject included in the plurality of captured images are obtained, and And storing, based on the plurality of captured images, the same three-dimensional position as that indicated by the first three-dimensional data of each point in the second subject whose distance from the first subject is equal to or less than a predetermined threshold. A processing unit that generates second three-dimensional data indicating a three-dimensional position in a coordinate space and stores the generated two-dimensional data in the storage unit;
A data generation device having: Obtain a plurality of captured images captured from a plurality of positions, and position information of each part on any subject included in the plurality of captured images,
Based on the acquired position information, generate a first display model according to the three-dimensional shape of any of the objects,
Based on the acquired plurality of captured images, a second display model corresponding to a three-dimensional shape of another subject included in the plurality of captured images and having a distance from any one of the subjects equal to or less than a threshold is generated. ,
Generate display data including the generated first display model, the generated second display model, and positional relationship information indicating a positional relationship between the first display model and the second display model. ,
A data generation device having a processing unit. On the computer,
Acquiring a plurality of captured images captured from a plurality of positions and first three-dimensional data indicating a three-dimensional position of each point in a first subject included in the plurality of captured images;
Based on the plurality of captured images, each point in the second subject whose distance from the first subject is equal to or less than a predetermined threshold value in the same coordinate space as the three-dimensional position indicated by the first three-dimensional data. Generating second three-dimensional data indicating a three-dimensional position;
Data generation program to execute the process. On the computer,
Obtain a plurality of captured images captured from a plurality of positions, and position information of each part on any subject included in the plurality of captured images,
Based on the acquired position information, generate a first display model according to the three-dimensional shape of any of the objects,
Based on the acquired plurality of captured images, a second display model corresponding to a three-dimensional shape of another subject included in the plurality of captured images and having a distance from any one of the subjects equal to or less than a threshold is generated. ,
Generate display data including the generated first display model, the generated second display model, and positional relationship information indicating a positional relationship between the first display model and the second display model. ,
Data generation program to execute the process.

Images