JP7564737B2 - Photographing device and photographing method - Google Patents

Description

Embodiments of the present invention relate to an imaging processing device and an imaging processing method.

At power plants, inspectors regularly patrol the equipment to perform maintenance inspections on a huge amount of equipment to ensure the plant operates safely. During their patrols, the inspectors also visually check for equipment abnormalities. Traditional maintenance inspections are generally done by handwriting the results of the inspectors' inspections on-site and creating inspection records in the office. As such, much of the work is done manually, and efforts are being made to reduce the labor required for this work. Many of these labor-saving efforts are services that use mobile devices. For example, by presenting inspection instructions to inspectors and inputting the values of meters and other items that the inspectors have visually checked into a mobile device, handwriting is eliminated and inspection results are digitized.

A system is also under consideration that would record footage from cameras held by inspectors on patrol, like a dashcam, to record any equipment abnormalities and allow the footage to be reviewed at a later date if necessary. Because the footage recorded is long, it is desirable to review footage from a specific shooting location or only footage that may show abnormalities. There are also plans to use fisheye lenses on cameras to record and review spherical images of a wide area at once.

JP 2019-045163 A

However, while GPS is commonly used as a method of determining location outdoors, it is difficult to use GPS indoors, such as in power plants, where radio waves cannot reach, making it even more difficult to identify the shooting location. In addition, spherical images are curved, and processing using spherical images may reduce the accuracy of identifying the shooting location.

The embodiment of the present invention has been made in consideration of these circumstances, and provides an image capture processing device and an image capture processing method that can acquire the image capture position at which an image was captured.

The image capture processing device according to this embodiment includes a conversion unit, a feature point generation unit, and an image capture position estimation unit. The conversion unit converts a spherical image of an object in real space into a flat elemental image based on a first reference point. The feature point generation unit generates first position coordinates for each of a plurality of first feature points in the object obtained based on the elemental images. The image capture position estimation unit estimates the image capture position of the spherical image using the first position coordinates.

According to this embodiment, it is possible to obtain the shooting position where the captured image was taken.

1 is a block diagram showing a configuration of an imaging processing system 1 according to a first embodiment. 1 is a block diagram showing a configuration of an image capturing and processing device 2 according to a first embodiment. FIG. 2 is a diagram showing a movable range as two-dimensional map information. FIG. 4 is a diagram illustrating an example of processing by an image conversion unit. A conceptual diagram of converting a spherical panoramic image into a flat image. FIG. 2 is a diagram showing a spherical panoramic image and each element image. FIG. 4 is a diagram showing a schematic relationship between pre-registered three-dimensional coordinate points and shooting positions. FIG. 13 is a schematic diagram showing a front image and an instrument image with reduced distortion. 1 is a schematic diagram showing edge intersections, which are feature points in a device image. FIG. 4 is a diagram showing an example of position estimation by a shooting position estimation unit. FIG. 2 is a block diagram showing an example of the configuration of a monitoring device. FIG. 2 is a diagram illustrating map information generated by a map information generating unit. 11 is a flowchart showing an example of processing by the image capture processing device. 10 is a flowchart showing an example of processing by an image capture processing device according to a modified example of the first embodiment. FIG. 11 is a block diagram showing the configuration of an image capturing and processing device according to a second embodiment. FIG. 2 is a diagram showing a schematic diagram of a traveling direction and obstacles. FIG. 13 is a block diagram showing the configuration of an image capturing and processing device according to a third embodiment. FIG. 4 is a detailed block diagram of a three-dimensional image processing unit. FIG. 11 is a diagram illustrating an example of processing by a small region comparison unit. FIG. 4 is a diagram illustrating an example of processing by a three-dimensional coordinate generation unit. 4A and 4B are diagrams showing examples of voxel data generated by a three-dimensional image reconstruction processing unit. 11A and 11B are diagrams showing examples of voxel data and differential processing results. FIG. 13 is a diagram showing an example in which difference information is reprojected onto each element image. FIG. 13 is a block diagram showing the configuration of an image capturing processing device according to a fourth embodiment. FIG. 4 is a detailed block diagram of a two-dimensional image processing unit.

Hereinafter, an image capture processing device and an image capture processing method according to an embodiment of the present invention will be described in detail with reference to the drawings. Note that the embodiment shown below is an example of an embodiment of the present invention, and the present invention is not limited to these embodiments. In addition, in the drawings referred to in this embodiment, the same parts or parts having similar functions are given the same or similar symbols, and repeated explanations may be omitted. In addition, the dimensional ratios of the drawings may differ from the actual ratios for the convenience of explanation, and some components may be omitted from the drawings.
First Embodiment

Figure 1 is a block diagram showing the configuration of an image capture processing system 1 according to the first embodiment. As shown in Figure 1, the image capture processing system 1 includes an image capture processing device 2 and a monitoring device 4. The image capture processing device 2 is a device capable of identifying the image capture position, and transmits the image capture position and image information related to the image capture position to the monitoring device 4, for example, via wireless communication. On the other hand, the monitoring device 4 is a device that monitors indoor equipment such as a power generation plant based on information supplied from the image capture processing device 2. The image capture processing device 2 and the monitoring device 4 can communicate information with each other via wireless communication via a repeater or the like placed in a position free of radio interference. Note that the image capture processing device 2 and the monitoring device 4 according to this embodiment are configured as separate entities, but are not limited to this. For example, the image capture processing device 2 and the monitoring device 4 may be configured as an integrated device to form a portable image capture processing device 2.

Fig. 2 is a block diagram showing the configuration of the image capture device 2 according to the first embodiment. As shown in Fig. 2, the image capture device 2 is capable of identifying the image capture position, and is configured with an image capture unit 10, a motion sensor 15, a storage unit 20, a communication unit 25, and an information processing unit 30. Note that the image capture device 2 may be configured without including the motion sensor 15.

The photographing unit 10 is, for example, an omnidirectional camera (360-degree camera), and can generate a omnidirectional panoramic image as a continuous spherical image, that is, a video of the spherical image. More specifically, the photographing unit 10 photographs a omnidirectional panoramic image including an image of an object inside a power plant. The photographing unit 10 is set, for example, in a moving unit equipped with a drive mechanism, and automatically photographs a omnidirectional panoramic image according to the movement of the moving unit. Furthermore, the photographing unit 10 supplies the omnidirectional panoramic image to the storage unit 20 and the information processing unit 30. In this embodiment, the photographing processing device 2 is set in a moving unit equipped with a drive mechanism, but is not limited to this. For example, the photographing unit 10 and the motion sensor 15 may be set in a helmet of an observer such as a worker. In this case, the storage unit 20 and the information processing unit 30 may be configured, for example, in the monitoring device 4, and the photographing unit 10, the motion sensor 15, the storage unit 20, and the information processing unit 30 may communicate information by wireless communication.

The motion sensor 15 is, for example, a nine-axis sensor that combines an inertial sensor (a three-axis acceleration sensor and a three-axis angular velocity sensor) with a three-axis geomagnetic sensor. This motion sensor 15 is mounted on the image capture unit 10 and detects the acceleration that occurs when the image capture unit 10 moves. This motion sensor 15 can also detect gravitational acceleration. Furthermore, the motion sensor 15 detects angular velocity that occurs when the attitude of the image capture unit 10 changes. It is also possible to grasp the attitude of the image capture unit 10 from geomagnetism. The acceleration values and angular velocity values detected by the motion sensor 15 are supplied to the information processing unit 30.

The storage unit 20 is realized by, for example, a semiconductor memory element such as a RAM (Random Access Memory) or a flash memory, a hard disk, etc. This storage unit 20 has a feature point recording unit 200, an image recording unit 202, and a shooting position recording unit 204. The feature point recording unit 200 records a feature point group, and primary feature amount and position information are recorded for each feature point. The feature point recording unit 200 may also record point cloud data indicating the three-dimensional position of each structure of the power plant measured by a 3D laser scanner or the like. The storage unit 20 also stores various processing programs executed by the information processing unit 30.

The coordinate system of the primary feature points recorded in the feature point recording unit 200 has a scale adjusted in advance, and has a scale in millimeters or meters. The origin and orientation may also be adjusted to match drawings or CAD data of the power plant, etc.

Figure 3 is a diagram showing a schematic representation of the range in which the imaging and processing device 2 can move within a site, such as a power plant building, as two-dimensional map information 8. As described above, the feature point recording unit 200 records a group of primary feature points for each imaging position in the map information 8 as primary feature amounts. Details of the primary feature amounts will be described later with reference to Figure 7.

As shown in FIG. 2 again, the video recording unit 202 records the spherical panoramic image captured by the imaging unit 10, and each of the front, rear, left, right, top, and bottom element images described below.
In the photographing position recording unit 204, the photographing position of the photographing unit 10 processed by the information processing unit 30 is recorded in association with the spherical panoramic image, each element image, and the photographing time.

The communication unit 25 is capable of wireless communication with the monitoring device 4 and the like.
The information processing unit 30 can generate the shooting position of the shooting unit 10 using the omnidirectional panoramic image, and is configured to include, for example, a CPU (Central Processing Unit). That is, the information processing unit 30 is configured as a computer in which the CPU executes various programs stored in the storage unit 20, thereby realizing information processing by software using hardware resources. As a result, the information processing unit 30 is configured with a posture information acquisition unit 300, an image conversion unit 302, a feature point generation unit 304, a feature point comparison unit 306, a shooting position estimation unit 308, a verification unit 310, and a control unit 312. Note that the posture information acquisition unit 300, the image conversion unit 302, the feature point generation unit 304, the feature point comparison unit 306, the shooting position estimation unit 308, the verification unit 310, and the control unit 312 may be configured as a circuit.

The orientation information acquisition unit 300 generates the shooting direction and orientation of the image capture unit 10 when capturing a celestial sphere panoramic image. That is, the orientation information acquisition unit 300 generates the shooting direction when capturing a celestial sphere panoramic image by combining the angular velocity value and acceleration value supplied from the motion sensor 15 to identify the orientation of the image capture unit 10. For example, the orientation information acquisition unit 300 identifies the shooting start angle in the world coordinate system of the celestial sphere panoramic image.

Figure 4 is a diagram showing a schematic example of processing by the image conversion unit 302. The left diagram shows a spherical panoramic image P600, and the right diagram shows a flat image cube-mapped into each of the front, back, left, right, top, and bottom element images G402.

As shown in FIG. 4, the image conversion unit 302 performs planar expansion on each element image based on the information from the orientation information acquisition unit 300 so that the center m30 of the previous image coincides with the true north direction. The image conversion unit 302 converts, for example, an equirectangular omnidirectional panoramic image into a cube map format. As a result, the 360° omnidirectional panoramic image is expanded into a cube map. Furthermore, the image conversion unit 302 associates the omnidirectional panoramic image P600 with each element image G402 and records them in the shooting position recording unit 204.

FIG. 5 is a conceptual diagram of a case where an equirectangular omnidirectional panoramic image P600 is converted into a cube map flat image. The omnidirectional panoramic image P600 is, for example, an image captured on a spherical surface p0 of radius r. For this reason, each pixel S1 of the omnidirectional panoramic image P600 can be represented by polar coordinates. As a result, since the arrangement relationship of each element image q0 (front, back, left, right, top, and bottom) and the polar coordinates of each pixel S1 are known, each pixel S1 can be projected onto each element image G402 as each pixel S2. Note that point S3 indicates an example of a target point of pixel S2 in real space.

FIG. 6 is a diagram showing an actually captured omnidirectional panoramic image P600 and each of the element images G402 of the front, back, left, right, top, and bottom. The left figure shows the omnidirectional panoramic image P600, and the right figure shows a flat image cube-mapped into each of the element images G402 of the front, back, left, right, top, and bottom. As shown in FIG. 6, the processing of the image conversion unit 302 generates each of the element images G402 of the front, back, left, right, top, and bottom, which have less distortion than the omnidirectional panoramic image P600. Here, a case where the omnidirectional panoramic image is projected onto each plane of a hexahedron is described, but the plane onto which the image is projected can also be a polyhedron having six or more planes. Also, instead of the omnidirectional panoramic image, a hemispherical panoramic image can also be used. In this case, the same processing can be performed by projecting each of the element images of the front, left, right, top, and bottom instead of projecting onto a hexahedron.

FIG. 7 is a diagram showing a schematic relationship between the three-dimensional coordinate points of primary feature points in a power plant building that have been registered in advance and the shooting positions of a celestial sphere panoramic image P600. The left diagram shows multiple celestial sphere panoramic images p600 captured along a travel route L600, and the right diagram is a feature point diagram G600 showing the shooting positions of the multiple celestial sphere panoramic images p600 and the three-dimensional coordinate points of the primary feature points. As described above, the three-dimensional coordinate points of the primary feature points are generated from point cloud data that indicates the three-dimensional positions of each structure of the power plant measured by a 3D laser scanner or the like. Primary feature points are, for example, the intersections of edges of peripheral equipment.

In the feature point recording unit 200, as described above, a group of two-dimensional coordinates of the feature points generated in association with the two-dimensional map information 8 (see FIG. 2) is recorded as a primary feature. For example, the feature points shown in the feature point map G600 are projected onto each of the element images, front, back, left, right, top, and bottom, with the reference direction being due north, and a group of two-dimensional coordinates of each feature point for each element image is generated as a primary feature. For example, by a prior computer simulation, each element image surface, front, back, left, right, top, and bottom, with the reference direction being due north, is virtually generated for each plane coordinate (x, y) in the feature point map G600, the feature points shown in the feature point map G600 are projected onto each element image surface, and a group of two-dimensional coordinates of each feature point for each element image surface is generated as a primary feature. These primary feature amounts are associated with plane coordinates (x, y) (see FIG. 3) and recorded in the feature point recording unit 200. In addition, the two-dimensional coordinate group of the primary features is associated with the three-dimensional coordinates in real space shown in the feature point map G600 (see FIG. 7).

Figure 8 is a schematic diagram showing a previous image 700q and an equipment image 702q in which distortion has been reduced by processing of the image conversion unit 302. Figure 9 is a schematic diagram showing an edge intersection 800, which is a feature point in the equipment image 702q (see Figure 8).

As shown in FIG. 9, the feature point generation unit 304 extracts feature points from each of the generated front, back, left, right, top, and bottom element images through processing by the image conversion unit 302, and records the two-dimensional coordinate group of each feature point for each element image in the feature point recording unit 200 as secondary feature amounts.

When feature points are extracted from a spherical panoramic image, the image is curved, making it difficult to extract the feature points. Even if feature points can be extracted, the spatial resolution in the curved image varies depending on the position of the feature points, reducing the accuracy of comparing feature points between images. In response to this, the coordinates of feature points generated as secondary feature values by the image conversion unit 302 and the coordinates of feature points generated as primary feature values are both generated to correspond to the front, back, left, right, top, and bottom element images developed on a plane. This makes the extraction of feature points more stable. Furthermore, since the coordinates of feature points can be extracted for the same reference plane with reduced curvature, the shift in coordinates between images is uniform, making it possible to compare feature points with higher accuracy.

Another method is to determine the position using a sensor that can measure distance, such as a stereo camera or laser (LiDAR). With a stereo camera, the range captured by the camera is unidirectional, i.e., in the direction of travel. Therefore, if there are no structures that serve as feature points in the direction of travel or if there are obstacles, and feature points cannot be extracted continuously over time, it may not be possible to determine the position. Additionally, stereo cameras have a narrow range of capture, which can reduce the accuracy of their characteristics.

In contrast, with a spherical panoramic image, images can be acquired not only in the direction of travel, but also above, below, left, right, and behind, reducing the occurrence of events in which feature points cannot be extracted. In addition, when used indoors, such as in a power generation plant, the ceiling can be continuously photographed, making it possible to stably extract feature points.

For drones and other mechanisms that move indoors in three dimensions, aircraft equipped with 3D LiDAR that measures the situation of surrounding structures in three dimensions are used. 3D LiDAR is a mechanism that rotates while scanning in the vertical direction with a laser, and there is an issue with installing it, as it requires large expectations. In other words, when using LiDAR, it is necessary to equip it with a sensor in addition to the imaging device. Also, some have a rotation mechanism, which poses the issue of a complex configuration.

In response to this, our proposal is to equip the drone with a camera capable of capturing spherical images, making it possible to identify the image and its location. It is also superior in terms of payload capacity.

The feature point comparison unit 306 compares the coordinate group of the secondary feature amounts generated by the feature point generation unit 304 with the coordinate group of the primary feature amounts recorded in the feature point recording unit 200, and extracts the most similar primary feature amount. Then, the feature point comparison unit 306 selects the coordinates of feature points from the primary feature amounts that correspond to the coordinates of at least two feature points in the secondary feature amounts generated by the feature point generation unit 304.

The shooting position estimation unit 308 estimates the shooting position at which the omnidirectional panoramic image was shot, based on the comparison result of the feature point comparison unit 306. For example, the shooting position estimation unit 308 sets the position of the most similar primary feature selected by the feature point comparison unit 306 as the primary estimated shooting position. In other words, when calculating this primary feature, the central coordinates used in the simulation development of each element image plane of the front, back, left, right, top, and bottom are set as the primary estimated shooting position.

Furthermore, the shooting position estimation unit 308 estimates the shooting position with higher accuracy from the coordinates of the feature points in the primary feature quantity that correspond to the coordinates of at least two feature points selected by the feature point comparison unit 306.

Figure 10 is a diagram that shows a schematic example of position estimation by the shooting position estimation unit 308. A processing example of more accurate position estimation by the shooting position estimation unit 105 will be described with reference to Figures 5 and 10.

As shown in FIG. 10, the feature points S2 in the previous image 700q are associated with the feature points of the primary feature amount based on the comparison result of the feature point comparison unit 306. The feature points of the primary feature amount are two-dimensional coordinates, but are also associated with three-dimensional coordinates in real space shown in the feature point map G600 (see FIG. 7). This allows the shooting position estimation unit 308 to associate the multiple points S3 having three-dimensional coordinates in real space with the multiple feature points S2 by using multiple points S3 having three-dimensional coordinates in real space of the feature points of the primary feature amount corresponding to the multiple feature points S2. The multiple feature points S2 in the previous image 700q are projected onto the previous image 700q with the multiple points S3 having three-dimensional coordinates in real space as the center point g0. Therefore, the position where the projection lines connecting the multiple feature points S2 and the multiple points S3 having corresponding three-dimensional coordinates converge to one point is estimated as the shooting position. In this way, the shooting position estimation unit 308 re-estimates the position where the projection lines connecting the multiple points S3 having the three-dimensional coordinates corresponding to the multiple feature points S2 converge to one point as the secondary estimated shooting position. Then, the shooting position estimation unit 308 records the primary estimated shooting position and the secondary estimated shooting position in the shooting position record 204 in association with the shooting time, the omnidirectional panoramic image, and each element image.

The verification unit 310 verifies the position estimation by the shooting position estimation unit 308. If the position estimation by the shooting position estimation unit 308 is not valid, the verification unit 310 does not transmit the estimation result from the communication unit 25. On the other hand, if the position estimation by the shooting position estimation unit 308 is valid, the verification unit 310 communicates the position estimation result of the shooting position estimation unit 308, the shooting time, the omnidirectional panoramic image, and each element image to the communication unit 25. For example, the verification unit 310 verifies that the position estimation is not valid when the difference in distance between the previous estimated shooting position and the current estimated shooting position deviates from the distance based on the moving speed of the shooting processing device 2 by, for example, 10% or more.
The control unit 312 controls each unit of the image capturing and processing device 2 .

Next, an example of the configuration of the monitoring device 4 will be described. FIG. 11 is a block diagram showing an example of the configuration of the monitoring device 4. The monitoring device 4 includes a communication unit 400, a memory unit 402, a feature point generation unit 404, a map information generation unit 406, an operation unit 408, a display device 410, and an input device 412. The monitoring device 4 has, for example, a CPU, and configures the feature point generation unit 404, the map information generation unit 406, and the operation unit 408 by executing a processing program stored in the memory unit 402. Note that the feature point generation unit 404, the map information generation unit 406, and the operation unit 408 may be configured as a circuit.

The communication unit 400 communicates with the image capturing and processing device 2 .
The storage unit 402 is realized by, for example, a semiconductor memory element such as a RAM (Random Access Memory), a flash memory, a hard disk, etc. The storage unit 402 stores information transmitted from the imaging processing device 2. The storage unit 402 stores, for example, an estimated imaging position, an imaging time, a spherical panoramic image, and each element image in association with each other.

The feature point generation unit 404 generates point cloud data indicating the three-dimensional position of each structure of the power plant measured by the above-mentioned 3D laser scanner or the like, and primary feature values. The feature point generation unit 404 also stores the point cloud data indicating the three-dimensional position and the primary feature values in the storage unit 402, and transmits the primary feature values to the imaging and processing device 2 via the communication unit 400.

The map information generating unit 406 generates map information indicating the movement route of the image capturing device 2 based on the image capturing position transmitted from the image capturing device 2. The map information generating unit 406 also causes the display device 410 to display the generated map information.

Figure 12 is a diagram showing a schematic diagram of map information generated by the map information generating unit 406. A plurality of points Z12 indicate the estimated shooting positions of the shooting processing device 2. When the operator designates the estimated shooting position point Z12, the map information generating unit 406 causes the display device 410 to display map information 8 indicating the estimated shooting position, the shooting time, the spherical panoramic image, and each element image side by side.

The operation unit 408 can control the operation of the image capture device 2 in response to the information input by the monitor via the input unit 412. Note that the image capture device 2 is usually operated automatically.

The display device 410 is, for example, a liquid crystal monitor, and as described above, is capable of displaying the map information 8 generated by the map information generating unit 406, the spherical panoramic image, and each element image side by side. The input unit 412 is, for example, configured with a mouse, a keyboard, etc.

Figure 13 is a flowchart showing an example of processing by the image capture device 2. First, the image capture unit 10 captures the on-site conditions inside a building or other target object while moving using a moving unit (not shown), and outputs a celestial sphere panoramic image P600 (see Figure 4) including an image of the target object to the storage unit 20 and the information processing unit 30 (step S100).

Next, the image conversion unit 302 converts the spherical panoramic image P600 (see FIG. 4) into each element image G402 (see FIG. 4) based on the reference orientation using information from the orientation information acquisition unit 300 (step S102).

Next, the feature point generation unit 304 extracts feature points from each element image G402, and records the two-dimensional coordinate group of each feature point for each element image G402 as a secondary feature in the feature point recording unit 200 (step S104). Next, the feature point comparison unit 306 compares the coordinate group of the secondary feature generated by the feature point generation unit 304 with the coordinate group of the primary feature recorded in the feature point recording unit 200, and extracts the most similar primary feature (step S105).

Then, the shooting position estimation unit 308 estimates the shooting position where the spherical panoramic image was taken based on the comparison result of the feature point comparison unit 306 (step S106).

As described above, according to this embodiment, the feature point generation unit 304 generates secondary feature amounts based on each of the element images (front, back, left, right, top, and bottom) generated using the spherical panoramic image, and the shooting position estimation unit 308 estimates the shooting position based on the secondary feature amounts. This makes it possible to extract the coordinates of feature points for each element image with reduced curvature, thereby enabling the shooting position to be estimated with higher accuracy.

(Modification of the first embodiment)
The photographing processing system 1 according to the modified example of the first embodiment differs from the photographing processing device 2 according to the first embodiment in that the photographing position estimation unit 308 can estimate the position by using only images selected from the front, back, left, right, top, and bottom element images G402. The differences from the photographing processing system 1 according to the first embodiment will be described below.

FIG. 14 is a flowchart showing an example of processing by the imaging processing device 2 according to a modified example of the first embodiment. As shown in FIG. 14, the image conversion unit 302 develops, for example, only the front image (see FIG. 4) so that the center m30 of the front image coincides with the true north direction based on the captured spherical panoramic image P600 and the information of the orientation information acquisition unit 300. Then, the feature point generation unit 304, the feature point comparison unit 306, and the imaging position estimation unit 308 perform processing similar to that of the first embodiment using only the front image (step S200). Note that the processing is not limited to the front image, and five or less element images from among the front, back, left, right, top, and bottom element images G402 may be used for processing. That is, in step S200, position estimation is possible for each imaging session using only the front image according to the progress of the imaging processing device 2.

Next, the control unit 312 determines whether the number of times photographing has reached a predetermined number, for example, 10 times (step S202). If the control unit 312 determines that the predetermined number of times has not been reached (NO in step S202), it repeats photographing and photographing position estimation from step S202. During this time, the photographing and processing device 2 is in progress and is capable of position estimation for each photographing.

On the other hand, if it is determined that the number of shots has reached the predetermined number (YES in step S202), the control unit 312 performs position estimation using each of the element images of the front, back, left, right, top, and bottom, as shown in FIG. 13 (step S204).

The control unit 312 then determines whether or not to end the overall process (step S206), and if not (NO in step S206), repeats the process from step S200. On the other hand, if it is to end (YES in step S206), the overall process ends.

As described above, according to this embodiment, the shooting position estimation unit 308 can perform the estimation process using an image selected from the front, back, left, right, top, and bottom element images, which makes it possible to speed up the shooting position estimation process. In addition, the shooting position estimation process is performed using each element image at a predetermined cycle, making it possible to perform the shooting position estimation process with higher accuracy at a predetermined cycle.

Second Embodiment
The photographing and processing system 1 according to the second embodiment differs from the photographing and processing device 2 according to the modified example 1 of the first embodiment in that the photographing and processing device 1 according to the second embodiment is further capable of detecting an obstacle in the travel direction of the photographing and processing device 2. The differences from the photographing and processing device 2 according to the modified example 1 of the first embodiment will be described below.

Fig. 15 is a block diagram showing the configuration of the image capture processing device 2 according to the second embodiment. As shown in Fig. 15, the image capture processing device 2 according to the second embodiment is further configured to include an obstacle detection unit 314.

Figure 16 is a diagram that shows a schematic diagram of the traveling direction L14 and an obstacle B16 when the photography processing device 2 is located at an estimated photography position Z14. As shown in Figure 16, the obstacle detection unit 314 estimates the traveling direction L14 from the current estimated photography position Z14, and detects the presence of an obstacle B16 when the obstacle B16 is in the photography direction.

More specifically, the obstacle detection unit 314 acquires the estimated photographing position including the photographing position Z13 immediately before the current photographing time from the photographing position recording unit 204 based on the current photographing estimated position Z14 and the current photographing time. Next, the obstacle detection unit 314 estimates the traveling direction L14, for example, by using the difference in coordinates between the estimated shadow position Z14 and the immediately previous photographing position Z13. Next, the obstacle detection unit 314 determines whether or not a three-dimensional coordinate point (see FIG. 7) of a feature point in the power plant building registered in advance exists within a predetermined range, for example, within 10 meters from the estimated shadow position Z14 in the traveling direction L14 and within the vertical height of the photographing processing device 2. When the obstacle detection unit 314 determines that the three-dimensional coordinate point (see FIG. 7) of the feature point exists, it detects that an obstacle B16 exists. Then, the obstacle detection unit 314 changes the moving direction of the photographing processing device 2 to the correction direction L16 or stops it via the control unit 312. In addition, the obstacle detection unit 314 transmits the travel direction L14, the correction direction L16, and the coordinates of the obstacle B16 to the monitoring device 4 via the communication unit 25. The monitoring device 4 displays the information on the travel direction L14, the correction direction L16, and the obstacle B16 on the display device 410 together with the map information. This allows the observer to more easily observe the traveling environment of the imaging and processing device 2.

As described above, according to this embodiment, the obstacle detection unit 314 estimates the travel direction L14 from the current estimated shooting position Z14, and if an obstacle B16 is present in the shooting direction L14, it detects the presence of the obstacle B16. This makes it possible to perform avoidance control of the obstacle B16 before the photography processing device 2 comes into contact with the obstacle B16.

Third Embodiment
The imaging processing system 1 according to the third embodiment differs from the imaging processing system 1 according to the second embodiment in that a three-dimensional image can be reconstructed using elemental images of the front, back, left, right, top, and bottom. The differences from the imaging processing system 1 according to the second embodiment will be described below.

Fig. 17 is a block diagram showing the configuration of the imaging and processing device 2 according to the third embodiment. As shown in Fig. 17, the imaging and processing device 2 according to the third embodiment is further configured to include a three-dimensional image processing unit 316.

The three-dimensional image processing unit 316 generates three-dimensional coordinates of the feature points extracted by the feature point generation unit 304, and reconstructs a three-dimensional image in which the feature points are distributed in three dimensions. FIG. 18 is a detailed block diagram of the three-dimensional image processing unit 316. As shown in FIG. 18, the three-dimensional image processing unit 316 has a small area comparison unit 316a, a three-dimensional coordinate generation unit 316b, a three-dimensional image reconstruction processing unit 316c, a three-dimensional difference processing unit 316d, a video processing unit 316e, and a three-dimensional density calculation unit 316f.

Figure 19 is a diagram showing a schematic example of a processing example of the small area comparison unit 316a. As shown in Figure 19, previous images 702q and 704q at estimated shooting positions g0L and g0R at different times are shown. The small area comparison unit 316a divides the previous image 702q into multiple small image areas 80SL. Although only two small image areas 80SL are shown in Figure 19, the small area comparison unit 316a divides the previous image 702q into multiple small image areas 80SL in a matrix shape. Next, the small area comparison unit 316a extracts multiple small image areas 80RL with the highest correlation value for each of the multiple small image areas 80SL of the previous image 704q. For example, the small image area 80SL obtained by capturing the area 80S from the estimated shooting position g0L has the highest correlation value with the small image area 80SR obtained by capturing the area 80D from the estimated shooting position g0R. In other words, if a small image region 80SR with a high correlation value is extracted with respect to a small image region 80SL, the probability that the same region 80S is being captured increases. In this way, the small region comparison unit 316a divides the previous image 702q into multiple small image regions 80SL, and selects the small image region 80SR with a high correlation from the corresponding previous image 704q.

Figure 20 is a diagram showing a typical processing example of the three-dimensional coordinate generation unit 316b. As shown in Figure 20, the three-dimensional coordinate generation unit 316b obtains corresponding feature points 82L, 82R from the small image area 80SL and small image area 80SR with high correlation extracted by the small area comparison unit 316a, and records the intersections of the projection lines L82L, L82R connecting the feature points 82L, 82R and the estimated shooting positions g0L, g0R, respectively, as the three-dimensional coordinates of the feature points 82L, 82R in the feature point recording unit 200. Note that, although the previous image 702q has been described as an example, the small area comparison unit 316a and the three-dimensional coordinate generation unit 316b perform the same processing for each element image G402. In addition, the three-dimensional coordinate generation unit 316b adds the three-dimensional coordinates of the feature points 82L, 82R to the feature point recording unit 200. This makes it possible to increase the number of feature points in the feature point map G600 (see Figure 7).

By repeating this process for each estimated shooting position g0L, new three-dimensional coordinates of feature points are generated based on each element image G402 obtained for each image capture by the shooting processing device 2.

Figure 21 is a diagram showing an example of voxel data 210V generated by the three-dimensional image reconstruction processing unit 316c. The three-dimensional image reconstruction processing unit 316c generates the three-dimensional voxel data 210V using the three-dimensional coordinates of feature points 82L, 82R recorded in the feature point recording unit 200. The three-dimensional image reconstruction processing unit 316c assigns a predetermined value, for example 500, to the voxels corresponding to the three-dimensional coordinates of feature points 82L, 82R.

Figure 22 is a diagram showing an example of voxel data 210a, b generated by the 3D image reconstruction processing unit 316c, and voxel data 210c that is the processing result by the 3D difference processing unit 316d. As shown in Figure 22, voxel data 210a is a 3D reconstructed image of feature points 82L, 82R generated in April 2018, and voxel data 210b is a 3D reconstructed image of feature points 82L, 82R generated in September 2018. In this way, the 3D image reconstruction processing unit 316c can limit the generation period of feature points 82L, 82R to be used.

As shown in FIG. 22, the three-dimensional difference processing unit 316d performs a process of generating a difference between the voxel data 210a and the voxel data 210b. The voxel data 210c is the processing result of the dimension difference processing unit 316d, and indicates the difference between the voxel data 210a and the voxel data 210b. The square area indicates a three-dimensional area of a feature point that was not present in April 2018 but was present in September 2018. On the other hand, the circle area indicates a three-dimensional area of a feature point that was present in April 2018 but was no longer present in September 2018. In this way, the three-dimensional difference processing unit 316d performs a difference process between different periods, and assigns a first predetermined value, for example, 1000, to the newly generated voxel data 210c and a second predetermined value, for example, 1500, to the lost voxel data 210c.

The video processing unit 316e converts the processing result of the three-dimensional difference processing unit 316d into video information. For example, the video processing unit 316e generates video information as shown in FIG. 22 by associating the voxel data 210c, voxel data 210a, and voxel data 210b. The processed video information is then associated with the voxel data 210c, voxel data 210a, and voxel data 210b and transmitted to the monitoring device 4. This allows the monitoring device 4 to display, for example, an image as shown in FIG. 22 on the display device 410. This allows the monitor to more easily detect abnormalities indoors, such as in a power generation plant.

Figure 23 is a diagram showing an example of reprojecting the difference information generated by the three-dimensional difference processing unit 316d onto each element image. The square area indicates a two-dimensional area of feature points that were not present in April 2018 but appeared in September 2018. On the other hand, the circle area indicates a two-dimensional area of feature points that were present in April 2018 but no longer exist in September 2018.

As shown in FIG. 23, the image processing unit 316e converts the three-dimensional coordinates of the newly generated or disappeared voxel data 210c into two-dimensional coordinates on each element image, and displays them, for example, in a predetermined shape or color. For example, the image processing unit 316e displays the area corresponding to the newly generated feature point as a rectangle. Alternatively, the image processing unit 316e may display the area corresponding to the newly generated feature point in green. Similarly, the image processing unit 316e displays the area corresponding to the disappeared feature point as a circle. Alternatively, the image processing unit 316e may display the area corresponding to the disappeared feature point in red. This allows the monitor to easily check changes in the on-site situation, such as changes in the arrangement of equipment indoors at a power generation plant, using each element image.

The three-dimensional density calculation unit 316f can perform noise reduction processing on the voxel data 210a and the voxel data 210b. The three-dimensional difference processing unit 316d can perform difference processing on the voxel data 210a and the voxel data 210b that have been noise-reduced by the three-dimensional density calculation unit 316e. More specifically, the three-dimensional density calculation unit 316f calculates the density of three-dimensional feature points for each predetermined range for the voxel data 210a and the voxel data 210b. Then, when there is a difference of a predetermined value or more in the density calculated by the three-dimensional density calculation unit 316f, the three-dimensional difference processing unit 316d assigns a first predetermined value to the newly generated voxel data 210c and a second predetermined value to the lost voxel data 210c. This makes it possible to suppress the effects of noise.

As described above, according to this embodiment, the three-dimensional image reconstruction processing unit 316c generates voxel data 210a, 210b of feature points for each period. The three-dimensional difference processing unit 316d then performs difference processing of the voxel data 210a, 210b between different periods, assigning a first predetermined value to the newly generated voxel data 210c and a second predetermined value to the lost voxel data 210c. This makes it possible to grasp in three dimensions the areas that have been generated and the areas that have disappeared within a predetermined period. This allows the monitor to more easily detect changes in the on-site situation, such as changes in the arrangement of equipment indoors at a power generation plant, etc.

(Fourth embodiment)
The photography processing system 1 according to the fourth embodiment differs from the photography processing system 1 according to the fourth embodiment in that the photography processing system 1 according to the fourth embodiment can generate areas generated within a predetermined period and areas that have disappeared as two-dimensional image information using each of the element images of the front, back, left, right, top, and bottom, and can convert the generated areas and the disappeared areas into three-dimensional coordinates. The differences from the photography processing system 1 according to the fourth embodiment will be described below.

Fig. 24 is a block diagram showing the configuration of the imaging and processing device 2 according to the fourth embodiment. As shown in Fig. 24, the imaging and processing device 2 according to the fourth embodiment is further configured to include a two-dimensional image processing unit 318.

The two-dimensional image processing unit 318 uses each of the element images (front, back, left, right, top, and bottom) to generate two-dimensional image information of the areas generated within a predetermined period and the areas that have disappeared. FIG. 25 is a detailed block diagram of the two-dimensional image processing unit 318. As shown in FIG. 25, the two-dimensional image processing unit 318 has a shooting position comparison unit 318a, a correction image generation unit 318b, and a difference image display processing unit 318c.

The shooting position comparison unit 318a compares the estimated shooting positions estimated by the shooting position estimation unit 308. More specifically, it compares the current estimated shooting position with the estimated shooting positions within a specified period of time in the past recorded in the shooting position recording unit 204, and extracts the estimated shooting position within a specified period of time in the past that is closest to the current estimated shooting position.

The corrected image generating unit 318b performs affine transformation so that each elemental image recorded in the video recording unit 202 overlaps based on the current estimated shooting position and the estimated shooting position within a specified period of time in the past.The corrected image generating unit 318b then supplies each elemental image after the affine transformation to the small area comparing unit 316a (see FIG. 18).

The small area comparison unit 316a divides each element image into small areas, calculates the similarity of the corresponding areas, and extracts small areas with a similarity lower than a predetermined value as newly generated or disappeared small areas. The small area comparison unit 316a then supplies the two-dimensional coordinates of the small areas with a similarity lower than the predetermined value and each estimated shooting position to the three-dimensional coordinate generation unit 316b (see FIG. 18). As shown in FIG. 20, the three-dimensional coordinate generation unit 316b uses the two-dimensional coordinates of the small areas and each estimated shooting position to generate three-dimensional coordinates of the small areas. Furthermore, since only the small areas are converted into three-dimensional coordinates, processing can be made faster.

The differential image display processing unit 318c superimposes the three-dimensional coordinates of the small area, for example, on a three-dimensional image, such as the feature point map G600 (see FIG. 7), records it in the video recording unit 202, and transmits it to the monitoring device 4. This makes it possible for the monitoring device 4 to display the changed area on the display device 410 together with the three-dimensional image. This allows the monitor to more easily detect abnormalities indoors, such as in a power generation plant.

The differential image display processing unit 318c can also convert the three-dimensional coordinates of newly generated or disappeared small areas into two-dimensional coordinates on each element image, as shown in FIG. 23, and display them, for example, in a specified shape or color.

As described above, according to this embodiment, the shooting position comparison unit 318a extracts the estimated shooting position within a predetermined period of time that is closest to the current estimated shooting position, and the small area comparison unit 316a divides each element image of each estimated shooting position into small areas, calculates the similarity of the corresponding areas, and extracts small areas with a similarity lower than a predetermined value. Then, the difference image display processing unit 318c generates three-dimensional coordinates of the small areas with a similarity lower than a predetermined value, and superimposes the three-dimensional coordinates of these small areas, for example, on the three-dimensional image. This makes it possible to display the changed area on the display device 410 together with the three-dimensional image. This allows the monitor to more easily and quickly detect changes in the on-site situation, such as changes in the arrangement of equipment indoors at a power generation plant, etc.

At least a part of the imaging processing system 1 described in the above embodiment may be configured with hardware or software. If configured with software, a program that realizes at least a part of the functions of the control unit may be stored on a recording medium such as a flexible disk or CD-ROM, and may be read and executed by a computer. The recording medium is not limited to removable ones such as magnetic disks or optical disks, but may be fixed recording media such as a hard disk unit or memory.

In addition, a program that realizes at least a part of the functions of the image processing system 1 may be distributed via a communication line (including wireless communication) such as the Internet. Furthermore, the program may be encrypted, modulated, or compressed and distributed via a wired line or wireless line such as the Internet, or stored on a recording medium.

Although several embodiments have been described above, these embodiments are presented only as examples and are not intended to limit the scope of the invention. The novel parts, methods, and programs described in this specification can be embodied in various other forms. Furthermore, various omissions, substitutions, and modifications can be made to the forms of the parts, methods, and programs described in this specification without departing from the spirit of the invention.

1: Image capture processing system, 2: Image capture processing device, 4: Monitoring device, 304: Feature point generation unit, 308: Image capture position estimation unit, 314: Obstacle detection unit, 316c: 3D coordinate generation unit, 316d: 3D difference processing unit, 318: 2D image processing unit.

Claims (10)

an image conversion unit that converts a spherical image obtained by capturing an object in a real space into a flat element image in a specific direction based on a first reference point;
a feature point recording unit which records a plurality of feature points having three-dimensional coordinates in a three-dimensional space that is a measurement target range, in a flat measurement element image in the specific direction with respect to a measurement reference point, a group of two-dimensional coordinates of each feature point projected with the measurement reference point as a center point, as a primary feature amount, in association with the three-dimensional coordinates, the feature point recording unit recording the group of two-dimensional coordinates as a primary feature amount, in association with the three-dimensional coordinates, for each of measurement reference points at a plurality of different positions;
a feature point generating unit that generates, as secondary feature amounts, a group of two-dimensional coordinates of each of a plurality of first feature points in the object obtained based on the element image in the specific direction;
a feature point comparison unit that compares a coordinate group of the secondary feature amounts generated by the feature point generation unit with a coordinate group of the primary feature amounts recorded in the feature point recording unit, extracts the most similar primary feature amount, and associates each feature point of the secondary feature amounts with each feature point of the primary feature amounts;
an imaging position estimation unit that associates three-dimensional coordinates in real space of feature points of the primary feature amounts corresponding to at least two feature points of the secondary feature amounts, and estimates a position where projection lines connecting the at least two feature points and at least two points having corresponding three-dimensional coordinates converge as an imaging position of the spherical image;
An imaging processing device comprising: The image capture processing device according to claim 1, further comprising an obstacle detection unit that detects an obstacle based on a course direction estimated using the estimated image capture positions at multiple different times estimated by the image capture position estimation unit. The imaging processing device according to claim 2, further comprising a three-dimensional coordinate generation unit that generates three-dimensional coordinates of each of the first feature points based on the first position coordinates corresponding to each of the estimated imaging positions estimated by the imaging position estimation units and the estimated imaging positions. A small area comparison unit is further provided which divides flat elemental images in a specific direction at the estimated photographing positions at the plurality of different time points estimated by the photographing position estimation unit into a plurality of small image areas and selects small image areas having high mutual correlation,
The imaging and processing device according to claim 3 , wherein each of the plurality of first feature points is selected from small image regions that are highly correlated with each other. The imaging and processing device according to claim 4, further comprising a three-dimensional image reconstruction processing unit that generates three-dimensional voxel data using each of the three-dimensional coordinates generated by the three-dimensional coordinate generation unit. The imaging processing device according to claim 5, further comprising a three-dimensional difference processing unit that extracts an increase or decrease in a structure in a specified space using three-dimensional voxel data of three-dimensional coordinates in the specified space generated in a first period and three-dimensional voxel data of three-dimensional coordinates in the specified space generated in a second period different from the first period. The imaging processing device according to claim 6, wherein the three-dimensional difference processing unit assigns a first predetermined value to newly generated voxel data and assigns a second predetermined value different from the first predetermined value to lost voxel data. The imaging processing device according to claim 6 , wherein the three-dimensional difference processing section extracts an increase or decrease in a structure within the predetermined space based on a density of points indicated by three-dimensional coordinates within the predetermined space. The imaging processing device according to any one of claims 1 to 4, further comprising a two-dimensional image processing unit that matches first video information based on a spherical image captured from a first estimated imaging position estimated by the imaging position estimation unit with second video information based on a spherical image captured from a second estimated imaging position estimated by the imaging position estimation unit, and extracts areas where changes have occurred. an image conversion step of converting a spherical image obtained by capturing an object in a real space into a flat element image in a specific direction based on a first reference point;
a feature point recording step of recording a plurality of feature points having three-dimensional coordinates in a three-dimensional space that is a measurement target range, in a flat measurement element image in the specific direction relative to a measurement reference point, in which a group of two-dimensional coordinates of each feature point projected with the measurement reference point as a center point is recorded as a primary feature value in association with the three-dimensional coordinates, the feature point recording step recording the group of two-dimensional coordinates as a primary feature value in association with the three-dimensional coordinates for each of measurement reference points at a plurality of different positions;
a feature point generating step of generating, as secondary feature amounts, a group of two-dimensional coordinates of each of a plurality of first feature points in the object obtained based on the element image in the specific direction;
a feature point comparison step of comparing a coordinate group of the secondary feature amounts with a coordinate group of the primary feature amounts, extracting a most similar primary feature amount, and associating each feature point of the secondary feature amounts with each feature point of the primary feature amounts;
a photographing position estimating step of associating three-dimensional coordinates in real space of feature points of the primary feature amounts corresponding to at least two feature points of the secondary feature amounts, and estimating a position where projection lines respectively connecting the at least two feature points and at least two points having corresponding three-dimensional coordinates converge as a photographing position of the spherical image;
An imaging processing method comprising: