KR20180061803A - Apparatus and method for inpainting occlusion of road surface - Google Patents

Abstract

The present invention discloses a device and a method for a road surface occlusion region, which restore an occlusion region of a road surface three-dimensional point group by using multi-viewpoint images, thereby being utilized for lane extraction and map construction. The device for restoring a road surface occlusion region comprises: an abnormal point group removing part removing an abnormal point group from a three-dimensional point group obtained by a mobile mapping system (MMS); an occlusion region detecting part detecting the occlusion region from an image in which the abnormal point group is removed by the abnormal point group removing part; a point group generating part generating a virtual point group on the occlusion region detected by the occlusion region detecting part; and an occlusion region restoring part projecting each virtual point generated by the point group generating part to the multi-viewpoint image, and changing each color of the virtual point in the multi-viewpoint image into grayscale to restore the occlusion region.

Description

[0001] APPARATUS AND METHOD FOR INPAINTING OCCLUSION OF ROAD SURFACE [0002]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an apparatus and method for restoring a road surface occluded region, and more particularly, to an apparatus and method for restoring a road surface occluded region that restores an occluded region of a three-dimensional point cloud on a road surface using multi-view images.

Recently, 3D spatial information has been applied to various application fields such as city landscape planning, disaster management system, navigation and internet map service.

As a method for constructing the most important three-dimensional building model in the three-dimensional spatial information, there are a method of using the numerical map and the layer number information of the building complex, a method of using stereo stereoscopic images accompanied by sensor modeling, (LIDAR) data obtained by directly acquiring the elevation values of the points constituting the ground surface using the laser distance measuring equipment mounted on the aircraft, etc. have.

Particularly, in the method using the LIDAR data directly acquiring the height values of the points constituting the surface of the ground with the distance measurement equipment, projection stitching or warping was used for color restoration but relatively clear color estimation was difficult, There is a problem that a part which is difficult to restore due to a lack of available image or effective color information occurs.

The background art of the present invention is disclosed in Korean Patent Registration No. 10-1289885 (July 31, 2013) entitled " Apparatus and Method for Building Modeling. &Quot;

SUMMARY OF THE INVENTION The present invention has been made to solve the above problems, and it is an object of one aspect of the present invention to restructure an obstruction area of a road surface three-dimensional point cloud using multi-viewpoint images, And more particularly, to an apparatus and method for restoring a road surface obstruction area.

According to an aspect of the present invention, there is provided an apparatus for restoring a road surface occluded region, comprising: an abnormal point group removing unit that removes an abnormal point group from a three-dimensional point cloud acquired by an MMS (Mobile Mapping System); An occluded region detecting unit for detecting an occluded region in the image by the abnormal point group removed by the abnormal point group removing unit; A point cloud generation unit for generating a virtual cloud point in the occluded region detected by the closed region detection unit; And a blocking region restoring unit for restoring each of the virtual points generated by the point group generating unit to a multi-point image and changing the hue of each virtual point in the multi-point image to gray scale, .

In the present invention, the closed region detecting unit may include a point aligning unit for aligning the order of the peripheral point groups in a clockwise or counterclockwise direction on the basis of a certain point; An outer point detector for detecting an outer point of the obstruction area using an angle between two consecutive points centered on any one of the points arranged by the point aligner; And a closed region dividing unit for dividing the closed region by generating an outline connecting the outer points detected by the outer point detecting unit.

In the present invention, the outer point detection unit detects the current point as an outer point when the angle between two consecutive points around a certain point is greater than or equal to a preset angle between the surrounding points.

In the present invention, the point-group generating unit generates a virtual point cloud with a uniform density.

In the present invention, the occlusion region restoration unit may include a projection unit for projecting coordinates of virtual points in the occlusion region into two-dimensional images; An effective color determination unit for determining whether an effective color is based on a point-by-point color in each image corresponding to a virtual point projected by the projection unit; And an average color application unit for changing the hue of each point in the image to gray scale according to the determination result of the valid color determination unit and setting the changed hue to a virtual point color.

In the present invention, the valid color determination unit determines that the effective color is within a range where the variance of each point in each image is within a predetermined variance value.

In the present invention, the valid color determination unit may average the colors of the points in each image, change the average color to gray scale, and then set the color of the changed gray scale as the color of the virtual point.

The present invention further includes an example-based restoration unit for setting a color of a virtual point by using surrounding point group information around a virtual point, if the valid color determination unit determines that the closed area restoration unit is not an effective color .

In the present invention, the example-based restoration unit detects different points within a setting region centered on a virtual point, sets a color of a virtual point according to the reflection intensity between the detected other point and the virtual point and the similarity of colors .

According to an aspect of the present invention, there is provided a road surface clogging area restoration method comprising: removing an abnormal point group in a three-dimensional point cloud group obtained by an abnormal point cloud removal unit (MMS); Detecting an occlusion region in an image in which the abnormal point group is removed by the abnormal point group removing unit; Generating a point cloud group in the occluded region detected by the occluded region detecting unit; And restoring the occluded region by projecting each of the virtual points generated by the point cloud generating unit into a multi-point image by changing the color of each virtual point in the multi-point image to gray scale, .

In the present invention, the closed area detecting unit may arrange the order of the peripheral point groups in a clockwise or counterclockwise direction on the basis of a certain point, and determine an angle between two consecutive points centered on any one of the aligned points An outline point of the occlusion region is detected according to whether or not an abnormality is detected, and then an outline connecting the detected outline points is generated to distinguish the occlusion region.

In the present invention, the point-group generating unit generates a virtual point cloud with a uniform density.

In the present invention, the occluded region restoration unit may be configured to project the coordinates of virtual points in the occlusion region into two-dimensional images, and determine whether the target color is an effective color based on the distribution of colors in each of the images corresponding to the projected virtual points And changes the hue of each point in each image to gray scale according to the determination result, and sets the changed hue to the color of the virtual point.

In the present invention, the valid color determination unit may average the colors of the points in each image, change the average color to gray scale, and then set the color of the changed gray scale as the color of the virtual point.

The present invention is characterized in that the example-based restoration unit detects a different point within a setting area centered on a virtual point and sets a color of a virtual point according to the reflection intensity between the detected other point and the virtual point and the similarity of colors And further comprising:

The road surface clogging area restoration apparatus and method according to an aspect of the present invention can fully acquire lane information on the road surface by restoring the three-dimensional information that is not obtainable due to clogging, The lane information can be utilized and lane information can be utilized as high quality input data such as an automatic lane extraction module.

The road surface occluded region restoration apparatus and method according to another aspect of the present invention enable more accurate color estimation than the conventional method using stitching or warping by using accurate projection and multi-color image information.

The road surface occluded region restoration apparatus and method according to another aspect of the present invention can significantly reduce the cost of obtaining lane information on the road surface.

1 is a block diagram of a road surface occluded region restoration apparatus according to an embodiment of the present invention.
2 is a top view of a mobile mapping system in accordance with an embodiment of the present invention.
3 is a side view of a mobile mapping system according to an embodiment of the present invention.
FIG. 4 is a view illustrating a road surface definition and an abnormal point group removal method from a three-dimensional point cloud according to an embodiment of the present invention.
5 is a diagram illustrating a method of detecting an obstruction area of a road surface point cloud according to an embodiment of the present invention.
FIG. 6 is a diagram illustrating a method of correcting color through three-dimensional point cloud generation and projection according to an exemplary embodiment of the present invention.
FIG. 7 is a diagram illustrating an example-based color restoration method according to an embodiment of the present invention.
FIG. 8 is a diagram illustrating a road surface occluded region restoration method according to an embodiment of the present invention.

Hereinafter, an apparatus and method for restoring a road surface occluded region according to an embodiment of the present invention will be described in detail with reference to the accompanying drawings. In this process, the thicknesses of the lines and the sizes of the components shown in the drawings may be exaggerated for clarity and convenience of explanation. Further, the terms described below are defined in consideration of the functions of the present invention, which may vary depending on the user, the intention or custom of the operator. Therefore, definitions of these terms should be made based on the contents throughout this specification.

FIG. 2 is a plan view of a mobile mapping system according to an embodiment of the present invention. FIG. 3 is a block diagram of an embodiment of the present invention FIG. 4 is a view illustrating a road surface definition and an abnormal point group removal method from a three-dimensional point cloud according to an embodiment of the present invention. FIG. 5 is a side view of a road mapping method according to an embodiment of the present invention. FIG. 6 is a diagram illustrating a method of correcting color through three-dimensional point cloud generation and projection according to an embodiment of the present invention. FIG. FIG. 4 is a diagram illustrating an example-based color restoration method according to an exemplary embodiment of the present invention.

Referring to FIG. 1, the road surface occluded region restoration apparatus according to an embodiment of the present invention receives various measurement information from an MMS (Mobile Mapping System) 10, .

Referring to FIGS. 2 and 3, the MMS 10 is mounted on a vehicle and photographs or scans an image of a peripheral road to acquire measurement information on a surrounding area.

The MMS 10 includes a plurality of cameras 11 for photographing a surrounding image to acquire image information, a plurality of GPS (Global Positioning System) 13 for detecting the position of the vehicle, A Light Detection and Ranging (LiDAR) 12 for directly acquiring a height value, and an IMU (Inertial Measurement Unit) 14 for measuring the speed, direction, gravity and acceleration of the vehicle.

Particularly, a plurality of cameras 11 are provided, each of which photographs the periphery thereof, so that a multi-point image can be obtained even at the same point.

Referring to FIG. 1, a road surface occluded region restoration apparatus 20 according to an exemplary embodiment of the present invention restores a closed area of a road surface using measurement information transmitted from the MMS 10, A rejection region 21, a closed region detection unit 22, a point cloud generation unit 23, and a blocked region restoration unit 24.

The abnormal point group removing unit 21 removes the abnormal point group from the three-dimensional point cloud of the image acquired by the rider 12 of the MMS 10.

That is, the abnormal point group removing unit 21 extracts only the point group corresponding to the road surface from the three-dimensional point group obtained through the rider 12 as shown in FIG. 4 (a). In this case, the abnormal point group removing unit 21 estimates the height value of the vehicle and the elevation of the surface of the ground using the measurement information of the GPS 13 and the IMU 14, and recognizes the point group around the vehicle position as the point group of the road surface .

As shown in FIG. 4 (b), the abnormal point group eliminating unit 21 examines a point group in various directions based on the vehicle position. If the distance between the point group and the previous point group is longer than a predetermined distance, If the direction angle with respect to the survey point group is equal to or greater than the set angle, the three-dimensional point group survey is stopped and the point groups irradiated up to that point are merged into the road surface.

Based on this, the abnormal point group removing unit 21 defines the point group on the road surface as shown in FIG. 4 (c), and can remove the abnormal point group of the object that caused the obstruction area.

The occluded region detecting section 22 detects the occluded region in the image removed by the abnormal point group removing section 21. The closed area detecting unit 22 includes a point aligning unit 221, an outer edge detecting unit 222, and a closed region dividing unit 223.

As shown in FIG. 5A, the point alignment unit 221 detects any one point among the three-dimensional point cloud and arranges the order of the peripheral point cloud group clockwise or counterclockwise on the basis of the detected current point do. Here, the current point is a point that is currently detected among the plurality of points and is the object to be processed.

The outline point detecting unit 222 detects an angle between two consecutive points centered on the current point among the points arranged by the point aligning unit 221. When the detected angle is greater than a set angle The current point is recognized as an outer point.

In FIG. 5 (b), when the angle between the current points is significantly larger than the angle between the current points, the current point is recognized as an outer point.

The closed region classifying unit 223 generates an outline connecting the outer points detected by the outer point detecting unit 222 to classify the closed region. FIG. 5 (c) shows an example in which a convex hull is generated based on an outline generated by connecting outer points and the occlusion region is divided.

The point cloud generation unit 23 generates a virtual point cloud as shown in Fig. 6 (a) in the occlusion area detected by the occluded region detection unit 22. [ In this case, the point cloud generation unit 23 generates a virtual point cloud at a uniform density in the occlusion region. The point cloud generation unit 23 defines three-dimensional coordinates for each virtual point, and does not define reflection intensity and color.

The clogging area restoration unit 24 projects each virtual point generated by the point cloud generation unit 23 as a two-dimensional multi-point image, changes the color of each virtual point in the multi-point image to gray scale, Restore the area. In this case, the occluded region restoration unit 24 performs the above-described process for all virtual points in the occluded region.

The closed region restoration unit 24 includes a projection unit 241, an effective color determination unit 242, an average color application unit 243, and an example-based restoration unit 244.

The projection unit 241 projects the three-dimensional coordinates of each virtual point on the images acquired at the other viewpoint via the camera 11. [ In this case, as shown in FIG. 6, corresponding virtual points are located in each of the plurality of multi-viewpoint images. That is, since three-dimensional coordinates are defined in each virtual point generated by the point-group generating unit 23, when a three-dimensional coordinate is projected onto a two-dimensional multi-point image, a point corresponding to the corresponding coordinate is recognized As a result, one virtual point can be recognized as a point again.

The effective color determination unit 242 determines whether the color is an effective color based on a point-by-point color in each image corresponding to a virtual point. In this case, the valid color determination unit 242 determines that the effective color is within the range of the variance of each color in each image within the set variance value. For example, when the corresponding point is normally photographed by each camera 11, the color of the corresponding point in each image appears to be similar to each other. On the other hand, when the point is abnormally photographed, . For example, when the variance is relatively large, it is highly likely that the color of the road surface is not obtained correctly because different objects are photographed at various points in time.

Therefore, when the variance of the color per point in the image is within the set variance value, the valid color determiner 242 determines the color of the corresponding point as the valid color, whereas if the variance of the color per point in the image is not within the set variance value It is determined that the color of the corresponding point is not an effective color.

The average color application unit 243 sets the colors of the virtual points using the colors of the points in each image according to the determination result of the valid color determination unit 242. [

That is, if the color of the corresponding point is an effective color as a result of the determination of the valid color determination unit 242, the average color application unit 243 averages the color values of the colors and changes the average color to the rail scale. Since the intensity of the reflection is relatively high at the point where the paint is applied on the road surface, the corresponding color appears bright when converted to gray scale, and the point where the paint is not applied appears dark because the reflection intensity is relatively low. Accordingly, when the average color application unit 243 converts the hue to gray scale, the average color application unit 243 appears bright or dark depending on whether the paint is applied to the corresponding point.

Accordingly, the average color application unit 243 changes the average color obtained by averaging the color of the corresponding point in the image to gray scale, and sets the changed color as the color of the corresponding virtual point. Accordingly, if the virtual point is the point where the paint is applied, the virtual point is displayed in bright color, and if the point is not coated, the virtual point is displayed in dark color. This process is performed for all virtual points, and the occlusion region can be restored according to the result.

If the color of the corresponding point is not an effective color as a result of the determination of the valid color determination unit 242 in the above process, the example-based restoration unit 244 restores the virtual color using the point group information around the virtual point, Set the color of the point.

In other words, the example-based restoration unit 244 sets an arbitrary area around a virtual point, and detects another point within the setting area. The example-based restoring unit 244 then sets the color of the virtual point according to the intensity of reflection and the similarity of color between the detected other point and the virtual point. In this case, the detected other point is a restored virtual point or a detected point existing outside the obstruction area, and is not a virtual point not restored in the obstruction area.

7A and 7B, a virtual point cloud in the occlusion region is set to a uniform density, and in particular, a virtual point cloud is set in accordance with the reflection intensity and color similarity between different points in the setting region (target restoration region) The setting area is restored as shown in Fig. 7 (c).

Hereinafter, a method for restoring a road surface occluded region according to an embodiment of the present invention will be described in detail with reference to FIG.

FIG. 8 is a diagram illustrating a road surface occluded region restoration method according to an embodiment of the present invention.

Referring to FIG. 8, first, the rider 12 inside the MMS 10 detects a three-dimensional point cloud. In this case, the abnormal point group removing unit 21 extracts a point group corresponding to the road surface from the three-dimensional point group obtained through the rider 12 (S10).

Next, the abnormal point group removing unit 21 defines the point group on the road surface and removes the abnormal point group of the object that caused the obstruction area (S20).

Next, the occluded region detecting section 22 detects the occluded region in the image removed by the abnormal point group removing section 21 (S30).

That is, the closed-area detecting unit 22 detects any one point among the three-dimensional point cloud and aligns the order of the neighboring point cloud group clockwise or counterclockwise on the basis of the detected current point. Then, the closed-area detecting unit 22 detects an angle between two consecutive points around the current point, and recognizes the current point as an outlier point if the detected inter-angle is greater than or equal to a predetermined value among the surrounding inter-angle differences. As the outline points are recognized as described above, the occlusion area detecting unit 22 generates an outline connecting the outline points to separate the occlusion area. In this case, a convex hull composed of outer points is generated, and the occlusion region can be distinguished.

As the occlusion detection region is detected by the occlusion region detection unit 22, the point cloud generation unit 23 generates a virtual point cloud having a uniform density in the occlusion region detected by the occlusion region detection unit 22 (S40). In this case, three-dimensional coordinates are defined in each virtual point, while reflection intensity and color are not defined.

When a virtual point cloud in the obstruction area is generated by the point cloud generator 23, the obstruction area restoration unit 24 restores the three-dimensional coordinates of each virtual point generated by the point cloud generator 23 to the camera 11 (Step S50). The color of each point in each image corresponding to the projected virtual point is detected.

In operation S60, the clogging area restoration unit 24 determines whether the color is an effective color based on a point-by-point color in each image corresponding to a virtual point projected by the projection unit 241. In this case, the valid color determination unit 242 determines that the color of the corresponding point is an effective color when the variance of the point-by-point color in the image is within the set variance value, It is determined that the color of the corresponding point is not an effective color.

As a result of the determination in step S60, if the color of the corresponding point is an effective color, the average color application unit 243 averages color values of the colors, changes the average color to the rail scale, (S70). In this case, the virtual point is displayed in bright color when the paint is applied, and dark color when the paint is not applied.

On the other hand, if it is determined in step S60 that the color of the corresponding point is not an effective color, the average color application unit 243 sets an arbitrary area around a virtual point and detects another point within the set point After that, the color of the virtual point is set according to the similarity degree of the reflection intensity between the detected other point and the virtual point and the similarity degree of the color, thereby restoring the occluded region (S80).

As described above, the road surface clogging area restoration apparatus and method according to the embodiment of the present invention can completely acquire lane information on the road surface by restoring the three-dimensional information that is not obtainable due to occlusion, It is possible to utilize the lane information as high-quality input data such as an automatic lane extraction module.

In addition, the apparatus and method for restoring a road surface occluded region according to an exemplary embodiment of the present invention can perform more accurate color estimation than a conventional method using stitching or warping by using accurate projection and multi-color image information. And the cost of acquiring the lane information on the road surface can be greatly reduced.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is clearly understood that the same is by way of illustration and example only and is not to be taken by way of limitation, I will understand. Accordingly, the true scope of the present invention should be determined by the following claims.

10: MMS 11: Camera
12: Rider 13: GPS
14: IMU
20: road surface obstruction area restoration device
21: abnormal point removing unit 22: closed area detecting unit
221: point alignment unit 222: outer edge point detection unit
223: Closed area dividing section 23: Point group generating section
24: closed region restoring unit 241:
242: valid color determination unit 243: average color application unit
244: Example-Based Restoration

Claims (15)

Anomaly group elimination which removes abnormal point group from 3 - D point group obtained by MMS (Mobile Mapping System);
An occluded region detecting unit for detecting an occluded region in the image by the abnormal point group removed by the abnormal point group removing unit;
A point cloud generation unit for generating a virtual cloud point in the occluded region detected by the closed region detection unit; And
A blockage area restoration unit for restoring a blockage area by projecting each virtual point generated by the point group generation unit onto a multi-point image and changing the hue of each virtual point in the multi-point image to gray scale, Area restoration device.
The apparatus according to claim 1, wherein the closed region detecting unit comprises: a point aligning unit for aligning the order of peripheral point groups in a clockwise direction or a counterclockwise direction on the basis of a point;
An outer point detector for detecting an outer point of the obstruction area using an angle between two consecutive points centered on any one of the points arranged by the point aligner; And
And an obstruction area dividing unit for dividing the obstruction area by generating an outline connecting outer edge points detected by the outer edge point detecting unit.
3. The road surface obstruction detecting apparatus according to claim 2, wherein the outer point detecting section detects the current point as an outer point when the angle between two consecutive points around a certain point is greater than or equal to a predetermined value, Area restoration device.
The apparatus of claim 1, wherein the point cloud generator generates a virtual point cloud at a uniform density.
The apparatus of claim 1, wherein the occlusion region restoration unit
A projection unit for projecting coordinates of virtual points in the occlusion region into two-dimensional images;
An effective color determination unit for determining whether an effective color is based on a point-by-point color in each image corresponding to a virtual point projected by the projection unit; And
And an average color application unit for changing the color of a point in each image to gray scale according to the determination result of the valid color determination unit and setting the changed color as a color of a virtual point.
6. The apparatus of claim 5, wherein the valid color determination unit determines that the effective color is a color if the variance of each point in each image is within a predetermined variance value.
The method according to claim 5, wherein the valid color determination unit averages the colors of the points in each image, changes the average color to gray scale, and then sets the color of the changed gray scale to a virtual color of the point A facet clogging area restoration device.
6. The apparatus of claim 5, wherein the closed region restoration unit further includes an example-based restoration unit that sets a color of a virtual point using the nearby point group information about the virtual point as a result of the determination by the valid color determination unit And the road surface clogging area restoration device.
9. The image processing apparatus according to claim 8, wherein the example-based restoration unit detects another point within a setting area centered on a virtual point, detects a color of a virtual point according to the reflection intensity between the detected other point and the virtual point, Is set to the road surface clogging area.
Removing the abnormal point group from the three-dimensional point cloud obtained by the abnormal point cloud removal unit by the MMS (Mobile Mapping System);
Detecting an occlusion region in an image in which the abnormal point group is removed by the abnormal point group removing unit;
Generating a point cloud group in the occluded region detected by the occluded region detecting unit; And
A step of reconstructing a closed area by projecting each virtual point generated by the point group generating unit into a multi-point image by changing the color of each virtual point in the multi-point image into gray scale, A method for restoring a surface occluded region.
The apparatus according to claim 10, wherein the occlusion area detecting unit aligns the order of the peripheral point groups in a clockwise or counterclockwise direction on the basis of any one point, and calculates a distance between two consecutive points centered on any one of the aligned points Wherein the method comprises the steps of: detecting an outline point of the occlusion region according to whether the set angle is greater than or equal to the set angle, and generating an outline connecting the detected outline points to distinguish the occlusion region.
11. The method of claim 10, wherein the point cloud generator generates a virtual point cloud at a uniform density.
11. The image processing apparatus according to claim 10, wherein the occluded region restoration unit projects the coordinates of virtual points in the occlusion region into two-dimensional images, and based on the distribution of the per-point color in each image corresponding to the projected virtual point, And changing the color of a point in each image to gray scale and then setting the changed color as a color of a virtual point.
11. The method according to claim 10, wherein the valid color determination unit averages the colors of the points in each image, changes the average color to gray scale, and then sets the color of the changed gray scale to the virtual color of the point A method for restoring a surface occluded region.
11. The method of claim 10, wherein the example-based restoration unit detects another point within a setting area centered on a virtual point, and calculates a color of a virtual point according to the reflection intensity and color similarity between the detected other point and the virtual point Further comprising the step of setting the road surface clogging area.

Images