KR20230086921A - Method and device for mapping lidar data and color data - Google Patents

Abstract

The present invention relates to a LiDAR data and color information mapping method and device. The present invention relates to a mapping device which maps color information of a color image to LiDAR data. The mapping device includes: a LiDAR image generation unit which generates a LiDAR image as a perspective image corresponding to the viewpoint of the color image using the LiDAR data; a color transfer unit which color-transfers the color information to the LiDAR image; and a color information mapping unit which maps the color information color-transferred to the pixels of the LiDAR image to the LiDAR data corresponding to the LiDAR image.

Description

Method and device for mapping lidar data and color information {METHOD AND DEVICE FOR MAPPING LIDAR DATA AND COLOR DATA}

The present invention relates to a lidar data and color information mapping method and apparatus. More specifically, the present invention relates to a lidar data and color information mapping method and apparatus for mapping a camera image and a lidar image by mapping color information obtained from a camera to a point cloud for generating a lidar image.

Due to the demand for convenience and safety, the development and commercialization of self-driving cars are progressing very rapidly. Self-driving cars calculate the location of the vehicle through complex operations of sensors (GPS, radar, LiDAR, camera, etc.) mounted on the vehicle, and control the vehicle by recognizing the surrounding situation. However, it may be difficult to accurately recognize surroundings only by the operation of sensors, and the importance of high-precision maps including various road information, facility properties, and absolute coordinate information is increasing for more accurate vehicle control.

In order to produce a high-precision map, a point cloud having spatial location information is required. A point cloud can be obtained by scanning roads and road-side facilities using MMS (Mobile Mapping System). Point cloud information is converted into LAS data through post-processing, and a high-precision map is built based on the LAS data.

MMS consists of a Global Positioning System (GPS), an Inertial Measurement Unit (IMU), a camera, LIDAR, and a mileage sensor. Images and point clouds (point cloud data) collected from cameras and LIDAR are used to construct object location values, surface shape, and attribute information required for high-precision map construction.

LiDAR data and camera image data can be matched through calibration between each sensor module, but matching errors occur due to problems such as vibration, inertia, or processing delay when driving a vehicle equipped with an MMS system. That is, there is a problem that it is difficult to match the data of the two sensors only with the calibration information between the camera and lidar.

Registered Patent No. 10-1105361

In order to solve the above problems, the present invention is a lidar data and color information mapping method for adding color image information to lidar data by mapping color information obtained from a camera to a point cloud for generating lidar images. and to provide an apparatus.

In the present invention, in a mapping device for mapping color information of a color image to lidar data, a lidar image generator for generating a lidar image as a perspective image corresponding to a viewpoint of the color image using the lidar data ; a color transfer unit for color transferring the color information to the lidar image; and a color information mapping unit that maps the color information transferred to pixels of the lidar image to the lidar data corresponding to the lidar image.

In one embodiment, the color transfer unit may perform the color transfer by using the LIDAR image as a content image and the color image as a color image.

In addition, the lidar data is a point cloud, and the lidar image generating unit converts lidar data into voxels using the point clad, and converts the voxelized voxel cube to match the color image capture time point. By projecting, the LIDAR image may be generated.

In one embodiment, the color information mapping unit may map the color information to a point cloud constituting the lidar data.

In addition, the mapping device includes a lidar data pre-processing unit that removes a moving object excluding road information and facility information from the lidar data as noise and restores the removed part, and removes the moving object as noise from the color image. and a color image pre-processing unit for restoring the removed portion.

In addition, the mapping device may further include a lidar image pre-processor for pre-processing to enhance features of the lidar image generated by the lidar image generator.

In one embodiment, the lidar image pre-processing unit may merge the lidar image after applying a first filter to a plurality of divided images according to a distance from a vanishing point.

Also, the first filter is a bi-directional filter, and a different filter size may be applied to each of the divided images based on a distance from the vanishing point.

In one embodiment, the filter size is set smaller as the distance from the vanishing point increases.

In addition, the lidar image pre-processing unit may apply a second filter to the merged lidar image to enhance the feature, and then detect an edge in the lidar image.

In the present invention, in a mapping method of a mapping device for mapping color information of a color image to lidar data, a lidar image as a perspective image corresponding to a viewpoint of the color image is generated using the lidar data. image generating step; A color transfer step of color transferring the color information to the lidar image; and a color information mapping step of mapping the color information transferred to pixels of the lidar image to the lidar data corresponding to the lidar image.

The mapping method may include, before the lidar image generation step, a lidar data preprocessing step of removing a moving object excluding road information and facility information from the lidar data as noise and restoring the removed part; and a color image pre-processing step of removing the moving object as noise from the color image and restoring the removed part.

In addition, the mapping method may further include a lidar image preprocessing step of preprocessing to enhance features of the lidar image generated in the lidar image generating step.

In one embodiment, the lidar image preprocessing step includes generating split images by dividing the lidar images, applying a first filter to the split images and then merging them, and in the merged lidar images. Edge detection may be included.

In the generating of the divided images, the LIDAR image may be divided into a plurality of parts according to a distance from a vanishing point.

Also, the first filter is a bi-directional filter, and a different filter size may be applied to each of the divided images based on a distance from the vanishing point.

In the detecting of the edge, a second filter for enhancing the feature may be applied to the lidar image, and the edge may be detected after inverting the lidar image.

According to the present invention, it is possible to solve the conventional problem that it is difficult to match data of two sensors only with calibration information between a camera and a lidar.

According to the present invention, there is an effect of reducing the matching error by mapping the color of the camera image to lidar data by applying a color transfer technique.

Furthermore, LiDAR data to which color information is mapped by the mapping device according to the present invention can be easily utilized for simulation in virtual reality such as a digital twin.

1 is a diagram illustrating an apparatus for mapping lidar data and color information according to an embodiment of the present invention.
2 is a flowchart illustrating a method for mapping lidar data and color information according to an embodiment of the present invention.
3 is a flowchart illustrating a lidar image pre-processing process by a lidar image pre-processing unit of the lidar data and color information mapping apparatus according to an embodiment of the present invention.
4 is a diagram exemplarily illustrating a process of processing a lidar image processed by a lidar image pre-processing unit of an apparatus for mapping lidar data and color information according to an embodiment of the present invention.
5 is a diagram exemplarily illustrating a color transfer unit of an apparatus for mapping lidar data and color information according to an embodiment of the present invention.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. First, in adding reference numerals to components of each drawing, it should be noted that the same components have the same numerals as much as possible even if they are displayed on different drawings. In addition, in describing the present invention, if it is determined that a detailed description of a related known configuration or function may obscure the gist of the present invention, the detailed description will be omitted. In addition, although preferred embodiments of the present invention will be described below, the technical idea of the present invention is not limited or limited thereto and can be modified and implemented in various ways by those skilled in the art.

1 is a diagram illustrating an apparatus for mapping lidar data and color information according to an embodiment of the present invention.

In the present invention, the lidar data and color information mapping device 20 (hereinafter also referred to as a 'mapping device') uses lidar data and color images obtained from the mobile mapping system (MMS) 10 to Map color information to lidar data. MMS 10 may include lidar, cameras, GPS, and IMUs. In FIG. 1, the mapping device 20 is shown as receiving LIDAR data and camera images from the MMS 10, but in the practice of the present invention, the MMS 10 and the mapping device 20 are spatially separated may be provided. In addition, after the information acquired in the MMS 10 is stored in a separate storage device, the stored data may be transferred to the mapping device 20 .

In the present invention, lidar data may be lidar scan data scanned by lidar (LiDAR). LiDAR scan data may be referred to as point cloud data, that is, a point cloud. Also, a color image may be acquired using a camera. LiDAR data and color images may be obtained through a mobile mapping system (MMS) including a lidar and a camera.

Referring to FIG. 1 , a mapping device 20 according to an embodiment of the present invention includes a lidar image generating unit 40, a color transfer unit 70, and a color information mapping unit 80. The mapping device 20 may further include a lidar data preprocessing unit 30 for preprocessing lidar data (eg, point cloud). In addition, the mapping device 20 includes a lidar image pre-processing unit 50 for pre-processing the lidar image generated by the lidar image generator 40 and a color image pre-processing unit for pre-processing the color image acquired by the camera of the MMS. A portion 60 may be further included. In addition, the mapping device 20 may be configured to store lidar data including color information generated by the color information mapping unit 80 in a separate storage device (not shown).

It may be desirable to set the field of view (FOV) of lidar data and color images transmitted to the mapping device 20 to be as similar as possible. In an embodiment, a color image captured at a specific time or point of view and a point cloud around a location of the color image captured by a camera may be selected as lidar data. In one embodiment, it may be possible to transfer the color image and lidar data calibrated using the calibration information of the camera and lidar to the mapping device 20 . Even if the FOV of the lidar data and the color image are different, it may be possible to match the color information to the image according to the lidar data in the color transfer unit 70 to be described later. Processing time can be shortened.

The lidar data pre-processing unit 30 may remove noise from lidar data. The noise may be information other than roads or facilities that may be included in point cloud information. For example, moving objects such as pedestrians or vehicles may be removed as noise. In one embodiment, the lidar data preprocessing unit 30 performs preprocessing on lidar data by recognizing a moving object such as a vehicle by applying a deep learning-based object detection algorithm and removing the moving object from the point cloud. can do. In one embodiment, the lidar data pre-processing unit 30 restores bin information generated by removing a moving object from a point cloud using a recovery method such as multi-path based or space-time based temporal inpainting. can

The lidar image generator 40 generates a lidar image corresponding to a color image. In an embodiment, the lidar image generating unit 40 may generate a lidar image in consideration of a camera photographing viewpoint of a color image. Specifically, the lidar image generating unit 40 may generate a lidar image by using a point cloud at a time when the camera is photographed and GPS and IMU information.

In an embodiment, the lidar image generation unit 40 may convert lidar data into voxels using a point cloud, and project the voxel cube to match the camera's shooting point of view to generate lidar images. The LIDAR image may be a perspective view according to a camera shooting point of view. As an example of a method for generating a perspective image, first, GPS, IMU, and camera calibration parameters (Calibration parameters: Intrinsic/Extrinsic Matrix) are used to obtain placement information in an image sequence of the camera, and thus a voxel cube is converted into a camera The voxel cube is 2D mapped by projecting in correspondence with the shooting point.

The lidar image pre-processing unit 50 may enhance features by pre-processing the lidar image generated by the lidar image generating unit 40 . The lidar image pre-processing unit 50 may perform pre-processing to enhance intensity by applying image processing to the lidar image in order to effectively perform subsequent color transfer.

The color image pre-processing unit 60 may perform pre-processing on the transferred color image. Pre-processing of the color image may be performed by removing noise from the color image. In an embodiment, the color image pre-processing unit 60 may remove information other than roads or facilities included in the image, for example, moving objects such as pedestrians or vehicles as noise. In an embodiment, the color image preprocessing unit 60 may perform preprocessing of the color image by recognizing a moving object such as a vehicle by applying a deep learning-based object detection algorithm and removing the moving object from the color image. there is. In an embodiment, the color image pre-processing unit 60 may restore empty pixels in a color image generated by removing a moving object using a multi-path based or space-time based temporal inpainting restoration method. there is.

The color transfer unit 70 applies color transfer to the LIDAR image based on the color image. When a content image and a color image are given, color transfer maintains the main form of the image similar to the content image, while changing the color information to resemble the color image. say In the present invention, color information of a color image may be allocated to a lidar image by performing color transfer using a lidar image as a content image and a color image as a color image for color transfer. As an embodiment for color transfer, a deep learning network of an examiner-based colorization scheme may be used.

The color information mapping unit 80 maps the color information allocated in the color transfer to the point cloud used to generate the lidar image. When generating a perspective image in the LiDAR image generating unit 40, pixels of the perspective image are mapped to a point cloud. At least one point cloud is mapped to one pixel of the perspective image. If the perspective image is back-projected onto the point cloud using information used to generate the perspective image, color information may be mapped to each point of the point cloud.

2 is a flowchart illustrating a method for mapping lidar data and color information according to an embodiment of the present invention.

LiDAR data and color images obtained from MMS are prepared (S10, S20). LiDAR data can be presented as a point cloud.

Pre-processing, such as removing noise from lidar data, is performed by the lidar data pre-processing unit 30 (S12). Pre-processing of lidar data may include removing information other than roads or facilities that may be included in point cloud information and restoring information removed from the point cloud.

In order to generate a lidar image of the same viewpoint as the camera's viewpoint using lidar data, lidar data may be voxelized (S14).

The lidar image generation unit 40 generates a lidar image by projecting the voxel cube according to the camera's shooting point (S16). The LIDAR image may be a perspective view according to a camera shooting point of view.

After the lidar image is generated, a pre-processing process may be performed on the lidar image to enhance features of the lidar image (S18).

Meanwhile, a preprocessing process for the color image prepared in step S20 may be performed (S22). In one embodiment, the color image pre-processing unit 60 removes information other than roads or facilities included in the image, for example, moving objects such as pedestrians or vehicles, as noise, and removes the removed pixels as multi-path. It can be restored using a recovery method such as based or space-time based temporal inpainting.

The color transfer is performed by the color transfer unit 70 by using the lidar image as the content image and the color image obtained and processed by the camera as the color image (S30). In step S30, color information extracted from a color image is mapped to pixels of a lidar image, which is a perspective image generated by projecting a specific viewpoint from lidar data (ie, a point cloud).

By inversely transforming the lidar image into a 3-dimensional space, color information included in pixels of the lidar image is mapped to lidar data (S40). Accordingly, lidar data including color information may be provided.

In the foregoing description, it has been described that the lidar image pre-processing unit 50 enhances features by pre-processing the lidar image generated by the lidar image generating unit 40. The LiDAR image pre-processing unit 50 will be described in more detail.

3 is a flow chart illustrating a lidar image preprocessing process by the lidar image preprocessor 50 in the lidar data and color information mapping apparatus according to an embodiment of the present invention. 4 is a diagram exemplarily illustrating a process of processing a lidar image processed by the lidar image pre-processing unit 50 of the lidar data and color information mapping apparatus according to an embodiment of the present invention.

An example of a lidar image preprocessing process by the lidar image preprocessor 50 will be described.

The lidar image is generated according to the viewpoint of the color image by the lidar image generator 40 (see FIG. 4(a)).

The lidar image is divided into a predetermined number according to the distance from the vanishing point (S100). This is to maximize the noise removal effect by applying different filters according to the distance in the LIDAR image. In (b) of FIG. 4, it is exemplified that the LIDAR image is divided into three. In the case of an image including a road, the road is included up to about 2/3 from the bottom of the lidar image, and more than that may be a background such as the sky. Figure 4 (b) is presented in consideration of this situation. However, in the practice of the present invention, the number of divisions of the LIDAR image may be set differently according to the surrounding environment scanned by the LIDAR or photographed by the camera.

A first filter is applied to the segmented LiDAR image to remove noise (S110). In one embodiment, the first filter may be a bilateral filter that is a nonlinear filter that removes noise while preserving edges. In addition, a different filter size may be applied to each divided lidar image. In a lidar image having perspective, the closer the vanishing point is, the more ambiguous the information of the objects represented in the lidar perspective image can be distinguished from the road surface due to the smoothing effect accompanying noise removal. Accordingly, it may be desirable to set the filter size smaller as the distance increases. In addition, in the case of objects located close to each other, since the pixel area representing the object is widened, the noise removal effect can be maximized by increasing the filter size.

In the example of FIG. 4(b), a bidirectional filter with a small filter size is applied to the upper segmented image, a bidirectional filter with a larger filter size is applied to the middle segmented image, and a bidirectional filter with a smaller filter size is applied to the lower segmented image. A bidirectional filter with the largest filter size was applied.

After applying a first filter with different filter sizes to the divided lidar images, the divided images are merged (S120). (c) of FIG. 4 illustrates a merged LIDAR image after application of the first filter.

A second filter is applied to the merged LIDAR image (S130). A second filter may be applied to enhance features included in the lidar image. As an example, the second filter may be a Gaussian filter, and more specifically, an Extend Difference of Gaussian Filter may be applied. (d) of FIG. 4 illustrates a lidar image in which a feature is enhanced by applying an Extend Difference of Gaussian Filter.

Next, the LIDAR image is inverted for edge detection (S140). An inverted image is illustrated in FIG. 4(e).

Finally, an edge detection technique is applied (S150). As an example, a structured edge detection technique may be applied as an edge detection technique.

5 is a diagram exemplarily illustrating a color transfer unit 70 of an apparatus for mapping lidar data and color information according to an embodiment of the present invention, and FIG. 6 illustrates an image obtained by synthesizing a lidar image and a color image. It is an enemy drawing.

The lidar image (A1) and the color image (A2) are transferred to the color transfer unit 70, and the color transfer unit 70 transfers color information to the lidar image to obtain a lidar image (A3) including color information. generate In one embodiment, the color transfer unit 70 may use a deep learning network of an Emplar-based colorization scheme.

Referring to FIG. 6 , it can be exemplarily confirmed that a lidar image A1 and a color image A2 are synthesized by the color transfer unit 70 to generate a lidar image A3 including color information. there is.

The above description is merely an example of the technical idea of the present invention, and those skilled in the art can make various modifications, changes, and substitutions without departing from the essential characteristics of the present invention. will be. Therefore, the embodiments disclosed in the present invention and the accompanying drawings are not intended to limit the technical idea of the present invention, but to explain, and the scope of the technical idea of the present invention is not limited by these embodiments and the accompanying drawings. . The protection scope of the present invention should be construed according to the claims below, and all technical ideas within the equivalent range should be construed as being included in the scope of the present invention.

10: Mobile Mapping System (MMS)
20: mapping device
30: lidar data pre-processing unit
40: lidar image generating unit
50: lidar image pre-processing unit
60: color image pre-processing unit
70: color transfer unit
80: color information mapping unit

Claims (17)

A mapping device for mapping color information of a color image to lidar data,
a LiDAR image generating unit generating a lidar image as a perspective image corresponding to the viewpoint of the color image by using the lidar data;
a color transfer unit for color transferring the color information to the lidar image; and
a color information mapping unit for mapping the color information transferred to pixels of the lidar image to the lidar data corresponding to the lidar image;
Mapping device comprising a. According to claim 1,
The color transfer unit performs the color transfer on the color image by using the LIDAR image as a content image. According to claim 1,
The lidar data is a point cloud, and the lidar image generating unit converts the lidar data into voxels using the point cloud and projects the voxelized voxel cube to match the color image capturing time point. Mapping device, characterized in that for generating the lidar image. According to claim 1,
The color information mapping unit maps the color information to a point cloud constituting the lidar data. According to claim 1,
A lidar data preprocessing unit that removes moving objects excluding road information and facility information from the lidar data as noise and restores the removed parts, and
and a color image pre-processor configured to remove the moving object as noise from the color image and restore the removed part. According to any one of claims 1 to 5,
The mapping device further comprises a lidar image pre-processor for pre-processing to enhance features of the lidar image generated by the lidar image generator. According to claim 6,
The lidar image pre-processing unit applies a first filter to the divided images divided into a plurality according to the distance from the vanishing point and then merges the lidar image. According to claim 7,
The first filter is a bi-directional filter, and a different filter size is applied to each of the divided images based on a distance from the vanishing point. According to claim 8,
The filter size is set smaller as the distance from the vanishing point increases. According to claim 7,
The lidar image pre-processing unit applies a second filter to the merged lidar image to enhance the feature, and then detects an edge in the lidar image. In the mapping method of a mapping device for mapping color information of a color image to lidar data,
a lidar image generation step of generating a lidar image as a perspective image corresponding to a viewpoint of the color image by using the lidar data;
A color transfer step of color transferring the color information to the lidar image; and
A color information mapping step of mapping the color information transferred to pixels of the lidar image to the lidar data corresponding to the lidar image;
Mapping method including. According to claim 11,
Prior to the LiDAR image generation step,
A lidar data preprocessing step of removing moving objects excluding road information and facility information from the lidar data as noise and restoring the removed parts; and
a color image pre-processing step of removing the moving object as noise from the color image and restoring the removed part;
A mapping method further comprising a. According to claim 11 or 12,
A lidar image preprocessing step of preprocessing to enhance features of the lidar image generated in the lidar image generating step;
A mapping method further comprising a. According to claim 13,
The lidar image preprocessing step,
Generating a segmented image by segmenting the LIDAR image;
merging after applying a first filter to the divided images; and
Detecting an edge in the merged lidar image
A mapping method comprising a. 15. The method of claim 14,
The generating of the divided images may include dividing the LIDAR image into a plurality of parts according to a distance from a vanishing point. 15. The method of claim 14,
The first filter is a bi-directional filter, and a different filter size is applied to each of the divided images based on a distance from the vanishing point. 15. The method of claim 14,
The detecting of the edge may include applying a second filter for enhancing the feature to the lidar image, inverting the lidar image, and then detecting the edge.

Images