CN111462135A - Semantic Mapping Method Based on Visual SLAM and 2D Semantic Segmentation - Google Patents

Abstract

The invention relates to the field of cross fusion of computer vision and deep learning, and more particularly, to a semantic mapping method based on visual SLAM and two-dimensional semantic segmentation. The method of the present invention includes the following steps: S1, calibrating camera parameters, and correcting camera distortion; S2, acquiring image frame sequence; S3, image preprocessing; S4, judging whether the current image frame is a key frame, and if so, go to the step S6, if not, go to step S5; S5, dynamic blur compensation; S6, semantic segmentation, extract ORB feature points for the image frame, and use the mask area convolutional neural network algorithm model to perform semantic segmentation; S7, pose Calculate, use the sparse SLAM algorithm model to calculate the camera pose; S8, assist the semantic information to construct a dense semantic map to realize the three-dimensional semantic map of the global point cloud map. The invention can improve the performance of the UAV semantic mapping system, and significantly improve the robustness of extracting and matching feature points for dynamic scenes.

Description

Semantic Mapping Method Based on Visual SLAM and 2D Semantic Segmentation

technical field

The invention relates to the field of cross fusion of computer vision and deep learning, and more specifically, to a semantic mapping method based on visual SLAM and two-dimensional semantic segmentation.

Background technique

UAVs are generally composed of three modules: intelligent decision-making, environmental perception, and motion control, of which environmental perception is the basis of everything.

For drones to perceive the surrounding environment, a stable and powerful sensor system is needed to act as "eyes", and corresponding algorithms and powerful processing units are needed to "read objects".

In the environmental perception module of the drone, the visual sensor is an indispensable part. The visual sensor can be a camera. Compared with lidar and millimeter-wave radar, the resolution of the camera is higher and can obtain sufficient environmental details. Describe the appearance and shape of objects, read logos, etc.

Although the Global Positioning System (GPS) helps the positioning process, the visual sensor cannot be replaced by the GPS system because of the interference caused by tall trees, buildings, tunnels, etc., which can make GPS positioning unreliable.

Simultaneous Localization and Mapping (SLAM) refers to the fact that a subject carrying a specific sensor can estimate the trajectory of its own motion by calculating the image frame obtained by a specific sensor without prior information, and establish a map of the surrounding environment. It is widely used in robotics, drones, autonomous driving, augmented reality, virtual reality and other applications.

SLAM can be divided into two categories: laser SLAM and visual SLAM.

Due to its early start, laser SLAM is relatively mature in theoretical technology and engineering applications, but laser SLAM has a fatal disadvantage in the application of robots, that is, the structural information of lidar intelligent perception is two-dimensional information, and the amount of information is small, resulting in A lot of environmental information is lost. At the same time, its high cost, huge volume and lack of semantic information make it limited in some specific application scenarios.

The perceptual information source of visual SLAM is the camera image.

According to the camera type, visual SLAM can be divided into three types: monocular, binocular, and depth SLAM. Similar to lidar, depth cameras can directly calculate distances to obstacles by collecting point clouds. The depth camera has a simple structure, is easy to install and operate, and has low cost and a wide range of usage scenarios.

With the rise of deep learning, visual SLAM has also made great progress in recent years.

Most of the visual SLAM solutions are at the feature point or pixel level. In order to complete a specific task or intelligently interact with the surrounding environment, UAVs need to obtain semantic information.

The visual SLAM system can select useful information and eliminate invalid information.

With the development of deep learning, many mature object detection and semantic segmentation methods provide conditions for accurate semantic mapping. Semantic maps are conducive to improving the autonomy and robustness of UAVs, completing more complex tasks, and transforming from path planning to mission planning.

With the improvement of hardware computing power and the optimization of algorithm structure, deep learning has made more and more remarkable achievements.

A huge leap has been made in the field of computer vision. As far as RGB image segmentation is concerned, it can be roughly divided into object detection and semantic segmentation.

In the early stage, the main target detection framework was proposed, which achieved more and more accurate target detection.

The mainstream deep learning target detection framework is mainly based on CNN (Convolutional Neural Networks, convolutional neural network), among which the more efficient are the YOLO (You Only Look Once, you only need to look once) series and R-CNN (Region-CNN, Regional Convolutional Neural Networks) series.

Object perception technology in 3D images is becoming more and more mature, and the demand for 3D understanding is becoming more and more urgent. Due to the irregularity of point clouds, most researchers convert the points into regular voxel or grid models and use deep neural networks for prediction.

The direct semantic segmentation of point cloud space consumes a lot of computing resources, and the relationship between spatial points is weakened.

PointNet, proposed in 2017, is the first deep neural network that can directly process raw 3D point clouds.

The dense mapping method adopted by most of the existing visual SLAM systems lacks semantic information and cannot meet the requirements of intelligence.

A typical assumption of the visual SLAM algorithm is that the scene is fixed, and the appearance of some dynamic objects not only affects the estimation of the camera pose but also leaves an afterimage in the map, which affects the quality of the map.

The photos captured by the camera in the case of high-speed motion are easy to be blurred, which greatly affects the extraction and matching of feature points.

SUMMARY OF THE INVENTION

The purpose of the present invention is to provide a semantic mapping method based on visual SLAM and two-dimensional semantic segmentation, so as to solve the technical problem that high-speed moving dynamic objects affect the quality of the established map.

In order to achieve the above object, the present invention provides a semantic mapping method based on visual SLAM and two-dimensional semantic segmentation, comprising the following steps:

S1. Calibrate camera parameters and correct camera distortion;

S2, obtain an image frame sequence, the image frame sequence includes an RGB image and a depth image;

S3, image preprocessing, using the pinhole camera model to obtain the coordinates of the real three-dimensional space point corresponding to each pixel point in the RGB image;

S4, determine whether the current image frame is a key frame, if so, then go to step S6, if not, then go to step S5;

S5, dynamic blur compensation, calculating the centroid of the image block of the current image frame as a semantic feature point, as a supplement to the ORB feature point;

S6, semantic segmentation, extracting ORB feature points for the image frame, using the mask area convolutional neural network algorithm model to perform semantic segmentation, and obtaining the semantic information of each pixel of the frame image;

S7, pose calculation, use the sparse SLAM algorithm model to calculate the camera pose;

S8. Input the semantic information into the sparse SLAM algorithm model, assist the construction of the dense semantic map, complete the traversal of key frames, and realize the three-dimensional semantic map of the global point cloud map.

In one embodiment, the correction of camera distortion in step S1 further includes the following steps:

S11, project the three-dimensional space point P(X, Y, Z) of the camera coordinate system to the normalized image plane to form the normalized coordinates of the point as [x, y] ^T ;

S12. Perform radial distortion and tangential distortion correction on the point [x, y] ^T on the normalized plane, which is achieved by the following formula:

Among them, [x _corrected , y _corrected ] ^T is the corrected point coordinates, p ₁ , p ₂ are the tangential distortion coefficients of the camera, k ₁ , k ₂ , k ₃ are the radial distortion coefficients of the camera, and r is the point P the distance from the origin of the coordinate system;

S13. Project the corrected point [x _corrected ,y _corrected ] ^T through the internal parameter matrix to the pixel plane to obtain its correct position [u,v] ^T on the image, which is achieved by the following formula:

Among them, f _x , f _y , c _x , and _cy are the internal parameters of the camera.

In one embodiment, the image preprocessing in step S3 further includes that the mapping relationship between the pixel point [u, v] ^T to the real three-dimensional space point P(X, Y, Z) satisfies the following formula:

Among them, K is called the internal parameter matrix, f _x , f _y , c _x , _cy are the internal parameters of the camera, P is the real three-dimensional space point coordinates, [u, v] ^T is the pixel point coordinates.

In one embodiment, the key frame of step S4 is screened using a sparse SLAM algorithm model.

In one embodiment, the centroid of the image block in step S5 is obtained through the following steps:

Label each object in the frame image as a specific class;

For each segmented object, there is a corresponding marked area, and the segmented image is called an image block;

p,q＝{0,1}；Calculate moments of image patches

p,q={0,1};

Calculate the corresponding centroid C as a semantic feature point to supplement the ORB feature point, where

In one embodiment, the step S6 further includes:

The semantic information of each pixel includes semantic classification label, bounding box coordinates and confidence score of the classification;

Based on the results of semantic segmentation, the ORB feature points extracted from the regions that specify a certain category as dynamic objects are eliminated.

In one embodiment, the Mask R-CNN algorithm model of described step S6 carries out semantic segmentation, and further comprises:

Extract the features at different levels of the input image through the feature map pyramid network;

Propose proposals of interest through a region generation network;

Proposal area alignment using ROI alignment;

Mask segmentation using fully convolutional networks;

The region coordinate determination and the class classification are performed using fully connected layers.

In one embodiment, the sparse SLAM algorithm model further includes a tracking thread, a local mapping thread, and a loop closure detection thread:

Described tracking thread, by looking for the local map feature to be matched, utilize pure motion beam adjustment method to minimize the reprojection error to locate the camera of each frame of picture;

The local map building thread manages and optimizes the local map by executing the local beam adjustment method, maintains the co-view relationship between key frames through map points, and optimizes the co-view key frame pose and map point through the local beam adjustment method. ;

The loop closure detection thread detects large loops and corrects drift errors by performing pose graph optimization, accelerates the screening of closed-loop matching frames, optimizes scale, and optimizes essence maps and map points by global beam adjustment.

In one embodiment, the sparse SLAM algorithm model further includes a global beam adjustment method optimization thread, which is triggered after the loop closure detection thread is confirmed, and after the pose graph optimization, calculates the optimal structure and motion results of the entire system.

In one embodiment, the pose calculation in step S7 further includes: initially calculating the camera pose through PnP solution, using the back-end pose graph to optimize the calculation of the camera pose, and constructing a minimized reprojection error of the camera pose estimation :

Among them, _ui is the pixel coordinate, P _i is the camera coordinate, ξ^ is the Lie algebra corresponding to the camera pose, _si is the depth of the feature point, and K is the camera internal parameter matrix.

The present invention provides a semantic mapping method based on visual SLAM and two-dimensional semantic segmentation, based on the Mask R-CNN algorithm model of ORB feature points and the sparse SLAM algorithm model, establishes a dense semantic map for eliminating dynamic objects, and uses inter-frame information. And semantic information on image frames to improve the performance of UAV semantic mapping system, and improve the robustness of feature point extraction and matching for dynamic scenes.

Description of drawings

The above and other features, properties and advantages of the present invention will become more apparent from the following description taken in conjunction with the accompanying drawings and embodiments, wherein like reference numerals refer to like features throughout, wherein:

FIG. 1 discloses a flow chart of a method according to an embodiment of the present invention;

FIG. 2 discloses a calibration board for camera calibration according to an embodiment of the present invention;

Fig. 3a discloses a pinhole imaging model diagram of a pinhole camera according to an embodiment of the present invention;

Fig. 3b discloses a similar triangular schematic diagram of a pinhole camera according to an embodiment of the present invention;

FIG. 4 discloses a system flowchart of Mask RCNN according to an embodiment of the present invention.

Detailed ways

In order to make the objects, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the invention, but not to limit the invention.

The concept of semantic map refers to a map containing rich semantic information, which represents the abstraction of semantic information such as spatial geometric relationship and existing object types and locations in the environment. Semantic maps contain both the spatial information of the environment and the semantic information of the environment, so that the mobile robot, like a human, knows both the objects in the environment and what the objects are.

In view of the problems and deficiencies in the prior art, the present invention proposes a semantic mapping system based on visual SLAM and two-dimensional semantic segmentation, and uses feature points based on ORB (Oriented FAST and Rotated BRIEF, fast orientation and brief rotation) for semantic segmentation, Combined with the sparse SLAM algorithm model, semantic mapping is completed while positioning is achieved.

1 shows a flowchart of a semantic mapping method based on visual SLAM and two-dimensional semantic segmentation according to an embodiment of the present invention. In the embodiment shown in FIG. 1 , the present invention proposes a method based on visual SLAM and two-dimensional semantic segmentation. Semantic mapping method, the specific steps are as follows:

S1. Calibrate camera parameters and correct camera distortion;

S2, obtain an image frame sequence, the image frame sequence includes an RGB image and a depth image;

S3, image preprocessing, using the pinhole camera model to obtain the coordinates of the real three-dimensional space point corresponding to each pixel point in the RGB image;

S4, determine whether the current image frame is a key frame, if so, then go to step S6, if not, then go to step S5;

S5, dynamic blur compensation, calculating the centroid of the image block of the current image frame as a semantic feature point, as a supplement to the ORB feature point;

S6. Semantic segmentation, extracting ORB feature points for the image frame, using the Mask R-CNN algorithm model to perform semantic segmentation, and obtaining the semantic information of each pixel of the frame image;

S7, pose calculation, use the sparse SLAM algorithm model to calculate the camera pose;

S8. Input the semantic information into the sparse SLAM algorithm model, assist the construction of the dense semantic map, complete the traversal of key frames, and realize the three-dimensional semantic map of the global point cloud map.

Each step is described in detail below.

Step S1: calibrate camera parameters and correct camera distortion.

In the process of image measurement and machine vision applications, in order to determine the relationship between the three-dimensional geometric position of a point on the surface of a space object and its corresponding point in the image, a geometric model of camera imaging must be established, and these geometric model parameters are camera parameters.

The distortion coefficient is one of the camera parameters, which corresponds to the camera distortion phenomenon. Under most conditions, these camera parameters must be obtained through experiments and calculations, and this process of solving parameters is called camera calibration (or camera calibration).

Camera distortion includes radial distortion and tangential distortion.

The radial distortion is caused by the lens shape.

More specifically, in the pinhole model, a straight line projected onto the pixel plane is still a straight line.

However, in actual photos, the lens of the camera often turns a straight line in the real environment into a curve in the picture, and this distortion is called radial distortion.

The tangential distortion is formed during the assembly process of the camera because the lens and the imaging plane cannot be strictly parallel.

Due to ray projection, the actual object is inconsistent with the image projected to the 2D plane. This inconsistency is stable, and the distortion correction of subsequent images can be achieved by calibrating the camera and calculating the distortion parameters.

For radial distortion, correct with quadratic and higher-order polynomial functions related to the distance from the center:

Among them, [x,y] ^T is the coordinate of the uncorrected point, [x _corrected ,y _corrected ] ^T is the corrected point coordinate, k ₁ , k ₂ , k ₃ are the radial distortion coefficients of the camera, and r is the point The distance of P from the origin of the coordinate system.

For tangential distortion, two other parameters p ₁ , p ₂ can be used to correct:

Among them, [x,y] ^T is the coordinate of the uncorrected point, [x _corrected ,y _corrected ] ^T is the corrected point coordinate, p ₁ , p ₂ are the tangential distortion coefficients of the camera, and r is the point P away from the coordinate The distance from the origin.

Before the camera is used, by calibrating the radial distortion coefficient and tangential distortion coefficient of the camera, the three-dimensional information is obtained from the two-dimensional image, and the image distortion correction, object measurement, and three-dimensional reconstruction are realized.

Fig. 2 discloses a calibration plate for camera calibration according to an embodiment of the present invention. The calibration plate shown in Fig. 2 is placed within the visible range of the camera. From the feature points in the image, the internal parameters and external parameters of the camera are obtained, and then the distortion coefficients are obtained.

Preferably, the Camera Calibrator (camera calibration) toolbox in MATLAB is used to solve the camera parameters.

For the point P (X, Y, Z) in the camera coordinate system, step S1 of the present invention performs camera distortion correction through 5 distortion coefficients to find the correct position of this point on the pixel plane.

The steps to correct camera distortion are as follows:

S11. Project the three-dimensional space point to the normalized image plane. Let its normalized coordinates be [x,y] ^T .

S12. Perform radial distortion and tangential distortion correction on the points on the normalized plane.

Among them, [x _corrected , y _corrected ] ^T is the corrected point coordinates, p ₁ , p ₂ are the tangential distortion coefficients of the camera, k ₁ , k ₂ , k ₃ are the radial distortion coefficients of the camera, and r is the point P The distance from the origin of the coordinate system.

S13. Project the corrected point [x _corrected , y _corrected ] ^T to the pixel plane through the internal parameter matrix to obtain the correct position coordinates [u, v] ^T of the point on the image.

Among them, f _x , f _y , c _x , and _cy are the internal parameters of the camera.

Step S2, acquiring a sequence of image frames.

The RGB-D image frame sequence is obtained by using the Kinect camera, and the image frame sequence includes RGB image and depth image.

Step S3, image preprocessing

In one embodiment, an RGB-D camera is used as the main sensor to simultaneously obtain an RGB image and a depth image, and a pinhole camera model is used to map the pixels of the RGB image to the real three-dimensional space.

Fig. 3a discloses a pinhole imaging model diagram of a pinhole camera according to an embodiment of the present invention, and Fig. 3b discloses a similar triangular schematic diagram of a pinhole camera according to an embodiment of the present invention, as shown in Figs. 3a and 3b, Establish the camera coordinate system O-x-y-z, take the position of the optical center of the camera as the origin O of the coordinate system, and agree that the direction of the arrow is the positive direction.

Through the mapping transformation of similar triangles shown in Figure 3b, a coordinate system O'-x'-y'-z' is established on the imaging plane of the camera, and it is agreed that the direction of the arrow is positive.

Assuming that the coordinates of point P are [X, Y, Z] ^T , the focal length of the camera lens is f, and the focal length is the distance from the optical center of the camera to the physical imaging plane.

The point P is projected through the optical center to the point P' of the imaging plane, and the coordinates of the pixel point P' are [u, v] ^T .

According to the corresponding relationship, the mapping relationship corresponds to the amount of scaling and translation of a scale, and the derivation can be obtained:

Among them, K is called the camera internal parameter matrix, which is an intrinsic parameter, which has been calibrated in step S1, f _x , f _y , c _x , _cy are the internal parameters of the camera, P is the real three-dimensional space point coordinates, [u, v] ^T is the pixel coordinate.

Step S4, judge whether it is a key frame, if yes, then go to step S6, if not, then go to step S5;

If each frame of image is used for visual SLAM and semantic segmentation calculation, the amount of calculation is too large, therefore, the one with high quality is selected as the key frame.

In the present invention, a sparse SLAM algorithm model based on ORB (Oriented FAST and Rotated BRIEF, fast orientation and brief rotation) feature points is used to screen key frames.

Each keyframe contains an RGB image and a depth image.

Step S5, dynamic blur

Since there may be dynamic objects in each frame of image, each time the semantic mapping task is performed, certain kinds of objects are designated as dynamic objects. In the image sequence, if the dynamic target is identified in the frame image, the present invention will remove the corresponding point cloud when the two-dimensional pixel point is converted to the three-dimensional space coordinate, so as to prevent the dynamic object from leaving afterimages in the map. , which affects the quality of the map.

In step S5 of the present invention, if this frame image is not a key frame, because of motion blur, ORB feature point extraction is insufficient, before the image semantic segmentation step of step S6, carry out the following operations as a supplement:

Each object in the frame image is marked as a specific class, and there is a corresponding marked area for each segmented object. The segmented image is called an image block, and the moment m _pq of the image block B is calculated:

The centroid position C is:

The centroid serves as a semantic feature point to supplement the insufficiency of ORB feature points.

For the blurred images with serious loss of ORB feature points, the semantic feature points are supplemented, and the tracking algorithm is suppressed from using the matching belonging to the dynamic objects, and then the key frames are comprehensively screened, and the camera pose is estimated to prevent the mapping algorithm from including the moving objects as 3D maps. a part of.

Step S6, Semantic Segmentation

The ORB feature points are extracted for each image frame, and the Mask RCNN (Mask Region-CNN, mask region convolutional neural network) algorithm model is used to perform semantic segmentation to obtain the semantic information of each pixel of the frame image.

Based on the results of semantic segmentation, if a dynamic object is identified, the ORB feature points extracted from the region that specifies a certain category as a dynamic object are eliminated.

Suppressed vision SLAM algorithms include moving objects as part of the 3D map during the mapping process.

In step S6 of the present invention, the Mask RCNN algorithm model adopts the COCO data set for training.

The full name of COCO is Common Objects in COntext. It is a data set provided by the Microsoft team that can be used for image recognition. Classification information of 80 categories can be obtained.

FIG. 4 discloses a system flowchart of Mask RCNN according to an embodiment of the present invention. As shown in FIG. 4 , RGB image semantic segmentation of image frames is implemented based on the Mask R-CNN algorithm model. Convolutional neural network framework, the steps for semantic segmentation are as follows:

Extract features at different levels of the input image through FPN (Feature Pyramid Networks);

Propose proposals of interest through RPN (Region Proposal Network);

Use RoI Align (Region of Interest Align, region of interest alignment) to align proposal regions;

Use FCN (Fully Convolutional Networks, fully convolutional network) for mask segmentation;

Use FC (Fully Connected Layers) to determine the regional coordinates and classify the category.

The frame image is processed by the Mask RCNN algorithm model to generate pixel-level semantic classification results, that is, the semantic classification label of each pixel point, and output the bounding box coordinates and the confidence score of the classification.

The present invention uses ORB feature points for tracking, mapping and position recognition tasks. The advantages of ORB feature points are that they have rotation invariance and scale invariance, and can quickly extract features and perform matching, which can meet the needs of real-time operations, and can be used in real-time operations. The bag-of-words-based position recognition process shows good accuracy.

S7 pose calculation

The estimation of the visual odometry pose is for two adjacent frames of images. It is not difficult to understand that the accumulation of multiple such inter-frame pose estimates is the motion trajectory of the camera.

The camera pose is calculated using a sparse SLAM algorithm model based on ORB (Oriented FAST and Rotated BRIEF, fast orientation and brief rotation) feature points.

After extracting image frame feature points, PnP is used to estimate the camera pose based on key frames.

PnP is the abbreviation of Perspective-n-Point (n-point perspective), which is a method to solve the motion of 3D to 2D point pairs: that is, how to solve the pose of the camera when n 3D space points and their projection positions are given.

Assuming that at time k, the camera's position is x _k , the camera input data is u _k , and w _k is noise, construct the motion equation:

x _k =f(x _k-1 , _{uk , w k )} .

The landmark point y _j is observed at the position of x _k , a series of observation data z _k,j are generated, v _k,j is the observation noise, and the observation equation is constructed:

z _k,j =h(y _j ,x _k ,v _k,j ).

In step S7 of the present invention, the camera pose can be preliminarily calculated by solving the PnP problem, and then a more accurate camera pose can be further calculated by using the back-end pose graph optimization.

In step S7 of the present invention, the PnP problem of camera pose estimation is constructed as a nonlinear least squares problem on Lie algebra of definition domain.

Further, the camera pose estimation in step S7 of the present invention is constructed as a BA (Bundle Adjustment, beam adjustment method) problem, and the minimum reprojection error of the camera pose estimation is constructed:

Among them, _ui is the pixel coordinate, P _i is the camera coordinate, ξ^ is the Lie algebra corresponding to the camera pose, s _i is the depth of the feature point, K is the camera internal parameter matrix, and n is the number of points.

Further, step S7 of the present invention further includes, using a DBOW (Direct index Bag of words, bag of words model) embedded location recognition model for relocation to prevent tracking failure, or re-initialization of known map scenes, loopback detection, etc. .

In the present invention, the sparse SLAM algorithm model is used to screen key frames and calculate the camera pose. The sparse SLAM algorithm model is in ORB-SLAM2 (Oriented FAST and Rotated BRIEF- Simultaneous Localization and Mapping 2, the second generation of fast guidance and brief Rotation real-time positioning and map construction) are improved on the basis of.

The SLAM algorithm model consists of 4 parallel threads, including a tracking thread, a local mapping thread, a loopback detection thread and a global BA optimization thread.

Furthermore, the global BA optimization thread is executed only after the loopback detection thread has confirmed it.

The first three threads are parallel threads, which are defined as follows:

1) Trace the thread.

By finding matches to local map features, pure motion BA is used to minimize the reprojection error to locate the camera of each frame.

Preferably, a constant velocity model is used for matching.

2) Local mapping thread.

The local map is managed and optimized by performing local BA, the co-view relationship between key frames is maintained through MapPoints (map points), and the co-view key frame pose and MapPoints are optimized through local BA.

3) Loopback detection thread.

Detect large loops and correct drift errors by performing pose graph optimization, speed up the screening of closed-loop matching frames by Bow, optimize scale by Sim3, and optimize Essential Graph (essential graph) and MapPoints by global BA. The Sim3 transformation is the similarity transformation.

The loopback detection thread triggers the global BA optimization thread.

The global BA thread, after the pose graph optimization, calculates the optimal structure and motion results of the entire system.

Compared with the dense SLAM algorithm model of the prior art, in the sparse SLAM algorithm model of the present invention, through the fusion of semantic information, in the final mapping process, rich semantic segmentation information of the image is added.

Step S8, 3D semantic mapping

Using the semantic segmentation result of step S6, combined with the inter-frame pose information obtained in step S7 and the real three-dimensional coordinates of the pixel points of the image frame, the semantic information is input into the sparse SLAM algorithm model, and the same kind of object semantically contained in the frame image, The same label color is projected into the 3D point cloud map to assist in the construction of dense semantic maps, complete the traversal of key frames, and realize 3D semantic map construction of the global point cloud map.

Step S8 of the present invention further comprises:

S81. Project the three-dimensional space pixels generated by the key frame of the first frame into an initial point cloud;

S82, generate a point cloud map through the three-dimensional space coordinates corresponding to each pixel point of the current key frame calculated by the pinhole model;

S83. Calculate the pose change between the current key frame and the previous key frame;

S84. The two point cloud maps are superimposed and fused with three-dimensional coordinate points through the pose transformation matrix to generate a point cloud map with more information;

S85. The above steps are continuously iterated. When the traversal of all key frames is completed, the construction of the global point cloud map is realized.

The experimental results of the UAV semantic mapping method based on visual SLAM and two-dimensional semantic segmentation of the present invention will be further described in detail below in conjunction with specific experiments.

This experiment is based on the operating system Ubuntu16.04 and the hardware graphics card Nvidia Geforce GTX 1050. With the help of software tools such as Tensorflow, OpenCV, g2o, and Point Cloud Library, the real scene is used as the experimental condition, and the real shooting data of the Kinect V1 camera is used.

For 3D semantic mapping evaluation, Q ₁ represents the number of correctly detected objects, Q ₂ represents the number of objects detected but classified incorrectly and the number of objects that actually exist but not detected, Q ₃ represents the number of detected objects without objects, and P represents the number of detected results The correct detection rate of 3D objects is calculated as follows:

P=Q ₁ /(Q ₁ +Q ₂ +Q ₃ )

Through 9 times of dense semantic mapping for experimental records, the average correct detection rate of 3D objects in the map is calculated to be 48.1086%. The specific experimental results are shown in the following table:

Table 1

The present invention provides a semantic mapping method based on visual SLAM and two-dimensional semantic segmentation, based on the Mask R-CNN algorithm model of ORB feature points and the sparse SLAM algorithm model, establishes a dense semantic map for eliminating dynamic objects, and uses inter-frame information. And semantic information on image frames to improve the performance of UAV semantic mapping system, and improve the robustness of feature point extraction and matching for dynamic scenes.

Although the above-described methods are illustrated and described as a series of acts for simplicity of explanation, it should be understood and appreciated that these methods are not limited by the order of the acts, as some acts may occur in a different order in accordance with one or more embodiments and/or occur concurrently with other actions from or not shown and described herein but understood by those skilled in the art.

As shown in this application and in the claims, unless the context clearly dictates otherwise, the words "a", "an", "an" and/or "the" are not intended to be specific in the singular and may include the plural. In general, the terms "comprising" and "comprising" only imply that the clearly identified steps and elements are included, and these steps and elements do not constitute an exclusive list, and the method or apparatus may also include other steps or elements.

The above-mentioned embodiments are provided for those skilled in the art to realize or use the present invention. Those skilled in the art can make various modifications or changes to the above-mentioned embodiments without departing from the inventive concept of the present invention. The protection scope of the present invention is not limited by the above-mentioned embodiments, but should be the maximum scope conforming to the innovative features mentioned in the claims.

Claims (10)

1. A semantic mapping method based on visual S L AM and two-dimensional semantic segmentation is characterized by comprising the following steps:

s1, calibrating camera parameters, and correcting camera distortion;

s2, acquiring an image frame sequence, wherein the image frame sequence comprises an RGB image and a depth image;

s3, preprocessing the image, and obtaining the coordinates of a real three-dimensional space point corresponding to each pixel point in the RGB image by adopting a pinhole camera model;

s4, judging whether the current image frame is a key frame, if so, turning to the step S6, and if not, turning to the step S5;

s5, dynamic fuzzy compensation, wherein the centroid of the image block of the current image frame is obtained through calculation and is used as a semantic feature point and is used as a supplement of an ORB feature point;

s6, semantic segmentation, namely, extracting ORB feature points of the image frame, and performing semantic segmentation by using a mask region convolution neural network algorithm model to obtain semantic information of each pixel point of the image frame;

s7, calculating the pose, and calculating the pose of the camera by using a sparse S L AM algorithm model;

and S8, inputting semantic information into the sparse S L AM algorithm model, assisting the construction of a dense semantic map, completing the traversal of a key frame, and realizing the three-dimensional semantic map construction of the global point cloud map.

2. The semantic mapping method based on visual S L AM and two-dimensional semantic segmentation according to claim 1, wherein the step S1 corrects camera distortion, further comprising the steps of:

s11, projecting the three-dimensional space point P (X, Y, Z) of the camera coordinate system to the normalized image plane to form the normalized coordinate of the point as [ X, Y]^T；

S12, normalizing points [ x, y ] on the plane]^TPerforming radial distortion and tangential distortion correction byThe formula is realized as follows:

wherein, [ x ]_corrected,y_corrected]^TIs the corrected point coordinate, p₁，p₂Is the tangential distortion coefficient, k, of the camera₁，k₂，k₃Is the radial distortion coefficient of the camera, r is the distance of the point P from the origin of the coordinate system;

s13, correcting the point [ x ]_corrected,y_corrected]^TProjecting to a pixel plane to obtain the correct position [ u, v ] of the pixel plane on the image through an internal parameter matrix]^TThe method is realized by the following formula:

wherein f is_x，f_y，c_x，c_yIs an intrinsic parameter of the camera.

3. The semantic mapping method based on visual S L AM and two-dimensional semantic segmentation as claimed in claim 2, wherein the image preprocessing of step S3 further comprises pixel points [ u, v [ ]]^TThe mapping to the true three-dimensional spatial point P (X, Y, Z) satisfies the following formula:

where K is called the in-camera parameter matrix, f_x，f_y，c_x，c_yIs the intrinsic parameter of the camera, P is the real three-dimensional space point coordinate, [ u, v]^TIs the pixel point coordinate.

4. The semantic mapping method based on visual S L AM and two-dimensional semantic segmentation as claimed in claim 1, wherein the key frame of step S4 is filtered by using sparse S L AM algorithm model.

5. The semantic mapping method based on visual S L AM and two-dimensional semantic segmentation according to claim 1, wherein the centroids of the image blocks of step S5 are obtained by:

marking each object of the frame image as a specific class;

each divided object is provided with a corresponding label area, and the divided image is called an image block;

computing moments of image blocks

p,q＝{0,1}；

Calculating corresponding centroid C as semantic feature point to supplement ORB feature point, wherein

6. The semantic mapping method based on visual S L AM and two-dimensional semantic segmentation according to claim 1, wherein the step S6 further comprises:

the semantic information of each pixel point comprises a semantic classification label, an enclosure coordinate and a confidence score of the classification;

and based on the semantic segmentation result, eliminating ORB characteristic points extracted from the region of which a certain category is designated as the dynamic object.

7. The semantic mapping method based on visual S L AM and two-dimensional semantic segmentation of claim 1, wherein the Mask R-CNN algorithm model of step S6 is used for semantic segmentation, further comprising:

extracting features on different levels of the input image through a feature map pyramid network;

an interesting proposal is proposed through a regional generation network;

carrying out proposal area alignment by using the arrangement of the interested areas;

performing mask segmentation by using a full convolution network;

and utilizing the full connection layer to determine the region coordinates and classify the categories.

8. The semantic mapping method based on visual S L AM and two-dimensional semantic segmentation according to claim 1 or claim 4, wherein the sparse S L AM algorithm model further comprises a tracking thread, a local mapping thread, a loop detection thread:

the tracking thread is used for locating the camera of each frame of picture by searching and matching the local map features and minimizing the reprojection error by using a pure motion beam adjustment method;

the local map building thread manages and optimizes a local map by executing a local light beam adjustment method, maintains the common-view relation among key frames through map points, and optimizes the pose of the common-view key frame and the map points through the local light beam adjustment method;

and the loop detection thread detects a large loop, corrects drift errors by executing pose graph optimization, accelerates screening of closed loop matching frames, optimizes dimensions and optimizes essential graph and map points by a global beam adjustment method.

9. The semantic mapping method based on visual S L AM and two-dimensional semantic segmentation as claimed in claim 1, wherein the sparse S L AM algorithm model further comprises a global beam-balancing optimization thread, which is triggered after the loop detection thread is confirmed, and after the pose map is optimized, the optimal structure and motion result of the whole system is calculated.

10. The semantic mapping method based on vision S L AM and two-dimensional semantic segmentation as claimed in claim 1, wherein the pose calculation of step S7 further comprises solving preliminary calculated camera pose by PnP, optimizing calculated camera pose by using rear-end pose map, constructing minimized re-projection error of camera pose estimation:

wherein u is_iIs a pixel coordinate, P_iξ is the corresponding lie algebra of the camera pose, s_iAnd K is an intra-camera parameter matrix.

Images