CN116429082A - Visual SLAM method based on ST-ORB feature extraction - Google Patents

Abstract

The invention relates to a visual SLAM method based on ST-ORB feature extraction, firstly constructing a multi-scale space through an image pyramid, and then adopting a Shi-Tomasi algorithm to detect corner points of a gray image so as to improve the quality of feature points; then homogenizing the characteristic points by using a quadtree algorithm, and marking the characteristic points by using a binary descriptor BRIEF so as to match the characteristics of the front and rear frame images; finally, the algorithm is applied to an ORB-SLAM2 system to estimate the pose of the moving object. The method has good stability and improves the track precision of the visual SLAM.

Description

Visual SLAM method based on ST-ORB feature extraction

Technical Field

The invention relates to the field of visual SLAM, in particular to a visual SLAM method based on ST-ORB (Shi-Tomasi-ORB) feature extraction.

Background

V-SLAM (Visual Simultaneous Localization and Mapping) is visual synchronous positioning and map construction technology, which is to perform autonomous positioning of an unmanned system while mapping an environment through a visual sensor. With the rapid development of unmanned aerial vehicles and automatic driving industries, the visual SLAM becomes a research hotspot of vast students. The feature points detected by the feature point extraction method of the existing visual SLAM scheme are redundant and poor in uniformity, and because the extracted feature points cannot accurately describe image information, the problem of key point loss occurs in the tracking process, the accuracy of pose estimation is affected, and therefore the track positioning and map construction accuracy are finally affected; in addition, on platforms with limited computing resources such as unmanned aerial vehicles and unmanned vehicles, the common feature detection method has low computing efficiency and cannot meet the real-time requirement of SLAM algorithm.

Feature extraction is one of the important links of V-SLAM and can be generally classified into learning-based and design-based methods. In recent years, a machine learning method is widely applied to image feature extraction, arnfred et al propose a general framework for image feature extraction and feature matching without geometric constraint, zeng et al propose an image extraction matching method using parameterized Koopmans-Beckmann supervised learning, but the machine learning method has high requirements on computing resources and poor universality, and cannot be applied to complex and changeable environments. Bay proposes an accelerated robust feature algorithm based on a scale-invariant feature algorithm, which uses a Hessian matrix-based detector and a distribution-based descriptor to accelerate the extraction process, but still has difficulty in meeting the real-time requirements of visual SLAM.

The ORB algorithm proposed by Rublee et al employs FAST algorithm to extract feature points, and uses anti-rotation descriptors for feature modification and feature matching. The ORB algorithm greatly improves the calculation efficiency, but the characteristic points extracted by the method have poor stability and uneven distribution, which also leads to the reduction of pose estimation accuracy. Therefore, there is a need to design a new visual SLAM method to solve the problems of the prior art.

Disclosure of Invention

The invention aims to provide a visual SLAM method based on ST-ORB feature extraction, which has good stability and improves the track precision of the visual SLAM.

In order to achieve the above purpose, the invention adopts the following technical scheme: a visual SLAM method based on ST-ORB feature extraction firstly constructs a multi-scale space through an image pyramid, and then carries out corner detection on a gray level image by adopting a Shi-Tomasi algorithm so as to improve the quality of feature points; then homogenizing the characteristic points by using a quadtree algorithm, and marking the characteristic points by using a binary descriptor BRIEF so as to match the characteristics of the front and rear frame images; finally, the algorithm is applied to an ORB-SLAM2 system to estimate the pose of the moving object.

Further, the method specifically comprises the following steps:

1) In order to keep the scale invariance of the image features, the number of layers of an image pyramid is set, each frame of image is downsampled by adopting a Gaussian image pyramid, and a multi-scale space is constructed so as to improve the accuracy of feature point matching in a feature matching link;

2) Extracting feature points of each level image by adopting a Shi-Tomasi algorithm;

3) Homogenizing the characteristic points through a quadtree algorithm;

4) Carrying out feature description on the feature points by adopting a BRIEF description operator;

5) Performing feature matching on the front frame image and the rear frame image according to the binary descriptor BRIEF, screening key frames, and generating map points;

6) The above algorithm is applied to an ORB-SLAM2 system to realize pose estimation of the visual SLAM.

Further, the Shi-Tomasi algorithm judges the corner points according to the gray level change trend of the pixel points in the sliding window; the method comprises the following steps:

a window is placed over the image and moved; let the gray value I (x, y) of the image be shifted by an offset of (u, v) at the point (x, y) to produce a gray difference E (u, v) of:

where S is the moving window region and h (x, y) is the Gaussian weighting function; approximation of I (x+u, y+v) using the taylor formula yields:

substitution of formula (2) into formula (1) yields:

let M matrix be:

after simplification, the gray scale difference is expressed as:

in the formula (5), the M matrix is a second-order function about x and y, and pixel points are divided into three conditions of angular points, straight lines and planes according to different characteristic values; if the point is a corner point, the moving window moves towards any direction to cause larger change of gray scale in the window, and at the moment, two characteristic values of the M matrix are larger;

a corner response value R is generally adopted to judge the corner quality detected in the Shi-Tomasi algorithm;

R＝min(λ ₁ ，λ ₂ ) (6)

and when R is larger than the set threshold value, judging the pixel point as a corner point.

Compared with the prior art, the invention has the following beneficial effects: the invention provides a visual SLAM method based on ST-ORB feature extraction, which solves the problems of uneven distribution and poor stability of feature points extracted by an ORB algorithm, and an ORB-SLAM2 system extracted by the ST-ORB feature has stronger stability, can reduce absolute track error to a certain extent, and improves track precision of the visual SLAM.

Drawings

FIG. 1 is a flow chart of a method implementation of an embodiment of the present invention.

Fig. 2 is a schematic diagram of FAST feature points in an embodiment of the present invention.

FIG. 3 is a schematic diagram of an image pyramid structure in an embodiment of the present invention.

FIG. 4 is a schematic diagram of ORB feature extraction in an embodiment of the invention.

FIG. 5 is a schematic diagram of ST-ORB feature extraction in an embodiment of the invention.

FIG. 6 is a graph of trace error versus trace error in an embodiment of the invention.

FIG. 7 is a graph of APE indicator error analysis in an embodiment of the invention.

Detailed Description

The invention will be further described with reference to the accompanying drawings and examples.

It should be noted that the following detailed description is exemplary and is intended to provide further explanation of the present application. Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs.

It is noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of example embodiments in accordance with the present application. As used herein, the singular is also intended to include the plural unless the context clearly indicates otherwise, and furthermore, it is to be understood that the terms "comprises" and/or "comprising" when used in this specification are taken to specify the presence of stated features, steps, operations, devices, components, and/or combinations thereof.

As shown in fig. 1, the present embodiment provides a visual SLAM method based on ST-ORB feature extraction, firstly, constructing a multi-scale space through an image pyramid, and then, performing corner detection on a gray image by using a Shi-Tomasi algorithm to improve the quality of feature points; then homogenizing the characteristic points by using a quadtree algorithm, and marking the characteristic points by using a binary descriptor BRIEF so as to match the characteristics of the front and rear frame images; finally, the algorithm is applied to an ORB-SLAM2 system to estimate the pose of the moving object. Specifically, the method comprises the following steps:

1) In order to keep the scale invariance of the image features, the number of layers of the image pyramid is set, each frame of image is downsampled by adopting a Gaussian image pyramid, and a multi-scale space is constructed so as to improve the accuracy of feature point matching in a feature matching link.

2) And extracting the characteristic points of each level image by adopting a Shi-Tomasi algorithm.

3) And homogenizing the characteristic points through a quadtree algorithm.

4) And carrying out feature description on the feature points by adopting a BRIEF description operator.

5) And performing feature matching on the front frame image and the rear frame image according to the binary descriptor BRIEF, screening key frames, and generating map points.

6) The above algorithm is applied to an ORB-SLAM2 system to realize pose estimation of the visual SLAM.

The method avoids the rotation invariance problem in the FAST corner detection in the ORB algorithm and the problem of dense ORB characteristic point distribution, and further improves the stability of the characteristic points of the input image.

The Shi-Tomasi algorithm judges the corner points according to the gray level change trend of the pixel points in the sliding window; the method comprises the following steps:

a window is placed over the image and moved; let the gray value I (x, y) of the image be shifted by an offset of (u, v) at the point (x, y) to produce a gray difference E (u, v) of:

where S is the moving window region and h (x, y) is the Gaussian weighting function; approximation of I (x+u, y+v) using the taylor formula yields:

substitution of formula (2) into formula (1) yields:

let M matrix be:

after simplification, the gray scale difference is expressed as:

in the formula (5), the M matrix is a second-order function about x and y, and pixel points are divided into three conditions of angular points, straight lines and planes according to different characteristic values; if the point is a corner point, the moving window moves towards any direction to cause larger change of gray scale in the window, and at the moment, two characteristic values of the M matrix are larger;

a corner response value R is generally adopted to judge the corner quality detected in the Shi-Tomasi algorithm;

R＝min(λ ₁ ，λ ₂ ) (6)

and when R is larger than the set threshold value, judging the pixel point as a corner point.

The invention is further described below in relation to the following.

ORB-SLAM2 algorithm

The visual SLAM algorithm frame mainly comprises three parts, namely a front-end visual odometer, a rear-end nonlinear optimization and loop detection. The traditional visual SLAM algorithm mainly reads environment information through a visual sensor, completes local map building through a visual odometer, then carries out nonlinear optimization on a local map through a rear end part, and further improves SLAM track positioning accuracy through loop detection.

ORB-SLAM is a visual SLAM algorithm published by Raul Mur-atlal et al in 2015 on IEEE Transaction on Robotics. ORB-SLAM is a real-time SLAM system based on characteristic points, and can work in large-scale, small-scale, indoor and outdoor environments. The system is also robust to strenuous movements, supporting closed loop detection and repositioning of wide baselines, including full-automatic initialization. The system comprises a module common to all SLAM systems: tracking, mapping, repositioning and loop detection. Because the ORB-SLAM system is a SLAM system based on a characteristic point method, the process can calculate the motion trail of a camera in real time and generate a sparse three-dimensional reconstruction result of a scene.

The ORB-SLAM2 not only supports a monocular camera, but also supports a binocular camera and an RGB-D camera on the basis of the ORB-SLAM, and compared with the monocular camera, the depth camera can actively measure the distance between each pixel of an image and the camera through the infrared structured light principle, so that complex operation for acquiring depth information can be avoided, and track positioning errors are reduced to a certain extent. Compared with the traditional visual SLAM, the ORB-SLAM2 algorithm framework mainly comprises Tracking, localMapping threads and LoopCloing threads, and the multithreading parallel form greatly shortens the running time of the SLAM and improves the real-time performance of a visual SLAM system.

The main work of the Tracking part is to extract ORB features from the image, estimate the pose according to the previous frame, or initialize the pose by global repositioning, then track the already reconstructed local map, optimize the pose, and then determine new key frames according to some rules.

The LocalMapping part mainly completes local map construction, and comprises the steps of inserting a key frame, verifying and screening the most recently generated map points, then generating new map points, using local adjustment, and finally screening the inserted key frame to remove redundant key frame information.

The LoopCloing part is mainly divided into two processes, namely closed loop detection and closed loop correction. Closed loop detection is first detected using WOB and then similarity transformation is calculated by Sim3 algorithm. The closed loop correction is mainly a closed loop fusion and Essential Graph optimization process.

2. Feature extraction

2.1ORB feature extraction

ORB (Oriented FAST and Rotated BRIEF) is an algorithm for fast feature point extraction and description. The ORB algorithm is divided into two parts, feature point extraction and feature point description, and the ORB feature point extraction is mainly developed by FAST (Features fromAccelerated Segment Test) algorithm. The FAST algorithm extracts the definition of the image feature point as a corner point if the value of a sufficient number of pixel points is taken as the center point in the area with the center point not being the same. For a gray image, i.e. if the gray value of the point is larger or smaller than the gray value of a sufficient number of pixels in the surrounding area, the point may be a corner point.

Fast corner detection

The FAST corner detection process can be divided into the following steps:

1) A pixel point P is selected from the gray image as shown in fig. 2.

2) A threshold n is set.

3) 16 pixel points on a circle with radius of 3 pixels are extracted by taking the P point as the circle center, and are classified into three types according to the formula (1).

In formula (1.1): ip represents the gray value of the pixel point P; ip-x represents the gray value of the x-th pixel point on the periphery of the P point; wherein x has a value of 1-16; n is a threshold value during detection, typically an empirical value; a represents that the points on the circle are darker than p; b represents that the points on the circle are similar in brightness to P; c denotes that the pixels on the circle are brighter than P.

4) And when the classification result of at least 12 continuous pixel points on the circumference circle is a or c, judging the P point as a corner point. In actual detection, the efficiency of the method for directly traversing 16 points on the circle is lower, in order to increase the speed, gray values of 1/5/9 and 13 points on the circle can be detected first, if the P point is a corner point, at least 3 of the four points satisfy that the P point is greater than ip+n or less than Ip-n, and if the P point does not satisfy the condition, the P point is directly removed. This method can rapidly drain most non-corner points.

2.1.2. Image pyramid method

FAST feature points have neither scale nor rotational invariance. To solve the problem of scale invariance, the ORB algorithm is solved by building several layers of image pyramids. The image pyramid is divided into a laplacian image pyramid and a gaussian image pyramid according to the up-sampling and down-sampling modes.

The Gaussian image pyramid expresses the original image in a multi-scale structure, the given image is set with a scale factor to be sampled downwards step by step, namely the original image is scaled, and the sampled images are arranged upwards from high to low according to the resolution to form the image pyramid. The n layer of the pyramid is subjected to mean filtering and downsampling to obtain an n+1th layer image, wherein the calculation formula is as follows:

wherein Gn and Gn+1 are images of the nth layer and the n+1 th layer of the image pyramid; h is the mean filter template and is axa in size, where a is the downsampled line spacing. An 8-layer image pyramid is constructed for the input image in the ORB-SLAM2 algorithm framework, and the constructed Gaussian image pyramid is shown in FIG. 3.

2.1.3. Gray centroid method

The problem that the feature points after FAST corner detection do not have rotation invariance can be solved through a gray centroid method.

The basic principle of the gray centroid method is as follows: assuming that there is an offset between the gray level of a certain feature point and the centroid of the field, the principal direction of the feature point is calculated from the vector of the feature point to the centroid. Centroid is calculated by moment, which is defined as follows:

in the formula (1.3), I (x, y) is the gray value of the image; r is the radius of the field. The centroid of the moment is then:

therefore, as obtained in (1.4), the direction of the feature point is:

θ＝arctan(m ₀₁ /m ₁₀ ) (1.5)

the ORB feature extraction part mainly comprises a FAST corner detection method, an image pyramid method and a gray centroid method; of course, in order to solve the problem of uneven distribution of image feature points, a quadtree method is used in the ORB algorithm to perform homogenization treatment on the feature points in each layer of Gaussian pyramid image China. The problems of scale transformation and rotation invariance in the FAST algorithm are solved through an image pyramid and gray level centroid method, the stability of characteristic points in a visual SLAM system is improved, and the problem of key frame missing caused by mismatching can be effectively prevented.

Based on the stability problem of feature points and the problem that key frames in visual SLAM are easy to lose, the invention optimizes an ORB feature extraction algorithm, and provides a novel feature point extraction method, namely ST-ORB (Shi-Tomasi-ORB) feature extraction, and the method is applied to an ORB-SLAM2 algorithm frame so as to explore the track positioning problem of an indoor mobile robot.

3. Experimental results and analysis

In order to evaluate the quality of feature points of the ST-ORB feature extraction method and the track positioning effect of the optimized ORB-SLAM2 in the method, the embodiment adopts indoor images to perform feature extraction experiments, and performs result analysis by comparing the traditional ORB and ST-ORB feature extraction algorithms; the accuracy of the improved ORB-SLAM2 algorithm localization was verified by employing the TUM dataset. The PC platform operated in this embodiment is Intel Core i5-7200U CPU@2.5GHz, the memory is 12GB, and the system is Ubuntu 20.04.

3.1 feature Point extraction

The number of feature points extracted by the target was set to 200, and feature point extraction experiments were performed in indoor and outdoor environments using two feature extraction methods, ORB and ST-ORB, respectively, and the results are shown in fig. 4 and 5, respectively.

As can be seen from fig. 4 and fig. 5, for the ORB feature extraction algorithm, the extracted feature points are too densely distributed, only the local information in the image can be identified, and the pose estimation of the visual SLAM is affected to some extent; for the ST-ORB feature extraction algorithm, we can see that the feature point distribution in fig. 5 is more uniform, and compared with the ORB algorithm, the quality of the feature point is higher, so that the loss of the key frame can be avoided to a certain extent.

3.2 trajectory estimation

In order to verify the effect of both ORB and ST-ORB feature extraction methods on the accuracy of the visual SLAM method, ORB-SLAM2 monocular vision was used as the subject SLAM protocol, and the data set used was TUM data set. Table 1 shows the root mean square error of the trajectories obtained by the present method and ORB algorithm through experiments in four TUM datasets.

TABLE 1 root mean square error of trajectories (RMSE)/(m)

The results in table 1 show that: the absolute track error of the ORB method on the fre1_ rpy data set is low, and the track positioning accuracy of the data set is lower than that of an ORB scheme because part of key frame tracking loss occurs in the SLAM method using ST-ORB in the running process; the method has obviously stronger performance than ORB in the other three data sets, especially the ORB method has serious tracking loss problem in the fr1_desk2 data set, and the ST-ORB method can maintain good stability and reduce the track error to a great extent. From the performance of both ORB and ST-ORB methods on the TUM data set, the performance of the ST-ORB feature extraction method on SLAM pose estimation is obviously due to the traditional ORB method.

FIG. 6 shows the comparison of the track errors obtained on the fr1_desk data sequence by two methods, wherein the thermal bar shows that the track error is gradually increased from bottom to top, the broken line shows the reference track (i.e. true value), and the other lines show the estimated calculated values, so that compared with the conventional ORB-SLAM2 algorithm, the estimated track of the method is closer to the reference track.

FIG. 7 shows the analysis of the trace error APE, i.e., the absolute trace error performance, obtained on the fr1_desk dataset by two methods, and shows that the root mean square error of ORB-SLAM2 is 0.135m when the ORB method is used, and the root mean square error of ORB-SLAM2 is 0.071m when the ST-ORB method is used, and the trace positioning accuracy on the dataset is improved by 47%, so the method provided by the invention has more advantages.

It will be appreciated by those skilled in the art that embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.

The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

The above description is only a preferred embodiment of the present invention, and is not intended to limit the invention in any way, and any person skilled in the art may make modifications or alterations to the disclosed technical content to the equivalent embodiments. However, any simple modification, equivalent variation and variation of the above embodiments according to the technical substance of the present invention still fall within the protection scope of the technical solution of the present invention.

Claims (3)

1. A visual SLAM method based on ST-ORB feature extraction is characterized in that a multi-scale space is constructed through an image pyramid, and then corner detection is carried out on a gray image by adopting a Shi-Tomasi algorithm so as to improve the quality of feature points; then homogenizing the characteristic points by using a quadtree algorithm, and marking the characteristic points by using a binary descriptor BRIEF so as to match the characteristics of the front and rear frame images; finally, the algorithm is applied to an ORB-SLAM2 system to estimate the pose of the moving object.

2. The visual SLAM method based on ST-ORB feature extraction of claim 1, comprising the steps of:

1) In order to keep the scale invariance of the image features, the number of layers of an image pyramid is set, each frame of image is downsampled by adopting a Gaussian image pyramid, and a multi-scale space is constructed so as to improve the accuracy of feature point matching in a feature matching link;

2) Extracting feature points of each level image by adopting a Shi-Tomasi algorithm;

3) Homogenizing the characteristic points through a quadtree algorithm;

4) Carrying out feature description on the feature points by adopting a BRIEF description operator;

5) Performing feature matching on the front frame image and the rear frame image according to the binary descriptor BRIEF, screening key frames, and generating map points;

6) The above algorithm is applied to an ORB-SLAM2 system to realize pose estimation of the visual SLAM.

3. The visual SLAM method based on ST-ORB feature extraction of claim 2, wherein the Shi-Tomasi algorithm performs corner judgment according to gray scale variation trend of pixel points in a sliding window; the method comprises the following steps:

a window is placed over the image and moved; let the gray value I (x, y) of the image be shifted by an offset of (u, v) at the point (x, y) to produce a gray difference E (u, v) of:

where S is the moving window region and h (x, y) is the Gaussian weighting function; approximation of I (x+u, y+v) using the taylor formula yields:

substitution of formula (2) into formula (1) yields:

let M matrix be:

after simplification, the gray scale difference is expressed as:

in the formula (5), the M matrix is a second-order function about x and y, and pixel points are divided into three conditions of angular points, straight lines and planes according to different characteristic values; if the point is a corner point, the moving window moves towards any direction to cause larger change of gray scale in the window, and at the moment, two characteristic values of the M matrix are larger;

a corner response value R is generally adopted to judge the corner quality detected in the Shi-Tomasi algorithm;

R＝min(λ ₁ ，λ ₂ ) (6)

and when R is larger than the set threshold value, judging the pixel point as a corner point.

Images