CN116429082A - A Visual SLAM Method Based on ST-ORB Feature Extraction - Google Patents

Abstract

The invention relates to a visual SLAM method based on ST-ORB feature extraction, firstly constructing a multi-scale space through an image pyramid, and then adopting a Shi-Tomasi algorithm to detect corner points of a gray image so as to improve the quality of feature points; then homogenizing the characteristic points by using a quadtree algorithm, and marking the characteristic points by using a binary descriptor BRIEF so as to match the characteristics of the front and rear frame images; finally, the algorithm is applied to an ORB-SLAM2 system to estimate the pose of the moving object. The method has good stability and improves the track precision of the visual SLAM.

Description

A Visual SLAM Method Based on ST-ORB Feature Extraction

technical field

The invention relates to the field of visual SLAM, in particular to a visual SLAM method based on ST-ORB (Shi-Tomasi-ORB) feature extraction.

Background technique

V-SLAM (Visual Simultaneous Localization and Mapping), that is, visual synchronous localization and map construction technology, uses visual sensors to map the environment while performing autonomous positioning of unmanned systems. With the rapid development of drones and autonomous driving industries, visual SLAM has become a research hotspot for scholars. The feature points detected by the feature point extraction method of the existing visual SLAM scheme are redundant and poor in uniformity. Since the extracted feature points cannot accurately describe the image information, the problem of key point loss occurs during the tracking process, which affects the location. The accuracy of attitude estimation, which ultimately affects the accuracy of trajectory positioning and map construction; and on platforms with limited computing resources such as unmanned aerial vehicles and unmanned vehicles, the commonly used feature detection methods have low computational efficiency and cannot meet the real-time performance of SLAM algorithms. Require.

Feature extraction is one of the important links of V-SLAM, which can usually be divided into learning-based and design-based methods. In recent years, machine learning methods have been widely used in image feature extraction. Arnfred et al. proposed a general framework for image feature extraction and feature matching without geometric constraints. Zeng et al. proposed image extraction using parametric Koopmans-Beckmann supervised learning. However, due to the high requirements on computing resources and poor universality of machine learning methods, they cannot be applied to complex and changeable complex environments. Bay proposed an accelerated robust feature algorithm based on the scale-invariant feature algorithm. This algorithm uses a Hessian matrix-based detector and a distribution-based descriptor to accelerate the extraction process, but it is still difficult to meet the real-time requirements of visual SLAM.

The ORB algorithm proposed by Rublee et al. uses the FAST algorithm to extract feature points, and uses anti-rotation descriptors for feature modification and feature matching. The ORB algorithm greatly improves the computational efficiency, but the feature points extracted by this method have poor stability and uneven distribution, which also leads to a decrease in the accuracy of pose estimation. Therefore, it is necessary to design a new visual SLAM method to solve the problems existing in the existing technology.

Contents of the invention

The purpose of the present invention is to provide a visual SLAM method based on ST-ORB feature extraction, which has good stability and improves the trajectory accuracy of visual SLAM.

In order to achieve the above purpose, the technical solution adopted by the present invention is: a visual SLAM method based on ST-ORB feature extraction, first constructing a multi-scale space through the image pyramid, and then using the Shi-Tomasi algorithm to detect the corners of the gray image, To improve the quality of feature points; then use the quadtree algorithm to homogenize the feature points, and use the binary descriptor BRIEF to mark the feature points so that the features of the front and rear frame images can be matched; finally, the algorithm is applied to the ORB-SLAM2 system Estimating the pose of a moving object.

Further, the method specifically includes the following steps:

1) In order to maintain the scale invariance of image features, set the number of image pyramid layers, use Gaussian image pyramid to down-sample each frame of image, and construct a multi-scale space to improve the accuracy of feature point matching in the feature matching process;

2) Using the Shi-Tomasi algorithm to extract the feature points of each level of image;

3) Homogenize the feature points through the quadtree algorithm;

4) Use the BRIEF description operator to describe the features of the feature points;

5) According to the binary descriptor BRIEF, perform feature matching on the two frames of images before and after, filter key frames, and generate map points;

6) Apply the above algorithm to the ORB-SLAM2 system to realize the pose estimation of visual SLAM.

Further, the Shi-Tomasi algorithm performs corner judgment according to the trend of grayscale variation of pixels in the sliding window; specifically:

Place a window on the image and move it; if the image gray value I(x,y) is moved at the point (x,y) by the offset of (u,v), the resulting grayscale difference The value E(u,v) is:

In the formula, S is the moving window area, h(x, y) is the Gaussian weighting function; use the Taylor formula to approximate I(x+u,y+v), and get:

Substituting formula (2) into formula (1) can get:

Let the M matrix be:

After simplification, the gray level difference is expressed as:

In formula (5), the M matrix is a second-order function about x and y. According to the difference of its eigenvalues, the pixel point is divided into three cases: corner point, straight line and plane; if the point is a corner point, due to its Moving the moving window in any direction will cause a large change in the gray level in the window. At this time, the two eigenvalues of the M matrix are both large;

Usually a corner point response value R is used to judge the quality of the corner points detected in the Shi-Tomasi algorithm;

R=min(λ ₁ ,λ ₂ ) (6)

When R is greater than the set threshold, it is judged that the pixel point is a corner point.

Compared with the prior art, the present invention has the following beneficial effects: the present invention provides a visual SLAM method based on ST-ORB feature extraction, which overcomes the problems of uneven distribution and poor stability of feature points extracted by the ORB algorithm, The ORB-SLAM2 system using ST-ORB feature extraction is more stable, and can reduce the absolute trajectory error to a certain extent, and improve the trajectory accuracy of visual SLAM.

Description of drawings

Fig. 1 is a flow chart of the implementation of the method of the embodiment of the present invention.

Fig. 2 is a schematic diagram of FAST feature points in an embodiment of the present invention.

Fig. 3 is a schematic diagram of an image pyramid structure in an embodiment of the present invention.

Fig. 4 is a schematic diagram of ORB feature extraction in an embodiment of the present invention.

Fig. 5 is a schematic diagram of ST-ORB feature extraction in the embodiment of the present invention.

Fig. 6 is a comparison chart of track errors in the embodiment of the present invention.

Fig. 7 is an analysis diagram of APE index error in the embodiment of the present invention.

Detailed ways

The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

It should be pointed out that the following detailed description is exemplary and is intended to provide further explanation to the present application. Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs.

It should be noted that the terminology used here is only for describing specific implementations, and is not intended to limit the exemplary implementations according to the present application. As used herein, unless the context clearly dictates otherwise, the singular is intended to include the plural, and it should also be understood that when the terms "comprising" and/or "comprising" are used in this specification, they mean There are features, steps, operations, means, components and/or combinations thereof.

As shown in Figure 1, this embodiment provides a visual SLAM method based on ST-ORB feature extraction, first constructing a multi-scale space through an image pyramid, and then using the Shi-Tomasi algorithm to detect corners on gray images to improve The quality of the feature points; then use the quadtree algorithm to homogenize the feature points, and use the binary descriptor BRIEF to mark the feature points so that the features of the front and rear frame images can be matched; finally, the algorithm is applied to the ORB-SLAM2 system for moving object pose estimation. Specifically, the method includes the following steps:

1) In order to maintain the scale invariance of image features, set the number of image pyramid layers, use Gaussian image pyramid to down-sample each frame of image, and construct a multi-scale space to improve the accuracy of feature point matching in the feature matching process.

2) Using the Shi-Tomasi algorithm to extract the feature points of each level of image.

3) Homogenize the feature points through the quadtree algorithm.

4) Use the BRIEF description operator to describe the features of the feature points.

5) According to the binary descriptor BRIEF, perform feature matching on the two frames of images before and after, filter key frames, and generate map points.

6) Apply the above algorithm to the ORB-SLAM2 system to realize the pose estimation of visual SLAM.

This method avoids the problem of rotation invariance in FAST corner detection in ORB algorithm and the problem of dense distribution of ORB feature points, and further improves the stability of input image feature points.

Described Shi-Tomasi algorithm carries out corner point judgment according to the gray level change trend of pixel point in the sliding window; Specifically:

Place a window on the image and move it; if the image gray value I(x,y) is moved at the point (x,y) by the offset of (u,v), the resulting grayscale difference The value E(u,v) is:

In the formula, S is the moving window area, h(x, y) is the Gaussian weighting function; use the Taylor formula to approximate I(x+u,y+v), and get:

Substituting formula (2) into formula (1) can get:

Let the M matrix be:

After simplification, the gray level difference is expressed as:

In formula (5), the M matrix is a second-order function about x and y. According to the difference of its eigenvalues, the pixel point is divided into three cases: corner point, straight line and plane; if the point is a corner point, due to its Moving the moving window in any direction will cause a large change in the gray level in the window. At this time, the two eigenvalues of the M matrix are both large;

Usually a corner point response value R is used to judge the quality of the corner points detected in the Shi-Tomasi algorithm;

R=min(λ ₁ ,λ ₂ ) (6)

When R is greater than the set threshold, it is judged that the pixel point is a corner point.

The relevant content involved in the present invention will be further described below.

1. ORB-SLAM2 algorithm

The visual SLAM algorithm framework mainly includes three parts: front-end visual odometer, back-end nonlinear optimization and loop closure detection. The traditional visual SLAM algorithm mainly reads the environmental information through the visual sensor, completes the local mapping through the visual odometer, and then performs nonlinear optimization on the local map through the back-end part, and further improves the SLAM trajectory positioning accuracy through loop detection.

ORB-SLAM is a visual SLAM algorithm published in IEEE Transaction on Robotics by Raul Mur-Atral et al. in 2015. ORB-SLAM is a real-time SLAM system based on feature points, which can work in large-scale, small-scale, indoor and outdoor environments. The system is also robust to severe motion, supporting wide-baseline loop closure detection and relocalization, including fully automatic initialization. The system contains modules common to all SLAM systems: tracking, mapping, relocation, and loop detection. Since the ORB-SLAM system is a SLAM system based on the feature point method, this process can calculate the motion trajectory of the camera in real time and generate a sparse 3D reconstruction result of the scene.

Based on ORB-SLAM, ORB-SLAM2 supports not only monocular cameras, but also binocular cameras and RGB-D cameras. The distance between the pixel and the camera can avoid complex calculations to obtain depth information, and reduce the trajectory positioning error to a certain extent. Compared with traditional visual SLAM, the ORB-SLAM2 algorithm framework mainly includes three threads: Tracking, LocalMapping and LoopClosing. This multi-threaded parallel form greatly shortens the running time of SLAM and improves the real-time performance of the visual SLAM system.

The main work of the Tracking part is to extract ORB features from the image, perform pose estimation based on the previous frame, or initialize the pose through global relocation, then track the reconstructed local map, optimize the pose, and then determine the new key according to some rules frame.

The LocalMapping part mainly completes local map construction, including inserting key frames, verifying and filtering recently generated map points, then generating new map points, using local adjustments, and finally filtering the inserted key frames to remove redundant keys frame information.

The LoopClosing part is mainly divided into two processes, which are closed-loop detection and closed-loop correction. The closed-loop detection first uses WOB to detect, and then calculates the similarity transformation through the Sim3 algorithm. Closed-loop correction, mainly closed-loop fusion and the graph optimization process of EssentialGraph.

2. Feature extraction

2.1 ORB Feature Extraction

ORB (Oriented FAST and Rotated BRIEF) is an algorithm for fast feature point extraction and description. The ORB algorithm is divided into two parts: feature point extraction and feature point description, and ORB feature point extraction is mainly developed by the FAST (Features from Accelerated Segment Test) algorithm. The image feature points extracted by the FAST algorithm are defined as the area centered on a certain pixel point. If there are enough pixel points whose values are not in the same area as the center point, the point may be a corner point. For a grayscale image, that is, if the grayscale value of the point is larger or smaller than the grayscale value of enough pixels in the surrounding area, the point may be a corner point.

2.1.1.FAST corner detection

The FAST corner detection process can be divided into the following steps:

1) Select a pixel point P from the grayscale image, as shown in Figure 2.

2) Set a threshold n.

3) Take point P as the center to extract 16 pixel points on a circle with a radius of 3 pixels, and divide them into three categories according to formula (1).

In the formula (1.1): Ip represents the gray value of the pixel point P; Ip-x represents the gray value of the xth pixel point around the point P; where the value of x is between 1-16; n is the detection time The threshold value of is generally an empirical value; a means that the points on the circle are darker than p; b means that the points on the circle are similar to the brightness of P; c means that the pixels on the circle are brighter than P.

4) When the classification result of at least 12 consecutive pixel points on the surrounding circle is a or c, it is determined that point P is a corner point. In actual detection, the efficiency of directly traversing the 16 points on the circle is relatively low. In order to improve the speed, you can first detect the gray values of the four points 1/5/9 and 13 on the circle. If point P is an angle point, then at least three of these four points are greater than Ip+n or less than Ip-n. If the condition is not met, the point is directly eliminated. This method can quickly exclude most of the non-corner points.

2.1.2. Image pyramid method

FAST feature points have neither scale invariance nor rotation invariance. In order to solve the problem of scale invariance, the ORB algorithm solves it by establishing several layers of image pyramids. Image pyramids are divided into Laplacian image pyramids and Gaussian image pyramids according to the way of upsampling and downsampling.

The Gaussian image pyramid expresses the multi-scale structure of the original image, and sets the scale factor of the given image to downsample step by step, that is, the original image is scaled, and the sampled images are arranged from high to low in resolution to form an image pyramid. Perform mean filtering and down-sampling on the nth layer of the pyramid to obtain the n+1th layer image, and its calculation formula is:

In the formula, Gn and Gn+1 are the images of the nth layer and the n+1th layer of the image pyramid; h is the mean filter template, and the size is axa, where a is the row and column interval of the downsampling. In the ORB-SLAM2 algorithm framework, an 8-layer image pyramid is constructed for the input image, and the constructed Gaussian image pyramid is shown in Figure 3.

2.1.3. Gray scale centroid method

For the problem that the feature points after FAST corner point detection do not have rotation invariance, it can be solved by the gray centroid method.

The basic principle of the gray-scale centroid method is: assuming that there is an offset between the gray level of a feature point and the centroid of the field, the main direction of the feature point is calculated by the vector from the feature point to the centroid. The centroid is calculated via moments, which are defined as follows:

In formula (1.3), I(x, y) is the gray value of the image; r is the radius of the field. Then the centroid of the moment is:

Therefore, according to (1.4), the direction of the feature point is:

θ=arctan(m ₀₁ /m ₁₀ ) (1.5)

The ORB feature extraction part is mainly composed of FAST corner detection, image pyramid method and gray centroid method; of course, in order to solve the problem of uneven distribution of image feature points, the ORB algorithm uses the method of quadtree for each layer of Gaussian pyramid The feature points of the image are homogenized. The problem of scale transformation and rotation invariance in the FAST algorithm is solved through the image pyramid and gray-scale centroid method, which improves the stability of feature points in the visual SLAM system, and can effectively prevent the missing key frames caused by mismatching.

It is based on the stability of feature points and the problem that key frames are easily lost in visual SLAM. The present invention optimizes the ORB feature extraction algorithm and proposes a new feature point extraction method, namely ST-ORB (Shi- Tomasi-ORB) feature extraction, and this method is applied in the framework of ORB-SLAM2 algorithm in order to explore the trajectory localization of indoor mobile robots.

3. Experimental results and analysis

In order to evaluate the quality of the feature points of the ST-ORB feature extraction method in this method and the trajectory positioning effect of the optimized ORB-SLAM2, this example uses indoor images for feature extraction experiments, by comparing the traditional ORB and ST-ORB feature extraction algorithms Analyze the results; use the TUM data set to verify the accuracy of the improved ORB-SLAM2 algorithm positioning. The PC platform running in this embodiment is Intel Core i5-7200U CPU@2.5GHz, the memory is 12GB, and the system is Ubuntu 20.04.

3.1 Feature point extraction

Set the number of feature points for target extraction to 200, and use two feature extraction methods, ORB and ST-ORB, to conduct feature point extraction experiments in indoor and outdoor environments, and the results are shown in Figure 4 and Figure 5, respectively.

It can be seen from Figure 4 and Figure 5 that for the ORB feature extraction algorithm, the extracted feature points are too densely distributed, and can only recognize local information in the image, and have a certain impact on the pose estimation of visual SLAM; while for ST- ORB feature extraction algorithm, we can see that the distribution of feature points in Figure 5 is more uniform. Compared with the ORB algorithm, the quality of the feature points is higher, which can avoid the loss of key frames to a certain extent.

3.2 Trajectory Estimation

In order to verify the influence of the ORB and ST-ORB two feature extraction methods on the accuracy of the visual SLAM method, the ORB-SLAM2 monocular vision is used as the main SLAM solution, and the dataset used is the TUM dataset. Table 1 shows the root mean square error of the trajectory obtained by the method and the ORB algorithm in four TUM data sets through experiments.

Table 1 Trajectory root mean square error (RMSE)/(m)

The results in Table 1 show that the absolute trajectory error of the ORB method on the fr1_rpy dataset is low. During the operation of the dataset, some key frame tracking is lost due to the SLAM method using ST-ORB, which leads to its The trajectory positioning accuracy is lower than that of the ORB scheme; the performance of this method in the other three data sets is obviously better than that of ORB, especially the ORB method has a serious tracking loss problem on the fr1_desk2 data set, and the ST-ORB method can maintain good stability , which reduces the trajectory error to a large extent. From the perspective of the performance of the two methods of ORB and ST-ORB on the TUM dataset, the performance of the ST-ORB feature extraction method proposed by the present invention in SLAM pose estimation is obviously superior to the traditional ORB method.

Figure 6 is a comparison of the trajectory errors obtained by the two methods on the fr1_desk data sequence, where the thermal bar indicates that the trajectory error size gradually increases from bottom to top, the dotted line indicates the reference trajectory (that is, the true value), and other lines indicate the estimated calculated value , from which we can see that compared with the traditional ORB-SLAM2 algorithm, the estimated trajectory of this method is closer to the benchmark trajectory.

Figure 7 is the trajectory error APE analysis obtained by the two methods on the fr1_desk data set, that is, the absolute trajectory error performance analysis. It can be seen that the root mean square error obtained by ORB-SLAM2 using the ORB method is 0.135m, using ST- The root mean square error obtained by the ORB method is 0.071m, and the trajectory positioning accuracy on this data set is increased by 47%, so the method proposed by the present invention has more advantages.

Those skilled in the art should understand that the embodiments of the present application may be provided as methods, systems, or computer program products. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.

The present application is described with reference to flowcharts and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present application. It should be understood that each procedure and/or block in the flowchart and/or block diagram, and a combination of procedures and/or blocks in the flowchart and/or block diagram can be realized by computer program instructions. These computer program instructions may be provided to a general purpose computer, special purpose computer, embedded processor, or processor of other programmable data processing equipment to produce a machine such that the instructions executed by the processor of the computer or other programmable data processing equipment produce a An apparatus for realizing the functions specified in one or more procedures of the flowchart and/or one or more blocks of the block diagram.

These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing apparatus to operate in a specific manner, such that the instructions stored in the computer-readable memory produce an article of manufacture comprising instruction means, the instructions The device realizes the function specified in one or more procedures of the flowchart and/or one or more blocks of the block diagram.

These computer program instructions can also be loaded onto a computer or other programmable data processing device, causing a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process, thereby The instructions provide steps for implementing the functions specified in the flow chart or blocks of the flowchart and/or the block or blocks of the block diagrams.

The above is only a preferred embodiment of the present invention, and is not intended to limit the present invention to other forms. Any skilled person who is familiar with this profession may use the technical content disclosed above to change or modify the equivalent of equivalent changes. Example. However, any simple modifications, equivalent changes and modifications made to the above embodiments according to the technical essence of the present invention without departing from the content of the technical solution of the present invention still belong to the protection scope of the technical solution of the present invention.

Claims (3)

1. A visual SLAM method based on ST-ORB feature extraction is characterized in that a multi-scale space is constructed through an image pyramid, and then corner detection is carried out on a gray image by adopting a Shi-Tomasi algorithm so as to improve the quality of feature points; then homogenizing the characteristic points by using a quadtree algorithm, and marking the characteristic points by using a binary descriptor BRIEF so as to match the characteristics of the front and rear frame images; finally, the algorithm is applied to an ORB-SLAM2 system to estimate the pose of the moving object.

2. The visual SLAM method based on ST-ORB feature extraction of claim 1, comprising the steps of:

1) In order to keep the scale invariance of the image features, the number of layers of an image pyramid is set, each frame of image is downsampled by adopting a Gaussian image pyramid, and a multi-scale space is constructed so as to improve the accuracy of feature point matching in a feature matching link;

2) Extracting feature points of each level image by adopting a Shi-Tomasi algorithm;

3) Homogenizing the characteristic points through a quadtree algorithm;

4) Carrying out feature description on the feature points by adopting a BRIEF description operator;

5) Performing feature matching on the front frame image and the rear frame image according to the binary descriptor BRIEF, screening key frames, and generating map points;

6) The above algorithm is applied to an ORB-SLAM2 system to realize pose estimation of the visual SLAM.

3. The visual SLAM method based on ST-ORB feature extraction of claim 2, wherein the Shi-Tomasi algorithm performs corner judgment according to gray scale variation trend of pixel points in a sliding window; the method comprises the following steps:

a window is placed over the image and moved; let the gray value I (x, y) of the image be shifted by an offset of (u, v) at the point (x, y) to produce a gray difference E (u, v) of:

where S is the moving window region and h (x, y) is the Gaussian weighting function; approximation of I (x+u, y+v) using the taylor formula yields:

substitution of formula (2) into formula (1) yields:

let M matrix be:

after simplification, the gray scale difference is expressed as:

in the formula (5), the M matrix is a second-order function about x and y, and pixel points are divided into three conditions of angular points, straight lines and planes according to different characteristic values; if the point is a corner point, the moving window moves towards any direction to cause larger change of gray scale in the window, and at the moment, two characteristic values of the M matrix are larger;

a corner response value R is generally adopted to judge the corner quality detected in the Shi-Tomasi algorithm;

R＝min(λ ₁ ，λ ₂ ) (6)

and when R is larger than the set threshold value, judging the pixel point as a corner point.

Images