CN110631588B - Unmanned aerial vehicle visual navigation positioning method based on RBF network - Google Patents

Abstract

The invention discloses an RBF network-based visual navigation and positioning method for an unmanned aerial vehicle. The scheme of the present invention is: when the GNSS signal is not lost, collect images through the camera, and extract image frames from the camera, detect feature points for each image, extract descriptors, and retain the feature point information of the image; The descriptor processing of the image frame stores the descriptor information and positioning information in the visual database; in the case of GNSS signal loss, extracts the camera to capture the image, and also performs descriptor extraction, and uses the visual database information to train the RBF network classifier : Then according to the RBF network classifier, the generated descriptors are searched in the neighborhood, the optimal matching position is estimated and the current positioning information is obtained based on the recorded positioning information. In the case of GNSS signal loss, the visual database constructed based on the present invention can realize the positioning and navigation processing of the UAV, and the visual database only stores the feature point descriptor information of the image, and the memory occupies a small space.

Description

A method of visual navigation and positioning of UAV based on RBF network

technical field

The invention belongs to the technical field of unmanned aerial vehicle navigation and positioning, and in particular relates to an unmanned aerial vehicle visual navigation and positioning method based on an RBF (Radial Basis Function) network.

Background technique

The UAV integrated positioning system plays a vital role in its stability and integrity. The most common solutions for localization include the combination of Global Navigation Satellite System (GNSS) and Inertial Navigation System (INS) within a multi-sensor fusion framework. In this case, GNSS is used as a concise and economical method to constrain the unbounded errors produced by INS sensors during the positioning process. But in fact, INS integrates time in the iterative process of continuously acquiring data from multiple sensors to obtain an approximate drone position. During this process, the measurement errors generated by the sensors will quickly accumulate and grow without limit . Therefore, most UAVs utilize the Extended Kalman Filtering (EKF) framework to fuse data from INS and GNSS, which combines the short-term accuracy of INS with the long-term accuracy of GNSS to effectively Positioning errors are suppressed. Therefore, GNSS is widely used in various UAVs.

Despite their advantages, GNSS has proven to be unreliable in several documented instances. Outdoor scenarios such as urban canyons, forests, jungles, and rainy areas also prove to be vulnerable to intentional attacks and unintentional environmental disturbances. In addition to this, drones using GNSS have proven to be vulnerable to signal spoofing on multiple occasions, and such attacks are now a reality. The disadvantage of using GNSS for drone navigation is the radio communication necessary to obtain positioning data, and radio communication systems are generally prone to availability issues, interference and signal changes. The fundamental reason for using GNSS/INS fusion is to rely on global information obtained from GNSS to solve local positioning problems. In order to solve these problems, suitable navigation sensors and new navigation algorithms are introduced to solve the problem of UAV navigation and positioning when it is interfered by wireless communication and GNSS/INS short-term or long-term failure.

A popular approach to reliably determine drone positioning in GNSS-denied/GNSS-degraded outdoor environments uses a combination of monocular 2D cameras and vision-based techniques. These techniques fall into two categories: one is techniques that use prior knowledge of the environment and techniques that do not use prior knowledge of the environment. Map-based navigation appears to be advanced in the field of visual navigation using prior knowledge. This approach matches images taken by drones with previously flown high-resolution satellite images of landmarks or landmark images. Limitations of this solution Limitations include the need to have a large geographic imagery database that can be accessed by network-connected onboard devices, and another important limitation is the need for prior knowledge of origins or pre-defined boundaries. Therefore, map-based solutions suffer from severe limitations that hinder their application in real-world scenarios. A second class of vision-based techniques does not have this limitation because they do not require prior knowledge of the environment. Solutions in this category include vision measurement and simultaneous localization and mapping (SLAM), among others. In vision measurement, the motion of a drone is estimated by tracking features or pixels between consecutive images obtained from a monocular camera. However, even state-of-the-art monocular vision measurements suffer over time because current localization estimates are based on previous localization estimates, leading to an accumulation of errors. Compared to visual measurement, SLAM solves the problem of localization while building a map of the environment. Map building requires multiple steps, such as tracking, relocalization, and loop closing, but such solutions are always accompanied by heavy computation and memory usage.

Contents of the invention

The purpose of the present invention is to: aim at the above-mentioned problems, provide a UAV visual navigation and positioning method based on RBF network, which collects ground image feature descriptors in the process of UAV navigation, and uses feature description sub-data sets The trained RBF network classifier performs a neighborhood search on the feature point descriptors of the currently collected image to obtain the optimal matching position of the current image, thereby estimating more accurate positioning information of the UAV location.

The RBF network-based UAV visual navigation positioning method of the present invention comprises the following steps:

Step S1: setting the RBF neural network used for feature point descriptor matching of the image, and performing neural network training;

Among them, the training sample is: the image collected by the airborne camera during the flight of the UAV; the feature vector of the training sample is: the feature point descriptor of the image obtained through the ORB feature point detection process;

Step S2: Construct the visual database when the drone is flying:

During the flight of the UAV, the image is collected by the onboard camera, and the ORB feature point detection process is performed on the collected image, and the descriptor of each feature point is extracted to obtain the feature point descriptor of the current image; the feature point of the image is The descriptor is stored in the visual database together with the positioning information when the image is collected;

Step S3: UAV visual navigation and positioning based on visual database:

Based on the fixed interval period, the image collected by the airborne camera is extracted as the image to be matched;

Perform ORB feature point detection processing on the image to be matched, extract the descriptor of each feature point, and obtain the feature point descriptor of the image to be matched;

Input the feature point descriptor of the image to be matched into the trained RBF neural network, perform a neighborhood search, and obtain the optimal matching feature point descriptor of the image to be matched in the visual database;

And based on the positioning information recorded in the database by the optimal matching feature point descriptor, the current visual navigation positioning result of the UAV is obtained.

Further, step S3 also includes: detecting whether the similarity between the optimal matching feature point descriptor and the feature point descriptor of the image to be matched is less than a preset similarity threshold; if so, based on the optimal matching feature point descriptor in Based on the positioning information recorded in the database, the current visual navigation and positioning results of the UAV are obtained; otherwise, the navigation is continued based on the recently obtained visual navigation and positioning results.

In summary, owing to adopting above-mentioned technical scheme, the beneficial effect of the present invention is:

(1) The visual database only stores the feature point descriptor information of the image, reducing the memory footprint;

(2) It can directly access the visual database to match the images taken by the UAV without a reference gallery;

(3) Based on the trained RBF network, the feature description sub-neighborhood search is realized, the best matching position is obtained, and the location information is estimated.

Description of drawings

Figure 1 is the overall system framework of visual positioning;

Fig. 2 is the flowchart of ORB feature point detection;

Fig. 3 is a schematic diagram of feature point rough extraction during ORB feature point extraction;

Figure 4 is a schematic diagram of the RBF network structure;

Fig. 5 is a schematic diagram of a process of RBF network matching and positioning.

Detailed ways

In order to make the purpose, technical solution and advantages of the present invention clearer, the present invention will be further described in detail below in conjunction with the implementation methods and accompanying drawings.

A vision-based UAV navigation and positioning method of the present invention, by collecting the ground image feature descriptor of the area where the UAV is currently located, using the RBF network classifier trained by the feature descriptor data set to describe the feature points of the collected image Neighborhood search is carried out to obtain the optimal matching position of the image, so as to estimate the more accurate positioning information of the UAV location.

Referring to Fig. 1, the vision-based UAV navigation and positioning method of the present invention mainly includes two parts: the one is the data collection when going out, and the second is the positioning estimation when returning;

In the data acquisition part, the image is collected by the camera, and the image frame is extracted from the camera, the feature point is detected for each image, and the descriptor is extracted. This process discards the image data and retains the feature point information of the image; repeat the extraction of each image frame Descriptor processing, storing descriptor information and positioning information in the visual database;

In the positioning estimation process, in the case of GNSS signal loss, extract the image taken by the camera, also perform descriptor extraction, and use the visual database information to train the RBF network classifier: and then perform neighborhood analysis on the generated descriptor according to the RBF network classifier Search and estimate the optimal matching position; finally, estimate the positioning information of the current image according to the positioning information of the optimal matching position stored in the visual database.

Concrete implementation steps of the present invention are as follows:

(1) Data collection.

Collect images from the on-board camera, then perform ORB (ORiented Brief) feature point detection on each frame of image, and extract the descriptor of each feature point, then create and store database entries, where the database entries consist of previously extracted feature point descriptors set and the corresponding positioning information. Positioning information is composed of attitude information and position information provided by the UAV onboard equipment application program, and the format or nature of this part of information is highly dependent on the specific application program used.

(2) Feature extraction.

ORB feature point detection uses the FAST (features from accelerated segment test) algorithm to detect feature points on each layer of the scale pyramid. Based on the gray value of the image around the feature point, detect the pixel value of a circle around the candidate feature point, if there are enough pixels in the area around the candidate point and the gray value of the candidate point is different enough If the difference is greater than the preset threshold), the candidate point is considered to be a feature point.

Specific steps are as follows:

1) ORB feature point detection.

See Figure 2. During the ORB feature point detection process, first use FAST corner point detection for the input image; then use the Harris corner point measurement method to calculate the Harris corner point response value from the selected FAST feature point; then according to the corner point response value After sorting the results, pick out the N feature points with the largest response value; then, use the gray centroid method to calculate the direction of the ORB feature points, and use BRIEF as the feature point description method; finally, each feature point generates a 256-bit binary point pair.

That is, the ORB feature uses the FAST feature point detection method to detect the FAST feature points, and then uses the Harris corner point measurement method to calculate the Harris corner point response value from the selected FAST feature points, and pick out the top N feature points with the largest response value.

Among them, the corner response function f _CRF of FAST feature points is defined as:

Among them, ε _d is the threshold value, I(x) is the pixel value of the pixel point in the neighborhood of the point to be measured, and I*p) is the pixel value of the current point to be measured.

The sum of the corner response function values of the point to be tested and all corresponding surrounding points is recorded as N. When N is greater than the set threshold value, the point to be tested is a FAST feature point, and the threshold is usually 12.

The specific processing flow of ORB feature point extraction is as follows:

The first step: rough extraction of feature points. Select a point in the image and mark it as p, take p as the center of the circle, and 3 pixels as the radius, and detect the pixel values of the corresponding points on the circumference whose position numbers are 1, 5, 9, and 13 (as shown in Figure 3, it includes 16 positions in total , during rough extraction, that is, to detect four points on the circumference located in the four directions of the center p, up, down, left, and right), if the pixel value of at least 3 points among these 4 points is greater than or smaller than the pixel value of point p, then point p is considered is a feature point.

The second step: remove local dense points. The non-maximum suppression algorithm is used to calculate, retain the feature points at the maximum position, and delete the rest of the feature points.

Step 3: Scale invariance of feature points. Build a pyramid to achieve multi-scale invariance of feature points. Set a scale factor scale (eg 1.2) and the number of layers nlevels of the pyramid (eg 8 layers). The original image is down-sampled into an nlevels layer image according to the scale factor, and the relationship between the down-sampled image I' of each layer and the original image I is:

I'=I/scale ^k (k=1,2,...,8)

Step 4: Rotation invariance of feature points. The gray-scale centroid method is used to calculate the direction of the feature point. The moment within the radius r of the feature point is the centroid, and the vector formed between the feature point and the centroid is the direction of the feature point.

The vector angle θ between the feature point and the centroid C is the main direction of the feature point:

θ＝arctan(C _x ,C _y )

Wherein, (C _x ,C _y ) represents the coordinates of the centroid C.

2) Feature point descriptor generation.

The ORB feature uses the descriptor BRIEF as the feature point description method. The BRIEF descriptor is composed of a binary string with a length of n. In this specific implementation, n is 256. The calculation formula of a binary value τ(p:x,y) in the descriptor is as follows:

Among them, p(x) and p(y) are the respective gray levels of two points in a pair of points, and the feature descriptor f _n (p) composed of n pairs of point pairs can be expressed as:

f _n (p)＝∑ _1≤i≤n 2 ^i-1 τ(p:x,y)

Construct the affine transformation matrix R _θ to make the descriptor invariant to rotation, and obtain the rotation-corrected version S _θ of the generator matrix S:

S _θ = R _θ S

θ为特征点主方向。The generating matrix S is composed of n point pairs (xi _, y _i ), i=1,2,...,2n,

θ is the main direction of the feature point.

The finally obtained feature point descriptor g _n (p, θ) = f _n (p) | x _i , y _i ∈ S _θ constitutes a 256-bit feature point descriptor.

(3) Matching positioning based on RBF neural network.

When the UAV GNSS/INS signal is unavailable, the system prompts the UAV to return to the home. Using the motion information of the UAV stored in the feature database, the image descriptors extracted on the return journey are matched with the descriptors previously inserted into the database to obtain the positioning information. The matching localization system based on RBF neural network consists of network pattern training and pattern localization. The specific way is:

1) Set the training mode.

Set the training mode, learn the training samples, and provide classification decisions.

The RBF network contains only one hidden layer, uses the distance between the input value and the center vector as the independent variable of the function, and uses the radial basis function as the activation function. The local approximation method can simplify the amount of calculation, because for an input X, only some neurons respond, and other neurons are approximately 0, then the response w adjusts the parameters.

Referring to Figure 4, the RBF neural network consists of an input layer, a hidden layer and an output layer, where

In the input layer, the transformation from the input space to the hidden layer space is nonlinear;

The hidden layer uses the radial basis function as the activation function of neurons, and the transformation from the hidden layer to the output layer space is linear;

The output layer, neurons using linear functions, is a linear combination of the output of hidden layer neurons;

The RBF network uses RBF as the "base" of the hidden unit to form the hidden layer space, and directly maps the input vector to the hidden space. When the center point is determined, the mapping relationship can be determined. The mapping from input to output of the network is nonlinear, while the output of the network is linear to the adjustable parameters, and the connection weights can be directly solved by the linear equations, thus greatly speeding up the learning speed and avoiding local minimum problems.

In this specific embodiment, the weight between the input layer and the hidden layer of the RBF neural network is fixed to 1, the transfer function of the hidden layer unit adopts the radial basis function, and the hidden layer neuron uses the layer weight vector w The vector distance between _i and the input vector X _i is multiplied by the bias b _i as the input of the neuron activation function. Taking the radial basis function as a Gaussian function, the output of the neuron is:

Among them, x represents the input data, that is, the input vector, _xi is the center of the basis function, and σ is the function width parameter, which is used to determine the input vector of each radial basic neuron.

2) RBF neural network learning.

There are three parameters to be learned by the RBF network: the center _xi and variance σ of the basis function and the weight w between the hidden layer and the output layer.

i. Determine the basis function center x _i .

The feature database is generated by the feature descriptor vector of the image collected by the camera, the center x _i of the kernel function is determined by the k-means clustering algorithm, and I different samples are randomly selected from the training samples as the initial center x _i (0), and randomly input into the training Sample X _k , determine which center the training sample is closest to, and find it to satisfy:

i(X _k )＝argmin||X _k -x _i (n)||

Where i=1,2,...,I, x _i (n) represents the i-th center of the radial basis function at the nth iteration, and the number of iteration steps is set to n=0. Adjust the basis function center by the following formula:

Among them, γ is the learning step size, 0<γ<1.

That is, iteratively updates the center of the basis function through iterative training. The update formula is: x _i (n+1)= _xi (n)+γ[X _k (n) _-xi (n)], when the last two iterations When the change of the updated processing result does not exceed the preset threshold, the update is stopped (the learning ends). At this time, it is considered that x _i (n+1) is approximately equal to x _i (n), and the center of the last updated basis function As the final iterative training output result x _i (i=1, 2, . . . , I). Otherwise, n=n+1.

ii. Determine the basis function variance σ.

After determining the center of the RBF neural network, its width is expressed as:

Among them, M is the number of hidden layer units, d _max is the maximum distance between the selected centers.

iii. Determine the weight w from the hidden layer to the output layer.

The unit connection weights from the hidden layer to the output layer are calculated by the least square method, that is,

In the formula, g _qi represents the weight of the vector of the qth input sample and the center of the basis function, X _q is the vector of the qth input sample, q=1,2,...,N, i=1,2, ..., I, N represent the number of samples.

3) Match positioning.

In view of the timing of the images captured by the UAV, when returning to the voyage, the images captured by the camera can be extracted at fixed intervals and the features can be extracted. For example, the image extraction features can be extracted every ten frames, and the feature descriptor vector can be generated. Using the trained RBF The network classifier conducts a neighborhood search to obtain the optimal matching position, that is, based on the trained RBF network classifier, the currently extracted feature descriptor and the shooting process (from the departure point to the destination voyage) saved in the database are taken. The optimal matching result of the feature descriptor of the picture, and check whether the similarity between the currently extracted descriptor and the optimal matching result does not exceed the preset similarity threshold, and if so, the position of the current optimal matching result Positioning information is obtained as the current position estimation result of the UAV when returning home.

Furthermore, error compensation of the navigation system can also be performed on the obtained position estimation result to obtain positioning information. If the similarity with the optimal matching position is lower than the predefined similarity threshold, the position can be defined as unknown, continue to collect the ground image of the area where the UAV is located, and further base on the speed and attitude information of the UAV and the latest obtained The positioning results for navigation.

The error formula of the navigation system is:

表示位置估计结果(当前最优匹配结果的位置)，

表示最近的前n次的最终位置估计结果(也可以是位置估计结果

)的平均标准误差，j＝1,2,...,n，n为预设值。即，将已获得的最近多次定位结果的平均标准误差作为当前补偿量，对当前的位置估计结果进行误差补偿后作为当前的最终位置估计结果

Among them, the symbol

Indicates the position estimation result (the position of the current best matching result),

Indicates the final position estimation result of the last n times (it can also be the position estimation result

), j=1,2,...,n, where n is a preset value. That is, the average standard error of the most recent positioning results that have been obtained is used as the current compensation amount, and the error compensation is performed on the current position estimation result as the current final position estimation result

Referring to Fig. 5, the RBF network matching and positioning process of the present invention is:

First, randomly sample the feature descriptor information stored in the visual database; and train the sampled feature descriptor secondary data (training RBF network): set the training mode, use K-means clustering, and determine the RBF center ; Determine the RBF width d according to the obtained center; according to the RBF center and width, use the least squares method to determine the connection weight from the hidden layer to the output layer, and finally determine the RBF network structure;

According to the RBF network structure obtained through training, the neighborhood matching is performed on the feature point descriptors generated by the collected images during the return to the voyage, and the optimal matching position is judged.

To sum up, in the data collection process, the present invention starts from the image acquisition by the camera, uses the ORB feature point extraction technology to perform feature point detection on the image, and extracts a descriptor for each key point. Create and store a database entry consisting of previously extracted descriptors and positioning information. The positioning information includes attitude information and position information of the UAV. In the RBF network, there are mainly three parameters that need to be solved: including the center of the basis function, the variance, and the weight from the hidden layer to the output layer. The present invention adopts self-organization to select the center learning method, and the first step uses the unsupervised learning process to solve the center and variance of the hidden layer basis function; the second step uses a supervised learning process, and finally uses the least squares method to directly obtain the hidden layer to Weights between output layers. In the process of visual matching and positioning, it also starts from the capture of images in the return operation state of the UAV. In order to reduce the similarity of adjacent images, an image is extracted every fixed frame, and then the key points are detected, and the same features as the data collection process are used. Point extraction methods extract descriptors for each keypoint. Using the RBF network, the shortest distance between the current image and the previously inserted database descriptor is obtained, and the optimal matching position is found. The location information of the current image is estimated according to the optimal matching position.

The above is only a specific embodiment of the present invention. Any feature disclosed in this specification, unless specifically stated, can be replaced by other equivalent or alternative features with similar purposes; all the disclosed features, or All method or process steps may be combined in any way, except for mutually exclusive features and/or steps.

Claims (5)

1. An unmanned aerial vehicle visual navigation positioning method based on an RBF network is characterized by comprising the following steps:

step S1: setting an RBF neural network for matching the feature point descriptors of the image, and training the neural network;

the RBF neural network comprises an input layer, a hidden layer and an output layer, wherein a transfer function of the hidden layer adopts a radial basis function;

the training samples are: in the navigation process of the unmanned aerial vehicle, images are acquired through an onboard camera; the feature vectors of the training samples are: feature point descriptors of the image obtained through ORB feature point detection processing;

step S2: constructing a visual database of the unmanned aerial vehicle during navigation:

in the navigation process of the unmanned aerial vehicle, images are collected through an airborne camera, ORB feature point detection processing is carried out on the collected images, descriptors of all feature points are extracted, and feature point descriptors of the current images are obtained; storing the feature point descriptors of the image and positioning information during image acquisition into a visual database;

and step S3: unmanned aerial vehicle vision navigation positioning based on visual database:

based on a fixed interval period, extracting an image acquired by an airborne camera to serve as an image to be matched;

carrying out ORB feature point detection processing on the image to be matched, and extracting a descriptor of each feature point to obtain a feature point descriptor of the image to be matched;

inputting the feature point descriptor of the image to be matched into a trained RBF neural network, and performing neighborhood search to obtain the optimal matching feature point descriptor of the image to be matched in a visual database;

detecting whether the similarity between the optimal matching feature point descriptor and the feature point descriptor of the image to be matched is smaller than a preset similarity threshold value or not; if so, continuing navigating based on the recently obtained visual navigation positioning result; if not, obtaining the current position estimation result of the unmanned aerial vehicle based on the positioning information recorded in the database by the optimal matching feature point descriptor

And by estimating the result of the obtained position

Error compensation of the navigation system is carried out to obtain the current visual navigation positioning result of the unmanned aerial vehicle

Wherein,

or alternatively

Visual navigation positioning result representing last previous n times

The average standard error of (a) is,

indicating the position estimation result of the most recent previous n times

J =1, 2.. And n, n is a preset value.

2. The method of claim 1, wherein a weight between an input layer to a hidden layer of the RBF neural network is fixed to 1.

3. The method of claim 2, wherein in training the RBF neural network, a plurality of basis function centers of the radial basis functions are determined using a k-means clustering algorithm;

the variance σ of the radial basis function is set as:

wherein M is the number of cells in the hidden layer, d _max Is the maximum distance between the centers of the basis functions;

the weight W between the weight of the hidden layer and the weight of the output layer is:

wherein x is _q Feature vector, x, representing the q-th input sample _i Representing the ith basis function center.

4. The method of claim 1, wherein the positioning information comprises pose information and position information of the drone.

5. The method according to claim 1, wherein in step S3, the interval for extracting the images collected by the onboard camera is: extracted once every ten frames.

Images