CN110750094A - Method, apparatus and system for determining pose change information of movable equipment - Google Patents

Abstract

The present disclosure relates to a method, device and system for determining pose change information of a movable device, which belongs to the technical field of machine vision. The method includes: according to the three-dimensional position information in the image of at least one feature point in the image captured by each three-dimensional image capturing component when the movable device is in the initial position, determining the physical point corresponding to each feature point relative to the movable device. In the process of moving the movable device, in the image currently captured by each three-dimensional image capturing component, determine the three-dimensional position information of the feature points contained in each image; according to the relative position information corresponding to the determined physical point , The currently determined three-dimensional position information of the feature point and the pre-stored relative pose information between different three-dimensional image capturing components, determine the pose change information of the movable device relative to the initial position. With the present disclosure, the pose change information can also be determined reliably under harsh environmental conditions.

Description

Method, apparatus and system for determining pose change information of movable equipment

technical field

The present disclosure relates to the technical field of machine vision, and in particular, to a method, apparatus and system for determining pose change information of a movable device.

Background technique

In the related art, the mobile device can be navigated in an unfamiliar environment by relying on images captured by a three-dimensional image capturing component installed in the movable device through the visual navigation technology. Wherein, the three-dimensional image capturing component may include two cameras. The images captured by the three-dimensional image capturing component may include two two-dimensional images, namely, two-dimensional image A and two-dimensional image B. The depth information of the pixel points in the two-dimensional image A may be determined based on the two-dimensional image A, the two-dimensional image B, and the baseline distance between the two cameras. The three-dimensional position information of the pixel point may include the two-dimensional position information and depth information of the pixel point in the two-dimensional image A.

When the removable device is powered on, the removable device can be considered to be in the initial position. The movable device can identify whether the image captured by the 3D image capturing component contains an image of an obstacle. If so, it is necessary to continuously detect the relative position information of the obstacle relative to the moved movable device during the movement of the movable device. , so that it is possible to determine how to control the movable device to move to avoid the obstacle based on the relative position information of the obstacle relative to the moved movable device. The relative position information of the obstacle relative to the moved movable device can be determined according to the relative position information of the obstacle relative to the movable device at the initial position and the pose change information of the movable device relative to the initial position. Among them, the position and attitude change information records the degree of change in position and attitude of the movable device relative to the initial position. The above-mentioned relative position information is the relative position information of the obstacle relative to the movable device when the position and attitude of the movable device are considered. For example, the relative position information can be that the obstacle is in the coordinate system established based on the movable device. The coordinates of , or the relative position information can be that the obstacle is at a certain distance from a certain angle to the left (or right) of the movable device.

The method of determining the pose change information is described below: the relative position information of the movable device relative to the initial position of the physical point corresponding to the feature point and the relative position information of the movable device in the moving process of the three-dimensional image capturing part can be used. The three-dimensional position information of the feature points in the image determines the pose change information. Among them, the feature points can be pixels with certain image characteristics in the image, which can be extracted from the image through a preset feature extraction algorithm, such as FAST (Features from Accelerated Segment Test, a corner detection method) corner points, for example The four corners of the display are identified as FAST corners.

In the process of realizing the present disclosure, the inventor found at least the following problems:

There is only one 3D image capturing component installed in the movable device. If the 3D image capturing component is shooting strong light, the captured image will be too bright, making it difficult to accurately identify the feature points in the image, and then determine the change in pose. The information is also not accurate.

SUMMARY OF THE INVENTION

In order to overcome the problems existing in the related art, the present disclosure provides the following technical solutions:

According to a first aspect of the embodiments of the present disclosure, there is provided a method for determining pose change information of a movable device, where the movable device includes at least two three-dimensional image capturing components, and the method includes:

According to the three-dimensional position information in the image of at least one feature point in the image captured by each three-dimensional image capturing component when the movable device is in the initial position, determine the physical point corresponding to each feature point relative to the movable device. relative location information;

During the movement of the movable device, in the images currently captured by each 3D image capturing component, determine the 3D position information of the feature points included in each image;

Determine the relative position information of the movable device relative to the initial position according to the relative position information corresponding to the determined physical point, the three-dimensional position information of the currently determined feature point, and the pre-stored relative pose information between different three-dimensional image capturing components. Pose change information.

Optionally, according to the three-dimensional position information in the image of at least one feature point in the image captured by each three-dimensional image capturing component when the movable device is in the initial position, it is determined that the physical point corresponding to each feature point is relative to the actual point. The relative position information of the movable device, including:

According to the three-dimensional position information in the image of at least one feature point in the image captured by each 3D image capturing component when the movable device is in the initial position, it is determined that the physical point corresponding to each feature point is relative to the corresponding 3D image capturing component relative position information;

According to the relative position information of the physical point corresponding to each feature point relative to the corresponding 3D image capturing component, and the relationship between the pre-stored reference 3D image capturing component and other 3D image capturing components in the pre-stored at least two 3D image capturing components Determine the relative position information of the physical point corresponding to each feature point relative to the reference three-dimensional image capturing component, as the relative position information of the physical point corresponding to each feature point relative to the movable device.

Optionally, according to the relative position information corresponding to the determined physical point, the three-dimensional position information of the currently determined feature point, and the relative pose information between the pre-stored different three-dimensional image capturing components, determine the relative position of the movable device. The pose change information at the initial position, including:

According to the currently determined 3D position information of the feature point and the pre-stored relative pose information between the reference 3D image capturing component and other 3D image capturing components, determine the current position of each feature point relative to the reference 3D image capturing component. The three-dimensional position information of the captured image;

According to the relative position information of the physical point corresponding to each feature point with respect to the reference 3D image capturing part, and the 3D position information of each feature point with respect to the image currently captured by the reference 3D image capturing part, determine the possible The pose change information of the mobile device relative to the initial position.

Optionally, the method further includes:

In the feature points contained in the image currently captured by each three-dimensional image capturing component, determine a target feature point that matches the image feature of any feature point in the at least one feature point, and determine the three-dimensional position information of each target feature point;

The relative position information corresponding to the determined physical point, the currently determined three-dimensional position information of the feature point, and the pre-stored relative pose information between different three-dimensional image capturing components, determine the relative position of the movable device relative to the initial The pose change information of the position, including:

Determine the relative position information of the movable device relative to the initial position according to the relative position information corresponding to the determined physical point, the three-dimensional position information of each target feature point, and the pre-stored relative pose information between different three-dimensional image capturing components. Pose change information.

Optionally, according to the determined relative position information corresponding to the physical point, the three-dimensional position information of each target feature point, and the pre-stored relative pose information between different three-dimensional image capturing components, determine the relative position of the movable device. The pose change information at the initial position, including:

If the number of the target feature points is greater than or equal to the preset number threshold, then according to the relative position information corresponding to the determined physical point, the three-dimensional position information of each target feature point and the pre-stored difference between different three-dimensional image capturing components The relative pose information is to determine the pose change information of the movable device relative to the initial position.

Optionally, the method further includes:

If the number of the target feature points is less than the preset number threshold, the current position is set as the initial position, according to at least one of the images captured by each three-dimensional image capturing component when the movable device is in the reset initial position The three-dimensional position information of the feature points in the image determines the relative position information of the physical point corresponding to each feature point relative to the movable device.

According to a second aspect of the embodiments of the present disclosure, there is provided an apparatus for determining pose change information of a movable device, where the movable device includes at least two three-dimensional image capturing components, and the apparatus includes:

The determining module is configured to determine, according to the three-dimensional position information in the image of at least one feature point in the image captured by each three-dimensional image capturing component when the movable device is in the initial position, the physical point corresponding to each feature point is relative to all the feature points. the relative location information of the movable device;

The determining module is further configured to, in the process of moving the movable device, determine the three-dimensional position information of the feature points included in each image in the image currently captured by each three-dimensional image capturing component;

The determining module is further configured to determine the movable relative position information according to the relative position information corresponding to the determined physical point, the three-dimensional position information of the currently determined feature point, and the relative pose information between different pre-stored three-dimensional image capturing components. The pose change information of the device relative to the initial position.

Optionally, the determining module is used for:

According to the three-dimensional position information in the image of at least one feature point in the image captured by each 3D image capturing component when the movable device is in the initial position, it is determined that the physical point corresponding to each feature point is relative to the corresponding 3D image capturing component relative position information;

According to the relative position information of the physical point corresponding to each feature point relative to the corresponding 3D image capturing component, and the relationship between the pre-stored reference 3D image capturing component and other 3D image capturing components in the pre-stored at least two 3D image capturing components Determine the relative position information of the physical point corresponding to each feature point relative to the reference three-dimensional image capturing component, as the relative position information of the physical point corresponding to each feature point relative to the movable device.

Optionally, the determining module is used for:

According to the currently determined 3D position information of the feature point and the pre-stored relative pose information between the reference 3D image capturing component and other 3D image capturing components, determine the current position of each feature point relative to the reference 3D image capturing component. The three-dimensional position information of the captured image;

According to the relative position information of the physical point corresponding to each feature point with respect to the reference 3D image capturing part, and the 3D position information of each feature point with respect to the image currently captured by the reference 3D image capturing part, determine the possible The pose change information of the mobile device relative to the initial position.

Optionally, the determining module is also used for:

In the feature points contained in the image currently captured by each three-dimensional image capturing component, determine a target feature point that matches the image feature of any feature point in the at least one feature point, and determine the three-dimensional position information of each target feature point;

Determine the relative position information of the movable device relative to the initial position according to the relative position information corresponding to the determined physical point, the three-dimensional position information of each target feature point, and the pre-stored relative pose information between different three-dimensional image capturing components. Pose change information.

Optionally, the determining module is used for:

When the number of the target feature points is greater than or equal to the preset number threshold, according to the relative position information corresponding to the determined physical point, the three-dimensional position information of each target feature point and the pre-stored difference between different three-dimensional image capturing components The relative pose information is to determine the pose change information of the movable device relative to the initial position.

Optionally, the determining module is also used for:

When the number of the target feature points is less than the preset number threshold, the current position is set as the initial position, according to at least one of the images captured by each three-dimensional image capturing component when the movable device is in the reset initial position The three-dimensional position information of the feature points in the image determines the relative position information of the physical point corresponding to each feature point relative to the movable device.

According to a third aspect of the embodiments of the present disclosure, there is provided a system for determining pose change information of a movable device, the system includes a server and a movable device, the movable device includes at least two three-dimensional image capturing components, wherein :

The movable device is configured to capture an image through each 3D image capturing component whenever a preset period arrives, and send the image captured by each 3D image capturing component to the server;

The server is configured to, according to the three-dimensional position information in the image of at least one feature point in the image captured by each three-dimensional image capturing component when the movable device is in the initial position, determine that the physical point corresponding to each feature point is relative to the actual point. relative position information of the movable device; during the movement of the movable device, in the images currently captured by each 3D image capturing component, determine the 3D position information of the feature points included in each image; according to the determined The relative position information corresponding to the physical point, the three-dimensional position information of the currently determined feature points, and the pre-stored relative pose information between different three-dimensional image capturing components, to determine the pose change of the movable device relative to the initial position information.

According to a fourth aspect of embodiments of the present disclosure, there is provided a server, the server comprising a processor, a communication interface, a memory and a communication bus, wherein:

The processor, the communication interface and the memory communicate with each other through the communication bus;

the memory for storing computer programs;

The processor is configured to execute the program stored in the memory, so as to realize the above-mentioned method for determining the pose change information of the movable device.

According to a fifth aspect of the embodiments of the present disclosure, there is provided a computer-readable storage medium, where a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, the above-mentioned determination of the pose of a movable device is implemented Methods of changing information.

The technical solutions provided by the embodiments of the present disclosure may include the following beneficial effects:

With the method provided by the embodiment of the present disclosure, the pose change information of the movable device relative to the initial position can be determined through images captured by at least two three-dimensional image capturing components. Even if one of the three-dimensional image capturing parts is affected by strong light or other factors, the pose change information of the movable device relative to the initial position can be determined by using the images captured by the remaining three-dimensional image capturing parts. Furthermore, the pose change information can also be reliably determined under harsh environmental conditions.

It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure.

Description of drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the disclosure and together with the description serve to explain the principles of the disclosure. In the attached image:

1 is a schematic flowchart of a method for determining pose change information of a movable device according to an exemplary embodiment;

FIG. 2 is a schematic structural diagram of determining a base of a movable device according to an exemplary embodiment;

3 is a schematic diagram of a method for determining pose change information of a movable device according to an exemplary embodiment;

4 is a schematic diagram of a transformation relationship of a method for determining pose change information of a movable device according to an exemplary embodiment;

5 is a schematic structural diagram of an apparatus for determining pose change information of a movable device according to an exemplary embodiment;

Fig. 6 is a schematic structural diagram of a server according to an exemplary embodiment.

illustration:

1-6 cameras; 7 hardware trigger boards;

8 metal bottom plate; 9-11 wiring holes;

12-17 Rectangular groove

The above-mentioned drawings have shown clear embodiments of the present disclosure, and will be described in more detail hereinafter. These drawings and written descriptions are not intended to limit the scope of the disclosed concepts in any way, but rather to illustrate the disclosed concepts to those skilled in the art by referring to specific embodiments.

Detailed ways

Exemplary embodiments will be described in detail herein, examples of which are illustrated in the accompanying drawings. Where the following description refers to the drawings, the same numerals in different drawings refer to the same or similar elements unless otherwise indicated. The implementations described in the illustrative examples below are not intended to represent all implementations consistent with this disclosure. Rather, they are merely examples of apparatus and methods consistent with some aspects of the present disclosure as recited in the appended claims.

The embodiments of the present disclosure provide a method for determining pose change information of a movable device, and the method can be implemented by a movable terminal device and a server in cooperation. Among them, the movable terminal equipment may be a sweeper, a robot, an unmanned vehicle, or the like.

A server may include components such as processors, memory, and the like. The processor, which can be a CPU (Central Processing Unit, central processing unit), etc., can be used for the three-dimensional position information in the image of at least one feature point in the image captured by each three-dimensional image capturing component when the movable device is in the initial position, Determine the relative position information of the physical point corresponding to each feature point relative to the movable device, etc. The memory, which can be RAM (Random Access Memory, Random Access Memory), Flash (flash memory), etc., can be used to store the received data, the data required for the processing process, the data generated in the processing process, etc., such as physical points corresponding to relative position information, etc.

The server may also include transceivers and the like. The transceiver may be used for data transmission with the movable device, and the transceiver may include a Bluetooth component, a WiFi (Wireless-Fidelity, wireless high-fidelity technology) component, an antenna, a matching circuit, a modem, and the like.

An exemplary embodiment of the present disclosure provides a method for determining pose change information of a movable device, wherein the movable device includes at least two three-dimensional image capturing components. As shown in FIG. 1 , the processing flow of the method may include: Follow the steps below:

Step S110, according to the three-dimensional position information in the image of at least one feature point in the image captured by each three-dimensional image capturing component when the movable device is in the initial position, determine the relative position of the physical point corresponding to each feature point relative to the movable device. location information.

In the embodiment of the present disclosure, at least two three-dimensional image capturing components may be provided in the movable device, and three three-dimensional image capturing components are recommended (including six cameras in total). As shown in FIG. 2 , it is a schematic diagram of the structure of the base of the movable device. In the base of the movable device, the parts corresponding to the numbers 1-6 are six different cameras. The component corresponding to the label 7 is a hardware trigger board, which is used to control six cameras to perform synchronous exposure. The component corresponding to the numeral 8 is a metal bottom plate, and a camera and a hardware trigger board can be arranged above the metal bottom plate. The metal base plate can also be provided with three wiring holes corresponding to the numbers 9-11, and the wiring holes can be used to output the relevant wires of the camera and the hardware trigger board to the bottom of the base of the movable device. The metal base plate may also be provided with six rectangular grooves corresponding to the reference numerals 12-17, and the rectangular grooves are used to fix the camera on the metal base plate.

The six cameras can be independent industrial cameras, and the types of cameras can include ordinary pinhole cameras, fisheye cameras, and the like. The two adjacent cameras that are placed in parallel are a pair of binocular cameras (three-dimensional image capturing components). The viewing angles between different pairs of binocular cameras may or may not overlap.

The metal base plate can be set as a triangle with three sides in total, three pairs of binocular cameras are respectively arranged on one of the three sides, and two cameras in the pair of binocular cameras shoot in the same direction. The baseline distance of each pair of binocular cameras can be set to 12 cm. The movable device may include three 3D image capturing components, and the included angle between any two 3D image capturing components and the line connecting the preset reference center of the movable device is equal to 120 degrees, that is, adjacent pairs of binocular cameras in different pairs. The angle between the two cameras can be set to 120 degrees.

After the hardware trigger board controls the six cameras for synchronous exposure, the captured images can be sent to the server for subsequent processing, and the processing results are returned to the removable device. The method provided in this embodiment is mainly executed by the server, and executed by the movable device.

In the embodiment of the present disclosure, one three-dimensional image capturing component may include two cameras. Since the pixels in the two-dimensional image A captured by a single camera lack depth information, the two-dimensional image B captured by another camera needs to be used to determine the depth information of the pixels in the two-dimensional image A. Thus, a three-dimensional image can be established based on the two-dimensional image A and the depth information of the pixels in the two-dimensional image A, and each pixel in the three-dimensional image has corresponding three-dimensional position information.

Next, feature points in images captured by all cameras can be extracted. All can be extracted by algorithms such as ORB (Oriented FAST and Rotated BRIEF, a fast feature point extraction and description algorithm), DAISY (a feature point extraction algorithm) and SIFT (Scale-invariant feature transform, scale-invariant feature transform) algorithm. Feature points in the image captured by the camera. In the embodiments of the present disclosure, the corner points of FAST (Features from Accelerated Segment Test, a corner point detection method) extracted by the ORB algorithm are mainly used as an example for description, and other algorithms are similar, and are not repeated here.

The ORB algorithm mainly uses any pixel in the captured image as the center of the circle, and analyzes the characteristics of the gray value of the surrounding 16 pixels. If the gray value of the 16 pixels is greater than the gray value of the center of the circle plus the preset gray value threshold, and all of them are smaller than the gray value of the center of the circle minus the preset gray value threshold, there is a continuous N (generally Below, N is 9) pixels, then the center of the circle is the FAST corner.

Taking the point P in the captured image as the center of the circle, there are three states between the 16 pixels around it and the gray value of point P, that is, the gray value of the pixels in the 16 pixels is brighter than the point P, and the gray value of the pixels in the 16 pixels The grayscale of the pixels in the pixel points is similar to that of the point P, and the grayscale of the pixels in the 16 pixel points is darker than that of the point P. It can be determined by Equation 1:

Among them, S _p→x is the three states when the grayscale of the pixel point in the 16 pixel points is compared with the point P, namely d (darker), s (similar) and b (bright). I identifies the gray value, and T is a preset gray value threshold, which can be adjusted according to the environment.

The properties of FAST corners can be described by the BRIEF descriptor. A circular area with a diameter of 31 can be selected around the FAST corner, and according to predetermined rules, N pairs of pixels are selected to compare the gray values, and a 256-bit binary descriptor is generated according to the comparison result. It can be used as the Brief descriptor of the FAST corner.

Before determining the relative position information of the physical point corresponding to each feature point with respect to the movable device, the captured image can be corrected. Since the captured image has certain distortion, the captured image can be corrected to obtain an ideal image. Ideally, as shown in Figure 3, the camera coordinate system of a pair of binocular cameras is to translate the origin from point O2 to point O1, and the coordinate system does not rotate.

After the ideal image is obtained, for each pair of binocular cameras, the FAST corners that match the FAST corners in the image captured by the left camera can be determined in the image captured by the right camera. And the depth information of FAST corner points can be determined according to the baseline distance of the camera.

In the embodiment of the present disclosure, it can be considered that the right camera assists the left camera in determining the depth information of the FAST corner points in the image captured by the left camera, and mainly performs the determination processing of the pose change information based on the image captured by the left camera.

The camera coordinate system can be established separately based on the left camera in each pair of binocular cameras. Among them, the camera coordinate system takes the optical center of the camera as the origin, the optical axis as the Z axis, the line where the long side of the captured image is located as the X axis, and the line where the wide side of the captured image is located as the Y axis. coordinate system. Thus, for the device in Figure 2, three camera coordinate systems can be obtained. Among the three camera coordinate systems, the camera coordinate system established by the camera corresponding to label 1 may be used as the body coordinate system.

Before performing step S110, the relative pose information of the camera coordinate system and the body coordinate system in the three camera coordinate systems other than the body coordinate system can be pre-calibrated, which can also be referred to as the one of the three camera coordinate systems except the body coordinate system. The external parameters of the external camera coordinate system relative to the body coordinate system are respectively denoted as T _bl (the camera corresponding to the label 4) and T _br (the camera corresponding to the label 5).

Next, according to the three-dimensional position information in the image of at least one feature point in the image captured by each three-dimensional image capturing component when the movable device is in the initial position, it can be determined that the physical point corresponding to each feature point is relative to the corresponding three-dimensional image capturing The relative position information of the part. Finally, according to the relative position information of the physical point corresponding to each feature point with respect to the corresponding 3D image capture component, and the relationship between the pre-stored reference 3D image capture component and other 3D image capture components in the pre-stored at least two 3D image capture components The relative pose information between each feature point is determined, and the relative position information of the physical point corresponding to each feature point relative to the reference three-dimensional image capturing component is determined as the relative position information of the physical point corresponding to each feature point relative to the movable device.

Wherein, the above-mentioned relative position information is the relative position information of the obstacle relative to the movable device in the case of the current position and attitude of the movable device. If the current movable device is at the initial position, the relative position information is the relative position information of the obstacle relative to the initial position of the movable device. The relative position information of the physical point relative to the movable device can be the coordinate information of the physical point in the coordinate system established with the preset point in the movable device as the origin, or the physical point can be captured with the reference 3D image of the movable device. The optical center of the component is the origin, the line where the optical axis is located is the Z axis, the vertical line is the Y axis, and the line perpendicular to both the Z axis and the Y axis is the coordinate information in the coordinate system established by the X axis.

In the application, the relative position information of the physical point relative to the movable device at the initial position A can indicate that the physical point is 30 degrees and 20 meters in front of the movable device. After the movable device moves to position B, the relative position information of the physical point relative to the movable device at position B can indicate that the physical point is located 60 degrees and 15 meters in front of the movable device to the right.

The at least two 3D image capturing components may include a reference 3D image capturing component (camera corresponding to reference numeral 1) and a non-reference 3D image capturing component (cameras corresponding to reference numerals 4 and 5 respectively) other than the reference 3D image capturing component.

First, the three-dimensional position information in the image of at least one feature point in the image captured by each non-reference 3D image capturing component and the pre-stored corresponding to each non-reference 3D image capturing component can be based on The conversion relationship between the three-dimensional position information of the feature points and the relative position information of the physical point relative to the non-reference three-dimensional image capturing component (also called the camera internal reference), respectively determines the value of the image captured by each non-reference three-dimensional image capturing component. The relative position information of the physical point corresponding to the at least one feature point with respect to the non-reference three-dimensional image capturing component.

Next, according to the relative position information of the physical point corresponding to at least one feature point in the image captured by each non-reference 3D image capturing component relative to the non-reference 3D image capturing component, and the relative position information of each non-reference 3D image capturing component relative to the reference The relative pose information of the three-dimensional image capture component (also referred to as the external parameters of the camera coordinate system other than the body coordinate system in the three camera coordinate systems relative to the body coordinate system), to determine the capture of each non-reference three-dimensional image capture component. Relative position information (also referred to as map points) of the physical point corresponding to at least one feature point in the image relative to the reference three-dimensional image capturing component. Wherein, the initial position where the reference three-dimensional image capturing component is located is the initial position where the movable device is located.

Finally, the three-dimensional position information in the image of at least one feature point in the image captured by the reference 3D image capturing component when the movable device is in the initial position, and the 3D position information of the pre-stored feature point corresponding to the reference 3D image capturing component can be used. and the conversion relationship between the physical point relative to the relative position information of the reference 3D image capturing component, to determine the relative position information of the physical point corresponding to at least one feature point in the image captured by the reference 3D image capturing component relative to the reference 3D image capturing component (also known as map points).

Based on the following formula, the relative position information (also referred to as a map point) of the physical point corresponding to each feature point relative to the movable device can be determined.

为FAST角点，Z为该FAST角点的深度信息。in, is the coordinates of the map point in the world coordinate system. K is the camera internal parameter corresponding to different 3D image capturing components,

is the FAST corner, and Z is the depth information of the FAST corner.

Among them, the world coordinate system is the body coordinate system of the movable device when it is powered on, the world coordinate system is fixed, and the body coordinate system will translate and rotate with the movement of the movable device.

Through the internal parameters of the camera, the position information of the pixel points in the captured image can be converted into the position information of the physical point in the camera coordinate system. Among them, the camera internal parameters include the camera's inherent property parameters such as the shooting focal length of the camera.

After getting multiple map points, the map is established. The location information of different map points in the world coordinate system and their corresponding Brief descriptors are recorded in the map. It should be noted that the selection of the number of map points can be determined according to the length of the distance that the movable device moves each time. During the movement of the movable device, new images are continuously captured, and map points are continuously matched with feature points in the captured new images. When the number of map points that match the feature points in the new captured image is less than a preset number threshold, the map is rebuilt to generate a new map point. The currently matched map points are no longer involved in the process of generating new map points.

Since the generation of the map point is related to the frame number of the captured image, the frame number of the image can be recorded corresponding to the map point. In practical applications, for example, the frame number of the currently captured image is N, when performing matching processing, the inserted map point is generated based on the image whose frame number is N-T, and the projection processing of the map point in the currently captured image is performed. Among them, T is a fixed value, which can be adjusted according to the specific situation.

In step S120, during the movement of the movable device, in the images currently captured by each 3D image capturing component, determine the 3D position information of the feature points included in each image.

During the movement of the movable device, the three-dimensional position information of the feature points included in each image may be determined according to a preset period in the images currently captured by each three-dimensional image capturing component.

The three-dimensional position information of the pixel point in the image can be converted to the position information of the physical point in the camera coordinate system through the camera internal parameters. For the device in FIG. 2 , the position information of the physical point corresponding to the pixel point in the image captured by the camera corresponding to the label 4 and the camera corresponding to the label 5 can be obtained through the camera internal parameters of the camera corresponding to the label 4 and the camera corresponding to the label 5. The camera internal parameters are converted into the camera coordinate system of the camera corresponding to the label 4 and the camera coordinate system of the camera corresponding to the label 5 . As shown in FIG. 4 , the relative position information between different three-dimensional image capturing components can be the external parameters of the camera coordinate systems other than the body coordinate system in the three camera coordinate systems relative to the body coordinate system, which are respectively denoted as T _b1 (the camera corresponding to the number 4) and T _br (the camera corresponding to the number 5). The position information of the physical point in the camera coordinate system of the camera corresponding to the label 4 and the camera coordinate system of the camera corresponding to the label 5 can be converted into the position information of the physical point in the body coordinate system through the external parameters T _bl and T _br . . In this way, the position information of the physical point corresponding to the images captured by the three different three-dimensional image capturing components can be converted into the position information of the physical point in the body coordinate system. Furthermore, the position information of the physical point in the body coordinate system can be converted into the position information of the physical point in the world coordinate system through the pose change information.

The method provided by the embodiment of the present disclosure further includes: among the feature points included in the image currently captured by each three-dimensional image capturing component, determining a target feature point that matches the image feature of any feature point in the at least one feature point, and determining each feature point. The three-dimensional position information of the target feature point.

Since the current pose change information has not yet been determined, the map point (position information of the physical point in the world coordinate system) can be projected into the currently captured image with the help of the last determined pose change information. Projection points are obtained in the captured image. Among the Brief descriptors corresponding to the FAST corner points contained in the field of projection points, select the target feature point m whose distance from the feature vector of the Brief descriptor of the map point is less than the preset distance threshold, then the target feature point m is the same as the map point. point to match the projected point. Determine the three-dimensional position information of each target feature point.

As shown in Figure 4, if there are three map points, namely P ₁ , P ₂ , and P _n , and the matching projection points are m ₁ , m ₂ , and m _n , the following formula can be used to determine the map points P ₁ , m 2 , and m n . P ₂ , P _n and other related parameters to determine the three-dimensional position information of m ₁ , m ₂ , and m _n .

m ₁ =K ₁ T ^-1 P ₁ ; m ₂ =K ₂ T _bl ^-1 T ^-1 P ₂ ; m ₃ =K ₃ T _br ^-1 T ^-1 P ₃ (Equation 3)

Among them, K is the camera internal parameters corresponding to different cameras, T _bl and T _br are external parameters, and T is the last determined pose change information. The last determined pose change information is only the estimated pose change information, and cannot be directly used as the current pose change information.

Step S130: Determine the position of the movable device relative to the initial position according to the relative position information corresponding to the determined physical point, the three-dimensional position information of the currently determined feature point, and the pre-stored relative pose information between different three-dimensional image capturing components. posture change information.

First, according to the currently determined three-dimensional position information of the feature point and the pre-stored relative pose information between the reference three-dimensional image capturing component and other three-dimensional image capturing components, it is possible to determine the current capture of each feature point relative to the reference three-dimensional image capturing component. 3D position information of the image. Next, according to the relative position information of the physical point corresponding to each feature point relative to the reference 3D image capturing component, and the 3D position information of each feature point relative to the image currently captured by the reference 3D image capturing component, determine the relative position of the movable device relative to the reference 3D image capturing component. Pose change information at the initial position.

The at least two three-dimensional image capturing parts include a reference three-dimensional image capturing part and a non-reference three-dimensional image capturing part other than the reference three-dimensional image capturing part.

First, in the process of moving the movable device, determine the three-dimensional position information of the feature points in the image contained in the image currently captured by each non-reference 3D image capturing component, and determine the 3D position information contained in the image currently captured by the reference 3D image capturing component. The three-dimensional position information of feature points in the image.

Next, according to the three-dimensional position information in the image of the feature points contained in the image currently captured by each non-reference 3D image capturing component, and the relative pose information of each non-reference 3D image capturing component relative to the reference 3D image capturing component, The three-dimensional position information of the feature points included in the image currently captured by each non-reference 3D image capturing component relative to the image currently captured by the reference 3D image capturing component is respectively determined.

Finally, according to the relative position information corresponding to the physical point, the three-dimensional position information of the feature points contained in the image currently captured by each non-reference 3D image capturing part relative to the image currently captured by the reference 3D image capturing part, and the reference 3D image capturing part The three-dimensional position information of the feature points contained in the currently captured image in the image determines the pose change information of the movable device relative to the initial position.

Optionally, among the feature points included in the image currently captured by each three-dimensional image capturing component, a target feature point that matches the image feature of any feature point in the at least one feature point can be determined, and the three-dimensional feature point of each target feature point can be determined. location information. Next, according to the relative position information corresponding to the determined physical point, the three-dimensional position information of each target feature point, and the relative pose information between different pre-stored three-dimensional image capturing components, determine the pose of the movable device relative to the initial position change information.

Optionally, if the number of target feature points is greater than or equal to a preset number threshold, then according to the relative position information corresponding to the determined physical point, the three-dimensional position information of each target feature point and the pre-stored difference between the three-dimensional image capturing components. The relative pose information between them is used to determine the pose change information of the movable device relative to the initial position. If the number of target feature points is less than the preset number threshold, the current position is set as the initial position, according to at least one feature point in the image captured by each 3D image capturing component when the movable device is in the reset initial position The three-dimensional position information in the , determine the relative position information of the physical point corresponding to each feature point relative to the movable device.

During the movement of the movable device, new images are continuously captured, and map points are continuously matched with feature points in the captured new images. When the matching number of map points and feature points in the new captured image is less than a preset number threshold, the map is re-established to generate a new map point. If there are currently matched map points, they no longer participate in the process of generating new map points.

The current pose change information can be determined according to formula 4.

min[||(m ₁ -K ₁ T ^-1 P ₁ )|| ² +||(m ₂ -K ₂ T _bl ^-1 T ^-1 P ₂ )|| ² +||(m ₃ -K ₃ T _br ^-1 T ^-1 P ₃ )|| ² ] (Equation 4)

In the above formula, T is unknown, it is the current pose change information, and the rest of the parameters are known. The value of T can be optimized iteratively, so that when T is a certain value, the final result of formula 4 is the smallest, that is, the three-dimensional position information of the target feature point that can be matched with the map point is determined and calculated theoretically by formula 3. The error T with the smallest error between the three-dimensional position information of the projection points is the current pose change information.

With the method provided by the embodiment of the present disclosure, the pose change information of the movable device relative to the initial position can be determined through images captured by at least two three-dimensional image capturing components. Even if one of the three-dimensional image capturing parts is affected by strong light or other factors, the pose change information of the movable device relative to the initial position can be determined by using the images captured by the remaining three-dimensional image capturing parts. Furthermore, the pose change information can also be reliably determined under harsh environmental conditions.

Yet another exemplary embodiment of the present disclosure provides an apparatus for determining pose change information of a movable device, where the movable device includes at least two three-dimensional image capturing components. As shown in FIG. 5 , the apparatus includes:

The determination module 510 is configured to determine, according to the three-dimensional position information in the image of at least one feature point in the image captured by each three-dimensional image capturing component when the movable device is in the initial position, the physical point corresponding to each feature point relative to the actual point. relative position information of the movable device;

The determining module 510 is further configured to determine, in the image currently captured by each 3D image capturing component, the 3D position information of the feature points included in each image during the movement of the movable device;

The determining module 510 is further configured to determine the possible relative position information according to the relative position information corresponding to the determined physical point, the three-dimensional position information of the currently determined feature point, and the pre-stored relative pose information between different three-dimensional image capturing components. The pose change information of the mobile device relative to the initial position.

Optionally, the determining module 510 is configured to:

According to the three-dimensional position information in the image of at least one feature point in the image captured by each 3D image capturing component when the movable device is in the initial position, it is determined that the physical point corresponding to each feature point is relative to the corresponding 3D image capturing component relative position information;

According to the relative position information of the physical point corresponding to each feature point relative to the corresponding 3D image capturing component, and the relationship between the pre-stored reference 3D image capturing component and other 3D image capturing components in the pre-stored at least two 3D image capturing components Determine the relative position information of the physical point corresponding to each feature point relative to the reference three-dimensional image capturing component, as the relative position information of the physical point corresponding to each feature point relative to the movable device.

Optionally, the determining module 510 is configured to:

According to the currently determined 3D position information of the feature point and the pre-stored relative pose information between the reference 3D image capturing component and other 3D image capturing components, determine the current position of each feature point relative to the reference 3D image capturing component. The three-dimensional position information of the captured image;

According to the relative position information of the physical point corresponding to each feature point with respect to the reference 3D image capturing part, and the 3D position information of each feature point with respect to the image currently captured by the reference 3D image capturing part, determine the possible The pose change information of the mobile device relative to the initial position.

Optionally, the determining module 510 is further configured to:

In the feature points contained in the image currently captured by each three-dimensional image capturing component, determine a target feature point that matches the image feature of any feature point in the at least one feature point, and determine the three-dimensional position information of each target feature point;

Determine the relative position information of the movable device relative to the initial position according to the relative position information corresponding to the determined physical point, the three-dimensional position information of each target feature point, and the pre-stored relative pose information between different three-dimensional image capturing components. Pose change information.

Optionally, the determining module 510 is configured to:

When the number of the target feature points is greater than or equal to the preset number threshold, according to the relative position information corresponding to the determined physical point, the three-dimensional position information of each target feature point and the pre-stored difference between different three-dimensional image capturing components The relative pose information is to determine the pose change information of the movable device relative to the initial position.

Optionally, the determining module 510 is further configured to:

When the number of the target feature points is less than the preset number threshold, the current position is set as the initial position, according to at least one of the images captured by each three-dimensional image capturing component when the movable device is in the reset initial position The three-dimensional position information of the feature points in the image determines the relative position information of the physical point corresponding to each feature point relative to the movable device.

Regarding the apparatus in the above-mentioned embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be described in detail here.

With the apparatus provided by the embodiment of the present disclosure, the pose change information of the movable device relative to the initial position can be determined through images captured by at least two three-dimensional image capturing components. Even if one of the three-dimensional image capturing parts is affected by strong light or other factors, the pose change information of the movable device relative to the initial position can be determined by using the images captured by the remaining three-dimensional image capturing parts. Furthermore, the pose change information can also be reliably determined under harsh environmental conditions.

It should be noted that: when the device for determining the pose change information of the movable device provided in the above embodiment determines the pose change information of the movable device, only the division of the above functional modules is used as an example for illustration. The above-mentioned function distribution is completed by different function modules as required, that is, the internal structure of the server is divided into different function modules, so as to complete all or part of the functions described above. In addition, the apparatus for determining the pose change information of the movable device provided by the above embodiments and the method embodiment for determining the pose change information of the movable device belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here. .

Yet another exemplary embodiment of the present disclosure provides a system for determining pose change information of a movable device, the system includes a server and a movable device, the movable device includes at least two three-dimensional image capturing components, wherein:

The movable device is configured to capture an image through each 3D image capturing component whenever a preset period arrives, and send the image captured by each 3D image capturing component to the server;

The server is configured to, according to the three-dimensional position information in the image of at least one feature point in the image captured by each three-dimensional image capturing component when the movable device is in the initial position, determine that the physical point corresponding to each feature point is relative to the actual point. relative position information of the movable device; during the movement of the movable device, in the images currently captured by each 3D image capturing component, determine the 3D position information of the feature points included in each image; according to the determined The relative position information corresponding to the physical point, the three-dimensional position information of the currently determined feature points, and the pre-stored relative pose information between different three-dimensional image capturing components, to determine the pose change of the movable device relative to the initial position information.

Regarding the system in the above embodiment, the specific manner in which the movable device and the server perform operations have been described in detail in the embodiment of the method, and will not be described in detail here.

FIG. 6 shows a schematic structural diagram of a server 1900 provided by an exemplary embodiment of the present disclosure. The server 1900 may vary greatly due to different configurations or performances, and may include one or more processors (central processing units, CPU) 1910 and one or more memories 1920 . Wherein, the memory 1920 stores at least one instruction, and the at least one instruction is loaded and executed by the processor 1910 to implement the method for determining the pose change information of the movable device described in the above embodiments.

Other embodiments of the present disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or techniques in the technical field not disclosed by the present disclosure . The specification and examples are to be regarded as exemplary only, with the true scope and spirit of the disclosure being indicated by the claims.

It is to be understood that the present disclosure is not limited to the precise structures described above and illustrated in the accompanying drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims (13)

1. A method of determining pose change information of a movable device, the movable device including at least two three-dimensional image capturing components, the method comprising:

determining the relative position information of a real object point corresponding to each feature point relative to the movable equipment according to the three-dimensional position information of at least one feature point in the image shot by each three-dimensional image shooting component when the movable equipment is at the initial position;

determining three-dimensional position information of a feature point contained in each image in the image currently captured by each three-dimensional image capturing element during the movement of the movable device;

and determining the pose change information of the movable equipment relative to the initial position according to the relative position information corresponding to the determined real object point, the three-dimensional position information of the currently determined characteristic point and the pre-stored relative pose information among different three-dimensional image shooting components.

2. The method according to claim 1, wherein the determining the relative position information of the object point corresponding to each feature point with respect to the movable device according to the three-dimensional position information of at least one feature point in the image captured by each three-dimensional image capturing means when the movable device is at the initial position in the image comprises:

determining the relative position information of a real object point corresponding to each characteristic point relative to the corresponding three-dimensional image shooting component according to the three-dimensional position information of at least one characteristic point in the image shot by each three-dimensional image shooting component when the movable equipment is at the initial position;

and determining the relative position information of the real object point corresponding to each characteristic point relative to the reference three-dimensional image shooting component according to the relative position information of the real object point corresponding to each characteristic point relative to the corresponding three-dimensional image shooting component and the pre-stored relative pose information between the reference three-dimensional image shooting component and other three-dimensional image shooting components in the at least two three-dimensional image shooting components, wherein the relative position information of the real object point corresponding to each characteristic point relative to the reference three-dimensional image shooting component is used as the relative position information of the real object point corresponding to each characteristic point relative to the movable equipment.

3. The method according to claim 2, wherein the determining pose change information of the movable device relative to the initial position according to the relative position information corresponding to the determined real object point, the three-dimensional position information of the currently determined feature point and the pre-stored relative pose information between different three-dimensional image capturing means comprises:

determining three-dimensional position information of each feature point relative to the image currently shot by the reference three-dimensional image shooting component according to the currently determined three-dimensional position information of the feature point and pre-stored relative pose information between the reference three-dimensional image shooting component and other three-dimensional image shooting components;

and determining the pose change information of the movable equipment relative to the initial position according to the relative position information of the object point corresponding to each characteristic point relative to the reference three-dimensional image shooting component and the three-dimensional position information of each characteristic point relative to the image currently shot by the reference three-dimensional image shooting component.

4. The method of claim 1, further comprising:

determining target feature points matched with the image features of any feature point in the at least one feature point from the feature points contained in the image shot by each three-dimensional image shooting component at present, and determining three-dimensional position information of each target feature point;

the determining pose change information of the movable equipment relative to the initial position according to the relative position information corresponding to the determined real object point, the three-dimensional position information of the currently determined characteristic point and the pre-stored relative pose information among different three-dimensional image shooting components comprises the following steps:

and determining the pose change information of the movable equipment relative to the initial position according to the determined relative position information corresponding to the real object point, the three-dimensional position information of each target characteristic point and the pre-stored relative pose information among different three-dimensional image shooting components.

5. The method according to claim 4, wherein the determining pose change information of the movable device relative to the initial position according to the relative position information corresponding to the determined real object points, the three-dimensional position information of each target feature point and the pre-stored relative pose information between different three-dimensional image capturing means comprises:

and if the number of the target characteristic points is greater than or equal to a preset number threshold, determining pose change information of the movable equipment relative to the initial position according to the determined relative position information corresponding to the real object point, the three-dimensional position information of each target characteristic point and pre-stored relative pose information among different three-dimensional image shooting components.

6. The method of claim 5, further comprising:

and if the number of the target characteristic points is smaller than a preset number threshold, setting the current position as an initial position, and determining the relative position information of the real object point corresponding to each characteristic point relative to the movable equipment according to the three-dimensional position information of at least one characteristic point in the image shot by each three-dimensional image shooting component in the image when the movable equipment is at the reset initial position.

7. An apparatus that determines pose change information of a movable device, the movable device including at least two three-dimensional image capturing means, the apparatus comprising:

the determining module is used for determining the relative position information of a real object point corresponding to each feature point relative to the movable equipment according to the three-dimensional position information of at least one feature point in the image shot by each three-dimensional image shooting component when the movable equipment is at the initial position;

the determining module is further used for determining three-dimensional position information of a feature point contained in each image in the image currently shot by each three-dimensional image shooting component in the moving process of the movable equipment;

the determining module is further configured to determine pose change information of the movable device relative to the initial position according to the relative position information corresponding to the determined real object point, the three-dimensional position information of the currently determined feature point, and pre-stored relative pose information between different three-dimensional image capturing components.

8. The apparatus of claim 7, wherein the determining module is configured to:

determining the relative position information of a real object point corresponding to each characteristic point relative to the corresponding three-dimensional image shooting component according to the three-dimensional position information of at least one characteristic point in the image shot by each three-dimensional image shooting component when the movable equipment is at the initial position;

and determining the relative position information of the real object point corresponding to each characteristic point relative to the reference three-dimensional image shooting component according to the relative position information of the real object point corresponding to each characteristic point relative to the corresponding three-dimensional image shooting component and the pre-stored relative pose information between the reference three-dimensional image shooting component and other three-dimensional image shooting components in the at least two three-dimensional image shooting components, wherein the relative position information of the real object point corresponding to each characteristic point relative to the reference three-dimensional image shooting component is used as the relative position information of the real object point corresponding to each characteristic point relative to the movable equipment.

9. The apparatus of claim 8, wherein the determining module is configured to:

determining three-dimensional position information of each feature point relative to the image currently shot by the reference three-dimensional image shooting component according to the currently determined three-dimensional position information of the feature point and pre-stored relative pose information between the reference three-dimensional image shooting component and other three-dimensional image shooting components;

and determining the pose change information of the movable equipment relative to the initial position according to the relative position information of the object point corresponding to each characteristic point relative to the reference three-dimensional image shooting component and the three-dimensional position information of each characteristic point relative to the image currently shot by the reference three-dimensional image shooting component.

10. The apparatus of claim 7, wherein the determining module is further configured to:

determining target feature points matched with the image features of any feature point in the at least one feature point from the feature points contained in the image shot by each three-dimensional image shooting component at present, and determining three-dimensional position information of each target feature point;

and determining the pose change information of the movable equipment relative to the initial position according to the determined relative position information corresponding to the real object point, the three-dimensional position information of each target characteristic point and the pre-stored relative pose information among different three-dimensional image shooting components.

11. The apparatus of claim 10, wherein the determining module is configured to:

and when the number of the target characteristic points is greater than or equal to a preset number threshold, determining pose change information of the movable equipment relative to the initial position according to the determined relative position information corresponding to the real object point, the three-dimensional position information of each target characteristic point and pre-stored relative pose information among different three-dimensional image shooting components.

12. The apparatus of claim 11, wherein the determining module is further configured to:

and when the number of the target characteristic points is smaller than a preset number threshold, setting the current position as an initial position, and determining the relative position information of the real object point corresponding to each characteristic point relative to the movable equipment according to the three-dimensional position information of at least one characteristic point in the image shot by each three-dimensional image shooting component in the image when the movable equipment is at the reset initial position.

13. A system that determines pose change information of a movable device, the system comprising a server and the movable device, the movable device comprising at least two three-dimensional image capture components, wherein:

the mobile device is used for shooting images through each three-dimensional image shooting component every time a preset period is reached, and sending the images shot by each three-dimensional image shooting component to the server;

the server is used for determining the relative position information of a real object point corresponding to each feature point relative to the movable equipment according to the three-dimensional position information of at least one feature point in the image shot by each three-dimensional image shooting component when the movable equipment is at the initial position; determining three-dimensional position information of a feature point contained in each image in the image currently captured by each three-dimensional image capturing element during the movement of the movable device; and determining the pose change information of the movable equipment relative to the initial position according to the relative position information corresponding to the determined real object point, the three-dimensional position information of the currently determined characteristic point and the pre-stored relative pose information among different three-dimensional image shooting components.

Images