CN113920174A - Point cloud registration method, apparatus, device, medium and autonomous vehicle - Google Patents

Abstract

The present disclosure provides a point cloud registration method, device, equipment, medium and automatic driving vehicle, and relates to the field of artificial intelligence, in particular to automatic driving technology, computer vision technology and point cloud registration technology. The method includes: performing a matching operation on the first point cloud input and the second point cloud input based on the estimated pose to obtain at least one key point pair, wherein each key point pair includes a first key point and a second key point performing a registration operation on at least one key point pair, including: for each key point pair: performing coordinate transformation on the first key point at least based on the estimated pose to obtain a third key point; and based on the second key point and the relationship between the second key point and the third key point, determine the estimated color parameters of the third key point; based on the real color parameters of the first key point and the estimated color parameters of the third key point , and determine the first target update information for the estimated pose to update the estimated pose.

Description

Point cloud registration method, apparatus, apparatus, medium and autonomous vehicle

technical field

The present disclosure relates to the field of artificial intelligence, in particular to automatic driving technology, computer vision technology and point cloud registration technology, and in particular to a point cloud registration method, a point cloud registration device, an electronic device, a computer-readable storage medium and a computer program products, autonomous vehicles.

Background technique

Artificial intelligence is the study of making computers to simulate certain thinking processes and intelligent behaviors of people (such as learning, reasoning, thinking, planning, etc.), both hardware-level technology and software-level technology. AI hardware technologies generally include technologies such as sensors, dedicated AI chips, cloud computing, distributed storage, and big data processing: AI software technologies mainly include computer vision technology, speech recognition technology, natural language processing technology, and machine learning/depth Learning, big data processing technology, knowledge graph technology and other major directions.

Point cloud registration technology is used to transform a set of point cloud data in order to match with another set of point cloud data. In many scenarios including autonomous driving tasks, point cloud registration technology is of great significance. By achieving high-precision point cloud registration, the precise position of the current vehicle can be determined, thus providing guarantee for autonomous driving tasks.

The approaches described in this section are not necessarily approaches that have been previously conceived or employed. Unless otherwise indicated, it should not be assumed that any of the approaches described in this section qualify as prior art merely by virtue of their inclusion in this section. Similarly, unless otherwise indicated, the issues raised in this section should not be considered to be recognized in any prior art.

SUMMARY OF THE INVENTION

The present disclosure provides a point cloud registration method, a point cloud registration device, an electronic device, a computer-readable storage medium, a computer program product, and an autonomous driving vehicle.

According to an aspect of the present disclosure, a point cloud registration method is provided. The method includes: acquiring a first point cloud, a second point cloud, and an estimated pose of a reference frame of the second point cloud to the reference frame of the first point cloud; determining a first point cloud input based on the first point cloud; Two point clouds determine the second point cloud input; based on the estimated pose, perform a matching operation on the first point cloud input and the second point cloud input to obtain at least one key point pair, wherein each of the at least one key point pair A keypoint pair includes a first keypoint belonging to the first point cloud input and a second keypoint belonging to the second point cloud input; and performing a registration operation on at least one keypoint pair, wherein the registration operation includes: for at least one Each key point pair in a key point pair: coordinate transformation is performed on the first key point based on at least the estimated pose to obtain a third key point under the reference frame of the second point cloud; and based on the second key point The real color parameter, and the relationship between the second key point and the third key point, determine the estimated color parameter of the third key point; based on the real color parameter of the first key point in at least one key point pair and the The estimated color parameter of the third key point corresponding to a key point determines the first target update information for the estimated pose; and at least based on the first target update information, the estimated pose is updated.

According to another aspect of the present disclosure, a point cloud registration apparatus is provided. Wherein, the device includes: an acquisition unit, configured to acquire the first point cloud, the second point cloud, and the estimated pose of the reference frame of the second point cloud with respect to the reference frame of the first point cloud; the determination unit, configured to determining a first point cloud input based on the first point cloud and determining a second point cloud input based on the second point cloud; a matching unit configured to perform matching on the first point cloud input and the second point cloud input based on the estimated pose operating to obtain at least one keypoint pair, wherein each keypoint pair in the at least one keypoint pair includes a first keypoint belonging to a first point cloud input and a second keypoint belonging to a second point cloud input; and a registration unit configured to perform a registration operation on at least one keypoint pair, wherein the registration unit includes: a color estimation subunit, configured to: for each keypoint pair in the at least one keypoint pair: at least based on The estimated pose performs coordinate transformation on the first key point to obtain the third key point under the reference frame of the second point cloud; and the real color parameters based on the second key point, as well as the second key point and the third key point The relationship between, determine the estimated color parameter of the third key point; determine the subunit, be configured to be based on the real color parameter of the first key point in at least one key point pair and the third corresponding to the first key point. The estimated color parameter of the key point determines the first target update information for the estimated pose; and the update subunit is configured to update the estimated pose based on at least the first target update information.

According to another aspect of the present disclosure, there is provided an electronic device comprising: at least one processor; and a memory communicatively coupled to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor. The at least one processor executes to enable the at least one processor to perform the above method.

According to another aspect of the present disclosure, there is provided a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to perform the above-described method.

According to another aspect of the present disclosure, there is provided a computer program product, comprising a computer program, wherein the computer program, when executed by a processor, implements the above-described method.

According to another aspect of the present disclosure, there is provided an autonomous vehicle comprising: at least one processor; and a memory communicatively coupled to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor. The at least one processor executes to enable the at least one processor to perform the above method.

According to one or more embodiments of the present disclosure, by using an objective function including a point cloud reflection value, a constraint on the point cloud reflection value (ie, the color of the point) is introduced, so that the estimation result of the pose can be improved. Specifically, coordinate transformation is performed on the first key point based on the estimated pose to obtain a third key point located in the same reference frame as the second key point. By using the real color parameters of the corresponding second key point and the positional relationship between the second key point and the third key point, an accurate estimation of the color at the location of the third key point can be obtained. The estimated color parameters of the first key point and the real color parameters of the first key point update the estimated pose to obtain accurate pose estimation results.

It should be understood that what is described in this section is not intended to identify key or critical features of embodiments of the disclosure, nor is it intended to limit the scope of the disclosure. Other features of the present disclosure will become readily understood from the following description.

Description of drawings

The accompanying drawings illustrate the embodiments by way of example and constitute a part of the specification, and together with the written description of the specification serve to explain exemplary implementations of the embodiments. The shown embodiments are for illustrative purposes only and do not limit the scope of the claims. Throughout the drawings, the same reference numbers refer to similar but not necessarily identical elements.

1 shows a schematic diagram of an exemplary system in which various methods described herein may be implemented, according to embodiments of the present disclosure;

FIG. 2 shows a flowchart of a point cloud registration method according to an exemplary embodiment of the present disclosure;

FIG. 3 shows a flowchart of a registration operation according to an exemplary embodiment of the present disclosure;

FIG. 4 shows a flowchart of a point cloud registration method according to an exemplary embodiment of the present disclosure;

FIG. 5 shows a structural block diagram of a point cloud registration apparatus according to an exemplary embodiment of the present disclosure; and

6 shows a block diagram of an exemplary electronic device that can be used to implement embodiments of the present disclosure.

Detailed ways

Exemplary embodiments of the present disclosure are described below with reference to the accompanying drawings, which include various details of the embodiments of the present disclosure to facilitate understanding and should be considered as exemplary only. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope of the present disclosure. Also, descriptions of well-known functions and constructions are omitted from the following description for clarity and conciseness.

In the present disclosure, unless otherwise specified, the use of the terms "first", "second", etc. to describe various elements is not intended to limit the positional relationship, timing relationship or importance relationship of these elements, and such terms are only used for Distinguish one element from another. In some examples, the first element and the second element may refer to the same instance of the element, while in some cases they may refer to different instances based on the context of the description.

The terminology used in the description of the various described examples in this disclosure is for the purpose of describing particular examples only and is not intended to be limiting. Unless the context clearly dictates otherwise, if the number of an element is not expressly limited, the element may be one or more. Furthermore, as used in this disclosure, the term "and/or" covers any and all possible combinations of the listed items.

In the related art, the existing point cloud registration method has no constraints on the reflection value of the point cloud, so an accurate estimation of the estimated pose cannot be obtained when other constraints are ineffective.

In order to solve the above problems, the present disclosure introduces a constraint on the point cloud reflection value (ie, the color of the point) by using an objective function including the point cloud reflection value, so that the estimation result of the pose can be improved. Specifically, coordinate transformation is performed on the first key point based on the estimated pose to obtain a third key point located in the same reference frame as the second key point. By using the real color parameters of the corresponding second key point and the positional relationship between the second key point and the third key point, an accurate estimation of the color at the location of the third key point can be obtained. The estimated color parameters of the first key point and the real color parameters of the first key point update the estimated pose to obtain accurate pose estimation results.

1 shows a schematic diagram of an exemplary system 100 in which the various methods and apparatuses described herein may be implemented in accordance with embodiments of the present disclosure. Referring to FIG. 1 , the system 100 includes a motor vehicle 110 , a server 120 , and one or more communication networks 130 that couple the motor vehicle 110 to the server 120 .

In an embodiment of the present disclosure, the motor vehicle 110 may include and/or be configured to perform a method according to an embodiment of the present disclosure.

Server 120 may run one or more services or software applications that enable the method of point cloud registration. In some embodiments, server 120 may also provide other services or software applications that may include non-virtual and virtual environments. In the configuration shown in FIG. 1 , server 120 may include one or more components that implement the functions performed by server 120 . These components may include software components executable by one or more processors, hardware components, or a combination thereof. A user of motor vehicle 110 may in turn utilize one or more client applications to interact with server 120 to utilize the services provided by these components. It should be understood that a variety of different system configurations are possible, which may differ from system 100 . Accordingly, FIG. 1 is one example of a system for implementing the various methods described herein, and is not intended to be limiting.

Server 120 may include one or more general purpose computers, special purpose server computers (eg, PC (personal computer) servers, UNIX servers, midrange servers), blade servers, mainframe computers, server clusters, or any other suitable arrangement and/or combination . Server 120 may include one or more virtual machines running virtual operating systems, or other computing architectures that involve virtualization (eg, may be virtualized to maintain one or more flexible pools of logical storage devices of the server's virtual storage devices). In various embodiments, server 120 may run one or more services or software applications that provide the functionality described below.

The computing units in server 120 may run one or more operating systems including any of the operating systems described above, as well as any commercially available server operating systems. Server 120 may also run any of a variety of additional server applications and/or middle-tier applications, including HTTP servers, FTP servers, CGI servers, JAVA servers, database servers, and the like.

In some implementations, server 120 may include one or more applications to analyze and incorporate data feeds and/or event updates received from motor vehicle 110 . Server 120 may also include one or more applications to display data feeds and/or real-time events via one or more display devices of motor vehicle 110 .

Network 130 may be any type of network known to those skilled in the art that may support data communications using any of a variety of available protocols, including but not limited to TCP/IP, SNA, IPX, and the like. By way of example only, the one or more networks 110 may be a satellite communications network, a local area network (LAN), an Ethernet-based network, a token ring, a wide area network (WAN), the Internet, a virtual network, a virtual private network (VPN), an intranet , extranet, public switched telephone network (PSTN), infrared network, wireless network (including eg Bluetooth, WiFi) and/or any combination of these and other networks.

System 100 may also include one or more databases 150 . In some embodiments, these databases may be used to store data and other information. For example, one or more of the databases 150 may be used to store information such as audio files and video files. Data repository 150 may reside in various locations. For example, the data repository used by server 120 may be local to server 120, or may be remote from server 120 and may communicate with server 120 via a network-based or dedicated connection. Data repository 150 may be of different types. In some embodiments, the data repository used by server 120 may be a database, such as a relational database. One or more of these databases may store, update, and retrieve data to and from the databases in response to commands.

In some embodiments, one or more of the databases 150 may also be used by applications to store application data. Databases used by applications can be different types of databases such as key-value stores, object stores, or regular stores backed by a file system.

Motor vehicle 110 may include sensors 111 for sensing the surrounding environment. Sensors 111 may include one or more of the following sensors: vision cameras, infrared cameras, ultrasonic sensors, millimeter-wave radar, and laser radar (LiDAR). Different sensors can provide different detection accuracy and range. Cameras can be installed in front of, behind, or elsewhere in the vehicle. Vision cameras can capture what's inside and outside the vehicle in real time and present it to the driver and/or passengers. In addition, by analyzing the footage captured by the visual camera, information such as traffic light indications, intersection conditions, and other vehicle operating states can be obtained. Infrared cameras can capture objects with night vision. Ultrasonic sensors can be installed around the vehicle to measure the distance of objects outside the vehicle from the vehicle by using the characteristics of the strong directionality of ultrasonic waves. Millimeter-wave radar can be installed in the front, rear or other positions of the vehicle to measure the distance of objects outside the vehicle from the vehicle using the characteristics of electromagnetic waves. Lidar can be installed in the front, rear or other positions of the vehicle to detect object edges and shape information for object recognition and tracking. Due to the Doppler effect, the radar unit can also measure changes in the speed of vehicles and moving objects.

The motor vehicle 110 may also include a communication device 112 . Communication device 112 may include a satellite positioning module capable of receiving satellite positioning signals (eg, BeiDou, GPS, GLONASS, and GALILEO) from satellites 141 and generating coordinates based on these signals. The communication device 112 may also include a module for communicating with the mobile communication base station 142, and the mobile communication network may implement any suitable communication technology, such as GSM/GPRS, CDMA, LTE and other current or developing wireless communication technologies (eg, 5G technology) . The communication device 112 may also have a vehicle-to-vehicle (Vehicle-to-Everything, V2X) module configured to implement, for example, vehicle-to-vehicle (V2V) communication with other vehicles 143 and with infrastructure The facility 144 performs vehicle-to-infrastructure (V2I) communication with the outside world. In addition, the communication device 112 may also have a module configured to communicate with a user terminal 145 (including but not limited to a smartphone, tablet, or wearable device such as a watch), eg, via wireless local area network or Bluetooth using the IEEE 802.11 standard. Using the communication device 112 , the motor vehicle 110 may also access the server 120 via the network 130 .

The motor vehicle 110 may also include a control device 113 . Control device 113 may include a processor in communication with various types of computer-readable storage devices or media, such as a central processing unit (CPU) or graphics processing unit (GPU), or other special purpose processors, or the like. The control device 113 may include an automated driving system for automatically controlling various actuators in the vehicle. The automated driving system is configured to control the motor vehicle 110 (not shown) powertrain, steering system, and braking system, etc., via a plurality of actuators in response to inputs from a plurality of sensors 111 or other input devices to control acceleration, respectively , steering and braking without or with limited human intervention. Part of the processing functions of the control device 113 can be realized by cloud computing. For example, some processing may be performed using an on-board processor, while other processing may be performed using computing resources in the cloud. The control device 113 may be configured to perform the method according to the present disclosure. Furthermore, the control device 113 may be implemented as one example of a computing device on the motor vehicle side (client side) according to the present disclosure.

The system 100 of FIG. 1 may be configured and operated in various ways to enable application of the various methods and apparatuses described in accordance with the present disclosure.

According to an aspect of the present disclosure, a point cloud registration method is provided. As shown in FIG. 2, the method includes: step S201, obtaining the first point cloud, the second point cloud and the estimated pose of the reference frame of the second point cloud to the reference frame of the first point cloud; step S202, based on the first point cloud One point cloud determines the first point cloud input and the second point cloud input is determined based on the second point cloud; Step S203, based on the estimated pose, perform a matching operation on the first point cloud input and the second point cloud input to obtain at least A keypoint pair, wherein each keypoint pair in the at least one keypoint pair includes a first keypoint belonging to the first point cloud input and a second keypoint belonging to the second point cloud input; and step S204, pairing at least A registration operation is performed on a key point pair, wherein the registration operation includes: step S205 , for each key point pair in the at least one key point pair, at least perform coordinate transformation on the first key point based on the estimated pose to obtain The third key point under the reference frame of the second point cloud, and the relationship between the real color parameter of the second key point and the second key point and the third key point, determine the estimated color parameter of the third key point; Step S206, determining the first target update information for the estimated pose based on the real color parameter of the first key point in the at least one key point pair and the estimated color parameter of the third key point corresponding to the first key point ; and step S207 , updating the estimated pose based on at least the first target update information.

Thus, by using the objective function including the point cloud reflection value, a constraint on the point cloud reflection value (ie, the color of the point) is introduced, so that the estimation result of the pose can be improved. Specifically, coordinate transformation is performed on the first key point based on the estimated pose to obtain a third key point located in the same reference frame as the second key point. By using the real color parameters of the corresponding second key point and the positional relationship between the second key point and the third key point, an accurate estimation of the color at the location of the third key point can be obtained. The estimated color parameters of the first key point and the real color parameters of the first key point update the estimated pose to obtain accurate pose estimation results.

According to some embodiments, the first point cloud may be generated based on data collected by on-board sensors, and the second point cloud may be generated based on high-precision map data. It can be understood that, the first point cloud and the second point cloud may also be point cloud data obtained in other ways, which are not limited here.

According to some embodiments, step S202 , determining the first point cloud input based on the first point cloud and determining the second point cloud input based on the second point cloud may include: down-sampling the first point cloud and the second point cloud, respectively, to Get the first point cloud input and the second point cloud input. The meaning and effect of the above downsampling will be described later.

According to some embodiments, the estimated pose may be, for example, an estimated value of the current position and pose of the vehicle. The estimated pose may be obtained by, for example, a satellite navigation system, or by calculation based on accelerometer data, or by other methods, which are not limited here.

According to some embodiments, the estimated pose T ^k may include a rotation component ω ^k and a translation component t ^k . Updates to the estimated pose can be performed iteratively, so k can be used to denote the current iteration round. In an exemplary embodiment, the rotational component ω ^k further includes three sub-components α ^k , β ^k , γ ^k , and the translational component t ^k further includes three sub-components ^ak , b ^k , ^ck . The estimated pose can be expressed as the following matrix:

According to some embodiments, the matching operation on the first point cloud input and the second point cloud input may include: performing coordinate transformation on the first point cloud input based on the estimated pose; and transforming the coordinate transformed first point cloud input with The second point cloud input is matched. In some embodiments, performing coordinate transformation on the input of the first point cloud based on the estimated pose may be, for example, performing coordinate transformation on all points in the first point cloud based on the estimated pose. Therefore, by first performing pose transformation on the first point cloud and then matching with the second point cloud, the matching accuracy and the point matching rate can be improved.

After the matching results of the first point cloud and the second point cloud are obtained, the at least one matched key point pair can be registered.

According to some embodiments, for each key point pair in the at least one key point pair, coordinate transformation is performed on the corresponding first key point based on at least the estimated pose, so as to obtain the third point cloud input reference frame in the second point cloud. key point.

According to some embodiments, the true color parameters include the true color value of the second keypoint and the color gradient at the second keypoint.

According to some embodiments, a virtual orthographic camera may be introduced for the second keypoint, which is configured to observe the second keypoint _p along the normal np, and the image plane of this virtual camera is the tangent plane at p, the th The reference surface at the two key points. A point on the reference plane can be represented as a continuous color function C _p (u), where u is a vector emanating from the second keypoint p along the tangent plane: u*n _p =0. The function C _p (u) can be approximated by its first-order approximation:

Among them, C(p) is the true color value of the second key point p, and d _p is the gradient of C _p (u), that is, the color gradient at the second key point. After the reference surface at the second key point is obtained, the corresponding first key point may be coordinately transformed based on the estimated pose and further projected onto the reference surface to obtain the third key point.

It can be understood that, the registration operation in step S204 may include, for example, steps S205 to S207.

According to some embodiments, as shown in FIG. 3 , the registration operation may further include: step S301 , based on the real color value of the second key point, the respective real color values of the plurality of fourth key points in the second point cloud input, and The positional relationship between the second key point and the plurality of fourth key points determines the color gradient at the second key point. Wherein, the plurality of fourth key points are all located within a first preset range centered on the second key point. The operations of step S302 , step S303 and step S306 in FIG. 3 are similar to the operations of steps S205 to S207 in FIG. 2 , which are not limited herein.

Therefore, by determining the color gradient at the second key point according to the true color value of the local neighbors of the second key point and the positional relationship between the second key point and these local neighbors, it is possible to calculate the accurate color gradient, and then Get an accurate estimate of the color of the third keypoint.

According to some embodiments, determining the color gradient at the second key point may include: for each fourth key point in the plurality of fourth key points, based on the actual color value of the fourth key point, the actual color value of the second key point The color value, the estimated color gradient, and the positional relationship between the fourth key point and the second key point, determine the target update sub-information corresponding to the fourth key point; sub-information, determining the second target update information for the estimated color gradient; and determining the color gradient at the second key point based on the second target update information. Therefore, the accuracy of the obtained color gradient can be further improved by using the method of determining the target update information, thereby further improving the accuracy of estimating the color of the third key point.

In one exemplary embodiment, the color gradient is estimated by applying least squares to C _p' | _p ' ∈ N _p , which are the local neighbors of p, ie, a plurality of fourth keypoints. Let f(s) be a function of the projection of a 3D point s to the reference plane at the second keypoint p:

f(s)=sn _p (sp) ^T n _p

The second target update information for calculating the color gradient _dp is:

Wherein, the second target update sub-information may be, for example, the item for each local neighbor p' in the above formula. The estimated color gradient is an initial value of the color gradient used when calculating the color gradient based on the second target update information. It can be understood that a person skilled in the art can choose a corresponding initial value, that is, estimate the color gradient, to solve the color gradient, which is not limited here.

n_p＝0。可以理解的是，第二关键点处的颜色梯度可以在配准操作期间计算，也可以在匹配操作期间计算，也可以在预处理阶段通过对第二点云所包括的所有点计算颜色梯度，以得到每一个关键点对中的第二关键点的颜色梯度，在此不做限定。According to some embodiments, the color gradient at the second keypoint may be coplanar with the reference plane, ie

n _p =0. It can be understood that the color gradient at the second key point can be calculated during the registration operation, can also be calculated during the matching operation, or can be calculated in the preprocessing stage by calculating the color gradient for all points included in the second point cloud, to obtain the color gradient of the second key point in each key point pair, which is not limited here.

According to some embodiments, determining the first target update information about the estimated pose may include: for each keypoint pair, based on the true color value of the first keypoint in the keypoint pair and the corresponding first keypoint The difference value of the estimated color value of the third key point, determine the first target update sub-information corresponding to this key point pair; The first target update information of the pose. Therefore, by determining the first target update sub-information based on the difference between the real color value of the first key point and the estimated color value of the third key point, the constraint on the reflection value of the point cloud is realized, so as to obtain a more accurate bit cloud. Pose estimation result.

In some embodiments, the first target update information may be, for example, the target function E _C (T) constrained by the reflection value of the point cloud, as shown in the following formula. For at least one keypoint pair K={(p,q)}, the first keypoint q is transformed s(·) based on the estimated pose T and then projected to the third keypoint on the reference surface of the second keypoint p point q':

s(q,T)=T·q

q′=f(s(q,T))

Among them, C(q) is the real color value of the first key point, and C _p (q') is the estimated color value of the third key point.

According to some embodiments, updating the estimated pose based on at least the first target update information may include: determining a first residual and a first Jacobian matrix based on the first target update information and the estimated pose; based on the first residual and the first Jacobian matrix to determine the pose update information; and update the estimated pose based on the pose update information. Therefore, by using the residuals and Jacobian matrix for solution optimization, rapid convergence of the estimated pose can be achieved.

According to some embodiments, the first residual r _C may be obtained, for example, based on the residual corresponding to the item of each keypoint pair (p, q) in the first target update information, as follows:

例如可以通过第一残差r_C对预估位姿T^k中的每一个分量进行求导而得到。可以使用高斯牛顿法基于残差和雅可比矩阵对下式进行求解优化：First Jacobian

For example, it can be obtained by derivation of each component in the estimated pose T ^k by the first residual r _C . The following can be optimized using the Gauss-Newton method based on the residuals and the Jacobian:

Among them, the pose update information ξ can be converted with the estimated pose T. The pose update information ξ includes all components of the one-dimensional ^vector estimated pose Tk. In an exemplary embodiment, ξ=(α', β', γ', a', b', c'). The pose update information ξ can also be mapped to SE(3). The update to the estimated pose T ^k can be expressed as:

On the basis of the point cloud reflection value constraints, other constraints can also be introduced to further improve the accuracy of the registration results.

According to some embodiments, as shown in FIG. 3 , the registration operation may further include: step S305 , at least based on at least one of the respective real color values and coordinate values of the first key point and the second key point in the at least one key point pair one and the estimated pose, and determine the third target update information for the estimated pose. Step S306 , updating the estimated pose based on at least the first target update information may include: updating the estimated pose based on the first target update information and the third target update information. Therefore, by introducing other constraints and updating information based on targets corresponding to different constraints, the accuracy of the registration result can be further improved.

According to some embodiments, as shown in FIG. 3 , the registration operation may further include: Step S304 , for each key point pair, determining a first key based on respective coordinate values of a plurality of fifth key points in the first point cloud input The first covariance matrix at the point, and the second covariance matrix at the second key point is determined based on the respective coordinate values of the multiple sixth key points in the second point cloud input, wherein the multiple fifth key points are are located in the second preset range centered on the first key point, and the plurality of sixth key points are located in the third preset range centered on the second key point. The plurality of fifth keypoints and the plurality of sixth keypoints may be local neighbors of the first keypoint and the second keypoint.

According to some embodiments, determining the third target update information for the estimated pose may include: based on the positional relationship between the second key point and the third key point corresponding to the first key point, the first covariance matrix, the first The second covariance matrix and the estimated pose are used to determine the third target update information. Thus, by using the respective covariance matrices of the first keypoint and the second keypoint, geometric constraints can be implemented to complement the color constraints to improve the effect of point cloud registration.

In some embodiments, the third target update information may be, for example, a geometrically constrained target function EG ( _T ), as shown in the following formula. Assuming that Cov _p and Cov _q are covariance matrices calculated according to the coordinates of the points near the second key point p and the first key point q, respectively, there are:

为第二关键点和第三关键点之间的位置关系。in,

is the positional relationship between the second key point and the third key point.

According to some embodiments, updating the estimated pose based on the first target update information and the third target update information includes: determining respective weights of the first target update information and the third target update information; Pose, determine the first residual and the first Jacobian matrix; determine the second residual and the second Jacobian matrix based on the third target update information and the estimated pose; update the information based on the first target and the third target update The respective weights of the information splices the first residual and the second residual to obtain the fusion residual; based on the respective weights of the first target update information and the third target update information, the first Jacobian matrix and the second Jacobian matrix are combined. Splicing is performed to obtain a fusion Jacobian matrix; based on at least the fusion residuals and the fusion Jacobian matrix, it is determined that the information for pose update is used; and the estimated pose is updated based on the pose update information. In this way, it is realized that multiple target update information corresponding to different constraints are jointly and optimally solved to obtain the pose estimation results under these constraints.

In some embodiments, using σ∈[0,1] and (1-σ) as weights for balancing color constraints and geometric constraints, there are:

E(T)=(1-σ)E _C (T)+σ·E _G (T)

For example, the second residual r _G can be obtained based on the residual corresponding to the item of each key point pair (p, q) in the second target update information, as follows:

例如可以通过第二残差r_G对预估位姿T^k中的每一个分量进行求导而得到。可以基于第一目标更新信息和第三目标更新信息各自的权重将第一残差和第二残差进行拼接，以得到融合残差：Second Jacobian

For example, it can be obtained by derivation of each component in the estimated pose T ^k by the second residual r _G . The first residual and the second residual can be spliced based on the respective weights of the first target update information and the third target update information to obtain a fusion residual:

The first Jacobian matrix and the second Jacobian matrix can be concatenated based on the respective weights of the first target update information and the third target update information to obtain a fusion Jacobian matrix:

It can be understood that other constraints and corresponding target update information can also be used, or more constraints and corresponding target update information can be introduced to update the estimated pose, which is not limited herein.

According to some embodiments, as shown in FIG. 4 , the point cloud registration method may further include: step S405 , after updating the estimated pose, inputting the first point cloud and the second point based on the updated estimated pose Perform a matching operation on the cloud input to obtain the updated at least one key point pair; and step S406, in response to determining that the first preset condition is not met, perform a registration operation on the updated at least one key point pair. The operations of steps S401 to S404 in FIG. 4 are similar to the operations of steps S201 to S204 in FIG. 2 , and details are not described here. Therefore, by setting the preset conditions, iterative update of the estimated pose can be achieved, so as to further improve the accuracy of the registration result.

According to some embodiments, the first preset condition may include at least one of the following: a distance between the first keypoint and the second keypoint included in each keypoint pair in the updated at least one keypoint pair The difference between the average value of the average value and the average value of the distances between the first key point and the second key point included in each key point pair in the at least one key point pair before the update is less than the first preset threshold; after the update The first ratio of the number of at least one keypoint pair to the number of points included in the first point cloud input and the second ratio between the number of at least one keypoint pair before the update and the number of points included in the first point cloud input The difference of the ratios is less than the second preset threshold; and a third ratio of the number of the updated at least one keypoint pair to the number of points included in the second point cloud input and the number of the at least one keypoint pair before the update is the same as the The difference of the fourth ratio of the number of points included in the second point cloud input is smaller than the third preset threshold. Thus, the iteration can be ended when the distance of the keypoint pairs changes little or the ratio of the number of keypoint pairs to the number of points included in the point cloud changes little.

It can be understood that those skilled in the art can set the first preset threshold, the second preset threshold and the third preset threshold by themselves, and can also set the corresponding convergence conditions according to the requirements, such as the maximum number of iterations, etc. Do limit.

According to some embodiments, acquiring the first point cloud and the second point cloud may include down-sampling the source point cloud and the target point cloud, respectively, to obtain the first point cloud and the second point cloud. As shown in FIG. 4 , the point cloud registration method may further include: Step S407 , after updating the estimated pose, in response to determining that the first preset condition is met, based on the updated estimated pose, perform a pairing of the source point cloud and the The target point cloud performs the registration operation. Therefore, by using the estimated pose obtained based on the low-resolution first point cloud and the second point cloud as the initial value for high-resolution registration, it is possible to solve the problem when registering high-resolution point clouds. The problem of inaccurate initial value of , thereby improving the accuracy of point cloud registration results and shortening the time required for convergence.

According to some embodiments, the input point cloud may be downsampled using a grid of large to small voxels, a point cloud pyramid may be constructed, and registration may be performed starting from the lowest resolution level in the pyramid, and registering the lowest level The pose obtained after convergence is used as the next-lowest initial pose, and registration is performed one by one until the estimated pose at the highest resolution is obtained as the final pose. By using the above method, the layer-by-layer objective function is smoother, and the final estimated pose obtained is more accurate.

According to another aspect of the present disclosure, a point cloud registration apparatus is disclosed. As shown in FIG. 5 , the point cloud registration apparatus 500 includes: an acquisition unit 510 configured to acquire the first point cloud, the second point cloud, and the estimation of the reference frame of the second point cloud with respect to the reference frame of the first point cloud pose; the determining unit 520 is configured to determine the first point cloud input based on the first point cloud and determine the second point cloud input based on the second point cloud; the matching unit 530 is configured to determine the first point cloud input based on the estimated pose The point cloud input and the second point cloud input perform a matching operation to obtain at least one key point pair, wherein each key point pair in the at least one key point pair includes a first key point belonging to the first point cloud input and a first key point belonging to the first point cloud input. The second key point input by the two point clouds; and the registration unit 540 configured to perform a registration operation on at least one key point pair, wherein the registration unit includes: a color estimation subunit 541 configured to perform a registration operation for at least one key point pair For each key point pair in the key point pair: coordinate transformation is performed on the first key point based on at least the estimated pose to obtain the third key point under the reference frame of the second point cloud; The color parameters, and the relationship between the second key point and the third key point, determine the estimated color parameter of the third key point; the determination subunit 542 is configured to determine the estimated color parameter of the third key point based on the The real color parameter and the estimated color parameter of the third key point corresponding to the first key point, determine the first target update information for the estimated pose; and the update subunit 543 is configured to update based on at least the first target information, update the estimated pose.

It can be understood that the operations of the unit 510-unit 540 and the sub-unit 541-sub-unit 543 in the point cloud registration apparatus 500 are respectively similar to the operations of step S201-step S207 in FIG. 2, and will not be repeated here.

Thus, by using the objective function including the point cloud reflection value, a constraint on the point cloud reflection value (ie, the color of the point) is introduced, so that the estimation result of the pose can be improved. Specifically, coordinate transformation is performed on the first key point based on the estimated pose to obtain a third key point located in the same reference frame as the second key point. By using the real color parameters of the corresponding second key point and the positional relationship between the second key point and the third key point, an accurate estimation of the color at the location of the third key point can be obtained. The estimated color parameters of the first key point and the real color parameters of the first key point update the estimated pose to obtain accurate pose estimation results.

In the technical solutions of the present disclosure, the collection, storage, use, processing, transmission, provision, and disclosure of the user's personal information involved are all in compliance with relevant laws and regulations, and do not violate public order and good customs.

According to embodiments of the present disclosure, an electronic device, a readable storage medium, and a computer program product are also provided.

Referring to FIG. 6 , a structural block diagram of an electronic device 600 that can function as a server or client of the present disclosure will now be described, which is an example of a hardware device that can be applied to various aspects of the present disclosure. Electronic devices are intended to represent various forms of digital electronic computer devices, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic devices may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are by way of example only, and are not intended to limit implementations of the disclosure described and/or claimed herein.

As shown in FIG. 6 , the device 600 includes a computing unit 601 that can be executed according to a computer program stored in a read only memory (ROM) 602 or a computer program loaded from a storage unit 608 into a random access memory (RAM) 603 Various appropriate actions and handling. In the RAM 603, various programs and data necessary for the operation of the device 600 can also be stored. The computing unit 601 , the ROM 602 , and the RAM 603 are connected to each other through a bus 604 . An input/output (I/O) interface 605 is also connected to bus 604 .

Various components in the device 600 are connected to the I/O interface 605 , including: an input unit 606 , an output unit 607 , a storage unit 608 , and a communication unit 609 . The input unit 606 may be any type of device capable of inputting information to the device 600, the input unit 606 may receive input numerical or character information, and generate key signal input related to user settings and/or function control of the electronic device, and may Including but not limited to mouse, keyboard, touch screen, trackpad, trackball, joystick, microphone and/or remote control. Output unit 607 may be any type of device capable of presenting information, and may include, but is not limited to, a display, speakers, video/audio output terminals, vibrators, and/or printers. The storage unit 608 may include, but is not limited to, magnetic disks and optical disks. Communication unit 609 allows device 600 to exchange information/data with other devices over computer networks such as the Internet and/or various telecommunication networks, and may include, but is not limited to, modems, network cards, infrared communication devices, wireless communication transceivers, and/or chipsets , such as Bluetooth™ devices, 1302.11 devices, WiFi devices, WiMax devices, cellular communication devices and/or the like.

Computing unit 601 may be various general-purpose and/or special-purpose processing components with processing and computing capabilities. Some examples of computing units 601 include, but are not limited to, central processing units (CPUs), graphics processing units (GPUs), various specialized artificial intelligence (AI) computing chips, various computing units running machine learning network algorithms, digital signal processing processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 601 performs the various methods and processes described above, such as the point cloud registration method. For example, in some embodiments, the point cloud registration method may be implemented as a computer software program tangibly embodied on a machine-readable medium, such as storage unit 608 . In some embodiments, part or all of the computer program may be loaded and/or installed on device 600 via ROM 602 and/or communication unit 609 . When a computer program is loaded into RAM 603 and executed by computing unit 601, one or more steps of the point cloud registration method described above may be performed. Alternatively, in other embodiments, the computing unit 601 may be configured to perform the point cloud registration method by any other suitable means (eg, by means of firmware).

Various implementations of the systems and techniques described herein above may be implemented in digital electronic circuitry, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips system (SOC), load programmable logic device (CPLD), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include being implemented in one or more computer programs executable and/or interpretable on a programmable system including at least one programmable processor that The processor, which may be a special purpose or general-purpose programmable processor, may receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device an output device.

Program code for implementing the methods of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general purpose computer, special purpose computer or other programmable data processing apparatus, such that the program code, when executed by the processor or controller, performs the functions/functions specified in the flowcharts and/or block diagrams. Action is implemented. The program code may execute entirely on the machine, partly on the machine, partly on the machine and partly on a remote machine as a stand-alone software package or entirely on the remote machine or server.

In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in connection with the instruction execution system, apparatus or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media may include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media would include one or more wire-based electrical connections, portable computer disks, hard disks, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM or flash memory), fiber optics, compact disk read only memory (CD-ROM), optical storage, magnetic storage, or any suitable combination of the foregoing.

To provide interaction with a user, the systems and techniques described herein may be implemented on a computer having a display device (eg, a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user ); and a keyboard and pointing device (eg, a mouse or trackball) through which a user can provide input to the computer. Other kinds of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (eg, visual feedback, auditory feedback, or tactile feedback); and can be in any form (including acoustic input, voice input, or tactile input) to receive input from the user.

The systems and techniques described herein may be implemented on a computing system that includes back-end components (eg, as a data server), or a computing system that includes middleware components (eg, an application server), or a computing system that includes front-end components (eg, a user's computer having a graphical user interface or web browser through which a user may interact with implementations of the systems and techniques described herein), or including such backend components, middleware components, Or any combination of front-end components in a computing system. The components of the system may be interconnected by any form or medium of digital data communication (eg, a communication network). Examples of communication networks include: Local Area Networks (LANs), Wide Area Networks (WANs), and the Internet.

A computer system can include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server arises by computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, also known as a cloud computing server or a cloud host. It is a host product in the cloud computing service system to solve the traditional physical host and VPS service ("Virtual Private Server", or "VPS" for short). , there are the defects of difficult management and weak business expansion. The server can also be a server of a distributed system, or a server combined with a blockchain.

It should be understood that steps may be reordered, added or deleted using the various forms of flow shown above. For example, the steps described in the present disclosure can be performed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in the present disclosure can be achieved, which are not limited herein.

Although the embodiments or examples of the present disclosure have been described with reference to the accompanying drawings, it should be understood that the above-described methods, systems and devices are merely exemplary embodiments or examples, and the scope of the present invention is not limited by these embodiments or examples, but is limited only by the appended claims and their equivalents. Various elements of the embodiments or examples may be omitted or replaced by equivalents thereof. Furthermore, the steps may be performed in an order different from that described in this disclosure. Further, various elements of the embodiments or examples may be combined in various ways. Importantly, as technology evolves, many of the elements described herein may be replaced by equivalent elements that appear later in this disclosure.

Claims (21)

1. A point cloud registration method, comprising:

acquiring a first point cloud, a second point cloud and an estimated pose of a reference system of the second point cloud to the reference system of the first point cloud;

determining a first point cloud input based on the first point cloud and a second point cloud input based on the second point cloud;

performing a matching operation on the first point cloud input and the second point cloud input based on the estimated pose to obtain at least one key point pair, wherein each key point pair in the at least one key point pair comprises a first key point belonging to the first point cloud input and a second key point belonging to the second point cloud input; and

performing a registration operation on the at least one keypoint pair, wherein the registration operation comprises:

for each of the at least one keypoint pair:

performing coordinate transformation on the first key point at least based on the estimated pose to obtain a third key point under a reference system of the second point cloud; and

determining an estimated color parameter of a third key point based on a real color parameter of a second key point and a relation between the second key point and the third key point;

determining first target update information aiming at the estimated pose based on the real color parameter of a first key point in the at least one key point pair and the estimated color parameter of a third key point corresponding to the first key point; and

and updating the estimated pose at least based on the first target updating information.

2. The method of claim 1, wherein the true color parameters include a true color value of the second keypoint and a color gradient at the second keypoint.

3. The method of claim 2, wherein the registration operation further comprises:

determining a color gradient at the second keypoint based on the real color value of the second keypoint, the real color values of a plurality of fourth keypoints in the second point cloud input, and the position relationship between the second keypoint and the plurality of fourth keypoints, wherein the plurality of fourth keypoints are all located within a first preset range with the second keypoint as the center.

4. The method of claim 3, wherein determining the color gradient at the second keypoint comprises:

for each fourth key point in the plurality of fourth key points, determining target updating sub-information corresponding to the fourth key point based on the real color value of the fourth key point, the real color value and the estimated color gradient of the second key point and the position relationship between the fourth key point and the second key point;

determining second target updating information aiming at the estimated color gradient based on target updating sub-information corresponding to the fourth key points respectively; and

determining a color gradient at the second keypoint based on the second target update information.

5. The method of claim 2, wherein the registration operation further comprises:

determining a reference plane at the second keypoint,

wherein the color gradient at the second keypoint is coplanar with the reference plane, and

wherein performing coordinate transformation on the first keypoint based at least on the estimated pose comprises:

and carrying out coordinate transformation on the corresponding first key point based on the estimated pose and further projecting the first key point to the reference surface to obtain the third key point.

6. The method of claim 1, wherein the true color parameters include true color values of second keypoints, the estimated color parameters include estimated color values, and determining first target update information for the estimated poses comprises:

for each key point pair, determining first target update sub-information corresponding to the key point pair based on the difference value between the real color value of a first key point in the key point pair and the estimated color value of a third key point corresponding to the first key point; and

determining first target update information regarding the pre-estimated pose input based on the at least one keypoint pair each corresponding to a first target update sub-information.

7. The method of claim 1, wherein updating the estimated pose based at least on the first target update information comprises:

determining a first residual error and a first Jacobian matrix based on the first target updating information and the estimated pose;

determining pose update information based on the first residual and the first Jacobian matrix; and

and updating the estimated pose based on the pose updating information.

8. The method of claim 1, wherein the true color parameters comprise true color values of the second keypoint, the estimated color parameters comprise estimated color values, and the registration operations further comprise:

determining third target update information for the estimated pose based at least on the estimated pose input and at least one of true color values and coordinate values of each of the first keypoint and the second keypoint of the at least one keypoint pair,

wherein updating the estimated pose input based at least on the first target update information comprises:

and updating the estimated pose based on the first target update information and the third target update information.

9. The method of claim 8, wherein the registration operation further comprises:

for each key point pair:

determining a first covariance matrix at a first key point based on coordinate values of a plurality of fifth key points in the first point cloud input, wherein the fifth key points are all located in a second preset range with the first key point as a center; and

determining a second covariance matrix at the second keypoint based on respective coordinate values of a plurality of sixth keypoints in the second point cloud input, wherein the plurality of sixth keypoints are all located within a third preset range centered on the second keypoint,

wherein determining third target update information for the estimated pose comprises:

and determining the update information of the third target based on the position relationship between the second key point and a third key point corresponding to the first key point, the first covariance matrix, the second covariance matrix and the estimated pose.

10. The method of claim 8, wherein updating the estimated pose based on the first target update information and the third target update information comprises:

determining respective weights of the first target update information and the third target update information;

determining a first residual error and a first Jacobian matrix based on the first target updating information and the estimated pose;

determining a second residual error and a second Jacobian matrix based on the third target update information and the estimated pose;

splicing the first residual error and the second residual error based on respective weights of the first target update information and the third target update information to obtain a fused residual error;

splicing the first Jacobian matrix and the second Jacobian matrix based on respective weights of the first target updating information and the third target updating information to obtain a fused Jacobian matrix;

determining pose update information based at least on the fused residuals and the fused jacobian matrix; and

and updating the estimated pose based on the pose updating information.

11. The method of claim 1, further comprising:

after updating the estimated pose, performing matching operation on the first point cloud input and the second point cloud input based on the updated estimated pose to obtain at least one updated key point pair; and

in response to determining that a first preset condition is not satisfied, performing the registration operation on the updated at least one keypoint pair.

12. The method of claim 11, wherein the first preset condition comprises at least one of:

the difference value between the average value of the distance between the first key point and the second key point included by each key point pair in the at least one updated key point pair and the average value of the distance between the first key point and the second key point included by each key point pair in the at least one key point pair before updating is smaller than a first preset threshold value;

a difference value between a first ratio of the updated number of the at least one key point pair to the number of the points included in the first point cloud input and a second ratio of the updated number of the at least one key point pair to the number of the points included in the first point cloud input is smaller than a second preset threshold value; and

a difference between a third ratio of the updated number of the at least one key point pair to the number of the points included in the second point cloud input and a fourth ratio of the updated number of the at least one key point pair to the number of the points included in the second point cloud input is smaller than a third preset threshold.

13. The method of claim 11, wherein determining a first point cloud input based on the first point cloud and a second point cloud input based on the second point cloud comprises:

down-sampling the first point cloud and the second point cloud, respectively, to obtain the first point cloud input and the second point cloud input,

wherein the method further comprises:

after updating the estimated pose, in response to determining that the first preset condition is met, performing the registration operation on the first point cloud and the second point cloud based on the updated estimated pose.

14. The method of claim 1, wherein the matching operation comprises:

performing coordinate transformation on the first point cloud input based on the estimated pose input; and

and matching the first point cloud input and the second point cloud input after coordinate transformation.

15. The method of any of claims 1-14, wherein the estimated pose includes a rotation component and a translation component.

16. The method of any of claims 1-14, wherein the first point cloud is generated based on data acquired by an in-vehicle sensor and the second point cloud is generated based on high precision map data.

17. A point cloud registration apparatus, comprising:

the system comprises an acquisition unit, a processing unit and a display unit, wherein the acquisition unit is configured to acquire a first point cloud, a second point cloud and an estimated pose of a reference system of the second point cloud to the reference system of the first point cloud;

a determination unit configured to determine a first point cloud input based on the first point cloud and a second point cloud input based on the second point cloud;

a matching unit configured to perform a matching operation on the first point cloud input and the second point cloud input based on the estimated pose to obtain at least one keypoint pair, wherein each keypoint pair of the at least one keypoint pair comprises a first keypoint belonging to the first point cloud input and a second keypoint belonging to the second point cloud input; and

a registration unit configured to perform a registration operation on the at least one keypoint pair, wherein the registration unit comprises:

a color predictor subunit configured to, for each of the at least one keypoint pair: performing coordinate transformation on the first key point at least based on the estimated pose to obtain a third key point under a reference system of the second point cloud; determining an estimated color parameter of a third key point based on a real color parameter of a second key point and a relation between the second key point and the third key point;

a determining subunit, configured to determine first target update information for the estimated pose based on a true color parameter of a first keypoint of the at least one keypoint pair and an estimated color parameter of a third keypoint corresponding to the first keypoint; and

an update subunit configured to update the estimated pose based at least on the first target update information.

18. An electronic device, comprising:

at least one processor; and

a memory communicatively coupled to the at least one processor; wherein

The memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-16.

19. A non-transitory computer readable storage medium having stored thereon computer instructions for causing the computer to perform the method of any one of claims 1-16.

20. A computer program product comprising a computer program, wherein the computer program realizes the method of any one of claims 1-16 when executed by a processor.

21. An autonomous vehicle comprising:

at least one processor; and

a memory communicatively coupled to the at least one processor; wherein

The memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method performed by the vehicle of any one of claims 1-16.

Images