CN111415420B

CN111415420B - Spatial information determining method and device and electronic equipment

Info

Publication number: CN111415420B
Application number: CN202010221139.4A
Authority: CN
Inventors: 贺一家; 李鑫; 刘骁
Original assignee: Beijing Megvii Technology Co Ltd
Current assignee: Beijing Megvii Technology Co Ltd
Priority date: 2020-03-25
Filing date: 2020-03-25
Publication date: 2024-01-23
Anticipated expiration: 2040-03-25
Also published as: CN111415420A

Abstract

The invention provides a space information determining method, a space information determining device and electronic equipment, and relates to the technical field of image processing, wherein the method comprises the steps of determining space characteristics based on an image sequence acquired by a camera; the spatial features comprise three-dimensional feature points and three-dimensional feature lines; determining a triangular grid of the three-dimensional space through the three-dimensional characteristic points and the three-dimensional characteristic straight lines; the triangular mesh comprises a plurality of triangular faces; and determining the space information according to the triangular mesh and the three-dimensional characteristic straight line of the three-dimensional space. The method and the device can extract the space information more quickly and effectively.

Description

Spatial information determining method and device and electronic equipment

Technical Field

The present invention relates to the field of image processing technologies, and in particular, to a method and an apparatus for determining spatial information, and an electronic device.

Background

With the development of augmented reality (Augmented Reality, AR) technology, the method is widely applied to various fields such as medical treatment, military, entertainment, municipal construction, smart home and the like, and vision-based positioning and mapping technology (Visual Simultaneous Localization and Mapping, VSLAM) is widely applied to industries such as augmented reality and robots, and environment three-dimensional structure information calculated through the VSLAM technology has very important practical significance for augmented reality application and obstacle avoidance of the robots. However, VSLAM technology requires dense point cloud reconstruction of the environment, and is complex in calculation process and low in efficiency, while for areas with less textures (such as carton surfaces, desktops and the like), visual features are more rare, and space information is difficult to effectively determine.

Disclosure of Invention

In view of the above, the present invention aims to provide a method, an apparatus and an electronic device for determining spatial information, which can determine spatial information more quickly and effectively.

In order to achieve the above object, the technical scheme adopted by the embodiment of the invention is as follows:

in a first aspect, an embodiment of the present invention provides a method for determining spatial information, where the method includes: determining spatial features based on an image sequence acquired by a camera; the spatial features comprise three-dimensional feature points and three-dimensional feature lines; determining a triangular grid of the three-dimensional space through the three-dimensional characteristic points and the three-dimensional characteristic straight lines; the triangular mesh comprises a plurality of triangular faces; and determining the space information according to the triangular mesh and the three-dimensional characteristic straight line of the three-dimensional space.

Further, the step of determining the spatial feature based on the image sequence acquired by the camera includes: estimating a camera gesture track corresponding to the image sequence based on the image sequence acquired by the camera; the camera coordinate system of the initial moment corresponding to the camera gesture track is a world coordinate system aligned with the gravity direction; extracting two-dimensional feature points and two-dimensional feature lines based on the image sequence; and determining three-dimensional characteristic points and three-dimensional characteristic lines according to the gesture track of the camera, the two-dimensional characteristic points and the two-dimensional characteristic lines.

Further, the step of determining the triangular mesh in the three-dimensional space by the three-dimensional feature points and the three-dimensional feature straight lines includes: constructing a triangular grid of a two-dimensional plane based on the two-dimensional characteristic points and the two-dimensional characteristic straight lines; respectively calculating coordinate representation of the three-dimensional feature points corresponding to the two-dimensional feature points and the two-dimensional feature lines and vector representation of the three-dimensional feature lines; the triangular mesh of the three-dimensional space is determined by the coordinate representation of the three-dimensional feature points, the vector representation of the three-dimensional feature lines, and the triangular mesh of the two-dimensional plane.

Further, the step of determining the triangular mesh in the three-dimensional space by the three-dimensional feature points and the three-dimensional feature straight lines includes: according to the attribute feature range of the triangular mesh in the preset three-dimensional space, respectively screening out target three-dimensional feature points and target three-dimensional feature lines of the triangular mesh which cannot construct the three-dimensional space from the obtained three-dimensional feature points and three-dimensional feature lines; the attribute characteristic range comprises an inner angle degree range and a side length range of the triangular surface; a triangular mesh of the three-dimensional space is determined based on the three-dimensional feature points and the three-dimensional feature straight lines other than the target three-dimensional feature points and the target three-dimensional feature straight line.

Further, the step of determining the spatial information according to the triangular mesh and the three-dimensional characteristic line of the three-dimensional space includes: determining candidate planes in the three-dimensional space according to the spatial position relation among a plurality of triangular faces of the triangular mesh of the three-dimensional space and/or the spatial position relation between the triangular faces and the three-dimensional characteristic straight line; the candidate planes include one or more of a horizontal plane, a vertical plane, and an inclined plane; verifying the candidate planes; and determining the space information according to the candidate planes passing the verification.

Further, the candidate plane includes a horizontal plane, and the step of determining the candidate plane in the three-dimensional space according to the spatial position relationship between the triangular faces of the triangular mesh of the three-dimensional space and/or the spatial position relationship between the triangular faces and the three-dimensional feature straight line includes: when a first target three-dimensional characteristic straight line is parallel to a horizontal plane under a preset coordinate system in the three-dimensional space and/or a plurality of first target triangular faces are parallel to the horizontal plane in a triangular grid of the three-dimensional space, counting the heights of each first target triangular face and the first target three-dimensional characteristic straight line in the three-dimensional space and the first counted number of the corresponding first target triangular face and the first target three-dimensional characteristic straight line under each height; searching the target height, and taking a horizontal plane at the target height as a candidate plane in the three-dimensional space; wherein the first statistical number corresponding to the target height is higher than a first threshold.

Further, the step of verifying the candidate plane includes: for each target height, judging whether the distance between the first target triangular surface corresponding to the target height and the first target three-dimensional characteristic straight line is smaller than a preset first distance threshold value; if so, the horizontal plane at the target height is determined to be a valid candidate plane.

Further, the candidate plane includes a vertical plane, and the step of determining the candidate plane in the three-dimensional space according to the spatial position relationship between the triangular faces of the triangular mesh of the three-dimensional space and/or the spatial position relationship between the triangular faces and the three-dimensional characteristic straight line includes: when a vertical plane normal vector of a second target three-dimensional characteristic straight line perpendicular to a preset coordinate system exists in the three-dimensional space, and/or a plurality of second target triangular planes perpendicular to a horizontal plane exist in a triangular grid of the three-dimensional space, calculating azimuth angles of each second target triangular plane and each second target three-dimensional characteristic straight line in the three-dimensional space and distances between each second target triangular plane and an origin of the preset coordinate system, and calculating second statistical numbers of the second target triangular planes and the second target three-dimensional characteristic straight lines corresponding to each azimuth angle and each distance; wherein the vertical plane normal vector is a normal vector set parallel to the horizontal plane; the azimuth angle comprises a first azimuth angle between a second target triangular surface and a horizontal plane under a preset coordinate system and a second azimuth angle of a second target three-dimensional characteristic straight line in a three-dimensional space; the distance comprises a first distance between a triangular surface of the second target and an origin of a preset coordinate system and a second distance projected by a connecting line between a three-dimensional characteristic straight line of the second target and the origin of the preset coordinate system under the preset coordinate system; searching a target azimuth angle and a target distance, and taking a vertical plane positioned at the target azimuth angle and the target distance as a candidate plane in a three-dimensional space; and the second statistical number corresponding to the target azimuth angle and the target distance is higher than a second threshold value.

Further, the step of verifying the candidate plane includes: for each target azimuth angle and target distance, judging whether the distance between a second target triangular surface corresponding to the target azimuth angle and the target distance and a second target three-dimensional characteristic straight line is smaller than a preset second distance threshold value; if so, the vertical plane at the target azimuth and target distance is determined to be a valid candidate plane.

Further, the candidate plane includes an inclined plane, and the step of determining the candidate plane in the three-dimensional space based on the spatial positional relationship between the triangular faces of the triangular mesh in the three-dimensional space and/or the spatial positional relationship between the triangular faces and the three-dimensional feature straight line includes: when a plurality of third target triangular faces are not parallel to or perpendicular to the horizontal plane in the triangular mesh of the three-dimensional space, calculating included angles and space distances between every two of the plurality of third target triangular faces; and determining two triangular surfaces with included angles smaller than a preset included angle threshold and space distances smaller than a preset distance threshold to be attributed to the same inclined plane, and taking the determined inclined plane as a candidate plane in the three-dimensional space.

Further, the step of verifying the candidate plane includes: for each inclined plane, when the third target three-dimensional characteristic straight line exists in the three-dimensional space and is neither parallel nor perpendicular to the horizontal plane under the preset coordinate system, and the distance between the third target three-dimensional characteristic straight line and the inclined plane is smaller than a third specified distance and/or the direction vectors of at least two third target three-dimensional characteristic straight lines exist in the third specified distance are parallel, determining that the third target three-dimensional characteristic straight line belongs to the inclined plane; judging whether a third statistical number of third target triangular surfaces and third target three-dimensional characteristic straight lines belonging to the inclined plane exceeds a third threshold value; if so, the inclined plane is determined to be a valid candidate plane.

Further, the spatial information includes plane information, and the method further includes: and performing a spatial plane extraction operation according to the determined plane information.

Further, the method further comprises the steps of: establishing a virtual three-dimensional scene according to the extracted space plane; wherein the virtual three-dimensional scene comprises a target object; when input information of a user is received, a target object is controlled according to the input information.

In a second aspect, an embodiment of the present invention further provides a spatial information determining apparatus, including: the spatial feature determining module is used for determining spatial features based on the image sequence acquired by the camera; the spatial features comprise three-dimensional feature points and three-dimensional feature lines; the generating module is used for generating a triangular grid of the three-dimensional space through the three-dimensional characteristic points and the three-dimensional characteristic straight lines; the triangular mesh of the three-dimensional space comprises a plurality of triangular faces of the three-dimensional space; and the space information determining module is used for determining space information according to the triangular grids and the three-dimensional characteristic straight lines of each three-dimensional space.

In a third aspect, an embodiment of the present invention provides an electronic device, including: a processor and a storage device; the storage means has stored thereon a computer program which, when executed by a processor, performs a method according to any of the preceding embodiments.

In a fourth aspect, embodiments of the present invention provide a computer-readable storage medium having a computer program stored thereon, which when executed by a processor performs the steps of the method of any of the previous embodiments.

The embodiment of the invention provides a space information determining method, a space information determining device and electronic equipment, wherein space characteristics (comprising three-dimensional characteristic points and three-dimensional characteristic straight lines) are determined based on an image sequence acquired by a camera, then triangular grids in a three-dimensional space are determined through the three-dimensional characteristic points and the three-dimensional characteristic straight lines, the triangular grids comprise a plurality of triangular faces, and finally the space information is determined according to the triangular grids in the three-dimensional space and the three-dimensional characteristic straight lines. According to the method, the three-dimensional characteristic straight line and the triangular grid of the three-dimensional space are utilized to extract the space, and the characteristic straight line and the sparse point cloud are combined to extract the space, so that on one hand, dense point cloud reconstruction with complex calculation process is not needed, the calculated amount is reduced, the extraction efficiency of the space information is improved, and on the other hand, the problems that the robustness of extracting the space information directly based on the sparse characteristic points is poor and the space information is difficult to extract effectively are solved by combining the characteristic straight line and the sparse point cloud.

Additional features and advantages of embodiments of the invention will be set forth in the description which follows, or in part will be obvious from the description, or may be learned by practice of the embodiments of the invention.

In order to make the above objects, features and advantages of the present invention more comprehensible, preferred embodiments accompanied with figures are described in detail below.

Drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings that are needed in the description of the embodiments or the prior art will be briefly described, and it is obvious that the drawings in the description below are some embodiments of the present invention, and other drawings can be obtained according to the drawings without inventive effort for a person skilled in the art.

Fig. 1 shows a schematic diagram of the result of an electronic device according to an embodiment of the present invention;

fig. 2 is a flowchart of a method for determining spatial information according to an embodiment of the present invention;

FIG. 3 is a schematic diagram of plane detection through sparse feature points according to an embodiment of the present invention;

FIG. 4 is a schematic diagram of plane detection by sparse feature point and feature line combination according to an embodiment of the present invention;

FIG. 5 is a flowchart of another method for determining spatial information according to an embodiment of the present invention;

FIG. 6 is a schematic diagram of a triangular mesh for determining three-dimensional space according to an embodiment of the present invention;

fig. 7 is a schematic structural diagram of a spatial information determining apparatus according to an embodiment of the present invention.

Detailed Description

For the purpose of making the objects, technical solutions and advantages of the embodiments of the present invention more apparent, the technical solutions of the present invention will be described below with reference to the accompanying drawings, and it is apparent that the described embodiments are some embodiments of the present invention, but not all embodiments.

At present, a plane is mostly extracted from a dense point cloud through a vision sensor, and the method has the problems of huge calculation amount (such as a DTAM (DenseTracking and Mapping in Real-Time) algorithm) or the need of calculating a distance (Tof camera, structured light and the like) by means of an additional infrared sensor, and has complex process and low plane extraction efficiency. In order to solve the problem, although a visual SLAM method such as ORB-SLAM and DSO (Direct SparseOdometry) algorithm without using an infrared sensor and reducing the calculation amount is further proposed, the problem of poor robustness still exists in accurately extracting a plane in a sparse point cloud, and the main reason is that the structural information of feature points in the sparse point cloud is not strong, and the accuracy is poor when only a few feature points are used for extracting the plane in the sparse point cloud. In addition, for areas with less textures (such as carton surfaces, desktops and the like), visual features are more rare, so that information such as planes in space cannot be effectively and accurately extracted by the method. In view of the problem that a plane cannot be extracted effectively and accurately in the related art, the method, the device and the electronic equipment for determining the space information provided by the embodiment of the invention can be applied to intelligent terminals such as computers, robots, unmanned aerial vehicles, AR equipment and the like. Embodiments of the present invention are described in detail below.

Embodiment one:

first, an example electronic device 100 for implementing a spatial information determination method, apparatus, and electronic device according to an embodiment of the present invention is described with reference to fig. 1.

As shown in fig. 1, an electronic device 100 includes one or more processors 102, one or more storage devices 104, an input device 106, an output device 108, and an image capture device 110, which are interconnected by a bus system 112 and/or other forms of connection mechanisms (not shown). It should be noted that the components and structures of the electronic device 100 shown in fig. 1 are exemplary only and not limiting, as electronic devices may have other components and structures as desired.

Processor 102 may be implemented in at least one hardware form of a Digital Signal Processor (DSP), a Field Programmable Gate Array (FPGA), a Programmable Logic Array (PLA), processor 102 may be one or a combination of several of a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), or other form of processing unit with data processing and/or instruction execution capabilities, and may control other components in electronic device 100 to perform desired functions.

The storage 104 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and/or non-volatile memory. Volatile memory can include, for example, random Access Memory (RAM) and/or cache memory (cache) and the like. The non-volatile memory may include, for example, read Only Memory (ROM), hard disk, flash memory, and the like. One or more computer program instructions may be stored on a computer readable storage medium and executed by the processor 102 to implement client functionality and/or other desired functionality in embodiments of the present invention described below. Various applications and various data, such as various data used and/or generated by the applications, may also be stored in the computer readable storage medium.

The input device 106 may be a device used by a user to input instructions and may include one or more of a keyboard, mouse, microphone, touch screen, and the like.

The output device 108 may output various information (e.g., images or sounds) to the outside (e.g., a user), and may include one or more of a display, a speaker, and the like.

The image capture device 110 may capture images (e.g., photographs, videos, etc.) desired by the user and store the captured images in the storage device 104 for use by other components. For example, an example electronic device for implementing the spatial information determination method, apparatus and electronic device according to embodiments of the present invention may be implemented as a smart terminal such as a mobile phone, a computer, a robot, a wearable device, an AR device, or the like.

Embodiment two:

referring to a flowchart of a spatial information determination method shown in fig. 2, the method mainly includes steps S202 to S206 as follows:

step S202, determining spatial features based on an image sequence acquired by a camera; the spatial features include three-dimensional feature points and three-dimensional feature lines.

In order to determine the three-dimensional space structure of the space in which the camera is located, the camera pose track needs to be estimated from the image sequence acquired by the camera, for example, the camera pose is estimated while the camera acquires the image, so as to determine the camera pose track corresponding to the image sequence, and the camera coordinate system at the initial moment corresponding to the camera pose track is a world coordinate system aligned with the gravity direction. Both the three-dimensional feature points and the three-dimensional feature lines in space may be represented in a world coordinate system. In the implementation, the three-dimensional feature points and the three-dimensional feature lines can be obtained through a projection model of the camera, and the three-dimensional feature points and the three-dimensional feature lines are determined through observation corresponding to the image plane.

Step S204, determining a triangular grid of the three-dimensional space through the three-dimensional characteristic points and the three-dimensional characteristic straight lines; the triangular mesh includes a plurality of triangular faces.

The triangular mesh is composed of three-dimensional feature points and three-dimensional feature lines, such as one three-dimensional feature point and one three-dimensional feature line may construct a triangle (i.e., a triangle plane) of a space, and a triangular mesh composed of a plurality of triangle planes may be determined by determining the three-dimensional feature points and the three-dimensional feature lines.

Since the calculation amount of generating the triangular meshes is large directly from the sparse three-dimensional point cloud, two-dimensional feature points and two-dimensional feature lines can be extracted from the image sequence, triangular meshes of a two-dimensional plane can be constructed based on the two-dimensional feature points and the two-dimensional feature lines, the triangular meshes of the two-dimensional plane can be generated by adopting a triangulation algorithm (Constrained Delauany Triangulation, CDT) with constraint, and then the coordinate representation of the corresponding three-dimensional feature points and the vector representation of the three-dimensional feature lines are calculated by utilizing a visual SLAM technology, so that the triangular meshes of the three-dimensional space can be determined by the coordinate representation of the three-dimensional feature points, the vector representation of the three-dimensional feature lines and the triangular meshes of the two-dimensional plane. The visual SLAM technique may be any visual SLAM system, and is not particularly limited.

In consideration of the fact that more singular triangular surfaces may exist in the triangular meshes of the determined three-dimensional space, such as very slender triangular surfaces, the three-dimensional meshes of the three-dimensional space may be determined based on the attribute feature ranges (inner angle degree ranges, side length ranges, etc.) of the triangular meshes of the three-dimensional space, by respectively screening out target three-dimensional feature points and target three-dimensional feature straight lines from all the obtained three-dimensional feature points and three-dimensional feature straight lines, the screened out target three-dimensional feature points and target three-dimensional feature straight lines are mostly three-dimensional feature points and three-dimensional feature straight lines constituting the singular triangular surfaces, and then based on the rest of the three-dimensional feature points and three-dimensional feature straight lines except the target three-dimensional feature points and the target three-dimensional feature straight lines. After the screening of the preset attribute feature range, three-dimensional feature points and three-dimensional feature lines can be divided into two types: one is a point and a straight line for constructing a triangle mesh, and the other is a point and a straight line not for constructing a triangle mesh.

In addition, the constructed triangular grids can be screened through preset reserved conditions so as to enable planes determined based on the triangular grids to be more real and effective. The reservation conditions may include restrictions in terms of the interior angle size, the side length ratio, etc. of triangular faces, such as that all three interior angles of a triangle must be greater than a certain threshold (such as 5 degrees), the ratio of the longest side and the shortest side of a triangle should be less than a certain threshold (such as 20), all three sides of a triangle need be less than a certain threshold (such as 2 m), etc., just as examples and are not limiting herein.

Step S206, determining space information according to the triangular mesh of the three-dimensional space and the three-dimensional characteristic straight line.

In one embodiment, the spatial information may include various information in the space, such as a spatial structure, plane information, object information located in the space, and the like, and when determining the spatial information through the triangular mesh and the three-dimensional feature line of the three-dimensional space constructed as described above, it is first necessary to determine a plane normal vector of each triangular plane in the triangular mesh of the three-dimensional space and a distance from an origin of the world coordinate system to each triangular plane, and since the three-dimensional feature line may generally reflect a plane in the space, the three-dimensional feature line not used for constructing the triangular mesh may be added to the spatial information determination, so that the determination of the spatial information is performed according to the triangular mesh and the three-dimensional feature line.

The spatial information may include plane information, so when the spatial information is determined through the triangular mesh of the three-dimensional space and the three-dimensional feature straight line, the embodiment may further determine the plane information, perform a spatial plane extraction operation based on the determined plane information, and further perform a subsequent operation based on the extracted spatial plane, such as constructing a virtual scene based on the extracted spatial plane, or directly perform interaction between a person and an environment in the augmented reality (Augmented Reality, AR) scene based on the extracted spatial plane, such as placing a virtual object on the extracted spatial plane, thereby fusing the virtual information with the real world, applying the virtual information to the real world, and by superposing a virtual object (such as a virtual object, a virtual character, etc.) with the real scene, an effect that the virtual object and the real scene image are in the same picture or the same space may be presented, so that the user experiences a virtual and reality combined scene.

Since the plane detection may result in an error when only sparse feature points are used for plane detection in the three-dimensional space, referring to a schematic diagram of plane detection by sparse feature points shown in fig. 3, if only three feature points a, b, and c are detected by the image in the three-dimensional space, the three points can construct a triangular mesh (i.e. triangular plane), such as a hatched portion shown in fig. 3. And the triangle mesh constructed at this time does not conform to the environmental characteristics. Therefore, the embodiment of the invention further introduces the characteristic straight line, performs plane detection by extracting the characteristic points and the characteristic straight line, referring to a schematic diagram of plane detection by combining sparse characteristic points and the characteristic straight line shown in fig. 4, and supposing that the detected characteristic points in the space are the same as those in fig. 3 at this time, three characteristic points are represented by a ', b ', c ', and 4 straight lines on the outer side surface of the cuboid are detected for convenience of distinction, the triangular mesh at this time may form a shadow part shown in fig. 4, and obviously, the geometric structure information of the space can be more preserved by the characteristic points of the straight line compared with the characteristic points, and the high probability of the detected straight line in the image can correspond to the plane in the real physical world. Therefore, the three-dimensional characteristic points and the three-characteristic straight lines are used for detecting the plane in the three-dimensional space, so that the accuracy and the robustness of the space plane detection can be improved.

According to the space information determining method provided by the embodiment of the invention, the space characteristics (comprising the three-dimensional characteristic points and the three-dimensional characteristic straight lines) are determined based on the image sequence acquired by the camera, then the triangular grids of the three-dimensional space are determined through the three-dimensional characteristic points and the three-dimensional characteristic straight lines, wherein the triangular grids comprise a plurality of triangular faces, and finally the space information is determined according to the triangular grids of the three-dimensional space and the three-dimensional characteristic straight lines. The method utilizes the extracted three-dimensional characteristic straight line and the triangular grid of the three-dimensional space to determine the space information, and essentially combines the characteristic straight line and the sparse point cloud to extract the space plane, so that on one hand, dense point cloud reconstruction with complex calculation process is not needed, the calculation amount is reduced, the extraction efficiency of the space information is improved, and on the other hand, the problems that the robustness of extracting the space information directly based on the sparse characteristic point is poor and the space information is difficult to extract effectively are solved by combining the characteristic straight line and the sparse point cloud.

Considering that the real space is mostly in an artificial environment, many planes are horizontal or vertical in the artificial environment, such as walls, desktops, etc., so that the planes can be divided into horizontal planes (parallel to the ground) or vertical planes, a camera coordinate system at an initial moment can be determined as a world coordinate system, and the world coordinate system performs an alignment operation with the ground level, which can be completed by measuring the gravity direction by using an accelerometer, or can be completed in advance by calibration or specific markers. Only after the camera coordinate system at the initial moment is aligned with the world coordinate system, the plane is conveniently divided into a horizontal plane (parallel to the ground) and a vertical plane, so that the space plane is conveniently and rapidly extracted.

In the extraction of the space plane, the candidate plane in the three-dimensional space may be first determined according to a spatial position relationship between a plurality of triangular faces of the triangular mesh of the three-dimensional space and/or a spatial position relationship between the triangular faces and the three-dimensional characteristic straight line, the spatial position relationship may include, for example, determination in parallel, perpendicular, or at an angle, the candidate plane may also include one or more of a horizontal plane, a vertical plane, and an inclined plane, and since the candidate plane is not necessarily valid (such as the obtained candidate plane does not necessarily exist in reality), the candidate plane may be first verified, then the extraction operation may be performed on the candidate plane that has passed the verification, and the extraction of the space plane may be performed by a method such as least square fitting.

The present embodiment provides a method for determining and extracting a candidate plane as a horizontal plane, a vertical plane, and an inclined plane, and for brevity, the following preset coordinate systems are all world coordinate systems after the alignment operation in the foregoing embodiment, and the method for determining that the candidate plane is a horizontal plane will be described first, with reference to the following steps 1 and 2:

step 1, counting the height of each first target triangular surface and each first target three-dimensional characteristic straight line in the three-dimensional space and the first counted number of the corresponding first target triangular surface and first target three-dimensional characteristic straight line in each height when the first target three-dimensional characteristic straight line is parallel to a horizontal plane under a preset coordinate system in the three-dimensional space and/or a plurality of first target triangular surfaces are parallel to the horizontal plane in a triangular grid of the three-dimensional space.

In one embodiment, the first target three-dimensional feature line is a three-dimensional feature line parallel to a horizontal plane in the world coordinate system, and the first target triangular surface is a triangular surface parallel to the horizontal plane in the world coordinate system. When there are a plurality of three-dimensional feature lines parallel to the horizontal plane of the world coordinate system in the three-dimensional space, or there are a plurality of triangular faces parallel to the horizontal plane of the world coordinate system and a plurality of three-dimensional feature lines in the three-dimensional space, the heights of the three-dimensional feature lines and the triangular faces in the three-dimensional space are counted, and the numbers of the triangular faces and the three-dimensional feature lines at each height are counted, for example, the statistics may be performed in a manner of a histogram, the x-axis of the histogram is the height of the triangular faces and the three-dimensional feature lines at the world coordinate, and the y-axis is the numbers of the triangular faces and the three-dimensional feature lines corresponding to the height (i.e., the first counted number).

And 2, searching the target height, and taking the horizontal plane at the target height as a candidate plane in the three-dimensional space. Wherein the first statistical number corresponding to the target height is higher than a first threshold. The first threshold is the effective reference number of candidate planes for determining the target height, and can be determined according to the number of triangular faces in the triangular mesh and the number of three-dimensional characteristic lines, or according to the space condition of the three-dimensional space, which is not particularly limited herein.

For each target height, judging whether the distance between the triangular surface of the first target corresponding to the target height and the three-dimensional characteristic straight line of the first target is smaller than a preset first distance threshold value, wherein the first distance threshold value is a distance threshold value between the triangular surface and the three-dimensional characteristic straight line at the same horizontal height, namely when the distances between the triangular surface and the three-dimensional characteristic straight line at the same horizontal height are relatively close, the horizontal plane at the target height can be determined to be an effective candidate plane.

In one embodiment, the step of determining the candidate plane as a vertical plane may refer to the following steps (1) and (2):

and (1) counting azimuth angles of each second target triangular surface and each second target three-dimensional characteristic straight line in the three-dimensional space and distances between the second target triangular surface and an origin of a preset coordinate system when a normal vector of the vertical surface of the second target three-dimensional characteristic straight line in the three-dimensional space is perpendicular to a preset coordinate system and/or a plurality of second target triangular surfaces in a triangular grid of the three-dimensional space are perpendicular to a horizontal plane, and counting second statistics numbers of the second target triangular surfaces and the second target three-dimensional characteristic straight lines corresponding to each azimuth angle and each distance.

Since each azimuth angle corresponds in space to a plane perpendicular to the horizontal plane, the vertical plane normal vector is a set of normal vectors parallel to the horizontal plane, including normal vectors of any azimuth angle parallel to the horizontal plane. The second target three-dimensional characteristic line is a three-dimensional characteristic line perpendicular to a horizontal plane in the world coordinate system, and the second target triangular surface is a triangular surface perpendicular to the horizontal plane in the world coordinate system. For whether the triangular surface is perpendicular to the horizontal plane in the world coordinate system, the determination can be carried out by calculating the plane normal vector of the triangular surface and the horizontal plane normal vector in the world coordinate system; for whether the three-dimensional characteristic straight line is perpendicular to a certain vertical plane normal vector in the world coordinate system, whether the vertical plane normal vector corresponding to any azimuth angle is perpendicular to the three-dimensional characteristic straight line direction vector can be judged by traversing the vertical plane normal vector, for example, when the azimuth angle in the space is divided by 1 degree, the vertical plane normal vector in the space can be divided into 180 pieces, and if the vertical plane normal vector corresponding to a certain azimuth angle is perpendicular to the direction vector of the three-dimensional characteristic straight line, the candidate plane corresponding to the azimuth angle is determined.

Since planes in the space may have different angles perpendicular to the horizontal plane of the world coordinate system, such as two adjacent sides of a rectangular parallelepiped, which are perpendicular to the horizontal plane, but are not perpendicular to the horizontal plane, it is necessary to count the first azimuth angle between each second target triangular surface and the horizontal plane in the world coordinate system and the second azimuth angle of the second target three-dimensional feature straight line in the three-dimensional space. In addition, a first distance between the triangular surface of the second target and the origin of the world coordinate system and a second distance projected by a connecting line between the three-dimensional characteristic straight line of the second target and the origin of the world coordinate system under a preset coordinate system are also required to be counted. Statistics can be carried out through a two-dimensional histogram, the x-axis of the histogram is the included angle between a vertical plane and a horizontal plane, the y-axis of the histogram is the distance, the z-axis is the number count, and if the normal vector of the triangular plane corresponds to a certain included angle x _i The distance from the triangle surface to the origin of the world coordinate system is y _i Then the coordinates (x _i ，y _i ) The count z value at this point is incremented by 1. If the direction vector of the three-dimensional characteristic straight line is perpendicular to the normal vector of the vertical plane corresponding to the included angle xi, and the projection distance of the connecting line between any point on the three-dimensional characteristic straight line and the origin of the world coordinate system on the normal vector is y _i Then the coordinates (x _i ，y _i ) The count z value at this point is also incremented by 1.

Step (2), searching a target azimuth angle and a target distance, and taking a vertical plane positioned at the target azimuth angle and the target distance as a candidate plane in a three-dimensional space; and the second statistical number corresponding to the target azimuth angle and the target distance is higher than a second threshold value. The second threshold is the effective reference number of candidate planes for determining the target azimuth and the target distance, and can be specifically limited according to actual needs.

In one embodiment, for each target azimuth and target distance, it is determined whether the distance between every two second target triangular surfaces and second target three-dimensional characteristic straight lines corresponding to the target azimuth and the target distance is smaller than a preset second distance threshold, where the first distance threshold is a distance threshold between triangular surfaces and three-dimensional characteristic straight lines at the same target azimuth and target distance, that is, when the triangular surfaces and three-dimensional characteristic straight lines at the same target azimuth and target distance are closer, it may be determined that a vertical plane at the target azimuth and target distance is a valid candidate plane.

In one embodiment, after extraction of the horizontal plane and the vertical plane, there are generally fewer triangle planes or three-dimensional feature lines left, so traversing the remaining triangle planes and three-dimensional features can be used to quickly determine and extract the inclined plane, and the step of determining the inclined plane can refer to the following steps a and B:

And step A, calculating included angles and space distances between every two of the plurality of third target triangular surfaces when the plurality of third target triangular surfaces are neither parallel nor perpendicular to the horizontal plane in the triangular grid of the three-dimensional space. The third target triangular surface is a triangular surface that is neither parallel nor perpendicular to a horizontal plane in the world coordinate system.

And B, determining two triangular surfaces with included angles smaller than a preset included angle threshold and space distances smaller than a preset distance threshold to be attributed to the same inclined plane, and taking the determined inclined plane as a candidate plane in the three-dimensional space. If the angle between two triangular faces is small and the spatial distance between them is small, it can be determined that they belong to the same inclined plane.

In order to improve the robustness of plane extraction, for each inclined plane, if there is a third target three-dimensional feature line in the three-dimensional space and the distance between the inclined planes determined in the above step B is smaller than a third specified distance, the third target feature line is also determined to be attributed to the inclined plane. The third target feature line is a three-dimensional feature line that is neither parallel nor perpendicular to a horizontal plane in the world coordinate system, and the third specified distance is a distance of the three-dimensional feature line from the inclined plane. In addition, when the direction vectors of at least two third target three-dimensional feature lines existing at the third specified distance are parallel, it may also be determined that the third target three-dimensional feature lines belong to the inclined plane. Judging whether the third statistical number of the third target triangular surfaces and the third target three-dimensional characteristic straight lines belonging to the inclined plane exceeds a third threshold value, wherein the third distance threshold value is the number threshold value of the triangular surfaces and the three-dimensional characteristic straight lines which are positioned on the same inclined plane, namely, when the numbers of the triangular surfaces and the three-dimensional characteristic straight lines which are positioned on the same inclined plane reach a preset reference, the inclined plane can be determined to be an effective candidate plane.

In addition, after the spatial plane is extracted, a virtual three-dimensional scene may be established according to the extracted spatial plane, wherein the virtual three-dimensional scene includes a target object, that is, the target object is a virtual object or a virtual character or the like in the virtual three-dimensional scene. Alternatively, the target object may be placed in a virtual three-dimensional scene by AR technology, so that the target object is subsequently controlled. Taking an AR device as an example, acquiring images at a plurality of moments through the AR device, estimating the posture of a camera according to the acquired image sequences at the plurality of moments, further estimating three-dimensional characteristic points and three-dimensional characteristic straight lines, finally determining a space plane, and constructing a virtual three-dimensional scene according to the space plane. When input information of a user is received, a target object in the virtual three-dimensional scene may be controlled according to the input information, which may be motion control information, chat information, or the like, such as when the input information is motion control information, a corresponding motion may be performed on the target object, such as controlling the target object to move in the virtual three-dimensional scene, or the like. When the input information is chat information, chat interaction between the user and the target object can be realized through the input information. It should be noted that the above is illustrative only and should not be taken as limiting.

In summary, according to the spatial information determining method provided by the embodiment, the spatial information is extracted by using the extracted three-dimensional characteristic straight line and the triangular mesh of the three-dimensional space, and the spatial information is extracted by combining the characteristic straight line and the sparse point cloud, so that on one hand, dense point cloud reconstruction with complex calculation process is not needed, the calculated amount is reduced, the spatial information extraction efficiency is improved, and on the other hand, the problems that the robustness of spatial information extraction based on sparse characteristic points is poor and the spatial information is difficult to extract effectively are solved by combining the characteristic straight line and the sparse point cloud.

Embodiment III:

on the basis of the foregoing embodiments, the present embodiment provides a specific example of a spatial information determination method, which is described taking plane information contained in extracted spatial information as an example, referring to a flowchart of another spatial information determination method as shown in fig. 5, and the method mainly includes the following steps S502 to S506:

step S502, determining a camera gesture track, three-dimensional feature points and three-dimensional feature lines according to an image sequence acquired by a camera by utilizing a visual SLAM technology.

In one embodiment, the camera acquired image sequence is an image sequence of images acquired at a plurality of successive moments in time by the camera, and the camera pose locus is determined from the camera acquired image sequence, such as specifying the kth moment camera C _k The gesture is denoted as x _k ＝(R _k ，t _k ) Wherein R epsilon SO (3) represents a rotation matrix in three-dimensional space,representing translation vectors in three-dimensional space. The camera pose trajectory may be determined from the camera poses at a plurality of moments. The observation of the three-dimensional characteristic point and the three-dimensional characteristic line corresponding to the image plane corresponds to coordinates +.>And->Wherein,coordinates of the ith three-dimensional feature point in space, < >>Is the first in spacej three-dimensional characteristic straight linesCoordinates representing a point on a three-dimensional feature line, +.>Representing the direction vector of the three-dimensional feature line in space).

In addition, the two-dimensional feature point z of the image observation _f And a two-dimensional characteristic straight line z _L Extraction may be performed using image processing algorithms such as ORB (Oriented Fast and Rotated Brief), LSD (Line Segment Detector), etc., without limitation.

Step S504, generating a triangular grid of the three-dimensional space based on the three-dimensional feature points and the three-dimensional feature straight lines. That is, triangular meshes are generated using sparse points and straight lines in three-dimensional space.

In one embodiment, the triangular mesh formed in the three-dimensional space is a patch formed by triangles, and is composed of three-dimensional feature points and/or three-dimensional feature lines, such as a spatial triangle may be constructed from one three-dimensional feature point and one three-dimensional feature line. Because the calculation amount for generating triangular surfaces from the sparse three-dimensional point cloud is large, the three-dimensional triangular grid can be generated by firstly determining a two-dimensional image plane I through an imaging plane of an image shot by a camera, referring to a schematic diagram of the triangular grid for determining a three-dimensional space shown in fig. 6, and extracting two-dimensional characteristic points z on the two-dimensional image plane I _f And a two-dimensional characteristic straight line z _L The two-dimensional feature points and the two-dimensional feature straight lines are then constructed as triangular meshes of the two-dimensional image plane (dotted triangles on the two-dimensional image plane I in the figure). The triangular mesh of the two-dimensional image plane may be constructed using, for example, a constrained triangulation algorithm (Constrained Delauany Triangulation, CDT). And finally, calculating corresponding three-dimensional feature points f and three-dimensional feature lines L based on a visual SLAM technology according to the two-dimensional feature points and the two-dimensional feature lines, and generating a three-dimensional triangular grid (a shadow part in the figure).

For triangle meshes formed in the three-dimensional space, more singular triangles exist, selective elimination can be performed, the triangle meshes are screened by setting a reservation conforming condition, the reservation condition can comprise limitations on the size of internal angles of triangular faces, the side length proportion and the like, for example, all three internal angles of the triangle are required to be larger than a certain threshold (such as 5 degrees), the ratio of the longest side and the shortest side of the triangle is required to be smaller than a certain threshold (such as 20), all three sides of the triangle are required to be smaller than a certain threshold (such as 2 m) and the like. After screening and removing, three-dimensional characteristic points and three-dimensional characteristic straight lines used for constructing the triangular meshes are left. In addition, the high probability of the detected straight line in the image can be corresponding to the plane in the real physical world, so that in order to improve the robustness of plane extraction, only three-dimensional feature points which are not used for constructing the triangular mesh can be removed, and the three-dimensional feature straight line is reserved.

And S506, extracting an effective plane through the three-dimensional triangular mesh and the three-dimensional characteristic straight line.

Considering that a straight line easily appears with a plane in an actual environment, the triangular mesh constructed in the above step S304 and the three-dimensional feature straight line not used for constructing the triangular mesh are used for spatial information determination, the plane normal vector thereof is calculated for each triangular face in the extracted triangular meshAnd the distance h from the origin of the world coordinate system to the plane of the triangular surface so as to extract the space plane. When the method is used for determining the space information, multiple iterations are not needed, and plane extraction can be completed only once, so that the extraction speed is improved to a greater extent.

If a plurality of triangular surfaces or three-dimensional characteristic lines are at the same level and the distances of the triangular surfaces or three-dimensional characteristic lines in three-dimensional space are smaller than a certain threshold value, they can construct a horizontal plane. When the horizontal plane extraction is carried out, a one-dimensional histogram is constructed by utilizing the distance h from all triangular surfaces to the origin of a coordinate system and the z-axis coordinates of any two three-dimensional points on a three-dimensional straight line, the x-axis of the histogram is high, the y-axis is the number, in order to restrain the situation that a plurality of maximum values exist locally, a one-dimensional Gaussian filter can be used for carrying out smoothing operation on the histogram, then the histogram is used as a candidate horizontal plane higher than a specified threshold value, for example, the statistical number at the position of the distance h=1m is higher than the specified threshold value, a horizontal plane with the height of 1m possibly exists is indicated, all triangular surfaces and three-dimensional characteristic straight lines belonging to the plane are counted, if the triangular surfaces and the three-dimensional characteristic straight lines are relatively close to each other, the horizontal plane is determined, and the three-dimensional characteristic points and the three-dimensional characteristic straight lines on the determined horizontal plane are subjected to the parameters of a least square fitting plane to complete the extraction of the horizontal plane.

If the triangular surfaces are perpendicular to the horizontal plane or the direction vectors of the three-dimensional characteristic straight lines are perpendicular to the normal vector of a certain vertical plane, and the distances between the triangular surfaces or the three-dimensional characteristic straight lines are smaller than a certain threshold value, a vertical plane can be constructed. When the vertical plane extraction is carried out, triangular planes and three-dimensional characteristic straight lines which are vertical to the horizontal direction can be firstly screened out for subsequent construction of the vertical plane, wherein the screening method is that the plane normal vector of the triangular planes is close to the horizontal plane normal vector or the direction vector of the three-dimensional characteristic straight lines is vertical to a certain vertical plane normal vector, and the vertical plane normal vector is a group of normal vector sets parallel to the horizontal planex _i =1, 2 …,180, wherein the angle x _i Is azimuth.

Counting triangular surfaces and three-dimensional characteristic straight lines which can be used for constructing a vertical plane after screening by using a two-dimensional histogram, wherein one dimension in the two-dimensional histogram is used for recording an included angle between the vertical plane and a horizontal plane, such as what angle is perpendicular to the horizontal plane, and the angle can be called azimuth; the other dimension is used to record the distance to the normal vector of the plane, such as with the center of the bottom of the cube as the origin, and their azimuth angles are the same in the horizontal plane for the front and back sides, but the distances are different, one distance being positive and the other negative. The x-axis of the two-dimensional histogram is the included angle between the vertical plane and the horizontal plane, the y-axis is the distance, and the z-axis is the number count.

Traversing all triangular patches, if the normal vector of the patch corresponds to a certain included angle x _i The distance from the plane to the origin of the world coordinate system is y _i Then the coordinates (x _i ，y _i ) The count z value at this point is incremented by 1. Traversing all three-dimensional characteristic straight lines in the same way, if the direction and the included angle x of the three-dimensional characteristic straight lines are _i The corresponding normal vector of the vertical plane is vertical, and the projection distance of the connecting line between any point on the straight line and the origin of the coordinate system on the normal vector is y _i Then the coordinates (x _i ，y _i ) The count z value at this point is also incremented by 1. And finally, in order to restrain the situation that a plurality of local maxima exist, smoothing the histogram by using two-dimensional Gaussian filtering, taking the statistical two-dimensional histogram z value higher than a certain threshold value as a vertical plane, and carrying out least square fitting on parameters of a three-dimensional characteristic point and a three-dimensional characteristic straight line on the determined vertical plane to finish the extraction of the vertical plane.

In addition, after the extraction of the horizontal plane and the vertical plane, the residual triangle surface or the three-dimensional characteristic straight line is usually less, and the plane can be rapidly determined by a traversing mode. Similarly to the above manner, the angles between normal vectors of the triangular patches are calculated first, and if the angles between any two triangular faces are smaller than a certain threshold and the spatial distance between the two triangular faces is smaller than a certain threshold, the two triangles can be classified as a certain candidate plane. Meanwhile, if the distance from the empty three-dimensional characteristic straight line to the candidate plane class is smaller than a certain threshold value, the straight line is classified as the candidate plane class. Alternatively, if two three-dimensional feature straight line direction vectors are parallel, a candidate plane class may be formed. When the number of three-dimensional characteristic straight lines or triangular faces belonging to the same candidate plane class is higher than a certain threshold, the candidate plane is determined to be effective, and then the spatial information of the candidate plane is determined through least square fitting.

After the above-mentioned extraction of different types of space planes, a virtual three-dimensional scene can be established according to the space planes, and further, a target object in the virtual three-dimensional scene is controlled, so that a user can conveniently interact with the virtual object, for example, the user can control the active state of the virtual object through a mobile phone AR, or perform a communication session with a virtual character, etc., and the specific interaction mode of the user and the AR device is only taken as an example and is not limited in detail herein.

According to the space information determining method provided by the embodiment of the invention, firstly, a visual SLAM technology is utilized to determine a camera gesture track, three-dimensional feature points and three-dimensional feature lines according to an image sequence acquired by a camera, then a three-dimensional triangular grid is generated based on the three-dimensional feature points and the three-dimensional feature lines, and finally an effective plane is extracted through the three-dimensional triangular grid and the three-dimensional feature lines, wherein the effective plane comprises a horizontal plane, a vertical plane and other planes except the horizontal plane and other planes. By combining the three-dimensional characteristic straight lines with the three-dimensional characteristic points, the three-dimensional space triangular grid is constructed, and then the three-dimensional space plane is extracted, so that the problem that accuracy and robustness are poor due to the fact that sparse point clouds are adopted when space information is determined can be solved, the accuracy and robustness of space information determination are effectively improved, and because the method does not need to iterate for many times, accurate space information determination can be carried out only through one-time statistics, and therefore the extraction efficiency of the space information can be improved.

Embodiment four:

for the spatial information determining method provided in the second embodiment, the embodiment of the present invention provides a spatial information determining apparatus, referring to a schematic structure diagram of a spatial information determining apparatus shown in fig. 7, the apparatus includes the following modules:

a spatial feature determination module 702 for determining spatial features based on a sequence of images acquired by the camera; the spatial features comprise three-dimensional feature points and three-dimensional feature lines;

a generating module 704, configured to generate a triangular mesh in a three-dimensional space through the three-dimensional feature points and the three-dimensional feature lines; the triangular mesh of the three-dimensional space comprises a plurality of triangular faces of the three-dimensional space;

the spatial information determining module 706 is configured to determine spatial information according to the triangular mesh and the three-dimensional feature line of each three-dimensional space.

The embodiment of the invention provides a space information determining device, which utilizes the triangular grids of an extracted three-dimensional characteristic line and a three-dimensional space to extract a space plane, and essentially combines the characteristic line with a sparse point cloud to extract space information.

In one embodiment, the spatial feature determining module 702 is further configured to estimate a camera pose track corresponding to the image sequence based on the image sequence acquired by the camera; the camera coordinate system at the initial moment corresponding to the camera gesture track is a world coordinate system aligned with the gravity direction; extracting two-dimensional feature points and two-dimensional feature lines based on the image sequence; and determining three-dimensional characteristic points and three-dimensional characteristic lines according to the gesture track of the camera, the two-dimensional characteristic points and the two-dimensional characteristic lines.

In one embodiment, the generating module 704 is further configured to construct a triangular mesh of the two-dimensional plane based on the two-dimensional feature points and the two-dimensional feature straight line; respectively calculating coordinate representation of the three-dimensional feature points corresponding to the two-dimensional feature points and the two-dimensional feature lines and vector representation of the three-dimensional feature lines; the triangular mesh of the three-dimensional space is determined by the coordinate representation of the three-dimensional feature points, the vector representation of the three-dimensional feature lines, and the triangular mesh of the two-dimensional plane.

In one embodiment, the generating module 704 is further configured to screen out, according to a preset attribute feature range of the triangular mesh in the three-dimensional space, a target three-dimensional feature point and a target three-dimensional feature line of the triangular mesh that cannot construct the three-dimensional space from the obtained three-dimensional feature point and the three-dimensional feature line, respectively; the attribute characteristic range comprises an inner angle degree range and a side length range of the triangular surface; a triangular mesh of the three-dimensional space is determined based on the three-dimensional feature points and the three-dimensional feature straight lines other than the target three-dimensional feature points and the target three-dimensional feature straight line.

In one embodiment, the spatial information determining module 706 is further configured to determine the candidate plane in the three-dimensional space according to a spatial position relationship between a plurality of triangular faces of the triangular mesh of the three-dimensional space and/or a spatial position relationship between the triangular faces and the three-dimensional feature line; the candidate planes include one or more of a horizontal plane, a vertical plane, and an inclined plane; verifying the candidate planes; and determining the space information according to the candidate planes passing the verification.

In one embodiment, the candidate plane includes a horizontal plane, and the spatial information determining module 706 is further configured to, when there is a first target three-dimensional feature line in the three-dimensional space that is parallel to a horizontal plane under a preset coordinate system, and/or there is a plurality of first target triangular planes in a triangular grid of the three-dimensional space that are parallel to the horizontal plane, count a height of each of the first target triangular planes and the first target three-dimensional feature line in the three-dimensional space, and a first counted number of corresponding first target triangular planes and first target three-dimensional feature lines at each height; searching the target height, and taking a horizontal plane at the target height as a candidate plane in the three-dimensional space; wherein the first statistical number corresponding to the target height is higher than a first threshold.

In one embodiment, the spatial information determining module 706 is further configured to determine, for each target height, whether a distance between a first target triangular surface corresponding to the target height and a first target three-dimensional feature line is smaller than a preset first distance threshold; if so, the horizontal plane at the target height is determined to be a valid candidate plane.

In one embodiment, the candidate plane includes a vertical plane, and the spatial information determining module 706 is further configured to, when there is a vertical plane normal vector of the second target three-dimensional feature line in the three-dimensional space perpendicular to the preset coordinate system, and/or there is a plurality of second target triangular planes in the triangular grid of the three-dimensional space perpendicular to the horizontal plane, count an azimuth angle of each of the second target triangular planes and a distance of each of the second target three-dimensional feature lines in the three-dimensional space from an origin of the preset coordinate system, and count a second count of the second target triangular planes and the second target three-dimensional feature lines corresponding to each azimuth angle and distance; wherein the vertical plane normal vector is a normal vector set parallel to the horizontal plane; the azimuth angle comprises a first azimuth angle between a second target triangular surface and a horizontal plane under a preset coordinate system and a second azimuth angle of a second target three-dimensional characteristic straight line in a three-dimensional space; the distance comprises a first distance between a triangular surface of the second target and an origin of a preset coordinate system and a second distance projected by a connecting line between a three-dimensional characteristic straight line of the second target and the origin of the preset coordinate system under the preset coordinate system; searching a target azimuth angle and a target distance, and taking a vertical plane positioned at the target azimuth angle and the target distance as a candidate plane in a three-dimensional space; and the second statistical number corresponding to the target azimuth angle and the target distance is higher than a second threshold value.

In one embodiment, the spatial information determining module 706 is further configured to determine, for each target azimuth and target distance, whether a distance between a second target triangular surface corresponding to the target azimuth and the target distance and a second target three-dimensional feature line is less than a preset second distance threshold; if so, the vertical plane at the target azimuth and target distance is determined to be a valid candidate plane.

In one embodiment, the candidate plane includes an inclined plane, and the spatial information determining module 706 is further configured to calculate an included angle and a spatial distance between every two third target triangular planes when there are a plurality of third target triangular planes in the triangular mesh of the three-dimensional space that are neither parallel nor perpendicular to the horizontal plane; and determining two triangular surfaces with included angles smaller than a preset included angle threshold and space distances smaller than a preset distance threshold to be attributed to the same inclined plane, and taking the determined inclined plane as a candidate plane in the three-dimensional space.

In one embodiment, the spatial information determining module 706 is further configured to determine, for each inclined plane, that the third target three-dimensional feature line belongs to the inclined plane when the third target three-dimensional feature line exists in the three-dimensional space and is neither parallel nor perpendicular to the horizontal plane under the preset coordinate system, and the distance between the third target three-dimensional feature line and the inclined plane is smaller than the third specified distance and/or the direction vectors of at least two third target three-dimensional feature lines exist within the third specified distance are parallel; judging whether a third statistical number of third target triangular surfaces and third target three-dimensional characteristic straight lines belonging to the inclined plane exceeds a third threshold value; if so, the inclined plane is determined to be a valid candidate plane.

In one embodiment, the spatial information includes plane information, and the apparatus further includes: and the plane extraction module is used for executing the space plane extraction operation according to the determined plane information.

In one embodiment, the spatial information includes plane information, and the apparatus further includes: the interaction module is used for establishing a virtual three-dimensional scene according to the extracted space plane; wherein the virtual three-dimensional scene comprises a target object; when input information of a user is received, a target object is controlled according to the input information.

In summary, the spatial information determining method, the spatial information determining device and the electronic equipment provided by the embodiment of the invention utilize the extracted three-dimensional characteristic straight line and the triangular grid of the three-dimensional space to extract the spatial plane, and essentially combine the characteristic straight line with the sparse point cloud to extract the plane.

The method, the apparatus and the computer program product of the electronic device for determining spatial information provided in the embodiments of the present invention include a computer readable storage medium storing program codes, and instructions included in the program codes may be used to execute the method in the foregoing method embodiment, and specific implementation may refer to the method embodiment and will not be described herein.

The functions, if implemented in the form of software functional units and sold or used as a stand-alone product, may be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention may be embodied essentially or in a part contributing to the prior art or in a part of the technical solution, in the form of a software product stored in a storage medium, comprising several instructions for causing a computer device (which may be a personal computer, a server, a network device, etc.) to perform all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), a magnetic disk, or an optical disk, or other various media capable of storing program codes.

In the description of the present invention, it should be noted that the terms "first," "second," "third," and the like are used for descriptive purposes only and are not to be construed as indicating or implying relative importance.

Finally, it should be noted that: the above examples are only specific embodiments of the present invention, and are not intended to limit the scope of the present invention, but it should be understood by those skilled in the art that the present invention is not limited thereto, and that the present invention is described in detail with reference to the foregoing examples: any person skilled in the art may modify or easily conceive of the technical solution described in the foregoing embodiments, or perform equivalent substitution of some of the technical features, while remaining within the technical scope of the present disclosure; such modifications, changes or substitutions do not depart from the spirit and scope of the technical solutions of the embodiments of the present invention, and are intended to be included in the scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A method for determining spatial information, the method comprising:

Determining spatial features based on an image sequence acquired by a camera; the spatial features comprise three-dimensional feature points and three-dimensional feature straight lines;

determining a triangular grid of a three-dimensional space through the three-dimensional characteristic points and the three-dimensional characteristic straight lines; the triangular mesh comprises a plurality of triangular faces;

determining space information according to the triangular mesh of the three-dimensional space and the three-dimensional characteristic straight line;

wherein the step of determining spatial information according to the triangular mesh of the three-dimensional space and the three-dimensional characteristic line comprises the following steps:

determining candidate planes in the three-dimensional space according to the spatial position relation among a plurality of triangular faces of the triangular mesh of the three-dimensional space and/or the spatial position relation between the triangular faces and the three-dimensional characteristic straight line; the candidate plane comprises one or more of a horizontal plane, a vertical plane and an inclined plane, and the three-dimensional characteristic straight line reflects a plane in space;

verifying the candidate plane;

and determining spatial information according to the candidate planes passing the verification.

2. The method of claim 1, wherein the step of determining spatial features based on the sequence of images acquired by the camera comprises:

Estimating a camera gesture track corresponding to an image sequence based on the image sequence acquired by a camera; the camera coordinate system of the initial moment corresponding to the camera gesture track is a world coordinate system aligned with the gravity direction;

extracting two-dimensional feature points and two-dimensional feature lines based on the image sequence;

and determining a three-dimensional characteristic point and a three-dimensional characteristic straight line according to the camera gesture track, the two-dimensional characteristic point and the two-dimensional characteristic straight line.

3. The method according to claim 2, wherein the step of determining a triangular mesh of a three-dimensional space by the three-dimensional feature points and the three-dimensional feature straight lines comprises:

constructing a triangular grid of a two-dimensional plane based on the two-dimensional feature points and the two-dimensional feature straight lines;

respectively calculating coordinate representation of the three-dimensional feature points and vector representation of the three-dimensional feature lines, wherein the coordinate representation of the three-dimensional feature points corresponds to the two-dimensional feature points and the two-dimensional feature lines;

and determining the triangular grid of the three-dimensional space through the coordinate representation of the three-dimensional characteristic points, the vector representation of the three-dimensional characteristic straight lines and the triangular grid of the two-dimensional plane.

4. The method of claim 1, wherein the step of determining a triangular mesh of a three-dimensional space by the three-dimensional feature points and the three-dimensional feature lines comprises:

According to the attribute feature range of the triangular mesh in the preset three-dimensional space, respectively screening out target three-dimensional feature points and target three-dimensional feature lines which cannot construct the triangular mesh in the three-dimensional space from the obtained three-dimensional feature points and three-dimensional feature lines; the attribute characteristic range comprises an inner angle degree range and a side length range of the triangular surface;

and determining a triangular grid of the three-dimensional space based on the three-dimensional feature points and the three-dimensional feature straight lines except the target three-dimensional feature points and the target three-dimensional feature straight line.

5. The method according to claim 1, wherein the candidate plane comprises a horizontal plane, and the step of determining the candidate plane in the three-dimensional space based on a spatial positional relationship between a plurality of triangular faces of a triangular mesh of the three-dimensional space and/or a spatial positional relationship between the triangular faces and the three-dimensional feature straight line comprises:

when a first target three-dimensional characteristic straight line exists in the three-dimensional space and is parallel to a horizontal plane under a preset coordinate system, and/or a plurality of first target triangular faces exist in a triangular grid of the three-dimensional space and are parallel to the horizontal plane, counting the heights of each first target triangular face and each first target three-dimensional characteristic straight line in the three-dimensional space, and the first counted number of the corresponding first target triangular face and first target three-dimensional characteristic straight line under each height;

Searching a target height, and taking a horizontal plane at the target height as a candidate plane in the three-dimensional space; wherein the first statistical number corresponding to the target height is higher than a first threshold.

6. The method of claim 5, wherein said validating said candidate planes comprises:

for each target height, judging whether the distance between the first target triangular surface corresponding to the target height and the first target three-dimensional characteristic straight line is smaller than a preset first distance threshold value;

if so, the horizontal plane at the target height is determined to be a valid candidate plane.

7. The method according to claim 1, wherein the candidate plane comprises a vertical plane, and the step of determining the candidate plane in the three-dimensional space based on a spatial positional relationship between a plurality of triangular faces of a triangular mesh of the three-dimensional space and/or a spatial positional relationship between the triangular faces and the three-dimensional feature straight line comprises:

when a vertical plane normal vector of a second target three-dimensional characteristic straight line perpendicular to a preset coordinate system exists in the three-dimensional space, and/or when a plurality of second target triangular planes perpendicular to a horizontal plane under the preset coordinate system exist in a triangular grid of the three-dimensional space, calculating azimuth angles of each second target triangular plane and the second target three-dimensional characteristic straight line in the three-dimensional space and distances between each second target triangular plane and an origin of the preset coordinate system, and calculating second statistical numbers of the second target triangular planes and the second target three-dimensional characteristic straight lines corresponding to each azimuth angle and each distance; wherein the vertical plane normal vector is a normal vector set parallel to a horizontal plane; the azimuth angle comprises a first azimuth angle between the second target triangular surface and a horizontal plane under the preset coordinate system and a second azimuth angle of the second target three-dimensional characteristic straight line in a three-dimensional space; the distance comprises a first distance between the triangular surface of the second target and the origin of the preset coordinate system and a second distance between the line of the three-dimensional characteristic of the second target and the origin of the preset coordinate system projected under the preset coordinate system;

Searching a target azimuth angle and a target distance, and taking a vertical plane positioned in the target azimuth angle and the target distance as a candidate plane in the three-dimensional space; and the second statistical number corresponding to the target azimuth angle and the target distance is higher than a second threshold value.

8. The method of claim 7, wherein the step of validating the candidate plane comprises:

judging whether the distance between every two second target triangular surfaces corresponding to the target azimuth angle and the target distance and the second target three-dimensional characteristic straight line is smaller than a preset second distance threshold value or not for each target azimuth angle and each target distance;

if so, the vertical plane at the target azimuth and target distance is determined to be a valid candidate plane.

9. The method according to claim 1, wherein the candidate plane comprises an inclined plane, and the step of determining the candidate plane in the three-dimensional space based on a spatial positional relationship between a plurality of triangular faces of a triangular mesh of the three-dimensional space and/or a spatial positional relationship between the triangular faces and the three-dimensional feature straight line comprises:

when a plurality of third target triangular surfaces are not parallel or perpendicular to a horizontal plane under a preset coordinate system in the triangular grid of the three-dimensional space, calculating included angles and space distances between every two of the plurality of third target triangular surfaces;

And determining two triangular surfaces with the included angle smaller than a preset included angle threshold and the space distance smaller than a preset distance threshold to be attributed to the same inclined plane, and taking the determined inclined plane as a candidate plane in the three-dimensional space.

10. The method of claim 9, wherein the step of validating the candidate plane comprises:

for each inclined plane, when a third target three-dimensional characteristic straight line exists in the three-dimensional space and is neither parallel nor perpendicular to a horizontal plane under a preset coordinate system, and the distance between the third target three-dimensional characteristic straight line and the inclined plane is smaller than a third specified distance and/or the direction vectors of at least two third target three-dimensional characteristic straight lines exist in the third specified distance are parallel, determining that the third target three-dimensional characteristic straight line belongs to the inclined plane;

judging whether a third statistical number of third target triangular surfaces and third target three-dimensional characteristic straight lines belonging to the inclined plane exceeds a third threshold value;

if so, the inclined plane is determined to be a valid candidate plane.

11. The method of claim 1, wherein the spatial information comprises plane information, the method further comprising:

And executing a space plane extraction operation according to the determined plane information.

12. The method of claim 11, wherein the method further comprises:

establishing a virtual three-dimensional scene according to the extracted space plane; wherein the virtual three-dimensional scene comprises a target object;

when input information of a user is received, the target object is controlled according to the input information.

13. A spatial information determination apparatus, characterized in that the apparatus comprises:

the spatial feature determining module is used for determining spatial features based on the image sequence acquired by the camera; the spatial features comprise three-dimensional feature points and three-dimensional feature straight lines;

the triangular mesh generation module is used for generating triangular meshes of a three-dimensional space through the three-dimensional characteristic points and the three-dimensional characteristic straight lines; the triangular mesh of the three-dimensional space comprises a plurality of triangular faces of the three-dimensional space;

the space information determining module is used for determining space information according to the triangular grids of each three-dimensional space and the three-dimensional characteristic straight lines;

wherein, the spatial information determining module is further configured to: determining candidate planes in the three-dimensional space according to the spatial position relation among a plurality of triangular faces of the triangular mesh of the three-dimensional space and/or the spatial position relation between the triangular faces and the three-dimensional characteristic straight line; the candidate plane comprises one or more of a horizontal plane, a vertical plane and an inclined plane, and the three-dimensional characteristic straight line reflects a plane in space; verifying the candidate plane; and determining spatial information according to the candidate planes passing the verification.

14. An electronic device, comprising: a processor and a storage device;

the storage means has stored thereon a computer program which, when executed by the processor, performs the method of any of claims 1 to 12.

15. A computer-readable storage medium, on which a computer program is stored, characterized in that the computer program, when being executed by a processor, performs the steps of the method of any of the preceding claims 1 to 12.