CN112446905B - Three-dimensional real-time panoramic monitoring method based on multi-degree-of-freedom sensing association - Google Patents

Abstract

The invention belongs to the technical field of real-time positioning, image construction and computer vision, and particularly relates to a three-dimensional real-time panoramic monitoring method, system and device based on multi-degree-of-freedom sensing association, aiming at solving the problems that the existing monitoring technology cannot realize large-range three-dimensional panoramic video monitoring, and is low in monitoring efficiency and poor in effect. The system method comprises the steps of obtaining real-time observation data of sensors with N different degrees of freedom, and constructing a three-dimensional semantic map corresponding to each sensor to serve as a local map; integrating local maps generated by all sensors to obtain a panoramic map serving as a first map; acquiring an external reference matrix correspondingly estimated in a first map by each sensor through a RANSAC algorithm; and calculating the error between the real external parameter matrix and the estimated external parameter matrix, and updating the first map to obtain the panoramic map finally obtained at the current moment of the scene to be monitored. The invention realizes the three-dimensional panoramic video monitoring in a large range, improves the monitoring efficiency and ensures the monitoring quality and effect.

Description

Three-dimensional real-time panoramic monitoring method based on multi-degree-of-freedom sensor correlation

technical field

The invention belongs to the technical fields of real-time positioning, mapping and computer vision, and in particular relates to a three-dimensional real-time panoramic monitoring method, system and device based on multi-degree-of-freedom sensing correlation.

Background technique

Video surveillance is an important and challenging classic computer vision task, which has a wide range of applications in the fields of security monitoring, intelligent video analysis, personnel search and rescue and retrieval. Usually surveillance cameras are installed in a fixed position to collect multi-angle and multi-pose 2D pedestrian images. If the monitoring personnel want to track the real-time position and movement trajectory of pedestrians, they often need to accumulate some experience and cannot obtain this information intuitively. Sensors that only rely on a single degree of freedom cannot meet the needs of video surveillance very well. This method proposes a three-dimensional panoramic monitoring method based on multi-degree-of-freedom sensor correlation. By combining three-dimensional environment modeling, instance segmentation, three-dimensional model projection and other technologies to achieve three-dimensional video monitoring, it can better solve this problem. question.

SUMMARY OF THE INVENTION

In order to solve the above problems in the prior art, that is, in order to solve the problem that most of the existing video surveillance technologies rely on a single degree of freedom sensor with a fixed viewing angle, it is impossible to realize a large-scale three-dimensional panoramic video surveillance, and the experience requirements of the monitoring personnel are high, and the monitoring efficiency is low. , the problem of poor effect, the first aspect of the present invention proposes a three-dimensional real-time panoramic monitoring method based on multi-degree-of-freedom sensing correlation, the method includes:

Step S10: Obtain real-time observation data of N sensors with different degrees of freedom in the scene to be monitored, and construct a three-dimensional semantic map corresponding to each sensor as a local map; N is a positive integer; the real-time observation data includes observation time, real external parameter matrix;

Step S20, integrating the local maps generated by each sensor to obtain a panoramic map of the scene to be monitored, as the first map;

Step S30, sequentially registering the first map with each local map, and obtaining the corresponding estimated external parameter matrix of each sensor in the first map through the RANSAC algorithm;

Step S40: Calculate the error between the real extrinsic parameter matrix and the estimated extrinsic parameter matrix; based on each error, update the first map to obtain a second map as the final panoramic map obtained at the current moment of the scene to be monitored.

In some preferred embodiments, the N sensors with different degrees of freedom include fixed viewing angle surveillance cameras, PTZ surveillance cameras, movable surveillance robots, and visual surveillance drones.

In some preferred embodiments, the three-dimensional semantic map includes a static background model and a dynamic semantic instance; as shown in the following formula:

表示

时刻的三维语义地图，

表示静态背景模型，

表示动态语义 实例，

表示实例的类别，

表示实例对应的三维模型，

表示实例的空间位置和方向。 in,

express

3D semantic map of moments,

represents a static background model,

represents a dynamic semantic instance,

represents the class of the instance,

represents the 3D model corresponding to the instance,

Represents the spatial location and orientation of the instance.

In some preferred embodiments, the panoramic map, namely the panoramic three-dimensional semantic map, the acquisition method is:

During the navigation process of the mobile monitoring robot, the static background model of the panoramic map is automatically constructed through the real-time positioning and mapping algorithm based on TSDF;

For the semantic instances of pedestrian categories, the pedestrian re-identification algorithm based on RGB images is used to match the same semantic instance in each local map; for non-pedestrian category semantic instances, the volume overlap ratio between the 3D models corresponding to the semantic instances in each local map is calculated. , take the semantic instances whose volume overlap ratio is higher than the set threshold as the same semantic instance; combine the matched same semantic instance to obtain the dynamic semantic instance in the panoramic map;

The panoramic map is constructed by combining the obtained static background model of the panoramic map and the dynamic semantic instances in the panoramic map.

In some preferred embodiments, in step S30, "acquiring the corresponding estimated external parameter matrix of each sensor in the first map through the RANSAC algorithm", the method is as follows:

selecting the shared semantic instance of the first map and the local map corresponding to each sensor;

According to the position of each shared semantic instance, the RANSAC algorithm is used to obtain the extrinsic parameter matrix estimated by each sensor.

In some preferred embodiments, in step S40, "update the first map based on each error", and the method is:

Determine whether the error is less than or equal to the set threshold, and if so, do not update;

Otherwise, the static background model in the first map is not updated, and the dynamic semantic instance in the first map is updated with the spatial position and direction of the dynamic semantic instance in combination with the error.

In some preferred embodiments, "in combination with the error, the spatial position and orientation of the dynamic semantic instance are updated", and the method is:

观测到，则更新后的空间位置和方向为： If the dynamic semantic instance is only used by the sensor

observed, the updated spatial position and orientation are:

If the dynamic semantic instance is observed by multiple sensors, the updated spatial position and orientation are:

表示所有观测到动态语义实例的传感器集合，

、

表示全景地图与第

、

个传感器对应的局部地图之间的误差。 in,

represents the set of all sensors that observe dynamic semantic instances,

,

Represents a panoramic map with the

,

The error between the local maps corresponding to each sensor.

In a second aspect of the present invention, a three-dimensional real-time panoramic monitoring system based on multi-degree-of-freedom sensing correlation is proposed, the system includes a local map acquisition module, a panoramic map acquisition module, a registration module, and an update output module;

The local map acquisition module is configured to acquire real-time observation data of N sensors with different degrees of freedom in the scene to be monitored, and construct a three-dimensional semantic map corresponding to each sensor as a local map; N is a positive integer; the real-time observation data includes Observation time, real external parameter matrix;

The panoramic map acquisition module is configured to integrate the local maps generated by each sensor to obtain a panoramic map of the scene to be monitored as the first map;

The registration module is configured to sequentially register the first map with each local map, and obtain the corresponding estimated external parameter matrix of each sensor in the first map through the RANSAC algorithm;

The update output module is configured to calculate the error between the real extrinsic parameter matrix and the estimated extrinsic parameter matrix; based on each error, the first map is updated to obtain a second map, which is used as the final map at the current moment of the scene to be monitored. Acquired panoramic map.

In a third aspect of the present invention, a storage device is provided, wherein a plurality of programs are stored, and the programs are suitable for being loaded and executed by a processor to realize the above-mentioned three-dimensional real-time panoramic monitoring method based on multi-degree-of-freedom sensor correlation.

In a fourth aspect of the present invention, a processing device is proposed, including a processor and a storage device; the processor is adapted to execute various programs; the storage device is adapted to store multiple programs; the programs are adapted to be loaded by the processor And execute to realize the above-mentioned three-dimensional real-time panoramic monitoring method based on multi-degree-of-freedom sensor correlation.

Beneficial effects of the present invention:

The present invention can realize three-dimensional panoramic video monitoring in a large range with continuous monitoring pictures, improve monitoring efficiency, and ensure monitoring quality and effect.

(1) The present invention introduces a multi-degree-of-freedom sensor, and integrates the observation data of the multi-degree-of-freedom sensor to construct a dynamic and semantically rich three-dimensional panoramic monitoring map. The map not only includes a static background model, but also includes a dynamic semantic instance model. Realize three-dimensional panoramic video surveillance in a large range and the monitoring picture is continuous.

(2) The present invention introduces an automatic calibration method for multi-degree-of-freedom sensors, and uses the semantic instances in the three-dimensional panoramic map as calibration templates to automatically calculate the transformation matrix between the local map generated by the multi-degree-of-freedom sensor observation and the semantic instances in the panoramic map. , calibrating the extrinsic parameter matrix of the multi-degree-of-freedom sensor. Then, the error matrix between the local map and the panoramic map is calculated with the currently estimated external parameter matrix, and the panoramic map is updated to obtain a more accurate external parameter matrix and a more accurate three-dimensional panoramic map, which improves the monitoring efficiency and ensures the quality and effect of monitoring. .

Description of drawings

Other features, objects and advantages of the present application will become more apparent upon reading the detailed description of non-limiting embodiments taken with reference to the following drawings.

1 is a schematic flowchart of a three-dimensional real-time panoramic monitoring method based on multi-degree-of-freedom sensing correlation according to an embodiment of the present invention;

2 is a schematic diagram of a framework of a three-dimensional real-time panoramic monitoring system based on multi-degree-of-freedom sensing correlation according to an embodiment of the present invention;

FIG. 3 is a schematic flowchart of a three-dimensional real-time panoramic monitoring method based on multi-degree-of-freedom sensing correlation according to an embodiment of the present invention.

Detailed ways

In order to make the objectives, technical solutions and advantages of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not All examples. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

The present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the related invention, but not to limit the invention. In addition, it should be noted that, for the convenience of description, only the parts related to the related invention are shown in the drawings.

It should be noted that the embodiments in the present application and the features of the embodiments may be combined with each other in the case of no conflict.

The three-dimensional real-time panoramic monitoring method based on multi-degree-of-freedom sensing correlation of the present invention, as shown in FIG. 1 , includes the following steps:

Step S10: Obtain real-time observation data of N sensors with different degrees of freedom in the scene to be monitored, and construct a three-dimensional semantic map corresponding to each sensor as a local map; N is a positive integer; the real-time observation data includes observation time, real external parameter matrix;

Step S20, integrating the local maps generated by each sensor to obtain a panoramic map of the scene to be monitored, as the first map;

Step S30, sequentially registering the first map with each local map, and obtaining the corresponding estimated external parameter matrix of each sensor in the first map through the RANSAC algorithm;

Step S40: Calculate the error between the real extrinsic parameter matrix and the estimated extrinsic parameter matrix; based on each error, update the first map to obtain a second map as the final panoramic map obtained at the current moment of the scene to be monitored.

In order to more clearly describe the three-dimensional real-time panoramic monitoring method based on multi-degree-of-freedom sensing correlation of the present invention, each step in an embodiment of the method of the present invention will be described in detail below with reference to the accompanying drawings.

The invention introduces a multi-degree-of-freedom sensor to provide more abundant visual information, and uses three-dimensional modeling, instance segmentation and other methods to construct a three-dimensional panoramic semantic map of the scene, and finally iteratively updates the external parameter matrix and panoramic map of the multi-degree-of-freedom sensor. Real-time 3D panoramic map update. details as follows:

Step S10: Obtain real-time observation data of N sensors with different degrees of freedom in the scene to be monitored, and construct a three-dimensional semantic map corresponding to each sensor as a local map; N is a positive integer; the real-time observation data includes observation time, real external parameter matrix;

In this embodiment, N kinds of sensors with different degrees of freedom preferably adopt fixed viewing angle surveillance cameras (zero degrees of freedom), PTZ surveillance cameras (camera attitude degrees of freedom: 2 dimensions and scaling degrees of freedom: 1 dimension), movable surveillance robots ( Camera attitude freedom: 2D, scaling freedom: 1D, robot pose freedom: 6D), visual surveillance drone (camera attitude freedom: 2D, scaling freedom: 1D, unmanned The degree of freedom of the camera position and attitude: 6 dimensions), in other embodiments, the sensor can be selected according to actual needs.

在

时刻的观测数据表示为：

，其 中，

为摄像机的外参矩阵（真实的外参矩阵），表示传感器的位置和姿态，即位姿，对于PTZ 监控摄像机、可移动监控机器人、视觉监控无人机，外参矩阵为时变函数

；

为传感器 采样矩阵。同样，摄像机的成像方式不同，

表示的含义不同，对于RGB相机，

为相机内参。 对于变焦相机，

时变函数

，对于激光雷达等能够直接测量外部环境三维坐标的传感 器，

为单位矩阵。 In the present invention, multiple sensors

exist

The observation data at time is expressed as:

,in,

is the extrinsic parameter matrix of the camera (the real extrinsic parameter matrix), which represents the position and attitude of the sensor, that is, the pose. For PTZ surveillance cameras, mobile surveillance robots, and visual surveillance drones, the extrinsic parameter matrix is a time-varying function

;

Sample matrix for the sensor. Similarly, the imaging method of the camera is different,

The meaning of the representation is different, for RGB cameras,

Internal parameters for the camera. For zoom cameras,

time-varying function

, for sensors such as lidar that can directly measure the three-dimensional coordinates of the external environment,

is the identity matrix.

和动态语义实例模型

，如式（1）所示： Each sensor builds its corresponding three-dimensional semantic map according to the observation data as a local map. 3D semantic map including static background model

and dynamic semantic instance model

, as shown in formula (1):

（1）

(1)

其如式（2）所示： Dynamic Semantic Instance Model

It is shown in formula (2):

=

（2）

=

(2)

表示动态语义实例的类别，

表示动态语义实例的三维模型，

表示动 态语义实例的空间位置和方向。由于监控环境中动态语义实例

的位置、姿态甚至模型都 会发生改变，因而

可表示为时间的函数

。 in,

a class representing a dynamic semantic instance,

3D models representing dynamic semantic instances,

Represents the spatial location and orientation of dynamic semantic instances. Due to dynamic semantic instances in the monitoring environment

The position, attitude and even the model will change, so

can be expressed as a function of time

.

Step S20, integrating the local maps generated by each sensor to obtain a panoramic map of the scene to be monitored, as the first map;

In this embodiment, through the registration and synchronization of space and time, multiple perception information sources with different degrees of freedom are integrated to construct a panoramic map of the scene to be monitored.

When constructing a panoramic map, the static background model is automatically constructed by the real-time positioning and mapping algorithm based on TSDF during the navigation process of the mobile monitoring robot. Dynamic semantic instances are constructed based on the observation data acquired by real-time multi-DOF sensors through three steps: semantic instance extraction, 3D model mapping, and cross-sensor instance re-identification. details as follows:

Step S21, performing semantic instance extraction on the observation data of the multi-degree-of-freedom sensor acquired in real time;

Among them, for the visual sensor, the instance segmentation algorithm based on RGB image is used for extraction; for the lidar sensor, the 3D instance segmentation algorithm based on point cloud is used for extraction.

Step S22, one-to-one correspondence between the extracted semantic instance and the three-dimensional model of the category, and combining the depth sensor information to obtain the three-dimensional space position and direction of the model;

After steps S21 and S22, local maps corresponding to each sensor are obtained. Next, each local map is integrated to obtain a panoramic map, that is, a panoramic three-dimensional semantic map.

Step S23, re-identification of cross-sensor semantic instances.

For the semantic instance of pedestrian category, a pedestrian re-identification algorithm based on RGB images is used (since most of the sensors in the present invention are visual sensors) to match the same semantic instance in the field of view (local map) of different sensors; in other embodiments, pedestrian category semantics The instance can be obtained by selecting a suitable re-identification algorithm according to the sensor.

For non-pedestrian category instances, the overlapping proportions of the volumes between the three-dimensional models corresponding to the semantic instances under observation by different sensors are calculated. the same semantic instance of .

Step S30, sequentially registering the first map with each local map, and obtaining the corresponding estimated external parameter matrix of each sensor in the first map through the RANSAC algorithm;

与全局地图

进行配 准，计算

时刻多自由度传感器各自在地图中的观测数据

。具体 来说，在全景地图中选取与当前传感器对应的局部地图共有（同一）的语义实例，根据语义 实例的位置，使用RANSAC算法获取当前传感器估计的外参矩阵。 In this embodiment, the local map generated by the multi-degree-of-freedom sensor

with global map

register, calculate

The observation data of each multi-degree-of-freedom sensor in the map at time

. Specifically, the semantic instance shared (identical) with the local map corresponding to the current sensor is selected in the panoramic map, and the extrinsic parameter matrix estimated by the current sensor is obtained using the RANSAC algorithm according to the location of the semantic instance.

Step S40, calculating the error between the real extrinsic parameter matrix and the estimated extrinsic parameter matrix; based on each error, the first map is updated to obtain a second map, which is used as the panorama map finally obtained at the current moment of the scene to be monitored, details as follows:

投影到全局坐标系，然后将投影后的局部地 图与全景地图

进行配准，计算全景地图与真实观测的误差，并矫正全景地图。 In this embodiment, each local map is

Project to the global coordinate system, and then combine the projected local map with the panoramic map

Perform registration, calculate the error between the panoramic map and the real observation, and correct the panoramic map.

（或简称误差矩阵），当误差大于设定阈值，则矫正更新全景地图。 Specifically, the shared semantic instance is selected on the current local map and the panoramic map (ie, the first map), as shown in Figure 3, using the external parameter matrix of each sensor, the shared semantic instance is extracted from the sensor (given in the figure). The coordinate system of sensor 1 and sensor 2 is projected into the global coordinate system. In the global coordinate system, there is a certain spatial pose (spatial position and orientation) error (ie, the error of the external parameter matrix) between the sensor (local map) and the corresponding semantic instance in the panoramic map. Using the RANSAC method to calculate the error between the panoramic map and the local map according to the location set of the shared instances

(or error matrix for short), when the error is greater than the set threshold, the panoramic map is corrected and updated.

Among them, the method of "correcting and updating the panoramic map when the error is greater than the set threshold" is:

，更新其空间位姿，若动态语义实例仅在传感器

中被观测到，则更新后的空间位姿为：

，其中×为矩阵乘法。若动态语义实例

在多个传感器中被观测到，则更新后 的空间位姿为：

，其中

为所有可观测到实例

的传感器集合。 For static background models in panorama maps, no update is performed, for dynamic semantic instances in panorama maps

, update its spatial pose, if the dynamic semantic instance is only in the sensor

is observed in , the updated spatial pose is:

, where × is the matrix multiplication. If dynamic semantic instance

is observed in multiple sensors, the updated spatial pose is:

,in

for all observable instances

collection of sensors.

与全景地图。 Repeat the above steps S30, S40 to iteratively update the multi-degree-of-freedom sensor observation data

with panoramic map.

After acquiring the panoramic map, the present invention converts the panoramic map into GLB format and imports it into the Habitat-sim simulator, and uses the Habitat-lab library to train the visual navigation algorithm model based on reinforcement learning. It should be noted that the virtual agent in the simulator The loaded sensor parameters (including sensor type and external parameters) are consistent with the real environment.

Among them, "Visual Navigation Algorithm Based on Reinforcement Learning" consists of three modules, which are as follows:

，

为传感器，

为时间戳，将局部空间地图拼接到一起生成 全局地图

，维度为2×M×M； Real-time positioning and mapping module (SLAM module), input real-time data of multi-degree-of-freedom sensors into the neural network model to generate local spatial maps

,

for the sensor,

For timestamps, stitch the local spatial maps together to generate a global map

, the dimension is 2×M×M;

规划虚拟智能体的全局行动路径； Global decision module, according to the global map

Plan the global action path of the virtual agent;

The local decision-making module plans the local action path of the virtual agent according to the global path and the current reachable area.

A three-dimensional real-time panoramic monitoring system based on multi-degree-of-freedom sensing correlation according to the second embodiment of the present invention, as shown in FIG. 2, includes: a local map acquisition module 100, a panoramic map acquisition module 200, a registration module 300, and an update output module 400;

The local map acquisition module 100 is configured to acquire real-time observation data of N sensors with different degrees of freedom in the scene to be monitored, and construct a three-dimensional semantic map corresponding to each sensor as a local map; N is a positive integer; the real-time observation data Including observation time, real external parameter matrix;

The panoramic map acquisition module 200 is configured to integrate the local maps generated by each sensor to obtain a panoramic map of the scene to be monitored, as the first map;

The registration module 300 is configured to sequentially register the first map with each local map, and obtain the corresponding estimated external parameter matrix of each sensor in the first map through the RANSAC algorithm;

The update output module 400 is configured to calculate the error between the real extrinsic parameter matrix and the estimated extrinsic parameter matrix; based on each error, update the first map to obtain a second map, which is used as the current moment of the scene to be monitored The final obtained panoramic map.

Those skilled in the technical field can clearly understand that, for the convenience and brevity of description, for the specific working process and related description of the system described above, reference may be made to the corresponding process in the foregoing method embodiments, which will not be repeated here.

It should be noted that the three-dimensional real-time panoramic monitoring system based on multi-degree-of-freedom sensing correlation provided by the above embodiments is only illustrated by the division of the above functional modules. In practical applications, the above functions can be allocated by It can be completed by different functional modules, that is, the modules or steps in the embodiments of the present invention are decomposed or combined. For example, the modules in the above-mentioned embodiments can be combined into one module, and can also be further split into multiple sub-modules, so as to complete the above description. All or part of the functionality. The names of the modules and steps involved in the embodiments of the present invention are only for distinguishing each module or step, and should not be regarded as an improper limitation of the present invention.

A storage device according to a third embodiment of the present invention stores a plurality of programs, and the programs are suitable for being loaded by a processor and implementing the above-mentioned three-dimensional real-time panoramic monitoring method based on multi-degree-of-freedom sensing correlation.

A processing device according to a fourth embodiment of the present invention includes a processor and a storage device; the processor is adapted to execute various programs; the storage device is adapted to store multiple programs; the programs are adapted to be loaded and executed by the processor In order to realize the above-mentioned three-dimensional real-time panoramic monitoring method based on multi-degree-of-freedom sensor correlation.

Those skilled in the technical field can clearly understand that the undescribed convenience and brevity are not described. Repeat.

Those skilled in the art should be aware that the modules and method steps of each example described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, computer software or a combination of the two, and the programs corresponding to the software modules and method steps Can be placed in random access memory (RAM), internal memory, read only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or as known in the art in any other form of storage medium. In order to clearly illustrate the interchangeability of electronic hardware and software, the components and steps of each example have been described generally in terms of functionality in the foregoing description. Whether these functions are performed in electronic hardware or software depends on the specific application and design constraints of the technical solution. Skilled artisans may use different methods of implementing the described functionality for each particular application, but such implementations should not be considered beyond the scope of the present invention.

The terms "first," "second," etc. are used to distinguish between similar objects, and are not used to describe or indicate a particular order or sequence.

The term "comprising" or any other similar term is intended to encompass a non-exclusive inclusion such that a process, method, article or device/means comprising a list of elements includes not only those elements but also other elements not expressly listed, or Also included are elements inherent to these processes, methods, articles or devices/devices.

So far, the technical solutions of the present invention have been described with reference to the preferred embodiments shown in the accompanying drawings, however, those skilled in the art can easily understand that the protection scope of the present invention is obviously not limited to these specific embodiments. Without departing from the principle of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will fall within the protection scope of the present invention.

Claims (8)

1. A three-dimensional real-time panoramic monitoring method based on multi-degree-of-freedom sensing association is characterized by comprising the following steps:

step S10, acquiring real-time observation data of N sensors with different degrees of freedom in a scene to be monitored, and constructing a three-dimensional semantic map corresponding to each sensor as a local map; n is a positive integer; the real-time observation data comprises observation time and a real external parameter matrix; the three-dimensional semantic map comprises a static background model and a dynamic semantic instance, and is represented by the following formula:

wherein,

to represent

A three-dimensional semantic map of the time of day,

a static background model is represented that represents a static background model,

a dynamic instance of semantics is represented that,

the categories of the instances are shown in the figure,

a three-dimensional model corresponding to the instance is represented,

representing the spatial position and orientation of the instance;

step S20, integrating local maps generated by each sensor to obtain a panoramic map of a scene to be monitored as a first map;

the construction method of the panoramic map, namely the panoramic three-dimensional semantic map, comprises the following steps:

in the navigation process of the movable monitoring robot, a static background model of the panoramic map is automatically constructed through a real-time positioning and mapping algorithm based on TSDF;

aiming at the pedestrian category semantic instances, matching the same semantic instance in each local map by using a pedestrian re-recognition algorithm based on an RGB image; calculating volume overlap ratio between three-dimensional models corresponding to semantic instances in each local map aiming at the non-pedestrian category semantic instances, and taking the semantic instances with the volume overlap ratio higher than a set threshold value as the same semantic instance; acquiring a dynamic semantic instance in the panoramic map by combining the matched same semantic instance;

constructing a panoramic map by combining the obtained static background model of the panoramic map and the dynamic semantic instances in the panoramic map;

step S30, registering the first map and each local map in sequence, and acquiring the corresponding estimated external reference matrix of each sensor in the first map through RANSAC algorithm;

step S40, calculating the error between the real external parameter matrix and the estimated external parameter matrix; and updating the first map based on each error to obtain a second map serving as a panoramic map finally obtained at the current moment of the scene to be monitored.

2. The three-dimensional real-time panoramic monitoring method based on multi-degree-of-freedom sensing association as claimed in claim 1, wherein the sensors with N different degrees of freedom include fixed view monitoring cameras, PTZ monitoring cameras, movable monitoring robots, and visual monitoring unmanned aerial vehicles.

3. The three-dimensional real-time panoramic monitoring method based on multiple degrees of freedom sensing association as claimed in claim 2, wherein in step S30, "obtaining the corresponding estimated external reference matrix of each sensor in the first map by RANSAC algorithm" comprises:

selecting a common semantic instance of the first map and local maps corresponding to the sensors;

and acquiring the external parameter matrix estimated by each sensor by adopting a RANSAC algorithm according to the position of each common semantic instance.

4. The three-dimensional real-time panoramic monitoring method based on multiple degrees of freedom sensing association according to claim 3, wherein in step S40, "update the first map based on each error" includes:

judging whether the error is less than or equal to a set threshold value, if so, not updating;

otherwise, the static background model in the first map is not updated, and the space position and the direction of the dynamic semantic instance in the first map are updated by combining the error with the dynamic semantic instance in the first map.

5. The three-dimensional real-time panoramic monitoring method based on multi-degree-of-freedom sensing association according to claim 4, wherein the method for updating the spatial position and direction of the dynamic semantic instance in combination with the error comprises the following steps:

if the dynamic semantic instance is only sensed by the sensor

It is observed that then the updated spatial position and orientation are:

if the dynamic semantic instance is observed by multiple sensors, the updated spatial position and direction are:

wherein,

represents the set of all sensors observing dynamic semantic instances,

、

representing a first map and a second map

、

Error between the local maps corresponding to the individual sensors.

6. A three-dimensional real-time panoramic monitoring system based on multi-degree-of-freedom sensing association is characterized by comprising a local map acquisition module, a panoramic map acquisition module, a registration module and an update output module;

the local map acquisition module is configured to acquire real-time observation data of the sensors with N different degrees of freedom in a scene to be monitored, and construct a three-dimensional semantic map corresponding to each sensor as a local map; n is a positive integer; the real-time observation data comprises observation time and a real external parameter matrix; the three-dimensional semantic map comprises a static background model and a dynamic semantic instance, and is represented by the following formula:

wherein,

to represent

A three-dimensional semantic map of the time of day,

a static background model is represented that represents a static background model,

a dynamic instance of semantics is represented that,

the categories of the instances are shown in the figure,

a three-dimensional model corresponding to the instance is represented,

representing the spatial position and orientation of the instance;

the panoramic map acquisition module is configured to integrate local maps generated by the sensors to obtain a panoramic map of a scene to be monitored as a first map;

the construction method of the panoramic map, namely the panoramic three-dimensional semantic map, comprises the following steps:

in the navigation process of the movable monitoring robot, a static background model of the panoramic map is automatically constructed through a real-time positioning and mapping algorithm based on TSDF;

aiming at the pedestrian category semantic instances, matching the same semantic instance in each local map by using a pedestrian re-recognition algorithm based on an RGB image; calculating volume overlap ratio between three-dimensional models corresponding to semantic instances in each local map aiming at the non-pedestrian category semantic instances, and taking the semantic instances with the volume overlap ratio higher than a set threshold value as the same semantic instance; acquiring a dynamic semantic instance in the panoramic map by combining the matched same semantic instance;

constructing a panoramic map by combining the obtained static background model of the panoramic map and the dynamic semantic instances in the panoramic map;

the registration module is configured to sequentially register the first map with each local map, and acquire an external reference matrix corresponding to and estimated by each sensor in the first map through a RANSAC algorithm;

the update output module is configured to calculate an error between the real external parameter matrix and the estimated external parameter matrix; and updating the first map based on each error to obtain a second map serving as a panoramic map finally obtained at the current moment of the scene to be monitored.

7. A storage device having a plurality of programs stored therein, wherein the programs are adapted to be loaded and executed by a processor to implement the three-dimensional real-time panoramic monitoring method based on multiple degrees of freedom sensing correlation according to any one of claims 1 to 5.

8. A processing device comprising a processor, a storage device; a processor adapted to execute various programs; a storage device adapted to store a plurality of programs; characterized in that the program is suitable for being loaded and executed by a processor to realize the three-dimensional real-time panoramic monitoring method based on multi-degree-of-freedom sensing association as set forth in any one of claims 1 to 5.

Images