CN113810736B - An AI-driven real-time point cloud video transmission method and system - Google Patents

Abstract

The invention discloses an AI-driven real-time point cloud video transmission method and a system, wherein the method comprises the following steps: acquiring video data information by using AI (Artificial intelligence) generating equipment, and processing the video data information to obtain point cloud video data; extracting hierarchical features of the point cloud video data, determining key point cloud features in the point cloud video frame, and transmitting the key point cloud features; and receiving the key point cloud characteristics, expanding and reconstructing the received key point cloud characteristics to obtain point cloud information similar to the original input point cloud, and forming an original point cloud video on the visual effect. According to the invention, the original point cloud video stream to be transmitted is subjected to feature extraction, only part of key point cloud features are transmitted, and finally recovery reconstruction is carried out at the receiving end, so that the effect that the original point cloud video is transmitted visually is achieved, the transmission quantity and energy consumption of the point cloud video stream can be reduced remarkably, complex multiple processing in the traditional transmission scheme is avoided, and the data transmission quantity is greatly reduced.

Description

An AI-driven real-time point cloud video transmission method and system

technical field

The present invention relates to the field of point cloud video stream transmission, in particular to an AI-driven real-time point cloud video transmission method and system.

Background technique

Point clouds are a typical and popular data format for describing volumetric media and holographic videos, which can be captured by RGB-D cameras with depth sensors. It allows users to experience a motion scene with six degrees of freedom, and can change the position and direction in it, as there are only three degrees of freedom in traditional virtual reality (Vitual Reality, VR) videos. Volumetric media provides 3D scenes from multiple angles and is widely used in various fields, including education, medical treatment, and entertainment.

At present, it is very challenging to transmit dense point cloud streams even in the existing network environment including 5G, for the following reasons: (1) Transmitting a large amount of point cloud video requires extremely high bandwidth, even after traditional compression, it may Reaching the gigabit per second (Gbps) level, exceeding the capabilities of current 5G networks; (2) 3D volumetric media is computationally expensive because only software codecs can be used, and inefficient codecs will also slow down Transmission speed; (3) Traditional technologies such as rate adaptation and buffer control of Adaptive Bitrate Streaming (ABR) are not suitable for 3D volumetric media, and advanced technologies for volumetric media need to be explored.

At present, most of the existing point cloud video transmission technologies can be realized through traditional compression (including providing lossy and lossless compression), reducing the size of the data to be transmitted, but there are still many deficiencies, such as: On the one hand, point cloud video Advanced lossless compression technology is still not enough to achieve efficient transmission and better user experience. On the other hand, under limited network conditions, it is difficult for lossy compression to guarantee the authenticity of the recovered point cloud to match the actual original video. Other point cloud video transmission technologies, such as those that extend current VR video streams, are transmitted at the data block level. These methods have high mobile energy consumption, and the processing delay on the receiving device is often unacceptable. Additionally, each transport block is susceptible to various packet losses during network fluctuations and reassembly.

To sum up, the transmission capacity of traditional technologies is far from meeting the bandwidth requirements of real-time point cloud video streaming. Therefore, it is necessary to explore an advanced transmission scheme to ensure good service under the existing network.

Contents of the invention

In view of the problems in related technologies, the present invention proposes an AI-driven real-time point cloud video transmission method and system, which can significantly reduce the transmission volume and energy consumption of point cloud video streams. The system avoids the cumbersome multi-processing in traditional transmission schemes, and designs and trains an end-to-end deep learning network from raw data acquisition to final rendering and playback.

Technical scheme of the present invention is realized like this:

According to one aspect of the present invention, an AI-driven real-time point cloud video transmission method is provided.

The AI-driven real-time point cloud video transmission method includes:

Use AI generation equipment to obtain video data information, and process the video data information to obtain point cloud video data;

Perform layered feature extraction on the point cloud video data, determine the key point cloud features in the point cloud video frame, and transmit the key point cloud features;

The received key point cloud features are expanded and reconstructed to obtain point cloud information similar to the original input point cloud to form the original point cloud video in terms of visual effects.

Among them, using AI generation equipment to obtain video data information, and processing the video data information to obtain point cloud video data includes: using multiple depth cameras from different angles to scan the 3D model to be transmitted, and obtaining the point of each depth camera Cloud flow; and splicing multiple point clouds collected from multi-view cameras into a complete point cloud to obtain point cloud video data;

In addition, this AI-driven real-time point cloud video transmission method also includes:

The pre-built deep neural network will be trained offline using a large number of training sets consisting of 3D models of various sizes, and multiple candidate neural network models will be obtained through training;

Each trained neural network model is split into a layered feature extraction module and a point cloud restoration and reconstruction module based on a generative confrontation network; and are respectively deployed to high-performance edge servers and user terminal devices near the input end;

While deploying the layered feature extraction module and the point cloud restoration and reconstruction module based on the generative confrontation network to high-performance edge servers and user terminal devices, an adaptive matcher is deployed to prompt the adaptive matcher to monitor changes in network bandwidth in real time Adaptively match the neural network model that satisfies the current network's real-time point cloud video frame feature extraction and reconstruction recovery.

Further, the hierarchical feature extraction module includes three series of set abstraction levels for hierarchical feature learning to capture the local structure of the original point cloud; the set abstraction layer is composed of three basic layers, including sampling layer, grouping layer and mini-PointNet layer;

Among them, the sampling layer uses the farthest point sampling technique to select a subset from the output of the previous layer, and each point in the subset represents the center of a local area; the grouping layer finds n nearest neighbors around the center of the local area Points are combined to construct a local area set; the mini PointNet layer uses 3 two-dimensional convolutional layers and 1 maximum pooling layer to transform the local area set into a feature vector; the feature vector output by the mini PointNet layer of the last collection abstraction layer is the data to be transmitted.

Further, the point cloud restoration and reconstruction module includes a point cloud feature extension part and a final point set generation part, wherein the point cloud feature extension part is used to receive and transmit key point cloud features through a multi-layer perception The dimensionality of the feature is unified by the device; and through an up-down-up extension unit, a more diverse number of point clouds and feature dimensions are generated; the final point set generation part contains two layers of multi-layer perceptrons, and passed The two multilayer perceptron layers reconstruct the expanded point cloud features into a 3D coordinate form.

In addition, the layered feature extraction of the point cloud video data and the determination of the key point cloud features in the point cloud video frame include: through the layered feature extraction module deployed to a high-performance edge server close to the input end, the point cloud video data is analyzed. Layer feature extraction to identify key point cloud features in point cloud video frames.

In addition, expanding and reconstructing the received key point cloud features, obtaining point cloud information similar to the original input point cloud, and forming the original point cloud video in terms of visual effects include: The point cloud recovery and reconstruction module of the generated confrontation network expands and reconstructs the key point cloud features, obtains point cloud information similar to the original input point cloud, and forms the original point cloud video in terms of visual effects.

According to another aspect of the present invention, an AI-driven real-time point cloud video transmission system is provided.

This AI-driven real-time point cloud video transmission system includes:

The point cloud video data acquisition module is used to obtain video data information using AI generation equipment, and process the video data information to obtain point cloud video data;

The key point cloud feature extraction module is used to extract the layered features of the point cloud video data, determine the key point cloud features in the point cloud video frame, and transmit the key point cloud features;

The point cloud recovery and reconstruction module is used to receive the key point cloud features, expand and reconstruct the received key point cloud features, obtain point cloud information similar to the original input point cloud, and form the original point cloud video in visual effect .

Wherein, the point cloud video data acquisition module uses AI generating equipment to acquire video data information, and processes the video data information to obtain point cloud video data, and uses a plurality of depth cameras at different angles to scan the 3D model that needs to be transmitted , and obtain the point cloud stream of each depth camera; and splicing multiple point clouds collected from multi-view cameras into a complete point cloud to obtain point cloud video data

In addition, the AI-driven real-time point cloud video transmission system also includes:

The neural network training module is used to train the pre-built deep neural network offline using a large number of training sets consisting of 3D models of various sizes, and obtain multiple candidate neural network models through training;

The neural network deployment module is used to split each trained neural network model into a layered feature extraction module and a point cloud restoration and reconstruction module based on the generation confrontation network; and deploy them to high-performance edge servers and user terminal devices close to the input terminal respectively ;

The neural network matching module is used to deploy the adaptive matcher while deploying the layered feature extraction module and the point cloud restoration and reconstruction module based on the generative confrontation network to high-performance edge servers and user terminal devices, so that the adaptive matcher can be deployed according to The real-time monitored network bandwidth changes are adaptively matched to the neural network model that satisfies the current network's real-time point cloud video frame feature extraction and reconstruction recovery.

Further, the hierarchical feature extraction module includes three series of set abstraction levels for hierarchical feature learning to capture the local structure of the original point cloud; the set abstraction layer is composed of three basic layers, including sampling layer, grouping layer and mini-PointNet layer;

Among them, the sampling layer uses the farthest point sampling technique to select a subset from the output of the previous layer, and each point in the subset represents the center of a local area; the grouping layer finds n nearest neighbors around the center of the local area Points are combined to construct a local area set; the mini PointNet layer uses 3 two-dimensional convolutional layers and 1 maximum pooling layer to transform the local area set into a feature vector; the feature vector output by the mini PointNet layer of the last collection abstraction layer is the data to be transmitted.

Further, the point cloud restoration and reconstruction module includes a point cloud feature extension part and a final point set generation part, wherein the point cloud feature extension part is used to receive and transmit key point cloud features through a multi-layer perception The dimension of the feature is unified by the device; and through an up-down-up expansion unit, a more diverse number of point clouds and feature dimensions are generated; the final point set generation part contains two multi-layer perceptron layers for The expanded point cloud features are reconstructed into a three-dimensional coordinate form through the two multi-layer perceptron layers.

In addition, when the key point cloud feature extraction module performs layered feature extraction on point cloud video data and determines key point cloud features in point cloud video frames, the layered feature extraction module is deployed to a high-performance edge server close to the input end. The module extracts layered features from the point cloud video data, and determines the key point cloud features in the point cloud video frame.

In addition, when the point cloud restoration and reconstruction module expands and reconstructs the received key point cloud features, obtains point cloud information similar to the original input point cloud, and forms the original point cloud video on the visual effect, it is deployed close to The point cloud recovery and reconstruction module based on the generative confrontation network on the user terminal device at the input end expands and reconstructs key point cloud features, obtains point cloud information similar to the original input point cloud, and forms the original point cloud video in visual effect .

Beneficial effect:

The present invention extracts the features of the original point cloud video stream that needs to be transmitted, only transmits part of the key point cloud features, and finally restores and reconstructs at the receiving end, so as to achieve the effect of visually transmitting the original point cloud video, which can significantly reduce the point cloud Transmission volume and energy consumption of video streams. It avoids the cumbersome multi-processing in the traditional transmission scheme, greatly reduces the amount of data transmission, and makes it more suitable for the existing network environment. The present invention also considers the dynamics and instability of the network environment, incorporates it into end-to-end network design and training, and provides an adaptive transmission control algorithm to balance transmission delay and reconstruction accuracy.

Description of drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the accompanying drawings required in the embodiments. Obviously, the accompanying drawings in the following description are only some of the present invention. Embodiments, for those of ordinary skill in the art, other drawings can also be obtained based on these drawings without any creative effort.

Fig. 1 is a schematic flow chart of real-time point cloud video transmission driven by a kind of AI according to an embodiment of the present invention;

Fig. 2 is a structural schematic block diagram of an AI-driven real-time point cloud video transmission system according to an embodiment of the present invention;

3 is a schematic diagram of the principle of an AI-driven real-time point cloud video transmission method according to an embodiment of the present invention;

FIG. 4 is a schematic diagram of a structural design of a deep neural network model according to an embodiment of the present invention.

Detailed ways

The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some of the embodiments of the present invention, not all of them. All other embodiments obtained by persons of ordinary skill in the art based on the embodiments of the present invention belong to the protection scope of the present invention.

According to an embodiment of the present invention, an AI-driven real-time point cloud video transmission method is provided.

As shown in Figure 1, the AI-driven real-time point cloud video transmission method according to the embodiment of the present invention includes:

Step S101, using the AI generating device to obtain video data information, and processing the video data information to obtain point cloud video data;

Step S103, performing layered feature extraction on the point cloud video data, determining key point cloud features in the point cloud video frame, and transmitting the key point cloud features;

Step S105, the received key point cloud features, and expand and reconstruct the received key point cloud features, to obtain point cloud information similar to the original input point cloud, and form the original point cloud video in visual effect.

Among them, using AI generation equipment to obtain video data information, and processing the video data information to obtain point cloud video data includes: using multiple depth cameras from different angles to scan the 3D model to be transmitted, and obtaining the point of each depth camera Cloud flow; and splicing multiple point clouds collected from multi-view cameras into a complete point cloud to obtain point cloud video data;

In addition, this AI-driven real-time point cloud video transmission method also includes:

The pre-built deep neural network will be trained offline using a large number of training sets consisting of 3D models of various sizes, and multiple candidate neural network models will be obtained through training;

Each trained neural network model is split into a layered feature extraction module and a point cloud restoration and reconstruction module based on a generative confrontation network; and are respectively deployed to high-performance edge servers and user terminal devices near the input end;

While deploying the layered feature extraction module and the point cloud restoration and reconstruction module based on the generative confrontation network to high-performance edge servers and user terminal devices, an adaptive matcher is deployed to prompt the adaptive matcher to monitor changes in network bandwidth in real time Adaptively match the neural network model that satisfies the current network's real-time point cloud video frame feature extraction and reconstruction recovery.

Further, the hierarchical feature extraction module includes three series of set abstraction levels for hierarchical feature learning to capture the local structure of the original point cloud; the set abstraction layer is composed of three basic layers, including sampling layer, grouping layer and mini-PointNet layer;

Among them, the sampling layer uses the farthest point sampling technique to select a subset from the output of the previous layer, and each point in the subset represents the center of a local area; the grouping layer finds n nearest neighbors around the center of the local area Points are combined to construct a local area set; the mini PointNet layer uses 3 two-dimensional convolutional layers and 1 maximum pooling layer to transform the local area set into a feature vector; the feature vector output by the mini PointNet layer of the last collection abstraction layer is the data to be transmitted.

Further, the point cloud restoration and reconstruction module includes a point cloud feature extension part and a final point set generation part, wherein the point cloud feature extension part is used to receive and transmit key point cloud features through a multi-layer perception The dimensionality of the feature is unified by the device; and through an up-down-up extension unit, a more diverse number of point clouds and feature dimensions are generated; the final point set generation part contains two layers of multi-layer perceptrons, and passed The two multilayer perceptron layers reconstruct the expanded point cloud features into a 3D coordinate form.

In addition, the layered feature extraction of the point cloud video data and the determination of the key point cloud features in the point cloud video frame include: through the layered feature extraction module deployed to a high-performance edge server close to the input end, the point cloud video data is analyzed. Layer feature extraction to identify key point cloud features in point cloud video frames.

In addition, expanding and reconstructing the received key point cloud features, obtaining point cloud information similar to the original input point cloud, and forming the original point cloud video in terms of visual effects include: The point cloud recovery and reconstruction module of the generated confrontation network expands and reconstructs the key point cloud features, obtains point cloud information similar to the original input point cloud, and forms the original point cloud video in terms of visual effects.

According to an embodiment of the present invention, an AI-driven real-time point cloud video transmission system is provided.

As shown in Figure 2, the AI-driven real-time point cloud video transmission system according to the embodiment of the present invention includes:

The point cloud video data acquisition module 201 is used to obtain video data information using AI generating equipment, and process the video data information to obtain point cloud video data;

The key point cloud feature extraction module 203 is used to extract the layered features of the point cloud video data, determine the key point cloud features in the point cloud video frame, and transmit the key point cloud features;

The point cloud restoration and reconstruction module 205 is used for receiving key point cloud features, and expanding and reconstructing the received key point cloud features, to obtain point cloud information similar to the original input point cloud, and to form the original point cloud in visual effect video.

Wherein, when the point cloud video data acquisition module 201 acquires video data information by using an AI generating device, and processes the video data information to obtain point cloud video data, it uses multiple depth cameras at different angles to scan the 3D data that needs to be transmitted. model, and obtain the point cloud stream of each depth camera; and splicing multiple point clouds collected from multi-view cameras into a complete point cloud to obtain point cloud video data

In addition, the AI-driven real-time point cloud video transmission system also includes: a neural network training module (not shown in the figure), which is used to combine a pre-built deep neural network with a large number of 3D models of various sizes used offline The training set is trained, and multiple candidate neural network models are obtained through training; the neural network deployment module (not shown in the figure) is used to split each trained neural network model into a hierarchical feature extraction module and a GAN-based The point cloud restoration and reconstruction module; and respectively deployed to high-performance edge servers and user terminal devices close to the input; neural network matching module (not shown in the figure), used to combine the layered feature extraction module and the generation confrontation network-based When the point cloud recovery and reconstruction module is deployed to the high-performance edge server and user terminal equipment, an adaptive matcher is deployed to prompt the adaptive matcher to adaptively match the real-time point cloud video frame that satisfies the current network according to the real-time monitored network bandwidth changes Feature extraction and reconstruction of restored neural network models.

The hierarchical feature extraction module contains three series of set abstraction levels for hierarchical feature learning to capture the local structure of the original point cloud; the set abstraction layer is composed of three basic layers, including sampling layer, grouping layer and mini PointNet layer;

Among them, the sampling layer uses the farthest point sampling technique to select a subset from the output of the previous layer, and each point in the subset represents the center of a local area; the grouping layer finds n nearest neighbors around the center of the local area Points are combined to construct a local area set; the mini PointNet layer uses 3 two-dimensional convolutional layers and 1 maximum pooling layer to transform the local area set into a feature vector; the feature vector output by the mini PointNet layer of the last collection abstraction layer is the data to be transmitted.

The point cloud restoration and reconstruction module includes a point cloud feature extension part and a final point set generation part, wherein the point cloud feature extension part is used to unify the key point cloud features received and transmitted through a multi-layer perceptron The dimension of the feature; and through an up-down-up expansion unit, a more diverse number of point clouds and feature dimensions are generated; the final point set generation part contains two multi-layer perceptron layers, which are used to pass the two A multi-layer perceptron layer reconstructs the expanded point cloud features into a three-dimensional coordinate form.

In addition, when the key point cloud feature extraction module 203 performs layered feature extraction on point cloud video data and determines key point cloud features in point cloud video frames, the layered feature deployed to a high-performance edge server close to the input end The extraction module performs layered feature extraction on the point cloud video data, and determines the key point cloud features in the point cloud video frame.

In addition, when the point cloud recovery and reconstruction module 205 expands and reconstructs the received key point cloud features, obtains point cloud information similar to the original input point cloud, and forms an original point cloud video with visual effects, it deploys to The point cloud recovery and reconstruction module based on the generative confrontation network on the user terminal device near the input end expands and reconstructs the key point cloud features, obtains point cloud information similar to the original input point cloud, and forms the original point cloud in visual effect video.

In order to facilitate a clearer understanding of the above-mentioned technical solution of the present invention, the above-mentioned technical solution of the present invention will be described in detail below from the perspective of working principles.

Fig. 3 is a principle schematic diagram of the present invention, as can be seen from Fig. 3, method of the present invention is as follows:

(1) Multi-view camera. The system uses multiple depth cameras placed at different angles to capture the original point cloud, and uses a USB cable for preprocessing, and synchronizes the point cloud stream of each camera to a high-performance edge server for stitching.

(2) Point cloud feature extraction and point cloud information restoration and reconstruction. The key point cloud feature extraction is the layered feature extraction module provided by the present invention, which extracts the key features of the spliced point cloud; the point cloud information restoration and reconstruction based on the generative confrontation network is the point cloud reconstruction recovery module provided by the present invention, which will receive The point cloud feature recovery reconstruction.

(3) Adaptive matcher. It is used to perceive the network status of the connected terminal and select the optimal transmission reasoning model to maintain the smooth operation of communication and improve user experience.

(4) Base station. The system can provide real-time point cloud transmission under the current network environment, and use existing base stations to wirelessly transmit key point cloud features to various terminals.

(5) User terminal. The system can be applied to various terminals and has a wide range of application scenarios. For example, real-time holographic communication is realized on smartphones. A more immersive experience such as using AR glasses or AR headsets to render point clouds, users can interact in the point cloud video immersively.

Figure 4 is a schematic diagram of the structural design of the deep neural network model. It can be seen from Figure 4 that the deep neural network model structure includes a layered feature extraction module and a point cloud restoration and reconstruction module, specifically including:

(1) Hierarchical feature extraction module. It is used to learn the input point cloud and extract the features of some key point clouds for transmission. Specifically, this module performs hierarchical feature learning based on multiple sets of abstraction levels to capture the local structure of raw point clouds. Among them, each collection abstraction layer consists of three basic layers, including sampling layer, grouping layer and mini-PointNet layer.

The sampling layer is used to select a subset from the output of the previous layer to represent the center of the local area; the grouping layer is used to find n nearest neighbors around the center to construct a local area set, and the mini PointNet is used to use 3 two-dimensional Convolutional layers and 1 max pooling layer transform the set of local regions into feature vectors.

(2) Point cloud restoration and reconstruction module. It is used to restore and reconstruct the key point cloud features received by the receiving end. Specifically, this module uses part of the generation idea in the confrontational network, which has fewer parameters and calculations than the entire generative confrontational network, and is easy to deploy on resource-constrained terminals. It includes a point cloud feature extension part and a final point set generation part.

The point cloud feature extension part first receives the transmitted point cloud feature matrix, and unifies the dimension of the feature through a multi-layer perceptron layer. Then through an up-down-up extension unit, a more diverse number of point clouds and feature dimensions are generated.

In the final point set generation part, the expanded features are reconstructed into a three-dimensional coordinate form through two multi-layer perceptron layers.

In actual application, when the point cloud feature extraction is performed on the original data through the layered feature extraction module in the deep neural network, the specific implementation scheme can be as follows:

(1) Training stage

a. On the surface of each point cloud model in the training set, 200 points are randomly selected as the seed coordinates, centered on each seed coordinate, and the farthest point sampling technology is used to find 256 points around it, so that these 256 points The constituted area occupies approximately 5% of the model surface, the seed coordinates and these 256 points are defined as a fragment, and the point set coordinates within the fragment are normalized into a unit sphere. In this embodiment, each sample input into the neural network is a fragment, which includes 256 points, each point has three-dimensional coordinates, and the input can be expressed as (256, 3).

b. Input the input samples of (256, 3) to the sampling layer. The specific steps are: use the farthest point sampling technology to select 128 points to obtain a sparse point set, expressed as (128, 3). The reason for choosing to use the farthest point sampling technique is that it can better cover the entire point set compared with random sampling. The number of specifically selected center points is specified manually, and is specified as 128 in this embodiment.

c. Input the information of the sparse point set (128, 3) into the grouping layer. The specific steps are, centering on 128 points, use the ball query method to generate 128 local areas, each area contains 32 points, and the radius of the ball is is 0.2, and the grouping feature is obtained, expressed as (128, 32, 3). The number of points in each region and the radius of the ball are specified manually, and are specified as 32 and 0.2 in this embodiment. This step can also be implemented by the K-nearest neighbor method, both of which have little influence on the result.

d. Input the information of the grouping features (128, 32, 3) into the mini PointNet layer. The specific steps are, the grouping features (128, 32, 3) have passed through three two-dimensional convolutional layers and a maximum pooling layer successively, and output Hierarchical feature information (128, 64). Wherein, in this embodiment, the number of output channels of the three two-dimensional convolutional layers is 64, 64, and 64, the size of the convolution kernel is 1×1, the step size is 1×1, and there is no padding.

e. Input the layered feature information (128, 64) into the sampling layer, the specific steps are: use the farthest point sampling technology, select 64 points, and output the sparse point set feature (64, 64). The number of specifically selected center points is specified manually, and is specified as 64 in this embodiment.

f. Input the information of the sparse point set features (64, 64) into the grouping layer. The specific steps are: centering on 64 points, use the ball query method to generate 64 local areas, each area contains 64 points, The radius of the ball is 0.3, and the grouping feature is obtained, expressed as (64, 64, 64). The number of points in each region and the radius of the ball are specified manually, and are specified as 64 and 0.3 in this embodiment. This step can also be implemented by the K-nearest neighbor method.

g. Input the information of the grouping features (64, 64, 64) into the mini PointNet layer. The specific steps are: the grouping features (64, 64, 64) successively pass through three two-dimensional convolutional layers and a maximum pooling layer, and output Hierarchical feature information (64, 32). Wherein, in this embodiment, the number of output channels of the three two-dimensional convolutional layers is 64, 64, and 32, the size of the convolution kernel is 1×1, the step size is 1×1, and there is no padding.

h. Input the layered feature information (64, 32) into the sampling layer again. The specific steps are: use the farthest point sampling technology to select N points to obtain the sparse point set feature (N, 32). The number N of specifically selected center points is a variable.

i. Input the information of sparse point set features (N, 32) to the grouping layer. The specific steps are, centering on N points, use the ball query method to generate N local areas, each area contains 64 points, and the radius is 0.4, the grouping feature is obtained, expressed as (N, 64, 32). The number of points in each region and the radius of the ball are specified manually, and are specified as 64 and 0.3 in this embodiment. N is a variable. This step can also be implemented by the K-nearest neighbor method.

j. Input the information of the grouping features (N, 64, 32) into the mini PointNet layer. The specific steps are as follows: the grouping features (N, 64, 32) successively pass through three two-dimensional convolutional layers and a maximum pooling layer to obtain Hierarchical point cloud feature information (N, M). Wherein, in this embodiment, the number of output channels of the three two-dimensional convolutional layers is 32, 32, M, the size of the convolution kernel is 1×1, the step size is 1×1, and there is no padding.

For the hierarchical feature extraction module, the sampling layer, grouping layer and mini-PointNet layer are a set abstraction level, steps b-d are set abstraction level 1, e-g are set abstraction level 2, and h-j are set abstraction level 3. The number of collection abstraction levels is manually specified, and it is specified as 3 in this embodiment, and the output of the last abstraction level is point cloud features (N, M). Input all the training samples into the deep neural network, after steps b-j, forward propagation and back propagation, calculate the loss function, update the weight of the network, and train the neural network model. By setting N and M to different combinations, several candidate models can be trained.

(2) Reasoning stage

a. Randomly select several points on the surface of the target object in the video frame to be transmitted as the seed coordinates, and use the farthest point sampling technology to find 256 points around it with each seed coordinate as the center to form a fragment , and normalize the point set coordinates within the fragment into a unit sphere. Wherein the number of seeds is specified manually, and in the present invention, it is specified as the number of target point clouds divided by 256.

b. According to network fluctuations, select the optimal reasoning model.

c. Input the optimal inference model in units of fragments, perform forward inference, and obtain the final point cloud features that need to be transmitted.

In actual use, when the point cloud information is reconstructed for point cloud features through the point cloud restoration and reconstruction module, the specific implementation plan can be as follows:

(1) Training stage

a. Point cloud features (N, M), after a two-dimensional convolutional layer, a uniform dimension point cloud feature (N, 128) is obtained. The function of this layer makes it possible to output the number of features of the same dimension even through key point cloud features compressed by different inference models. Among them, the number of output channels of the two-dimensional convolutional layer is 128, the convolution kernel size is 1×1, the step size is 1×1, and there is no padding.

，（N，128），使用一个上采样操作增大点云特征数量，得到上采样后的点云特征

，表示为（256，128）。其中，上采样操作的具体步骤为：b. Point cloud features of uniform dimension, expressed as

, (N, 128), use an upsampling operation to increase the number of point cloud features, and get the upsampled point cloud features

, expressed as (256, 128). Among them, the specific steps of the upsampling operation are:

After copying the feature (N, 128) r times to get (rN, 128), r is the sampling magnification, which is equal to 256/N.

Using the 2D grid mechanism, a unique two-dimensional vector is generated for each feature, and this vector is appended to the feature vector of each corresponding point in the same feature to obtain the feature (rN, 128+2).

。其中两个二维卷积层的输出通道为256，128，卷积核大小为1×1，步长为1×1，无填充。Generate upsampled point cloud features using a self-attention unit and two 2D convolutional layers

. The output channels of the two two-dimensional convolutional layers are 256, 128, the convolution kernel size is 1×1, the step size is 1×1, and there is no padding.

（256，128），使用下采样操作得到与

规模相同的点云特征

，表示为（N，128）。其中，下采样操作的具体步骤为：c. For point cloud features

(256, 128), using the downsampling operation to get the same as

point cloud features at the same scale

, expressed as (N, 128). Among them, the specific steps of the downsampling operation are:

通过简单地移动行列，改造成（N，r×128）的特征，point cloud features

By simply moving the ranks and columns, transformed into (N, r×128) features,

。其中两个二维卷积层的输出通道为256，128，卷积核大小为1×1，步长为1×1，无填充。Generate point cloud features using two 2D convolutional layers

. The output channels of the two two-dimensional convolutional layers are 256, 128, the convolution kernel size is 1×1, the step size is 1×1, and there is no padding.

与点云特征

相减，得到点云特征

，维度为（N，128）。d. Point cloud features

and point cloud features

Subtract to get point cloud features

, with dimension (N, 128).

，（N，128）使用相同的上采样操作得到上采样点云特征

，维度为（256，128）。e. For point cloud features

, (N, 128) use the same upsampling operation to get upsampled point cloud features

, the dimension is (256, 128).

与点云特征

相加，得到点云特征

，维度为（256，128）。f. Point cloud features

and point cloud features

Add to get point cloud features

, the dimension is (256, 128).

，维度为（256，128）经过两个二维卷积层，进行坐标重构，得到重建点云，维度为（256，3）。其中两个二维卷积层的输出通道为64，3，卷积核大小为1×1，步长为1×1，无填充。g. Point cloud features

, the dimension is (256, 128) After two two-dimensional convolutional layers, the coordinates are reconstructed, and the reconstructed point cloud is obtained, and the dimension is (256, 3). The output channels of the two two-dimensional convolutional layers are 64, 3, the convolution kernel size is 1×1, the stride is 1×1, and there is no padding.

For the point cloud restoration and reconstruction module, steps c-f are the up-down-up expansion unit, and step g is the final point set generation part. Input all the point cloud feature samples obtained from the transmission into the deep neural network, after steps a-g, forward propagation and back propagation, calculate the loss function, update the weight of the network, and train the neural network model.

(2) Reasoning stage

a. According to network fluctuations, select the optimal inference model.

b. Input all the transmitted point cloud feature samples into the deep neural network, and perform forward reasoning through steps A-G to obtain the restored and reconstructed fragmented point cloud information.

c. Add all fragment reconstruction information together, use the farthest point sampling technology, select the same number of points as the original input point cloud, and reconstruct the final point cloud.

With the help of the above-mentioned technical solution of the present invention, the present invention extracts features from the original point cloud video stream that needs to be transmitted, only transmits some key point cloud features, and finally restores and reconstructs at the receiving end, so that what is visually transmitted is the original point cloud video effects.

At runtime, the present invention is completely driven by AI, that is, using a deep neural network to automatically achieve feature extraction and reconstruction goals, directly using the original point cloud coordinates as input, and does not require other operations, including information preprocessing, artificially obtained The geometric distribution of the point cloud, the attribute importance information, the selection of encoding and decoding methods, etc., only need to train the deep neural network and then deploy it to the actual application, and the separation of the deep neural network in the present invention is feature extraction and Rebuild both processes.

The system of the present invention is a neural network for end-to-end training, including feature extraction and reconstruction, that is, training can be performed as long as there is data input, it is more intelligent, and it does not need to pay attention to internal specific operations, and can also select a suitable reasoning model according to network conditions , to give users a better experience.

At the same time, the present invention can achieve a high compression ratio of 30.72 times under the condition of acceptable loss of precision, and realize real-time transmission under the existing environment below 5G, greatly reducing the amount of transmitted data.

In summary, the present invention can significantly reduce the transmission volume and energy consumption of point cloud video streams. It avoids the cumbersome multiple processing in the traditional transmission scheme, greatly reduces the amount of data transmission, and makes it more suitable for the existing network environment. The present invention also considers the dynamics and instability of the network environment and incorporates it into the end-to-end In the design and training of terminal network, an adaptive transmission control algorithm is provided to balance transmission delay and reconstruction accuracy

The above descriptions are only preferred embodiments of the present invention, and are not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention shall be included in the scope of the present invention. within the scope of protection.

Claims (8)

1. An AI-driven real-time point cloud video transmission method is characterized by comprising the following steps:

acquiring video data information by using AI (Artificial intelligence) generating equipment, and processing the video data information to obtain point cloud video data;

extracting hierarchical features of the point cloud video data, determining key point cloud features in the point cloud video frame, and transmitting the key point cloud features;

the method comprises the steps of receiving key point cloud characteristics, expanding and reconstructing the received key point cloud characteristics to obtain point cloud information similar to original input point cloud and form an original point cloud video with a visual effect;

training a pre-built deep neural network on line by using a training set consisting of a large number of 3D models with various scales to obtain a plurality of candidate neural network models;

each trained neural network model is divided into a hierarchical feature extraction module and a point cloud recovery reconstruction module based on the generated countermeasure network; respectively deploying the high-performance edge server and the user terminal equipment close to the input end;

when a hierarchical feature extraction module and a point cloud recovery reconstruction module based on a generated countermeasure network are deployed to a high-performance edge server and user terminal equipment, a self-adaptive matcher is deployed to enable the self-adaptive matcher to adaptively match a neural network model meeting the requirements of real-time point cloud video frame feature extraction and reconstruction recovery of a current network according to network bandwidth change monitored in real time.

2. The AI-driven real-time point cloud video transmission method according to claim 1, wherein the hierarchical feature extraction module includes three sets of abstraction levels connected in series for performing hierarchical feature learning to capture a local structure of an original point cloud; the set abstraction layer consists of three basic layers including a sampling layer, a grouping layer and a mini PointNet layer; wherein,

a sampling layer for selecting a subset from the output of the previous layer using a farthest point sampling technique, each point in the subset representing the center of a local area;

grouping layers, finding n nearest neighbor points around the center of a local area, and combining to construct a local area set;

the mini PointNet layer converts a local region set into a characteristic vector by adopting 3 two-dimensional convolution layers and 1 maximum pooling layer;

and the eigenvector output by the mini PointNet layer of the last set abstraction layer is the data to be transmitted.

3. The AI-driven real-time point cloud video transmission method according to claim 2, wherein the point cloud restoration reconstruction module comprises a point cloud feature expansion portion and a final point set generation portion, wherein,

the point cloud feature expansion part is used for unifying the dimensions of the features through a multilayer perceptron when the transmitted key point cloud features are received; and generating more diversified point cloud number and characteristic dimension through an upward-downward-upward expansion unit;

and the final point set generating part comprises two multilayer sensor layers, and reconstructs the expanded point cloud characteristics into a three-dimensional coordinate form through the two multilayer sensor layers.

4. The AI-driven real-time point cloud video transmission method of claim 3,

the method comprises the following steps of acquiring video data information by using AI (Artificial intelligence) generating equipment, and processing the video data information to obtain point cloud video data:

scanning a 3D model to be transmitted by using a plurality of depth cameras with different angles, and acquiring a point cloud flow of each depth camera; splicing a plurality of point clouds collected by the multi-view camera into a complete point cloud to obtain point cloud video data;

extracting hierarchical features of the point cloud video data, and determining key point cloud features in the point cloud video frame comprises the following steps:

performing hierarchical feature extraction on point cloud video data through a hierarchical feature extraction module deployed to a high-performance edge server close to an input end to determine key point cloud features in a point cloud video frame;

expanding and reconstructing the received key point cloud characteristics to obtain point cloud information similar to the original input point cloud, and forming an original point cloud video with a visual effect comprises the following steps:

and expanding and reconstructing key point cloud characteristics through a point cloud recovery reconstruction module which is deployed on user terminal equipment close to the input end and is based on the generation countermeasure network, so that point cloud information similar to the original input point cloud is obtained, and an original point cloud video on the visual effect is formed.

5. An AI-driven real-time point cloud video transmission system, comprising:

the point cloud video data acquisition module is used for acquiring video data information by using AI (artificial intelligence) generation equipment and processing the video data information to obtain point cloud video data;

the key point cloud feature extraction module is used for extracting hierarchical features of point cloud video data, determining key point cloud features in a point cloud video frame and transmitting the key point cloud features;

the point cloud recovery and reconstruction module is used for receiving the key point cloud characteristics, expanding and reconstructing the received key point cloud characteristics to obtain point cloud information similar to the original input point cloud and form an original point cloud video on the visual effect;

the neural network training module is used for training a pre-built deep neural network on line by using a training set consisting of a large number of 3D models with various scales to obtain a plurality of candidate neural network models;

the neural network deployment module is used for splitting each trained neural network model into a hierarchical feature extraction module and a point cloud recovery reconstruction module based on the generated countermeasure network; respectively deploying the high-performance edge server and the user terminal equipment close to the input end;

and the neural network matching module is used for deploying the hierarchical feature extraction module and the point cloud restoration and reconstruction module based on the generated countermeasure network to the high-performance edge server and the user terminal equipment, and deploying the self-adaptive matcher to enable the self-adaptive matcher to adaptively match the neural network model meeting the requirements of the real-time point cloud video frame feature extraction and reconstruction and restoration of the current network according to the network bandwidth change monitored in real time.

6. The AI-driven real-time point cloud video transmission system of claim 5, wherein the hierarchical feature extraction module comprises three sets of abstraction levels connected in series for performing hierarchical feature learning to capture the local structure of the original point cloud; the set abstraction layer consists of three basic layers including a sampling layer, a grouping layer and a mini PointNet layer; wherein,

a sampling layer for selecting a subset from the output of the previous layer using a farthest point sampling technique, each point in the subset representing the center of a local area;

grouping layers, finding n nearest neighbor points around the center of a local area, and combining to construct a local area set;

the mini PointNet layer converts a local region set into a characteristic vector by adopting 3 two-dimensional convolution layers and 1 maximum pooling layer;

and the eigenvector output by the mini PointNet layer of the last collection abstraction layer is the data to be transmitted.

7. The AI-driven real-time point cloud video transmission system of claim 6, wherein the point cloud restoration reconstruction module comprises a point cloud feature expansion portion and a final point set generation portion, wherein,

the point cloud feature expansion part is used for unifying the dimensions of the features through a multilayer perceptron when the transmitted key point cloud features are received; and generating more diversified point cloud number and characteristic dimension through an upward-downward-upward expansion unit;

and the final point set generating part comprises two multilayer sensor layers and is used for reconstructing the expanded point cloud characteristics into a three-dimensional coordinate form through the two multilayer sensor layers.

8. The AI-driven real-time point cloud video transmission system of claim 7,

the point cloud video data acquisition module is used for acquiring video data information by using AI (artificial intelligence) generation equipment, processing the video data information and scanning a 3D (three-dimensional) model to be transmitted by using a plurality of depth cameras at different angles when point cloud video data are obtained, and acquiring a point cloud stream of each depth camera; splicing a plurality of point clouds collected by a multi-view camera into a complete point cloud to obtain point cloud video data;

the key point cloud feature extraction module extracts the hierarchical features of the point cloud video data through the hierarchical feature extraction module arranged to the high-performance edge server close to the input end when the hierarchical features of the point cloud video data are extracted and the key point cloud features in the point cloud video frame are determined;

the point cloud recovery reconstruction module expands and reconstructs the key point cloud characteristics to obtain point cloud information similar to the original input point cloud, and when an original point cloud video on the visual effect is formed, the point cloud recovery reconstruction module which is deployed on user terminal equipment close to the input end and is based on the generation countermeasure network expands and reconstructs the key point cloud characteristics to obtain point cloud information similar to the original input point cloud, and the original point cloud video on the visual effect is formed.

Images