CN114554220B - A Fixed Scene Video Overlimit Compression and Decoding Method Based on Abstract Features - Google Patents

Abstract

The invention discloses a fixed scene video overrun compression and decoding method based on abstract features, which comprises the following steps: 1) Extracting a background image of an original video by adopting a background modeling method and performing compression coding; 2) Extracting abstract features from a foreground target by adopting a foreground target extraction module comprising algorithms such as instance segmentation, key point detection and the like; 3) Taking a snapshot of the foreground object and compressing the snapshot; 4) Compressing video background compression data, foreground object abstract features and snapshot compression data; 5) Decompressing the video compression data by pre-decoding; 6) Inputting the abstract features of the foreground target and the snapshot of the foreground target into a trained generator for generating an countermeasure network; 7) Fusing the generated foreground target decoding image of each frame with the background image; 8) Reconstructing the fused video frames to obtain a decoded video. The invention has extremely high compression ratio aiming at the video of the fixed scene, remarkably improves the storage efficiency and prolongs the storage time of the monitoring video.

Description

A Fixed Scene Video Overlimit Compression and Decoding Method Based on Abstract Features

technical field

The invention relates to the technical field of deep learning of computer vision, in particular to an abstract feature-based overlimit compression and decoding method for fixed-scene video.

Background technique

The common compression coding of video data is mainly based on low-level features such as texture, edge, and image block movement to remove redundant information, and does not fully consider the high-level abstract features contained in video content. The vigorous development of deep learning in the field of computer vision has brought technical feasibility to high-level abstract understanding of images and videos. With the support of big data and high-performance parallel computing, deep convolutional neural networks have brought revolutionary changes to the extraction of high-level features such as images and videos. Unlike traditional image feature extraction methods based on manual design, convolutional neural networks can automatically extract high-level features with stronger expressive capabilities in big data. These high-level features play a crucial role in image understanding and video structuring. With the help of the high-level feature extraction capability of the deep convolutional neural network model, on the basis of widely available video big data, extracting more expressive high-level abstract feature information in the video and removing a large number of abstract redundancy in the video will be able to Greatly improve video compression performance, reduce storage space and transmission bandwidth, and bring new ideas for better persistent video storage and transmission.

Therefore, how to provide a video compression method that improves the compression ratio by extracting high-level abstract feature information from the video is an urgent problem to be solved by those skilled in the art.

Contents of the invention

(1) Solved technical problems

Aiming at the deficiencies of the prior art, the present invention provides a fixed-scene video overcompression and decoding method based on abstract features. By extracting high-level abstract feature information from the video for storage, the storage space is greatly reduced to solve the above-mentioned technical problems. question.

(2) Technical solutions

In order to achieve the above object, the present invention provides the following technical solution: a fixed-scene video overcompression and decoding method based on abstract features, which includes an encoder and a decoder. This method comprises the following steps:

1. Video compression.

Disassemble the original video into image frames and send them to the encoder for processing. The encoder consists of two modules: background modeling and foreground object extraction.

The background modeling module uses a background modeling algorithm based on a mixed Gaussian model to perform foreground subtraction on each frame of video to obtain a background image. After all the video frames are processed, the multi-frame background images are combined to obtain a single background image, and then discrete cosine transform, quantization and entropy coding are performed to obtain video background compressed data.

The foreground target extraction module is composed of an instance segmentation model and a key point detection model based on a convolutional neural network, and performs object instance segmentation and key point detection on image frames to obtain abstract features of foreground targets. The abstract feature of the foreground object includes the shape feature and key point feature of the foreground object.

After processing all the video frames, use the method based on the IOU threshold of the target detection frame to perform inter-frame target matching to obtain the corresponding relationship between foreground targets between frames, and then extract a snapshot for each foreground target. The algorithm steps for extracting snapshots are as follows: For each multi-frame shape feature of the foreground object, only retain the shape feature of the frame with the highest confidence output by the instance segmentation model, and use the shape feature to extract the image of the foreground object from the original video frame , the snapshot of the foreground object is obtained, and the snapshot is subjected to discrete cosine transform, quantization and entropy coding to obtain the compressed data of the foreground object snapshot. The purpose of extracting snapshots is to preserve the detailed features of the foreground object, such as color texture and so on.

Finally, the abstract feature of the foreground object, the snapshot compressed data and the background compressed data are compressed and packaged to obtain the video compressed data. Video compression is complete.

In the encoder, the background modeling module encodes the background of the original video into a single compressed image, thereby realizing the removal of background redundant information; the foreground target extraction module extracts the abstract features and snapshots of the foreground target, The goal is to save only multi-frame abstract features and single-frame snapshot compressed data, so as to achieve the removal of foreground redundant information. Compared with traditional video compression coding, the coding method of the present invention greatly reduces the data capacity to be saved, thereby realizing over-limit compression.

2. Video pre-decoding

When a user needs to watch a video, first perform video pre-decoding. Decompress the video compression data compressed and packaged by the encoder at the end, and restore the abstract features of the foreground object, the snapshot of the foreground object and the video background image.

3. Video decoding.

The decoder of the present invention is composed of a convolutional neural network model based on a generated confrontation network architecture, which includes a generator and a discriminator. The input of the generator is the snapshot of the foreground target and the abstract features of the foreground target, and the output is the decoded image of the foreground target; the discriminator is responsible for assisting the generator to improve the quality of the generated image during the training of the generator, and the input is the decoded image of the foreground target generated by the generator and the real The foreground target image in the video frame is output as a value between 0 and 1, which means that the discriminator judges that the input image may be a generated image (0) or a real image (1).

(1) Decoder training process

The objective function of the training process is: L=L _GAN +L _L1 +L _VGG

in:

To generate the adversarial loss, _IS and _It are the foreground target snapshot and the real foreground target image to be generated respectively, _RS and R _t are the response maps generated according to the key points of the _IS image and _It image, so as to be input to the generation device. Decoded image for the foreground object generated by the generator, z is random noise.

in:

For the L1 loss, the minimum absolute error between the generator-generated image and the real image is computed.

in:

For the perceptual loss, by inputting the decoded image of the foreground object generated by the generator and the real foreground object image into the public VGG pre-trained network model, the least square difference between the two in the deep feature map is calculated.

After training, the decoder only needs to keep the generator.

(2) Decoder decoding process

The multi-frame abstract features and snapshots of each foreground object are read and fed to the generator in the decoder. The generator model obtains information such as the pose and skeleton of the target from the abstract features of multiple frames, and obtains information such as the color and texture of the target from the snapshot, and fuses the above information to generate a decoded image of the foreground target.

The video background image is read, and all generated foreground target decoded images are fused with the background image to obtain the reconstructed video frame. Merge and reconstruct all the reconstructed video frames to obtain the decoded video.

(3) Beneficial effects

Compared with the prior art, the present invention provides a fixed-scene video overcompression and decoding method based on abstract features, which has the following beneficial effects: the abstract feature-based fixed-scene video over-limit compression and decoding method is aimed at fixed scene Video has a high compression ratio, which greatly saves storage space resources. Experiments have proved that for fixed-scene videos with different lengths and different numbers of targets, the compressed data capacity stored by this method is only 1/40 to 1/3 of the video encoded by H264, achieving a high compression ratio beyond traditional video compression coding. The invention can be applied to various intelligent monitoring systems, significantly prolonging the storage time of monitoring videos, and the abstract features of objects extracted during the compression process can be used for abnormal behavior detection, traffic flow monitoring and the like.

Description of drawings

FIG. 1 is a frame diagram of an abstract feature-based fixed-scene video overcompression and decoding method proposed by the present invention.

Detailed ways

The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some, not all, embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the present invention.

The overall structure of the fixed scene video overcompression and decoding method proposed by the present invention is shown in FIG. 1 , which mainly consists of two parts, an encoder and a decoder. When compressing, the original video is input to the video encoder to obtain video compressed data; when decoding, the video compressed data is firstly pre-decoded, and then input to the decoder to generate decoded video.

1. Video compression steps

Step 1) The convolutional neural network is initialized, and the instance segmentation model and the key point detection model are loaded into the GPU.

Step 2) Mixed Gaussian background model initialization.

Step 3) read the i-th frame of the original video.

Step 4) Input the video frame into the mixed Gaussian background model, perform matching and model weight update, and obtain the Gaussian background modeling result bg _i .

Step 5) Use the instance segmentation model to perform instance segmentation on the current video frame, and obtain the instance segmentation results of m foreground objects S _i ={box _j , mask _j |j=1,2,...,m}; where box _j is the video The rectangular detection frame (x_min, y_min, x_max, y_max) of the jth foreground object in the frame, mask _j is the mask of the jth foreground object in the video frame, and the mask is a binary image whose length and width are equal to the video frame , the area corresponding to the target appearance is 1, and the other areas are 0. In the subsequent steps, the detection frame of the foreground object is equivalent to box _j in this step, the shape feature of the foreground object is equivalent to mask _j in this step, and the spatiotemporal information of the foreground object is equal to the current frame number i plus box _j in this step (i, x_min, y_min, x_max, y_max).

Step 6) Use the key point detection model to perform key point detection on each detected foreground object, and obtain the key point coordinates (x0, y0, x1, y1...) of the foreground object.

Step 7) Repeat steps 3 to 6 until all video frames are processed.

Step 8) Read the spatio-temporal information (i, x_min, y_min, x_max, y_max) of the foreground object detected by the instance segmentation model in step 5). Using the method based on the IOU threshold of the target detection frame for inter-frame foreground target matching, multiple matching lists are obtained, and each list contains multi-frame spatio-temporal information of each foreground target, arranged in time order. For example, p targets appear in the video, and these p targets appear in q ₁ , q ₂ ... q _p frames respectively, then p matching lists with lengths q ₁ , q ₂ ... q _p respectively are obtained, and each Item is the spatio-temporal information of the target in different frames.

Step 9) Take a snapshot for each foreground object, the steps are: according to the foreground object spatio-temporal information of each matching list in step 8), read the multi-frame shape features of each foreground object, and then only keep the confidence output of the instance segmentation model The shape feature of a frame with the highest degree is used, and the image of the foreground object is extracted from the original video frame by using the shape feature, and a snapshot of the foreground object is obtained. Perform discrete cosine transform, quantization, and entropy coding on the snapshot to obtain the compressed data of the foreground target snapshot, and use the spatiotemporal information of the snapshot as the file name (i _s , x_min _s , y_min _s , x_max _s , y_max _s .jpg) to save.

Step 10) Combine the snapshot file name (i _s , x_min _s , y_min _s , x_max _s , y_ max _s .jpg) of each foreground target with the multi-frame spatio-temporal information (i, x_min, y_min, x_max, y_max), multi-frame The key point coordinates (x0, y0, x1, y1...) are merged and written into a csv file for saving, which is an abstract feature file of the foreground target. So far, for each foreground object, only single-frame snapshot + multi-frame object abstract features are preserved.

Step 11) Find the union bg={bg _{1 U} bg ₂ U bg ₃ U bg _n } to obtain a complete video background image, and then perform discrete cosine transform, quantization and entropy coding to obtain video background compressed data.

Step 12) compress and package the foreground object snapshot compressed data obtained in step 9), the foreground object abstract feature file obtained in step 10) and the video background compressed data obtained in step 11) as a whole to obtain video compressed data.

2. Video decoding steps

Step 1) pre-decoding, decompressing the compressed video data, recovering the abstract features of the foreground object, the snapshot of the foreground object and the video background image.

Step 2) Initialize the convolutional neural network, and load the trained generator network model into the GPU.

Step 3) Read the abstract feature file of the foreground object, and obtain the snapshot file name of the foreground object and its corresponding multi-frame abstract feature.

Step 4) Input the snapshot of the foreground object, the abstract features of the snapshot of the foreground object, and the abstract features of the decoded image of the foreground object to be generated into the generator model to generate the decoded image of the foreground object.

Step 5) Repeat step 3 to step 4 until the abstract feature file of the foreground object is read and all the foreground objects are decoded.

Step 6) read the video background image.

Step 7) Fusion the decoded image of the foreground target with the video background image of each frame to generate a reconstructed video frame until all frames of the video are reconstructed, and merge all the reconstructed video frames to obtain the decoded video.

It should be noted that in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is a relationship between these entities or operations. There is no such actual relationship or order between them. Furthermore, the term "comprises", "comprises" or any other variation thereof is intended to cover a non-exclusive inclusion such that a process, method, article, or apparatus comprising a set of elements includes not only those elements, but also includes elements not expressly listed. other elements of or also include elements inherent in such a process, method, article, or device.

Although the embodiments of the present invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and variants, the scope of the invention is defined by the appended claims and their equivalents.

Claims (1)

1. a fixed scene video overcompression and decoding method based on abstract features, is characterized in that comprising the following steps: Use the encoder to compress the fixed scene video data; Using a decoder to decode the compressed video data to obtain a decoded video; In the encoder, a background modeling method is used to extract the background image of the original video, and then the extracted background image is compressed and encoded to obtain video background compressed data; In the encoder, the foreground object extraction module including the object instance segmentation and key point detection algorithm is used to extract the features of the foreground object in the video frame to obtain the abstract feature of the foreground object, and the abstract feature of the foreground object includes the shape feature of the foreground object and key features; The encoder utilizes the foreground target snapshot extraction algorithm to extract a snapshot of the foreground target obtained by the foreground target extraction module, compress and encode the snapshot, and obtain the compressed data of the foreground target snapshot; The encoder extracts abstract features and snapshots of foreground objects, and only saves abstract features and snapshot compressed data for each foreground object in the video; For the video background, only the video background compressed data is saved, and the abstract features of the foreground target, the snapshot compressed data and the background compressed data are compressed and packaged to obtain the video compressed data; When decoding, the compressed video data is decompressed, and the abstract features of the foreground object, the snapshot of the foreground object and the video background image are recovered; In the decoder, a deep learning model based on a Generative Adversarial Network is used for video decoding; In the decoder, the abstract feature of the foreground target and the snapshot of the foreground target are input into the generator of the generative confrontation network, and the decoded image of the foreground target is reconstructed; The decoder fuses the decoded image of the foreground target with the background image of each frame in the video to obtain a reconstructed video frame, and merges and reconstructs all the reconstructed video frames to obtain a decoded video.