CN112261409A - Residual encoding method, residual decoding method, residual encoding device, residual decoding device, storage medium and electronic device - Google Patents

Abstract

The present invention provides a residual coding and decoding method and device, a storage medium and an electronic device. In the residual coding method, the residual of the reference block of the block to be coded is determined; based on the residual of the reference block of the to-be-coded block, the residual of the to-be-coded block is predicted to obtain the to-be-coded block The prediction residual of the block; the residual difference value between the prediction residual of the to-be-coded block and the actual residual of the to-be-coded block is encoded into the code stream. The invention solves the problem of how to reduce the bit rate of the coding residual in the related art, thereby achieving the effect of saving the code rate.

Description

Residual coding and decoding method and device, storage medium and electronic device

technical field

The present invention relates to the field of communications, and in particular, to a residual coding and decoding method and device, a storage medium and an electronic device.

Background technique

In recent years, with the rapid development of network communication and multimedia technologies, video content has begun to be presented to viewers in high-resolution and ultra-high-resolution formats. Compared to standard resolution videos, the higher the resolution of the video, the better the visual experience for people watching the video content. At the same time, high-resolution video also puts forward higher requirements for video coding technology. To address this challenge, the International Joint Collaborative Team on Video Coding (JCT-VC) developed a new generation of High Efficiency Video Coding (HEVC). Compared with the previous generation video coding standard H.264/MPEG-4 AVC, HEVC improves the compression efficiency by 50% and maintains the same visual quality. On September 1, 2015, Google announced the establishment of the Alliance of Open Media (AOM) with Amazon, Cisco, Intel, Microsoft, Firefox, and Netflix. AOM developed a new generation of video encoding format AV1. In December 2017, China's Audio Video Coding Standard (AVS for short) proposed a new generation of AVS3 video coding. The Joint Video Exploration Team (JVET for short) determined the name of the latest generation of video coding standards at the San Diego Conference on April 10, 2018 as Versatile Video Coding (VVC for short), and the main goal is to improve Existing HEVC, which provides higher compression performance, will be optimized for emerging applications (360° panoramic video and High Dynamic Range Imaging (HDR or HDRI) for short). VVC is expected to be standardized by 2020, and the current proposed solution has improved by more than 40% compared to HEVC.

Inter-frame prediction is the core component of the video coding standard HEVC/AV1/AVS. It uses the correlation in the video time domain to predict the pixels of the current image using the adjacent encoded image pixels in the time domain to effectively remove the video time domain. redundant purpose. The current mainstream video coding standards all use block-based motion compensation technology. The principle is to find a best matching block for each pixel block of the current image through motion estimation in the previously coded image. The image used for prediction is called a reference image, and the displacement from the reference block to the current pixel block is called a motion vector (MV for short), and the difference between the original pixel value of the current block and the pixel value of the prediction block after motion compensation of the reference block is performed. The difference between them is called residual (also called residual, residual value, residual block). Inter-frame prediction only needs to write the optimal MV, reference frame index and residual value of the coding block into the code stream and transmit it to the decoder. The decoder side finds the corresponding reference block in the reference frame according to the optimal MV and the reference frame index, and then adds the decoded residual value to restore the original pixel value of the decoded block. Inter-frame prediction mainly consumes bit rate is the coding of residual blocks, and traditional inter-frame prediction directly encodes the actual residual obtained after prediction. However, for most complex motion cases, the value of the residual block is very large, which results in a very high bit rate of the encoded residual.

Therefore, how to reduce the bit rate of the coding residual to achieve the effect of saving the code rate is an urgent problem to be solved at present.

SUMMARY OF THE INVENTION

Embodiments of the present invention provide a residual coding and decoding method and device, a storage medium, and an electronic device, so as to at least solve the problem of how to reduce the bit rate of the coding residual in the related art.

According to an embodiment of the present invention, a residual coding method is provided, comprising: determining a residual of a reference block of a block to be coded, wherein the reference block of the block to be coded includes: the block to be coded is in at least two first reference blocks in the time domain; or, at least two second reference blocks in the spatial domain of the block to be encoded; or, at least one first reference block in the time domain of the block to be encoded, and At least one second reference block of the block to be coded in the spatial domain; based on the residual of the reference block of the block to be coded, predict the residual of the block to be coded to obtain the block to be coded. Prediction residual; encode the residual difference between the predicted residual of the block to be encoded and the actual residual of the block to be encoded into a code stream.

In at least one exemplary embodiment, when the image frame where the block to be encoded is located is a P frame, the first reference block of the block to be encoded in the time domain includes: the block to be encoded is in the The optimal prediction unit block PU in the forward reference frame, denoted as the first optimal PU, and/or the optimal PU of the block to be encoded in the frame preceding the forward reference frame, denoted as the first optimal PU Two optimal PUs; or, when the image frame where the block to be encoded is located is a B frame, the first reference block of the block to be encoded in the time domain includes: the block to be encoded is in the forward direction The optimal PU in the reference frame is recorded as the third optimal PU, and/or the optimal PU of the block to be encoded in the backward reference frame is recorded as the fourth optimal PU.

In at least one exemplary embodiment, when the image frame in which the block to be encoded is located is a P frame, the first reference block of the block to be encoded in the time domain is determined in the following manner: through motion estimation Determine the first optimal PU of the block to be coded in the forward reference frame, and determine the motion vector MV of the block to be coded relative to the first optimal PU; according to the first optimal PU The position of the best PU and the MV, the second best PU is determined by motion estimation in the frame preceding the forward reference frame.

In at least one exemplary embodiment, the second reference block of the block to be encoded in the spatial domain is located in an image frame where the block to be encoded is located, and is adjacent to the block to be encoded in the spatial domain .

In at least one exemplary embodiment, the second reference block of the block to be encoded in the spatial domain includes: in the image frame where the block to be encoded is located, the left block adjacent to the block to be encoded and / or the block above.

In at least one exemplary embodiment, the reference block of the block to be encoded includes:

The corresponding residuals of the block to be encoded in the time domain are two first reference blocks that are not all zeros, and the corresponding residuals of the block to be encoded in the spatial domain are two non-all zeros. a second reference block; or,

The corresponding residual of the block to be encoded in the time domain is a first reference block that is not all zero, and the corresponding residual of the block to be encoded in the spatial domain is a first reference block that is not all zero. 2 reference blocks; or,

The corresponding residuals of the to-be-coded block in the time domain are two first reference blocks that are not all zeros, and the corresponding residuals of the to-be-coded block in the spatial domain are one of the non-all-zero residuals the second reference block; or,

The corresponding residual of the block to be coded in the time domain is a first reference block that is not all zero, and the corresponding residual of the block to be coded in the space domain is two non-all zeros. the second reference block; or,

The corresponding residuals of the block to be coded in the time domain are two first reference blocks that are not all zeros; or,

The corresponding residuals of the block to be encoded in the spatial domain are two second reference blocks that are not all zeros.

In at least one exemplary embodiment, predicting the residual of the block to be encoded based on the residual of the reference block of the block to be encoded, and obtaining the prediction residual of the block to be encoded includes: The residual of the reference block is input to the residual prediction model to obtain the prediction residual of the to-be-coded block, wherein the residual prediction model is obtained based on training samples using deep learning network training, and the training samples include: The residual of the reference block of the coding block with the known residual, and the actual residual of the coding block with the known residual; or, performing linear weighting on the residual of the reference block to obtain the The prediction residual of the block to be coded, wherein the linear weighting includes linear weighting of a single weight or linear weighting of multiple weights.

In at least one exemplary embodiment, the actual residual of the block to be encoded is a difference between a pixel value of an original image block of the block to be encoded and a pixel value of a predicted block of the block to be encoded , wherein the prediction block of the block to be coded is a block obtained after motion compensation is performed on the reference block of the coding block.

According to another embodiment of the present invention, a residual decoding method is provided. difference, wherein the reference blocks of the block to be decoded include: at least two first reference blocks of the block to be decoded in the time domain; or at least two second reference blocks of the block to be decoded in the spatial domain reference block; or, at least one first reference block in the time domain of the block to be decoded and at least one second reference block in the spatial domain of the block to be decoded; based on the residual, predict the residual of the block to be decoded to obtain the predicted residual of the to-be-decoded block; according to the predicted residual of the to-be-decoded block and the to-be-decoded The residual difference between the prediction residual of the block and the actual residual of the block to be decoded determines the actual residual of the block to be decoded.

In at least one exemplary embodiment, when the image frame in which the block to be decoded is located is a P frame, the first reference block of the block to be decoded in the time domain includes: the block to be decoded is in the The optimal prediction unit block PU in the forward reference frame is denoted as the first optimal PU, and/or the optimal PU of the block to be decoded in the frame before the forward reference frame is denoted as the first optimal PU Two optimal PUs; or, when the image frame where the block to be decoded is located is a B frame, the first reference block of the block to be decoded in the time domain includes: the block to be decoded is in the forward direction The optimal PU in the reference frame is recorded as the third optimal PU, and/or the optimal PU of the block to be decoded in the backward reference frame is recorded as the fourth optimal PU.

In at least one exemplary embodiment, when the image frame where the block to be decoded is located is a P frame, the first reference block of the block to be decoded in the time domain is determined in the following manner: according to the The MV parsed from the code stream determines the first optimal PU of the block to be decoded in the forward reference frame; according to the first optimal PU in the forward reference frame For co-located PUs in a frame, the second optimal PU is determined in a frame preceding the forward reference frame by motion estimation.

In at least one exemplary embodiment, the second reference block of the block to be decoded in the spatial domain is located in an image frame where the block to be decoded is located, and is adjacent to the block to be decoded in the spatial domain .

In at least one exemplary embodiment, the second reference block of the block to be decoded in the spatial domain includes: in the image frame where the block to be decoded is located, the left block adjacent to the block to be decoded and / or the block above.

In at least one exemplary embodiment, predicting the residual of the block to be decoded based on the residual of the reference block of the block to be decoded, and obtaining the prediction residual of the block to be decoded includes: The residual prediction model of the reference block is input to the residual prediction model to obtain the prediction residual of the block to be decoded, wherein the residual prediction model is the same as the residual prediction model at the encoder end; Linear weighting is performed on the residual to obtain the prediction residual of the block to be decoded, wherein the linear weighting is the same as the linear weighting at the encoder side.

In at least one exemplary embodiment, between the prediction residual of the block to be decoded and the prediction residual of the block to be decoded parsed from the code stream and the actual residual of the block to be decoded After determining the actual residual difference of the block to be decoded, the method further includes: adding the actual residual difference on the basis of the predicted block of the block to be decoded, and recovering the block to be decoded the original image block.

According to yet another embodiment of the present invention, a residual coding apparatus is provided, comprising: an encoder-side reference residual determination module, configured to determine a residual of a reference block of a block to be coded, wherein the The reference blocks include: at least two first reference blocks of the block to be encoded in the time domain; or, at least two second reference blocks of the block to be encoded in the spatial domain; or, the block to be encoded at least one first reference block in the time domain and at least one second reference block in the spatial domain of the block to be coded; a residual prediction module at the encoder end, set to be based on the reference block of the block to be coded residual, predicting the residual of the to-be-coded block to obtain the predicted residual of the to-be-coded block; a residual coding module, set to combine the predicted residual of the to-be-coded block with the to-be-coded block The residual difference between the actual residuals of the block is encoded into the code stream.

According to yet another embodiment of the present invention, a residual decoding apparatus is provided, comprising: a reference residual determination module at the decoder side, configured to determine a prediction block of a block to be decoded based on a motion vector MV parsed from a code stream, and Obtain the residual of the reference block of the block to be decoded, wherein the reference block of the block to be decoded includes: at least two first reference blocks of the block to be decoded in the time domain; or, the block to be decoded Decoding at least two second reference blocks of the block in the spatial domain; or, at least one first reference block of the block to be decoded in the time domain and at least one second reference block of the block to be decoded in the spatial domain; decoding The device-end residual prediction module is set to predict the residual of the to-be-decoded block based on the residual of the reference block of the to-be-decoded block to obtain the predicted residual of the to-be-decoded block; residual decoding module, set to be based on the prediction residual of the to-be-decoded block and the residual difference value between the prediction residual of the to-be-decoded block parsed from the code stream and the actual residual of the to-be-decoded block , to determine the actual residual of the block to be decoded.

According to yet another embodiment of the present invention, a storage medium is also provided, wherein a computer program is stored in the storage medium, wherein the computer program is configured to execute the steps in any one of the above method embodiments when running.

According to yet another embodiment of the present invention, there is also provided an electronic device comprising a memory and a processor, wherein the memory stores a computer program, the processor is configured to run the computer program to execute any of the above Steps in Method Examples.

According to the present invention, the residual error of the block to be coded is predicted according to the residual error of the reference block to obtain the prediction residual error, and the residual difference value between the predicted residual error and the actual residual error is encoded into the code stream, because the encoder only needs to transmit The difference between the actual residual and the predicted residual of the current block to be coded, and the residual difference is much smaller than the actual residual, so the bit rate required for the coding residual is effectively reduced, and the code saving is achieved. rate effect. On the decoder side, the same residual prediction process of the encoder side is arranged. The difference value determines the actual residual error of the block to be decoded, which can ensure that at the decoder side, the actual residual error of the block to be decoded can still be correctly recovered based on the rate-saving code stream.

Description of drawings

The accompanying drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the attached image:

1 is a block diagram of the hardware structure of a mobile terminal 10 provided with an inter-frame prediction module capable of applying a residual coding method and a residual decoding method according to an embodiment of the present invention;

2 is a flowchart of a residual coding method according to Embodiment 1 of the present invention;

3 is a structural block diagram of a residual coding apparatus according to Embodiment 1 of the present invention;

4 is a flowchart of a residual decoding method according to Embodiment 2 of the present invention;

5 is a structural block diagram of a residual decoding apparatus according to Embodiment 2 of the present invention;

6 is a schematic diagram of a convolutional neural network for generating a residual prediction module according to Embodiment 3 of the present invention;

7 is a schematic diagram of selection of a time-domain reference residual block in the case of a P frame and a B frame according to Embodiment 3 of the present invention;

8 is a schematic diagram of selection of a spatial reference residual block according to Embodiment 3 of the present invention;

FIG. 9 is a schematic process diagram of a method for improving video inter-frame coding performance by utilizing spatiotemporal correlation according to Embodiment 3 of the present invention.

Detailed ways

Hereinafter, the present invention will be described in detail with reference to the accompanying drawings and in conjunction with embodiments. It should be noted that the embodiments in the present application and the features of the embodiments may be combined with each other in the case of no conflict.

It should be noted that the terms "first", "second" and the like in the description and claims of the present invention and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific sequence or sequence.

The embodiment of the present invention provides a scheme for improving coding performance (eg, video inter-frame coding performance) by utilizing spatiotemporal correlation. The code rate of the coding residual is used to improve the coding efficiency of inter-frame prediction in video coding. The codec adds a residual prediction module to the inter-frame prediction, and predicts the residual value of the current block to be encoded (hereinafter referred to as prediction residual) by using the residual of the reference block in the spatial domain and the time domain of the current block to be encoded. Compared with the coding scheme that directly encodes the residual block of the block to be encoded into the code stream, the encoder only needs to transmit the difference between the actual residual of the current block and the predicted residual, thereby reducing the bits required to encode the residual. rate, in order to achieve the effect of saving the code rate.

The following embodiment 1 describes from the encoder side a residual coding scheme that can reduce the bit rate required for coding residuals, and the second embodiment of the present application describes from the decoder side the correct parsing to be decoded corresponding to the encoder side Residual decoding scheme for residual blocks of blocks. The encoder-side embodiments provided in Embodiment 1 and the decoder-side embodiments provided in Embodiment 2 can be respectively applied to the inter-frame prediction module of the codec-side (for example, can be applied to the inter-frame prediction of existing video coding standards. modules), these inter-frame prediction modules may be arranged in mobile terminals, computer terminals or similar computing devices. Taking the inter-frame prediction module provided on a mobile terminal as an example, FIG. 1 is a hardware structural block diagram of a mobile terminal 10 provided with an inter-frame prediction module capable of applying residual coding and residual decoding methods according to an embodiment of the present invention. As shown in FIG. 1 , the mobile terminal 10 may include one or more (only one is shown in FIG. 1 ) processors 102 (the processors 102 may include but are not limited to processing devices such as a microprocessor MCU or a programmable logic device FPGA, etc. , on which a corresponding program can be run for realizing the function of an inter-frame prediction module capable of applying the residual coding method and the residual decoding method) and a memory 104 for storing data, optionally, the above-mentioned mobile terminal may also include a Transmission device 106 and input and output device 108 for communication functions. Those of ordinary skill in the art can understand that the structure shown in FIG. 1 is only a schematic diagram, which does not limit the structure of the above-mentioned mobile terminal. For example, the mobile terminal 10 may also include more or fewer components than those shown in FIG. 1 , or have a different configuration than that shown in FIG. 1 .

The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer programs corresponding to the embodiments of the encoder end provided by Embodiment 1 of the present invention and the embodiments of the decoder end provided by Embodiment 2, and the processor. 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, the above-mentioned method is implemented. Memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory located remotely from the processor 102, and these remote memories may be connected to the mobile terminal 10 through a network. Examples of such networks include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

Transmission means 106 are used to receive or transmit data via a network. The specific example of the above-mentioned network may include a wireless network provided by the communication provider of the mobile terminal 10 . In one example, the transmission device 106 includes a network adapter (Network Interface Controller, NIC for short), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 may be a radio frequency (Radio Frequency, RF for short) module, which is used to communicate with the Internet in a wireless manner.

The residual coding scheme is described below from the encoder side and the decoder side through Embodiments 1 and 2, respectively.

Example 1

A residual coding method is provided in this embodiment. FIG. 2 is a flowchart of the residual coding method according to Embodiment 1 of the present invention. As shown in FIG. 2 , the process includes the following steps:

Step S202: Determine the residual of the reference block of the block to be coded (in this embodiment of the present invention, this term is also referred to as residual, residual block, reference residual, and reference residual block), wherein the to-be-coded block The reference blocks of the coding block include: at least two first reference blocks of the block to be coded in the time domain; or, at least two second reference blocks of the block to be coded in the spatial domain; or, the at least one first reference block of the block to be encoded in the time domain and at least one second reference block of the block to be encoded in the spatial domain;

Step S204, predicting the residual of the to-be-coded block based on the residual of the reference block of the to-be-coded block to obtain the predicted residual of the to-be-coded block;

Step S206: Encode the residual difference between the prediction residual of the block to be encoded and the actual residual of the block to be encoded into a code stream (eg, a video inter-coded code stream).

The residual of the reference block mentioned in step S202 refers to obtaining the actual coding residual of the reference block itself according to the traditional method in the coding process. Because the coding of the reference block has been completed, the residual of the reference block can be obtained by technical means are available.

The actual residual of the block to be encoded mentioned in step S206 is the difference between the pixel value of the original image block of the block to be encoded and the pixel value of the predicted block of the block to be encoded, The prediction block of the block to be encoded is a block obtained after motion compensation is performed on the reference block of the block to be encoded.

Through the above steps, predict the residual of the block to be coded according to the residual of the reference block to obtain the predicted residual, and encode the residual difference between the predicted residual and the actual residual into the code stream, because the encoder only needs to transmit The difference between the actual residual and the predicted residual of the current block to be coded, and the residual difference is much smaller than the actual residual, so the bit rate required for the coding residual is effectively reduced, and the code saving is achieved. rate effect.

Regarding the selection of the reference block of the block to be coded, in view of the continuity of the video image, selecting an appropriate reference block can accurately predict the residual of the block to be coded.

In at least one exemplary embodiment, when the image frame where the block to be encoded is located is a P frame, the first reference block of the block to be encoded in the time domain may include: the block to be encoded The optimal prediction unit block PU in the forward reference frame is denoted as the first optimal PU, and/or the optimal PU of the to-be-coded block in the frame preceding the forward reference frame is denoted as The second best PU.

When the image frame where the block to be encoded is located is a B frame, the first reference block of the block to be encoded in the time domain includes: the optimal PU of the block to be encoded in the forward reference frame , denoted as the third optimal PU, and/or, the optimal PU of the block to be encoded in the backward reference frame, denoted as the fourth optimal PU.

In at least one exemplary embodiment, when the image frame in which the block to be encoded is located is a P frame, the first reference block of the block to be encoded in the time domain may be determined in the following manner: through motion estimating to determine the first optimal PU of the block to be coded in the forward reference frame, and to determine the motion vector MV of the block to be coded relative to the first optimal PU; according to the first The position of the optimal PU and the MV, the second optimal PU is determined by motion estimation in the frame preceding the forward reference frame.

In at least one exemplary embodiment, the second reference block of the block to be encoded in the spatial domain is located in an image frame where the block to be encoded is located, and is adjacent to the block to be encoded in the spatial domain .

In at least one exemplary embodiment, the second reference block of the block to be encoded in the spatial domain includes: in the image frame where the block to be encoded is located, the left block adjacent to the block to be encoded and /or upper block (preferably may be adjacent left and/or upper blocks in the airspace with the same size).

Those skilled in the art should understand that, in practical applications, the first reference block of the block to be encoded in the time domain and/or the second reference block of the block to be encoded in the spatial domain may be determined in the above manner. The residual corresponding to the first reference block in the time domain may be referred to as a time domain reference residual block, and the residual corresponding to the second reference block in the spatial domain may be referred to as a spatial reference residual block, based on at least one time domain reference The residual block and/or at least one spatial reference residual block can predict residual information of the block to be coded.

In a preferred embodiment, for the sake of data symmetry, two temporal reference residual blocks and two spatial reference residual blocks may be selected, and residual prediction is performed based on these four reference residual blocks. At this time, the reference blocks of the to-be-coded block include: two first reference blocks whose corresponding residuals of the to-be-coded block in the time domain are not all zeros, and the to-be-coded block in the The corresponding residuals on the spatial domain are two second reference blocks that are not all zeros.

In another preferred embodiment, when only one time-domain reference residual block and one spatial-domain reference residual block are non-all-zero blocks, one time-domain and one spatial-domain reference residual block may be selected, and Residual prediction is performed based on these two reference residual blocks. In this case, the reference block of the block to be coded includes: a first reference block whose residual error corresponding to the block to be coded in the time domain is not all zeros, and the block to be coded in the The corresponding residual on the space domain is a second reference block that is not all zero.

In another preferred embodiment, when only two time-domain reference residual blocks and one spatial-domain reference residual block are non-all-zero blocks, two time-domain and one spatial-domain reference residual blocks may be selected , and perform residual prediction based on these three reference residual blocks. At this time, the reference blocks of the to-be-coded block include: two first reference blocks whose corresponding residuals of the to-be-coded block in the time domain are not all zeros, and the to-be-coded block in the The corresponding residual on the spatial domain is a second reference block that is not all zero.

In another preferred embodiment, when only one time-domain reference residual block and two spatial-domain reference residual blocks are non-all-zero blocks, one time-domain and two spatial-domain reference residual blocks can be selected , and perform residual prediction based on these three reference residual blocks. In this case, the reference block of the block to be coded includes: a first reference block whose residual error corresponding to the block to be coded in the time domain is not all zeros, and the block to be coded in the The corresponding residuals on the above-mentioned spatial domain are two second reference blocks that are not all zeros.

In another preferred embodiment, when only two time-domain reference residual blocks in the time domain are non-all-zero blocks, two time-domain reference residual blocks may be selected, and based on the two time-domain reference residual blocks The residual block performs residual prediction. At this time, the reference blocks of the to-be-coded block include: two first reference blocks whose corresponding residuals of the to-be-coded block in the time domain are not all zeros.

In another preferred embodiment, when only the two reference residual blocks in the spatial domain are non-all-zero blocks, two spatial reference residual blocks can be selected, and based on the two spatial reference residual blocks Residual prediction. At this time, the reference blocks of the to-be-coded block include: two second reference blocks whose corresponding residuals of the to-be-coded block in the spatial domain are not all zeros.

The process of predicting the residual of the to-be-coded block based on the residual of the reference block can be implemented by adopting various prediction methods. In at least one exemplary embodiment, the residual of the block to be encoded is predicted based on the residual of the reference block of the block to be encoded, and obtaining the prediction residual of the block to be encoded may include the following: one:

(1) Input the residual of the reference block into a residual prediction model to obtain the prediction residual of the to-be-coded block, wherein the residual prediction model may be a residual prediction trained by a deep learning network A model, that is, it is obtained by training a deep learning network based on training samples, the training samples including: the residuals of the reference blocks of the coding blocks with known residuals, and the codings with known residuals the actual residual of the block;

(2) Perform linear weighting on the residual of the reference block to obtain the prediction residual of the to-be-coded block, where the linear weighting includes linear weighting of a single weight or linear weighting of multiple weights.

For example, the linear weighted sum of a single weight can be calculated using the following formula:

ReiPred(i,j) ₌ W1ResiA(i,j) ₊ W2ResiB(i,j)

W1 and W2 are weights, ResiA(i, j), ResiB(i, j) are the pixel values of the selected reference residual block at pixel point (i, j), ReiPred(i, j) is the prediction residual block The pixel value at pixel point (i, j).

For example, the linear weighted sum of multiple weights can be calculated using the following formula:

W1 _ij and W2 _ij are the weight values corresponding to each pixel of the reference residual block, which can be obtained through training. ResiA(i, j), ResiB(i, j) are the pixel values of the selected reference residual block at pixel point (i, j), ReiPred(i, j) is the prediction residual block at pixel point (i, j) the pixel value at j).

From the description of the above embodiments, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is better implementation. Based on this understanding, the technical solutions of the present invention can be embodied in the form of software products in essence or the parts that make contributions to the prior art, and the computer software products are stored in a storage medium (such as ROM/RAM, magnetic disk, CD-ROM), including several instructions to make a terminal device (which may be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present invention.

In this embodiment, a residual coding apparatus is provided corresponding to the above residual coding method, which is used to implement the above embodiments and preferred implementations, and what has been described will not be repeated. As used below, the term "module" may be a combination of software and/or hardware that implements a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, implementations in hardware, or a combination of software and hardware, are also possible and contemplated.

FIG. 3 is a structural block diagram of a residual coding apparatus according to Embodiment 1 of the present invention. As shown in FIG. 3 , the apparatus includes:

The encoder-side reference residual determination module 32 is configured to determine the residual of the reference block of the block to be encoded, wherein the reference block of the block to be encoded includes: at least two of the blocks to be encoded in the time domain the first reference block; or, at least two second reference blocks of the block to be coded in the spatial domain; or, at least one first reference block of the block to be coded in the time domain and the block to be coded in the spatial domain at least one second reference block on;

The encoder-side residual prediction module 34 is configured to predict the residual of the to-be-encoded block based on the residual of the reference block of the to-be-encoded block to obtain the predicted residual of the to-be-encoded block;

The residual coding module 36 is configured to encode the residual difference value between the predicted residual of the to-be-coded block and the actual residual of the to-be-coded block into the code stream.

It should be noted that the above modules can be implemented by software or hardware, and the latter can be implemented in the following ways, but not limited to this: the above modules are all located in the same processor; or, the above modules can be combined in any combination The forms are located in different processors.

This embodiment also provides a storage medium, where a computer program is stored in the storage medium, wherein the computer program is configured to execute the steps in the above residual coding method in this embodiment when running.

Optionally, in this embodiment, the above-mentioned storage medium may be configured to store a computer program for executing the following steps:

S1, determine the residual of the reference block of the block to be encoded (in this embodiment of the present invention, this term is also referred to as residual, residual block, reference residual, and reference residual block), wherein the to-be-encoded block The reference blocks of the block include: at least two first reference blocks in the time domain of the block to be coded; or, at least two second reference blocks in the spatial domain of the block to be coded; or, the to-be-coded block at least one first reference block of the coding block in the time domain and at least one second reference block of the block to be coded in the spatial domain;

S2, based on the residual of the reference block of the to-be-coded block, predict the residual of the to-be-coded block to obtain a prediction residual of the to-be-coded block;

S3: Encode the residual difference value between the prediction residual of the block to be encoded and the actual residual of the block to be encoded into a code stream (for example, a video inter-coded code stream).

Optionally, in this embodiment, the above-mentioned storage medium may include but is not limited to: a USB flash drive, a read-only memory (Read-Only Memory, referred to as ROM), a random access memory (Random Access Memory, referred to as RAM), Various media that can store computer programs, such as removable hard disks, magnetic disks, or optical disks.

An embodiment of the present invention further provides an electronic device including a memory and a processor, where a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps of the above residual coding method in this embodiment.

Optionally, the above-mentioned electronic device may further include a transmission device and an input-output device, wherein the transmission device is connected to the above-mentioned processor, and the input-output device is connected to the above-mentioned processor.

Optionally, in this embodiment, the above-mentioned processor may be configured to execute the following steps through a computer program:

S1, determine the residual of the reference block of the block to be encoded (in this embodiment of the present invention, this term is also referred to as residual, residual block, reference residual, and reference residual block), wherein the to-be-encoded block The reference blocks of the block include: at least two first reference blocks in the time domain of the block to be coded; or at least two second reference blocks in the spatial domain of the block to be coded; or, the to-be-coded block at least one first reference block of the coding block in the time domain and at least one second reference block of the block to be coded in the spatial domain;

S2, based on the residual of the reference block of the to-be-coded block, predict the residual of the to-be-coded block to obtain a prediction residual of the to-be-coded block;

S3: Encode the residual difference value between the prediction residual of the block to be encoded and the actual residual of the block to be encoded into a code stream (for example, a video inter-coded code stream).

Optionally, for specific examples in this embodiment, reference may be made to the examples described in the foregoing embodiments and optional implementation manners, and details are not described herein again in this embodiment.

Example 2

A residual decoding method is provided in this embodiment. FIG. 4 is a flowchart of the residual decoding method according to Embodiment 2 of the present invention. As shown in FIG. 4 , the process includes the following steps:

Step S402: Determine the prediction block of the block to be decoded based on the motion vector MV in the code stream (for example, the video inter-coded code stream), and obtain the residual of the reference block of the block to be decoded (in this embodiment of the present invention, This term is also referred to as residual, residual block, reference residual, reference residual block), wherein the reference block of the block to be decoded includes: at least two of the block to be decoded in the time domain or, at least two second reference blocks in the spatial domain of the block to be decoded; or, at least one first reference block in the time domain of the block to be decoded and the block to be decoded in the at least one second reference block over the airspace;

Step S404, predicting the residual of the to-be-decoded block based on the residual of the reference block of the to-be-decoded block to obtain the predicted residual of the to-be-decoded block;

Step S406, according to the prediction residual of the to-be-decoded block and the residual difference value between the predicted residual of the to-be-decoded block parsed from the code stream and the actual residual of the to-be-decoded block, The actual residual of the block to be decoded is determined.

The residual of the reference block mentioned in step S402 refers to the actual coding residual of the reference block itself obtained according to the traditional method in the coding process. Because the coding of the reference block has been completed, the residual of the reference block can be available through technical means.

In step S402, the prediction block of the block to be decoded is a block obtained after motion compensation is performed on the block to be decoded.

The actual residual of the block to be decoded mentioned in step S406 is the difference between the pixel value of the original image block of the block to be decoded and the pixel value of the prediction block of the block to be decoded.

Through the above steps, since the same residual prediction process of the encoder is arranged at the decoder, the decoder predicts the residual of the block to be decoded based on the residual of the reference block of the block to be decoded, and predicts the residual of the block to be decoded based on the residual of the reference block of the block to be decoded. The residual difference value carried in determines the actual residual error of the block to be decoded, which can ensure that the actual residual error of the block to be decoded can still be correctly recovered based on the rate-saving code stream at the decoder side.

In at least one exemplary embodiment, when the image frame in which the block to be decoded is located is a P frame, the first reference block of the block to be decoded in the time domain includes: the block to be decoded is in the The optimal prediction unit block PU in the forward reference frame is denoted as the first optimal PU, and/or the optimal PU of the block to be decoded in the frame before the forward reference frame is denoted as the first optimal PU The second optimal PU.

In at least one exemplary embodiment, when the image frame where the block to be decoded is located is a B frame, the first reference block of the block to be decoded in the time domain includes: the block to be decoded is in the The optimal PU in the forward reference frame is recorded as the third optimal PU, and/or the optimal PU of the block to be decoded in the backward reference frame is recorded as the fourth optimal PU.

It should be noted that, in order to ensure the accuracy of residual prediction, the decoder side adopts the same reference block selection method as the encoder side, so as to ensure the consistency with the prediction residual obtained by the encoder side through residual prediction.

In at least one exemplary embodiment, when the image frame where the block to be decoded is located is a P frame, the first reference block of the block to be decoded in the time domain is determined in the following manner: according to the The MV parsed from the code stream determines the first optimal PU of the block to be decoded in the forward reference frame; according to the first optimal PU in the forward reference frame For co-located PUs in a frame, the second optimal PU is determined in a frame preceding the forward reference frame by motion estimation.

In at least one exemplary embodiment, the second reference block of the block to be decoded in the spatial domain is located in an image frame where the block to be decoded is located, and is adjacent to the block to be decoded in the spatial domain .

In at least one exemplary embodiment, the second reference block of the block to be decoded in the spatial domain includes: in the image frame where the block to be decoded is located, the left block adjacent to the block to be decoded and /or upper block (preferably may be adjacent left and/or upper blocks in the airspace with the same size).

Those skilled in the art should understand that, in practical applications, the first reference block of the block to be decoded in the time domain and/or the second reference block of the block to be decoded in the spatial domain may be determined in the above manner. The residual corresponding to the first reference block in the time domain may be referred to as a time domain reference residual block, and the residual corresponding to the second reference block in the spatial domain may be referred to as a spatial reference residual block, based on at least one time domain reference The residual block and/or at least one spatial reference residual block can predict residual information of the block to be decoded.

In a preferred embodiment, for the sake of data symmetry, two time-domain reference residual blocks and two spatial-domain reference residual blocks can be selected, and residual prediction is performed based on these four reference residual blocks. , the reference blocks of the to-be-decoded block include: two first reference blocks whose corresponding residuals of the to-be-decoded block in the time domain are not all zeros, and the to-be-decoded block in the The corresponding residuals in the spatial domain are two second reference blocks that are not all zeros.

In another preferred embodiment, when only one time-domain reference residual block and one spatial-domain reference residual block are non-all-zero blocks, one time-domain and one spatial-domain reference residual block may be selected, and Residual prediction is performed based on these two reference residual blocks. At this time, the reference block of the block to be decoded includes: a first reference block in which the corresponding residual of the block to be decoded in the time domain is not all zeros, and the block to be decoded in the The corresponding residual on the above-mentioned spatial domain is a second reference block that is not all zero.

In another preferred embodiment, when only two time-domain reference residual blocks and one spatial-domain reference residual block are non-all-zero blocks, two time-domain and one spatial-domain reference residual blocks may be selected , and perform residual prediction based on these three reference residual blocks. At this time, the reference blocks of the to-be-decoded block include: two first reference blocks whose corresponding residuals of the to-be-decoded block in the time domain are not all zeros, and the to-be-decoded block in the The corresponding residual on the spatial domain is a second reference block that is not all zero.

In another preferred embodiment, when only one time-domain reference residual block and two spatial-domain reference residual blocks are non-all-zero blocks, one time-domain and two spatial-domain reference residual blocks can be selected , and perform residual prediction based on these three reference residual blocks. At this time, the reference block of the block to be decoded includes: a first reference block in which the corresponding residual error of the block to be decoded in the time domain is not all zeros, and the block to be decoded in the The corresponding residuals on the above-mentioned spatial domain are two second reference blocks that are not all zeros.

In another preferred embodiment, when only two time-domain reference residual blocks in the time domain are non-all-zero blocks, two time-domain reference residual blocks may be selected, and based on the two time-domain reference residual blocks The residual block performs residual prediction. At this time, the reference blocks of the block to be decoded include: two first reference blocks whose corresponding residuals of the block to be decoded in the time domain are not all zeros.

In another preferred embodiment, when only the two reference residual blocks in the spatial domain are non-all-zero blocks, two spatial reference residual blocks can be selected, and based on the two spatial reference residual blocks Residual prediction. At this time, the reference blocks of the block to be decoded include: two second reference blocks whose corresponding residuals of the block to be decoded in the spatial domain are not all zeros.

The process of predicting the residual of the block to be decoded based on the residual of the reference block can be implemented by using various prediction methods. In at least one exemplary embodiment, the residual of the block to be decoded is predicted based on the residual of the reference block of the block to be decoded, and obtaining the prediction residual of the block to be decoded may include the following: one:

(1) Inputting the residual of the reference block into a residual prediction model to obtain the prediction residual of the block to be decoded, wherein the residual prediction model is the same as the residual prediction model at the encoder end;

(2) Perform linear weighting on the residual of the reference block to obtain the prediction residual of the block to be decoded, wherein the linear weighting is the same as the linear weighting at the encoder side.

For example, the linear weighted sum of a single weight can be calculated using the following formula:

ReiPred(i,j) ₌ W1ResiA(i,j) ₊ W2ResiB(i,j)

W1 and W2 are weights, ResiA(i, j), ResiB(i, j) are the pixel values of the selected reference residual block at pixel point (i, j), ReiPred(i, j) is the prediction residual block The pixel value at pixel point (i, j).

For example, the linear weighted sum of multiple weights can be calculated using the following formula:

W1 _ij and W2 _ij are the weight values corresponding to each pixel of the reference residual block, which can be obtained through training. ResiA(i, j), ResiB(i, j) are the pixel values of the selected reference residual block at pixel point (i, j), ReiPred(i, j) is the prediction residual block at pixel point (i, j) the pixel value at j).

In at least one exemplary embodiment, between the prediction residual of the block to be decoded and the prediction residual of the block to be decoded parsed from the code stream and the actual residual of the block to be decoded After determining the actual residual difference of the block to be decoded, the method further includes: adding the actual residual difference on the basis of the predicted block of the block to be decoded, and recovering the block to be decoded the original image block.

From the description of the above embodiments, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is better implementation. Based on this understanding, the technical solutions of the present invention can be embodied in the form of software products in essence or the parts that make contributions to the prior art, and the computer software products are stored in a storage medium (such as ROM/RAM, magnetic disk, CD-ROM), including several instructions to make a terminal device (which may be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present invention.

In this embodiment, a residual decoding apparatus is provided corresponding to the above residual decoding method, which is used to implement the above embodiments and preferred implementations, and what has been described will not be repeated. As used below, the term "module" may be a combination of software and/or hardware that implements a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, implementations in hardware, or a combination of software and hardware, are also possible and contemplated.

FIG. 5 is a structural block diagram of a residual decoding apparatus according to Embodiment 2 of the present invention. As shown in FIG. 5 , the apparatus includes:

The decoder side refers to the residual error determination module 52, and is set to determine the prediction block of the block to be decoded based on the motion vector MV parsed in the code stream, and obtain the residual of the reference block of the block to be decoded, wherein the to-be-decoded block The reference blocks of the block include: at least two first reference blocks in the time domain of the block to be decoded; or at least two second reference blocks in the spatial domain of the block to be decoded; or, the to-be-decoded block at least one first reference block in the time domain of the decoding block and at least one second reference block in the spatial domain of the block to be decoded;

The decoder-side residual prediction module 54 is configured to predict the residual of the to-be-decoded block based on the residual of the reference block of the to-be-decoded block to obtain the predicted residual of the to-be-decoded block;

The residual decoding module 56 is set to be based on the prediction residual of the to-be-decoded block and the prediction residual of the to-be-decoded block parsed from the code stream and the actual residual of the to-be-decoded block. The residual difference value determines the actual residual of the block to be decoded.

It should be noted that the above modules can be implemented by software or hardware, and the latter can be implemented in the following ways, but not limited to this: the above modules are all located in the same processor; or, the above modules can be combined in any combination The forms are located in different processors.

This embodiment also provides a storage medium, where a computer program is stored in the storage medium, wherein the computer program is configured to execute the steps in the above residual decoding method in this embodiment when running.

Optionally, in this embodiment, the above-mentioned storage medium may be configured to store a computer program for executing the following steps:

S1, determine the prediction block of the block to be decoded based on the motion vector MV in the code stream (for example, the video inter-coded code stream), and obtain the residual of the reference block of the block to be decoded (in this embodiment of the present invention, this A term also referred to as residual, residual block, reference residual, reference residual block), wherein the reference block of the block to be decoded includes: at least two of the blocks to be decoded in the time domain the first reference block; or, at least two second reference blocks in the spatial domain of the block to be decoded; or, at least one first reference block in the time domain of the block to be decoded and the block to be decoded in the spatial domain at least one second reference block on;

S2, based on the residual of the reference block of the to-be-decoded block, predict the residual of the to-be-decoded block to obtain the predicted residual of the to-be-decoded block;

S3, according to the prediction residual of the to-be-decoded block and the residual difference value between the prediction residual of the to-be-decoded block parsed from the code stream and the actual residual of the to-be-decoded block, determine The actual residual of the block to be decoded.

Optionally, in this embodiment, the above-mentioned storage medium may include but is not limited to: a USB flash drive, a read-only memory (Read-Only Memory, referred to as ROM), a random access memory (Random Access Memory, referred to as RAM), Various media that can store computer programs, such as removable hard disks, magnetic disks, or optical disks.

This embodiment also provides an electronic device, including a memory and a processor, where a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps of the above residual decoding method in this embodiment.

Optionally, the above-mentioned electronic device may further include a transmission device and an input-output device, wherein the transmission device is connected to the above-mentioned processor, and the input-output device is connected to the above-mentioned processor.

Optionally, in this embodiment, the above-mentioned processor may be configured to execute the following steps through a computer program:

S1, determine the prediction block of the block to be decoded based on the motion vector MV in the code stream (for example, the video inter-coded code stream), and obtain the residual of the reference block of the block to be decoded (in this embodiment of the present invention, this A term also referred to as residual, residual block, reference residual, reference residual block), wherein the reference block of the block to be decoded includes: at least two of the blocks to be decoded in the time domain the first reference block; or, at least two second reference blocks in the spatial domain of the block to be decoded; or, at least one first reference block in the time domain of the block to be decoded and the block to be decoded in the spatial domain at least one second reference block on;

S2, based on the residual of the reference block of the to-be-decoded block, predict the residual of the to-be-decoded block to obtain the predicted residual of the to-be-decoded block;

S3, according to the prediction residual of the to-be-decoded block and the residual difference value between the prediction residual of the to-be-decoded block parsed from the code stream and the actual residual of the to-be-decoded block, determine The actual residual of the block to be decoded.

Optionally, for specific examples in this embodiment, reference may be made to the examples described in the foregoing embodiments and optional implementation manners, and details are not described herein again in this embodiment.

The following embodiment 3 takes the training of a residual prediction model (also referred to as a residual prediction module) by using a deep learning network as an example, and describes the specific implementation of the residual encoding and decoding scheme in detail. It should be noted that, for the scheme of using the linear weighted sum to perform residual prediction, the principle of the overall residual encoding and decoding is similar to the following embodiments, and will not be repeated here.

Example 3

In this embodiment, a method for improving the performance of video inter-frame coding by utilizing the spatiotemporal correlation is designed for the coding unit of video coding. In actual use, the encoder will call this method to complete the inter-frame prediction. The content involved in the method will be described in detail below.

(1) Residual prediction module

The residual prediction module can be generated by a convolutional neural network. FIG. 6 is a schematic diagram of a convolutional neural network for generating a residual prediction module according to Embodiment 3 of the present invention. As shown in FIG. 6 , the convolutional neural network uses two time-domain reference residuals related to the current coding block block and two spatial reference residual blocks as input (it can also be other numbers of temporal and spatial reference residual blocks, or only temporal reference residual blocks or only spatial reference residual blocks), after convolution, pooling Extract the feature information of the residual image, and then output the prediction residual through the deconvolution process.

The input of the convolutional neural network is four reference residual blocks, two from the temporal reference frame and two from the adjacent spatial domain.

In the time domain, different operations are performed according to the different prediction modes of the P frame and the B frame. 7 is a schematic diagram of selection of a time domain reference residual block in the case of a P frame and a B frame according to Embodiment 3 of the present invention, as shown in FIG. 7 :

For the P frame, since it is unidirectional prediction, there is only one forward reference frame. First, the optimal prediction unit block (PU) is found in the forward reference frame through motion estimation, and the motion vector of the current coding block is generated ( motion vector, MV). Then, based on the position of the optimal reference PU of the current coding block and the MV of the current coding block, another optimal reference PU is searched from the previous frame of the forward reference frame through motion estimation, and the two optimal reference PUs are The corresponding residual blocks are input as two temporal reference residual blocks of the P frame.

For the B frame, there is a forward reference frame and a backward reference frame. In the forward reference frame and the backward reference frame, an optimal PU residual block is found through motion estimation as the input of the temporal reference residual block.

In the spatial domain, FIG. 8 is a schematic diagram of selection of a spatial reference residual block according to Embodiment 3 of the present invention. As shown in FIG. 8 , the left and upper blocks adjacent to the current block and of the same size are selected as reference blocks, and the residual block corresponding to the reference block is used as the input of the spatial reference residual block.

The output of the residual prediction module is the prediction residual block of the current block. On the encoder side, the prediction residual block can be subtracted from the actual residual block to obtain the coding residual. At the decoder side, the prediction residual block plus the coded residual can recover the actual residual block.

(2) Inter-coding block residual prediction method based on spatiotemporal correlation

9 is a schematic diagram of a method for improving the performance of video inter-frame coding by utilizing spatiotemporal correlation according to Embodiment 3 of the present invention. The residual encoding and decoding operations at the decoder side are described in detail below.

Operation on the encoder side:

The first step is to pre-encode the video sequence provided by the standard, and extract the residual data of the four reference blocks and the residual data of the current block as training data. Use the deep neural network embodiment constructed in FIG. 6 to perform network training, and then embed the trained deep neural network into the inter-frame prediction of the encoder.

The second step is to read the to-be-coded block to be coded from the input video image, obtain the predicted block of the to-be-coded block through motion estimation prediction, and subtract the pixel value of the predicted block from the original image pixel value to obtain the traditional residual block.

In the third step, according to the different prediction methods of the P frame and the B frame, the motion vector MV obtained by motion estimation is used to search for the two temporal reference residual blocks shown in Figure 7, and then two spatial reference residual blocks are obtained according to Figure 8. Difference block, input the obtained four reference residual blocks into the residual prediction module to obtain the prediction residual block.

In the fourth step, the difference between the traditional residual block and the prediction residual block is used as the encoding residual of the current block to be encoded, and is written into the code stream file. In this step, the traditional residual block is the residual value to be coded obtained by the current coding block according to the traditional inter-frame prediction method, and the predicted residual is the residual value predicted by the residual prediction module.

In summary, it can be seen that on the encoder side, motion estimation is used to find the reference block in the reference frame, and after the motion compensation operation, the predicted block is obtained, and the difference between the original block and the predicted block is the traditional inter prediction mode. the actual residual block. In this process, a residual prediction module is established to predict the residual of the current block. The input of the residual prediction module is two reference residual blocks in the time domain and the spatial domain corresponding to the reference block of the current coding block, and the output of the residual prediction module is the prediction residual of the current block. The actual residual block and the prediction The difference value of the residual block is used as the coding residual for subsequent transformation, quantization and entropy coding.

Operation on the decoder side:

In the first step, the residual prediction module is embedded in the decoder inter prediction.

In the second step, the prediction block of the current block to be decoded is obtained through the motion vector MV parsed by the code stream, and then the two temporal reference residual blocks shown in Figure 7 are found, and the two spatial domains shown in Figure 8 are obtained. Referring to the residual block, input the obtained four reference residual blocks into the residual prediction module to obtain the prediction residual block. In this step, the four reference residual blocks corresponding to the current block are generated in the same manner as the four reference residual blocks on the encoder side.

The third step is to read the coding residual of the block to be decoded from the code stream, then add the predicted residual block to the encoded residual to restore the actual residual of the to-be-decoded block, and finally add the predicted block to restore the original image block.

To sum up, it can be seen that in the inter prediction decoding operation at the decoder side, the same residual prediction module as the encoder is established to predict the residual of the current block. The input of the residual prediction module is two reference residual blocks in the time domain and spatial domain corresponding to the reference block of the current decoding block. The search method of the reference block is the same as that of the encoder, and the output is the prediction residual of the current decoding block. The reconstruction of the current block can be completed by adding the decoded residual and the predicted block of the block to be decoded.

Obviously, those skilled in the art should understand that the above-mentioned modules or steps of the present invention can be implemented by a general-purpose computing device, which can be centralized on a single computing device, or distributed in a network composed of multiple computing devices Alternatively, they may be implemented in program code executable by a computing device, such that they may be stored in a storage device and executed by the computing device, and in some cases, in a different order than here The steps shown or described are performed either by fabricating them separately into individual integrated circuit modules, or by fabricating multiple modules or steps of them into a single integrated circuit module. As such, the present invention is not limited to any particular combination of hardware and software.

The above descriptions are only preferred embodiments of the present invention, and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the principles of the present invention shall be included within the protection scope of the present invention.

Claims (19)

1. A method of residual coding, comprising:

determining a residual of a reference block of a block to be encoded, wherein the reference block of the block to be encoded comprises: at least two first reference blocks of the block to be coded in the time domain; or, the block to be coded is at least two second reference blocks in a space domain; or, at least one first reference block of the block to be coded in the time domain and at least one second reference block of the block to be coded in the space domain;

predicting the residual error of the block to be coded based on the residual error of the reference block of the block to be coded to obtain the predicted residual error of the block to be coded;

and coding a residual difference value between the prediction residual of the block to be coded and the actual residual of the block to be coded into a code stream.

2. The method of claim 1,

under the condition that the image frame where the block to be coded is located is a P frame, the first reference block of the block to be coded in the time domain comprises: recording an optimal prediction unit block (PU) of the block to be coded in a forward reference frame as a first optimal PU, and/or recording an optimal PU of the block to be coded in a previous frame of the forward reference frame as a second optimal PU;

or,

under the condition that the image frame where the block to be coded is located is a B frame, the first reference block of the block to be coded in the time domain comprises: and recording the optimal PU of the block to be coded in the forward reference frame as a third optimal PU, and/or recording the optimal PU of the block to be coded in the backward reference frame as a fourth optimal PU.

3. The method as claimed in claim 2, wherein in the case that the image frame where the block to be encoded is located is a P frame, the first reference block in the time domain of the block to be encoded is determined by:

determining the first optimal PU of the block to be coded in the forward reference frame through motion estimation, and determining a motion vector MV of the block to be coded relative to the first optimal PU;

determining the second optimal PU in a frame previous to the forward reference frame by motion estimation based on the location of the first optimal PU and the MV.

4. The method according to claim 1, wherein a second reference block of the block to be coded in the spatial domain is located in an image frame where the block to be coded is located and is adjacent to the block to be coded in the spatial domain.

5. The method of claim 4, wherein the second reference block of the block to be coded in the spatial domain comprises: and in the image frame where the block to be coded is located, the left block and/or the upper block adjacent to the block to be coded.

6. The method according to any of claims 1-5, wherein the reference block of the block to be encoded comprises:

the corresponding residual errors of the blocks to be coded in the time domain are two first reference blocks which are not all zero, and the corresponding residual errors of the blocks to be coded in the space domain are two second reference blocks which are not all zero; or,

a first reference block of which the corresponding residual error of the block to be coded in the time domain is non-all-zero, and a second reference block of which the corresponding residual error of the block to be coded in the space domain is non-all-zero; or,

two first reference blocks of which the corresponding residual errors of the blocks to be coded in the time domain are non-all-zero, and one second reference block of which the corresponding residual errors of the blocks to be coded in the space domain are non-all-zero; or,

the corresponding residual error of the block to be coded in the time domain is a first reference block which is not all zero, and the corresponding residual error of the block to be coded in the space domain is two second reference blocks which are not all zero; or,

two first reference blocks of which the corresponding residual errors of the blocks to be coded in the time domain are non-all-zero; or,

and the corresponding residual errors of the blocks to be coded on the spatial domain are two second reference blocks which are not all zero.

7. The method according to any of claims 1-5, wherein predicting the residual of the block to be coded based on the residual of the reference block of the block to be coded, and obtaining the predicted residual of the block to be coded comprises:

inputting the residual error of the reference block into a residual error prediction model to obtain a prediction residual error of the block to be coded, wherein the residual error prediction model is obtained by adopting deep learning network training based on training samples, and the training samples comprise: a residual of a reference block of a coding block having a known residual, and an actual residual of the coding block having the known residual;

or,

and performing linear weighting on the residual of the reference block to obtain a prediction residual of the block to be coded, wherein the linear weighting comprises linear weighting of single weight or linear weighting of multiple weights.

8. The method according to any of claims 1-5, wherein the actual residual of the block to be encoded is a difference between a pixel value of an original image block of the block to be encoded and a pixel value of a prediction block of the block to be encoded, wherein the prediction block of the block to be encoded is a block obtained after motion compensation of the reference block of the encoding block.

9. A residual decoding method, comprising:

determining a prediction block of a block to be decoded based on a motion vector MV analyzed from a code stream, and acquiring a residual of a reference block of the block to be decoded, wherein the reference block of the block to be decoded comprises: at least two first reference blocks of the block to be decoded in the time domain; or, the block to be decoded is at least two second reference blocks on a spatial domain; or, at least one first reference block of the block to be decoded in time domain and at least one second reference block of the block to be decoded in space domain;

predicting the residual of the block to be decoded based on the residual of the reference block of the block to be decoded to obtain a predicted residual of the block to be decoded;

and determining the actual residual of the block to be decoded according to the prediction residual of the block to be decoded and a residual difference value analyzed from the code stream, wherein the residual difference value is a difference value between the prediction residual of the block to be decoded and the actual residual of the block to be decoded.

10. The method of claim 9,

when the image frame where the block to be decoded is located is a P frame, the first reference block of the block to be decoded in the time domain includes: recording an optimal prediction unit block (PU) of the block to be decoded in a forward reference frame as a first optimal PU, and/or recording an optimal PU of the block to be decoded in a previous frame of the forward reference frame as a second optimal PU;

or,

when the image frame where the block to be decoded is located is a B frame, the first reference block of the block to be decoded in the time domain includes: and recording the optimal PU of the block to be decoded in the forward reference frame as a third optimal PU, and/or recording the optimal PU of the block to be decoded in the backward reference frame as a fourth optimal PU.

11. The method as claimed in claim 10, wherein in the case that the image frame in which the block to be decoded is located is a P frame, the first reference block in the time domain of the block to be decoded is determined by:

determining the first optimal PU of the block to be decoded in the forward reference frame according to the analyzed MV in the code stream;

determining the second optimal PU in a frame previous to the forward reference frame by motion estimation according to a co-located PU of the first optimal PU in a frame previous to the forward reference frame.

12. The method of claim 9, wherein a second reference block of the block to be decoded on the spatial domain is located in an image frame of the block to be decoded and is adjacent to the block to be decoded on the spatial domain.

13. The method of claim 12, wherein the second reference block of the block to be decoded on the spatial domain comprises: and in the image frame where the block to be decoded is located, the left block and/or the upper block adjacent to the block to be decoded.

14. The method according to any of claims 9-13, wherein predicting the residue of the block to be decoded based on the residue of the reference block of the block to be decoded, and wherein obtaining the predicted residue of the block to be decoded comprises:

inputting the residual of the reference block into a residual prediction model to obtain the prediction residual of the block to be decoded, wherein the residual prediction model is the same as a residual prediction model at an encoder end;

or,

and carrying out linear weighting on the residual error of the reference block to obtain the prediction residual error of the block to be decoded, wherein the linear weighting is the same as that of an encoder end.

15. The method according to any of claims 9-13, further comprising, after determining an actual residual of the block to be decoded from the difference between the prediction residual of the block to be decoded and a residual parsed from the code stream:

and adding the actual residual on the basis of the prediction block of the block to be decoded, and restoring an original image block of the block to be decoded.

16. A residual encoding apparatus, comprising:

an encoder-side reference residual determining module configured to determine a residual of a reference block of a block to be encoded, wherein the reference block of the block to be encoded includes: at least two first reference blocks of the block to be coded in the time domain; or, the block to be coded is at least two second reference blocks in a space domain; or, at least one first reference block of the block to be coded in the time domain and at least one second reference block of the block to be coded in the space domain;

the encoder-side residual error prediction module is used for predicting the residual error of the block to be coded based on the residual error of the reference block of the block to be coded to obtain the predicted residual error of the block to be coded;

and the residual coding module is used for coding a residual difference value between the prediction residual of the block to be coded and the actual residual of the block to be coded into a code stream.

17. A residual decoding apparatus, comprising:

a decoder-side reference residual determining module configured to determine a prediction block of a block to be decoded based on a motion vector MV analyzed in a code stream, and obtain a residual of a reference block of the block to be decoded, where the reference block of the block to be decoded includes: at least two first reference blocks of the block to be decoded in the time domain; or, the block to be decoded is at least two second reference blocks on a spatial domain; or, at least one first reference block of the block to be decoded in time domain and at least one second reference block of the block to be decoded in space domain;

a decoder-side residual prediction module configured to predict a residual of the block to be decoded based on a residual of the reference block of the block to be decoded, so as to obtain a predicted residual of the block to be decoded;

and the residual decoding module is used for determining the actual residual of the block to be decoded according to the prediction residual of the block to be decoded and a residual difference value analyzed from the code stream, wherein the residual difference value is a difference value between the prediction residual of the block to be decoded and the actual residual of the block to be decoded.

18. A storage medium, in which a computer program is stored, wherein the computer program is arranged to perform the method of any of claims 1 to 15 when executed.

19. An electronic device comprising a memory and a processor, wherein the memory has stored therein a computer program, and wherein the processor is arranged to execute the computer program to perform the method of any of claims 1 to 15.

Images