JPH11285007A - Method and device for encoding moving image - Google Patents

Abstract

PROBLEM TO BE SOLVED: To reduce an encoding quantity and to prevent deterioration in picture quality by performing warping motion compensating operation by making a jump by the number of frames determined by both the variation quantities of a moving image being inputted and an encoding quantity being outputted and multiplexing the encoded signal of a moving vector, the preliminary error signal of frames, and the encoded signal of the number of frames. SOLUTION: An analyzer 1 monitors a frame being inputted and the variation quantity in a buffer 20, analyzes the quantitative variation to determine the number of jumped frames, and informs a multiplexing means 2 and a timing generator 3 of it. The timing generator 3 decides whether or not the frame being inputted is jumped over and inputs a frame which is a specific number of frames ahead by operating a switch 5. Warping motion compensating operation is performed by inputted frames, the preliminary error signal of the difference from the frame being inputted is encoded, and the multiplexing means 2 encodes the number of jumped frames, multiplexes the encoded signal of the moving vector, the preliminary error signal of the frames, and the encoded signal of the number of frames, and outputs the resulting signal as an encoded sequence.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a moving picture coding technique, and more particularly to the storage and recording of moving picture signals represented by television signals.
The present invention relates to a moving picture coding technique used for transmission and broadcasting.

[0002]

2. Description of the Related Art In a conventional moving picture encoding and decoding method and apparatus, a motion-compensated inter-frame predictive coding method has recently been actively used as a part of a standard moving picture coding method. In particular, the motion-compensated inter-frame predictive coding method includes:
A hybrid coding method combining a discrete cosine coding method (hereinafter referred to as DCT) is most often used. In these systems, a DC that performs motion compensation using block matching and quantization in consideration of visual characteristics is used.
Usually, T coding is used.

[0003] The warping motion compensation prediction used in the present invention is a type of motion compensation prediction, and is a method of interpolating a motion vector obtained in a block unit to obtain a pixel unit and performing motion compensation in a pixel unit. Regarding this technology, J.
J. Niewglowski, TGCampbell, a
nd P. Haavisto, A novel video coding scheme based on temporal prediction using digital image warping
IEEE using International Conference on Consumer Electronics, June 8-10, 1993, in Chicago, USA entitled prediction using digital image warping,
International Conference on Consumer Electronics
), And in the application number Heisei 5-330556 (“moving picture coding method using control grid”) by the inventor of the present invention, there is a detailed description under the name of “control grid motion compensation”. G. Wolberg's digital imagery
19, entitled Digital Image Warping,
Descriptions at the IEEE Computer Society Press, published in 1990 by Los Alamitos, California, USA, are also known. Furthermore, this method is also called morphing motion compensation, and is used by GJ Sullivan et al.
J. Sullivan and RL Baker,) Motion compensation for video compression using control grid interpolation (Motion compensation for video compression us)
ingcontrol grid interpolation, 199
International Conference on Acoustics held in Toronto, Canada in May of January
Speech and signal processing (Int. C
onf. Acoustics, Speech, and Signal Processing,)
c.) on pages 2713 to 2716,
Motion compensation based on segmentation by integration of triangular patches by Nakatani and Harashima (Motion
compensation based on segmentation by integration
Specific proposals have been made in the 1991 Image Coding Symposium Proceedings, 3-1 entitled of triangular patches,).

Next, an outline of warping motion compensation will be described. FIG. 6 is an explanatory diagram for explaining how a motion vector is used in warping motion compensation.

[0005] First, equidistant grids are set in the screen.
A motion vector at each grid point is obtained, and for pixels other than the grid points, a motion vector for each pixel is obtained by interpolation from the motion vectors of a plurality of nearest grid points.

The warping motion compensation can be replaced with the conventional block-based motion compensation. Therefore, it is possible to easily cope with a hybrid coding scheme with DCT or the like. For example, as a conventional moving picture encoding device, as shown in a block diagram of FIG. 7, a first frame memory 11 for storing past frames of moving picture data composed of a plurality of frames, a first frame memory A motion vector estimating means 12 for estimating a motion vector from the output from the current frame 11 and the current frame; a warping motion compensating means 13 for performing a warping motion compensation operation based on the motion vector estimated by the motion vector estimating means 12; A differentiator 14 for calculating the difference between the compensation operation result of the compensator 13 and the currently input frame; a prediction error encoder 15 for encoding a prediction error signal output from the differentiator 14 as a prediction error code; An adder 16 for adding the obtained local decoded signal to the compensation operation result of the warping motion compensator 13 and outputting the result; A second frame memory 17 to be stored in order to use the force to warping motion compensation operation of the next frame, the motion vector coding unit 18 for coding the motion vector the motion vector estimation unit 12 has estimated,
A multiplexing unit 19 that multiplexes the output of the motion vector coding unit 18, the prediction error code, and the quantization characteristic of each frame and outputs the coded sequence, and a coded sequence output by the multiplexing unit 19 that is smoothed at a constant speed. A buffer 20 for transferring data; and an analyzer 21 for analyzing a currently input frame and a generated code amount in the buffer, determining a quantization characteristic, and supplying the quantized characteristic to the prediction error encoding means 15 and the multiplexing means 19. There is.

The operation of the moving picture coding apparatus is as follows. The moving picture data is composed of a plurality of frames, and the motion vector estimating means 12 is determined from the past frame stored in the first frame memory 11 and the currently input frame. Estimates the motion vector, and the warping motion compensating means 13 performs a warping motion compensation operation based on the motion vector. Next, a prediction error signal which is a difference between the warping motion compensation calculation result and the currently input frame is calculated by the differentiator 14, and the calculated prediction error signal is encoded by the prediction error encoding means 15 as a prediction error code. I do. Subsequently, the adder 16 adds this prediction error code to the warping motion compensation calculation result, and stores this addition result in the second frame memory 17 for use in the warping compensation calculation of the next frame. The motion vector estimated by the motion vector estimating means 12 is encoded by the motion vector encoding means 18 for transfer. next,
The multiplexing unit 19 multiplexes the output of the motion vector coding unit 18, the prediction error code, and the quantization characteristics of each frame, and outputs the multiplexed code string. Subsequently, the buffer 20 smoothes the code string output from the multiplexing means 19 and transfers the code string at a constant speed. Further, the analyzer 21 analyzes the amount of change in the content of the currently input frame and the amount of code generated in the buffer 20 (or the occupied amount of the buffer), determines a quantization characteristic, and multiplexes with the prediction error encoding means 15. To the means 19. That is,
If the generated code amount is large and it is necessary to reduce the generated code amount, or if the temporal change in the content of the input video signal is large and deterioration in image quality is not noticeable even when coarse Reference numeral 21 controls the prediction error encoding means 15 so that the quantization characteristic becomes coarse, and when the generated code amount is small or the temporal content change amount of the input moving image signal is small, the quantization characteristic The prediction error encoding means is controlled so as to be fine.

On the other hand, as shown in the block diagram of FIG. 8, a conventional moving picture decoding apparatus includes a buffer 31 for temporarily storing a received code string and a code string received via the buffer 31 as a motion vector. Demultiplexing means 32 for demultiplexing into a plurality of elements including the encoded signal, the prediction error signal code and the quantization characteristic, and the prediction error signal from the prediction error code and the quantization characteristic output from the demultiplexing means 32. A prediction error decoding unit 33 for decoding, a motion vector decoding unit 34 for decoding a motion vector from a coded signal of the motion vector, a warping motion compensation unit 35 for performing warping motion compensation using the decoded motion vector, and a prediction error signal And an adder 36 for adding the result of the warping motion compensation and an adder 3
And a frame memory 37 for storing the output of No. 6 for use in the warping motion compensation of the next frame.

The operation of this decoder is as follows.
Demultiplexes the received code string into a plurality of elements including a coded signal of a motion vector, a prediction error code, and a quantization characteristic by a demultiplexing unit 32. Subsequently, the prediction error decoding unit 33 The difference is decoded from the characteristic. At the same time, the motion vector decoding means 34 decodes the motion vector from the coded signal of the motion vector, and subsequently the warping motion compensation means 35 performs the warping motion compensation using this motion vector and the past frame stored in the frame memory. Do. Next, the adder 36 adds the result of the warping motion compensation and the difference, decodes and outputs moving image data composed of a plurality of frames. The frame memory 37 includes an adder 36
Is stored for use in warping motion compensation for the next frame.

[0010]

The conventional moving picture coding and decoding apparatus described above has a low operating principle whether the operation principle is the motion compensation prediction coding scheme or the warping motion compensation prediction coding scheme. When applied to encoding at a bit rate, there is a problem that image quality deteriorates.

An object of the present invention is to interpolate warping motion compensation and interpolate a frame by interpolating or extrapolating frames between frames, thereby preventing deterioration of image quality even if code amount is significantly reduced. It is an object of the present invention to provide a technique capable of performing efficient encoding and decoding.

[0012]

According to the present invention, frames constituting a moving image are classified into three types. The first is an intra-coded frame that does not perform inter-frame predictive coding, and is referred to as an I-frame. An I frame is a frame required to start encoding or decoding, and its insertion frequency changes depending on the application.

[0013] The second is a normal predictive coding frame,
This is encoded as described in the prior art. This is referred to as a P frame. In the present invention, this P frame is set while skipping every several frames. FIG. 3 is an explanatory diagram for explaining a relationship between an I frame and a P frame and other uncoded frames. As shown in FIG. 3, the skipped frames are not encoded. The number of jump frames may be arbitrarily changed. For this purpose, the number of skipped frames is required on the moving image decoding device side, and the moving image encoding device side encodes and multiplexes the number of skipped frames.

On the moving picture decoding apparatus side, which is a pair with the present invention,
Only the motion vector of the P frame, the prediction error, and the number of skipped frames are decoded. Here, the skipped frame is referred to as an F frame.

In reconstructing the F frame, the motion vector for the P frame can be interpolated according to the position of the F frame to obtain the motion vector for the F frame. Warping motion compensation is performed using the interpolated motion vector, and the signal is directly displayed as a reconstructed signal of the F frame without adding a prediction error. FIG. 4 is an explanatory diagram for explaining the relationship between the I frame and the P frame processed by the interpolation method at the time of decoding and the reconstructed F frame. In the example shown in FIG. 4, one F-frame exists between the I-frame and the immediately succeeding P-frame.
The frame is stuck. This F frame is reconstructed by warping motion compensation using (1/2) v0 obtained by interpolating the motion vector v0 for the P frame. Further, two F frames are sandwiched between the P frame and the next P frame, and the motion vector v1 for the next P frame is used.
/ 3) v1 and (2/3) v1 reconstructed by warping motion compensation.

In general, the motion of a moving image is continuous, and a natural image can be reconstructed by such processing. In particular, the warping motion compensation does not generate discontinuities in the motion compensation result, unlike the conventional block motion compensation. Therefore, even if such time-direction motion interpolation is performed, no unnatural feeling is generated.

FIG. 5 is an explanatory diagram for explaining the relationship between the I frame and the P frame processed by the interpolation method and the extrapolation method at the time of decoding and the reconstructed F frame. In the example shown in FIG. 5, 2 is between the first P frame and the next P frame.
Although the F frames are interposed, the motion vector of the first F frame is not (1/3) v1, but the (2/3) v0 using the motion vector v0 used immediately before, from the I frame. Warping motion compensation. Thereby, decoding delay can be suppressed.

[0018]

Next, embodiments of the present invention will be described with reference to the drawings.

FIG. 1 is a block diagram showing an embodiment of a moving picture coding apparatus according to the present invention.

The difference from the conventional moving picture coding apparatus is that the analyzer 1 determines the quantization characteristic while monitoring the amount of change in the content of the currently input frame and the generated code amount. Multiplexing means 2 has a function of determining and outputting the number of interleaved frames, and a function of encoding the number of interleaved frames and multiplexing it with another code, and further selects a frame to be encoded according to the number of interleaved frames. The point that a timing generator 3 for generating timing and a switch 5 combined with a delay unit 4 for taking in only a frame to be coded from a plurality of frames that continuously receive and receive the output of the timing generator 3 are added. is there. In addition, the ones whose names and codes are the same as those of the
Since the functions are the same, the description is omitted.

Next, the operation will be described. In order to prevent the description from being redundant, a description will be given mainly of a portion that performs an operation different from that of the conventional moving picture coding apparatus.

In general, when performing data compression, if the motion is monotonous and small, the number of skipped frames may be increased.
If the movement is severe or a scene change occurs, it is necessary to reduce the number of skipped frames. For this reason, the analyzer 1 monitors the change in the content of the currently input frame and the change in the occupation amount of the buffer 20 (or the amount of code input to the buffer per unit time), and the quantitative analysis of these two signals is performed. By analyzing the change, the number of interlaced frames for determining a frame to be quantized among a plurality of frames is determined, and the number of skipped frames is notified to the multiplexing means 2 and the timing generator 3. The timing generator 3 operates the switch 5 to take in the frames skipped by the specified number of frames while determining whether or not to skip the currently input frame. Thereafter, the captured frame operates in the same manner as the conventional video encoding device, performs a warping motion compensation operation for each corresponding frame, and encodes a prediction error signal, which is a difference from the currently input frame, as a prediction error code. And the number of frames skipped by the multiplexing means 2 is encoded.
The encoded signal of the motion vector, the prediction error code of the corresponding frame, and the encoded signal of the number of frames are multiplexed and output as a code string.

FIG. 2 is a block diagram showing an example of a moving picture decoding apparatus which can be paired with the encoding apparatus shown in FIG. FIG.
(A) is a block diagram of a first example of the video decoding device,
FIG. 2B is a block diagram of the second example.

The difference between the first example of the moving picture decoding apparatus and the conventional moving picture decoding apparatus is that the demultiplexing means 22 has a function of decoding the coded signal of the number of interlaced frames included in the output, and further has a function of decoding. A frame counter 23 for receiving the number of skipped frames and the end signal for each frame at the time of executing the warping motion compensation from the warping motion compensation means, and counting the distance between the frame being processed and the immediately preceding frame; 34, a memory 24 for storing the decoded motion vectors, a motion vector from the memory 24, an inter-frame distance and the number of interlaced frames from the frame counter 23, and a motion vector of the frame being processed is obtained by interpolation calculation. 35
And a first switch 2 connected only at the time of outputting a frame to be encoded at the time of transmission and to be decoded at the time of reception and adding a prediction error signal to the adder 36.
6 and a second switch 27 that is connected only when the first switch 26 is connected and stores the output of the adder 36 in the frame memory 37. The other components having the same names and codes as those of the conventional video decoding device have the same functions, and thus the description is omitted.

Next, the operation will be described. In order to prevent the description from becoming redundant, a description will be given mainly of a portion that performs an operation different from that of the conventional moving picture decoding apparatus.

After the motion vector is decoded from the received code string by the motion vector decoding means 34, the motion vector is stored in the memory 24 for use over several frames.
The motion vector interpolation means 25 receives the inter-frame distance and the number of skipped frames from the frame counter 23, and interpolates the motion vector in the memory 24 according to the distance between the decoded adjacent frames according to the number of skipped frames. The warping motion compensating means 35 performs only warping motion compensation using the interpolated motion vector, and
6 adds the result of the warping motion compensation and the difference via the first switch 26 under the control of the frame counter 23, and exists between each of the actual frames on the transmitting side, And decodes and outputs moving image data including a plurality of frames that did not exist. At this time, the frame memory 37 stores the output of the adder 36 for use in warping motion compensation of the next frame.

The second example of the moving picture decoding apparatus paired with the present invention has exactly the same configuration as the first example. The only difference is that in the first example, the motion vector interpolation means 25 is used. Frame counter 23 which receives a motion vector from memory 24
From the distance between frames and the number of skipped frames,
The point is that the motion vector of the frame being processed is obtained by extrapolation and interpolation calculation, and is replaced by the motion vector interpolation / extrapolation means 28 supplied to the warping motion compensation means 35.

Next, the operation will be described.

After the motion vector is decoded from the received code string by the motion vector decoding means 34, the motion vector is stored in the memory 24 for use over several frames.
The motion vector interpolation / extrapolation means 28 receives the inter-frame distance and the number of skipped frames from the frame counter 23, and interpolates according to the distance between adjacent frames obtained by decoding the motion vector in the memory 24 according to the number of skipped frames. And extrapolate. Warping motion compensation means 35
Performs only warping motion compensation with the interpolated and extrapolated motion vectors, and an adder 36 adds the result of the warping motion compensation and the difference via the first switch 26 under the control of the frame counter 23. Then, it decodes and outputs moving image data including a plurality of frames that exist between the actual frames on the transmission side and do not exist in the received code string. At this time, the frame memory 37 stores the output of the adder 36 for use in warping motion compensation of the next frame.

As described above, according to the present invention, the moving picture coding apparatus side determines the number of skipped frames by analyzing the input frame and the output code string, and Performs a warping motion compensation operation, encodes a prediction error signal, which is a difference from the currently input frame, as a prediction error code, encodes the number of skipped frames, and predicts a motion vector encoded signal and a corresponding frame. The error code and the coded signal of the number of frames are multiplexed and output as a code string, and the moving image decoding apparatus decodes the received code string and extracts the extracted frame and the adjacent frame of the decoded motion vector according to the number of skipped frames. Interpolation and extrapolation according to the distance between them, and only warping motion compensation is performed using the interpolated and extrapolated motion vectors. By reconstructing a plurality of skipped frames that are not present in the received code sequence, it is possible to prevent a deterioration in image quality even if the code amount is significantly reduced, and to perform more efficient encoding and decoding. There is an effect of becoming.

[Brief description of the drawings]

FIG. 1 is a block diagram of an embodiment of a moving image encoding apparatus according to the present invention.

FIG. 2 is a block diagram illustrating an example of a moving image decoding device that can be used in combination with the present invention.

FIG. 3 is an explanatory diagram for explaining a relationship between an I frame and a P frame and other non-coded frames.

FIG. 4 shows an I frame and P processed by interpolation during decoding.
FIG. 4 is an explanatory diagram for explaining a relationship between a frame and a reconstructed F frame.

FIG. 5 is an explanatory diagram for explaining a relationship between an I frame and a P frame processed by an interpolation method and an extrapolation method at the time of decoding and a reconstructed F frame;

FIG. 6 is an explanatory diagram illustrating how a motion vector is used in warping motion compensation.

FIG. 7 is a block diagram of a conventional video encoding device.

FIG. 8 is a block diagram of a conventional video decoding device.

[Description of Code] 1,21 Analyzer 2,19 Multiplexer 3 Timing Generator 4 Delayer 5 Switch 11 First Frame Memory 12 Motion Vector Estimator 13,35 Warping Motion Compensator 14 Differentiator 15 Prediction Error Code Coding means 16, 36 adder 17 second frame memory 18 motion vector coding means 20, 31 buffer 22, 32 demultiplexing means 23 frame counter 24 memory 25 motion vector interpolation means 26 first switch 27 second Switch 28 Motion vector interpolation / extrapolation means 33 Prediction error decoding means 34 Motion vector decoding means 37 Frame memory

Claims (2)

[Claims] A motion vector is estimated from a past frame of moving image data composed of a plurality of frames and a currently input frame, and a warping motion compensation operation is performed based on the motion vector. A prediction error signal which is a difference between the current frame and the currently input frame is encoded as a prediction error code, and the prediction error code is added to the warping motion compensation calculation result, and the added result is used for the warping compensation calculation of the next frame. Therefore, in the moving image encoding method of storing the motion vector in the frame memory, multiplexing the motion vector including the quantization characteristic, and multiplexing the motion vector with the prediction error code, and outputting the coded sequence as a code sequence, Interlace frame for determining the frame to be quantized among the plurality of frames by analyzing the amount of change in the code amount The number of the frame is determined, a plurality of frames currently being input, while performing the warping motion compensation calculation for each corresponding frame while skipping the number of the interlaced frame, a prediction error signal that is a difference from the previously encoded frame. Encoding the number of skipped frames while encoding as a prediction error code, multiplexing the coded signal of the motion vector and the prediction error code of the corresponding frame with the coded signal of the number of frames, and outputting as a code sequence A moving picture coding method characterized by the above-mentioned. 2. A first frame memory for storing a past frame of moving image data composed of a plurality of frames, and a motion vector estimation for estimating a motion vector from an output from the first frame memory and a current frame. Means, a warping motion compensation means for performing a warping motion compensation operation based on the motion vector estimated by the motion vector estimation means, and a differentiator for calculating a difference between a compensation operation result of the warping motion compensation means and a currently input frame. A prediction error encoding unit that encodes a prediction error signal output from the differentiator as a prediction error code, and an adder that adds the prediction error code to a compensation operation result of the warping motion compensation unit and outputs the result. A second frame memory for storing the output of the adder for use in the warping motion compensation operation of the next frame; Motion vector encoding means for encoding a motion vector, multiplexing means for multiplexing the output of the motion vector encoding means, the prediction error code, and the quantization characteristic of each frame, and outputting as a code string; A buffer for smoothing the code string output by the means and transferring it at a constant speed, and analyzing the amount of change in the content of the moving image currently being input and the amount of change in the amount of code being output, determining the quantization characteristic, and In a moving image encoding apparatus including an error encoding unit and an analyzer for supplying to a multiplexing unit, the analyzer monitors and analyzes a currently input frame and a generated code amount of a code string being output in the buffer. If the two change amounts are large, the number of interlaced frames is set to be small, and if the two change amounts are small, the number of interleaved frames is set to a large value. A multiplexing means having a function of encoding the number of interlaced frames and multiplexing with other codes, a timing generator for generating a timing for selecting a frame to be encoded according to the number of interleaved frames, and an output of the timing generator. And a switch for taking in only a frame to be coded from a plurality of frames that are continuously input.