KR100775143B1 - Video decoding - Google Patents

Abstract

The present invention is a method of selecting an intra frame (I frame) as a random access point in a H.264 based video decoder.

The present invention provides a method for performing a shift search (FF / REW) using a predetermined frame as an access point when decoding a video based on H.264, comprising: a pointer containing an address of a corresponding buffer to a frame received before an IDR NAL. resetting only the pointer) and keeping the contents of the physical memory (frame buffer) not to be erased, and searching for which time position the received NAL reconstructed frame belongs to at the decoder; Making an I frame accessible by determining whether to apply error concealment or jump to the next IDR based on the number of frames stored in the frame buffer when the number of reference frames (num_ref_frames) is 2 or more; In the case of referencing a reference frame that does not exist during the decoding, a method of selecting one of adjacent reference frames. The intra frame is a random access point in the H.264-based video decoder. It can be used.

H.264, video decoder, I-frame access, FF / REW

Description

Video information decoding method {VIDEO DECODING}

1 illustrates an example of a random access point at FF in a conventional H.264 based video decoder.

2 illustrates an example of a random access point at FF in an H.264 based video decoder according to the present invention.

3 illustrates a NAL Unit Syntax structure.

4 illustrates a structure of a NAL unit type syntax;

5 is a diagram showing the structure of the front portion of a slice header of H.264

6 is a diagram showing an example of a processing method when a frame reference does not exist in the present invention.

The present invention relates to a method of selecting an intra frame (I frame) as a random accessible point in an H.264 based video decoder, and in particular, a mobile handset, a PC-based player, or a ground / satellite satellite. Random access point used to play H.264-based content in a DMB broadcasting terminal, such as streaming, or fast-forward / fast-rewind (FF / REW), etc., in VOD or video file playback. ) Can also be used for I frames.

There are various standards for compressing and encoding video such as MPEGx and H.26x.

Among them, H.264, a high-efficiency compression technology, is basically composed of NAL (Network Abstract Layer) units. The NAL type containing video data can be largely composed of two types, IDR NAL and non-IDR NAL.

Instaneous Decoding Refresh (IDR) NAL is a random-access point that compresses using only spatial redundancy without using temporal redundancy, and removes all frames that come before IDR NAL from the frame buffer. It is no longer referenced for compression.

In comparison, the non-IDR NAL consists of an I type slice, a P type slice, and a B type slice. P-type slices and B-type slices are compressed with the same prediction coding as conventional codecs. Of these, I-type slices are compressed using only spatial redundancy like IDR, but the difference is that they do not erase the contents of the frame buffer. The reason for not erasing the contents of the frame buffer is that the P-type NAL or B-type NAL coming after the I-type slice can refer to the contents before the I-type slice.

An IDR frame composed of IDR NAL can be a random access point used for variable-speed playback mode (FF / REW) during video playback, while a non-IDR frame composed of non-IDR NAL uses I only using spatial redundancy. Even a frame is not utilized as a random accessible point. The reason for this is that, as described above, NALs coming in according to I frames may perform prediction encoding based on the contents before the I frame. Therefore, when referring to a frame that does not exist in the frame buffer, most of them show an error message or a decoder malfunction due to software processing.

1 illustrates an example of a random access point at FF in a conventional H.264 based video decoder. Referring to FIG. 1, it can be seen that frames are arranged in the order of IDR, P, P, I, P, P, and IDR, where the random access point at FF is IDR as described above. However, I frame is not a random access point.

Therefore, when playing a video in a H.264-based video decoder by a streaming method or a file playback method, an I frame can be used as a point that can be randomly accessed in FF / REW and the like. You will also need to deal with the case of referencing a frame.

 An object of the present invention is to provide a method for selecting an intra frame (I frame) as a randomly accessible point in a video decoder.

In addition, the present invention is an H.264-based random access intra frame (I Frame), I frame that can be used as a random access point I frame composed of I NAL, and when referring to a frame that does not exist Its purpose is to provide a response.

In addition, the present invention enables fine random access by using a non-IDR frame as a random access point in an H.264-based video service, and error concealment of a reference to a frame that does not exist when starting playback in a non-IDR frame. Its purpose is to make the technique referenceable.

In addition, an object of the present invention is to provide a random access point selection method that can improve the decoding performance and quality by making the maximum use of the reference picture existing in the physical memory when decoding.

In the random access point selection method of the present invention for achieving the above object, in the method of performing a shift search (FF / REW) using a predetermined frame as an access point when decoding a video based on H.264,

Reconstructs the frame that received the NAL from the decoder and resets only the pointer containing the address of the buffer for the frame that came before the IDR NAL, and keeps the contents of the physical memory (frame buffer) from being erased. Searching at which time position it belongs; Making an I frame accessible by determining whether to apply error concealment or jump to the next IDR based on the number of frames stored in the frame buffer when the number of reference frames (num_ref_frames) is 2 or more; Characterized in that comprises a.

In the present invention, when the reference frame does not exist during the decoding, one of the adjacent reference frames is selected.

In the present invention, when the reference frame does not exist, the reference frame is referred to the nearest reference frame in time, or when there are several reference frames closest in time, the reference frame is replaced with an average of a corresponding region of two frames, or all existing Among the reference frames, the one having the smallest error with the corresponding region is selected, or the region having the minimum error with the surrounding pixels of the corresponding region among all the existing reference frames is searched for.

In addition, the present invention is characterized in that it searches for which time position the frame reconstructed by receiving the NAL from the decoder according to whether gaps_in_frame_num_value_allowed_flag is used in the H.264 sequence parameter set.

Further, in the present invention, searching for which time position the frame reconstructed by receiving the NAL at the decoder belongs to is based on a time stamp.

In the present invention, if the num_ref_frames (number of frames referenced) is not '1', an error concealment method is used if there is an image to be referred to in the frame buffer, and if there is no image to be referred to in the frame buffer, it jumps to the next IDR. It is done.

The present invention prevents the frame buffer contents from being erased in order to make an I frame into a randomly accessible point, and searches for gaps_in_frame_num_value_allowed_flag so that if the frame number does not increase to a certain number, the frame obtained by receiving a NAL from the decoder and reconstructed at a certain time position Search for num_ref_frames and use error concealment if the number of frames referenced is two or more, or jump to the next IDR, and select one of the adjacent reference frames if it references a reference frame that does not exist. By doing so, the I frame / non-IDR frame can be accessed when decoding H.264 based video.

2 illustrates an example of a random access point at FF in an H.264 based video decoder according to the present invention. As shown in FIG. 2, FF can be performed by using only an IDR as a randomly accessible point (see FIG. 1). have. Here, not only the I frame but also the first P frame thereafter shows that the subsequent P frame can be used as a reference frame.

This is because the contents of the physical memory (frame buffer) are not erased as described above, so that when the contents in the physical memory are available, they can be used. This is possible because the error concealment or the jump to the next IDR is performed depending on whether the increment (gaps_in_frame_num_value_allowed_flag) is 1 or more, and the problem when the referenced frame does not exist is solved.

Therefore, the present invention can start playback not only in IDR but also in I frame, and can reduce packet loss rate during H.264 streaming, and FF / REW even in content in which only one IDR exists at the beginning of video streaming or video file. Becomes possible. In addition, the present invention can eliminate the phenomenon that all subsequent content cannot be seen by one or more packets by making an I frame / non-IDR frame accessible when streaming content having only one IDR at the beginning, and gaps_in_frame_num_value_allowed_flag Even if the frame is set, video playback can be performed without having an error search function according to random access.

Hereinafter, with reference to the accompanying drawings will be described an embodiment of the present invention.

First, in order to describe a method of making an I frame accessible in the present invention, attention is paid to the structure of the NAL unit syntax of H.264, and the problems to be considered in order to use the I frame as a randomly accessible point are described.

3 shows the NAL unit syntax of H.264. The NAL unit syntax 1 byte indicates the type of each NAL and whether the corresponding NAL is used as a reference. Here, FZB (forbidden_zero_bit) corresponding to the first byte of Zeal is fixed to '0' in ISO 14496-10 and is not used.

The portion used in FF / REW used for shift / search in video decoding / playback is NAL_UNIT_TYPE_syntax, and respective values are defined as shown in FIG. In the case of FF / REW, the NAL whose value is XXX1001 ₂ (x = don't care) can be found and used as the starting point.

However, when the IDR is used, the coding gain is reduced. The reason is that since the contents existing in the frame buffer are erased, the frames coming after the IDR cannot be predictively encoded using the image before the current IDR frame. In contrast, in the case of I frames, the coding gain is increased because the contents of the frame buffer are not deleted. That is, the same coding gain is applied to the currently compressed IDR frame or I frame, but the compression for the images following the IDR uses only the frames after the IDR, and in the case of the I frame, the content before the I frame can also be written. As a result, even if both perform compression using spatial redundancy, the coding gain is higher in the case of I frames than in the case of IDR frames.

On the other hand, apart from the coding gain, the content provider often inserts only I frames instead of IDR frames due to incorrect encoding. Such cases are often found in practically served mobile terminal wireless networks. In such a case, since an accessible IDR frame cannot be searched previously, the probability of generating a decoding error is absolute. Therefore, in case of inserting only I frame instead of IDR frame in this way, FF / REW is not possible and playback can be restarted when packet loss occurs. Therefore, the packet is downloaded by the user from the service provider. But you can't see the video. This causes a problem due to unnecessary billing and has the disadvantages and inconveniences of paying for the video that the user cannot see.

Therefore, a solution to the above-mentioned problem should be suggested. In the present invention, the use of an I frame as an accessible point overcomes the problems of inserting only an I frame during encoding and impossibility of playing a video when packet loss occurs.

Meanwhile, FIG. 5 shows the front portion of the slice header of H.264. Basically, considering the slice type (slice_type) of FIG. 5, if the frame is configured to be randomly accessed even if the frame is composed of only I slice, the following problems are excluded except the existence of the previous frame in the frame buffer. This remains.

That is, in the case of H.264, there is a syntax called gaps_in_frame_num_value_allowed_flag, which is information describing whether or not the frame number is allowed to be arbitrarily increased in the sequence parameter set. This is a syntax that allows the frame number to be incremented to any value greater than 1, rather than always being +1. This is a syntax for omitting an error search for a frame number so that it can be used in an error free situation. However, there is content created by setting a flag to enable gaps_in_frame_num_value in cases where the network is not actually error-free. Therefore, in this case, it is necessary to consider whether to use the gaps_in_frame_num_value_allowed_flag.

In summary, all the problems to be considered in order to use an I frame as a point that can be accessed randomly are as follows.

1. Make sure that the frame buffer contents are not erased.

2. Consider whether to use gaps_in_frame_num_value_allowed_flag.

3. The reference frame must be determined based on whether or not num_ref_frames (number of frames referenced) is one or more.

4. There is a need for a treatment for non-existent reference frames.

In the case of the first problem in the present invention, the pointer containing the frame buffer address is reset in the H.264 decoder, but the contents of the physical memory can be solved. In other words, for a frame before IDR NAL, only the pointer containing the address of the buffer is reset and the contents of the physical memory (frame buffer) are kept from being erased. This makes it possible to use the contents stored in the memory when the contents in the physical memory are available. In other words, the contents (video information) of the frame memory (buffer) are left as they are without being erased, and the pointer value is reset to the corresponding frame value (frame_number) the next time the frame is to be decoded or referenced.

The second problem, that is, a method of considering whether to use gaps_in_frame_num_value_allowed_flag can be solved by searching a temporal position of a frame. In other words, if gaps_in_frame_num_value_allowed_flag is used, since the frame number does not increase to a certain number, the decoder cannot know which temporal position the frame reconstructed by receiving NAL belongs to. Solve the problem.

One method is to judge based on a time stamp in the case of streaming. For example, the time stamp of the received packet and the time stamp of the last packet used to reconstruct the current frame are used to determine the temporal resolution. That is, the time stamp of the packet received before and the time stamp of the last packet used for reconstructing the current frame are used to determine which temporal position the frame reconstructed by receiving the NAL from the decoder belongs to.

Another case is that the file is stored in a file format. For example, the video file may be downloaded and stored and then played back. However, even in this case, the time stamp can be used to determine the temporal position of the frame. This method is also similar to streaming.

Alternatively, regardless of the methods listed above, if num_ref_frames (number of frames referenced) is '1', it can be played without considering the video file playback, and in case of streaming, only the packet loss is known. If you can.

Here, the gaps_in_frame_num_value_allowed_flag should be considered along with the number of frames (num_ref_frames) referenced.

If num_ref_frames (the number of frames referenced) is '1', only one frame buffer is stored, so it is irrelevant whether it is an IDR frame or an I frame. However, gaps_in_frame_num_value_allowed_flag is a problem.

For example, if the frame number increases from 0 → 1 → 2 → 3 → 4 → 5 → 6, whether or not an error occurs in frame 2 → frame 4 depends on whether gaps_in_frame_num_value_allowed_flag is set. If it is not set, an error has occurred. In this case, the decoder should respond according to the specified method (decoder frame error processing method ).

On the contrary, if the frame number increases by more than '1' even though gaps_in_frame_num_value_allowed_flag is not set, an appropriate error cnnceal method should be used. The most common and easiest way is to find the next IDR. In such a case, it is desirable to find the next IDR since there is no content available for error concealment and it will continue to affect the next frames.

On the other hand, if num_ref_frames (number of frames referenced) is not '1', it is determined whether to use error concealment or jump to the next IDR (NEXT IDR) based on the number of frames stored in the frame buffer.

For example, when num_ref_frames (number of frames referenced) is '3', an error concealment technique is used when the reference image present in the frame buffer is 2 or more. If there is no image to be referred to in the frame buffer, the next IDR is jumped to.

Here, the reference using the error concealment technique is possible when at least one reference picture exists in the frame buffer. If there is no image to reference in the frame buffer, the error concealment technique cannot be used, so there is no choice but to jump to the next IDR.

The fourth problem to consider in order to use the I frame as a randomly accessible point, that is, when referring to a frame that does not exist, is a problem that is substantially encountered in the aforementioned error concealment method. This is because error concealment is impossible if there is no previous frame to refer to in error concealment. Therefore, in case of referring to the frame which does not exist, the following methods are taken.

One way is to make the reference frame that does not exist modify the address of the frame buffer to the frame closest in time. In this way, if the address of the frame buffer is corrected to the frame closest to time, the existing reference frame is referred to, thereby solving the above problem.

Another method is to cope with the case where two frames close to the temporal reference frame can be generated when the frame nearest the temporal frame is designated as the reference frame. In this case, when there are two reference frames that are close in time, the reference frame may be replaced by an average of corresponding regions of the two frames. Alternatively, the smallest error (SAD: Sum of Absolute Difference or MAD: Mean Absolute Difference) is selected from two frames. By selecting the one with the smallest error, the quality of error concealment from the frame referenced in error concealment is improved.

Another approach to the case of referring to a non-existent reference frame is to select the one with the smallest error with the corresponding region among all existing reference frames. By selecting the one with the smallest error, the quality of error concealment from the frame referenced in error concealment is improved.

The final coping method for referring to a reference frame that does not exist is to search for a region where the error with the boundary pixel of the corresponding region is minimized among all the existing reference frames. For example, a technique such as motion estimation is applied. By selecting the one with the smallest error, the quality of error concealment from the frame referenced in error concealment is improved.

FIG. 6 schematically illustrates a method in which a P frame searches for a peripheral area of a corresponding area with reference to an I frame.

According to the present invention, when a video service is provided based on H.264, random access can be made possible even in an I frame that is a non-IDR picture. This enables FF / REW at more detailed intervals for file-based video services, while jumping to the next I-frame or next IDR instead of jumping to the next IDR for streaming-based video services. In addition, it is possible to reduce the packet loss rate in the streaming-based video service by enabling the packet to be determined to be packet loss.

Claims (9)

In the video information decoding method, Acquiring image information including reference information referring to a picture not available in the buffer; And, Decoding network abstract layer (NAL) unit information that does not have IDR (Instaneous Decoding Refresh) information included in the obtained image information; The decoding of the NAL unit information, Decoding using any one of a first method using an I frame included in the acquired image information, and a second method using at least one of next IDR information that is close in time or recovery information corresponding to an error concealment. Video information decoding method characterized in that. The method of claim 1, wherein the reference information includes: when decoding image information including reference information referring to a picture not available in the buffer; To reference the nearest reference frame in time, If there are several frames that are closest in time, you can refer to a frame that is the average of the pixel values of the corresponding region of two frames Select the smallest pixel error with the corresponding area among all existing reference frames, And retrieving and referring to an area of which there is a minimum error with neighboring pixels of a corresponding area among all existing frames. delete delete delete delete delete delete delete

Images