JP5256095B2 - Compressed image noise removal device and playback device - Google Patents

Description

  This patent is directed to digital video equipment that reproduces digitally compressed images.

  As background art in this technical field, for example, there is JP-A-6-70173 (Patent Document 1). According to the publication, “[Purpose] Prevents ringing from occurring in a reproduced image. [Configuration] The edge determination unit 22 determines whether an edge exists from the state of an image obtained by combining encoded images for each block. For the block having an edge, the plane approximation unit 32 performs plane approximation on the areas on both sides of the edge to create an approximate image, and DCT this approximate image to convert it to the frequency domain. This is supplied to the arithmetic unit 38 and added to the decoded image in the frequency domain at a predetermined internal ratio, that is, the high frequency region lost by the quantization processing is added in the arithmetic unit. Therefore, the ringing in the reproduced image can be prevented by the processing in the arithmetic unit. "

JP-A-6-70173

  In recent years, the spread of digital broadcasting has made it possible to easily handle high-definition images at home. This digital image has a large amount of information, and is usually compressed by MPEG2, H.264 or the like, which is an international standard system for moving image compression. In this compression, the amount of data is reduced by frequency-converting image information and cutting high-frequency components by quantization. By reducing the amount of data due to this quantization, in the compressed image, blurring of the image and mosquito noise occur due to lack of high-frequency components.

  With respect to the removal of these noises, a method for predicting reduced frequency components has been proposed (Japanese Patent Laid-Open No. 6-70173). However, in this patent, noise is reduced by creating an approximate image using only the image information of the target block and adding the high-frequency component thereof. Depending on the accuracy of the approximate image, the image may be different from the original image (original image) on which image compression is performed, and may be different from the corrected image and the peripheral image, thereby reducing the quality of the reproduced image.

  In particular, in the normal video compression, the quantization width control provides a wider quantization width for the edge portion based on human visual characteristics. It is considered that the quality of the corrected image is also lowered.

  When a digital compressed image is reproduced, a high-quality reproduced image is desired by accurately predicting a component lost due to quantization. An object of the present invention is to reproduce a compressed image with high image quality.

  In order to achieve the above object, the configuration described in the claims is adopted.

  According to the present invention, a compressed image can be reproduced with high image quality.

  Problems, configurations, and effects other than those described above will be clarified by the following description of embodiments.

FIG. 2 is a diagram showing one aspect of the basic concept of the present invention in the first embodiment. It is a figure which shows the structure of a 2nd Example. It is a figure which shows the structure of a 3rd Example. It is a figure which shows use of the peripheral block data in a 2nd Example. It is a figure which shows each signal component at the time of carrying out frequency conversion of the signal which assumes the image signal in a 2nd Example. FIG. 6 shows an image of quantizing the signal component shown in FIG. 5 in the second embodiment. It is a figure which shows the structure in the case of using peripheral block data in a 2nd Example. It is a figure which shows the structure of the noise removal using original image evaluation calculation in a 3rd Example. It is a figure which shows the specific table of a coefficient in the 3rd Example, and the specific example of coefficient correction. It is a figure which shows the structure of a 4th Example. It is a figure which shows the structure of a 5th Example. It is a conceptual diagram which performs noise removal using a prediction error. It is an example of a correlation between a motion vector and motion. It is a figure which shows the difference between a quantization object block and a motion estimation block. It is a figure which shows an input picture order and a compression order. It is a figure which shows an image application structure. It is a figure which shows an image application structure.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  The first embodiment will be described with reference to the drawings. This example illustrates one aspect of the basic concept of the present invention. FIG. 1 is a diagram showing one aspect of the basic concept of the present invention. (1) in FIG. 1 represents an original coefficient 101, which indicates a coefficient having a certain component (value). (2) of FIG. 1 shows an example in which the original coefficient is quantized with a certain quantization width 102. Quantization is to approximate a continuous amount of analog data or the like to a discrete value of digital data or the like. When the original coefficient is quantized with a certain quantization width, the component of the original coefficient is approximated by the quantization width and becomes the quantized coefficient 107.

  Even in digital data compression, rather than using the original coefficients as they are, compression processing of a large amount of data is realized by using the quantization width and the quantized coefficients.

  Next, a method for restoring the digital data compressed by this quantization will be described. For restoration, the restoration coefficient 105 is restored from the quantization width and the post-quantization coefficient. This process is sometimes called inverse quantization. This process is an example of restoring the value to the median value in the quantization width region. The position to be restored in this quantization width is defined on both the quantization side and the inverse quantization side.

  The difference between the restored coefficient, which is the restored component, and the value of the original coefficient before restoration is referred to as a quantization error 104. This is noise caused by digital data compression. Regarding the relationship between the quantization width and the compression ratio, generally, increasing the quantization width increases the data compression ratio but increases the quantization error, and decreasing the quantization width decreases the quantization error. The compression ratio becomes smaller.

A specific example of the processing so far will be shown. For example, consider that quantization is performed with a quantization width of 20 when the original component has a value of 55. In this case, the information used for compression is the post-quantization coefficient 2 and the quantization width 20. When inverse quantization is performed based on this information, when the value is restored to the intermediate value of the quantization width, the restoration coefficient is 50, and as a quantization error,
(Original coefficient) 55-(Restoration coefficient) 50 = 5
Will occur.

  As for the removal of this quantization error, noise is removed from the restoration coefficient based on the peripheral information and the nature of its components. It is necessary to correct the components in the quantization width region used for the quantization. If the reconstruction coefficient component exceeds the quantization width region used for inverse quantization shown in 106 by the process for removing the quantization error, it is a coefficient (image) that is different from the original coefficient (original image). It means that it has become. Therefore, in order to remove noise and faithfully restore the original coefficient, it is important to perform noise removal based on this principle, that is, noise removal within a range not exceeding the quantization width region.

  In this embodiment, an MPEG2 (ISO / IEC 13813-2) system is used as an image compression system. However, the present invention is not limited to the MPEG2 system, and can be realized by other compression systems (for example, H.264, MPEG1, MPEG4, JPEG, VC1).

  First, an outline of the present embodiment will be described. At the time of image compression (encoder side), image information is decomposed into frequency components and quantized. The quantization width at that time is stored in the stream with the quantization value, the quantization matrix, and the quantization width of the intra DC. On the decoder side, information on the quantized coefficient and the quantization width can be acquired, so it is possible to infer which value range the coefficient was in at the time of encoding. When noise removal is performed on the decoding side, values are corrected so that the coefficient after inverse quantization falls within the quantization width range.

  FIG. 5 shows respective signal components (503 to 510) when the frequency of a signal assuming an image signal is converted. FIG. 5 shows an example of conversion into 8-dimensional components for simplification. In MPEG2, the conversion to 64 dimensions is performed by using a frequency conversion called DCT (Discrete Cosine Transform) for performing this conversion every 8 × 8. The horizontal axis 501 in FIG. 5 indicates the frequency, and indicates that it is a high frequency component as it goes to the right. The vertical axis 502 in FIG. 5 indicates the value of each frequency component, and the larger the absolute value, the more frequency components exist. Next, in the image compression, quantization processing is performed on this frequency component.

  FIG. 6 shows an image for quantizing the signal component shown in FIG. In the quantization, the quantization width 601 is individually calculated for each signal, and quantization is performed for each component. The quantization width 601 is determined by a quantization value given for each macroblock which is a processing unit for encoding, and a quantization matrix given for each picture. In general, it is preferable for image compression to increase the quantization width for higher frequency components. Image compression is realized by performing quantization separately for each component with this quantization width and using the quantized coefficient and the quantization width as a stream.

  For decoding (restoration and expansion) of this stream, the frequency component is restored from the quantization coefficient and the quantization width, and the frequency component is restored to the image information by performing inverse frequency conversion. Incidentally, in the MPEG2 standard, information necessary for quantization includes a quantization value (quantizer_scale), a quantization matrix (non_intra / intra_quantizer_matrix), an intra DC quantization width (intra_dc_precision), and intra / inter switching information (macroblock_intra). It is written. Also in other standards, information necessary for performing quantization is targeted.

  In the pre-compression image and the post-image restoration image, noise is generated due to the quantization error as described in the first embodiment. This noise is generally called mosquito noise or block noise. The removal of these noises can be realized by applying various filters to the restored image. However, the restoration of the restored image for which noise removal is performed must be performed within the range of the quantization width used for compression, resulting in an image different from the image before compression. In order to remove noise and faithfully restore an image before compression, it is necessary to remove noise based on this principle.

  FIG. 2 shows the configuration of this embodiment. The left part of FIG. 2 shows the MPEG2 decoder means 201. An MPEG2 Video stream is input to a VLD (Variable Length Decoding) block 202, quantized information 205 for calculating a quantized coefficient 208 and a quantization width 209 from the stream by variable-length inverse coding, a motion vector, and the like The motion information 206 is converted. The quantized coefficient is inversely quantized by the inverse quantization unit 203, and inverse frequency transform is performed by the inverse frequency transform unit. The quantization information 205 that is the quantization width is output from the VLD unit to the inverse quantization unit and the information holding unit. The motion information 206 is output to the motion compensation unit 207. The motion compensation unit generates a predicted image from the previous decoded image based on the motion information, and adds it to the information of the inverse frequency transform unit, thereby generating a decoded image 210 to be displayed. This decoded image is stored in the motion compensation unit for use as a reference image for the next picture. The above is the description of the decoder means. This is the same operation as a normal MPEG2 decoder. Since the image decoded by MPEG2 includes mosquito noise and block noise that are quantization noise, noise removal is performed on the right block. This noise removal is performed for each 8 × 8 block which is a DCT unit of MPEG decoding processing.

  The noise determination unit 250 will be described. The decoded image generated by the decoder is input to the noise determination unit. Here, it is determined for each block to be decoded (8 × 8 pixel unit in MPEG2) whether the target block should perform noise removal based on decoded image information, quantization information, picture information, and peripheral image information. A lot of noise in the compressed image occurs when an edge image is included. This occurs because the edge contains a lot of high-frequency components and these high-frequency components are lost due to quantization. Therefore, in this block, it is analyzed whether or not edge information exists for the decoded image generated by the decoder means. As an analysis method, a difference between pixels is taken, and when the difference value is larger than a certain threshold value, detection is performed such as an edge. Further, an edge detection filter may be used. A method for determining noise by analyzing the quantization value of the target block is also conceivable. This has a characteristic that high-frequency noise is less noticeable due to human visual characteristics in the encoding of moving images, and a block including a steep edge tends to increase the quantization value. Therefore, when the quantization value of this block is large, it may be determined that the portion includes edge information. Furthermore, edge detection with higher accuracy is possible by making a comprehensive determination using both (image information and quantization information).

  In addition, a moving image has a feature that noise at a portion where the motion of the image is intense is not noticeable. Therefore, it may be determined not to remove noise in a portion where movement is intense.

  There may be a mode in which no noise is performed depending on the user's preference. It may be determined not to perform noise removal based on such user designation.

  The noise removal unit 251 will be described. In this block, noise removal is performed only on the blocks that are determined to be subjected to noise deletion by the noise determination unit. The noise removal method is performed by predicting missing data due to quantization in the frequency domain during image compression. The basic concept of this method is shown below. With respect to the decoded image, the edge is more likely to be an edge and the noise is removed by guessing the pattern from the surrounding image and separating the edge and the noise. This operation creates a predicted image from which noise has been removed. Information usage from the peripheral images will be described later.

  The frequency conversion unit 252 will be described. In this block, frequency conversion is performed on the image from which noise has been removed. The frequency conversion method is a method corresponding to the inverse frequency conversion. In the case of MPEG, DCT is used.

  The quantization width check unit 253 will be described. In this block, it is checked whether the image after noise removal is present in the region within the quantization width from the quantized coefficient and the quantization width.

  If all the coefficients fall within the quantization width by this quantization width check, it is determined that the correction is appropriate, and the image is transferred to the display image holding unit 254 as a display image. If the quantization width check results in out of the quantization width, the information may be transferred to the noise removal unit again to remove noise, or the image before noise removal is displayed as a display image. You may transfer to an image holding part.

  Next, the use of peripheral block data will be described with reference to FIG. FIG. 4 shows a decoded image 401 that has been decoded. Decoding is performed on a rectangular macroblock 402 basis. The quantization value can also be set for each macroblock. The diagram on the left side of FIG. 4 is a scene in which a background 404 and an object 403 are photographed, and a lot of quantization noise 405 occurs in a macroblock including both the object and the background. An example of the step 406 of the quantized value 407 at this time is shown in the diagram on the right side of FIG. This is a quantization value set at the time of encoding, and is generally quantized more finely at a flat portion and roughly at an edge portion due to human visual characteristics. In such a case, it is more likely that effective information can be acquired when using information from a block with a small quantized value in using peripheral block information to remove noise from a certain block. This is because the block having a small quantized value has few missing high-frequency data and the quality of the reproduced image is considered good. Further, in the left diagram, for a block including an object and a background, noise can be removed more efficiently by performing noise removal using background information 408 for the background and object information 409 for the object. Specifically, this peripheral information is used as information when creating a predicted image of the noise removal unit. In addition to the method using the peripheral information, it is effective to use information on past and future frames, information on luminance and color difference, and the like.

  FIG. 7 shows a configuration for using this peripheral block. By adding a memory unit 701 in addition to the configuration of FIG. 2, the quantized value and the decoded image are held and used in the noise removing unit.

  In this memory unit, a frame after noise removal may be held and used for noise removal after the next frame. In this case, motion compensation is performed on the image after noise removal using motion information, and noise is removed as a predicted image.

  Alternatively, the quantized value may be held in the memory unit and used for noise removal using the quantized value. This is because the quantization magnitude differs between I-pictures and P / B-pictures, and noise removal for each picture is evenly performed by removing noise using the quantized values in I-pictures continuously. It is possible to process, and it is possible to realize high-quality noise removal in moving images.

  The noise removal in the second embodiment is configured to perform noise removal on the decoded image and check whether the image after the noise removal is within the quantization width. In the third embodiment, in the middle of the decoder An embodiment for correcting the frequency component will be described.

  FIG. 3 shows the configuration of this embodiment. The decoder means 201 in the figure is the same as in the second embodiment.

  The frequency conversion unit 301 will be described. The frequency conversion unit performs frequency conversion on the predicted image output from the motion compensation unit. The frequency conversion method is a method corresponding to the inverse frequency conversion unit 204. DCT for MPEG.

  Then, the output 302 of the frequency conversion unit and the inverse quantization unit 203 is added 302. Thereby, the same information as that obtained by DCT of the decoded image is created. The operations 301 and 302 are operations for acquiring the frequency conversion information of the decoded image, and may be a system in which DCT is performed on the decoded image created by the decoder means.

  The correction width calculation unit 303 will be described. The information on the quantized coefficient and the quantization width is obtained from the decoder means, and the correction width of the frequency component that does not exceed the quantization width region is calculated for the frequency component of the decoded image.

  The inverse frequency conversion unit 305 will be described. This block is a block that creates a display image by performing inverse frequency conversion on the frequency components from which noise has been removed.

  The display image holding unit 254 is a block that holds a display image.

  The noise determination unit 306 will be described. The information obtained by applying DCT to the decoded image generated by the adder 302 is used as an input to determine whether noise removal is performed on the target block. Basically, it has the same function as the noise removal unit 250 in the figure. However, since the input is a frequency component, the determination is based on the frequency component.

  In this block, noise determination may be performed in the image area by performing inverse frequency conversion. Further, the determination may be made using the peripheral block, the past, and the future block. However, in that case, it is necessary to hold information to be determined using a memory.

The noise removal unit 304 will be described. The noise removing unit performs noise removal on the frequency component information of the decoded image output from the adding unit 302. The range of correction of the value by removing noise in the frequency component is corrected within the range calculated by the correction width calculation unit. At this time, the following method can be considered as a correction method.
-Pattern the components of the original image such as a natural image so that the pattern is close to the frequency component of the decoded image.
・ Method shown in FIG. 8 (described later)
The former method will be described in detail. This is a method of removing noise on frequency components. Natural images and artificial images are generally clear images with little noise. When these images are frequency-converted using frequency conversion such as DCT, a characteristic conversion result is obtained. Coefficients obtained by frequency conversion of various patterns of natural images and artificial images are tabulated. Then, the pattern closest to the quantized coefficient is searched from the table, and the quantized coefficient is corrected.

  FIG. 9 shows a specific example of coefficient table formation and coefficient correction. Reference numeral 901 denotes a frequency conversion coefficient. In this example, since a case where 8 × 8 DCT is used is assumed, there are 64 coefficients. The upper left coefficient is a DC component 904 (DC component), the right direction represents the low frequency component from the horizontal component to the high frequency component, and the lower direction represents the high frequency component from the low frequency component of the vertical component. When creating a table, DCT is applied to a natural image or an artificial image, and a high frequency component group 903 (solid color in the figure) is obtained by combining low frequency component groups 902 (hatched blocks in the figure) that are low-frequency clusters. Block) components are tabulated in advance. This is because it is desirable that the post-quantization coefficients have such a configuration because many high-frequency components are removed by quantization and a lot of information remains in the low-frequency components. Further, the separation between the low frequency component group and the high frequency component group can be determined by the memory capacity of the table that can be secured. If a large number of tables can be secured, it is necessary to make the low-pass county wider and vice versa. The coefficient can be corrected more correctly by increasing the low-frequency area county.

  Next, a method for correcting the quantized coefficient using the table will be described. With reference to the above table with the low-frequency component group coefficient of the coefficient after inverse quantization at the time of decoding, the high-frequency component group is extracted and the value of the high-frequency component group of the coefficient after inverse quantization is corrected. In particular, when the high-frequency component group of the coefficient after inverse quantization is 0, it is effective to replace the value with reference to the table value. At this time, it is desirable to check the quantization width described in the first embodiment for each wide-area component group.

  Also, the coefficient after inverse quantization may be corrected by referring to information on similar blocks from the peripheral blocks as shown in FIG. 4 and the frequency components of past and future images.

  Next, the latter will be described. FIG. 8 shows the configuration of noise removal using the original image evaluation calculation. In addition to the configuration of FIG. 3, an original image evaluation calculation 802 block is newly added.

  The original image evaluation calculation unit calculates an evaluation value for the display image input from the inverse frequency conversion unit using an evaluation function. This evaluation function is preferably an evaluation function that shows a more characteristic numerical value in a natural image, an artificial image, or the like. In calculating the evaluation function, the information in the processing target block is not used alone, but may be calculated by using information in the peripheral block and past / future frames. In that case, the data used for the evaluation value calculation is held using a separate memory.

  The noise removing unit 801 corrects the frequency component information of the decoded image generated by the adding unit 302 within the correction range in which the frequency component is changed by the noise removing unit calculated by the correction width calculating unit. Then, inverse frequency conversion is performed, and the evaluation value is calculated by the current picture evaluation value calculation unit. The value is fed back to the noise removal unit again. This operation is performed every time the frequency component of the decoded image is corrected. Then, the frequency component whose evaluation value indicates the most original image quality is adopted as the corrected component.

  Next, the use of surrounding block information when creating a predicted image will be described. In moving picture coding, the quantization value is switched in units of blocks inside the picture. For example, in MPEG2, image compression can be performed in accordance with the tendency of an image in a frame by switching in block units of 16 × 16 pixels. In a block with a small quantized value, high-frequency data is hardly lost and the quality of the reproduced image is considered good. Therefore, the adjacent block of the block to be processed is investigated, and if there is a block with a small quantized value, the information of that block is actively used. Specifically, in predictive image generation, an adjacent pixel of an adjacent block having a small quantized value is set as a predicted image value of the main block. By doing so, an effect of reducing the block distortion generated between the blocks can be expected. The quantization information of the peripheral blocks is realized by the information holding unit. An example is shown in FIG. The image data of the surrounding blocks is used for the decoded image in consideration of the quantization value increment.

  In this embodiment, H.264 (ITU-T H.264) / MPEG4 AVC (ISO / IEC 14496-10) and VC1 (moving image compression methods developed by Microsoft and standardized on Windows (registered trademark) Media Video 9 The configuration in the case of ( Also in these video coding standards, the original noise removal based on the principle of the first embodiment is performed. Further, the present embodiment has almost the same concept as the system of the second embodiment, but the encoding tool is different because the encoding standards to be handled are different, and the system of the second embodiment cannot be applied as it is. Therefore, the difference part from Example 2 is demonstrated in detail especially.

  In these standards, a filtering process called an in-loop filter (deblocking filter) is performed on the decoded image.

  FIG. 10 shows a configuration diagram of this embodiment. The video stream is decoded by variable length coding in the VLD unit 1002, and the quantization information 1005, the motion information 1006, and the block mode for calculating the quantization width such as the image data and the quantization value and matrix information. It is divided into sub-information such as information. The image data undergoes inverse quantization processing by the inverse quantization unit 1003, and inverse frequency conversion is performed by the inverse frequency conversion unit 1004. At this time, quantization information is supplied to the inverse quantization block. In H.264 and VC1, the block division size is 4x4 to 8x8, and the size is variable. Such information shall be supplied to each block from the VLD block.

  On the other hand, the motion information 1006 is supplied to the motion compensation unit 1007 to perform motion compensation. The decoded image 1010 is generated by adding the motion compensated data and the inverse frequency transform unit.

  In addition, in the standard of this embodiment, there is a mode of predictive encoding using only information in a decoded frame called intra prediction, but in this embodiment, this motion compensation unit also performs an intra prediction process. The decoded image is subjected to block noise removal by a DF unit 1011 that performs a deblocking filter. These processes are based on the image of a normal H.264 / VC1 decoder, and are referred to as decoder means 1001.

  The decoded image 1010 is input to the noise determination unit 1050 and determines whether noise removal is performed. Then, noise removal is performed by the noise removal unit 1051. The frequency conversion unit 1052 performs frequency conversion, and the quantization width check unit 1053 checks the quantization width. Then, the DF2 unit 1055 applies a deblocking filter, and the display image holding unit 1054 holds and displays it.

  The noise determination unit, noise removal unit, and quantization width check unit will be described. This block basically has the same functions as the noise determination unit, noise removal unit, and quantization width check unit in FIG. However, in the standard of this embodiment, since the frequency conversion size is variable from 4x4 to 8x8, processing is performed according to the frequency conversion size. The block division size can be known by acquiring the DCT size from the sub-information in the VLD block.

  The DF2 part will be described. In this block, a deblocking filter is applied to an image from which noise has been removed by the noise removal unit, or a block that has been judged to be unnecessary for noise judgment by the noise judgment unit. However, the image from which noise has been removed by the noise removing unit has already been subjected to noise removal, and therefore it is not necessary to apply a deblocking filter in this block.

  Further, even when noise removal has already been performed, whether to perform the deblocking filter or not may be switched depending on the result.

  The DF2 unit is a block that has the same function as the DF unit except that it determines whether or not to apply a deblocking filter when noise removal is performed by the noise removal unit. Also good.

  The fourth embodiment is an example in which noise is removed from a decoded image, but an example in which noise is removed in the frequency domain is described in the present embodiment. The basic idea is the same as that of the third embodiment, but since the handling standards are different, the encoding tools are different and cannot be applied as they are. In this embodiment, the difference between H.264 and VC1 will be described in detail.

  FIG. 11 shows a configuration for performing noise removal on the frequency domain in the case of standards such as H.264 and VC1. The decoder unit 1101 has the same configuration as the decoder unit 1001 of FIG.

  The frequency conversion unit 1151 is a block that converts the frequency of the motion-compensated data created in the decoder device. When performing frequency conversion, it is necessary to perform frequency conversion in accordance with the DCT block size. Information on these DCT blocks can be obtained from the VLD block.

  The adder 1152, the frequency conversion unit 1151, the noise determination unit 1160, the noise removal unit 1155, the correction width calculation unit 1153, the inverse frequency conversion unit 1154, and the display image holding unit 1157 have substantially the same functions as the configuration in FIG. doing. However, in the standard of the present embodiment, the DCT block size is variable block division of 4 × 4 to 8 × 8, and the quantization is also executed with the same block size. Therefore, the processing is performed in consideration of this block size.

  The DF2 unit 1156 will be described. This block has substantially the same function as DF2 in FIG.

  The basic concept of the present embodiment will be described using a one-dimensional signal as an example with reference to FIG. The present embodiment is a method of performing noise removal using noise prediction target data after subsequent noise reduction when performing noise removal of digital data.

  First, in order to explain the basic concept, the compression 1201 will be explained. The top diagram shows the original signal before compression, the horizontal axis 1202 indicates the time direction, and the original signal shows the nth signal 1203 and the n + 1th signal 1204 in time order. Yes. In this example, the nth signal will be described as the first signal.

  An example of compressing these original signals using quantization and prediction is shown in the next stage compression method 1205. Since the head signal n does not exist at the previous time, quantization 1206 is performed only on the n-th signal and compression is performed. Next, n + 1 compression is performed. At this time, prediction 1207 of the n + 1th data is performed using the nth data. In the case of performing this prediction, it is necessary to perform prediction using n 1206 after quantization after performing the nth quantization and performing data compression. This is because the quantization error 1208, which is the difference between the original signal and the quantized signal, propagates to the next signal due to prediction unless prediction is performed in consideration of errors generated by quantization.

  The difference between the n + 1 prediction data 1208 predicted using the quantized data and the original image n + 1 (same as 1204) is the prediction error n + 1 1209. Quantization 1210 is also performed on the prediction error to generate prediction error quantized data n + 1 1211. A method of performing compression while performing prediction and quantization is generally used.

  An example of post-compression data created by these compression methods is shown in 1212. The post-compression data is configured in the order of data 1213 of post-quantization data 1206, data 1214 at the time of prediction 1207, and quantized data n + 1 1215 of prediction error. After that, although not shown in the figure, it is generally configured in the order of prediction data of next signal information and prediction error data.

  Here, the data in the prediction 1207 is information on how the quantized data is used as the prediction data. For example, when the size is changed, an enlargement / reduction ratio or the like can be considered. In the case of two-dimensional data, vector information or the like applies to this information.

  Then, decompression processing and noise removal processing are performed using the compressed data. The decompression 1216 performs data restoration based on the compressed data 1212. The decompressed data n 1217 is created by performing inverse quantization on the data 1213 after the quantization. Next, the decompressed data n + 1 1218 is predicted from the decompressed data 1217 using the predicted data 1214, and the data obtained by dequantizing the prediction error data 1215 is added to the data.

  Noise removal is performed on the expanded data. When noise removal is performed on the expanded data n 1217, noise-removed data n 1219 is created using the expanded data 1218 using the expanded data 1217 as a prediction.

  This is because when a signal with little change in the original signal is compressed, the first signal and the quantized data (a signal with a lot of quantization) with a large quantization width are compressed as a prediction error of the next compressed data. Since it may exist in the data, it is possible to remove noise from a noisy signal using the prediction error signal.

  Next, this concept is applied to image compression. Also in image compression, most of the international standard methods of image compression are methods combining prediction and quantization, and this concept can be applied. However, the image is configured in two dimensions, and quantization is performed by defining a two-dimensional small block and performing conversion to the frequency domain, and for prediction, a two-dimensional motion vector is defined and the past Predicted using images already compressed in the future. Although it is not defined as an international standard system, prediction may be performed in a spatial direction in which an image formed in time is already compressed.

When performing noise removal, it is necessary to consider the following points when applying the prediction error of the data which is the prediction for the target image. That is, the original image of the target image for which noise removal is performed and the original image of the image using it as a prediction are greatly different, which may cause a prediction error. In that case, if a prediction error is added to the noise target image, the desired noise cannot be removed. It is desirable to confirm any or all of the following:
1. 1. There is no motion vector used for prediction (motion vector is 0 value), or the value is similar to surrounding motion vectors. 2. The quantization value obtained by quantizing the prediction error and the quantization value of the noise removal target block are the same or the former is smaller. Details of each of the confirmation of the quantization width as described in the first embodiment when applying the prediction error will be described.

  1. If the original image has changed significantly, the motion detection vector also tends to point in various directions. In this case, it is not desirable to perform noise removal of the present concept. In this noise removal, it is assumed that similar original images have a certain spatial size and continue in time. When the same original image continues still and continuously, the motion vector value is 0 in a certain area, and when the image is moving in parallel, the direction is generally indicated. Here, the certain area is at least twice the block size for performing vector search. 2. Will be described. Assume that there is no significant change in the original image. A quantization error (N1) generated when a certain picture is compressed appears in a prediction error (Y1) generated by using it for prediction. The prediction error is compressed by quantization, but if the width is not smaller than the width of the quantization value used for N1, the prediction error is lost due to quantization. If a prediction error is lost due to quantization, it cannot be used to compensate for N1 error (noise removal) to be performed in this embodiment.

  3. Will be described. As described in the first embodiment, when removing noise, it is desirable to adjust components without a quantization width. In order to compensate for the error of N1, Y1 referring to the data is used as noise removal data. However, when it is used, it is possible to confirm whether N1 falls within the quantized quantization width. desirable.

  Next, the difference in position between the area where noise is removed and the area used for prediction will be described with reference to FIG. When image compression is performed, the image is divided into certain blocks and compression such as quantization is performed. Here, the image n (1401) is a frame that is compressed before n + 1 (1402) in the compression order, and the image n + 1 is used to perform compression using the image n + 1 for prediction. To do. Assume that the image n is divided into a certain block (1403) and quantized. Incidentally, the block size to be quantized is, for example, 8 × 8 for MPEG1, MPEG2 and MPPEG4, 8 × 8 or 4 × 4 for H.264, and 8 × 8, 8 × 4, 4 × 8 and 4 × 4 pixel units for VC1.

  Now, a description will be given focusing on a block (1404) having an image n. When performing noise removal on this block, an area in which this block is used for prediction in the image n + 1 is calculated using the prediction vector of the image n + 1. Next, the prediction unit of the image n + 1 is shown at the bottom of the figure. When predicting and compressing the image n + 1, the image is divided into blocks of a certain size and a prediction vector is calculated in units of the block. For example, when attention is paid to two blocks (1405, 1406), the predictions are the positions of 1407, 1408 in the image n, respectively. From this, it is conceivable that predictions of various blocks overlap in the noise removal target block 1404.

  Some positions are not predicted from anywhere. For this, it is desirable to take the following scheme. For the pixel in the 1404 block, if the portion being predicted is alone, the prediction error of the data is applied only to the control portion. If no reference is made from anywhere, the noise of the pixel is not removed. Alternatively, the pixels at the relative positions of the portions used in the prediction of the surrounding blocks are used for noise removal. Next, consider a case where a plurality of predictions are made.

  As for prediction, FIG. 15 shows an example of a prediction frame for image compression. In this example, the MPEG2 prediction relationship is taken as an example. The upper part of the figure shows the input frame order (1501) when compression is performed. Assume that B0-> B1-> I2-> B3-> B4-> P5 are input in order of time. The lower part shows the compression order (1502) when compression is performed. At this time, frames using I2 as a prediction are B0 (1503), B1 (1504), and P5 (1505). Similarly, B3, B4, and P8 perform predictive compression using P5 as a prediction. Thus, there is a possibility that a plurality of pictures are referred to in I2 and P5. In this case, when I2 is considered from the order of the input images, B1, B0, and P5 are in order in time.

Based on this, it is desirable that which prediction is adopted when prediction is made from a plurality of FIG. 14 as follows.
・ Check the quantized value of each predicted block and the one with the smallest quantized value. ・ The frame that is close in time to playback. Things Confirm either or both of these to adopt the best prediction. Prediction data is selected by this process, and the above-described prediction error selection is executed to remove noise.

  Next, a configuration when the present method is applied to image processing is shown. FIG. 16 shows the configuration. An example of this application will be described with reference to FIG. Each block in FIG. 16 has the same function as in FIG. 7, but the noise removal of this embodiment is added as a function of the noise removal unit (251). Also, the quantization check unit 253 performs the quantization width check of the present embodiment.

  Next, the memory unit (1601) will be described. In the memory unit, decode information (VLD information, quantization information, decoded image, predicted image, etc.) is held, and when the frame is considered to be predicted, the information is read from the memory unit and removed. I do. The frames that are considered to be predicted can be determined by stream analysis such as IPB configuration in MPEG1 / 2/4 and ref_frame and DPB configuration information in H.264. However, in order to prevent an increase in the configuration size of the memory, for example, it is possible to simplify such as using prediction data of only I and P pictures.

  As another configuration, a configuration in which the retained information is restored after being read again can be considered. For example, in FIG. 17, the stream distribution unit (1702) decodes a frame that may have predicted the frame prior to decoding the noise-removed frame. For example, in the example of FIG. 15, B0, B1, and P5 are decoded in advance before decoding I2 to be denoised, and at that time, quantization information, motion information, error image, etc. Is stored in the memory unit (1701). Then, I2 which is a noise removal target frame is decoded, and necessary information is read from the memory at that time, and noise removal is performed by the noise removal unit.

  As mentioned above, although preferred embodiment concerning this invention was described referring an accompanying drawing, it cannot be overemphasized that this invention is not limited to this example. For example, the above-described embodiments have been described in detail for easy understanding of the present invention, and are not necessarily limited to those having all the configurations described. It is obvious for those skilled in the art that various changes or modifications can be conceived within the scope of the technical idea described in the claims. It is understood that it belongs to.

201 Decoder means
202 VLD (Variable Length Decoding) block
208 Post-quantization coefficient
209 Quantization width
205 Quantization information
206 Motion information
203 Inverse quantization section
207 Motion compensation unit
210 Decoded image
250 Noise judgment part
252 Frequency converter
253 Quantization width check section

Claims (12)

A VLD unit that decodes a digital image-compressed stream into image data and sub-information, an inverse quantization unit that inversely quantizes the image data decoded by the VLD unit, and an inverse frequency converter that performs inverse frequency conversion on the inversely quantized data. A decoder means comprising: a frequency conversion unit; and a motion compensation unit that performs motion compensation based on the sub-information decoded by the VLD unit;
A noise judgment unit for judging whether or not the noise removal for each block for the de coder means display viewing images produced from,
By correcting the coefficient output from the inverse quantization unit within the quantization width included in the sub-information output from the VLD unit for the block determined to perform noise removal by the noise determination unit , A noise removing unit for removing noise ;
The output image of the noise removal section have a, a display image holding unit for holding a Viewing image,
The noise removal unit performs noise removal based on a motion vector value of a block referring to the noise removal target block in a future or past frame in a display order with respect to the noise removal target frame or a block around the block. A compressed image noise removing apparatus characterized by performing noise removal on a block . 2. The compressed image noise removal apparatus according to claim 1, wherein the noise removal unit has a motion vector value of a block that refers to a noise removal target block in a future or past frame in a display order with respect to the noise removal target frame. In this case, the compressed image noise removing apparatus is characterized in that noise removal is performed on a target block for noise removal based on the block . 2. The compressed image noise removal apparatus according to claim 1, wherein the noise removal unit moves a block around a block that refers to a noise removal target block in a future or past frame in a display order with respect to the noise removal target frame. A compressed image noise removing apparatus , wherein when a vector value is similar, noise removal is performed on a target block for noise removal based on the block . 2. The compressed image denoising device according to claim 1, wherein when the quantization value obtained by quantizing the prediction error is equal to or smaller than the quantization value of the noise removal target block, noise removal is performed on the noise removal target block. Compressed image noise removal device. 2. The compressed image noise removing apparatus according to claim 1, wherein the sub information used in the noise removing unit includes a quantization value (quanter_scale) used for quantization, a quantization matrix (quantiser_matrix), and an intra DC quantization width (intra_dc_precision). ) Is a compressed image noise removing device. 2. The compressed image noise removing apparatus according to claim 1, wherein whether to remove noise is determined based on edge information and a quantization value of the block . 2. The compressed image noise removal apparatus according to claim 1, wherein the correction value of the inverse quantization coefficient when performing noise removal is calculated using image data of a block having a small quantization value of an adjacent block of this block. Compressed image noise removing device. 2. The compressed image noise removing apparatus according to claim 1, wherein the decoding means is MPEG2, which is an international standard system for image compression. 2. The compressed image noise removing apparatus according to claim 1, wherein the decoding means is H.264 which is an international standard system for image compression. Input means for inputting a compressed image stream to which image data is frequency converted into coefficients, the coefficients are quantized with a predetermined quantization width, and quantization information indicating the quantization width is added;
  Inverse quantization means for dequantizing the compressed image stream input to the input means using a quantization width indicated by the quantization information into a restoration coefficient;
  Correction means for correcting the restoration coefficient inversely quantized by the inverse quantization means;
  Correction range limiting means for limiting the range corrected by the correction means using the quantization width indicated by the quantization information;
  Reverse frequency conversion means for performing reverse frequency conversion on the restoration coefficient corrected by the correction means;
  Output means for outputting the image data subjected to inverse frequency conversion by the inverse frequency conversion means,
  The correction unit is configured to perform a noise removal target block based on a motion vector value of a block referring to the noise removal target block in a future or past frame in a display order with respect to the noise removal target frame or a block around the block. A playback device characterized in that noise removal is performed. 11. The reproducing apparatus according to claim 10, wherein the correction unit has a motion vector value of a block that refers to a noise removal target block in a future or past frame in a display order with respect to the noise removal target frame is zero. A reproduction apparatus characterized by performing noise removal on a target block for noise removal based on the block. 11. The reproduction apparatus according to claim 10, wherein the correction unit is a motion vector value of a block around a block that refers to a noise removal target block in a future or past frame in a display order with respect to the noise removal target frame. If the two are similar to each other, a reproducing apparatus is characterized in that noise removal is performed on a target block for noise removal based on the block.

Images