WO2013077305A1

WO2013077305A1 - Image decoding device, image decoding method, image coding device

Info

Publication number: WO2013077305A1
Application number: PCT/JP2012/080020
Authority: WO
Inventors: 貴也山本; 内海　端
Original assignee: シャープ株式会社
Priority date: 2011-11-22
Filing date: 2012-11-20
Publication date: 2013-05-30
Also published as: JP2013110643A

Abstract

Provided is an image coding method with which it is possible to obtain superior coding efficiency. An image coding method which codes each macroblock into which an image is segmented comprises a step of generating a predict image of a macroblock to be coded, on the basis of a predict mode when macroblocks which are adjacent to the macroblock to be coded are intra-predicted, and the pixel values of the adjacent macroblocks.

Description

[Name of invention determined by ISA based on Rule 37.2] Image decoding device, image decoding method, image encoding device

The present invention relates to an image encoding method, an image encoding device, an image decoding method, an image decoding device, and a program thereof.

In video coding, various methods such as intra-frame prediction and motion compensation inter-frame prediction have been proposed, and MPEG (Moving Picture Experts Group) -2, MPEG-4, MPEG-4 AVC (Advanced Video Coding) / H. H.264.
As described in Non-Patent Document 1, MPEG-4 AVC / H. One of the methods adopted in H.264 is a skipped macroblock mode. This method does not encode the information of the macroblock to be encoded at all, and when decoding, generates a prediction vector based on the motion vector of the macroblock adjacent to the encoded macroblock, and uses that prediction vector. An encoded macroblock is decoded by performing motion compensation.

In Patent Literature 1, when multi-view video is encoded, a global difference indicating a global difference between an encoding target image and an image of a viewpoint different from the encoding target image is used instead of such a prediction vector. A technique for predicting a motion vector of a current block based on a vector and using it as a prediction vector is disclosed. Patent Document 1 also discloses that encoding and decoding are performed in a skipped macroblock mode using this prediction vector.

Special table 2010-516158

However, in the techniques of Non-Patent Document 1 and Patent Document 1 described above, when the skipped macroblock mode is applied to an image having a high spatial correlation such as a depth map in which the encoding target image represents depth information, the skipped macroblock mode is applied. There is a problem that high coding efficiency may not be obtained because the macroblock mode is a method mainly based on correlation in the time direction.

The present invention has been made in view of the above situation, and an object of the present invention is to provide an image coding method and image code capable of obtaining excellent coding efficiency even in an image having a high spatial correlation. An image decoding apparatus, an image decoding method, an image decoding apparatus, and a program thereof are provided.

(1) The present invention has been made to solve the above-described problems, and one aspect of the present invention is an image encoding method for encoding each macroblock obtained by dividing an image, and includes a macro to be encoded. A generation step of generating a prediction image of the encoding target macroblock based on a prediction mode when the macroblock adjacent to the block is intra-predicted and a pixel value of the adjacent macroblock; To do.

(2) According to another aspect of the present invention, there is provided the above-described image encoding method, wherein the prediction image is generated by the generation step for each of the encoding target macroblocks based on the adjacent macroblocks. It has the step which selects whether it produces | generates, It is characterized by the above-mentioned.

(3) According to another aspect of the present invention, there is provided the above-described image encoding method, wherein the generation step generates the prediction image by intra prediction based on the prediction mode and the pixel value. Features.

(4) According to another aspect of the present invention, there is provided the above-described image encoding method, wherein the generation step includes: an adjacent direction of the adjacent macroblock to the macroblock to be encoded; The step of determining the prediction mode of the intra prediction is determined by a combination with the prediction mode of the adjacent macroblock.

(5) Further, another aspect of the present invention is an image encoding apparatus that encodes each macroblock obtained by dividing an image, and prediction when a macroblock adjacent to a macroblock to be encoded is intra-predicted A prediction image generation unit that generates a prediction image of the encoding target macroblock based on the mode and the pixel value of the adjacent macroblock is provided.

(6) According to another aspect of the present invention, there is provided an image decoding method for decoding an image encoded for each macroblock obtained by dividing an image, wherein a macroblock adjacent to a macroblock to be decoded is intra-predicted. And a generation step of generating a prediction image of the macroblock to be decoded based on a prediction mode and a pixel value of the adjacent macroblock.

(7) According to another aspect of the present invention, there is provided an image decoding apparatus for decoding an image encoded for each macroblock obtained by dividing an image, and intra prediction is performed on a macroblock adjacent to a decoding target macroblock. And a prediction image generation unit that generates a prediction image of the macroblock to be decoded based on the prediction mode and the pixel value of the adjacent macroblock.

(8) According to another aspect of the present invention, prediction is performed when a computer of an image encoding apparatus that encodes each macroblock obtained by dividing an image performs intra prediction on a macroblock adjacent to the macroblock to be encoded. This is a program for operating as a prediction image generation unit that generates a prediction image of the encoding target macroblock based on the mode and the pixel value of the adjacent macroblock.

(9) According to another aspect of the present invention, a computer of an image decoding apparatus that decodes an image encoded for each macroblock obtained by dividing an image, and intra-predicts a macroblock adjacent to the macroblock to be decoded. This is a program for operating as a prediction image generation unit that generates a prediction image of the macroblock to be decoded based on the prediction mode and the pixel value of the adjacent macroblock.

According to the present invention, it is possible to obtain excellent encoding efficiency even in an image having a high correlation in the spatial direction.

1 is a schematic diagram illustrating a configuration example of a three-dimensional image capturing system including an image encoding device according to a first embodiment of the present invention. It is the schematic which shows the structural example of the image coding apparatus which concerns on the 1st Embodiment of this invention. It is the schematic which shows the prediction mode of intra prediction. It is the schematic which shows the structural example of the skipped macroblock part which concerns on the 1st Embodiment of this invention. It is a conceptual diagram which shows an example of the encoding by the skipped macroblock mode which concerns on the 1st Embodiment of this invention. It is a flowchart for demonstrating an example of the image coding process which the image coding apparatus which concerns on the 1st Embodiment of this invention performs. It is a flowchart following FIG. It is a flowchart for demonstrating an example of the skipped macroblock prediction image macroblock production | generation process in the skipped macroblock part which concerns on the 1st Embodiment of this invention. It is a flowchart following FIG. It is the schematic which shows the structural example of the image decoding apparatus which concerns on the 1st Embodiment of this invention. It is a flowchart for demonstrating an example of the image decoding process which the image decoding apparatus concerning the 1st Embodiment of this invention performs.

In each of the following embodiments, the depth map is encoded or decoded in the skipped macroblock mode for each macroblock which is a partial area constituting the depth map including the depth value for each pixel. Of course, as will be apparent from the following description, encoding / decoding is not limited to skipped macroblock mode for all macroblocks. Each embodiment has a main feature in generating a predicted image. More specifically, in the skipped macroblock mode in each embodiment, when a macroblock is predictively encoded, the prediction image generation method is changed according to the encoding mode of an adjacent block.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.
(First embodiment)
FIG. 1 is a schematic diagram illustrating a configuration example of a three-dimensional image capturing system including an image encoding device according to the first embodiment of the present invention.
A three-dimensional image capturing system 10 illustrated in FIG. 1 includes an image capturing device 3a, an image capturing device 3b, an image pre-processing unit 2, an image encoding device 1, and a texture image encoding device 4.
The photographing device 3a and the photographing device 3b are installed at different positions (viewpoints), and images of subjects included in the same field of view (photographing space) are obtained at predetermined time intervals (for example, one every 1/30 seconds). Times) to shoot. The imaging device 3 a and the imaging device 3 b define an image signal indicating the captured image as a texture image (sometimes referred to as “two-dimensional image”), and output each texture image to the image preprocessing unit 2. To do.

This image signal is a signal value (pixel value, luminance value) representing the color or shade of the subject or background included in the subject space, and is an image signal composed of signal values for each pixel arranged on a two-dimensional plane. is there. This image signal is an image signal having a signal value representing a color space for each pixel, for example, an RGB signal. The RGB signal includes an R signal that represents the luminance value of the red component, a G signal that represents the luminance value of the green component, and a B signal that represents the luminance value of the blue component.

Of the images input to the image preprocessing unit 2, one of the image capturing device 3a and the image capturing device 3b, for example, a texture image input from the image capturing device 3a is defined as a reference image, and another image capturing device (in this example, The description will be made separately from the texture image input from the photographing apparatus 3b). Therefore, the reference image is also a signal value (luminance value) representing the color and shading of the subject and background included in the subject space, and is a general two-dimensional signal composed of signal values for each pixel arranged on the two-dimensional plane. It is an image signal.

The image preprocessing unit 2 uses the texture image input from the imaging device 3a as a reference image, calculates the parallax between the reference image and the texture image input from the other imaging device 3b for each pixel, and the imaging device A depth map corresponding to the texture image input from 3a is generated. The depth map (also referred to as depth map, “depth image”, or “distance image”) is a signal value (“depth value”) corresponding to the distance from the viewpoint (shooting device, etc.) of the subject or background included in the subject space. ”,“ Depth value ”,“ depth ”, etc.), and is an image signal composed of signal values (pixel values) for each pixel arranged in a two-dimensional plane. The pixels constituting the depth map correspond to the pixels constituting the reference image. Therefore, the depth map is a clue for representing the three-dimensional object space by using the reference image, which is a reference image signal obtained by projecting the object space onto the two-dimensional plane.

Next, the image preprocessing unit 2 uses the texture image input from the other image capturing device 3b as a reference image, and generates a depth map of the image capturing device 3a from the parallax with the texture image input from the image capturing device 3a. Similarly, a depth map corresponding to the texture image of the photographing apparatus 3b is generated. The image preprocessing unit 2 outputs the texture images of the image capturing device 3a and the image capturing device 3b to the texture image encoding device 4 alternately for each frame. In addition, the image preprocessing unit 2 outputs a depth map corresponding to the texture image to the image encoding device 1. That is, the image preprocessing unit 2 outputs a depth map corresponding to the texture image to the image encoding device 1 at the timing when the texture image is output to the texture image encoding device 4.

The texture image encoding device 4 converts the input texture image into, for example, ITU-T H. A texture image code is generated by encoding using an encoding method corresponding to a decoding method defined in the H.264 standard.
In the present embodiment, the number of imaging apparatuses provided in the image capturing system is described as two. However, the number is not limited to this, and may be three or more. Further, the depth map and the texture image respectively input to the image encoding device 1 and the texture image encoding device may not be based on images captured by the imaging device 3a and the imaging device 3b, for example, computer graphics These images may be pre-synthesized.

Next, a configuration example and functions of the image encoding device 1 according to the present embodiment will be described with reference to FIG. FIG. 2 is a schematic diagram illustrating a configuration example of the image encoding device 1 according to the present embodiment.
The image encoding apparatus 1 illustrated in FIG. 2 includes a depth map input unit 101, a reference image memory 102, an intra prediction unit 103, an inter prediction unit 104, a skipped macroblock unit 105, a coding mode determination unit 106, and a motion vector memory. 107, an encoding mode memory 108, a switch 109, a subtraction unit 110, a DCT transform / quantization unit 111, an inverse quantization / inverse DCT transform unit 112, an addition unit 113, and an information source coding unit 114. .

The depth map input unit 101 receives the depth map from the image preprocessing unit 2 for each frame, and extracts a macro block (hereinafter, depth macro block) from the input depth map. The depth map input unit 101 outputs the extracted depth macroblock to the encoding mode determination unit 106, the intra prediction unit 103, the inter prediction unit 104, and the subtraction unit 110. The depth macroblock includes a predetermined number of pixels, for example, 16 pixels in the horizontal direction × 16 pixels in the vertical direction.

The depth map input unit 101 shifts the position of the macro block from which the depth macro block is extracted in the raster scan order so that the blocks do not overlap. That is, the depth map input unit 101 sequentially moves the block from which the depth macroblock is extracted from the upper left end of the frame to the right by the number of pixels in the horizontal direction of the block. After the right end of the macroblock from which the depth macroblock is extracted reaches the right end of the frame, the depth map input unit 101 moves the macroblock downward by the number of pixels in the vertical direction of the macroblock and to the left end of the frame. The depth map input unit 101 moves the macroblock from which the depth macroblock is extracted in this way until it reaches the lower right of the frame. Hereinafter, the extracted depth macroblock may be referred to as an encoding target macroblock or a target macroblock.

The reference image memory 102 arranges the macro block of the reference image (hereinafter referred to as reference image macro block) input from the adding unit 113 at the position of the encoding target macro block in the corresponding frame and stores it as a reference image. This reference image macroblock is a macroblock obtained by encoding and decoding a depth macroblock input in the past. An image signal of a frame configured by arranging reference image macroblocks in this way is called a reference image. The reference image memory 102 sequentially stores a reference number of frames set in advance from the present, for example, 15 frames of past reference images. When the number of stored frames exceeds the preset number of frames, for example, When the 16th frame reference image is input, all the previous frame reference images are deleted.

The intra prediction unit 103 receives the target macroblock from the depth map input unit 101, and is a macroblock adjacent to the target macroblock in the same frame as the target macroblock from the reference image memory 102, and performs processing related to encoding. Read the reference image macroblock. Next, the intra prediction unit 103 performs intra-frame prediction based on the read reference image macroblock, and generates an intra predicted image macroblock. As a specific intra-frame prediction method, for example, ITU-T H. There is an intra-frame prediction method described in the H.264 standard.

FIG. 3 shows a prediction mode (prediction direction) for a luminance signal in intra prediction defined in Non-Patent Document 1. When the target macroblock is divided into blocks of 4 × 4 or 8 × 8 and intraframe prediction is performed, nine prediction modes from 0 to 8 are provided as shown in FIG. In the case where the target macroblock is not divided and the size is 16 × 16, four prediction modes from 0 to 3 are determined as shown in FIG. 3B. Note that Intra_DC in FIGS. 3A and 3B is a method of predicting a target macroblock or subblock with an average value of pixels adjacent on the left and above the target macroblock or subblock. Also, Intra_Plane in FIG. 3B is a method of interpolating pixel values in the diagonal direction between pixels adjacent to the target macroblock or sub-block and pixels adjacent to the left to obtain predicted values.

The intra prediction unit 103 regards the depth value of the read reference image macroblock as a luminance value, and performs the above-described intraframe prediction. In addition, the intra prediction unit 103 divides the target macroblock into 16 patterns of 4 × 4 blocks, a pattern that is divided into 8 × 8 blocks, and a pattern that is not divided (block size is 16 × 16). A prediction mode in which an intra prediction image macroblock that is most similar to the target macroblock is generated is selected. As an index of whether the target macroblock and the intra predicted image macroblock are similar, for example, there is SAD (SumSof Absolute Difference, sum of absolute differences). The intra prediction unit 103 compares the SADs of the intra predicted image macroblocks of each pattern, and outputs the intra predicted image macroblock of the pattern having the smallest SAD to the coding mode determination unit 106. In addition, this invention is not limited by the specific degree calculation method of a specific object macroblock and an intra prediction image macroblock.

The inter prediction unit 104 divides the target macroblock input from the depth map input unit 101 into one or a plurality of partitions, and sequentially detects a motion vector for each partition. The division into one partition means that the division is not substantially performed.
Specifically, the inter prediction unit 104 first reads a reference image block constituting a reference image from the reference image memory 102. The size (number of pixels) of the reference image block read by the inter prediction unit 104 is the same as the size of the partition to be processed (hereinafter referred to as the target partition) among the one or more partitions. Here, the inter prediction unit 104 performs block matching between the target partition in the depth macroblock input from the depth map input unit 101 and the reference image block. That is, the position where the reference image block is read from the frame of the past reference image stored in the reference image memory 102 is moved pixel by pixel in the horizontal direction or the vertical direction within a preset range from the position of the target partition. The inter prediction unit 104 detects a difference between the coordinates of the target partition and the coordinates of the reference image block most similar to the target partition as a vector.

Here, the inter prediction unit 104 uses an index value indicating the similarity between the signal value for each pixel included in the target partition and the signal value for each pixel included in the reference image block, for example, SAD (Sum of Absolute Difference, difference (Absolute value sum) is calculated. The inter prediction unit 104 determines the reference image block that minimizes the index value as the reference image block that is most similar to the target partition. The inter prediction unit 104 calculates a motion vector based on the coordinates of the target partition and the coordinates of the most similar reference image block.

Also, the inter prediction unit 104 selects a partition size that minimizes the coding cost among the sizes of the partitions constituting the target macroblock. For example, the inter prediction unit 104 performs the following processing on the sizes of all partitions. First, the inter prediction unit 104 calculates the sum of the absolute value of the calculated horizontal component of the motion vector, the absolute value of the vertical component thereof, and the index value between the target partition and the reference block most similar to the target partition as the cost value. To do. The above processing is performed on all partitions constituting the target macroblock to calculate the cost value, and the sum of the cost values is obtained for each partition size. Then, the partition size with the smallest sum of cost values is selected from among the partition sizes.

As another partition size selection method, for example, an RD cost is used. With the RD cost, the encoding cost J is calculated from the following equation.
J = D + λ × R
Here, D is a difference between the target macroblock and an image obtained by encoding and decoding the target macroblock, and R is an expected generated code amount. The coefficient λ may be a constant or a function of a quantization parameter that controls the roughness of quantization in the DCT transform / quantization unit 111. The present invention is not limited to a specific encoding cost calculation method.

Next, the inter prediction unit 104 performs motion compensation based on the size of the partition having the minimum cost value and the motion vector calculated for each partition of the size, and generates an inter predicted image macroblock. Specifically, first, the inter prediction unit 104 reads a reference image block at coordinates indicated by motion vectors for each partition from the reference image memory 102. The inter prediction unit 104 generates an inter predicted image macroblock by performing processing for matching the coordinates of the read reference image block with the coordinates of the partition for each partition.
The inter prediction unit 104 outputs the generated inter prediction image macroblock to the encoding mode determination unit 106, and outputs the generated motion vector to the information source encoding unit 114 and the switch 109.

The skipped macroblock unit 105 reads the encoding mode signal of the block adjacent to the target macroblock from the encoding mode memory 108. Similarly, the motion vector of the block adjacent to the target macroblock is read from the motion vector memory 107, and the pixel value of the reference image block adjacent to the target macroblock is read from the reference image memory 102.

The skipped macroblock unit 105 generates a skip prediction image macroblock based on the coding mode signal, the motion vector, and the pixel value of the block adjacent to the target macroblock. The skipped macroblock unit 105 outputs the skip predicted image macroblock to the encoding mode determination unit 106. Further, the skipped macroblock unit 105 may generate a prediction vector from the motion vectors of adjacent blocks. When the skipped macroblock unit 105 generates a prediction vector, the skipped macroblock unit 105 outputs the generated prediction vector to the switch 109. Details of the processing will be described later.

The coding mode determination unit 106 receives the target macroblock from the depth map input unit 101, the intra prediction image macroblock from the intra prediction unit 103, the inter prediction image macroblock from the inter prediction unit 104, and the skipped macroblock unit 105. Each skip prediction image macroblock is received as an input.

Then, the coding mode determination unit 106 determines a coding mode with the highest coding efficiency based on the three input predicted image macroblocks. Specifically, for example, the encoding mode determination unit 106 calculates the square error of the signal value for each pixel for each predicted image macroblock and the target macroblock. Then, the encoding mode determination unit 106 sets the sum of the square errors in each prediction image macroblock as the encoding cost of the prediction image macroblock. The encoding mode determination unit 106 calculates the encoding cost for all prediction image macroblocks, and sets the encoding mode corresponding to the prediction image macroblock with the lowest encoding cost to the encoding mode with the highest encoding efficiency. Choose as. As another determination method, for example, there is a method using RD cost. The present invention is not limited to a specific encoding cost calculation method.

The coding mode determination unit 106 outputs a coding mode signal indicating a coding mode with the highest coding efficiency to the information source coding unit 114, the coding mode memory 108, and the switch 109. The encoding mode determination unit 106 outputs the predicted image macroblock corresponding to the encoding mode with the highest encoding efficiency to the subtraction unit 110 and the addition unit 113. In the present embodiment, the coding mode that can be indicated by the coding mode signal includes a skipped macroblock mode indicating that the skipped macroblock unit 105 has generated a predicted image macroblock.

The motion vector memory 107 regards the motion vector or the prediction vector input from the switch 109 as a motion vector, and stores them in the order of the encoded macro blocks.
The encoding mode memory 108 stores the encoding mode signal indicating the encoding mode selected by the encoding mode determination unit 106 input from the encoding mode determination unit 106 in the order of the encoded macro blocks. The encoding mode signal indicates the encoding mode selected by the encoding mode determination unit 106 and also includes information indicating what partition the target macroblock is divided into. In this embodiment, the size of the partition is 16 pixels horizontally × 16 pixels vertically, 16 pixels × 8 pixels, 8 pixels × 16 pixels, 8 pixels × 8 pixels, 8 pixels × 4 pixels, 4 pixels × 8. Although it is a pixel or 4 pixels × 4 pixels, the present invention is not limited to a specific partition size. Further, when the encoding mode selected by the encoding mode determination unit 106 is intra prediction, the encoding mode signal indicates intra prediction as the encoding mode and also includes information indicating the prediction mode of intra prediction.

The switch 109 receives an encoding mode signal corresponding to the encoding mode selected from the encoding mode determination unit 106, receives a motion vector from the inter prediction unit 104, and receives a prediction vector from the skipped macroblock unit. The switch 109 outputs the received motion vector to the motion vector memory 107 when the received encoding mode signal indicates inter prediction. The switch 109 outputs the received prediction vector to the motion vector memory 107 when the received coding mode signal indicates the skipped macroblock mode.

The subtracting unit 110 uses the intra-predicted image macroblock, the inter-predicted image macroblock input from the coding mode determining unit 106, or the signal value of each pixel included in the depth macroblock input from the depth map input unit 101, or A residual signal macroblock is calculated by subtracting the signal value of each pixel included in the skip predicted image macroblock. The subtraction unit 110 outputs the calculated residual signal macroblock to the DCT transform / quantization unit 111.

The DCT transform / quantization unit 111 performs a two-dimensional DCT (Discrete Cosine 、 Transform) on the residual signal macroblock input from the subtraction unit 110 to calculate a DCT coefficient as a frequency domain signal. Coefficients are quantized to generate quantized DCT coefficients. Here, the quantization is a process of converting an input DCT coefficient into one of discrete representative values. The DCT transform / quantization unit 111 outputs the generated quantized DCT coefficient to the inverse quantization / inverse DCT transform unit 112 and the information source coding unit 114.

The inverse quantization / inverse DCT transform unit 112 performs de-quantization on the quantized DCT coefficient input from the DCT transform / quantization unit 111 to generate an inverse quantized DCT coefficient, and then performs inverse quantization. Two-dimensional inverse DCT (inverse Discrete Cosine Transform) is applied to the generalized DCT coefficient to generate a residual signal macroblock decoded as a spatial domain signal. Hereinafter, the decoded residual signal macroblock is referred to as a decoded residual signal macroblock. The inverse quantization / inverse DCT transform unit 112 outputs the generated decoded residual signal macroblock to the adder 113.

The addition unit 113 receives the signal value of each pixel included in the intra prediction image macroblock, the inter prediction image macroblock, or the skip prediction image macroblock input from the coding mode determination unit 106, and inverse quantization / inverse DCT transform The signal value of each pixel included in the decoded residual signal macroblock input from the unit 112 is added to generate a reference image macroblock. The adding unit 113 stores the generated reference image macroblock in the reference image memory 102.

The information source encoding unit 114 includes an encoding mode signal input from the encoding mode determination unit, a quantized DCT coefficient input from the DCT transform / quantization unit 111, and a motion vector input from the inter prediction unit 104. Are encoded to generate an encoded stream. The information source encoding unit 114 performs, for example, variable length encoding as encoding for generating an encoded stream with a smaller amount of information (number of bits) than input information. The variable-length coding is a coding method that compresses the amount of information by converting shorter codes for information with higher appearance frequency into longer codes for information with lower appearance frequency in input information. The information source encoding unit 114 outputs the generated encoded stream to the outside of the image encoding device 1.

Next, a configuration example and functions of the skipped macroblock unit 105 according to the present embodiment will be described with reference to FIG. FIG. 4 is a schematic diagram illustrating a configuration example of the skipped macroblock unit 105 according to the present embodiment.
The skipped macroblock unit 105 illustrated in FIG. 4 includes a prediction image generation method selection unit 1051, a prediction vector generation unit 1052, a prediction image macroblock generation unit 1053, a prediction mode determination unit 1054, and an intra prediction image macroblock. And a generation unit 1055.

The predicted image generation method selection unit 1051 reads the motion vector of the macroblock adjacent to the left, top, and top right of the target macroblock from the motion vector memory 107. When the block from which the motion vector can be read out of the three adjacent macroblocks is equal to or greater than a preset threshold (here, 2), the predicted image generation method selection unit 1051 sends the motion vector to the predicted vector generation unit 1052. Is output. If there is a block from which a motion vector cannot be read, vector information whose horizontal component and vertical component are 0 is output instead as the motion vector of the block. The prediction image generation method selection unit 1051 sends a signal indicating that processing is to be started to the prediction mode determination unit 1054 when the number of blocks from which motion vectors can be read out of three adjacent macroblocks is less than one. To do.

Thus, the predicted image generation method selection unit 1051 generates a predicted image of the target macroblock based on the pixel value of the adjacent block (generated by intra prediction) or based on the motion vector of the adjacent macroblock ( Whether to generate by inter prediction). The motion vector of each macroblock exists when the encoding mode of the macroblock is inter prediction or inter prediction of skipped macroblock mode. Therefore, as described above, the determination based on the number of adjacent macroblocks from which the motion vector can be read out means that the determination is based on the number of adjacent macroblocks for which the prediction image is generated by the intra prediction. Become. That is, the predicted image generation method selection unit 1051 generates the predicted image of the target macroblock by intra prediction when the number of adjacent macroblocks for which the predicted image has been generated by intra prediction is equal to or greater than a threshold (here, 2). Has been decided.

The prediction vector generation unit 1052 receives the motion vectors of the blocks adjacent to the left, top, and top right of the target macroblock from the prediction image generation method selection unit 1051. The prediction vector generation unit 1052 generates a prediction vector by calculating median values of the horizontal and vertical components of the received motion vector. The prediction vector generation unit 1052 outputs the generated prediction vector to the prediction image macroblock generation unit 1053 and the switch 109.

The prediction image macroblock generation unit 1053 receives the prediction vector from the prediction vector generation unit 1052. The prediction image macroblock generation unit 1053 generates a skip prediction image macroblock by performing motion compensation according to the prediction vector. The prediction image macroblock generation unit 1053 outputs the generated skip prediction image macroblock to the encoding mode determination unit 106.

The prediction mode determination unit 1054 starts processing after receiving a signal indicating that processing is to be started from the predicted image generation method selection unit 1051. The prediction mode determination unit 1054 reads the encoding mode of the block adjacent to the left, upper left, upper and upper right of the target macroblock from the encoding mode memory 108. And the prediction mode determination part 1054 determines the prediction mode of intra prediction according to the encoding mode of the adjacent block. A specific determination method will be described with reference to FIG.

As shown in FIG. 5, the macroblock adjacent to the left of the target macroblock is referred to as block A, the macroblock adjacent above is referred to as block B, the block adjacent to the upper right is referred to as block C, and the block adjacent to the upper left is referred to as block D. I decided to.
First, the prediction mode determination unit 1054 confirms whether the encoding mode of the block B indicates 0 of the prediction mode of intra prediction ((a) in FIG. 5). The prediction mode determination unit 1054 sets 0 as the intra prediction mode signal when the coding mode of the block B indicates 0 of the prediction mode of intra prediction.
Next, when the encoding mode of block B does not indicate 0 of the prediction mode of intra prediction, the prediction mode determination unit 1054 indicates that the encoding mode of block A indicates 1 of the prediction mode of intra prediction. Is confirmed ((b) of FIG. 5). The prediction mode determination unit 1054 sets 1 as the intra prediction mode signal when the coding mode of the block A indicates 1 of the prediction mode of intra prediction.

Next, when the encoding mode of block A does not indicate 1 of the prediction mode of intra prediction, the prediction mode determination unit 1054 determines that the encoding mode of block C is 16 × 16. And whether the

prediction mode

3 or 7 of the intra prediction is indicated ((c) in FIG. 5). When the block C encoding mode is not 16 × 16 and the prediction mode determination unit 1054 indicates the intra

prediction prediction mode

3 or 7, the prediction mode determination unit 1054 outputs the intra prediction mode signal. The same value as the prediction mode of block C is set.

Next, when the encoding mode of the block C is not 16 × 16 and the size of the partition of the block C is 16 × 16 and the prediction mode determination unit 1054 does not indicate the

prediction mode

3 or 7 of the intra prediction, the prediction mode determination unit 1054 It is confirmed whether the encoding mode of 4 indicates 4, 5 or 6 of the prediction mode of intra prediction ((d) in FIG. 5). The prediction mode determination unit 1054 sets the same value as the prediction mode of the block D as the intra prediction mode signal when the encoding mode of the block D indicates 4, 5 or 6 of the prediction mode of intra prediction.
Next, the prediction mode determination part 1054 sets 2 as an intra prediction mode signal, when the encoding mode of the block D has not shown 4, 5 or 6 of the prediction mode of intra prediction.
The prediction mode determination unit 1054 outputs the set intra prediction mode signal to the intra prediction image macroblock generation unit 1055.

In the present embodiment, the prediction mode determination unit 1054 performs the order of FIGS. 5A to 5D, that is, the prediction modes 0, 1, (3 or 7), (4, 5 or 6), and 2 in this order. Judgment was made as to whether it was applicable. This is generally the order from the highest occurrence frequency of the prediction mode, but other orders may be used.
The prediction mode of block B may be 0 when the block B, which is a macro block, is intra-predicted by 16 pixels × 16 pixels and the prediction mode is 0. Or it is good also as a block at the time of the block B being intra-predicted, Comprising: When the prediction mode of all the blocks adjacent to an object macroblock is 0. Or it is good also as a block at the time of the block B being intra-predicted, Comprising: When the prediction mode more than a predetermined number or more than a predetermined ratio is 0 among the blocks adjacent to an object macroblock. The same applies when the prediction mode of block A is 1.

The intra predicted image macroblock generation unit 1055 receives the intra prediction mode signal from the prediction mode determination unit 1054, and reads the pixel value of the macroblock adjacent to the target macroblock from the reference image memory 102. The intra prediction image macroblock generation unit 1055 generates a skip prediction image macroblock from the pixel values of the blocks adjacent to the target macroblock, using intra prediction based on the prediction direction of the intra prediction mode indicated by the intra prediction mode signal. To do. In addition, although an intra prediction image macroblock is 16 pixels x 16 pixels, the prediction direction according to the relationship between the prediction mode and prediction direction shown to Fig.3 (a) is used. The intra predicted image macroblock generation unit 1055 outputs the generated skip predicted image macroblock and a signal indicating that the skip predicted image macroblock has been generated by intra prediction to the encoding mode determination unit 106.

Next, image encoding processing performed by the image encoding device 1 according to the present embodiment will be described with reference to FIGS. 6 and 7. 6 and 7 are flowcharts for explaining an example of the image encoding process performed by the image encoding device 1 according to this embodiment.
First, in step S101, the depth map input unit 101 inputs a depth map from the image preprocessing unit 2 for each frame. Then, it progresses to step S102.
In step S102, the image encoding device 1 performs the processes in steps S103 to S120 for each macroblock included in the input frame. The order of steps S103 to S105 does not matter.

In step S103, the intra prediction unit 103 reads from the reference image memory 102 a reference image block that has been subjected to processing related to encoding adjacent to the target macroblock in the same frame as the target macroblock. The intra prediction unit 103 performs intra-frame prediction based on the read reference macroblock, and generates an intra predicted image macroblock. The intra prediction unit 103 outputs the generated intra predicted image macroblock to the encoding mode determination unit 106. Thereafter, the process proceeds to step S104.

In step S104, the inter prediction unit 104 inputs the target macroblock from the depth map input unit 101, and reads the reference image block constituting the reference image from the reference image memory 102. The inter prediction unit 104 divides the target macroblock into one or a plurality of partitions, and sequentially detects a motion vector for each partition. The inter prediction unit 104 selects the size of the partition that minimizes the coding cost from among the sizes of the partitions constituting the target macroblock. The inter prediction unit 104 performs motion compensation or disparity compensation based on the partition size and the motion vector of each partition with the minimum coding cost, and generates an inter prediction image macroblock. The inter prediction unit 104 outputs the calculated vector to the information source encoding unit 114 and the switch 109. The inter prediction unit 104 outputs the generated inter prediction image macroblock to the encoding mode determination unit 106. Thereafter, the process proceeds to step S105.

In step S105, the skipped macroblock unit 105 reads from the coding mode memory 108 the coding mode signal of the macroblock adjacent to the target macroblock, and from the motion vector memory 107 to the motion vector of the macroblock adjacent to the target macroblock. Then, the pixel value of the reference image macroblock adjacent to the target macroblock is read from the reference image memory 102. The skipped macroblock unit 105 generates a skip prediction image macroblock from the read encoding mode signal, the motion vector, and the pixel value of the adjacent reference image block. The skipped macroblock unit 105 outputs the skip predicted image macroblock to the encoding mode determination unit 106. If the skipped macroblock unit 105 generates a prediction vector, it outputs the generated prediction vector to the switch 109. Thereafter, the process proceeds to step S106.

In step S106, the encoding mode determination unit 106 skips the target macroblock from the depth map input unit 101, the intra prediction image macroblock from the intra prediction unit 103, the inter prediction image macroblock from the inter prediction unit 104, and the skipped macro. The skip prediction image macroblock is received as an input from the block unit 105. The encoding mode determination unit 106 selects an encoding mode with the highest encoding efficiency based on the three input predicted image macroblocks. The coding mode determination unit 106 outputs a prediction image macroblock corresponding to the selected coding mode to the subtraction unit 110 and the addition unit 113, and generates a coding mode signal corresponding to the selected coding mode. The information is output to the information source encoding unit 114, the encoding mode memory 108, and the switch 109. Thereafter, the process proceeds to step S107.

In step S107, the switch 109 receives an encoding mode signal corresponding to the encoding mode selected from the encoding mode determination unit 106, a motion vector from the inter prediction unit 104, and a prediction vector from the skipped macroblock unit 105. . The switch 109 outputs a motion vector to the motion vector memory 107 when the encoding mode signal corresponding to the selected encoding mode indicates inter prediction. The switch 109 outputs the prediction vector to the motion vector memory 107 when the encoding mode signal corresponding to the selected encoding mode indicates the skipped macroblock mode. The motion vector memory 107 stores the motion vector or prediction vector output from the switch 109 as a motion vector. Thereafter, the process proceeds to step S108.

In step S108, the encoding mode memory 108 stores the encoding mode signals corresponding to the selected encoding mode output from the encoding mode determination unit 106 in the order of the encoded macro blocks. Thereafter, the process proceeds to step S109.

In step 109, the subtraction unit 110 receives the intra prediction image macroblock and the inter prediction image macro input from the encoding mode determination unit 106 from the signal values of the respective pixels included in the target macroblock input from the depth map input unit 101. A residual signal macroblock is calculated by subtracting the signal value of each pixel included in the block or skip prediction image macroblock. The subtraction unit 110 outputs the calculated residual signal macroblock to the DCT transform / quantization unit 111. Then, it progresses to step S110.

In step S110, the DCT transform / quantization unit 111 performs two-dimensional DCT on the residual signal macroblock input from the subtraction unit 110, calculates the DCT coefficient, quantizes the DCT coefficient, Generalized DCT coefficients are generated. The DCT transform / quantization unit 111 outputs the generated quantized DCT coefficient to the inverse quantization / inverse DCT transform unit 112 and the information source coding unit 114. Then, it progresses to step S111.

In step S111, the inverse quantization / inverse DCT transformation unit 112 performs inverse quantization on the quantized DCT coefficient input from the DCT transformation / quantization unit 111 to generate an inverse quantization DCT coefficient, and then performs inverse quantization. Two-dimensional inverse DCT is performed on the DCT coefficients to generate a decoded residual signal macroblock. The inverse quantization / inverse DCT transform unit 112 outputs the generated decoded residual signal macroblock to the adder 113. Thereafter, the process proceeds to step S112.

In step S112, the addition unit 113 performs inverse quantization on the signal value of each pixel included in the intra prediction image macroblock, the inter prediction image macroblock, or the skip prediction image macroblock input from the encoding mode determination unit 106. A reference image macroblock is generated by adding the signal value of each pixel included in the decoded residual signal macroblock input from the inverse DCT transform unit 112. Thereafter, the process proceeds to step S113.
In step S 113, the adding unit 113 stores the generated reference image macroblock in the reference image memory 104. Then, it progresses to step S114 of FIG.

In step S114 of FIG. 7, according to the determination result of the encoding mode by the encoding mode determination unit 106, if the encoding mode is the skipped macroblock mode (YES in step S114), the process proceeds to step S115, otherwise (NO in step S114), the process proceeds to step S116.

In step S115, the information source encoding unit 114 encodes the encoding mode signal input from the encoding mode determination unit 106, and generates an encoded stream. Thereafter, the process proceeds to step S119.
In step S116, if the encoding mode is inter prediction (YES in step S116), the process proceeds to step S117. If the encoding mode is intra prediction (NO in step S116), the process proceeds to step S118.

In step S117, the information source encoding unit 114 receives the encoding mode signal input from the encoding mode determination unit 106, the quantized DCT coefficient input from the DCT transform / quantization unit 111, and the input from the inter prediction unit 104. The encoded motion vector is encoded to generate an encoded stream. Thereafter, the process proceeds to step S119.
In step S118, the information source encoding unit 114 encodes the encoding mode signal input from the encoding mode determination unit 106 and the quantized DCT coefficient input from the DCT transform / quantization unit 111 to generate an encoded stream. Is generated. Thereafter, the process proceeds to step S119.

In step S119, the information source encoding unit 114 outputs the generated encoded stream to the outside of the image encoding device 1. Thereafter, the process proceeds to step S120.
In step S120, the image encoding device 1 sequentially changes the macroblocks to be encoded in the raster scan order, and returns to step S103 in FIG. However, if the image encoding device 1 determines that there is no block that has not yet been subjected to encoding processing in the frame of the input depth map, it ends the processing for that frame.

Next, skip predicted image macroblock generation processing in the skipped macroblock unit 105 according to the present embodiment will be described with reference to FIGS. 8 and 9. 8 and 9 are flowcharts for explaining an example of the skip predicted image macroblock generation processing in the skipped macroblock unit 105 according to the present embodiment. This process is a process corresponding to step S105 described above.

In step S 201, the predicted image generation method selection unit 1051 reads from the motion vector memory 107 the motion vectors of macroblocks adjacent to the left, upper, and upper right of the target macroblock. Thereafter, the process proceeds to step S202.
In step S202, the predicted image generation method selection unit 1051 determines whether or not there are two or more blocks from which motion vectors can be read out among the three adjacent macroblocks. If there are two or more (Yes in step S202), the motion vector is output to the prediction vector generation unit 1052, and the process proceeds to step S203. If there are not two or more (No in step S202), the process proceeds to step S206 in FIG.

In step 203, the prediction vector generation unit 1052 receives the motion vector of the macroblock adjacent to the left, top and top right of the target macroblock from the prediction image generation method selection unit 1051. The prediction vector generation unit 1052 generates a prediction vector by calculating median values of the horizontal and vertical components of the received motion vector. The prediction vector generation unit 1052 outputs the generated prediction vector to the prediction image macroblock generation unit 1053. Thereafter, the process proceeds to step S204.
In step S 204, the predicted image macroblock generation unit 1053 receives the prediction vector from the prediction vector generation unit 1052. Thereafter, the process proceeds to step S205.
In step S205, the predicted image macroblock generation unit 1053 generates a skip predicted image macroblock by performing motion compensation according to the prediction vector. The prediction image macroblock generation unit 1053 outputs the generated skip prediction image macroblock to the encoding mode determination unit 106 and ends the process.

In step S206 of FIG. 9, the prediction mode determination unit 1054 reads out the encoding modes of block A, block B, block C, and block D adjacent to the target macroblock from the encoding mode memory 108. Thereafter, the process proceeds to step S207.
In step S207, the prediction mode determination unit 1054 confirms whether the encoding mode of block B indicates 0 of the prediction mode of intra prediction (Yes in step S207), and proceeds to step S208. If not shown (No in step S207), the process proceeds to step S209.
In step S208, the prediction mode determination unit 1054 sets 0 as the intra prediction mode signal of the target macroblock. The prediction mode determination unit 1054 outputs the set intra prediction mode signal to the intra prediction image macroblock generation unit 1055. Thereafter, the process proceeds to step S216.

In step S209, the prediction mode determination unit 1054 confirms whether the coding mode of block A indicates 1 of the prediction mode of intra prediction (Yes in step S209), and proceeds to step S210. If not shown (No in step S209), the process proceeds to step S211.
In step S210, the prediction mode determination part 1054 sets 1 as an intra prediction mode signal of an object macroblock. The prediction mode determination unit 1054 outputs the set intra prediction mode signal to the intra prediction image macroblock generation unit 1055. Thereafter, the process proceeds to step S216.

In step S211, the prediction mode determination unit 1054 confirms whether the coding mode of the block C indicates that the block size of the block C is not 16 × 16 and indicates the

prediction mode

3 or 7 of the intra prediction. If yes (Yes in step S211), the process proceeds to step S212. If not indicated (No in step S211), the process proceeds to step S213.
In step S212, the prediction mode determination part 1054 sets the same value as the prediction mode of the block C as an intra prediction mode signal of an object macroblock. The prediction mode determination unit 1054 outputs the set intra prediction mode signal to the intra prediction image macroblock generation unit 1055. Thereafter, the process proceeds to step S216.

In step S213, the prediction mode determination unit 1054 confirms whether the encoding mode of the block D indicates 4, 5, or 6 of the prediction mode of intra prediction, and if it indicates (Yes in step S213), The process proceeds to step S214, and if not shown (No in step S213), the process proceeds to step S215.
In step S214, the prediction mode determination unit 1054 sets the same value as the prediction mode of the block D as the intra prediction mode signal of the target macroblock. The prediction mode determination unit 1054 outputs the set intra prediction mode signal to the intra prediction image macroblock generation unit 1055. Thereafter, the process proceeds to step S216.

In step S215, the prediction mode determination part 1054 sets 2 as an intra prediction mode signal of an object macroblock. The prediction mode determination unit 1054 outputs the set intra prediction mode signal to the intra prediction image macroblock generation unit 1055. Thereafter, the process proceeds to step S216.
In step S216, the intra prediction image macroblock generation unit 1055 receives the intra prediction mode signal from the prediction mode determination unit 1054, and reads the pixel value of the block adjacent to the target macroblock from the reference image memory 102. The intra prediction image macroblock generation unit 1055 generates a skip prediction image macroblock from the pixel values of the blocks adjacent to the target macroblock, using intra prediction based on the prediction direction of the intra prediction mode indicated by the intra prediction mode signal. To do. The intra predicted image macroblock generation unit 1055 outputs the generated skip predicted image macroblock to the encoding mode determination unit 106, and ends the process.

As described above, in the image encoding device 1 according to the present embodiment, the encoding method when encoding in the skipped macroblock mode is changed according to the encoding mode of the block adjacent to the target macroblock. ing. Then, the prediction mode when the encoding method when encoding in the skipped macroblock mode is intra prediction is determined according to the prediction mode when the block adjacent to the target macroblock is encoded by intra prediction. is doing.

Next, the image decoding apparatus according to this embodiment will be described. The image decoding apparatus according to the present embodiment is a partial map that forms a depth map including depth values for each pixel, which is encoded in the skipped macroblock mode according to the image encoding method in the image encoding apparatus 1 described above. The image decoding part which decodes the macroblock which is an area | region is provided. Hereinafter, a configuration example of such an image decoding device will be described with reference to FIG. FIG. 10 is a schematic diagram illustrating a configuration example of the image decoding apparatus according to the present embodiment.

The image decoding apparatus 5 illustrated in FIG. 10 includes an information source decoding unit 501, an encoding mode memory 502, a motion vector memory 503, an inverse quantization / inverse DCT transform unit 504, an addition unit 505, a reference image memory 506, and an intra prediction image. A generation unit 507, an inter prediction image generation unit 508, a skipped macroblock unit 509, an encoding mode determination unit 510, a switch 511, and an image output unit 512 are configured.

The information source decoding unit 501 decodes an encoded stream for each block regarding the depth map input from the outside of the image decoding device 5, and generates a quantized DCT coefficient, a motion vector, and an encoding mode signal. The decoding method performed by the information source decoding unit 501 is the reverse of the encoding method performed by the information source encoding unit 114 of FIG. That is, when the information source encoding unit 114 performs variable length encoding, the information source decoding unit 501 performs variable length decoding. The information source decoding unit 501 outputs the generated motion vector to the inter predicted image generation unit 508 and the switch 511. When the information source decoding unit 501 generates the quantized DCT coefficient, the information source decoding unit 501 outputs the generated quantized DCT coefficient to the inverse quantization / inverse DCT transform unit 504. The information source decoding unit 501 stores the generated encoding mode signal in the encoding mode memory 502, and outputs it to the encoding mode determination unit 510, the intra prediction image generation unit 507, the inter prediction image generation unit 508, and the switch 511.

The coding mode memory 502 receives the coding mode signal from the information source decoding unit 501 and stores it in the decoding order of the macroblocks. The encoding mode memory 502 outputs the stored encoding mode signal to the skipped macroblock unit 509.
The motion vector memory 503 receives a motion vector or a prediction vector from the switch 511, and stores it in the decoding order of macroblocks. The motion vector memory 503 outputs the stored motion vector to the inter predicted image generation unit 508 and the skipped macroblock unit 509.

The inverse quantization / inverse DCT transform unit 504 performs inverse quantization on the quantized DCT coefficient input from the information source decoding unit 501 to generate an inverse quantized DCT coefficient, and then performs 2 for the inverse quantized DCT coefficient. Dimensional inverse DCT is performed to generate a decoded residual signal block as a spatial domain signal. The inverse quantization / inverse DCT transform unit 504 outputs the decoded residual signal block to the adder 505.

The adding unit 505 receives the signal value of each pixel included in any of the intra predicted image macroblock, the inter predicted image macroblock, or the skip predicted image macroblock input from the coding mode determination unit 510, and the inverse quantization / A reference image macroblock is generated by adding the signal value of each pixel included in the decoded residual signal block input from the inverse DCT transform unit 504. The adder 505 stores the generated reference image macroblock in the reference image memory 506 and outputs it to the image output unit 511.

The reference image memory 506 stores the reference image macroblock input from the adder 505 at the position of the decoding target macroblock in the corresponding frame and stores it as a reference image. The reference image memory 506 stores reference images of frames that have been set in advance from the current number of frames (for example, 15 frames), and when the number of stored frames exceeds the number of frames that has been set in advance, for example, the 16th frame. When the reference image is input, all the reference images in the past frames are deleted.

The intra prediction image generation unit 507 receives the encoding mode signal from the information source decoding unit 501. The intra predicted image generation unit 507 reads from the reference image memory 506 a reference image macroblock that has been subjected to processing related to encoding adjacent to the target macroblock in the same frame as the target macroblock. The intra-predicted image generation unit 507 performs intra-frame prediction based on the read reference image macroblock and the prediction mode indicated by the encoding mode signal, and generates an intra-predicted image macroblock. The intra predicted image generation unit 507 outputs the generated intra predicted image macroblock to the coding mode determination unit 510.

The inter prediction image generation unit 508 receives the motion vector and the encoding mode signal from the information source decoding unit 501. The inter prediction image generation unit 508 reads a reference image block constituting a reference image from the reference image memory 506 for each partition constituting the target macroblock. The size of this partition is determined based on the encoding mode signal. The coordinates of the reference image block read by the inter predicted image generation unit 508 are coordinates compensated by adding the motion vector input from the information source decoding unit 501 to the coordinates of the decoding target partition. Further, the size of the reference image block to be read is the same as the size of the corresponding partition. The inter prediction image generation unit 508 generates an inter prediction image macroblock by performing processing for matching the coordinates of the read reference image block with the coordinates of the corresponding partition. The inter prediction image generation unit 508 outputs the generated inter prediction image macroblock to the encoding mode determination unit 510.

The skipped macroblock unit 509 reads the encoding mode signal of the block adjacent to the target macroblock from the encoding mode memory 502. The skipped macroblock unit 509 reads the motion vector of the block adjacent to the target macroblock from the motion vector memory 503, and reads the pixel value of the reference image block adjacent to the target macroblock from the reference image memory 506.

The skipped macroblock unit 509 generates a skip predicted image macroblock based on the coding mode, motion vector, and pixel value of a block adjacent to the target macroblock. The skipped macroblock unit 509 outputs the skip predicted image macroblock to the encoding mode determination unit 510. Further, the skipped macroblock unit 509 may generate a prediction vector from the motion vectors of adjacent blocks. When the skipped macroblock unit 509 generates a prediction vector, the skipped macroblock unit 509 outputs the generated prediction vector to the switch 511. Details of processing and configuration in the skipped macroblock unit 509 are the same as those of the skipped macroblock unit 105 of the image encoding device 1.

The encoding mode determination unit 510 includes an encoding mode signal from the information source decoding unit 501, an intra prediction image macroblock from the intra prediction image generation unit 507, and an inter prediction image macroblock from the inter prediction image generation unit 508. The skip predicted image macroblock from the skipped macroblock unit 509 is input. The coding mode determination unit 510 outputs one of the intra prediction image macroblock, the inter prediction image macroblock, and the skip prediction image macroblock to the addition unit 505 based on the coding mode indicated by the coding mode signal. .

The switch 511 receives an encoding mode signal and a motion vector from the information source decoding unit 501 and receives a prediction vector from the skipped macroblock unit 509. The switch 511 outputs a motion vector to the motion vector memory 503 when the encoding mode signal indicates inter prediction. The switch 511 outputs the prediction vector to the motion vector memory 503 when the encoding mode signal indicates the skipped macroblock mode.

The image output unit 512 generates the reference image for each frame by arranging the reference image macroblock input from the adding unit 505 at the position of the decoding target macroblock in the corresponding frame. The image output unit 512 outputs the generated reference image to the outside of the image decoding device 5 for each frame as a decoded depth map.

Next, image decoding processing performed by the image decoding device 5 according to the present embodiment will be described with reference to FIG. FIG. 11 is a flowchart for explaining an example of image decoding processing performed by the image decoding device 5 according to the present embodiment.
In step S301, the image decoding device 5 performs the processing of steps S302 to S313 for each macroblock.
First, in step S 302, the information source decoding unit 501 inputs an encoded stream from the outside of the image decoding device 5. Thereafter, the process proceeds to step S303.

In step S303, the information source decoding unit 501 performs decoding (variable length decoding in this example) of the encoded stream input in step S302, and generates an encoding mode signal, a quantized DCT coefficient, and a motion vector. The information source decoding unit 501 outputs the generated encoding mode signal to the encoding mode determination unit 510, the encoding mode memory 502, the intra prediction image generation unit 507, and the inter prediction image generation unit 508, and generates the generated quantized DCT coefficients. Is output to the inverse quantization / inverse DCT transform unit 504, and the generated motion vector is output to the inter prediction image generation unit 508 and the motion vector memory. Thereafter, the process proceeds to step S304.

In step S304, the inverse quantization / inverse DCT transform unit 504 performs inverse quantization on the quantized DCT coefficient input from the information source decoding unit 501, generates an inverse quantized DCT coefficient, and then performs inverse quantization. A two-dimensional inverse DCT is performed on the DCT coefficients to generate a decoded residual signal block. The inverse quantization / inverse DCT transform unit 504 outputs the decoded residual signal block to the adder 505. Thereafter, the process proceeds to step S305.

In step S305, the inter prediction image generation unit 508 receives the encoding mode signal and motion vectors corresponding to all partitions in the target macroblock from the information source decoding unit 501. The inter prediction image generation unit 508 divides the macroblock into partitions based on the coding mode signal received from the information source decoding unit 501, and performs motion compensation or disparity compensation based on the motion vector for each partition. A predicted image macroblock is generated. The inter prediction image generation unit 508 outputs the generated inter prediction image macroblock to the encoding mode determination unit 510. Thereafter, the process proceeds to step S306.

In step S306, the intra predicted image generation unit 507 receives a coding mode signal from the information source decoding unit 501, and performs processing related to decoding adjacent to the target macroblock in the same frame as the target macroblock from the reference image memory 506. Read the reference image block. The intra predicted image generation unit 507 divides the macro block into blocks based on the received encoding mode signal, performs intra-frame prediction based on the reference image macro block read for each block, and converts the intra predicted image macro block into the block. Generate. The intra predicted image generation unit 507 outputs the generated intra predicted image macroblock to the coding mode determination unit 510. Thereafter, the process proceeds to step S307.

In step S307, the skipped macroblock unit 509 outputs the coding mode signal of the macroblock adjacent to the target macroblock from the coding mode memory 502, and the motion vector of the macroblock adjacent to the target macroblock from the motion vector memory 503. Then, the pixel value of the reference image block adjacent to the target macroblock is read from the reference image memory 506. The skipped macroblock unit 509 generates a skip prediction image macroblock from the read encoding mode signal, the motion vector, and the pixel value of the adjacent reference image block. The skipped macroblock unit 509 outputs the skip predicted image macroblock to the encoding mode determination unit 510. Thereafter, the process proceeds to step S308.

In step S308, the encoding mode determination unit 510 performs the encoding mode signal from the information source decoding unit 501, the intra prediction image macroblock from the intra prediction image generation unit 507, and the inter prediction from the inter prediction image generation unit 508. An image macroblock and a skip predicted image macroblock from the skipped macroblock unit 509 are input. The encoding mode determination unit 510 determines whether to output an intra prediction image macroblock, an inter prediction image macroblock, or a skip prediction image macroblock based on the encoding mode indicated by the encoding mode signal. The macro block according to the result is output to the adding unit 505. Thereafter, the process proceeds to step S309.

In step S309, the motion vector memory 503 receives the motion vector from the information source decoding unit 501, and stores it in the decoding order of the macroblocks. The motion vector memory 503 outputs the stored motion vector to the inter predicted image generation unit 508 and the skipped macroblock unit 509. Thereafter, the process proceeds to step S310.

In step S310, the encoding mode memory 502 receives the encoding mode signal from the information source decoding unit 501, and stores it in the decoding order of the macroblocks. The encoding mode memory 502 outputs the stored encoding mode signal to the skipped macroblock unit 509. Thereafter, the process proceeds to step S311.

In step S311, the addition unit 505 performs the inverse quantization on the signal value of each pixel included in the intra prediction image macroblock, the inter prediction image macroblock, or the skip prediction image macroblock input from the coding mode determination unit 510. A reference image macroblock is generated by adding the signal value of each pixel included in the decoded residual signal macroblock input from the inverse DCT transform unit 504. Thereafter, the process proceeds to step S312.

In step S 312, the adding unit 505 stores the generated reference image macroblock in the reference image memory 506 and outputs the reference image macroblock to the image output unit 512. Thereafter, the process proceeds to step S313.
In step S313, the image decoding apparatus 5 changes the decoding target macroblock in the raster scan order, and the process returns to step S302. However, if the image decoding device 5 determines that there is no macroblock that has not been decoded in the frame of the input encoded stream, the image decoding device 5 ends the processing for the frame.

As described above, in the present embodiment, when encoding is performed for each macroblock, which is a partial area constituting a depth map composed of depth values for each pixel, the encoding mode of a block adjacent to the one macroblock is encoded. Then, a depth prediction image macroblock is generated based on the motion vector and the pixel value.
Thereby, in the present embodiment, in a depth map having a high spatial direction correlation, a skipped macroblock mode is applied to an area where intra prediction frequently occurs in the depth map, that is, a region having a high spatial direction correlation. By using intra prediction at the time of encoding, a predicted image macroblock more similar to the target macroblock can be generated, and encoding efficiency can be improved.
In the present embodiment, the encoding target of the image encoding device 1 and the decoding target of the image decoding device 5 are the depth maps. However, the above-described two-dimensional texture image may be used.

(Second Embodiment)
The image encoding device 1 and the image decoding device 5 in the first embodiment described above record an image encoding program and / or image decoding program on a computer-readable recording medium, and execute the program on the computer. This can also be realized.

As shown in the first embodiment, the image encoding program uses the encoding mode signal, the motion vector, and the pixel value of the macroblock adjacent to the target macroblock when encoding the depth map. This is a program for causing a computer to execute an image encoding process for generating a skip predicted image macroblock and predictively encoding a depth map using the skip predicted image macroblock.

The image encoding processing includes a step of selecting a prediction image generation method for one macroblock based on a motion vector of an adjacent macroblock, and a step of generating a prediction vector based on a motion vector of the adjacent macroblock. , Determining a prediction mode for intra prediction from coding mode signals of adjacent macroblocks, and generating a skip prediction image macroblock based on the determined prediction mode and the pixel value of the adjacent macroblock. Other application examples are as described in the first embodiment, and a description thereof is omitted.

In addition, the above-described image decoding program is subjected to predictive coding using a skip prediction image macroblock according to the above-described image coding method (or using the above-described image coding apparatus or image coding program). This is a program for causing a computer to execute an image decoding step for decoding a macroblock which is a partial region constituting a depth map including a depth value indicating a distance from a viewpoint. Other application examples are as described in the first embodiment, and a description thereof is omitted.

In other words, the image encoding program and the image decoding program are executed when the entire image encoding device 1 and / or the entire image decoding device 5 in the first embodiment are configured to be realized by a computer. It is a program that can be incorporated. Of course, you may make it implement | achieve only a part of the image coding apparatus 1 in 1st Embodiment, and / or a part of the image decoding apparatus 5 with a computer.

The above recording medium may be a program medium such as a memory (not shown) such as a ROM (not shown) because processing is performed by a computer, and a program reading device as an external storage device (not shown) is provided in the recording medium. It may be a program medium that can be read by inserting. In any case, the stored program may be configured to be accessed and executed by a microprocessor that is a main computation / control unit of the computer, or the program read out and read out by the computer It may be downloaded to a program storage area provided and the program may be executed. In this case, it is assumed that the download program is stored in the main device in advance.

Here, the program medium is a recording medium configured to be separable from the computer main body, and includes a tape system such as a magnetic tape and a cassette tape, a magnetic disk such as a floppy disk (registered trademark) and a hard disk, and a CD-ROM / MO. / MD / DVD and other optical discs, IC cards (including memory cards) / optical cards, etc., or mask ROM, EPROM (Erasable Programmable Read Only Memory), EEPROM (Electrically Erasable Programmable Read Only Memory), It may be a medium that carries a fixed program including a semiconductor memory such as a flash ROM. In this case, since the system configuration is capable of connecting a communication network including the Internet, the medium may be a medium that fluidly carries the program so as to download the program from the communication network. When the program is downloaded from the communication network in this way, the download program may be stored in advance in the computer main body, or may be installed from another recording medium.

The above-mentioned recording medium is read by a digital color image forming apparatus or a program reading apparatus provided in a computer system, whereby the above-described image encoding method and image decoding method are executed. The computer system includes a general-purpose image input device such as a WEB camera, a computer that performs various processes such as the image processing method by loading a predetermined program, a display / liquid crystal display that displays the processing results of the computer, and the like. Image display device. Furthermore, a network card, a modem, and the like are provided as communication means for connecting to a server or the like via a network.

(Other)
In all the embodiments, encoding and decoding of a depth map corresponding to two-viewpoint image signals captured by two cameras have been described. However, for multi-viewpoint image signals captured by a plurality of cameras having more than two viewpoints. May be applied. For example, a depth map corresponding to a three-viewpoint image signal is input, and a depth map of one viewpoint is encoded by estimating a depth value using a texture image corresponding to the depth map, and an object to be encoded is encoded. By switching, the depth maps of the three viewpoints can be encoded.

Moreover, you may implement | achieve part or all of the image coding apparatus 1 and the image decoding apparatus 5 in embodiment mentioned above as integrated circuits, such as LSI (Large Scale Integration), or IC (Integrated Circuit) chip set. Each functional block of the image encoding device 1 and the image decoding device 5 may be individually made into a processor, or a part or all of them may be integrated into a processor. Further, the method of circuit integration is not limited to LSI, and may be realized by a dedicated circuit or a general-purpose processor. In addition, when an integrated circuit technology that replaces LSI appears due to the advancement of semiconductor technology, an integrated circuit based on the technology may be used.
In addition, as exemplified in the flow of processing in the image encoding device and the image decoding device, the present invention further describes the image encoding method and the image decoding method as described in the processing of the image encoding program and the image decoding program. The form as can also be taken.

In this image encoding method, the image encoding device uses the depth prediction image macroblock for each macroblock which is a partial area constituting a depth map including a depth value indicating the distance from the viewpoint for each pixel. This is a method for predictive coding of a depth map. In this method, the image encoding device selects a prediction image generation method for one macroblock based on the motion vector of the adjacent macroblock, and the prediction vector based on the motion vector of the adjacent macroblock. , A step of determining a prediction mode for intra prediction from a coding mode signal of an adjacent macroblock, and a skip prediction image macroblock based on the pixel value of the macroblock adjacent to the determined prediction mode Generating step. Other application examples are as described for the image coding apparatus in the first embodiment.

The above image decoding method is a partial area constituting a depth map composed of depth values indicating the distance from the viewpoint for each pixel, which is predictively encoded using the skip prediction image macroblock according to the above image encoding method. An image decoding step in which the image decoding apparatus decodes the macroblock. Other application examples are as described for the image decoding apparatus in the first embodiment.

DESCRIPTION OF SYMBOLS 1 ... Image encoding apparatus 2 ...

Image pre-processing part

3a, 3b ... Imaging | photography apparatus 4 ... Texture image encoding apparatus 5 ... Image decoding apparatus 10 ... Three-dimensional image imaging system 101 ... Depth map input part 102 ... Reference image memory 103 ... Intra prediction unit 104 ... Inter prediction unit 105 ... Skipped macroblock unit 106 ... Coding mode determination unit 107 ... Motion vector memory 108 ... Coding mode memory 109 ... Switch 110 ... Subtraction unit 111 ... DCT transform / quantization unit 112 ... inverse quantization / inverse DCT transform unit 113 ... addition unit 114 ... information source coding unit 501 ... information source decoding unit 502 ... coding mode memory 503 ... motion vector memory 504 ... inverse quantization / inverse DCT transform unit 505 ... addition Unit 506 ... reference image memory 507 ... intra prediction image generation unit 508 ... inter prediction image generation unit 509 Skipped macroblock unit 510 ... Coding mode determination unit 512 ... Image output unit 1051 ... Prediction image generation method selection unit 1052 ... Prediction vector generation unit 1053 ... Prediction image macroblock generation unit 1054 ... Prediction mode determination unit 1055 ... Intra prediction image Macroblock generator

Claims

An image decoding apparatus for decoding an image encoded for each macroblock obtained by dividing an image,
A prediction mode determination unit that determines a prediction mode of the decoding target macroblock from a prediction mode when the macroblock adjacent to the decoding target macroblock in the depth map is intra-predicted as an image for decoding the depth map; ,
An intra prediction image macroblock generation unit that generates a prediction image of the macroblock to be decoded by intra prediction based on the prediction mode determined by the prediction mode determination unit and the pixel value of the adjacent macroblock;
An image decoding apparatus comprising:
The prediction mode determination unit determines the prediction mode of the intra prediction based on a combination of an adjacent direction of the adjacent macroblock with respect to the encoding target macroblock and a prediction mode of the adjacent macroblock. The image decoding apparatus according to claim 1.
An image decoding method for decoding an image encoded for each macroblock obtained by dividing an image,
Determining the prediction mode of the decoding target macroblock from the prediction mode when the macroblock adjacent to the decoding target macroblock in the depth map is intra-predicted as an image for decoding the depth map;
Generating a prediction image of the decoding target macroblock by intra prediction based on the determined prediction mode and a pixel value of the adjacent macroblock;
An image decoding method characterized by comprising:
An image encoding device that encodes each macroblock obtained by dividing an image,
Prediction mode determination for determining the prediction mode of the macroblock to be encoded from the prediction mode when the macroblock adjacent to the macroblock to be encoded in the depth map is intra-predicted as an image for decoding the depth map And
A prediction image generation unit that generates a prediction image of the encoding target macroblock by intra prediction based on the determined prediction mode and the pixel value of the adjacent macroblock;
An image encoding device comprising: