RU2609083C2 - Image encoding method, image decoding method, image encoding device and image decoding device - Google Patents

Abstract

FIELD: data processing.

SUBSTANCE: invention relates to video encoding/decoding. Disclosed is images encoding method for encoding each of images units. Method includes stage, at which extracting bi-predicted candidate for current unit motion vectors, which must be encoded, from first unit first motion vector included into first image, wherein first image is different from image, which includes current unit. Further, according to method, adding extracted bi-predicted candidate into candidates list and selecting one bi-predicted candidate, which includes second motion vector and third motion vector, from candidates list. Encoding current unit using second motion vector and third motion vector, as well as second reference image and third reference image of current unit, wherein second reference image and third reference image correspond to second motion vector and third motion vector, respectively.

EFFECT: technical result is higher efficiency of images encoding.

7 cl, 41 dwg

Description

2420-510149EN / 17

FIELD OF THE INVENTION

[0001] The present invention relates to a method of encoding images for encoding each of the image blocks.

BACKGROUND

[0002] A technology associated with an image encoding method for encoding each of the image blocks is described in Non-Patent Document (NPL) 1.

LIST OF BIBLIOGRAPHIC REFERENCES

NON-PATENT DOCUMENTS

[0003] [NPL 1] ISO / IEC 14496-10 - MPEG-4 Part 10, Advanced Video Coding

SUMMARY OF THE INVENTION

TECHNICAL PROBLEM

[0004] However, there are cases when the traditional method of encoding images does not allow to achieve a sufficiently high encoding efficiency.

[0005] With this in mind, the present invention provides an image encoding method that can improve the encoding efficiency when encoding images.

THE SOLUTION OF THE PROBLEM

[0006] An image encoding method according to an aspect of the present invention is an image encoding method for encoding each of the image blocks, the image encoding method including: extracting a candidate for a motion vector of a current block to be encoded from a motion vector of a co-located block that is a block included in an image other than an image that includes the current block; adding the extracted candidate to the list; selection of the motion vector of the current block from the list to which the candidate is added; and encoding the current block using the selected motion vector and the reference image of the current block, the extraction includes: determining whether the reference image of the current block is a long-term reference image or a short-term reference image, and whether the reference image of the co-located block is long-term reference image or short-term reference image; extracting the candidate from the motion vector of the co-located block by means of a first extraction circuit that does not include scaling based on the time distance, in the case of determining that each of the reference image of the current block and the reference image of the co-located block is a long-term reference image; and extracting the candidate from the motion vector of the co-located block by means of a second extraction scheme that incorporates scaling based on the time distance, in the event that it is determined that each of the reference image of the current block and the reference image of the co-located block is a short-term reference image.

[0007] These general and specific aspects can be implemented using a system, device, integrated circuit, computer program or non-volatile computer-readable recording medium such as a CD-ROM, or any combination of systems, devices, methods, integrated circuits, computer programs of their media records.

Advantages of the Invention

[0008] An image encoding method according to the present invention provides an image encoding method that can improve the encoding efficiency when encoding images.

BRIEF DESCRIPTION OF THE DRAWINGS

[0009] FIG. 1 is a flowchart showing an operation of an image encoding apparatus according to a reference example.

FIG. 2 is a flowchart showing an operation of an image decoding apparatus according to a reference example.

FIG. 3 is a flowchart showing details of an extraction process according to a reference example.

FIG. 4 is a diagram for explaining a co-located unit according to a reference example.

FIG. 5 is a block diagram of an image decoding apparatus according to Embodiment 1.

FIG. 6 is a block diagram of an image decoding apparatus according to Embodiment 1.

FIG. 7 is a flowchart showing an operation of an image encoding apparatus according to Embodiment 1.

FIG. 8 is a flowchart showing an operation of an image decoding apparatus according to Embodiment 1.

FIG. 9 is a flowchart showing details of an extraction process according to Embodiment 1.

FIG. 10 is a flowchart showing details of an extraction process according to Embodiment 2.

FIG. 11 is a diagram for explaining a co-located unit according to Embodiment 2.

FIG. 12 is a flowchart showing details of an extraction process according to Embodiment 3.

FIG. 13A is a block diagram of an image decoding apparatus according to Embodiment 4.

FIG. 13B is a flowchart showing an operation of an image encoding apparatus according to Embodiment 4.

FIG. 14A is a block diagram of an image decoding apparatus according to Embodiment 4.

FIG. 14B is a flowchart showing an operation of an image decoding apparatus according to Embodiment 4.

FIG. 15A is a diagram showing a first example of a storage location for a parameter indicating a classification of reference images.

FIG. 15B is a diagram showing a second example of a storage location for a parameter indicating a classification of reference images.

FIG. 15C is a diagram showing a third example of a storage location for a parameter indicating a classification of reference images.

FIG. 16 is a diagram showing an example of a storage location for a parameter indicating a prediction mode.

FIG. 17 illustrates the overall configuration of a content delivery system for implementing content distribution services.

FIG. 18 illustrates the overall configuration of a digital broadcast system.

FIG. 19 is a block diagram illustrating an example configuration of a television receiver.

FIG. 20 is a block diagram illustrating an example configuration of an information reproducing / recording unit that reads and writes information from or to a recording medium that is an optical disk.

FIG. 21 shows an example configuration of a recording medium that is an optical disc.

FIG. 22A shows an example of a cell phone.

FIG. 22B shows an example configuration of a cell phone.

FIG. 23 shows the structure of multiplexed data.

FIG. 24 schematically illustrates how each of the streams is multiplexed into multiplexed data.

FIG. 25 illustrates how a video stream is stored in a stream of PES packets.

FIG. 26 shows the structure of TS packets and source packets in multiplexed data.

FIG. 27 shows a PMT data structure.

FIG. 28 shows an internal structure of multiplexed data information.

FIG. 29 shows an internal structure of stream attribute information.

FIG. 30 shows the steps for identifying video data.

FIG. 31 is a flowchart illustrating an example configuration of an integrated circuit for implementing a moving image coding method and a moving image decoding method according to each of the embodiments.

FIG. 32 shows a configuration for switching between driving frequencies.

FIG. 33 shows the steps for identifying video data and switching between driving frequencies.

FIG. 34 shows an example of a lookup table in which video data standards are associated with driving frequencies.

FIG. 35A shows an example configuration for sharing a signal processor module.

FIG. 35B shows another configuration example for sharing a signal processor module.

DETAILED DESCRIPTION OF EMBODIMENTS

[0010] BASIC BASIS FOR FORMING KNOWLEDGE OF THE PRESENT INVENTION

Regarding the method of encoding images disclosed in the section "prior art", the inventors have discovered the following problem. It should be noted that in the description below, the image may be any of a moving image, consisting of many images, a still image, consisting of one image, part of the image, etc.

[0011] Image encoding schemes in recent years include MPEG-4 AVC / H.264 and HEVC (high performance video encoding standard). In these image coding schemes, external (inter-) prediction using encoded reference images is available.

[0012] In addition, a reference image called a long-term reference image may be used in these image coding schemes. For example, if the reference image is stored in a DPB (buffer of decoded (graphic) images) for a long time, the reference image can be used as a long-term reference image.

[0013] In HEVC, there is a mode called a merge mode. In merge mode, the motion vector predictor obtained by predicting the motion vector of the current block from the motion vector of an adjacent block, etc., is used to encode the current block as the motion vector of the current block. In other words, in merge mode, the motion vector predictor is processed as the motion vector of the current block. The motion vector predictor and the motion vector of the current block in merge mode are also referred to as the merge vector.

[0014] In HEVC, a temporal motion vector predictor can also be used. The temporal predictor of the motion vector is extracted from the motion vector of the co-located block in the coded co-located image. The coordinates of the co-located block in the co-located image correspond to the coordinates of the current block in the current image to be encoded.

[0015] Further, the motion vector of the co-located block is also referred to as the co-located motion vector, and the reference image of the co-located block is also referred to as the co-located reference image. A co-located block is encoded using a co-located motion vector and a co-located reference image. It should be noted that “co-located” can also be written as “collocated”.

[0016] Similarly, the motion vector of the current block is also referred to as the current motion vector, and the reference image of the current block is also referred to as the current reference image. The current block is encoded using the current motion vector and the current reference image.

[0017] The aforementioned current block and co-located block are a prediction unit (PU). A prediction unit is an image block and is defined as a data unit for prediction. In HEVC, a coding unit (CU) is defined as a unit of data for coding, separate from a prediction unit. A prediction unit is a block in a coding unit. In the description below, the term “block” may be replaced by “prediction unit” or “coding unit”.

[0018] The coding unit and the prediction unit are not fixed in size. For example, a single image may include a plurality of coding units with different sizes, and a single image may include a plurality of prediction units with different sizes.

[0019] This may lead to a case where a block that exactly matches the region of the current block is not defined in the shared image. Accordingly, in HEVC, a co-located block is selected from a plurality of blocks included in the co-located image by a predetermined selection method.

[0020] A temporal motion vector predictor is generated by scaling a motion vector of a selected co-located block based on a POC distance (image sequence counter). POCs are ordinals assigned to images in display order. The POC distance corresponds to the time distance between two images. POC distance scaling is also referred to as POC based scaling. Expression 1 below is an arithmetic expression for performing POC-based scaling for a motion vector of a co-located block.

[0021] pmv = (tb / td) × colmv (expression 1).

[0022] Here, colmv is a motion vector of a co-located block; pmv is a temporary motion vector predictor extracted from the motion vector of a co-located block; tb is a signed POC distance representing the difference between the current image and the current reference image; td is a signed POC distance representing the difference between the co-located image and the co-located reference image.

[0023] In the event that a valid motion vector predictor is present, the motion vector predictor is inserted into the ordered list of candidates of the current motion vectors. The motion vector used to encode the current block is selected from an ordered list of candidates for the current motion vectors. The selected motion vector is indicated by a parameter in the bit stream.

[0024] FIG. 1 is a flowchart showing an operation of an image encoding apparatus according to a reference example. In particular, FIG. 1 shows an image encoding process by means of external prediction.

[0025] First, the image encoding apparatus classifies each of the reference images as a short-term reference image or as a long-term reference image (step S101). The image encoding device records information indicating the classification of each of the reference images in the header of the bit stream (step S102).

[0026] Then, the image encoding apparatus identifies the current reference image (step S103). The image encoding apparatus further extracts the current motion vector (step S104). The extraction process is described in detail below.

[0027] After that, the image encoding device generates a prediction block by performing motion compensation using the current reference image and the current motion vector (step S105).

[0028] After that, the image encoding device subtracts the prediction block from the current block to generate a residual block (step S106). Finally, the image encoding apparatus encodes a residual block to generate a bit stream including an encoded residual block (step S107).

[0029] FIG. 2 is a flowchart showing an operation of an image decoding apparatus according to a reference example. In particular, FIG. 2 shows an image decoding process by inter prediction.

[0030] First, the image decoding apparatus receives the bitstream and obtains information indicating the classification of each of the reference images by parsing the header of the bitstream (step S201). The image decoding apparatus also obtains a residual block by parsing the bitstream (step S202).

[0031] Then, the image decoding apparatus identifies the current reference image (step S203). The image decoding apparatus further extracts the current motion vector (step S204). The extraction process is described in detail below. After that, the image decoding apparatus generates a prediction block by performing motion compensation using the current reference image and the current motion vector (step S205). Finally, the image decoding apparatus summarizes the prediction block with the residual block to generate the reconstructed block (step S206).

[0032] FIG. 3 is a flowchart showing details of the extraction process shown in FIG. 1 and 2. The following describes the operation of the image encoding device. The operation of the image decoding device is identical to the operation of the image encoding device, while the "encoding" is replaced by "decoding".

[0033] First, the image encoding apparatus selects a co-located image (step S301). Then, the image encoding apparatus selects the co-located block in the co-located image (step S302). The image encoding apparatus further identifies a co-located reference image and a co-located motion vector (step S303). After that, the image encoding device extracts the current motion vector by an extraction circuit that incorporates POC-based scaling (step S304).

[0034] FIG. 4 is a diagram for explaining a co-located unit used in the extraction process shown in FIG. 3. A co-located block is selected from a plurality of blocks in a co-located image.

[0035] The co-located image is different from the current image, which includes the current block. For example, a co-located image is an image immediately preceding or coming immediately after the current image in the display order. In more detail, for example, a co-located image is a reference image listed first in either of the two lists of reference images used for encoding B-images (bi-predictive coding).

[0036] The first block, including the c0 sample in the co-located image, is the main candidate for the co-located block and is also referred to as the primary co-located block. A second block including sample c1 in the co-located image is the second main candidate for the co-located block and is also referred to as a secondary co-located block.

[0037] Let (x, y) be the coordinates of the tl sample from the top left in the current block, w is the width of the current block, and h is the height of the current block. The coordinates of the sample c0 are (x + w, y + h). The sampling coordinates c1 are (x + (w / 2) -1, y + (h / 2) -1).

[0038] In the event that the first block is unavailable, the second block is selected as the co-located block. Examples of a case where the first block is unavailable includes a case in which the first block is not present, since the current block is to the right of or least of all, and a case in which the first block is encoded by intra prediction.

[0039] A more specific example of the process of extracting a temporary motion vector predictor as the current motion vector is again described again with reference to FIG. 3.

[0040] First, the image encoding apparatus selects a co-located image (step S301). Then, the image encoding device selects the co-located unit (step S302). In the case where the first block including the sample c0 shown in FIG. 4 is available, the first block is selected as a co-located block. In the event that the first block is unavailable and the second block includes the sample c1 shown in FIG. 4 is available, the second block is selected as a co-located block.

[0041] In the case where an available co-located block is selected, the image encoding apparatus sets the time predictor of the motion vector to be available. In the event that an available co-located block is not selected, the image encoding device sets the time predictor of the motion vector as unavailable.

[0042] In the event that the temporal motion vector predictor is set as available, the image encoding apparatus identifies the co-located motion vector as the base motion vector. The image encoding apparatus also identifies a co-located reference image (step S303). The image encoding apparatus further extracts the time predictor of the motion vector from the base motion vector by scaling according to expression 1 (step S304).

[0043] Through the process described above, the image encoding apparatus and the image decoding apparatus extract a temporal motion vector predictor as the current motion vector.

[0044] However, there are times when it is difficult to retrieve the proper current motion vector, depending on the relationships between the current image, the current reference image, the co-located image and the co-located reference image.

[0045] For example, in the case where the current reference image is a long-term reference image, it is likely that the time distance between the current reference image and the current image is large. In case the co-located reference image is a long-term reference image, it is likely that the time distance between the co-located reference image and the co-located image is large.

[0046] These cases are susceptible to the possibility that an extremely large or small current motion vector is generated as a result of scaling based on POC. This leads to a decrease in prediction accuracy and a decrease in coding efficiency. In particular, an extremely large or small current motion vector cannot be properly expressed by a fixed number of bits, resulting in a significant decrease in prediction accuracy and a decrease in coding efficiency.

[0047] An image encoding method according to an aspect of the present invention is an image encoding method for encoding each of the image blocks, the image encoding method including: extracting a candidate for a motion vector of a current block to be encoded from a motion vector of a co-located block that is a block included in an image other than an image that includes the current block; adding the extracted candidate to the list; selection of the motion vector of the current block from the list to which the candidate is added; and encoding the current block using the selected motion vector and the reference image of the current block, the extraction includes: determining whether the reference image of the current block is a long-term reference image or a short-term reference image, and whether the reference image of the co-located block is long-term reference image or short-term reference image; extracting the candidate from the motion vector of the co-located block by means of a first extraction circuit that does not include scaling based on the time distance, in the case of determining that each of the reference image of the current block and the reference image of the co-located block is a long-term reference image; and extracting the candidate from the motion vector of the co-located block by means of a second extraction scheme that incorporates scaling based on the time distance, in the event that it is determined that each of the reference image of the current block and the reference image of the co-located block is a short-term reference image.

[0048] Thus, the candidate for the current motion vector is appropriately retrieved while it is not extremely large or small. This contributes to improved prediction accuracy and increased coding efficiency.

[0049] For example, when retrieving: extracting a candidate from a motion vector of a co-located block cannot be performed if it is determined that one of the reference image of the current block and the reference image of the co-located block is a long-term reference image, and the other from the reference image of the current block and the reference image of the co-located block is a short-term reference image; and extracting the candidate from the motion vector of the co-located block can be performed in the case of determining that each of the reference image of the current block and the reference image of the co-located block is a long-term reference image, or in the case of determining that each of the reference image of the current block and the reference images of a co-located block is a short-term reference image.

[0050] Thus, in the event that low prediction accuracy is expected, the candidate for the current motion vector is not extracted from the motion vector of the co-located block. Therefore, a decrease in prediction accuracy can be prevented.

[0051] For example, the encoding may further include encoding information indicating whether the reference image of the current block is a long-term reference image or a short-term reference image, and information indicating whether the reference image of a co-located block is a long-term reference image or a short-term reference image.

[0052] Thus, information indicating for each reference image, whether the reference image is a long-term reference image or a short-term reference image is provided from the encoding side to the decoding side. This enables the encoding side and the decoding side to obtain an identical determination result and perform an identical process.

[0053] For example, the extraction may include: determining whether the reference image of the current block is a long-term reference image or a short-term reference image using the time distance between the reference image of the current block and the image that includes the current block; and determining whether the reference image of the co-located block is a long-term reference image or a short-term reference image using the time distance between the reference image of the co-located block and the image that includes the co-located block.

[0054] Thus, for each reference image, whether the reference image is a long-term reference image or a short-term reference image is determined simply and appropriately based on a time distance.

[0055] For example, the retrieval may include determining whether the reference image of the co-located block is a long-term reference image or a short-term reference image during the period during which the co-located block is encoded.

[0056] Thus, whether the reference image of the co-located block is a long-term reference image or a short-term reference image is determined more accurately.

[0057] For example, the extraction may include determining whether the reference image of the co-located block is a long-term reference image or a short-term reference image during the period during which the current block is encoded.

[0058] Thus, information indicating whether the reference image of the co-located unit is a long-term reference image or a short-term reference image should not be stored for a long time.

[0059] For example, the extraction may include: extracting the motion vector of the co-located block as a candidate in the case of determining that each of the reference image of the current block and the reference image of the co-located block is a long-term reference image; and extracting the candidate by scaling the motion vector of the co-located block using the relation in the case of determining that each of the reference image of the current block and the reference image of the co-located block is a short-term reference image, wherein the ratio is the ratio of the time distance between the reference image of the current block and the image , which includes the current block, to the time distance between the reference image together Nogo and block image which includes co-located block.

[0060] Thus, in case the two reference images are a long-term reference image, scaling is omitted, thereby reducing the amount of computation. In case the two reference images are a short-term reference image, the candidate for the current motion vector is properly extracted based on the time distance.

[0061] For example, the retrieval may further include, without retrieving the candidate from the co-located block, selecting another co-located block and retrieving the candidate from the motion vector of the other co-located block by the second retrieval circuit in the event that it is determined that the reference image of the current block is a short-term reference image, and the reference image of the co-located block is a long-term reference image, with another co-located block of code This applies to a short reference image.

[0062] Thus, a block is selected for retrieving a candidate with high prediction accuracy. This contributes to improved prediction accuracy.

[0063] Further, an image decoding method according to an aspect of the present invention is an image decoding method for decoding each of the image blocks, the image decoding method including: extracting a candidate for a motion vector of a current block to be decoded from a motion vector co-located a block, which is a block included in an image other than an image that includes the current block; adding the extracted candidate to the list; selection of the motion vector of the current block from the list to which the candidate is added; and decoding the current block using the selected motion vector and the reference image of the current block, the extraction includes: determining whether the reference image of the current block is a long-term reference image or a short-term reference image, and whether the reference image of the co-located block is long-term reference image or short-term reference image; extracting the candidate from the motion vector of the co-located block by means of a first extraction circuit that does not include scaling based on the time distance, in the case of determining that each of the reference image of the current block and the reference image of the co-located block is a long-term reference image; and extracting the candidate from the motion vector of the co-located block by means of a second extraction scheme that incorporates scaling based on the time distance, in the event that it is determined that each of the reference image of the current block and the reference image of the co-located block is a short-term reference image.

[0064] Thus, the candidate for the current motion vector is appropriately retrieved despite being extremely large or small. This contributes to improved prediction accuracy and increased coding efficiency.

[0065] For example, when extracting: extracting a candidate from a motion vector of a co-located block cannot be performed if it is determined that one of the reference image of the current block and the reference image of the co-located block is a long-term reference image, and the other from the reference image of the current block and the reference image of the co-located block is a short-term reference image; and extracting the candidate from the motion vector of the co-located block can be performed in the case of determining that each of the reference image of the current block and the reference image of the co-located block is a long-term reference image, or in the case of determining that each of the reference image of the current block and the reference images of a co-located block is a short-term reference image.

[0066] Thus, in case a low prediction accuracy is expected, the candidate for the current motion vector is not extracted from the motion vector of the co-located block. Therefore, a decrease in prediction accuracy can be prevented.

[0067] For example, decoding may further include decoding information indicating whether the reference image of the current block is a long-term reference image or a short-term reference image, and information indicating whether the reference image of a co-located block is a long-term reference image or a short-term reference image, while the extraction includes: determining whether the reference image of the current block is a long-term reference image or a short-term reference image using information indicating whether the reference image of the current block is a long-term reference image or a short-term reference image; and determining whether the reference image of the co-located block is a long-term reference image or a short-term reference image using information indicating whether the reference image of the co-located block is a long-term reference image or a short-term reference image.

[0068] Thus, information indicating for each reference image whether the reference image is a long-term reference image or a short-term reference image is provided from the encoding side to the decoding side. This enables the encoding side and the decoding side to obtain an identical determination result and perform an identical process.

[0069] For example, the extraction may include: determining whether the reference image of the current block is a long-term reference image or a short-term reference image using the time distance between the reference image of the current block and the image that includes the current block; and determining whether the reference image of the co-located block is a long-term reference image or a short-term reference image using the time distance between the reference image of the co-located block and the image that includes the co-located block.

[0070] Thus, for each reference image, whether the reference image is a long-term reference image or a short-term reference image is determined simply and appropriately based on a time distance.

[0071] For example, the retrieval may include determining whether the reference image of the co-located block is a long-term reference image or a short-term reference image during the period during which the co-located block is decoded.

[0072] Thus, whether the reference image of the co-located block is a long-term reference image or a short-term reference image is determined more accurately.

[0073] For example, the extraction may include determining whether the reference image of the co-located block is a long-term reference image or a short-term reference image during the period during which the current block is decoded.

[0074] Thus, information indicating whether the reference image of the co-located unit is a long-term reference image or a short-term reference image should not be stored for a long time.

[0075] For example, the extraction may include: extracting the motion vector of the co-located block as a candidate in the event that it is determined that each of the reference image of the current block and the reference image of the co-located block is a long-term reference image; and extracting the candidate by scaling the motion vector of the co-located block using the relation in the case of determining that each of the reference image of the current block and the reference image of the co-located block is a short-term reference image, wherein the ratio is the ratio of the time distance between the reference image of the current block and the image , which includes the current block, to the time distance between the reference image together Nogo and block image which includes co-located block.

[0076] Thus, in case the two reference images are a long-term reference image, scaling is omitted, thereby reducing the amount of computation. In case the two reference images are a short-term reference image, the candidate for the current motion vector is appropriately retrieved based on the time distance.

[0077] For example, the extraction may further include, without extracting the candidate from the co-located block, selecting another co-located block and extracting the candidate from the motion vector of the other co-located block by the second extraction scheme in the event that it is determined that the reference image of the current block is a short-term reference image, and the reference image of the co-located block is a long-term reference image, wherein another co-located deco block It is based on a short reference image.

[0078] Thus, a block is selected for retrieving a candidate with high prediction accuracy. This contributes to improved prediction accuracy.

[0079] Furthermore, a method for providing content according to an aspect of the present invention is a method for providing content for transmission from a server to which image data encoded by the image encoding method described above of image data in response to a request from an external terminal is recorded.

[0080] These general and specific aspects can be implemented using a system, device, integrated circuit, computer program, or non-volatile computer-readable recording medium such as a CD-ROM, or any combination of systems, devices, methods, integrated circuits, computer programs and media records.

[0081] Hereinafter, certain exemplary embodiments are described in more detail with reference to the accompanying drawings. Each of the exemplary embodiments described below shows a general or specific example. The numerical values, forms, materials, structural elements, arrangement and connection of structural elements, steps, processing steps, etc., shown in the following exemplary embodiments are merely examples and, therefore, do not limit the scope of the present invention. Therefore, of the structural elements in the following exemplary embodiments, structural elements not set forth in any of the independent claims are described as arbitrary structural elements.

[0082] OPTION 1

FIG. 5 is a block diagram of an image encoding apparatus according to Embodiment 1. The image encoding device 500 shown in FIG. 5 encodes a block-based image and outputs a bitstream including an encoded image. In detail, the image encoding apparatus 500 includes a subtraction unit 501, a transform unit 502, a quantization unit 503, an entropy encoder 504, an inverse quantization unit 505, an inverse transform unit 506, an adder 507, a block memory 508, image memory 509, an internal memory unit 510 510 prediction module 511 external prediction module 512 selection.

[0083] Subtraction module 501 subtracts the prediction image from the image provided to the image encoding device 500 to generate a difference image. The conversion unit 502 performs frequency conversion for the differential image generated by the subtraction unit 501 to generate a plurality of frequency coefficients. The quantization unit 503 quantizes the plurality of frequency coefficients generated by the transform unit 502 to generate a plurality of quantization coefficients. Entropy encoder 504 encodes a plurality of quantization coefficients generated by quantization module 503 to generate a bit stream.

[0084] The inverse quantization unit 505 inverse quantizes a plurality of quantization coefficients generated by the quantization unit 503 to recover a plurality of frequency coefficients. The inverse transform unit 506 performs inverse frequency conversion for a plurality of frequency coefficients reconstructed by the inverse quantization unit 505 to reconstruct a difference image. The summing unit 507 sums the prediction image with the differential image reconstructed by the inverse transform unit 506 to reconstruct (reconstruct) the image. The summing unit 507 stores the reconstructed image (reconstructed image) in the block memory 508 and the image memory 509.

[0085] The block memory 508 is a memory for storing an image restored by the block-based summing unit 507. The image memory 509 is a memory for storing an image restored by the image-based summing unit 507.

[0086] The intra prediction unit 510 performs intra prediction by accessing the block memory 508. In other words, the intra prediction unit 510 predicts a pixel value in the image from another pixel value in the image. The intra prediction unit 510 thereby generates a prediction image. The external prediction module 511 performs external prediction by accessing the image memory 509. In other words, the inter prediction unit 511 predicts a pixel value in an image from a pixel value in another image. Inter prediction module 511 thereby generates a prediction image.

[0087] The selection module 512 selects any of the prediction image generated by the intra prediction module 510 and the prediction image generated by the inter prediction module 511, and outputs the selected prediction image to the subtraction module 501 and the summing module 507.

[0088] Although not shown in FIG. 5, the image encoding device 500 may include a filtering module for deblocking. The deblocking filtering unit may perform a deblocking filtering process for the image reconstructed by the summing unit 507 to remove noise near block boundaries. The image encoding device 500 may also include a control module that controls each process in the image encoding device 500.

[0089] FIG. 6 is a block diagram of an image decoding apparatus according to this embodiment. The image decoding apparatus 600 shown in FIG. 6, receives a bitstream and decodes the image based on blocks. In detail, the image decoding apparatus 600 includes an entropy decoder 601, an inverse quantization module 602, an inverse transform module 603, an adder 604, a block memory 605, an image memory 606, an intra prediction module 607, an inter prediction module 608, and a selection module 609.

[0090] The entropy decoder 601 decodes the encoded plurality of quantization coefficients included in the bitstream. The inverse quantization module 602 inverse quantizes a plurality of quantization coefficients decoded by an entropy decoder 601 to recover a plurality of frequency coefficients. The inverse transform unit 603 performs an inverse frequency transform for the plurality of frequency coefficients reconstructed by the inverse quantization unit 602 to reconstruct a difference image.

[0091] The summing unit 604 sums the prediction image with the difference image reconstructed by the inverse transform unit 603 to reconstruct (reconstruct) the image. Summing module 604 outputs a reconstructed image (reconstructed image). Summing unit 604 also stores the reconstructed image in block memory 605 and image memory 606.

[0092] The block memory 605 is a memory for storing an image restored by the block-based summing unit 604. The image memory 606 is a memory for storing images restored by the image-based summing unit 604.

[0093] The intra prediction unit 607 performs intra prediction by accessing the block memory 605. In other words, the intra prediction unit 607 predicts a pixel value in the image from another pixel value in the image. The intra prediction unit 607 thereby generates a prediction image. Inter prediction module 608 performs inter prediction by accessing the image memory 606. In other words, the inter prediction unit 608 predicts a pixel value in an image from a pixel value in another image. Inter prediction module 608 thereby generates a prediction image.

[0094] The selection module 609 selects any of the prediction image generated by the intra prediction module 607 and the prediction image generated by the inter prediction module 608, and outputs the selected prediction image to the summing module 604.

[0095] Although not shown in FIG. 6, the image decoding apparatus 600 may include a filtering module for deblocking. The deblocking filtering unit may perform a deblocking filtering process for the image reconstructed by the summing unit 604 to remove noise near block boundaries. The image decoding apparatus 600 may also include a control unit that controls each process in the image decoding apparatus 600.

[0096] The above encoding process and decoding process are performed based on the coding unit. The transform process, the quantization process, the inverse transform process, and the inverse quantization process are performed based on the transform unit (TU), and the transform unit is included in the coding unit. The prediction process is performed based on the prediction unit, with the prediction unit being included in the coding unit.

[0097] FIG. 7 is a flowchart showing the operation of the image encoding apparatus 500 shown in FIG. 5. In particular, FIG. 7 shows an image encoding process by external prediction.

[0098] First, the inter prediction module 511 classifies each of the reference images as a short-term reference image or a long-term reference image (block 701).

[0099] Long-term reference image is a reference image suitable for long-term use. A long-term reference image is defined as a reference image for longer use than a short-term reference image. Accordingly, there is a high probability that a long-term reference image is stored in the image memory 509 for a long time. A long-term reference image is indicated by an absolute POC that is independent of the current image. Meanwhile, the short-term reference image is indicated by POC relative to the current image.

[0100] Then, the entropy encoder 504 records information indicating the classification of each of the reference images in the header of the bit stream (step 702). In other words, the entropy encoder 504 records information indicating for each of the reference images whether the reference image is a long-term reference image or a short-term reference image.

[0101] After that, the inter prediction module 511 identifies a reference image of the current block to be encoded (which should be predicted) (step S703). Inter prediction module 511 may identify a reference image of a block adjacent to the current block as the current reference image. Alternatively, the inter prediction module 511 may identify the current reference image by a predetermined reference index. Inter prediction module 511 further extracts the current motion vector (step S704). The extraction process is described in detail below.

[0102] The inter prediction unit 511 generates a prediction unit by performing motion compensation using the current reference image and the current motion vector (step S705). After that, the subtraction unit 501 subtracts the prediction block from the current block (original image) to generate a residual block (step S706). Finally, the entropy encoder 504 encodes a residual block to generate a bit stream including a residual block (step S707).

[0103] FIG. 8 is a flowchart showing the operation 600 of the image decoding apparatus shown in FIG. 6. In particular, FIG. 8 shows an image decoding process by inter prediction.

[0104] First, the entropy decoder 601 receives the encoded stream and obtains information indicating the classification of each of the reference images by parsing the header of the bit stream (block 801). In other words, the entropy decoder 601 receives information indicating for each of the reference images whether the reference image is a long-term reference image or a short-term reference image. Entropy decoder 601 also obtains a residual block by parsing the bitstream (step S802).

[0105] Then, the inter prediction module 608 identifies the current reference image (step S803). Inter prediction module 608 may identify a reference image of a block adjacent to the current block as the current reference image. Alternatively, the inter prediction module 608 may identify the current reference image by a predetermined reference index.

[0106] After this, the inter prediction module 608 extracts the current motion vector (step S804). The extraction process is described in detail below. The inter prediction unit 608 further generates a prediction block by performing motion compensation using the current reference image and the current motion vector (step S805). Finally, the summing unit 604 sums the prediction block with the residual block to generate the reconstructed block (block 806).

[0107] FIG. 9 is a flowchart showing details of the extraction process shown in FIG. 7 and 8. The following describes mainly the operation of the inter prediction module 511 shown in FIG. 5. The operation of the external prediction module 608 shown in FIG. 6 is identical to the operation of the inter prediction module 511 shown in FIG. 5, wherein “encoding” is replaced by “decoding”.

[0108] First, the inter prediction unit 511 selects a co-located image from the plurality of available reference images (step S901). Many available reference images are encoded images and are stored in the image memory 509.

[0109] Then, the inter prediction unit 511 selects the co-located block in the co-located image (step S902). The inter prediction unit 511 further identifies the co-located reference image and the co-located motion vector (step S903).

[0110] After this, the inter prediction module 511 determines whether any of the current reference image and the co-located reference image are a long-term reference image (step S904). If it is determined that any of the current reference image and the shared reference image is a long-term reference image (step S904: “Yes”), the inter prediction unit 511 extracts the current motion vector by the first extraction circuit (step S905).

[0111] The first extraction scheme is a scheme using a co-located motion vector. In more detail, the first extraction scheme is a scheme for directly extracting a co-located motion vector as the current motion vector without scaling based on the POC. The first extraction scheme may be a scheme for extracting the current motion vector by scaling the co-located motion vector in a predetermined ratio.

[0112] If it is determined that none of the current reference image and the shared reference image is a long-term reference image (step S904: "No"), the inter prediction unit 511 extracts the current motion vector by the second extraction circuit (step S906) . In other words, in the event that it is determined that the current reference image and the jointly located reference image are a short-term reference image, the inter prediction unit 511 extracts the current motion vector by the second extraction circuit.

[0113] The second extraction circuit is a circuit using a current reference image, a co-located reference image, and a co-located motion vector. In more detail, the second extraction scheme is the extraction scheme of the current motion vector by performing POC-based scaling (expression 1) for the co-located motion vector.

[0114] The following is a more specific example of the process of extracting the current motion vector again with reference to FIG. 9. The extraction process described above can be modified as follows.

[0115] First, the inter prediction module 511 selects a co-located image (step S901). In more detail, if the slice_type parameter of the slice header is B and the collocated_from_l0_flag parameter of the slice header is 0, the image RefPicList1 [0] is selected as the co-located image. Image RefPicList1 [0] is a reference image listed first in the ordered list RefPicList1 of reference images.

[0116] If the slice_type parameter of the slice header is not equal to B, or if the collocated_from_l0_flag parameter of the slice header is not 0, the image RefPicList0 [0] is selected as the co-located image. Image RefPicList0 [0] is a reference image listed first in the ordered list RefPicList0 of reference images.

[0117] Then, the inter prediction unit 511 selects the co-located block (step S902). In the case where the first block including the sample c0 shown in FIG. 4 is available, the first block is selected as a co-located block. In the event that the first block is unavailable and the second block includes the sample c1 shown in FIG. 4 is available, the second block is selected as a co-located block.

[0118] In the case where an available co-located block is selected, the inter prediction unit 511 sets the time predictor of the motion vector to be available. In the event that an available co-located block is not selected, the inter prediction module 511 sets the time predictor of the motion vector as unavailable.

[0119] In the event that the time predictor of the motion vector is set as available, the inter prediction module 511 identifies the co-located motion vector as the base motion vector. Inter prediction unit 511 also identifies a co-located reference image (step S903). In case the co-located block has many motion vectors, i.e. in case the co-located block is encoded using a plurality of motion vectors, the external prediction module 511 selects a basic motion vector according to a predetermined priority order.

[0120] For example, in the case where the current reference image is a short-term reference image, the inter prediction module 511 may preferably select a motion vector that indicates a location in the short-term reference image from a plurality of motion vectors as the base motion vector.

[0121] In detail, if there is a motion vector that indicates a location in the short-term reference image, the inter prediction unit 511 selects the motion vector as the base motion vector. In case there is no motion vector that indicates the location in the short-term reference image, the inter prediction module 511 selects a motion vector that indicates the location in the long-term reference image as the base motion vector.

[0122] After that, if any of the current reference image and the shared reference image is a long-term reference image (step S904: "Yes"), the inter prediction module 511 extracts the base motion vector as a temporary predictor of the motion vector (step S905).

[0123] On the other hand, in the event that neither of the two reference images is a long-term reference image (step S904: "No"), the inter prediction unit 511 extracts the time motion vector predictor from the base motion vector by scaling based on POC ( step S906).

[0124] As described above, the temporal predictor of the motion vector is set as available or inaccessible. The external prediction unit 511 inserts a time motion vector predictor defined as available into an ordered list of candidates for the current motion vectors. An ordered list stores not only a temporary predictor of the motion vector, but also various motion vectors as candidates.

[0125] Inter prediction module 511 selects one motion vector from an ordered list as the current motion vector. Here, the inter prediction module 511 selects a motion vector with the highest prediction accuracy for the current block or a motion vector that enables encoding the current block with the highest coding efficiency from an ordered list. The index corresponding to the selected motion vector is written to the bit stream.

[0126] Through the process described above, the current motion vector is appropriately extracted from the co-located motion vector while it is not extremely large or small. This contributes to improved prediction accuracy and increased coding efficiency.

[0127] It should be noted that the state of each reference image with respect to whether the reference image is a long-term reference image or a short-term reference image can be changed according to time. For example, a short-term reference image may subsequently be changed to a long-term reference image, and a long-term reference image may subsequently be changed to a short-term reference image.

[0128] Furthermore, the inter prediction module 511 may determine whether the co-located reference image is a long-term reference image or a short-term reference image during the period during which the co-located block is encoded. The image encoding device 500 may further include additional memory for storing the determination result from the moment the co-located block is encoded to the moment the current block is encoded.

[0129] Thus, whether the co-located reference image is a long-term reference image or a short-term reference image is determined more accurately.

[0130] Alternatively, the inter prediction module 511 may determine whether the co-located reference image is a long-term reference image or a short-term reference image during the period during which the current block is encoded.

[0131] Thus, information regarding whether the co-located reference image is a long-term reference image or a short-term reference image should not be stored for a long time.

[0132] Furthermore, the inter prediction module 511 may determine whether the current reference image is a long-term reference image or a short-term reference image using the time distance between the current reference image and the current image.

[0133] As an example, in case the time distance between the current reference image and the current image exceeds a predetermined threshold value, the inter prediction unit 511 determines that the current reference image is a long-term reference image. In case the time distance does not exceed a predetermined threshold value, the external prediction unit 511 determines whether the current reference image is a short-term reference image.

[0134] Similarly, the inter prediction module 511 may determine whether the co-located reference image is a long-term reference image or a short-term reference image using the time distance between the co-located reference image and the co-located image.

[0135] As an example, if the time distance between the co-located reference image and the co-located image exceeds a predetermined threshold value, the external prediction unit 511 determines that the co-located reference image is a long-term reference image. In case the time distance does not exceed a predetermined threshold value, the external prediction unit 511 determines that the co-located reference image is a short-term reference image.

[0136] The inter prediction unit 608 in the image decoding apparatus 600 may determine for each reference image whether the reference image is a long-term reference image or a short-term reference image based on a time distance, in the same way as the inter prediction unit 511 in the encoding device 500 images. In such a case, information indicating for each reference image whether the reference image is a long-term reference image or a short-term reference image need not be encoded.

[0137] In addition, with respect to each of the other processes described in this embodiment, each structural element in the image decoding apparatus 600 performs a process identical to the process of the corresponding structural element in the image encoding apparatus 500, resulting in an image encoded with high efficiency encoding, properly decoded.

[0138] The operations described above are also applicable to other embodiments. Any of the structures and operations described in this embodiment may be included in other embodiments, and any of the structures and operations described in other embodiments may be included in this embodiment.

[0139] OPTION 2 IMPLEMENTATION

The image encoding device and image decoding device according to Embodiment 2 have structures identical to those in Embodiment 1. Therefore, the operations of the image encoding apparatus and the image decoding apparatus according to this embodiment are described below using the structure of the image encoding apparatus 500 shown in FIG. 5 and the structure of the image decoding apparatus 600 shown in FIG. 6.

[0140] The image encoding apparatus 500 according to this embodiment carries out the operation shown in FIG. 7, similarly to embodiment 1. An image decoding apparatus 600 according to this embodiment performs the operation shown in FIG. 8, similarly to embodiment 1. This embodiment is different from Embodiment 1 by the process of extracting current motion vectors. This is described in detail below.

[0141] FIG. 10 is a flowchart showing details of an extraction process according to this embodiment. The inter prediction module 511 according to this embodiment performs the operation shown in FIG. 10, instead of the operation shown in FIG. 9. The following describes mainly the operation of the inter prediction module 511 shown in FIG. 5. The operation of the external prediction module 608 shown in FIG. 6 is identical to the operation of the inter prediction module 511 shown in FIG. 5, wherein “encoding” is replaced by “decoding”.

[0142] First, the inter prediction unit 511 selects a co-located image from the plurality of available reference images (step S1001). Then, the inter prediction unit 511 selects the co-located block in the co-located image (step S1002). The inter prediction unit 511 further identifies the co-located reference image and the co-located motion vector (step S1003).

[0143] First, the inter prediction module 511 determines whether the current reference image is a long-term reference image (step S1004). If it is determined that the current reference image is a long-term reference image (step S1004: “Yes”), the inter prediction unit 511 extracts the current motion vector by the first extraction circuit identical to Embodiment 1 (step S1005).

[0144] If it is determined that the current reference image is not a long-term reference image (step S1004: "No"), the inter prediction unit 511 determines whether the shared reference image is a long-term reference image (step S1006).

[0145] If it is determined that the shared reference image is not a long-term reference image (step S1006: "No"), the inter prediction unit 511 extracts the current motion vector by a second extraction circuit identical to Embodiment 1 (step S1007). In other words, in the event that it is determined that the current reference image and the jointly located reference image are a short-term reference image, the inter prediction unit 511 extracts the current motion vector by the second extraction circuit.

[0146] If it is determined that the co-located reference image is a long-term reference image (step S1006: "Yes"), the inter prediction unit 511 selects another co-located block in the co-located image (step S1008). In the example shown in FIG. 10, a block encoded with respect to a short-term reference picture is selected as another co-located block.

[0147] Thereafter, the inter prediction unit 511 identifies the co-located reference image and the co-located motion vector corresponding to another co-located block (step S1009). The inter prediction unit 511 further extracts the current motion vector by means of a second extraction circuit that uses POC-based scaling (step S1010).

[0148] In detail, if the reference image of the current block is a short-term reference image, and the reference image of the co-located block is a long-term reference image, the inter prediction unit 511 does not extract the current motion vector from the motion vector of the co-located block. The inter prediction unit 511 instead selects another co-located block encoded with respect to the short-term reference image and extracts the current motion vector from the motion vector of the selected other co-located block.

[0149] As an example, in the case where the reference image of the current block is a short-term reference image and the reference image of the co-located block is a long-term reference image, the inter prediction unit 511 searches for a block encoded with respect to the short-term reference image. Inter prediction unit 511 selects a block encoded with respect to the short-term reference picture as another co-located block.

[0150] As another example, in the case where the reference image of the current block is a short-term reference image and the reference image of the co-located block is a long-term reference image, the inter prediction unit 511 first searches for a block encoded with respect to the short-term reference image.

[0151] In the event that a block encoded with respect to the short-term reference picture is present, the inter prediction unit 511 selects the block as another co-located block. In the event that a block encoded with respect to the short-term reference image is not present, the inter prediction unit 511 searches for a block encoded with respect to the long-term reference image. Inter prediction unit 511 selects a block encoded with respect to the long-term reference image as another co-located block.

[0152] For example, the inter prediction unit 511 first selects the first block shown in FIG. 4, as a co-located unit. In the event that the current reference image is a short-term reference image and the co-located reference image is a long-term reference image, then the external prediction unit 511 again selects the second block shown in FIG. 4, as a co-located unit.

[0153] In the above example, the inter prediction module 511 may select the second block shown in FIG. 4, as a co-located block, only if the reference image of the second block is a short-term reference image. The block selected here as the co-located block is not limited to the second block shown in FIG. 4, and a block other than the second block may be selected as a co-located block.

[0154] FIG. 11 is a diagram for explaining a co-located unit according to this embodiment. Samples c0, c1, c2 and c3 in a co-located image are shown in FIG. 11. Samples c0 and c1 in FIG. 11 are equal to samples c0 and c1 in FIG. 4. Not only the second block including the c1 sample, but also the third block including the c2 sample, or the fourth block including the c3 sample, can be selected as another co-located block.

[0155] The coordinates of the sample c2 are (x + w-1, y + h-1). The coordinates of the c3 sample are (x + 1, y + 1).

[0156] Inter prediction module 511 determines for each of the first, second, third, and fourth blocks in this order whether the block is available or not. Inter prediction module 511 determines the available block as the final co-located block. Examples of a case where a block is not available includes a case in which a block is not present and a case in which a block is encoded by intra prediction.

[0157] In the event that the current reference image is a short-term reference image, the inter prediction module 511 may determine that the block encoded with respect to the long-term reference image is not available.

[0158] Although an example of a method for selecting a co-located block is described above, the method for selecting the co-located block is not limited to the above example. A block including a sample other than samples c0, c1, c2 and c3 may be selected as a co-located block. In addition, the priority order of blocks is not limited to the example described in this embodiment.

[0159] The following describes a more specific example of the process of extracting the current motion vector again with reference to FIG. 10. The extraction process described above can be modified as follows.

[0160] First, the inter prediction unit 511 selects a co-located image in the same manner as in Embodiment 1 (step S1001). Then, the inter prediction unit 511 selects a first block including the sample c0 shown in FIG. 11, as a co-located block, and identifies a co-located reference image (steps S1002 and S1003).

[0161] After that, the inter prediction module 511 determines whether or not the co-located block is available. In the event that the current reference image is a short-term reference image and the co-located reference image is a long-term reference image, the inter prediction unit 511 determines that the co-located block is unavailable (steps S1004 and S1006).

[0162] In the event that the co-located block is unavailable, the inter prediction module 511 searches and selects another co-located block that is available (step S1008). In detail, the inter prediction unit 511 selects a block encoded with respect to the short-term reference picture from a second block including a sample c1, a third block including a sample c2, and a fourth block including a sample c3, in FIG. 11. The inter prediction unit 511 further identifies the reference image of the co-located block (step S1009).

[0163] In the case where an available co-located block is selected, the inter prediction module 511 sets the time predictor of the motion vector to be available. In the event that an available co-located block is not selected, the inter prediction module 511 sets the time predictor of the motion vector as unavailable.

[0164] In the event that the time predictor of the motion vector is set as available, the inter prediction module 511 identifies the co-located motion vector as the base motion vector (steps S1003 and S1009). In case the co-located block has many motion vectors, i.e. in case the co-located block is encoded using a plurality of motion vectors, the external prediction module 511 selects a basic motion vector according to a predetermined priority order similar to Embodiment 1.

[0165] In the event that any of the current reference image and the shared reference image is a long-term reference image (step S1004: "Yes"), the inter prediction unit 511 extracts the basic motion vector as a temporary motion vector predictor (step S1005) .

[0166] On the other hand, in the event that none of the current reference image and the shared reference image is a long-term reference image (step S1004: "No"), the inter prediction unit 511 extracts a time predictor of the motion vector from the base motion vector by scaling based on the POC (steps S1007 and S1010).

[0167] If the time predictor of the motion vector is set to be unavailable, the inter prediction module 511 does not retrieve the time predictor of the motion vector.

[0168] Similar to Embodiment 1, the inter prediction module 511 adds a time motion vector predictor, set as available, to the list as a candidate for the current motion vector. Inter prediction module 511 further selects the current motion vector from the list.

[0169] As described above, in this embodiment, if the reference image of the current block is a short-term reference image and the reference image of the co-located block is a long-term reference image, the current motion vector is not extracted from the motion vector of the co-located block.

[0170] It is extremely difficult to retrieve the current motion vector with high prediction accuracy if one of the current reference image and the co-located reference image is a long-term reference image, and the other from the current reference image and the co-located reference image is a short-term reference image. With this in mind, the image encoding apparatus 500 and the image decoding apparatus 600 according to this embodiment prevent a decrease in prediction accuracy by the operation described above.

[0171] OPTION 3 IMPLEMENTATION

The image encoding device and image decoding device according to Embodiment 3 have structures identical to those in Embodiment 1. Therefore, the operations of the image encoding apparatus and the image decoding apparatus according to this embodiment are described below using the structure of the image encoding apparatus 500 shown in FIG. 5 and the structure of the image decoding apparatus 600 shown in FIG. 6.

[0172] The image encoding apparatus 500 according to this embodiment carries out the operation shown in FIG. 7, similarly to embodiment 1. An image decoding apparatus 600 according to this embodiment performs the operation shown in FIG. 8, similarly to embodiment 1. This embodiment is different from Embodiment 1 by the process of extracting current motion vectors. This is described in detail below.

[0173] FIG. 12 is a flowchart showing details of an extraction process according to this embodiment. The inter prediction module 511 according to this embodiment performs the operation shown in FIG. 12, instead of the operation shown in FIG. 9. The following describes mainly the operation of the inter prediction module 511 shown in FIG. 5. The operation of the external prediction module 608 shown in FIG. 6 is identical to the operation of the inter prediction module 511 shown in FIG. 5, wherein “encoding” is replaced by “decoding”.

[0174] First, the inter prediction unit 511 selects a co-located image from the plurality of available reference images (step S1201). Then, the inter prediction unit 511 selects the co-located block in the co-located image (step S1202). The inter prediction unit 511 further identifies the co-located reference image and the co-located motion vector (step S1203).

[0175] First, the inter prediction module 511 determines whether the current reference image is a long-term reference image (step S1204). If it is determined that the current reference image is a long-term reference image (step S1204: “Yes”), the inter prediction unit 511 extracts the current motion vector by the first extraction circuit identical to Embodiment 1 (step S1205).

[0176] If it is determined that the current reference image is not a long-term reference image (step S1204: “No”), the inter prediction unit 511 determines whether the shared reference image is a long-term reference image (step S1206).

[0177] If it is determined that the shared reference image is not a long-term reference image (step S1206: “No”), the inter prediction unit 511 extracts the current motion vector by a second extraction circuit identical to Embodiment 1 (step S1207). In other words, in the event that it is determined that the current reference image and the jointly located reference image are a short-term reference image, the inter prediction unit 511 extracts the current motion vector by the second extraction circuit.

[0178] If it is determined that the co-located reference image is a long-term reference image (step S1206: "Yes"), the inter prediction unit 511 selects another co-located image (step S1208). The inter prediction unit 511 further selects another co-located block in another co-located image (step S1209). In the example shown in FIG. 12, the block encoded with respect to the short-term reference picture is selected as another co-located block.

[0179] Thereafter, the inter prediction unit 511 identifies the co-located reference image and the co-located motion vector corresponding to another co-located block (step S1210). The inter prediction unit 511 further extracts the current motion vector by means of a second extraction circuit that uses POC-based scaling (step S1211).

[0180] In detail, if the reference image of the current block is a short-term reference image, and the reference image of the co-located block is a long-term reference image, the external prediction module 511 does not extract the current motion vector from the motion vector of the co-located block.

[0181] The inter prediction unit 511 instead selects another co-located image. The inter prediction unit 511 further selects another co-located block encoded with respect to the short-term reference image from the selected other co-located image. Inter prediction module 511 extracts the current motion vector from the motion vector of a selected other co-located block.

[0182] As an example, in the case where the current reference image is a short-term reference image and the co-located reference image is a long-term reference image, the inter prediction unit 511 searches for an image that includes a block encoded with respect to the short-term reference image. The inter prediction unit 511 selects an image that includes a block encoded with respect to the short-term reference image as another co-located image.

[0183] As another example, in the case where the current reference image is a short-term reference image and the co-located reference image is a long-term reference image, the inter prediction unit 511 first searches for an image that includes a block encoded with respect to the short-term reference Images.

[0184] In the event that an image is present that includes a block encoded with respect to the short-term reference image, the inter prediction unit 511 selects the image as another co-located image.

[0185] In the event that an image that includes a block encoded with respect to the short-term reference image is not present, the inter prediction unit 511 searches for an image that includes a block encoded with respect to the long-term reference image. The inter prediction unit 511 selects an image that includes a block encoded with respect to the long-term reference image as another co-located image.

[0186] For example, if the image RefPicList0 [0] is a co-located image, the image RefPicList1 [0] is another co-located image. In case the image RefPicList1 [0] is a co-located image, the image RefPicList0 [0] is another co-located image.

[0187] In other words, an image listed first in one of two lists of reference images used for encoding B-images (bi-predictive coding) is a co-located image, and an image listed first in the other of the two lists of reference images is the other together posted image.

[0188] The following is a more specific example of the process of extracting the current motion vector again with reference to FIG. 12. The extraction process described above can be modified as follows.

[0189] First, the inter prediction unit 511 selects one of the image RefPicList0 [0] and the image RefPicList1 [0] as the co-located image (step S1201). The inter prediction unit 511 selects, from the selected co-located image, a first block including a sample c0 shown in FIG. 11, as a co-located block, and identifies a co-located reference image (steps S1202 and S1203).

[0190] After this, the inter prediction module 511 determines whether or not the co-located block is available. In the event that the current reference image is a short-term reference image and the co-located reference image is a long-term reference image, the inter prediction unit 511 determines that the co-located block is unavailable (steps S1204 and S1206).

[0191] In the event that the co-located block is unavailable, the inter prediction module 511 again selects the available co-located block. For example, the inter prediction module 511 selects a second block including the sample c1 shown in FIG. 11, as a co-located block. The inter prediction unit 511 further identifies a co-located reference image.

[0192] In the event that an available co-located block is not selected, the inter prediction unit 511 selects another co-located image. Here, the inter prediction unit 511 selects another from the image RefPicList0 [0] and the image RefPicList1 [0] as the co-located image (step S1208).

[0193] The inter prediction unit 511 selects, from the selected co-located image, a first block including a sample c0 shown in FIG. 1, as a co-located block, and identifies a co-located reference image (steps S1209 and S1210).

[0194] After that, the inter prediction module 511 determines whether or not the co-located block is available. Similar to the previous definition, in the event that the current reference image is a short-term reference image, and the co-located reference image is a long-term reference image, the inter prediction unit 511 determines that the co-located block is not available.

[0195] In the event that the co-located block is unavailable, the inter prediction unit 511 again selects the available co-located block (step S1209). In detail, the inter prediction unit 511 selects a second block including the sample c1 shown in FIG. 11, as a co-located block. The inter prediction unit 511 further identifies the co-located reference image (step S1210).

[0196] In the event that an accessible co-located block is ultimately selected, the inter prediction unit 511 sets the time predictor of the motion vector to be available. In the event that ultimately the available co-located block is not selected, the inter prediction module 511 sets the time predictor of the motion vector as unavailable.

[0197] In the event that the time predictor of the motion vector is set as available, the inter prediction module 511 identifies the motion vector of the co-located block as the base motion vector (steps S1203 and S1210). In case the co-located block has many motion vectors, i.e. in case the co-located block is encoded using a plurality of motion vectors, the external prediction module 511 selects a basic motion vector according to a predetermined priority order similar to Embodiment 1.

[0198] In the event that any of the current reference image and the shared reference image is a long-term reference image (step S1204: "Yes"), the inter prediction module 511 extracts the base motion vector as a temporary predictor of the motion vector (step S1205) .

[0199] On the other hand, in the event that none of the current reference image and the shared reference image is a long-term reference image (step S1204: "No"), the inter prediction unit 511 extracts a time motion vector predictor from the base motion vector by scaling based on the POC (steps S1207 and S1211).

[0200] In the event that the temporal predictor of the motion vector is set to be unavailable, the inter prediction module 511 does not retrieve the temporal predictor of the motion vector.

[0201] Similar to Embodiment 1, the inter prediction module 511 adds a time motion vector predictor defined as available to the list as a candidate for the current motion vector. Inter prediction module 511 further selects the current motion vector from the list.

[0202] As described above, the image encoding apparatus 500 and the image decoding apparatus 600 according to this embodiment select a block suitable for extracting the current motion vectors from the plurality of images, and extract the current motion vector from the motion vector of the selected block. This contributes to increased coding efficiency.

[0203] OPTION 4

Embodiment 4 reasonably describes representative structures and representative procedures included in embodiments 1-3.

[0204] FIG. 13A is a block diagram of an image decoding apparatus according to this embodiment. The image encoding device 1300 shown in FIG. 13A encodes each of the image blocks. The image encoding device 1300 includes an extraction module 1301, an add module 1302, a selection module 1303, and an encoder 1304.

[0205] For example, extraction module 1301, addition module 1302, and selection module 1303 correspond to the inter prediction module 511 shown in FIG. 5 and the like, and encoder 1304 corresponds to the entropy encoder 504 shown in FIG. 5 etc.

[0206] FIG. 13B is a flowchart showing the operation of the image encoding apparatus 1300 shown in FIG. 13A.

[0207] The extraction module 1301 extracts a candidate for a motion vector of a current block from a motion vector of a co-located block (step S1301). A co-located block is a block included in an image other than an image that includes the current block to be encoded.

[0208] When extracting the candidate, the extraction module 1301 determines whether the reference image of the current block is a long-term reference image or a short-term reference image. The extraction module 1301 also determines whether the reference image of the co-located block is a long-term reference image or a short-term reference image.

[0209] If it is determined that the reference image of the current block and the reference image of the co-located block are a long-term reference image, the extraction module 1301 extracts the candidate from the motion vector of the co-located block by the first extraction circuit. The first extraction scheme is an extraction scheme that does not include scaling based on a time distance.

[0210] On the other hand, in the event that it is determined that the reference image of the current block and the reference image of the co-located block are a short-term reference image, on the other hand, the extraction module 1301 extracts the candidate from the motion vector of the co-located block by the second extraction circuit. The second extraction scheme is an extraction scheme that incorporates time-based scaling.

[0211] The add module 1302 adds the extracted candidate to the list (step S1302). The selection module 1303 selects a motion vector of the current block from the list to which the candidate has been added (step S1303).

[0212] Encoder 1304 encodes the current block using the selected motion vector and a reference image of the current block (step S1304).

[0213] FIG. 14A is a block diagram of an image decoding apparatus according to Embodiment 4. The image decoding apparatus 1400 shown in FIG. 14A, decodes each of the image blocks. The image decoding apparatus 1400 includes an extraction module 1401, an add module 1402, a selection module 1403, and a decoder 1404.

[0214] For example, extraction module 1401, adding module 1402, and selection module 1403 correspond to the inter prediction module 608 shown in FIG. 6 and the like, and decoder 1404 corresponds to the entropy decoder 601 shown in FIG. 6 etc.

[0215] FIG. 14B is a flowchart showing the operation of the image decoding apparatus 1400 shown in FIG. 14A.

[0216] The extraction module 1401 extracts a candidate for the motion vector of the current block from the motion vector of the co-located block (step S1401). A co-located block is a block included in an image other than an image that includes a current block to be decoded.

[0217] When extracting the candidate, the extraction module 1401 determines whether the reference image of the current block is a long-term reference image or a short-term reference image. The extraction module 1401 also determines whether the reference image of the co-located unit is a long-term reference image or a short-term reference image.

[0218] In the case that it is determined that the reference image of the current block and the reference image of the co-located block are a long-term reference image, the extraction module 1401 extracts the candidate from the motion vector of the co-located block by the first extraction circuit. The first extraction scheme is an extraction scheme that does not include scaling based on a time distance.

[0219] On the other hand, in the event that it is determined that the reference image of the current block and the reference image of the co-located block are a short-term reference image, the extraction module 1401 extracts the candidate from the motion vector of the co-located block by the second extraction circuit. The second extraction scheme is an extraction scheme that incorporates time-based scaling.

[0220] The add module 1402 adds the extracted candidate to the list (step S1402). The selection module 1403 selects the motion vector of the current block from the list to which the candidate has been added (step S1403).

[0221] Decoder 1404 decodes the current block using the selected motion vector and a reference image of the current block (step S1404).

[0222] Through the process described above, the candidate for the current motion vector is appropriately extracted from the motion vector of the co-located block, although it is not extremely large or small. This contributes to improved prediction accuracy and increased coding efficiency.

[0223] Here, extraction modules 1301 and 1401 may not extract a candidate from the motion vector of the co-located block if it is determined that one of the reference image of the current block and the reference image of the co-located block is a long-term reference image, and the other from the reference image of the current block and the reference image of the co-located block is a short-term reference image.

[0224] In this case, the extraction modules 1301 and 1401 may further select another co-located block, encoded or decoded with respect to the short-term reference image, and extract a candidate from another co-located block by the second extraction circuit. Alternatively, extraction modules 1301 and 1401 may extract a candidate through another extraction scheme. As another alternative, retrieval modules 1301 and 1401 may ultimately not retrieve a candidate corresponding to a temporary motion vector predictor.

[0225] The extraction modules 1301 and 1401 can determine whether the reference image of the current block is a long-term reference image or a short-term reference image using the time distance between the reference image of the current block and the image that includes the current block.

[0226] The extraction modules 1301 and 1401 may determine whether the reference image of the co-located block is a long-term reference image or a short-term reference image using the time distance between the reference image of the co-located block and the image that includes the co-located block.

[0227] The extraction modules 1301 and 1401 may determine whether the reference image of the co-located block is a long-term reference image or a short-term reference image during the period during which the co-located block is encoded or decoded.

[0228] The extraction modules 1301 and 1401 may determine whether the reference image of the co-located block is a long-term reference image or a short-term reference image during the period during which the current block is encoded or decoded.

[0229] The first extraction scheme may be a motion vector extraction scheme of a co-located block as a candidate. The second extraction scheme may be a candidate extraction scheme by scaling the motion vector of the co-located block using the ratio of the time distance between the reference image of the current block and the image that includes the current block to the time distance between the reference image of the co-located block and the image that includes self-hosted block.

[0230] The encoder 1304 may further encode information indicating whether the reference image of the current block is a long-term reference image or a short-term reference image, and information indicating whether the reference image of the co-located block is a long-term reference image or a short-term reference image.

[0231] The decoder 1404 may further decode information indicating whether the reference image of the current block is a long-term reference image or a short-term reference image, and information indicating whether the reference image of a co-located block is a long-term reference image or a short-term reference image.

[0232] The extraction module 1401 may further determine whether the reference image of the current block is a long-term reference image or a short-term reference image using decoded information, and determine whether the reference image of the co-located block is a long-term reference image or a short-term reference image, with using decoded information.

[0233] Information indicating the classification of each reference image may be stored, as a parameter, in the bitstream at the location described below.

[0234] FIG. 15A is a diagram showing a first example of a storage location for a parameter indicating a classification of reference images. As shown in FIG. 15A, a parameter indicating the classification of the reference pictures can be stored in the header of the sequence. The sequence header is also referred to as a sequence parameter set.

[0235] FIG. 15B is a diagram showing a second example of a storage location for a parameter indicating a classification of reference images. As shown in FIG. 15B, a parameter indicating the classification of reference images may be stored in the image header. The image header is also referred to as a set of image parameters.

[0236] FIG. 15C is a diagram showing a third example of a storage location for a parameter indicating a classification of reference images. As shown in FIG. 15C, a parameter indicating the classification of reference images may be stored in the slice header.

[0237] Information indicating the prediction mode (external prediction or internal prediction) may be stored, as a parameter, in the bitstream at the location described below.

[0238] FIG. 16 is a diagram showing an example of a storage location for a parameter indicating a prediction mode. As shown in FIG. 16, the parameter may be stored in the CU header (coding unit header). The parameter indicates whether the prediction unit is encoded in the coding unit by means of external prediction or internal prediction. This parameter can be used to determine if a shared block is available or not.

[0239] Each of the structural elements in each of the above embodiments may be configured in the form of a fully hardware product or may be implemented by executing a program suitable for the structural element. Each of the structural elements can be implemented by a program execution module, such as a CPU and a processor, which reads and executes a program recorded on a recording medium, such as a hard disk or a semiconductor memory device. Here, a program for implementing an image encoding device, etc. according to each of the embodiments is a program described below.

[0240] The program instructs the computer to implement an image encoding method for encoding each of the image blocks, the image encoding method including: extracting a candidate for a motion vector of a current block to be encoded from a motion vector of a co-located block, which is a block included an image other than an image that includes the current block; adding the extracted candidate to the list; selection of the motion vector of the current block from the list to which the candidate is added; and encoding the current block using the selected motion vector and the reference image of the current block, the extraction includes: determining whether the reference image of the current block is a long-term reference image or a short-term reference image, and whether the reference image of the co-located block is long-term reference image or short-term reference image; extracting the candidate from the motion vector of the co-located block by means of a first extraction circuit that does not include scaling based on the time distance, in the case of determining that each of the reference image of the current block and the reference image of the co-located block is a long-term reference image; and extracting the candidate from the motion vector of the co-located block by means of a second extraction scheme that incorporates scaling based on the time distance, in the event that it is determined that each of the reference image of the current block and the reference image of the co-located block is a short-term reference image.

[0241] The program may instruct the computer to implement an image decoding method for decoding each of the image blocks, the image decoding method including: extracting a candidate for a motion vector of a current block to be decoded from a motion vector of a co-located block, which is a block, included in an image other than an image that includes the current block; adding the extracted candidate to the list; selection of the motion vector of the current block from the list to which the candidate is added; and decoding the current block using the selected motion vector and the reference image of the current block, the extraction includes: determining whether the reference image of the current block is a long-term reference image or a short-term reference image, and whether the reference image of the co-located block is long-term reference image or short-term reference image; extracting the candidate from the motion vector of the co-located block by means of a first extraction circuit that does not include scaling based on the time distance, in the case of determining that each of the reference image of the current block and the reference image of the co-located block is a long-term reference image; and extracting the candidate from the motion vector of the co-located block by means of a second extraction scheme that incorporates scaling based on the time distance, in the event that it is determined that each of the reference image of the current block and the reference image of the co-located block is a short-term reference image.

[0242] Each of the structural elements may be a circuit. These circuits can be a single circuit or be separate circuits. Each of the structural elements may be implemented by a general-purpose processor or implemented by a special-purpose processor.

[0243] Image encoding apparatuses according to one or more embodiments are described above, but the scope of the present invention is not limited to these embodiments. Specialists in the art should easily take into account that various modifications can be made in these exemplary embodiments, and changes can be obtained by arbitrarily combining structural elements of various embodiments without actually departing from the scope of the present invention. Accordingly, all such modifications and changes may in fact be included in one or more of the embodiments disclosed herein.

[0244] For example, an image encoding and decoding device may include an image encoding device and an image decoding device. A process performed by a particular processor may be performed by another processor. The processes can be performed in a different order, and two or more processes can be performed in parallel.

[0245] OPTION 5 IMPLEMENTATION

The processing described in each of the embodiments can be implemented simply in an independent computer system, by recording, on a recording medium, a program for implementing configurations of a moving image encoding method (image encoding method) and a moving image decoding method (image decoding method) described in each of the embodiments. Recording media may be any recording medium provided that the program can be recorded, such as a magnetic disk, an optical disk, a magneto-optical disk, an IC card, and a semiconductor memory device.

[0246] The rest of this document describes applications to the moving image coding method (image encoding method) and the moving image decoding method (image decoding method) described in each of the embodiments, and to systems using them. The system has a feature of having an image encoding and decoding device, which includes an image encoding device using an image encoding method and an image decoding device using an image decoding method. Other configurations in the system may be changed appropriately depending on the application.

[0247] FIG. 17 illustrates the overall configuration of a content providing system ex100 for implementing content distribution services. The area for the provision of communication services is divided into cells of the required size, and base stations ex106, ex107, ex108, ex109 and ex110, which are fixed wireless stations, are located in each of the cells.

[0248] The content providing system ex100 connects to devices such as a computer ex111, a personal digital device (PDA) ex112, a camera ex113, a cellphone ex114 and a gaming machine ex115, via the Internet ex101, an Internet service provider ex102, a telephone network ex104, and also base stations ex106-ex110, respectively.

[0249] However, the configuration of the content providing system ex100 is not limited to the configuration shown in FIG. 17, and the combination in which any of the elements are connected is valid. In addition, each device can connect to the telephone network ex104 directly, and not through the base stations ex106-ex110, which are fixed wireless stations. In addition, the devices can be connected to each other via short-range wireless communication, etc.

[0250] An ex113 camera, such as a digital video camera, allows video capture. An ex116 camera, such as a digital camera, can capture both still images and video. In addition, the ex114 cell phone can be a telephone that meets any of the standards such as Global System for Mobile Communications (GSM) (registered trademark), Code Division Multiple Access (CDMA), Code Division Multiple Access (W -CDMA), Long Term Evolution (LTE) and High Speed Packet Access (HSPA). Alternatively, the ex114 cell phone may comply with a Personal Mobile Phone System (PHS) standard.

[0251] In the content providing system ex100, the streaming server ex103 connects to the camera ex113 and the like. via the telephone network ex104 and the base station ex109, which provides the distribution of live broadcast images, etc. With this distribution, content (e.g., live music concert video) captured by the user using the camera ex113 is encoded as described above in each of the embodiments (i.e., the camera acts as an image encoding device according to an aspect of the present invention ), and the encoded content is transmitted to the streaming server ex103. On the other hand, the streaming server ex103 streams the transmitted content data to clients upon request. Clients include a computer ex111, a PDA ex112, a camera ex113, a cell phone ex114 and a gaming machine ex115 that allow decoding of the above encoded data. Each of the devices that receives the distributed data decodes and reproduces the encoded data (i.e., acts as an image decoding device according to an aspect of the present invention).

[0252] The captured data can be encoded by the camera ex113 or the streaming server ex103, which transmits the data, or the encoding processes can be shared between the camera ex113 and the streaming server ex103. Similarly, distributed data may be decoded by clients or a streaming server ex103, or decoding processes may be shared between clients and a streaming server ex103. In addition, data of still images and videos captured not only by the camera ex113, but also the camera ex116, can be transmitted to the streaming server ex103 through the computer ex111. Encoding processes may be performed by or shared between the camera ex116, the computer ex111, or the streaming server ex103.

[0253] In addition, encoding and decoding processes can be performed by the LSI ex500, generally included in each of the computer ex111 and devices. The LSI ex500 can be configured from a single chip or multiple crystals. Video encoding and decoding software can be integrated into a certain type of recording medium (such as a CD-ROM, floppy disk and hard disk) that is readable by an ex111 computer and the like, and encoding and decoding processes can be performed using software. In addition, when the cell phone ex114 comprises a camera, video data obtained by the camera can be transmitted. Video data is data encoded by the LSI ex500 included in the ex114 cell phone.

[0254] In addition, the streaming server ex103 may consist of servers and computers, and may decentralize data and process decentralized data, record or distribute data.

[0255] As described above, clients can receive and play encoded data in the content providing system ex100. In other words, clients can receive and decode user-transmitted information and play back the decoded data in real time in the content providing system ex100, so that a user who does not have specific rights and equipment can implement personal broadcast.

[0256] In addition to the exemplary content providing system ex100, at least one of the moving image encoding device (image encoding device) and the moving image decoding device (image decoding device) described in each of the embodiments may be implemented in a digital broadcast system ex200 illustrated in FIG. 18. More specifically, the broadcast station ex201 reports or transmits on the radio waves to the broadcast satellite ex202 the multiplexed data obtained by multiplexing audio data and the like. in the video data. The video data is data encoded by the moving image encoding method described in each of the embodiments (i.e., data encoded by the image encoding device according to an aspect of the present invention). When receiving multiplexed video data, broadcast satellite ex202 transmits radio waves for broadcast transmission. Then, the ex204 antenna for home use with the satellite broadcast reception function receives radio waves. Further, a device such as a television receiver (receiver) ex300 and a set-top box (STB) ex217 decodes the received multiplexed data and reproduces the decoded data (i.e., acts as an image decoding device according to an aspect of the present invention).

[0257] Furthermore, the reader / writer module ex218 (i) reads and decodes the multiplexed data recorded on the recording medium ex215, such as DVD and BD, or (i) encodes the video signals on the recording medium ex215, and in some cases records data, obtained by multiplexing an audio signal into encoded data. The reader / writer module ex218 may include a moving image decoding device or a moving image encoding device, as shown in each of the embodiments. In this case, the reproduced video signals are displayed on the monitor ex219 and can be reproduced by another device or system using the recording medium ex215 on which the multiplexed data is recorded. It is also possible to implement a moving picture decoding apparatus in an ex217 set-top box connected to an ex203 cable cable or an ex204 antenna for satellite and / or terrestrial broadcasting in order to display video signals on an ex219 monitor of an ex300 television receiver. A moving image decoding device may not be implemented in a set-top box, but in an ex300 television receiver.

[0258] FIG. 19 illustrates an ex300 television receiver (receiver) that uses a moving picture coding method and a moving picture decoding method described in each of the embodiments. The ex300 television receiver includes: a tuner ex301 that receives or provides multiplexed data obtained by multiplexing audio data into video data through an antenna ex204 or cable ex203, etc., which receives broadcast transmission; a modulation / demodulation module ex302 that demodulates the received multiplexed data or modulates the data into multiplexed data to be supplied to the outside; and a multiplexing / demultiplexing unit ex303 that demultiplexes the modulated multiplexed data into video and audio data or multiplexes the video and audio data encoded by the signal processor ex306 into data.

[0259] The television receiver ex300 further includes: a signal processor ex306 including an audio signal processor ex304 and a video signal processor ex305 that decode audio data and video data and encode audio data and video data, respectively (which act as an image encoding device and an image decoding device according to aspects of the present invention); and an output module ex309 including a speaker ex307 that provides the decoded audio signal, and a display module ex308 that displays the decoded video signal, such as a display. In addition, the television receiver ex300 includes an interface module ex317 including a function input module ex312 that receives a user operation input. In addition, the television receiver ex300 includes a control module ex310 that provides overall control of each constituent element of the television receiver ex300, and a power circuit module ex311 that provides power to each of the elements. In addition to the function input module ex312, the interface module ex317 may include: a bridge ex313 that connects to an external device, such as a read / write module ex218; a slot module ex314 for enabling the attachment of a recording medium ex216, such as an SD card; ex315 driver for connecting to an external recording medium such as a hard disk; and ex316 modem in order to connect to the telephone network. Here, the recording medium ex216 can electrically record information using a non-volatile / non-volatile semiconductor storage element. The constituent elements of the ex300 television receiver are connected to each other via a synchronous bus.

[0260] First, a configuration is described in which a television receiver ex300 decodes multiplexed data received externally through an antenna ex204 and the like, and reproduces decoded data. In a television receiver ex300, in a user operation through a remote controller ex220 or the like, the multiplexing / demultiplexing unit ex303 demultiplexes the multiplexed data demodulated by the modulation / demodulation unit ex302 under the control of the control unit ex310 including a CPU. In addition, the audio signal processor ex304 decodes the demultiplexed audio data, and the video signal processor ex305 decodes the demultiplexed video data using the decoding method described in each of the embodiments in the television receiver ex300. The output module ex309 provides the decoded video signal and the audio signal outward, respectively. When the output module ex309 provides a video signal and an audio signal, the signals may be temporarily stored in the buffers ex318 and ex319 and the like. so that the signals are reproduced synchronously with each other. In addition, the ex300 television receiver can read the multiplexed data not via broadcast or the like, but from recording media ex215 and ex216, such as a magnetic disk, an optical disk, and an SD card. The following describes a configuration in which an ex300 television receiver encodes an audio signal and a video signal and transmits data outward or writes data to a recording medium. In the television receiver ex300, in a user operation through a remote controller ex220 or the like, the audio signal processor ex304 encodes the audio signal, and the video signal processor ex305 encodes the video signal under the control of the ex310 control module using the encoding method as described in each of the embodiments. The multiplexing / demultiplexing unit ex303 multiplexes the encoded video signal and the audio signal and provides the resulting signal outward. When the multiplexing / demultiplexing unit ex303 multiplexes the video signal and the audio signal, the signals may be temporarily stored in the buffers ex320 and ex321 and the like. so that the signals are reproduced synchronously with each other. Here, the buffers ex318, ex319, ex320 and ex321 may be many, as illustrated, or at least one buffer may be shared in the television receiver ex300. In addition, data can be stored in a buffer, so that overflow and underflow in the system, for example, between modulation / demodulation module ex302 and multiplexing / demultiplexing module ex303, may not be allowed.

[0261] Furthermore, the ex300 television receiver may include a configuration for receiving AV input from a microphone or camera, different from a configuration for receiving audio and video data from a broadcast transmission or recording medium, and may encode the received data. Although the ex300 television receiver can encode, multiplex, and provide outward data in the description, it can only accept, decode, and provide outward data, and not encode, multiplex, and provide outward data.

[0262] Furthermore, when the reader / writer module ex218 reads or writes multiplexed data from or to the recording medium, one of the television receiver ex300 and the reader / writer module ex218 can decode or code the multiplexed data, and the television receiver ex300 and the reader / module ex218 / records can share decoding or encoding.

[0263] As an example, FIG. 20 illustrates the configuration of an information reproducing / recording unit ex400 when data is read or written from or to an optical disc. The information reproducing / recording unit ex400 includes constituent elements ex401, ex402, ex403, ex404, ex405, ex406 and ex407, which are described below. The optical head ex401 emits a laser spot on the recording surface of the recording medium ex215, which is an optical disc for recording information, and detects reflected light from the recording surface of the recording medium ex215 to read information. The modulation recording module ex402 electrically excites a semiconductor laser included in the ex401 optical head and modulates the laser radiation according to the recorded data. The demodulation reproducing module ex403 amplifies the reproduction signal obtained by electrically detecting reflected light from the recording surface using a photo detector included in the optical head ex401, and demodulates the reproduction signal by separating a signal component recorded on the recording medium ex215 to reproduce the necessary information. The buffer ex404 temporarily stores information to be recorded on the recording medium ex215 and information reproduced from the recording medium ex215. The disk motor ex405 rotates the recording medium ex215. The servo control module ex406 moves the optical head ex401 to a predetermined information track while simultaneously controlling the rotary drive of the disk motor ex405 to follow the laser spot. The system control unit ex407 exercises overall control of the information playback / recording unit ex400. The read and write processes can be implemented by the system control module ex407 using various information stored in the ex404 buffer and generating and adding new information as necessary, and by the modulation recording module ex402, a playback demodulation module ex403, and servo control module ex406 that record and reproduce information through the optical head ex401 in coordinated control. The system control module ex407 includes, for example, a microprocessor, and performs processing by instructing the computer to execute a program for reading and writing.

[0264] Although the ex401 optical head emits a laser spot in the description, it can record at high density using light in a near field.

[0265] FIG. 21 illustrates a recording medium ex215, which is an optical disc. On the recording surface of the recording medium ex215, the guide grooves are formed in a spiral, and the information track ex230 records, in advance, address information indicating an absolute position on the disc according to a change in the shape of the guide grooves. The address information includes information for determining positions of recording blocks ex231, which are a unit for recording data. Reproduction of the information track ex230 and reading of the address information in a device that records and reproduces data can lead to the determination of the positions of the recording blocks. In addition, the recording medium ex215 includes a data recording area ex233, an inner circle zone ex232, and an outer circle zone ex234. The data recording zone ex233 is a zone for use in recording user data. The inner circle zone ex232 and the outer circle zone ex234, which are inside and outside the data recording zone ex233, respectively, are for a specific application except for recording user data. The information reproducing / recording unit 400 reads and writes encoded audio, encoded video data or multiplexed data obtained by multiplexing encoded audio and encoded video data from and to a data recording area ex233 of a recording medium ex215.

[0266] Although an optical disc having a layer such as DVD and BD is described as an example in the description, the optical disc is not so limited and may be an optical disc having a multilayer structure and allowing recording on a part other than the surface. In addition, the optical disk may have a structure for multidimensional recording / reproducing, for example, recording information using light of colors with different wavelengths in one part of the optical disk and for recording information having different layers from different angles.

[0267] In addition, a vehicle ex210 having an antenna ex205 may receive data from a satellite ex202 and the like. and play the video on a display device, such as an ex211 car navigation system installed in an ex210 car, in an ex200 digital broadcast system. Here, the configuration of the car navigation system ex211 is a configuration, for example, including a GPS receiving module, from the configuration illustrated in FIG. 19. The same applies to the configuration of the computer ex111, cell phone ex114, etc.

[0268] FIG. 22A illustrates a cellular telephone ex114 that uses a moving picture coding method and a moving picture decoding method described in embodiments. The cell phone ex114 includes: an antenna ex350 for transmitting and receiving radio waves through the base station ex110; an ex365 camera module capable of capturing moving and still images; and a display module ex358, for example, a liquid crystal display for displaying data, such as decoded video, captured by a camera module ex365 or received by an ex350 antenna. The cell phone ex114 further includes: a main body module including a function key module ex366; an audio output module ex357, for example, a speaker for outputting audio; an audio input module ex356, for example, a microphone for inputting audio; a storage device ex367 for storing captured video or still images, recorded audio, encoded or decoded data of the received video, still images, mail messages, etc .; and a socket module ex364, which is an interface module for a recording medium that stores data in the same manner as the memory device ex367.

[0269] An example of a configuration of a cellular telephone ex114 will now be described with reference to FIG. 22B. In the ex114 cellphone, the main control module ex360, designed to fully control each module of the main body including the display module ex358, as well as the function key module ex366, is interconnected, via the synchronous bus ex370, with the power circuit module ex361, control module ex362 function input, video signal processor ex355, camera interface module ex363, liquid crystal display (LCD) control module ex359, modulation / demodulation module ex352, multiplexer module ex353 anija / demultiplexing audio processor ex354, ex364 module sockets and the memory ex367.

[0270] When the call end key or the power key is turned on by a user operation, the power circuit module ex361 provides power to the respective modules from the battery so as to activate the cell phone ex114.

[0271] In the cellular telephone ex114, the audio signal processor ex354 converts the audio signals collected by the audio input module ex356 in the voice communication mode into digital audio signals under the control of a main control module ex360 including CPU, ROM and RAM. Then, the modulation / demodulation module ex352 performs spread spectrum processing for digital audio signals, and the ex351 transceiver module performs digital-to-analog and frequency conversion for the data so as to transmit the resulting data through the ex350 antenna. Also, in the cell phone ex114, the transceiver module ex351 amplifies the data received by the ex350 antenna in voice communication mode and performs frequency conversion and analog-to-digital conversion for the data. Then, the modulation / demodulation module ex352 performs backward spread spectrum processing for the data, and the audio signal processor ex354 converts them to analog audio signals so as to output them through the audio output module ex357.

[0272] Furthermore, when email is transmitted in a data transmission mode, text email data inputted by operations with function key module ex366 and the like. of the main body are sent to the main control module ex360 through the function input control module ex362. The main control module ex360 instructs the modulation / demodulation module ex352 to perform spread spectrum processing for text data, and the transceiver module ex351 performs digital-to-analog and frequency conversion for the resulting data to transmit data to the base station ex110 via the ex350 antenna. When a mail message is received, processing that is approximately inverse with respect to processing for sending a mail message is performed on the received data, and the resulting data is provided to the display unit ex358.

[0273] When video, still images, or video and audio in data mode are transmitted, the video signal processor ex355 compresses and encodes the video signals provided from the camera module ex365 using the moving image encoding method shown in each of the embodiments (ie acts as an image encoding device according to an aspect of the present invention), and transmits the encoded video data to the multiplexing / demultiplexing unit ex353. In contrast, during the time that the camera module ex365 captures video, still images and the like, the audio signal processor ex354 encodes the audio signals collected by the audio input module ex356 and transmits the encoded audio data to the multiplexing / demultiplexing module ex353.

[0274] The multiplexing / demultiplexing unit ex353 multiplexes encoded video data supplied from the video signal processor ex355 and encoded audio data supplied from the audio signal processor ex354 using a predetermined method. Then, the modulation / demodulation module ex352 (modulation / demodulation scheme module) performs spread spectrum processing for the multiplexed data, and the transceiver module ex351 performs digital-to-analog and frequency conversion for the data so as to transmit the resulting data through the ex350 antenna.

[0275] When receiving data of a video file that is linked to a web page or the like, in a data transmission mode or when receiving a mail message with attached video and / or audio, to decode the multiplexed data received via the ex350 antenna, the multiplexing unit ex353 / demultiplexing demultiplexes the multiplexed data into a video bit stream and an audio data bit stream and provides encoded video data to an ex355 video signal processor and encoded audio data to an ex354 audio signal processor through synchronous bus ex370. The video signal processor ex355 decodes the video signal using the moving image decoding method corresponding to the moving image encoding method shown in each of the embodiments (i.e., acts as an image decoding apparatus according to an aspect of the present invention), and then the display unit ex358 displays, for example, video images and still images included in the video file associated with the web page via the ex359 LCD control module. In addition, the audio signal processor ex354 decodes the audio signal, and the audio output module ex357 provides audio.

[0276] Furthermore, similarly to the ex300 television receiver, a terminal, such as an ex114 cell phone, probably has 3 types of configurations implemented, including not only (i) a transceiver terminal including both an encoding device and a device decoding, but also (ii) a transmitting terminal including only an encoding device, and (iii) a receiving terminal including only a decoding device. Although in the description, the ex200 digital broadcast system receives and transmits multiplexed data obtained by multiplexing the audio data into video data in the description, the multiplexed data may be data obtained by multiplexing not the audio data, but the character data associated with the video, into the video data, and may not be multiplexed data, and the video data itself.

[0277] In fact, the moving picture coding method and the moving picture decoding method in each of the embodiments can be used in any of the described devices and systems. Thus, the advantages described in each of the embodiments can be obtained.

[0278] In addition, the present invention is not limited to the above embodiments, and various modifications and changes can be made in any of the embodiments in the present invention.

[0279] OPTION 6 IMPLEMENTATION

Video data may be generated by switching, as necessary, between (i) a moving image encoding method or a moving image encoding device shown in each of the embodiments, and (ii) a moving image encoding method or a moving image encoding device in accordance with another standard such as MPEG-2, MPEG-4 AVC and VC-1.

[0280] Here, when a plurality of video data that conforms to different standards is generated and then decoded, decoding methods must be selected so that they conform to different standards. However, since it cannot be detected which standard each of the plurality of video data to be decoded conforms to, there is a problem that the proper decoding method cannot be selected.

[0281] In order to solve the problem, multiplexed data obtained by multiplexing audio data or the like. in video data, have a structure including identification information indicating to which standard the video data conforms. The following describes a specific structure of multiplexed data including video data generated in a moving image coding method and by a moving image coding device shown in each of the embodiments. Multiplexed data is a digital stream in the MPEG-2 transport stream format.

[0282] FIG. 23 illustrates the structure of multiplexed data. As illustrated in FIG. 23, multiplexed data can be obtained by multiplexing at least one of a video stream, an audio stream, a presentation graphics stream (PG), and an interactive graphics stream. The video stream represents the primary video and the secondary video of the film, the audio stream (IG) represents the primary audio portion and the secondary audio portion to be mixed with the primary audio portion, and the presentation graphics stream represents the subtitles of the film. Here, the primary video is ordinary video that should be displayed on the screen, and the secondary video is video that should be displayed in a smaller window in the primary video. In addition, the interactive graphics stream represents an interactive screen, which should be formed by placing GUI components on the screen. The video stream is encoded in a moving picture coding method or through a moving picture coding device shown in each of the embodiments, or in a moving picture coding method or by a moving picture coding device in accordance with a conventional standard such as MPEG-2, MPEG-4 AVC and VC-1. The audio stream is encoded in accordance with such a standard as AC-3 Dolby, Dolby Digital Plus, MLP, DTS, DTS-HD and linear PCM.

[0283] Each stream included in the multiplexed data is identified by a PID. For example, 0 × 1011 is allocated to the video stream that should be used for the movie video, 0 × 1100-0 × 111F are allocated to audio streams, 0 × 1200-0 × 121F are allocated to presentation graphics streams, 0 × 1400-0 × 141F are allocated to interactive graphics streams, 0 × 1B00-0 × 1B1F are allocated to the video streams that should be used for the secondary video of the film, and 0 × 1A00-0 × 1A1F are allocated to the audio streams that should be used for the secondary audio, which should be mixed with the primary audio.

[0284] FIG. 24 schematically illustrates how data is multiplexed. Firstly, the video stream ex235, consisting of video frames, and the audio stream ex238, consisting of audio frames, are converted into a stream of PES packets ex236 and a stream of PES packets ex239 and additionally into TS packets ex237 and TS packets ex240, respectively. Similarly, the data of the presentation graphics stream ex241 and the interactive graphics stream ex244 are converted into a stream of PES packets ex242 and a stream of PES packets ex245 and further into TS packets ex243 and TS packets ex246, respectively. These TS packets are multiplexed into the stream to receive multiplexed ex247 data.

[0285] FIG. 25 illustrates how a video stream is stored in a stream of PES packets. The first panel in FIG. 25 shows a stream of video frames in a video stream. The second panel shows the flow of PES packets. As indicated by arrows denoted by yy1, yy2, yy3 and yy4 in FIG. 25, the video stream is divided into images, such as I-images, B-images and P-images, each of which is a unit of the video presentation, and the images are stored in the operational data of each of the PES packets. Each of the PES packets has a PES header, and the PES header stores a presentation time stamp (PTS) indicating an image display time and a decoding time stamp (DTS) indicating an image decoding time.

[0286] FIG. 26 illustrates the format of TS packets to be finally written to the multiplexed data. Each of the TS packets is a fixed length 188-byte packet including a 4-byte TS header having information such as a PID for identifying a stream, and 184-byte working TS data for storing data. PES packets are split and stored in the TS operational data, respectively. When a BD-ROM is used, each of the TS packets is assigned a 4-byte TP_Extra_Header, thereby resulting in 192-byte source packets. Source packets are written to multiplexed data. TP_Extra_Header stores information such as Arrival_Time_Stamp (ATS). ATS shows the start time of the transmission at which each of the TS packets should be transmitted to the PID filter. Source packets are placed in multiplexed data, as shown at the bottom of FIG. 26. Numbers increasing from the header of the multiplexed data are called source packet numbers (SPNs).

[0287] Each of the TS packets included in the multiplexed data includes not only audio, video, subtitle streams, etc., but also a program association table (PAT), a program structure table (PMT), and a time stamp programs (PCR). The PAT shows what the PID indicates in the PMT used in the multiplexed data, and the PID of the PAT itself is registered as zero. PMT stores the PIDs of video, audio, subtitle, and the like streams included in the multiplexed data and stream attribute information corresponding to the PIDs. PMT also has various descriptors associated with multiplexed data. Descriptors have information such as copy control information indicating whether copying of the multiplexed data is allowed or not. The PCR stores STC time information corresponding to the ATS, showing when the PCR packet is transmitted to the decoder in order to achieve synchronization between the arrival timer (ATC), i.e. time axis ATS, and system timer (STC), i.e. time axis of PTS and DTS.

[0288] FIG. 27 details the structure of the PMT data. The PMT header is located at the top of the PMT. The PMT header describes the length of the data included in the PMT, and the like. Many descriptors associated with the multiplexed data are located after the PMT header. Information, such as copy control information, is described in descriptors. After the descriptors are many pieces of stream information associated with the streams included in the multiplexed data. Each piece of stream information includes stream descriptors, each of which describes information such as a stream type for identifying a stream compression codec, a stream PID, and stream attribute information (such as frame rate or aspect ratio). The stream descriptors are equal in number to the number of streams in the multiplexed data.

[0289] When the multiplexed data is recorded on a recording medium, etc., it is recorded together with the information files of the multiplexed data.

[0290] Each of the multiplexed data information files is control information of the multiplexed data, as shown in FIG. 28. The multiplexed data information files are in one-to-one correspondence with the multiplexed data, and each of the files includes multiplexed data information, stream attribute information and an entry map.

[0291] As illustrated in FIG. 28, the multiplexed data information includes system speed, playback start time and playback end time. The system speed indicates the maximum transfer rate at which the system target decoder, which should be described below, transmits the multiplexed data to the PID filter. The ATS intervals included in the multiplexed data are set to no more than system speed. The start time of the playback indicates the PTS in the video frame in the header of the multiplexed data. The interval of one frame is added to the PTS in the video frame at the end of the multiplexed data, and the PTS is set to the end time of playback.

[0292] As shown in FIG. 29, a piece of attribute information is recorded in stream attribute information for each PID of each stream included in the multiplexed data. Each piece of attribute information has different information depending on whether the corresponding stream is a video stream, an audio stream, a presentation graphics stream, or an interactive graphics stream. Each piece of information of the attributes of the video stream carries information including which compression codec is used to compress the video stream, as well as the resolution, aspect ratio, and frame rate of the fragments of image data that are included in the video stream. Each piece of attribute information of an audio stream carries information including which compression codec is used to compress the audio stream, how many channels are included in the audio stream, which language supports the audio stream, and how high the sampling rate is. The video stream attribute information and the audio stream attribute information are used to initialize the decoder before the player reproduces the information.

[0293] In the present embodiment, the multiplexed data to be used has the stream type included in the PMT. In addition, when the multiplexed data is recorded on the recording medium, information of the attributes of the video stream included in the information of the multiplexed data is used. More specifically, the moving image encoding method or moving image encoding device described in each of the embodiments includes a step or module for extracting unique video data indicating information generated by the moving image encoding method or moving image encoding device in each of the embodiments , for the type of stream included in the PMT, or video stream attribute information. With this configuration, the video data generated by the moving image coding method or the moving image coding device described in each of the embodiments may differ from the video data that conform to another standard.

[0294] In addition, FIG. 30 illustrates the steps of a moving picture decoding method according to the present embodiment. In step exS100, the type of stream included in the PMT or attribute information of the video stream included in the information of the multiplexed data is obtained from the multiplexed data. Then, in step exS101, it is determined whether or not the type of stream or the attribute information of the video stream indicates that the multiplexed data is generated by the video encoding method or the video encoding device in each of the embodiments. When it is determined that the stream type or the video stream attribute information indicates that multiplexed data is generated by the moving image encoding method or the moving image encoding device in each of the embodiments, in step exS102, decoding is performed by the moving image decoding method in each of the embodiments . In addition, when the stream type or attribute information of the video stream indicates compliance with traditional standards such as MPEG-2, MPEG-4 AVC and VC-1 in step exS103, decoding is performed by a moving picture decoding method in accordance with traditional standards.

[0295] Also, highlighting a new unique value for the stream type or the attribute information of the video stream provides a determination of whether or not to perform decoding, a moving image decoding method or a moving image decoding device that are described in each of the embodiments. Even when multiplexed data that conforms to another standard is input, an appropriate decoding method or apparatus can be selected. Thus, it becomes possible to decode information without errors. In addition, the moving image encoding method or device or the moving image decoding method or device in the present embodiment can be used in the devices and systems described above.

[0296] OPTION 7 IMPLEMENTATION

Each of the moving picture coding method, the moving picture coding device, the moving picture decoding method, and the moving picture decoding device in each of the embodiments are typically implemented in the form of an integrated circuit or a large integrated circuit (LSI). As an example of LSI, FIG. 31 illustrates the configuration of an LSI ex500, which consists of a single chip. LSI ex500 includes elements ex501, ex502, ex503, ex504, ex505, ex506, ex507, ex508 and ex509, which should be described below, and the elements are connected to each other via bus ex510. The power circuit module ex505 is activated by providing each of the batteries when the power circuit module ex505 is turned on.

[0297] For example, when encoding is performed, the LSI ex500 receives an AV signal from a microphone ex117, a camera ex113, and the like. via the AV input / output ex509 under the control of the control unit ex501 including the CPU ex502, a storage controller ex503, a stream controller ex504, and a drive frequency control unit ex512. The received AV signal is temporarily stored in the external memory ex511, for example, SDRAM. Under the control of the control module ex501, the stored data is segmented into pieces of data according to the volume and processing speed so that they are transmitted to the signal processor ex507. Then, the signal processor ex507 encodes the audio signal and / or video signal. Here, the encoding of a video signal is the encoding described in each of the embodiments. In addition, the signal processor ex507 sometimes multiplexes the encoded audio data and the encoded video data, and the streaming input / output ex506 provides the multiplexed data to the outside. The provided bitstream is transmitted to the base station ex107 or recorded on the recording medium ex215. When the data sets are multiplexed, the data should be temporarily stored in the buffer ex508 so that the data sets are synchronized with each other.

[0298] Although the ex511 memory is an item outside the LSI ex500, it can be included in the LSI ex500. The ex508 buffer is not limited to one buffer, but may consist of buffers. In addition, the LSI ex500 can consist of a single crystal or multiple crystals.

[0299] Furthermore, although the control unit ex501 includes a CPU ex502, a storage controller ex503, a stream controller ex504, a drive frequency control unit ex512, the configuration of the control unit ex501 is not limited to this. For example, the signal processor ex507 may further include a CPU. The inclusion of another CPU in the signal processor ex507 may increase processing speed. In addition, as another example, the CPU ex502 may act as or be part of a signal processor ex507 and, for example, may include an audio signal processor. In such a case, the control unit ex501 includes a signal processor ex507 or a CPU ex502 including a portion of the signal processor ex507.

[0300] The name used here is LSI, but it may also be called IC, system LSI, super-LSI or ultra-LSI depending on the degree of integration.

[0301] In addition, methods to achieve integration are not limited to LSI, and a special circuit or general-purpose processor, etc. also allows integration. User-Programmable Gate Array (FPGA), which can be programmed after manufacturing the LSI, or a reconfigurable processor that allows reconfiguration of the connection or configuration of the LSI, can be used for a similar purpose.

[0302] In the future, with the development of semiconductor technology, a completely new technology may replace LSI. Function blocks can be integrated using this technology. It is possible that the present invention is applicable to biotechnology.

[0303] OPTION 8

When video data generated by the moving image coding method or by the moving image coding device described in each of the embodiments is decoded, compared to when video data that conforms to a conventional standard such as MPEG-2, MPEG-4 AVC and VC is decoded -1, the amount of processing is likely to increase. Thus, the LSI ex500 must be set at an excitation frequency higher than the frequency of the CPU ex502, which should be used when the video data is decoded in accordance with the traditional standard. However, when the excitation frequency is set higher, there is a problem in that the power consumption increases.

[0304] In order to solve the problem, a moving picture decoding device such as a television receiver ex300 and an LSI ex500 is configured to determine which standard the video data conforms to and switch between driving frequencies according to a specific standard. FIG. 32 illustrates a configuration of ex800 in the present embodiment. The driving frequency switching unit ex803 sets the driving frequency to a high driving frequency when the video data is generated by the moving image encoding method or the moving image encoding device described in each of the embodiments. Then, the driving frequency switching unit ex803 instructs the decoding processor ex801, which implements the moving image decoding method described in each of the embodiments, to decode the video data. When the video data complies with the traditional standard, the driving frequency switching unit ex803 sets the driving frequency to a lower driving frequency than the driving frequency of the video data generated by the moving image coding method or the moving image coding device described in each of the embodiments. Then, the driving frequency switching unit ex803 instructs the decoding processor ex802, which conforms to the conventional standard, to decode the video data.

[0305] More specifically, the driving frequency switching unit ex803 includes a CPU ex502 and the driving frequency control unit ex512 in FIG. 31. Here, each of the decoding processor ex801 that implements the moving image decoding method described in each of the embodiments and the decoding processor ex802 that conforms to the conventional standard corresponds to the signal processor ex507 of FIG. 31. The CPU ex502 determines to which standard the video data conforms. Then, the driving frequency control unit ex512 determines the driving frequency based on the signal from the CPU ex502. In addition, the signal processor ex507 decodes the video data based on the signal from the CPU ex502. For example, the identification information described in Embodiment 6 is likely to be used to identify video data. The identification information is not limited to the identification information described in Embodiment 6, but may be any information provided that the information indicates to which standard the video data conforms. For example, when what standard the standard video data conforms to can be determined based on an external signal to determine that the video data is used for a television receiver or disc, etc., determination can be made based on such an external signal. In addition, the CPU ex502 selects a drive frequency based on, for example, a lookup table in which video data standards are associated with drive frequencies, as shown in FIG. 34. The excitation frequency can be selected by storing the lookup table in the ex508 buffer and the LSI internal memory and by accessing the lookup table by the CPU ex502.

[0306] FIG. 33 illustrates the steps for implementing the method in the present embodiment. First, in step exS200, the signal processor ex507 obtains identification information from the multiplexed data. Then, in step exS201, the CPU ex502 determines based on the identification information whether or not video data is generated by the encoding method and the encoding device described in each of the embodiments. When video data is generated by the moving image encoding method and the moving image encoding device described in each of the embodiments in step exS202, the CPU ex502 transmits a signal for setting the driving frequency to a higher driving frequency to the driving frequency control unit ex512. Then, the driving frequency control unit ex512 sets the driving frequency to a higher driving frequency. On the other hand, when the identification information indicates that the video data conforms to a traditional standard such as MPEG-2, MPEG-4 AVC and VC-1, in step exS203, the CPU ex502 transmits a signal for setting the driving frequency to a lower driving frequency to the ex512 module control the frequency of excitation. Then, the driving frequency control unit ex512 sets the driving frequency to a lower driving frequency than the driving frequency if the video data is generated by the moving image encoding method and the moving image encoding device described in each of the embodiments.

[0307] In addition, along with switching the driving frequencies, the energy-saving effect can be enhanced by changing the voltage to be applied to the LSI ex500 or to a device including the LSI ex500. For example, when the drive frequency is set lower, the voltage to be applied to the LSI ex500 or to a device including the LSI ex500 is likely to be set to a voltage lower than the voltage if the drive frequency is set higher.

[0308] Further, when the processing amount for decoding is larger, the driving frequency may be set higher, and when the processing volume for decoding is smaller, the driving frequency may be set lower as a method for setting the driving frequency. Thus, the setting method is not limited to the methods described above. For example, when the processing amount for decoding video data in accordance with MPEG-4 AVC exceeds the processing amount for decoding video data generated by the moving image encoding method and the moving image encoding device described in each of the embodiments, the driving frequency is probably set in the reverse order regarding the task described above.

[0309] Furthermore, a method for setting a driving frequency is not limited to a method for setting a driving frequency less. For example, when the identification information indicates that video data is generated by the moving image encoding method and the moving image encoding device described in each of the embodiments, the voltage to be applied to the LSI ex500 or to a device including the LSI ex500 is probably set great. When the identification information indicates that the video data conforms to a traditional standard such as MPEG-2, MPEG-4 AVC and VC-1, the voltage to be applied to the LSI ex500 or to a device including the LSI ex500 is likely to be set lower. As another example, when the identification information indicates that video data is generated by the moving image encoding method and the moving image encoding device described in each of the embodiments, the excitation of the CPU ex502 probably should not be stopped. When the identification information indicates that the video data conforms to a traditional standard such as MPEG-2, MPEG-4 AVC and VC-1, the excitation of the CPU ex502 is probably stopped at this time, because the CPU ex502 has additional processing performance. Even when the identification information indicates that video data is generated by the moving image encoding method and the moving image encoding device described in each of the embodiments, if the CPU ex502 has additional processing performance, the excitation of the CPU ex502 is probably stopped at this time. In such a case, the pause time is probably set to be shorter than the pause time when the identification information indicates that the video data conforms to a traditional standard such as MPEG-2, MPEG-4 AVC and VC-1.

[0310] Accordingly, the energy-saving effect can be enhanced by switching between driving frequencies according to the standard to which the video data conforms. In addition, when the LSI ex500 or device including the LSI ex500 is energized using a battery, the battery life can be extended due to the energy-saving effect.

[0311] OPTION 9

There are cases when a lot of video data that meet various standards is provided to devices and systems, such as a television receiver and cell phone. In order to decode a plurality of video data that conforms to different standards, the ex507 LSI ex500 signal processor must conform to different standards. However, problems with scaling up the LSI ex500 circuitry and rising costs arise with the separate use of signal processors ex507 that comply with the relevant standards.

[0312] In order to solve the problem, a configuration has been devised in which a decoding processor is partially shared to implement a moving image decoding method described in each of the embodiments, and a decoding processor that conforms to a conventional standard such as MPEG-2, MPEG-4 AVC and VC-1. Ex900 in FIG. 35A shows an example configuration. For example, the moving image decoding method described in each of the embodiments and the moving image decoding method that corresponds to MPEG-4 AVC have, in part, jointly processing information such as entropy encoding, inverse quantization, filtering for deblocking, and prediction with motion compensation. Information about the processing to be shared probably includes the use of a decoding processor ex902 that corresponds to MPEG-4 AVC. In contrast, the dedicated decoding processor ex901 is probably used for other processing unique to an aspect of the present invention. Since an aspect of the present invention is specifically different by inter prediction, for example, a dedicated decoding processor ex901 is used for inter prediction. Otherwise, the decoding processor is probably shared for one of entropy decoding, filtering for deblocking and inverse quantization, or for all processing. A decoding processor for implementing the moving picture decoding method described in each of the embodiments may be shared for processing to be shared, and a dedicated decoding processor may be used for processing unique to MPEG-4 AVC.

[0313] Furthermore, ex1000 in FIG. 35B shows another example in which processing is partially shared. This example uses a configuration including a dedicated decoding processor ex1001 that supports processing unique to an aspect of the present invention, a dedicated decoding processor ex1002 that supports processing unique to another traditional standard, and a decoding processor ex1003 that supports processing that must be shared used by a moving image decoding method according to an aspect of the present invention and a traditional moving decoding method images. Here, the dedicated decoding processors ex1001 and ex1002 are not necessarily specialized for processing according to an aspect of the present invention and processing according to the traditional standard, respectively, and may be processors capable of implementing general processing. In addition, the configuration of the present embodiment may be implemented by the LSI ex500.

[0314] Likewise, reducing the scale of the LSI scheme and reducing costs is possible by sharing a decoding processor for processing to be shared by a moving image decoding method according to an aspect of the present invention and a moving image decoding method in accordance with a conventional standard.

[0315] INDUSTRIAL APPLICABILITY

The present invention is applicable to a television receiver, digital video recorder, car navigation system, cell phone, digital camera, digital video camera, and the like.

LIST OF REFERENCE NUMBERS

[0316] 500, 1300 - image encoding device

501 - subtraction module

502 - conversion module

503 - quantization module

504 - entropy encoder

505, 602 - inverse quantization module

506, 603 - inverse transformation module

507, 604 - summing module

508, 605 - block memory

509, 606 - image memory

510, 607 - internal forecasting module

511, 608 - external forecasting module

512, 609, 1303, 1403 - selection module

600, 1400 - image decoding device

601 - entropy decoder

1301, 1401 - extraction module

1302, 1402 - add module

1304 - encoder

1404 - decoder

Claims (30)

1. An image encoding method for encoding each block of image blocks, the image encoding method comprising the steps of: extracting a bi-predictive candidate for the motion vectors of the current block to be encoded from the first motion vector of the first block included in the first image, the first image being different from the image that includes the current block; add the extracted bi-predictive candidate to the candidate list; selecting one bi-predictive candidate, which includes a second motion vector and a third motion vector, from the list of candidates; and encode the current block using (i) a second motion vector and a third motion vector, as well as (ii) a second reference image and a third reference image of the current block, the second reference image and the third reference image corresponding to the second motion vector and the third motion vector, wherein the extraction includes the steps of: determining (i) whether each of the second reference image and the third reference image of the current block is a long-term reference image or a short-term reference image, and (ii) whether the first reference image is a first a block with a long-term reference image or a short-term reference image; extracting each of the second motion vector and the third motion vector so that they are identical to the first motion vector without scaling based on the time distance, in the event that it is determined that (i) the corresponding one of the second reference image and the third reference image is a long-term reference image and that (ii) the first reference image is a long-term reference image; and each of the second motion vector and the third motion vector is extracted from said first motion vector by scaling based on a time distance in the case of determining that (i) the corresponding one of the second reference image and the third reference image is a short-term reference image and (ii) the first reference The image is a short-term reference image. 2. The method of encoding images according to claim 1, in which upon extraction: for each of the second motion vector and the third motion vector, the bi-predictive candidate is not extracted from the first motion vector of the first block if it is determined that (i) one of the corresponding one of the second reference image and the third reference image of the current block and the first reference image the first block is a long-term reference image, a (ii) another one of the corresponding one of the second reference image and the third reference image of the current block and the first reference images of the first block is a short reference image; and for each of the second motion vector and the third motion vector, the bi-predictive candidate is extracted from the first motion vector of the first block (i) if it is determined that (a) the corresponding one of the second reference image and the third reference image of the current block is a long-term reference image and (b) the first reference image of the first block is a long-term reference image, or (ii) if it is determined that (a) the corresponding one of the second reference image and the third PORN image of the current block is a short reference image, and (b) the first reference image is the first block of the reference image briefly. 3. The method of encoding images according to claim 1, wherein the encoding further includes (i) encoding information indicating whether the second reference image of the current block is a long-term reference image or a short-term reference image, and (ii) encoding information indicating whether the third reference image of the current block is a long-term reference image image or short-term reference image, and (iii) information indicating whether the first reference image of the first block is a long-term reference image or short temporary reference image. 4. The method of encoding images according to claim 1, in which the extraction includes: determining whether the second reference image of the current block is a long-term reference image or a short-term reference image using the time distance between the second reference image of the current block and the image that includes the current block; determining whether the third reference image of the current block is a long-term reference image or a short-term reference image using the time distance between the third reference image of the current block and the image that includes the current block; and determining whether the first reference image of the first block is a long-term reference image or a short-term reference image using the time distance between the first reference image of the first block and the first image that includes the first block. 5. The method of encoding images according to claim 1, in which the extraction includes determining whether the first reference image of the first block is a long-term reference image or a short-term reference image, during the period during which the first block is encoded. 6. The method of encoding images according to claim 1, in which the extraction includes determining whether the first reference image of the first block is a long-term reference image or a short-term reference image, during the period during which the current block is encoded. 7. An image encoding device that encodes each block of image blocks, the image encoding device comprising: an extraction module configured to extract a bi-predictive candidate for the motion vectors of the current block to be encoded from the first motion vector of the first block included in the first image, the first image being different from the image that includes the current block; an add module configured to add the extracted bi-predictive candidate to the candidate list; a selection module, configured to select one bi-predictive candidate, which includes a second motion vector and a third motion vector, from the list of candidates; and an encoder configured to encode the current block using (i) a second motion vector and a third motion vector, as well as (ii) a second reference image and a third reference image of the current block, the second reference image and the third reference image corresponding to the second motion vector and third motion vector, wherein the extraction module is configured to: determining (i) whether each of the second reference image and the third reference image of the current block is a long-term reference image or a short-term reference image, and (ii) whether the first reference image of the first block is a long-term reference image or a short-term reference image; extracting each of the second motion vector and the third motion vector so that they are identical to the first motion vector without scaling based on the time distance, if it is determined that (i) the corresponding one of the second reference image and the third reference image is a long-term reference image and that (ii) the first reference image is a long-term reference image; and extracting each of the second motion vector and the third motion vector from said first motion vector by scaling based on a time distance, in the event that it is determined that (i) the corresponding one of the second reference image and the third reference image is a short-term reference image and (ii) the first the reference image is a short-term reference image.

Images