JP6614275B2 - Receiving device, receiving method, transmitting device, and transmitting method - Google Patents

Description

This technology, reception apparatus receiving method, related to the transmission apparatus and transmission method.

  When a compressed moving image is serviced by broadcast, network, etc., the upper limit of the frame frequency that can be reproduced is limited by the decoding capability of the receiver. Therefore, it is necessary for the service side to restrict the service to a low frame frequency service or to provide a plurality of high and low frame frequency services at the same time in consideration of the reproduction capability of popular receivers.

  The receiver is expensive to support a high frame frequency service, which is an obstacle to early diffusion. Only low-cost receivers dedicated to low frame frequency services are prevailing at the beginning, and if the service side starts high frame frequency services in the future, it will be impossible to view without a new receiver, and new services will spread. It becomes an obstruction factor.

  For example, H.M. In H.265 / HEVC (High Efficiency Video Coding), temporal direction scalability has been proposed by hierarchically encoding image data of each picture constituting moving image data (see Non-Patent Document 1). On the receiving side, the hierarchy of each picture can be identified based on the temporal ID (temporal_id) inserted in the header of the NAL (Network Abstraction Layer) unit, and selective decoding up to the hierarchy corresponding to the decoding capability becomes possible. .

Gary J. Sullivan, Jens-Rainer Ohm, Woo-Jin Han, Thomas Wiegand, “Overview of the High Efficiency Video Coding (HEVC) Standard” IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECNOROGY, VOL. 22, NO. 12, pp 1649-1668, DECEMBER 2012

  An object of the present technology is to enable a good decoding process on the receiving side.

The concept of this technology is
Classifying the image data of each picture constituting the moving image data into a plurality of layers, encoding the image data of the classified pictures of each layer, and dividing the plurality of layers into a predetermined number of layer sets; An image encoding unit for generating the predetermined number of video streams respectively having encoded image data of pictures of each of the divided hierarchical groups;
A transmission unit configured to transmit a container of a predetermined format including the generated predetermined number of video streams;
The image encoding unit is
At least the transmission apparatus performs encoding so that the decoding intervals of the encoded image data of the pictures in the lowest layer set are equal.

  In the present technology, the image encoding unit encodes the image data of each picture constituting the moving image data to generate a predetermined number of video streams. In this case, the image data of each picture constituting the moving image data is classified into a plurality of layers and encoded. Then, the plurality of hierarchies are divided into a predetermined number of hierarchies, and a predetermined number of video streams each having the encoded image data of the pictures of the divided hierarchies are generated.

  In the image encoding unit, encoding is performed so that at least the decoding intervals of the encoded image data of the pictures in the lowest layer set are equal. For example, the image encoding unit encodes all of the pictures in the hierarchical groups whose decoding timing of the encoded image data of the pictures in the hierarchical group positioned higher than the lowest hierarchical group is lower than the hierarchical group. The encoding may be performed so as to be an intermediate timing of the decoding timing of the image data. Thus, for example, when the receiving side has the ability to decode not only the lowest layer set but also the encoded image data of a picture of a layer set higher than that, the decoding process of each picture is performed. It becomes possible to proceed smoothly one after another.

  Further, for example, the image encoding unit includes a plurality of hierarchies so that the lowest hierarchy set includes a plurality of hierarchies, and the hierarchy set positioned higher than the lowest hierarchy set includes one hierarchy. It may be arranged such that it is divided into a number of hierarchical groups. Thus, for example, when the receiving side has a decoding capability capable of processing the encoded image data of the pictures of a plurality of hierarchies included in the lowest hierarchy set, the encoded image data of the pictures of the lowest hierarchy set is provided. Only a video stream having a video stream is selected and fetched into a buffer and decoded, and a complicated configuration such as a process of combining a plurality of video streams is unnecessary.

  The transmission unit transmits a container of a predetermined format including the predetermined number of video streams. For example, the container may be a transport stream (MPEG-2 TS) adopted in the digital broadcasting standard. Further, for example, the container may be MP4 used for Internet distribution or the like, or a container of other formats.

  As described above, according to the present technology, at least the decoding intervals of the encoded image data of the pictures in the lowest layer set are encoded so as to be equal intervals. Therefore, for example, when the receiving side has a decoding capability capable of processing the encoded image data of the pictures of a plurality of hierarchies included in the lowest hierarchy set, the decoding process of the encoded image data of each picture is continued without difficulty. Can be performed.

Other concepts of this technology are
Classifying the image data of each picture constituting the moving image data into a plurality of layers, encoding the image data of the classified pictures of each layer, and dividing the plurality of layers into a predetermined number of layer sets; An image encoding unit for generating the predetermined number of video streams respectively having encoded image data of pictures of each of the divided hierarchical groups;
A transmission unit for transmitting a container of a predetermined format including the generated predetermined number of video streams;
In the container layer, each of the predetermined number of video streams is a base stream having encoded image data of pictures of the lowest hierarchy set, or a hierarchy set positioned higher than the lowest hierarchy set The transmission apparatus includes an identification information insertion unit that inserts identification information for identifying whether the stream is an enhancement stream including encoded image data of a picture.

  In the present technology, the image encoding unit encodes the image data of each picture constituting the moving image data to generate a predetermined number of video streams. In this case, the image data of each picture constituting the moving image data is classified into a plurality of layers and encoded. Then, the plurality of hierarchies are divided into a predetermined number of hierarchies, and a predetermined number of video streams each having the encoded image data of the pictures of the divided hierarchies are generated.

  For example, the image encoding unit may perform encoding so that at least the decoding intervals of the encoded image data of the pictures in the lowest layer set are equal. In this case, for example, the image encoding unit may decode pictures of all hierarchical groups whose decoding timings of the encoded image data of the hierarchical group of pictures positioned higher than the lowest hierarchical group are lower than this hierarchical group. The encoded image data may be encoded so as to be intermediate in the decoding timing of the encoded image data.

  The transmission unit transmits a container of a predetermined format including the predetermined number of video streams. For example, the container may be a transport stream (MPEG-2 TS) adopted in the digital broadcasting standard. Further, for example, the container may be MP4 used for Internet distribution or the like, or a container of other formats.

The identification information insertion unit inserts identification information into the container layer. This identification information
Each of the predetermined number of video streams is a base stream having encoded image data of pictures of the lowest hierarchical group, or encoded image data of pictures of a hierarchical group positioned higher than the lowest hierarchical group. This is identification information for identifying whether or not the enhancement stream is included.

  For example, when there are a plurality of enhanced streams, the identification information may be such that each enhanced stream can be further identified. In addition, for example, the container is a transport stream, and the identification information insertion unit streams the identification information in a video elementary stream loop arranged corresponding to a predetermined number of video streams under the program map table. Insert as a type.

  As described above, in the present technology, identification information for identifying whether each of a predetermined number of video streams is a base stream or an enhanced stream is inserted into a container layer. Therefore, on the receiving side, by using this identification information, for example, it is possible to easily select only the base stream and selectively decode the encoded image data of the pictures in the lower layer set.

Other concepts of this technology are
Classifying the image data of each picture constituting the moving image data into a plurality of layers, encoding the image data of the classified pictures of each layer, and dividing the plurality of layers into a predetermined number of layer sets; An image encoding unit for generating the predetermined number of video streams respectively having encoded image data of pictures of each of the divided hierarchical groups;
A transmission unit for transmitting a container of a predetermined format including the generated predetermined number of video streams;
The transmission apparatus includes a configuration information insertion unit that inserts configuration information of the video stream corresponding to each of a predetermined number of video streams included in the container in the container layer.

  In the present technology, the image encoding unit encodes the image data of each picture constituting the moving image data to generate a predetermined number of video streams. In this case, the image data of each picture constituting the moving image data is classified into a plurality of layers and encoded. Then, the plurality of hierarchies are divided into a predetermined number of hierarchies, and a predetermined number of video streams each having the encoded image data of the pictures of the divided hierarchies are generated. Then, a container having a predetermined format including the predetermined number of video streams is transmitted by the transmission unit.

  The configuration information insertion unit inserts the configuration information of the video stream into the container layer corresponding to each of a predetermined number of video streams included in the container. For example, the container is a transport stream, and the configuration information insertion unit uses this configuration information as a descriptor in a video elementary stream loop arranged corresponding to a predetermined number of video streams under the program map table. It may be inserted.

  For example, the configuration information may include information indicating a service group to which the video stream belongs. Further, for example, the configuration information may include information indicating a dependency relationship between streams starting from a base stream having encoded image data of pictures in the lowest layer set. For example, the configuration information may include information indicating the number of hierarchies of a plurality of hierarchies classified by the image encoding unit.

  As described above, in the present technology, the configuration information of the video stream is inserted into the container layer corresponding to each of the predetermined number of video streams included in the container. Therefore, for example, on the receiving side, for each video stream included in the container, to which group it belongs, what kind of stream dependency it is, how many layers the hierarchy is related to, etc. It can be easily grasped.

Other concepts of this technology are
The picture data of each picture constituting the moving picture data is classified and encoded into a plurality of hierarchies, and the codes of the pictures in each hierarchic set obtained by dividing the plurality of hierarchies into a predetermined number of hierarchies. A receiving unit for receiving the predetermined number of video streams each having digitized image data;
A processing unit for processing the received predetermined number of video streams;
Among the predetermined number of video streams, at least the video stream having the encoded image data of the picture of the lowest layer set is in the receiving apparatus which is encoded so that the decoding interval of each picture is equal. .

  In the present technology, the image data of each picture constituting the moving image data is classified into a plurality of hierarchies and encoded by the receiving unit, and the plurality of hierarchies are divided into a predetermined number of hierarchies. Then, a predetermined number of video streams each having encoded image data of pictures in each hierarchical group are received. Then, the processing unit processes the received predetermined number of video streams.

  In this case, among the predetermined number of video streams, at least the video stream having the encoded image data of the pictures in the lowest hierarchical set is encoded so that the decoding intervals of each picture are equal. Therefore, for example, when there is a decoding capability capable of processing the encoded image data of a plurality of hierarchies included in the lowest hierarchy set, the decoding process of the encoded image data of each picture should be performed continuously without difficulty. Is possible.

  In the present technology, for example, a predetermined number of video streams have all the decoding timings of the encoded image data of the pictures of the hierarchical group positioned higher than the lowest hierarchical group at the lower level than this hierarchical group. The encoding may be performed so as to be an intermediate timing of the decoding timing of the encoded image data of the hierarchical set of pictures. As a result, for example, when there is an ability to decode not only the lowest hierarchical group but also the encoded image data of a picture of a hierarchical group positioned higher than the lowest hierarchical group, the decoding process of each picture proceeds sequentially and smoothly. It becomes possible.

Other concepts of this technology are
The picture data of each picture constituting the moving picture data is classified and encoded into a plurality of hierarchies, and the pictures of the respective hierarchies are obtained by dividing the plurality of hierarchies into a predetermined number of hierarchies. A receiving unit for receiving a container of a predetermined format including the predetermined number of video streams each having the encoded image data of
From the predetermined number of video streams included in the received container, the encoded image data of a picture of a predetermined hierarchy or lower according to the decoding capability is selectively taken into a buffer, and the code of each picture taken into the buffer is selected. An image decoding unit that decodes the converted image data and obtains image data of a picture of a layer below the predetermined layer,
Among the predetermined number of video streams, at least the video stream having the encoded image data of the picture of the lowest layer set is in the receiving apparatus which is encoded so that the decoding interval of each picture is equal. .

  In the present technology, a container having a predetermined format is received by the receiving unit. In this container, image data of each picture constituting moving image data is classified into a plurality of hierarchies and encoded, and one obtained by dividing the plurality of hierarchies into a predetermined number of hierarchies. A predetermined number of video streams having encoded image data of pictures in the above hierarchy are included.

  The image decoding unit selectively fetches the encoded image data of a picture of a layer below the predetermined layer corresponding to the decoding capability from the predetermined number of video streams included in the received container, and stores the encoded image data in the buffer. Then, the encoded image data of each picture is decoded, and image data of pictures in a hierarchy below a predetermined hierarchy is obtained. For example, the image decoding unit decodes the encoded image data of each picture as one stream based on the decoding timing information when the encoded image data of pictures of a predetermined hierarchy set is included in a plurality of video streams. You may be like.

  In this case, among the predetermined number of video streams, at least the video stream having the encoded image data of the pictures in the lowest hierarchical set is encoded so that the decoding intervals of each picture are equal. Therefore, for example, when there is a decoding capability capable of processing the encoded image data of a plurality of hierarchies included in the lowest hierarchy set, the decoding process of the encoded image data of each picture should be performed continuously without difficulty. Is possible.

  In the present technology, for example, each of a predetermined number of video streams in the container layer is a base stream including encoded image data of pictures in the lowest layer set or higher than this lowest layer set. The identification information for identifying whether or not the enhanced stream includes the encoded image data of the picture of the layer set positioned in is inserted, and based on the identification information, the image decoding unit includes a predetermined stream including the base stream The encoded image data of a predetermined layer set of pictures corresponding to the decoding capability from a number of video streams may be taken into a buffer and decoded. In this case, by using the identification information, for example, it is possible to easily select only the base stream and selectively decode the encoded image data of the low-layer set picture.

  In the present technology, for example, a post processing unit that matches the frame rate of the image data of each picture obtained by the image decoding unit with the display capability may be further provided. In this case, even if the decoding capability is low, it is possible to obtain image data with a frame rate suitable for the high display capability.

  According to the present technology, a good decoding process can be performed on the receiving side. Note that the effects described here are not necessarily limited, and may be any of the effects described in the present disclosure.

It is a block diagram which shows the structural example of the transmission / reception system as embodiment. It is a block diagram which shows the structural example of a transmitter. It is a figure which shows an example of the hierarchical encoding performed with an encoder. It is a figure which shows the content of the main parameter in the structural example of the NAL unit header, and the structural example. It is a figure for demonstrating the structure of the encoding image data of each picture by HEVC. It is a figure which shows an example of the encoding in the case of hierarchical encoding, decoding, a display order, and a delay. It is a figure which shows the encoding expectation of a hierarchical encoding, and the display expectation (display order) in a designated hierarchy. It is a figure for demonstrating the encoding timing (decoding timing) of the picture of a predetermined number of video streams (2 streams). It is a figure which shows an example of the encoding timing (decoding timing) of each picture in the case of producing | generating two video streams, a base stream and an enhancement stream. It is a figure which shows another example of the encoding timing (decoding timing) of each picture in the case of producing | generating two video streams, a base stream and an enhancement stream. It is a figure for demonstrating the encoding timing (decoding timing) of the picture of a predetermined number of video streams (3 streams). It is a figure which shows an example of the encoding timing (decoding timing) of each picture in the case of producing | generating three video streams of a base stream and two enhancement streams. It is a figure which shows another example of the encoding timing (decoding timing) of each picture in the case of producing | generating three video streams of a base stream and two enhancement streams. It is a figure which shows an example of HRD (Hypothetical Reference Decoder) control of an encoder. It is a block diagram which shows the structural example of an encoder. It is a figure which shows an example of the processing flow of an encoder. It is a figure which shows the structural example of a HEVC descriptor (HEVC_descriptor). It is a figure which shows the structural example of a multi-stream descriptor (multistream_descriptor). It is a figure which shows the content of the main information in the structural example of a multistream descriptor (multistream_descriptor). The figure which shows an example of "Stream_type", "Group_id", "max / min layer", "max_layer_in_group", "Stream_dependency_ordering" in the case where the transport stream TS includes, for example, video streams of services 1 and 2 It is. It is a block diagram which shows the structural example of a multiplexer. It is a figure which shows an example of the processing flow of a multiplexer. It is a figure which shows the structural example of the transport stream TS in 2 stream delivery. It is a figure which shows the structural example of the transport stream TS in 3 stream delivery. It is a block diagram which shows the structural example of a receiver. It is a block diagram which shows the structural example of a demultiplexer. It is a figure which shows an example of stream selection. It is a figure which shows an example of the processing flow of a demultiplexer. It is a block diagram which shows the structural example of a decoder. It is a figure which shows an example of stream coupling | bonding. It is a figure which shows the structural example of a post process part. It is a figure which shows an example of the processing flow of a decoder and a post-processing part.

Hereinafter, modes for carrying out the invention (hereinafter referred to as “embodiments”) will be described. The description will be given in the following order.
1. Embodiment 2. FIG. Modified example

<1. Embodiment>
[Transmission / reception system]
FIG. 1 shows a configuration example of a transmission / reception system 10 as an embodiment. The transmission / reception system 10 includes a transmission device 100 and a reception device 200.

  The transmission device 100 transmits a transport stream TS as a container on a broadcast wave. The transport stream TS is obtained by classifying and encoding image data of each picture constituting moving image data into a plurality of hierarchies, and dividing the plurality of hierarchies into a predetermined number of hierarchies. A predetermined number of video streams each including encoded image data of pictures in each hierarchical group is included. In this case, for example, H.264 / AVC, H.H. Encoding such as H.265 / HEVC is performed, and the referenced picture is encoded so as to belong to a self-layer and / or a layer lower than the self-layer.

  In this embodiment, when dividing a plurality of hierarchies into a predetermined number of hierarchies, the lowest hierarchy includes a plurality of hierarchies, and the hierarchy set higher than the lowest hierarchy set has one hierarchy. To be included. Due to such division, on the receiving side, for example, when there is a decoding capability capable of processing encoded image data of pictures of a plurality of hierarchies included in the lowest hierarchy set, the encoding of the pictures of the lowest hierarchy set is performed. Only the video stream having the converted image data can be selected and taken into the buffer to be decoded.

  Hierarchy identification information for identifying the belonging hierarchy is added to the encoded image data of the picture of each hierarchy for each picture. In this embodiment, hierarchical identification information (“nuh_temporal_id_plus1” meaning temporal_id) is arranged in the header portion of the NAL unit (nal_unit) of each picture. By adding the layer identification information in this way, on the receiving side, the layer identification of each picture becomes possible in the layer of the NAL unit, and the encoded image data of the layer below the predetermined layer is selectively extracted and decoded. It can be carried out.

  In this embodiment, among the predetermined number of video streams, at least the video stream having the encoded image data of the pictures in the lowest layer set is encoded so that the decoding interval of each picture is equal. . With this encoding, on the receiving side, when there is a decoding capability capable of processing the encoded image data of the pictures of a plurality of hierarchies included in the lowest hierarchy set, the decoding process of the encoded image data of each picture can be performed easily. It becomes possible to carry out continuously.

  In this embodiment, the decoding timing of the encoded image data of the pictures of the hierarchical group positioned higher than the lowest hierarchical group is the encoded image data of the pictures of all the hierarchical groups positioned lower than this hierarchical group. Encoding is performed so as to be an intermediate timing of the decoding timing. With this encoding, on the receiving side, if there is the ability to decode not only the lowest layer set but also the encoded image data of the pictures in the layer set higher than that, the decoding process of each picture is performed. It becomes possible to proceed smoothly one after another.

  In this embodiment, each of a predetermined number of video streams in the layer of the transport stream TS is a base stream having encoded image data of pictures in the lowest layer set, or from this lowest layer set Identification information for identifying whether the stream is an enhanced stream including encoded image data of a hierarchical set of pictures positioned at the upper level is inserted. This identification information is inserted as a stream type in a video elementary stream loop arranged corresponding to a predetermined number of video streams under the program map table. With this identification information, the receiving side can easily select only the base stream and selectively decode the encoded image data of the pictures in the lower layer set.

  In this embodiment, video stream configuration information is inserted into the transport stream TS layer corresponding to each of a predetermined number of video streams included therein. This configuration information is inserted as a descriptor in a video elementary stream loop arranged corresponding to a predetermined number of video streams under the program map table. With this configuration information, on the receiving side, for each video stream included in the container, to which group it belongs, what kind of stream dependency relationship, how many layers the hierarchy is related to, etc. Can be easily grasped.

  The receiving apparatus 200 receives the above-described transport stream TS transmitted from the transmitting apparatus 100 on a broadcast wave. The receiving apparatus 200 selectively fetches and decodes encoded image data of a picture of a hierarchy below a predetermined hierarchy selected according to the decoding capability from a predetermined number of video streams included in the transport stream TS, Image data of each picture is acquired and image reproduction is performed.

  As described above, the identification information for identifying whether the predetermined number of video streams is the base stream or the enhanced stream is included in the layer of the transport stream TS. Based on this identification information, a predetermined layer set of encoded image data corresponding to the decoding capability is fetched from the predetermined number of video streams including the base stream and processed.

  In addition, the receiving apparatus 200 performs post processing for matching the frame rate of the image data of each picture obtained by decoding as described above to the display capability. By this post processing, for example, even when the decoding capability is low, it is possible to obtain image data with a frame rate suitable for high display capability.

"Configuration of Transmitter"
FIG. 2 shows a configuration example of the transmission device 100. The transmission apparatus 100 includes a CPU (Central Processing Unit) 101, an encoder 102, a compressed data buffer (cpb: coded picture buffer) 103, a multiplexer 104, and a transmission unit 105. The CPU 101 is a control unit and controls the operation of each unit of the transmission device 100.

  The encoder 102 inputs non-compressed moving image data and performs hierarchical encoding. The encoder 102 classifies the image data of each picture constituting the moving image data into a plurality of hierarchies. Then, the encoder 102 encodes the image data of the classified pictures of each layer, and generates a video stream having the encoded image data of the pictures of each layer. The encoder 102 is, for example, H.264. H.264 / AVC, H.H. Encoding such as H.265 / HEVC is performed. At this time, the encoder 102 performs encoding so that a picture to be referred to (referenced picture) belongs to the own hierarchy and / or a hierarchy lower than the own hierarchy.

  FIG. 3 shows an example of hierarchical encoding performed by the encoder 102. In this example, the image data is classified into five layers from 0 to 4, and image data of pictures in each layer is encoded.

  The vertical axis represents the hierarchy. 0 to 4 are set as temporal_id (hierarchy identification information) arranged in the header portion of the NAL unit (nal_unit) constituting the encoded image data of the pictures of layers 0 to 4, respectively. On the other hand, the horizontal axis indicates the display order (POC: picture order of composition), the display time is on the left side, and the display time is on the right side.

  FIG. 4A shows a structure example (Syntax) of the NAL unit header, and FIG. 4B shows contents (Semantics) of main parameters in the structure example. In the 1-bit field of “Forbidden_zero_bit”, 0 is essential. The 6-bit field “Nal_unit_type” indicates the NAL unit type. The 6-bit field of “Nuh_layer_id” is assumed to be 0. A 3-bit field of “Nuh_temporal_id_plus1” indicates temporal_id and takes a value (1 to 7) obtained by adding 1.

  Returning to FIG. 3, each of the rectangular frames indicates a picture, and the numbers indicate the order of the encoded pictures, that is, the encoding order (decoding order on the receiving side). For example, a sub picture group (Sub group of pictures) is composed of 16 pictures from “2” to “17”, and “2” is the head picture of the sub picture group. “1” is a picture of the previous sub-picture group. Several of these sub-picture groups are gathered to form a GOP (Group Of Pictures).

  As shown in FIG. 5, the encoded image data of the first picture of the GOP is composed of NAL units of AUD, VPS, SPS, PPS, PSEI, SLICE, SSEI, and EOS. On the other hand, pictures other than the first picture of the GOP are configured by NAL units of AUD, PPS, PSEI, SLICE, SSEI, and EOS. VPS and SPS can be transmitted once in a sequence (GOP), and PPS can be transmitted in each picture.

  Returning to FIG. 3, the solid line arrows indicate the reference relationship of pictures in encoding. For example, the picture “2” is a P picture and is encoded with reference to the picture “1”. The picture “3” is a B picture and is encoded with reference to the pictures “1” and “3”. Similarly, other pictures are encoded with reference to nearby pictures in display order. Note that the picture of layer 4 is not referred to by other pictures.

  The encoder 102 divides a plurality of hierarchies into a predetermined number of hierarchies of two or more, and generates a predetermined number of video streams each having encoded image data of pictures in each hierarchic group. For example, the encoder 102 includes a plurality of hierarchies in the lowest hierarchy set and divides the hierarchy set higher than the lowest hierarchy set so as to include one hierarchy.

  For example, in the hierarchical coding example of FIG. 3, the encoder 102 has two hierarchies, with the hierarchies 0 to 3 being the lowest hierarchy group and the hierarchy 4 being the upper hierarchy group so as to be separated by a one-dot chain line. Divide into pairs. In this case, the encoder 102 generates two video streams (coded streams) each having the coded image data of each hierarchical set of pictures.

  Also, for example, in the example of hierarchical encoding in FIG. 3, the encoder 102 is configured such that the hierarchical levels 0 to 2 are the lowest hierarchical group and the hierarchical level 3 is positioned higher than the hierarchical level so as to be separated by a one-dot chain line and a two-dot chain line. Further, the hierarchy 4 is further divided into three hierarchy groups as the hierarchy group positioned at the higher level. In this case, the encoder 102 generates three video streams (coded streams) each having the coded image data of each hierarchical set of pictures.

In this case, a video stream having encoded image data of pictures in the lowest layer set is a base stream, and the stream type is “0x24”. In addition, a video stream including encoded image data of a picture of a hierarchical group positioned higher than the lowest hierarchical group is an enhanced stream, and the stream type is newly defined “0x25”.

  When there are a plurality of enhanced streams, it is possible to define a new stream type so that each enhanced stream can be identified instead of setting the stream types of all the enhanced streams to “0x25”. For example, when there are two enhanced streams, the stream type of the first enhanced stream is “0x25”, and the stream type of the second enhanced stream is “0x26”.

  This stream type constitutes identification information for identifying whether each of a predetermined number of video streams is a base stream or an enhanced stream. This stream type is inserted into the layer of the transport stream TS. That is, this stream type is inserted into a video elementary stream loop (Video ES loop) arranged corresponding to a predetermined number of video streams under a program map table (PMT).

  FIG. 6 shows an example of encoding, decoding, display order and delay at the time of hierarchical encoding. This example corresponds to the above-described hierarchical encoding example of FIG. This example shows a case where all layers (all layers) are hierarchically encoded at full time resolution. FIG. 6A shows the encoder input. As shown in FIG. 6B, each picture is encoded in the encoding order with a delay of 16 pictures to obtain an encoded stream. FIG. 6B shows the decoder input, and each picture is decoded in decoding order. Then, as shown in FIG. 6C, the image data of each picture is obtained in the display order with a delay of 4 pictures.

  FIG. 7A shows an encoded stream similar to the encoded stream shown in FIG. 6B described above, divided into three stages of layers 0 to 2, layer 3, and layer 4. Here, “Tid” indicates temporal_id. FIG. 7B shows the display expectation (display order) in the case of selectively decoding each picture in the partial hierarchies of hierarchies 0 to 2, that is, Tid = 0-2. FIG. 7C shows the display expectation (display order) in the case of selectively decoding the pictures of the layers 0 to 3, that is, the partial layers of Tid = 0-3. Further, FIG. 7D shows the display expectation (display order) in the case of selectively decoding the pictures of all the layers of layers 0 to 4, that is, Tid = 0 to 4.

  In order to decode the encoded stream of FIG. 7A according to the decoding capability, a decoding capability with a full resolution of time resolution is required. However, when decoding with Tid = 0-2, a decoder with 1/4 decoding capability should be able to process for the full encoded temporal resolution. Also, when decoding with Tid = 0-3, a decoder with 1/2 decoding capability should be able to process for the encoded full temporal resolution.

  However, if pictures belonging to a lower hierarchy that are referred to in hierarchical encoding are consecutive and are encoded at full timing with temporal resolution, the ability of the decoder to perform partial decoding cannot catch up. The period A in FIG. A decoder that decodes a partial hierarchy of Tid = 0 to 2 or Tid = 0 to 3 performs decoding and display with a capability of 1/4 or 1/2 of the time axis as shown in the display example. , A picture encoded in the period A and having full time resolution cannot be decoded.

  Ta indicates the time required for the decoding process for each picture in the decoder that decodes Tid = 0-2. Tb indicates the time required for the decoding process for each picture in the decoder that decodes Tid = 0-3. Tc indicates the time required for the decoding process for each picture in the decoder that decodes Tid = 0 to 4 (all layers). The relationship between these times is Ta> Tb> Tc.

  In this embodiment, the encoder 102 performs encoding so that at least the decoding intervals of the encoded image data of the pictures in the lowest hierarchy set are equal. FIG. 8A shows a case where each picture is encoded at a full 120 Hz timing with temporal resolution in the example of hierarchical encoding in FIG. 3, and layers 0 to 3 constitute a base stream (B stream). In this example, the hierarchy is divided into two hierarchy sets, and the hierarchy 4 is the hierarchy set constituting the enhancement stream (E stream) positioned at the upper level.

  In this case, although the temporal resolution of the pictures in the lowest layer set is 60 fps, there are pictures that are continuously encoded at a timing of 120 Hz, and a decoder having a decoding capability of 60 fps has a continuous and stable decoding process. Is impossible. For this reason, as shown in FIG. 8B, the encoding timing of the pictures in the lowest hierarchical group constituting the base stream is adjusted to be 60 Hz, and the encoded image data of the pictures in the lowest hierarchical group is set. Are encoded so that their decoding intervals are equal. As a result, a decoder having a decoding capability of 60 fps enables continuous and stable decoding processing on the encoded image data of the pictures in the lowest layer set constituting the base stream.

  Further, as shown in FIG. 8B, the encoding timing of the hierarchical group of pictures constituting the enhancement stream (E stream), and hence the decoding timing of the encoded image data of the picture, is the base stream (B stream). Encoding is performed so as to be an intermediate timing of the decoding timing of the encoded image data of the picture of the lowest layer group to be configured. As a result, when the receiving side has the ability to decode not only the base stream but also the encoded image data of the hierarchical set of pictures that make up the enhanced stream, the decoding process of each picture can proceed smoothly and smoothly. It becomes.

  FIG. 9 shows the encoding timing (decoding timing) of each picture in the case of generating two video streams of a base stream (B stream) and an enhancement stream (E stream) in the hierarchical encoding example of FIG. An example is shown. In this example, the decoding delay of the enhanced stream is minimized with respect to the base stream. The decoding delay in this case is 8 pictures at the encoding interval of full time resolution (1/2 of the encoding interval of the base stream).

  In this example, the encoding timing of the picture of the base stream (B stream) is an even timing, and the encoding timing of the enhancement stream (E stream) is an odd timing. In this example, the enhancement stream (E stream) is encoded immediately after the encoding order of the highest layer of the base stream (B stream). That is, the picture “9” of the enhancement stream (E stream) is encoded immediately after the picture “8” of the base stream (B stream).

  FIG. 10 shows the encoding timing (decoding timing) of each picture in the case of generating two video streams of a base stream (B stream) and an enhancement stream (E stream) in the example of hierarchical encoding of FIG. Another example is shown. In this example, the decoding delay of the enhanced stream is larger than that of the base stream. The decoding delay in this case is 16 pictures at the encoding interval of full time resolution (1/2 of the encoding interval of the base stream). When the decoding delay increases as described above, a large amount of reference memory is required inside dpb (decoded picture buffer).

  In this example, the encoding timing of the picture of the base stream (B stream) is an even timing, and the encoding timing of the enhancement stream (E stream) is an odd timing. In this example, the enhancement stream (E stream) is encoded after the highest layer encoding of the base stream (B stream) is completed. That is, the picture “17” of the enhancement stream (E stream) is encoded immediately after the picture “16” of the base stream (B stream).

  FIG. 11 (a) shows a case where each picture is encoded at a full 120 Hz timing with temporal resolution in the example of the hierarchical encoding of FIG. 3, and layers 0 to 2 constitute a base stream (B stream). A hierarchy in which the hierarchy 3 constitutes an enhanced stream (E stream 1) positioned higher than that, and a hierarchy 4 constitutes an enhanced stream (E stream 2) located in the upper hierarchy thereof. A case is shown in which a set is divided into three hierarchical sets.

  In this case, although the temporal resolution of the pictures in the lowest layer set is 30 fps, there are pictures that are continuously encoded at a timing of 120 Hz, and a decoder having a decoding capability of 30 fps has a continuous and stable decoding process. Is impossible. Therefore, as shown in FIG. 11 (b), the encoding timing of the pictures in the lowest layer set constituting the base stream is adjusted to be 30 Hz, and the encoded image data of the pictures in the lowest layer set Are encoded so that their decoding intervals are equal. As a result, a decoder having a decoding capability of 30 fps can continuously and stably decode the encoded image data of the pictures in the lowest layer set constituting the base stream.

  In addition, as shown in FIG. 11B, the encoding timing of the hierarchical group of pictures constituting the enhancement stream (E stream 1), and hence the decoding timing of the encoded image data of the picture, is the base stream (B stream). Encoding is performed so as to be an intermediate timing of the decoding timing of the encoded image data of the picture of the lowest layer group to be configured. Further, as shown in FIG. 11 (b), the encoding timing of the hierarchical group of pictures constituting the enhancement stream (E stream 2), and hence the decoding timing of the encoded image data of the picture, is the base stream (B stream) and Encoding is performed so as to be an intermediate timing of the decoding timing of the encoded image data of the hierarchical set of pictures constituting the enhancement stream (E stream 1). As a result, when the receiving side has the ability to decode not only the base stream but also the encoded image data of the hierarchical set of pictures that make up the two enhanced streams, the decoding process of each picture proceeds smoothly and sequentially. Is possible.

  FIG. 12 is a diagram illustrating an example of hierarchical encoding in FIG. 3 in which three video streams of a base stream (B stream), an enhanced stream (E stream 1), and an enhanced stream (E stream 2) are generated. An example of encoding timing (decoding timing) is shown. In this example, the decoding delay of the enhanced stream is minimized with respect to the base stream. The decoding delay in this case is 12 pictures at the encoding interval of full time resolution (1/4 of the encoding interval of the base stream).

  In this example, the encoding timing of the picture of the base stream (B stream) is a multiple of 4, the encoding timing of the enhancement stream (E stream 1) is a multiple of 4, and the base stream (B stream) The timing is intermediate between the picture encoding timings. The encoding timing of the enhancement stream (E stream 1) is an odd timing.

  In this example, the enhancement stream (E stream 1) is encoded immediately after the encoding order of the highest layer of the base stream (B stream). That is, the picture “10” of the enhanced stream (E stream 1) is encoded immediately after the picture “8” of the base stream (B stream). In this example, the enhancement stream (E stream 2) is encoded immediately after the encoding order of the enhancement stream (E stream 1). That is, the picture “11” of the enhanced stream (E stream 2) is encoded immediately after the picture “10” of the enhanced stream (E stream 1).

  FIG. 13 shows an example of hierarchical coding in FIG. 3, in which three video streams of a base stream (B stream), an enhanced stream (E stream 1), and an enhanced stream (E stream 2) are generated. The other example of the encoding timing (decoding timing) is shown. In this example, the decoding delay of the enhanced stream is larger than that of the base stream. The decoding delay in this case is 27 pictures at the encoding interval of full time resolution (1/4 of the encoding interval of the base stream). When the decoding delay increases as described above, a large amount of reference memory is required inside dpb (decoded picture buffer).

  In this example, the encoding timing of the picture of the base stream (B stream) is a multiple of 4, the encoding timing of the enhancement stream (E stream 1) is a multiple of 4, and the base stream (B stream) The timing is intermediate between the picture encoding timings. The encoding timing of the enhancement stream (E stream 1) is an odd timing.

  In this example, the enhancement stream (E stream 1) is encoded after the highest layer encoding of the base stream (B stream) is completed. That is, the picture “14” of the enhancement stream (E stream 1) is encoded immediately after the picture “12” of the base stream (B stream). In this example, the enhanced stream (E stream 2) is encoded after the encoding of the enhanced stream (E stream 1) is completed. That is, the picture “27” of the enhanced stream (E stream 2) is encoded immediately after the picture “26” of the enhanced stream (E stream 1).

  FIG. 14 shows an example of HRD (Hypothetical Reference Decoder) control of the encoder 102. In this example, two video streams of a base stream (B stream) and an enhanced stream (E stream) are generated. Here, the base stream is assumed to be substream 1 (Substream1) and the enhanced stream is assumed to be substream 2 (Substream2).

  A stair-like solid line a1 indicates the transition of the data amount of the substream 1 generated by encoding (encoding), and each stage corresponds to one picture unit. The height of the step indicates the amount of data generated by encoding.

  Timing P01 indicates the timing at which the first byte of the encoded image data of the first picture enters cpb1 (coded picture buffer 1: compressed data buffer). R1 indicates an input bit rate to the cpb1 of the encoded image data of the first picture. Here, when the amount of encoded data input to cpb1 with time T1 is Q1, R1 = Q1 / T1. In the example shown in the figure, the input bit rate to cpb1 of encoded image data of other pictures is also R1.

  A stair-like solid line b1 indicates a transition of the data amount in cpb1 consumed by decoding (decoding), and each stage corresponds to one picture unit. The step height indicates the amount of data consumed by decoding. Qcpb1 indicates the occupation amount of cpd1. The occupation amount is encoded so as to be within the size (memory capacity) of cpb1 at any timing.

  Further, a stair-like solid line a2 indicates the transition of the data amount of the substream 2 generated by encoding (encoding), and each stage corresponds to one picture unit. The height of the step indicates the amount of data generated by encoding.

  Timing P02 indicates the timing at which the first byte of the encoded image data of the first picture enters cpb2 (coded picture buffer 2: compressed data buffer). R2 indicates an input bit rate to the cpb2 of the encoded image data of the first picture. Here, when the amount of encoded data input to cpb2 with time T2 is Q2, R2 = Q2 / T2. In the illustrated example, the case where the input bit rate to the cpb2 of the encoded image data of other pictures is also R2 is shown.

  A stair-like solid line b2 indicates a change in the amount of data in cpb2 consumed by decoding (decoding), and each stage corresponds to one picture unit. The step height indicates the amount of data consumed by decoding. Qcpb2 indicates the occupation amount of cpd2. The occupation amount is encoded so as to be within the size (memory capacity) of cpb2 at any timing.

  In the illustrated example, sub-stream 1 is decoded in the order of pictures “1-0”, “1-1”, “1-2”, “1-3”... .., “0”, “2-1”, “2-2”, “2-3”... In this way, the picture of substream 1 and the picture of substream 2 are decoded alternately. The decoded image data of each picture is input to dpb (decoded picture buffer). In this example, the number of delayed pictures from the decoding to the start of display is 4 pictures.

  In the above description, both R1 and R2 are examples of a constant bit rate (constant_bit_rate). However, the concept is the same even if the variable bit rate (variable_bit_rate) is not limited thereto.

  FIG. 15 shows a configuration example of the encoder 102. The encoder 102 includes a temporal ID generation unit 121, a buffer delay control unit 122, an HRD (Hypothetical Reference Decoder) setting unit 123, a parameter set / SEI encoding unit 124, a slice encoding unit 125, and a NAL packetizing unit 126. have.

  The temporal ID generation unit 121 is supplied with information on the number of layers from the CPU 101. The temporal ID generation unit 121 generates temporal_id corresponding to the number of layers based on the information on the number of layers. For example, in the hierarchical code example of FIG. 3, temporal_id = 0-4 is generated.

  The buffer delay control unit 122 is supplied with information on minimum decoding capability (minimum_target_decoder_level_idc) from the CPU 101 and temporal_id generated by the temporal ID generation unit 121. The buffer delay control unit 122 calculates “initial_cpb_removal_delay”, which is an initial value of cpb buffering (buffering), and “cpb_removal_delay” and “dpb_output_delay” for each picture, for each video stream.

  The buffer delay control unit 122 controls “Cpb_removal_delay” in the cpb buffer for each substream (Sub-stream). The buffer delay control unit 122 controls the dpb buffer so that there is no buffer failure between the decoding timing of the decoder and the display timing. In this case, “cpb_removal_delay” is controlled so that the decoding timings of the pictures in the lowest layer set are equally spaced. In this case, the encoding timing of the encoded image data of the pictures of the hierarchical group positioned higher than the lowest hierarchical group is the same as the encoded image data of the pictures of all the hierarchical groups positioned lower than this hierarchical group. “Cpb_removal_delay” is controlled so as to be an intermediate timing of the encoding timing. In addition, “dpb_output_delay” is controlled so that the cpb buffer is not broken. The encode timing has the same meaning as the decode timing read from the compressed data buffer (cpb: coded picture buffer) on the receiving side.

  An HRD (Hypothetical Reference Decoder) setting unit 123 is supplied with “cpb_removal_delay” and “dpb_output_delay” of the picture of each video stream calculated by the buffer delay control unit 122 and the number of streams (Number of streams) from the CPU 101. Information is supplied. The HRD setting unit 123 performs HRD setting based on these pieces of information.

  The parameter set / SEI encoding unit 124 is supplied with temporal_id together with the HRD setting information. The parameter set / SEI encoding unit 124 generates a parameter set and SEI such as VPS, SPS, and PPS of pictures in each layer according to the number of streams to be encoded.

  For example, a picture timing SEI (Picture timing SEI) including “cpb_removal_delay” and “dpb_output_delay” is generated. Also, for example, a buffering period SEI (Buffere Perifod SEI) including “initial_cpb_removal_time” is generated. The buffering period SEI is generated corresponding to the first picture (access unit) of the GOP.

  “Initial cpb removal time” indicates a time (initial time) to be taken out when decoding the encoded image data of the first picture of the GOP (Group Of Picture) from the compressed data buffer (cpb). “Cpb_removal_delay” is a time for extracting the encoded image data of each picture from the compressed data buffer (cpb), and the time is determined together with “initial_cpb_removal_time”. “Dpb_output_delay” indicates the time taken to decode and enter the uncompressed data buffer (dpb).

  The slice encoding unit 125 encodes image data of pictures in each layer to obtain slice data (slice segment header, slice segment data). The slice encoding unit 125 inserts “ref_idx_l0_active (ref_idx_l1_active)” indicating the index of the prediction destination picture of the “Prediction Unit” into the “slice segment header” as information indicating the prediction state in the temporal direction using the frame buffer. As a result, at the time of decoding, the referenced picture is determined together with the hierarchical level indicated by temporal_id. Also, the slice encoding unit 125 inserts the index of the current slice (slice) into “slice segment header” as “short_term_ref_pic_set_idx” or “it_idx_sps”.

  The NAL packetizing unit 126 generates encoded image data of pictures in each layer based on the parameter set and SEI generated by the parameter set / SEI encoding unit 124 and the slice data generated by the slice encoding unit 125. The number of video streams (encoded streams) corresponding to the number of streams is output.

  In that case, temporal_id which shows the hierarchy is attached | subjected to a NAL unit header for every picture (refer FIG. 4). Also, pictures belonging to the layer indicated by temporal_id are bundled as a sublayer (sub_layer), and the bit rate level designation value “Level_idc” for each sublayer is set to “sublayer_level_idc” and inserted into the VPS or SPS.

  FIG. 16 shows a processing flow of the encoder 102. In step ST1, the encoder 102 starts processing, and then proceeds to processing in step ST2. In step ST2, the encoder 102 sets the number N of layers in the layer encoding. Next, in step ST3, the encoder 102 sets temporal_id of pictures in each layer to 0 to (N-1).

  Next, in step ST4, the encoder 102 sets the hierarchy level K that can be decoded by the decoder having the minimum capability among the target decoders within a range of 0 to N-1. In step ST5, the encoder 102 uses the buffer delay control unit 122 to set the picture encoding interval and the encoding timing in each layer set.

  Next, in step ST6, the encoder 102 reflects the picture encoding interval and encoding timing of each layer set obtained in step ST5 in “cpb_removal_delay” and “dpb_output_delay”, and performs HRD setting, parameter set / SEI encoding, and slice encoding. And transfer to the multiplexed block as a NAL unit. Then, the encoder 102 complete | finishes a process in step ST7.

  Returning to FIG. 2, the compressed data buffer (cpb) 103 temporarily stores the video stream generated by the encoder 102 and including the encoded data of the pictures of each layer. The multiplexer 104 reads the video stream stored in the compressed data buffer 103, converts it into a PES packet, further converts it into a transport packet, multiplexes it, and obtains a transport stream TS as a multiplexed stream.

  As described above, the transport stream TS includes a predetermined number of video streams having encoded image data of pictures in each layer set obtained by dividing a plurality of layers. The multiplexer 104 inserts identification information for identifying whether each of the predetermined number of video streams is a base stream or an enhanced stream into the transport stream TS. In this case, the identification information is inserted as a stream type in a video elementary stream loop (Video ES loop) arranged corresponding to a predetermined number of video streams under the program map table.

  In this case, the stream type of the base stream is “0x24”. The stream type of the enhanced stream is newly defined, for example, “0x25”. When there are multiple enhanced streams, the stream types of all enhanced streams are not the same, but multiple stream types are newly defined as stream types of enhanced streams so that each enhanced stream can be identified. Also good. For example, when there are two enhanced streams, the stream type of the first enhanced stream is “0x25”, and the stream type of the second enhanced stream is “0x26”.

  The multiplexer 104 inserts video stream configuration information corresponding to each of a predetermined number of video streams in the layer of the transport stream TS. The multiplexer 104 inserts this configuration information as a descriptor in a video elementary stream loop arranged corresponding to a predetermined number of video streams under the program map table.

  The multiplexer 104 inserts a newly defined multi-stream descriptor (multistream_descriptor) together with the HEVC descriptor (HEVC_descriptor). FIG. 17 shows a structural example (Syntax) of the HEVC descriptor (HEVC_descriptor). The 8-bit field of “descriptor_tag” indicates a descriptor type, and here indicates that it is a HEVC descriptor. The 8-bit field of “descriptor_length” indicates the length (size) of the descriptor, and indicates the number of subsequent bytes as the length of the descriptor.

  The 8-bit field of “level_idc” indicates a bit rate level designation value. When “temporal_layer_subset_flag = 1”, a 5-bit field of “temporal_id_min” and a 5-bit field of “temporal_id_max” exist. “Temporal_id_min” indicates the value of temporal_id of the lowest hierarchy of the hierarchically encoded data included in the corresponding video stream. “Temporal_id_max” indicates the value of temporal_id of the highest hierarchy of the hierarchically encoded data included in the corresponding video stream.

  FIG. 18 illustrates a structural example (Syntax) of a multistream descriptor (multistream_descriptor). FIG. 19 shows the contents (Semantics) of main information in the structural example.

  An 8-bit field of “multistream_descriptor_tag” indicates a descriptor type. Here, it indicates a multi-stream descriptor. The 8-bit field of “multistream_descriptor_length” indicates the length (size) of the descriptor, and indicates the number of subsequent bytes as the descriptor length. Here, 2 bytes are shown. A 4-bit field of “group_id” indicates an ID of a group associated with a series of services. In this case, the base stream and all non-base streams (non-base stream = enhanced stream) based on the base stream have the same ID.

  The 4-bit field of “stream_dependency_ordering” defines the dependency relationship between streams starting from the base stream in ascending order. “0001” indicates a basic stream. “0010” indicates the second stream (enhanced stream) from the basic stream. “0011” indicates the third stream from the basic stream. “Max_layer_in_group” indicates the maximum value of the layers encoded in the group.

  FIG. 20 shows “Stream_type”, “Group_id”, “max / min layer” when the transport stream TS includes, for example, video streams of service 1 (SERVICE 1) and service 2 (SERVICE 2). ], “Max_layer_in_group”, “Stream_dependenncy ordering”.

In this example, as a video stream of the service 1, a base stream (Base stream), an enhanced stream (Enhanced stream 1), an enhanced stream (Enhanced stream 2)
) Three video streams. In this service 1, the value of “Group_id” is “0”. Further, in the service 1, for example, the number of layers is five and the value of “max / min layer” is “4”, similarly to the example of the layer encoding shown in FIG.

  The service 1 is divided into three hierarchical groups. The value of “Stream_type” of the base stream is set to “0x24”, and “max / min layer” of the HEVC descriptor indicates that the pictures of layers 0 to 2 are included. Further, the value of “Stream_type” of the enhanced stream (Enhanced stream 1) is set to “0x25”, which indicates that “max / min layer” of the HEVC descriptor includes a picture of layer 3. Further, the value of “Stream_type” of the enhanced stream (Enhanced stream 2) is set to “0x25”, which indicates that “max / min layer” of the HEVC descriptor includes a picture of layer 4.

Also, in this example, as a video stream of service 2, a base stream (Base stream), an enhanced stream (Enhanced stream 1), and an enhanced stream (Enhanced stream 2)
) Three video streams. In this service 2, the value of “Group_id” is “1”. Further, in the service 2, for example, the number of hierarchies is four, and the value of “max / min layer” is “3”.

  The service 2 is divided into three hierarchical groups. The value of “Stream_type” of the base stream is set to “0x24”, and “max / min layer” of the HEVC descriptor indicates that a picture of layers 0 to 1 is included. In addition, the value of “Stream_type” of the enhanced stream (Enhanced stream 1) is set to “0x25”, and “max / min layer” of the HEVC descriptor indicates that a picture of layer 2 is included. Furthermore, the value of “Stream_type” of the enhanced stream (Enhanced stream 2) is set to “0x25”, which indicates that “max / min layer” of the HEVC descriptor includes a picture of layer 3.

  FIG. 21 shows a configuration example of the multiplexer 104. The multiplexer 104 includes a section coding unit 142, PES packetization units 143-1 to 143-N, a switch unit 144, and a transport packetization unit 145.

  The PES packetization units 143-1 to 143-N read the video streams 1 to N stored in the compressed data buffer 103, respectively, and generate PES packets. Here, the video streams 1 to N include one base stream and one or more enhancement streams.

  At this time, the PES packetizing units 143-1 to 143-N add DTS (Decoding Time Stamp) and PTS (Presentation Time Stamp) time stamps to the PES header based on the HRD information of the video streams 1 to N. In this case, “cpu_removal_delay” and “dpb_output_delay” of each picture are referred to, DTS and PTS are generated with accuracy synchronized with STC (System Time Clock) time, and are arranged at predetermined positions of the PES header.

  The switch unit 144 selectively extracts the PES packets generated by the PES packetization units 143-1 to 143-N based on the packet identifier (PID), and sends the PES packets to the transport packetization unit 145. The transport packetization unit 145 generates a TS packet including a PES packet in the payload, and obtains a transport stream TS.

  The section coding unit 142 generates various section data to be inserted into the transport stream TS. The section coding unit 142 is supplied with information such as the number of layers (Number of layers) and the number of streams (Number of streams) from the CPU 101. The section coding unit 142 generates the above-described HEVC descriptor (HEVC_descriptor) and multi-stream descriptor (multistream_descriptor) based on these pieces of information.

  The section coding unit 142 sends various section data to the transport packetizing unit 145. The transport packetization unit 145 generates a TS packet including this section data and inserts it in the transport stream TS. At this time, a stream type is also inserted into a video elementary stream loop (Video ES loop) arranged corresponding to each video stream. In this case, the stream type of the base stream is “0x24”, and the stream type of the enhanced stream is, for example, “0x25” that is newly defined.

  FIG. 22 shows a processing flow of the multiplexer 104. In step ST11, the multiplexer 104 starts processing, and then proceeds to processing in step ST12. In step ST12, the multiplexer 104 refers to the HRD information (cpu_removal_delay, dpb_output_delay), determines DTS and PTS, and inserts them at predetermined positions in the PES header.

  Next, in step ST13, the multiplexer 104 determines whether or not there is a multi-stream, that is, whether or not N is plural. In the case of multi-stream, the multiplexer 104 proceeds the multiplexing process with a plurality of PIDs in step ST14. In step ST15, the multiplexer 104 determines whether or not the stream is a base stream.

  When it is a base stream, the multiplexer 104 sets the stream type to “0x24” in step ST16, and then proceeds to the process of step ST18. On the other hand, when the stream is an enhanced stream, the multiplexer 104 sets the stream type to a value indicating the enhanced stream, for example, “0x25” that is newly defined in step ST17, and then proceeds to the process of step ST18.

  If the multi-stream is not determined in step ST13, the multiplexer 104 performs the multiplexing process with one PID in step ST19, and then proceeds to the process of step ST18.

  In step ST18, the multiplexer 104 section-codes HEVC descriptors, multi-stream descriptors, etc., and inserts an encoded stream (video elementary stream) into the PES payload to form a PES packet. Then, the multiplexer 104 converts it into a transport packet in step ST20, and obtains a transport stream TS. Thereafter, the multiplexer 104 ends the process in step ST21.

  FIG. 23 illustrates a configuration example of the transport stream TS when a certain service is distributed in two streams. The transport stream TS includes two video streams, a base stream and an enhanced stream. That is, in this configuration example, there is a base stream PES packet “video PES1” and an enhanced stream PES packet “video PES2”.

  The transport stream TS includes a PMT (Program Map Table) as one of PSI (Program Specific Information). This PSI is information describing to which program each elementary stream included in the transport stream belongs.

  In the PMT, there is a program loop that describes information related to the entire program. The PMT has an elementary stream loop having information related to each video stream. In this configuration example, there is a video elementary stream loop “video ES1 loop” corresponding to the base stream, and a video elementary stream loop “video ES2 loop” corresponding to the enhanced stream.

  In the “video ES1 loop”, information such as a stream type and a packet identifier (PID) is arranged corresponding to the base stream (video PES1), and a descriptor describing information related to the video stream is also arranged. The This stream type is “0x24” indicating the base stream. Further, the HEVC descriptor and the multi-stream descriptor described above are inserted as one of the descriptors.

  In addition, in the “video ES2 loop”, information such as a stream type and a packet identifier (PID) is arranged corresponding to the enhanced stream (video PES2), and a descriptor describing information related to the video stream is also provided. Be placed. This stream type indicates an enhanced stream, for example, “0x25” that is newly defined. Further, the HEVC descriptor and the multi-stream descriptor described above are inserted as one of the descriptors.

  FIG. 24 illustrates a configuration example of the transport stream TS when a certain service is distributed in three streams. This transport stream TS includes three video streams of a base stream and two enhanced streams. That is, in this configuration example, the base stream PES packet “video PES1” exists, and the enhanced stream PES packets “video PES2” and “video PES3” exist.

  The PMT has an elementary stream loop having information related to each video stream. In this configuration example, there is a video elementary stream loop “video ES1 loop” corresponding to the base stream, and a video elementary stream loop “video ES2 loop” and a video elementary stream loop “video” corresponding to two enhanced streams. ES3 loop "exists.

  In the “video ES1 loop”, information such as a stream type and a packet identifier (PID) is arranged corresponding to the base stream (video PES1), and a descriptor describing information related to the video stream is also arranged. The This stream type is “0x24” indicating the base stream. Further, the HEVC descriptor and the multi-stream descriptor described above are inserted as one of the descriptors.

  In addition, in the “video ES2 loop”, information such as a stream type and a packet identifier (PID) is arranged corresponding to the enhanced stream (video PES2), and a descriptor describing information related to the video stream is also provided. Be placed. This stream type indicates an enhanced stream, for example, “0x25” that is newly defined. Further, the HEVC descriptor and the multi-stream descriptor described above are inserted as one of the descriptors.

  In addition, in the “video ES3 loop”, information such as a stream type and a packet identifier (PID) is arranged corresponding to the enhanced stream (video PES3), and a descriptor describing information related to the video stream is also provided. Be placed. This stream type indicates an enhanced stream, for example, newly defined “0x25” or “0x26”. Further, the HEVC descriptor and the multi-stream descriptor described above are inserted as one of the descriptors.

  Returning to FIG. 2, the transmission unit 105 modulates the transport stream TS with a modulation scheme suitable for broadcasting such as QPSK / OFDM, and transmits an RF modulation signal from the transmission antenna.

  The operation of the transmission apparatus 100 shown in FIG. 2 will be briefly described. Uncompressed moving image data is input to the encoder 102. The encoder 102 performs hierarchical encoding on the moving image data. That is, in the encoder 102, the image data of each picture constituting the moving image data is classified into a plurality of layers and encoded, and a video stream having the encoded image data of the pictures of each layer is generated. At this time, encoding is performed so that the picture to be referenced belongs to the self hierarchy and / or a hierarchy lower than the self hierarchy.

  In the encoder 102, a plurality of hierarchies are divided into a predetermined number of hierarchies, and a predetermined number of video streams each having encoded image data of pictures in each hierarchic group are generated. In this case, a base stream having encoded image data of pictures of the lowest hierarchical group is generated, and a predetermined number of encoded image data of pictures of hierarchical groups positioned higher than the lowest hierarchical group are included. An enhanced stream is generated.

  The predetermined number of video streams generated by the encoder 102 is supplied to the compressed data buffer (cpb) 103 and temporarily accumulated. In the multiplexer 104, each video stream stored in the compressed data buffer 103 is read out, converted into a PES packet, further converted into a transport packet, and multiplexed to obtain a transport stream TS as a multiplexed stream.

  Also, in the multiplexer 104, identification information for identifying whether each of the predetermined number of video streams is a base stream or an enhanced stream is inserted into the layer of the transport stream TS. This identification information is, for example, a stream type inserted into a video elementary stream loop (Video ES loop) arranged corresponding to each video stream. In this case, the stream type of the base stream is “0x24”, and the stream type of the enhanced stream is, for example, “0x25” that is newly defined.

  Further, in the multiplexer 104, video stream configuration information is inserted into the layer of the transport stream TS corresponding to each of a predetermined number of video streams. That is, in the multiplexer 104, the HEVC descriptor and the multi-stream descriptor are inserted into the video elementary stream loop corresponding to each video stream.

  The transport stream TS generated by the multiplexer 104 is sent to the transmission unit 105. In the transmission unit 105, the transport stream TS is modulated by a modulation method suitable for broadcasting such as QPSK / OFDM, and an RF modulation signal is transmitted from the transmission antenna.

"Receiver configuration"
FIG. 25 illustrates a configuration example of the receiving device 200. The receiving apparatus 200 includes a CPU (Central Processing Unit) 201, a receiving unit 202, a demultiplexer 203, and a compressed data buffer (cpb: coded picture buffer) 204. The receiving apparatus 200 includes a decoder 205, an uncompressed data buffer (dpb: decoded picture buffer) 206, and a post processing unit 207. The CPU 201 constitutes a control unit and controls the operation of each unit of the receiving device 200.

  The receiving unit 202 demodulates the RF modulation signal received by the receiving antenna, and acquires the transport stream TS. The demultiplexer 203 selectively extracts, from the transport stream TS, encoded image data of a layered set of pictures corresponding to a decoding capability (Decoder temporal layer capability), and sends it to a compressed data buffer (cpb: coded picture buffer) 204. .

  FIG. 26 shows a configuration example of the demultiplexer 203. The demultiplexer 203 includes a TS adaptation field extraction unit 231, a clock information extraction unit 232, a TS payload extraction unit 233, a section extraction unit 234, a PSI table / descriptor extraction unit 235, and a PES packet extraction unit 236. ing. The demultiplexer 203 includes a PES header extraction unit 237, a time stamp extraction unit 238, a PES payload extraction unit 240, and a stream configuration unit (stream composer) 241.

  The TS adaptation field extraction unit 231 extracts the adaptation field from the TS packet having the adaptation field of the transport stream TS. The clock information extraction unit 232 extracts the PCR from an adaptation field including a PCR (Program Clock Reference), and sends the PCR to the CPU 201.

  The TS payload extraction unit 233 extracts the TS payload from the TS packet having the TS payload of the transport stream TS. The section extraction unit 234 extracts the section data from the TS payload including the section data. The PSI table / descriptor extraction unit 235 analyzes the section data extracted by the section extraction unit 234 and extracts a PSI table and a descriptor. Then, the PSI table / descriptor extraction unit 235 transmits the minimum value (min) and maximum value (max) of temporal_id, the maximum number of hierarchies, the stream dependency relationship, the group ID, and the like to the CPU 201 and to the stream configuration unit 241.

  The PES packet extraction unit 236 extracts the PES packet from the TS payload including the PES packet. The PES header extraction unit 237 extracts a PES header from the PES packet extracted by the PES packet extraction unit 236. The time stamp extraction unit 238 extracts the time stamp (DTS, PTS) inserted in the PES header for each picture, and sends it to the CPU 201 and also sends it to the stream configuration unit 241.

  The PES payload extraction unit 240 extracts the PES payload, that is, the encoded image data of the pictures of each layer, from the PES packet extracted by the PES packet extraction unit 236. From the encoded image data of each layer of pictures extracted by the PES payload extraction unit 240, the stream configuration unit 241 includes only the base stream or the base stream and a predetermined number of enhancement streams according to the decoding capability (Decoder temporal layer capability). The encoded image data of a picture is selectively extracted and sent to a compressed data buffer (cpb: coded picture buffer) 204. In this case, the stream configuration unit 241 refers to the hierarchy information obtained by the PSI table / descriptor extraction unit 235 and the like.

  For example, consider a case where the frame rate of a predetermined number of video streams (encoded streams) included in the transport stream TS is 120 fps. For example, it is assumed that a plurality of hierarchies are divided into a hierarchy set on the lower hierarchy side and a hierarchy set on the higher hierarchy side, and the frame rate of pictures in each hierarchy set is 60 fps. For example, in the above-described hierarchical coding example shown in FIG. 3, layers 0 to 3 are set to a lower layer set, and a decoder corresponding to level_idc of 60 fps can be decoded. Further, the layer 4 is a layer set on the higher layer side, and a 120 fps level_idc compatible decoder can be decoded.

  When the decoding capability corresponds to 120 fps, the stream configuration unit 241 extracts the encoded image data of both the base stream and the enhanced stream based on the packet ID (PID), and compresses the compressed data buffer (cpb). 204. On the other hand, when the decoding capability does not correspond to 120 fps but corresponds to 60 fps, the stream configuration unit 241 extracts only the encoded image data of the picture of the base stream based on the packet ID (PID) and compresses it. The data is sent to the data buffer (cpb) 204.

  FIG. 27 illustrates an example of picture (slice) selection by the stream configuration unit 241 when the transport stream TS includes two video streams (encoded streams) of a base stream and an enhanced stream. Assume that the packet identifier (PID) of the base stream is PID A, and the packet identifier (PID) of the enhanced stream is PID B. The example shown corresponds to the example shown in FIG. 9 described above, and shows only the nth sub-picture group (Sub group of pictures), and is attached to each picture indicated by a rectangular frame. These numbers indicate the decoding order (encoding order on the transmission side).

  When the decoding capability corresponds to 120 fps, the stream configuration unit 241 performs filtering based on the packet identifier (PID), and encodes image data of both pictures of the base stream that is PIDA and the enhanced stream that is PIDB. Is sent to the compressed data buffer (cpb) 204. In this case, the encoded image data of the base stream picture is accumulated in the region 1 (cpb_1), and the encoded image data of the enhanced stream picture is accumulated in the region 2 (cpb_2).

  On the other hand, when the decoding capability does not correspond to 120 fps but corresponds to 60 fps, the stream configuration unit 241 performs filtering based on the packet identifier (PID), and encodes an encoded image of a base stream picture that is PIDA Only the data is extracted, sent to the compressed data buffer (cpb) 204, and stored in the area 1 (cpb_1).

  FIG. 28 shows an example of the processing flow of the demultiplexer 203. This processing flow includes encoding a base stream having encoded image data of a picture in the lowest hierarchy set in a transport stream TS and a predetermined number of pictures in the hierarchy set located above the lowest hierarchy set. A case where a predetermined number of enhancement streams each having image data is included is shown.

  In step ST41, the demultiplexer 203 starts processing, and then proceeds to processing in step ST42. In step ST42, the CPU 201 sets a decoding capability (Decoder temporal layer capability). Next, in step ST43, the demultiplexer 203 determines whether or not it has the ability to decode all layers (layers).

  When there is an ability to decode the entire hierarchy, the demultiplexer 203 selects all the streams constituting the entire hierarchy by the PID filter in step ST44, and performs section parsing. Thereafter, the demultiplexer 203 proceeds to the process of step ST45.

  When there is no capability to decode the entire hierarchy in step ST43, the demultiplexer 203 selects a predetermined number of streams including the base stream that constitutes a decodable low hierarchy in step ST46, and also related section parsing (Section parsing). Thereafter, the demultiplexer 203 proceeds to the process of step ST45.

  In step ST45, the demultiplexer 203 reads the HEVC descriptor and the multistream descriptor in the section of the target PID, and obtains the dependency of the stream in the group, the maximum number of hierarchies, and the maximum and minimum values of temporal_id.

  Next, in step ST47, the demultiplexer 203 transfers the encoded image data of the picture of the stream selected in step ST44 or step ST46 to the compressed data buffer (cpb) 204. The demultiplexer 203 ends the process in step ST48 after the process of step ST47.

  Returning to FIG. 25, the compressed data buffer (cpb) 204 temporarily accumulates the video stream (encoded stream) extracted by the demultiplexer 203. The decoder 205 extracts from the video stream stored in the compressed data buffer 204 encoded image data of a picture of a hierarchy designated as a hierarchy to be decoded. Then, the decoder 205 decodes the encoded image data of each extracted picture at the decoding timing of the picture, and sends the decoded picture data to the uncompressed data buffer (dpb) 206.

  Here, in the decoder 205, the hierarchy to be decoded from the CPU 201 is specified by temporal_id. This designated layer is the entire layer included in the video stream (encoded stream) extracted by the demultiplexer 203 or a part of the lower layer, and is set automatically by the CPU 201 or in response to a user operation. The The decoder 205 is given a decoding timing from the CPU 201 based on a DTS (Decoding Time stamp). Note that the decoder 205 reads the image data of the referenced picture from the uncompressed data buffer 206 and uses it as necessary when decoding the encoded image data of each picture.

  FIG. 29 shows a configuration example of the decoder 205. The decoder 205 includes a temporal ID analysis unit 251, a target hierarchy selection unit 252, a stream combination unit 253, and a decoding unit 254. The temporal ID analysis unit 251 reads the video stream (encoded stream) stored in the compressed data buffer 204 and analyzes temporal_id inserted in the NAL unit header of the encoded image data of each picture.

  The target layer selection unit 252 extracts the encoded image data of the picture of the layer specified as the layer to be decoded from each video stream read from the compressed data buffer 204 based on the analysis result of the temporal ID analysis unit 251. Take out. In this case, the target layer selection unit 252 outputs one or a plurality of video streams (encoded streams) according to the number of video streams read from the compressed data buffer 204 and the specified layer.

  The stream combining unit 253 combines each video stream (encoded stream) output from the target layer selecting unit 252 into one. Although not shown in the figure, the stream combining unit 253 may combine each video stream (encoded stream) output from the cpb buffer 204 into one. In this case, the stream combination unit 253 performs target hierarchy selection together with temporal ID analysis. The stream combining unit 253 makes the encoded image data of each picture into one stream based on the decoding timing information. FIG. 30 shows an example of stream combination.

  This example corresponds to the example shown in FIG. 9 described above, and is an example in which a picture of a base stream at 60 Hz intervals and a picture of an enhanced stream at 60 Hz intervals are combined. In this case, each picture is a stream with a time stamp of 120 Hz.

  This one stream is sent to the decoding unit 254, and the encoded image data of each picture is decoded at a decoding timing and stored in a dpb (uncompressed data buffer) 206, as will be described later. Thereafter, the image data of each picture is reordered from the uncompressed data buffer 206 and sequentially read out at 120 Hz. In the example shown in the figure, first, a picture of a certain sub-picture group (indicated by right-down hatching) is read out, and then a picture of the next sub-picture group (indicated by left-downward hatching) is read. . After decoding, while a picture of a certain sub-picture group is displayed, a picture of the next sub-picture group is stored in the uncompressed data buffer 206 and becomes a subsequent reference picture.

  Note that the picture summarization process itself of the plurality of streams is performed on the plurality of selected streams in the above-described demultiplexer 203 and transferred to the compressed data buffer (cpb) 204 as one stream. Good. Similarly, the combining process is also performed based on the decode timing information. In this case, the combining process in the decoder is not necessary.

  The decoding unit 254 sequentially decodes the encoded image data of each picture included in the video stream (encoded stream) combined by the stream combining unit 253 at a decoding timing, and sends the decoded image data to the uncompressed data buffer (dpb) 206.

  In this case, the decoding unit 254 analyzes the VPS and SPS, for example, grasps the bit rate level designation value “sublayer_level_idc” for each sublayer, and confirms whether it can be decoded within the decoding capability. In this case, the decoding unit 254 analyzes the SEI, grasps, for example, “initial_cpb_removal_time” and “cpb_removal_delay”, and confirms whether the decoding timing from the CPU 201 is appropriate.

  When decoding the slice (Slice), the decoding unit 254 acquires “ref_idx_l0_active (ref_idx_l1_active) as information indicating a prediction destination in the time direction from the slice header (Slice header), and performs prediction in the time direction. The decoded picture is processed as a reference by another picture with “short_term_ref_pic_set_idx” or “it_idx_sps” obtained from the slice header as an index.

  Returning to FIG. 25, the uncompressed data buffer (dpb) 206 temporarily stores the image data of each picture decoded by the decoder 205. The post processing unit 207 performs processing for matching the frame rate with the display capability for the image data of each picture sequentially read from the uncompressed data buffer (dpb) 206 at the display timing. In this case, display timing is given from the CPU 201 based on a PTS (Presentation Time stamp).

  For example, when the frame rate of the image data of each picture after decoding is 120 fps and the display capability is 120 fps, the post processing unit 207 sends the image data of each picture after decoding to the display as it is. For example, when the frame rate of the image data of each picture after decoding is 120 fps and the display capability is 60 fps, the post processing unit 207 has a temporal resolution for the image data of each picture after decoding. Sub-sample processing is performed so as to be ½ times, and the image data is sent to the display as 60 fps image data.

  For example, when the frame rate of the image data of each picture after decoding is 60 fps and the display capability is 120 fps, the post processing unit 207 has a temporal resolution for the image data of each picture after decoding. Interpolation processing is performed so as to be doubled, and the image data is sent to the display as 120 fps image data. For example, when the frame rate of the image data of each picture after decoding is 60 fps and the display capability is 60 fps, the post processing unit 207 sends the image data of each decoded picture to the display as it is.

  FIG. 31 shows a configuration example of the post processing unit 207. In this example, as described above, it is possible to cope with the case where the frame rate of the image data of each picture after decoding is 120 fps or 60 fps and the display capability is 120 fps or 60 fps.

  The post processing unit 207 includes an interpolation unit 271, a subsample unit 272, and a switch unit 273. The decoded image data of each picture from the uncompressed data buffer 206 is directly input to the switch unit 273, or is input to the switch unit 273 after the frame rate is doubled by the interpolation unit 271, or the subsample unit After the frame rate is set to 1/2 the frame rate at 272, the frame rate is input to the switch unit 273.

  Selection information is supplied from the CPU 201 to the switch unit 273. The selection information is generated automatically by the CPU 201 with reference to the display capability or in response to a user operation. The switch unit 273 selectively outputs any of the inputs based on the selection information. As a result, the frame rate of the image data of each picture sequentially read from the uncompressed data buffer (dpb) 206 at the display timing is adapted to the display capability.

  FIG. 32 shows an example of the processing flow of the decoder 205 and post processing unit 207. In step ST51, the decoder 205 and the post processing unit 207 start the processing, and then move to the processing in step ST52. In step ST52, the decoder 205 reads out the decoding target video stream stored in the compressed data buffer (cpb) 204, and selects a picture in a hierarchy designated as a decoding target from the CPU 201 based on temporal_id.

  Next, in step ST53, the decoder 205 sequentially decodes the encoded image data of each selected picture at the decode timing, and transfers the decoded image data of each picture to the uncompressed data buffer (dpb) 206. , Temporarily accumulate. Next, in step ST54, the post processing unit 207 reads the image data of each picture from the uncompressed data buffer (dpb) 206 at the display timing.

  Next, the post processing unit 207 determines whether or not the frame rate of the read image data of each picture matches the display capability. When the frame rate does not match the display capability, the post processing unit 207 sends the frame rate to the display in accordance with the display capability in step ST56, and then ends the processing in step ST57. On the other hand, when the frame rate matches the display capability, the post processing unit 207 sends the frame rate as it is to the display in step ST58, and then ends the process in step ST57.

  The operation of receiving apparatus 200 shown in FIG. 25 will be briefly described. In the reception unit 202, the RF modulation signal received by the reception antenna is demodulated, and the transport stream TS is acquired. This transport stream TS is sent to the demultiplexer 203. In the demultiplexer 203, all or part of the video stream is PID-filtered from the transport stream TS according to the decoding capability (Decoder temporal layer capability).

  For example, when the decoding capability is high, all video streams of the base stream and the enhanced stream are selected. For example, when the decoding capability is low, a predetermined number of video streams including a base stream including a hierarchy capable of decoding are selected. Then, from the demultiplexer 203, the encoded image data of the picture of the selected video stream is sent to the compressed data buffer (cpb) 204 and temporarily stored.

  In the decoder 205, encoded image data of a picture of a hierarchy designated as a hierarchy to be decoded is extracted from the video stream stored in the compressed data buffer 204. Then, the decoder 205 decodes the extracted encoded image data of each picture at the decoding timing of the picture, sends it to the uncompressed data buffer (dpb) 206, and temporarily accumulates it. In this case, when the encoded image data of each picture is decoded, the image data of the referenced picture is read from the uncompressed data buffer 206 and used as necessary.

  The image data of each picture sequentially read from the uncompressed data buffer (dpb) 206 at the display timing is sent to the post processing unit 207. In the post processing unit 207, interpolation or sub-sampling is performed on the image data of each picture so that the frame rate matches the display capability. The image data of each picture processed by the post processing unit 207 is supplied to a display, and a moving image is displayed using the image data of each picture.

  As described above, in the transmission / reception system 10 shown in FIG. 1, encoding is performed so that at least the decoding intervals of the encoded image data of the pictures in the lowest hierarchy set are equal intervals on the transmission side. is there. Therefore, for example, when the receiving side has a decoding capability capable of processing the encoded image data of the pictures of a plurality of hierarchies included in the lowest hierarchy set, the decoding process of the encoded image data of each picture is continued without difficulty. Can be performed.

  Further, in the transmission / reception system 10 shown in FIG. 1, on the transmission side, the decoding timings of the encoded image data of the pictures in the hierarchical group positioned higher than the lowest hierarchical group are all positioned lower than the hierarchical group. Are encoded so as to be in the middle of the decoding timing of the encoded image data of the pictures in the layer set. Therefore, for example, when the receiving side has the ability to decode not only the lowest layer set but also the encoded image data of a picture of a layer set higher than that, the decoding process of each picture is sequentially performed. It becomes possible to proceed smoothly.

  Further, in the transmission / reception system 10 shown in FIG. 1, when dividing a plurality of hierarchies into a predetermined number of hierarchies on the transmission side, the lowest hierarchy includes a plurality of hierarchies and is higher than this lowest hierarchy The hierarchy set located at is to include one hierarchy. Therefore, for example, when the receiving side has a decoding capability capable of processing the encoded image data of a plurality of layers of pictures included in the lowest layer set, the encoded image data of the pictures of the lowest layer group is used. A configuration in which only a video stream possessed is selected and taken into a buffer for decoding processing is sufficient, and a complicated configuration such as processing for combining a plurality of video streams is not required.

  In the transmission / reception system 10 shown in FIG. 1, identification information for identifying whether each of a predetermined number of video streams is a base stream or an enhanced stream in the transport stream TS layer on the transmission side. Is inserted. Therefore, on the reception side, for example, it is possible to easily select only the base stream and selectively decode the encoded image data of a low-layer set picture by using this identification information.

  Also, in the transmission / reception system 10 shown in FIG. 1, on the transmission side, the video stream configuration information corresponding to each of a predetermined number of video streams included in the transport stream TS, in the transport stream TS layer. Is inserted. For this reason, for example, on the receiving side, for each video stream included in the transport stream TS, which group it belongs to, what stream dependency it has, and how many layer encodings it relates to , Etc. can be easily grasped.

  Further, in the transmission / reception system 10 shown in FIG. 1, on the receiving side, encoded image data of a picture in a layer below a predetermined layer corresponding to a decoding capability (Decoder temporal layer capability) is selectively compressed from a received video stream. The data is taken into the data buffer 204 and decoded. Therefore, for example, an appropriate decoding process according to the decoding capability can be performed.

  In the transmission / reception system 10 shown in FIG. 1, the frame rate of the image data of each picture after decoding is matched with the display capability by the post processing unit 207 on the receiving side. Therefore, for example, even when the decoding capability is low, it is possible to obtain image data with a frame rate suitable for high display capability.

<2. Modification>
In the above-described embodiment, “Max_layer_in_group”, which is information on the maximum layer value, is described in the multi-stream descriptor (see FIG. 18) and is sent to the receiving side. However, instead of describing the maximum layer information in the descriptor and supplying it to the receiving side, the notation of “specify the maximum layer” is used in the text of the file or paper standard document, and the layer maximum is stored in advance in the receiving device. It is also possible that a value is specified or set. In this case as well, the receiving side filters the stream including the layer corresponding to its own decoding capability by referring to the information on the maximum layer value as in the case where the descriptor is supplied from the transmitting side as described above. Thus, the decoding process is performed.

  Moreover, in the above-mentioned embodiment, the example of temporal scalability (temporal scalability) is shown and the configuration information of each stream is transmitted to the receiving side using a multi-stream descriptor. However, although the detailed description is omitted, with regard to the scalability such as spatial scalability or bit rate scalability, in the service in which the basic stream (base stream) and the extension stream (enhanced stream) coexist, the multi-stream descriptor described above is used. Can be applied. That is, the above-described multi-stream descriptor is a useful signaling method when performing multi-stream encoding.

  In the above-described embodiment, the transmission / reception system 10 including the transmission device 100 and the reception device 200 has been described. However, the configuration of the transmission / reception system to which the present technology can be applied is not limited thereto. For example, the receiving device 200 may be a configuration of a set-top box and a monitor connected via a digital interface such as (High-Definition Multimedia Interface (HDMI)). Trademark.

  Further, in the above-described embodiment, an example in which the container is a transport stream (MPEG-2 TS) is shown. However, the present technology can be similarly applied to a system configured to be distributed to receiving terminals using a network such as the Internet. In the Internet distribution, it is often distributed in a container of MP4 or other formats. In other words, containers of various formats such as transport stream (MPEG-2 TS) adopted in the digital broadcasting standard and MP4 used in Internet distribution correspond to the container.

Moreover, this technique can also take the following structures.
(1) The image data of each picture constituting the moving image data is classified into a plurality of hierarchies, the image data of the classified pictures of each hierarchy is encoded, and the plurality of hierarchies are grouped into a predetermined number of hierarchies. An image encoding unit that divides and generates the predetermined number of video streams each having the encoded image data of each of the divided groups of pictures,
The image encoding unit is
An encoding apparatus that performs encoding so that at least the decoding intervals of encoded image data of pictures in the lowest layer set are equal.
(2) The image encoding unit
The decoding timing of the encoded image data of the pictures of the hierarchical group positioned higher than the lowest hierarchical group is intermediate between the decoding timings of the encoded image data of the pictures of all the hierarchical groups positioned lower than the hierarchical group. The encoding device according to (1), wherein encoding is performed so that timing is reached.
(3) The image encoding unit
Dividing the plurality of hierarchies into a predetermined number of hierarchies so that the lowest hierarchic group includes a plurality of hierarchies and the hierarchic group positioned higher than the lowest hierarchic hierarchies includes one hierarchy; The encoding device according to (1) or (2).
(4) The image encoding unit classifies the image data of each picture constituting the moving image data into a plurality of layers, encodes the image data of the classified pictures in each layer, and Dividing into a predetermined number of layer sets, and generating the predetermined number of video streams respectively having encoded image data of pictures of each divided layer set,
The image encoding unit is
An encoding method that performs encoding so that at least the decoding intervals of encoded image data of pictures in the lowest layer set are equal.
(5) The image data of each picture constituting the moving image data is classified into a plurality of hierarchies, the image data of the classified pictures of each hierarchy is encoded, and the plurality of hierarchies are grouped into a predetermined number of hierarchies. An image encoding unit that divides and generates the predetermined number of video streams respectively having encoded image data of pictures of each divided hierarchical group;
A transmission unit configured to transmit a container of a predetermined format including the generated predetermined number of video streams;
The image encoding unit is
A transmission device that performs encoding so that at least the decoding intervals of encoded image data of pictures in the lowest layer set are equal.
(6) The image data of each picture constituting the moving image data is classified into a plurality of hierarchies, the image data of the classified pictures of each hierarchy is encoded, and the plurality of hierarchies are grouped into a predetermined number of hierarchies. An image encoding unit that divides and generates the predetermined number of video streams respectively having encoded image data of pictures of each divided hierarchical group;
A transmission unit for transmitting a container of a predetermined format including the generated predetermined number of video streams;
In the container layer, each of the predetermined number of video streams is a base stream having encoded image data of pictures of the lowest hierarchy set, or a hierarchy set positioned higher than the lowest hierarchy set A transmission apparatus comprising: an identification information insertion unit that inserts identification information for identifying whether or not an enhanced stream includes encoded image data of a picture.
(7) The container is a transport stream,
The identification information insertion unit
The transmission device according to (6), wherein the identification information is inserted as a stream type into a video elementary stream loop arranged corresponding to the predetermined number of video streams under a program map table.
(8) The image encoding unit
The transmission device according to (6) or (7), wherein encoding is performed so that at least the decoding intervals of the encoded image data of the pictures in the lowest hierarchy set are equal intervals.
(9) The image encoding unit
The decoding timing of the encoded image data of the pictures of the hierarchical group positioned higher than the lowest hierarchical group is intermediate between the decoding timings of the encoded image data of the pictures of all the hierarchical groups positioned lower than the hierarchical group. The transmission apparatus according to (8), wherein encoding is performed so that timing is reached.
(10) The image data of each picture constituting the moving image data is classified into a plurality of hierarchies, the image data of the classified pictures of each hierarchy is encoded, and the plurality of hierarchies are grouped into a predetermined number of hierarchies. An image encoding unit that divides and generates the predetermined number of video streams respectively having encoded image data of pictures of each divided hierarchical group;
A transmission unit for transmitting a container of a predetermined format including the generated predetermined number of video streams;
A transmission apparatus comprising: a configuration information insertion unit configured to insert configuration information of a video stream corresponding to each of a predetermined number of video streams included in the container in the container layer.
(11) The transmission device according to (10), wherein the configuration information includes information indicating a service group to which the video stream belongs.
(12) The configuration information includes information indicating a dependency relationship between streams starting from a base stream having encoded image data of a picture of the lowest hierarchical set. The information according to (10) or (11) Transmitter device.
(13) The transmission device according to any one of (10) to (12), wherein the configuration information includes information indicating a number of layers of the plurality of layers classified by the image encoding unit.
(14) The container is a transport stream,
The configuration information insertion unit
The configuration information is inserted as a descriptor in a video elementary stream loop arranged corresponding to each of the predetermined number of video streams under the program map table. Any one of (10) to (13) Transmitter.
(15) The image data of each picture constituting the moving image data is classified and encoded into a plurality of layers, and each layer obtained by dividing the plurality of layers into a predetermined number of layer sets A receiving unit for receiving the predetermined number of video streams each having encoded image data of a set of pictures;
A processing unit for processing the received predetermined number of video streams;
A receiving apparatus in which at least a video stream having encoded image data of a picture in the lowest hierarchical group among the predetermined number of video streams is encoded so that a decoding interval of each picture is equal.
(16) The predetermined number of video streams are
The decoding timing of the encoded image data of the pictures of the hierarchical group positioned higher than the lowest hierarchical group is intermediate between the decoding timings of the encoded image data of the pictures of all the hierarchical groups positioned lower than the hierarchical group. The reception apparatus according to (15), wherein the reception apparatus is encoded so as to be timing.
(17) Image data of each picture constituting the moving image data is classified and encoded into a plurality of layers, and each layer obtained by dividing the plurality of layers into a predetermined number of layer sets A receiving unit for receiving a container of a predetermined format including the predetermined number of video streams each having encoded image data of a set of pictures;
From the predetermined number of video streams included in the received container, the encoded image data of a picture of a predetermined hierarchy or lower according to the decoding capability is selectively taken into a buffer, and the code of each picture taken into the buffer is selected. An image decoding unit that decodes the converted image data and obtains image data of a picture of a layer below the predetermined layer,
A receiving apparatus in which at least a video stream having encoded image data of a picture in the lowest hierarchical group among the predetermined number of video streams is encoded so that a decoding interval of each picture is equal.
(18) In the container layer, each of the predetermined number of video streams is a base stream having encoded image data of pictures in the lowest hierarchy set, or is positioned higher than the lowest hierarchy set Identification information for identifying whether the stream is an enhanced stream including encoded image data of a hierarchical set of pictures to be inserted,
Based on the identification information, the image decoding unit fetches and decodes encoded image data of a predetermined layer set of pictures according to the decoding capability from a predetermined number of video streams including the base stream into the buffer. The receiving device according to (17).
(19) The image decoding unit
When the encoded image data of the picture of the predetermined hierarchy set is included in a plurality of video streams, the encoded image data of each picture is decoded as one stream based on the decoding timing information. (17) or ( The receiving device according to 18).
(20) The receiving device according to any one of (17) to (19), further including a post processing unit that adjusts a frame rate of image data of each picture obtained by the image decoding unit to display capability.

  The main feature of the present technology is that at least the reception side includes a plurality of images included in the lowest layer set by performing encoding so that the decoding intervals of the encoded image data of the pictures of the lowest layer set are equal. When there is a decoding capability capable of processing the encoded image data of the pictures in the hierarchy, the decoding processing of the encoded image data of each picture can be performed continuously without difficulty (FIGS. 8 and 11). reference).

  The main feature of the present technology is that by inserting identification information for identifying whether each of a predetermined number of video streams is a base stream or an enhanced stream into the layer of the transport stream TS, By using this identification information on the receiving side, for example, it is possible to easily selectively decode only the base stream (see FIGS. 20 and 23).

  The main feature of the present technology is that the configuration information of the video stream is inserted into the layer of the transport stream TS corresponding to each of a predetermined number of video streams included in the transport stream TS. For each video stream included in the transport stream TS, it is possible to easily understand which group it belongs to, what stream dependency relationship it has, and how many layer encodings the layer number relates to. (See FIGS. 20 and 23).

DESCRIPTION OF SYMBOLS 10 ... Transmission / reception system 100 ... Transmission apparatus 101 ... CPU
102: Encoder 103: Compressed data buffer (cpb)
104: Multiplexer 105: Transmitter 121 ... Temporal ID generator 122 ... Buffer delay controller 123 ... HRD setting unit 124 ... Parameter set / SEI encoder 125 ... Slice encoding 126: NAL packetization unit 142: Section coding unit 143-1 to 143-N: PES packetization unit 144: Switch unit 145: Transport packetization unit 200: Reception Device 201 ... CPU
202: receiving unit 203 ... demultiplexer 204 ... compressed data buffer (cpb)
205: Decoder 206: Uncompressed data buffer (dpb)
207: Post processing unit 231: TS adaptation field extracting unit 232: Clock information extracting unit 233: TS payload extracting unit 234: Section extracting unit 235: PSI table / descriptor extracting unit 236 ... PES packet extractor 237 ... PES header extractor 238 ... Time stamp extractor 240 ... PES payload extractor 241 ... Stream component 251 ... Temporal ID analyzer 252 ... Target Hierarchy Selection Unit 253... Stream Combining Unit 254... Decoding Unit 271... Interpolation Unit 272.

Claims (7)

A first stream and a higher hierarchy that are generated by hierarchically encoding the picture data of each picture constituting the moving picture data so that the decoding order and the display order are different, and having the encoded picture data of the pictures on the lower hierarchy side A receiving unit for receiving a second stream having encoded image data of a side picture;
A processing unit that performs decoding processing on the first stream or both the first stream and the second stream;
The encoded image data has a NAL unit structure, and the level designation value of the first stream is inserted into the SPS NAL unit of the first stream,
The encoded image data of the picture included in the first stream is encoded so that the decoding interval is equal ,
The receiving apparatus, wherein the decoding timing of the encoded image data of the picture included in the second stream is encoded so as to be an intermediate timing of the decoding timing of the encoded image data of the picture included in the first stream . The reception unit includes identification information for identifying that the first stream is a base stream having encoded image data of a lower layer picture, and the second stream is a code of a higher layer picture. The receiving apparatus according to claim 1, further receiving identification information for identifying that the stream is an enhanced stream having digitized image data. The processing unit performs a decoding process on the first stream or both the first stream and the second stream included in the received container based on the identification information according to a decoding capability. The receiving apparatus according to claim 2 , wherein a process of matching a frame rate of image data of each picture obtained by the decoding process with a display capability is performed. A first stream and a higher hierarchy that are generated by hierarchically encoding the picture data of each picture constituting the moving picture data so that the decoding order and the display order are different, and having the encoded picture data of the pictures on the lower hierarchy side Receiving a second stream having encoded image data of a picture on the side;
A procedure for decoding the first stream or both the first stream and the second stream;
The encoded image data has a NAL unit structure, and the level designation value of the first stream is inserted into the SPS NAL unit of the first stream,
The encoded image data of the picture included in the first stream is encoded so that the decoding interval is equal ,
A reception method in which the decoding timing of the encoded image data of the picture of the second stream is encoded so as to be an intermediate timing of the decoding timing of the encoded image data of the picture of the first stream . A first stream and a higher hierarchy that are generated by hierarchically encoding the picture data of each picture constituting the moving picture data so that the decoding order and the display order are different, and having the encoded picture data of the pictures on the lower hierarchy side A transmission unit for transmitting a second stream having encoded image data of a picture on the side,
The encoded image data has a NAL unit structure, and the level designation value of the first stream is inserted into the SPS NAL unit of the first stream,
The encoded image data of the picture included in the first stream is encoded so that the decoding interval is equal ,
The transmitting apparatus, wherein the decoding timing of the encoded image data of the picture included in the second stream is encoded to be an intermediate timing of the decoding timing of the encoded image data of the picture included in the first stream . The transmission unit includes identification information for identifying that the first stream is a base stream having encoded image data of a picture on the lower layer side, and a code of the picture on the higher layer side of the second stream. The transmission apparatus according to claim 5 , further transmitting identification information for identifying the enhanced stream having the converted image data. A first stream and a higher hierarchy that are generated by hierarchically encoding the picture data of each picture constituting the moving picture data so that the decoding order and the display order are different, and having the encoded picture data of the pictures on the lower hierarchy side Transmitting a second stream having encoded image data of a picture on the side,
The encoded image data has a NAL unit structure, and the level designation value of the first stream is inserted into the SPS NAL unit of the first stream,
The encoded image data of the picture included in the first stream is encoded so that the decoding interval is equal ,
The transmission method, wherein the decoding timing of the encoded image data of the picture included in the second stream is encoded so as to be an intermediate timing of the decoding timing of the encoded image data of the picture included in the first stream .

Images