JP2024096139A - Transmission method and reception method - Google Patents

Abstract

To reduce throughput pertaining to the generation of data to be decoded.SOLUTION: A transmission method includes: a dividing step for dividing a picture into a plurality of regions (S101); an encoding step for generating encoded data corresponding to each of the plurality of regions by encoding each of the plurality of regions so as to be independently decodable (S102); a packetizing step for storing a plurality of generated encoded data in a plurality of packets (S103); and a transmission step for transmitting the plurality of packets (S104). Header information of each of the packets includes identification information indicating that the packet stores a portion corresponding to a start of a basic data unit or the packet stores a portion other than the portion corresponding to the start of a basic data unit.SELECTED DRAWING: Figure 8

Description

The present invention relates to a transmission method, a reception method, a transmission device, and a reception device.

As broadcasting and communication services become more advanced, the introduction of ultra-high definition video content such as 8K (7680 x 4320 pixels, hereinafter also referred to as 8K4K) and 4K (3840 x 2160 pixels, hereinafter also referred to as 4K2K) is being considered. A receiving device needs to decode the encoded data of received ultra-high definition video in real time and display it, but video with a resolution such as 8K in particular imposes a large processing load during decoding, making it difficult to decode such video in real time with a single decoder. Therefore, a method is being considered that reduces the processing load per decoder and achieves real-time processing by parallelizing the decoding process using multiple decoders.

The encoded data is multiplexed based on a multiplexing method such as MPEG-2 TS (Transport Stream) or MMT (MPEG Media Transport) before being transmitted. For example, Non-Patent Document 1 discloses a technique for transmitting encoded media data packet by packet according to MMT.

Information technology - High efficiency coding and media delivery in heterogeneous environments - Part 1: MPEG media transport (MMT), ISO/IEC DIS 23008-1

However, since the encoded data is multiplexed based on a multiplexing method such as MPEG-2 TS or MMT before being transmitted, the receiving device must separate the encoded data of the video from the multiplexed data prior to decoding. In the following, the process of separating the encoded data from the multiplexed data is referred to as demultiplexing.

When parallelizing the decoding process, the receiving device needs to distribute the encoded data to be decoded to each decoder. At this time, the receiving device needs to analyze the encoded data itself. In particular, the bit rate of 8K content is very high, so the processing load related to the analysis is large. This causes the demultiplexing process to become a bottleneck, and there are cases where real-time playback is not possible.

The present invention aims to provide a transmission method or a reception method that can reduce the amount of processing involved in generating data to be decrypted.

In order to achieve the above object, a transmission method according to one aspect of the present invention includes a division step of dividing a picture into a plurality of regions, an encoding step of generating encoded data corresponding to each of the plurality of regions by encoding each of the plurality of regions so that each of the plurality of regions can be decoded independently, a packetization step of storing the generated plurality of encoded data in a plurality of packets, and a transmission step of transmitting the plurality of packets, wherein each of the plurality of encoded data is in one-to-one correspondence with a basic data unit which is a unit of data stored in one or more packets, each of the plurality of encoded data is stored in the one or more packets, and header information of each of the packets includes: (1) only the packet in question is included in the basic data unit; (2) the basic data unit includes a plurality of packets; (3) the basic data unit includes a packet and the packet is the first packet of the basic data unit, (4) the basic data unit includes a plurality of packets and the packet is the last packet of the basic data unit, and in the packetization step, control information used for the decoding unit in the picture is stored in a single packet that is different from the plurality of packets in which the plurality of encoded data are stored, and the header information of the packet includes offset information indicating the bit length from the beginning of the encoded data of the picture to the beginning of the encoded data included in the packet.

In addition, a receiving method according to one aspect of the present invention is a receiving method in a receiving device having a plurality of decoding units, and includes a receiving step of receiving a plurality of packets obtained by packetizing a plurality of encoded data obtained by encoding a plurality of areas obtained by dividing a picture so that the plurality of areas can be decoded independently, and a decoding step of the plurality of decoding units decoding the plurality of encoded data in parallel, wherein each of the plurality of encoded data is in one-to-one correspondence with a basic data unit which is a unit of data stored in one or more packets, each of the plurality of encoded data is stored in the one or more packets, and header information of each of the packets includes: (1) only the packet is included in the basic data unit; (2) the basic data unit includes (3) the basic data unit includes multiple packets, and the packet is the first packet of the basic data unit; (4) the basic data unit includes multiple packets, and the packet is the last packet of the basic data unit; and (5) the basic data unit includes multiple packets, and the packet is the last packet of the basic data unit. The control information used for the decoding unit in the picture is stored in a packet that is different from the multiple packets in which the multiple encoded data are stored, and the header information of the packet includes offset information that indicates the bit length from the beginning of the encoded data of the picture to the beginning of the encoded data included in the packet.

These general or specific aspects may be realized by a system, a method, an integrated circuit, a computer program, or a recording medium such as a computer-readable CD-ROM, or may be realized by any combination of a system, a method, an integrated circuit, a computer program, and a recording medium.

The present invention provides a transmission method or a reception method that can reduce the amount of processing involved in generating data to be decrypted.

FIG. 1 is a diagram illustrating an example of dividing a picture into slice segments. FIG. 2 is a diagram showing an example of a PES packet sequence in which picture data is stored. FIG. 2 is a diagram showing an example of dividing a picture according to an embodiment; FIG. 11 is a diagram showing an example of dividing a picture according to a comparative example of the embodiment; FIG. 2 is a diagram showing an example of data of an access unit according to an embodiment. FIG. 2 is a block diagram of a transmission device according to an embodiment. FIG. 2 is a block diagram of a receiving device according to an embodiment. FIG. 2 is a diagram showing an example of an MMT packet according to an embodiment. FIG. 13 is a diagram showing another example of an MMT packet according to an embodiment. 6 is a diagram showing an example of data input to each decoding unit according to the embodiment; FIG. FIG. 2 is a diagram showing an example of an MMT packet and header information according to an embodiment. FIG. 11 is a diagram showing another example of data input to each decoding unit according to the embodiment. FIG. 2 is a diagram showing an example of dividing a picture according to an embodiment; 1 is a flowchart of a transmission method according to an embodiment. FIG. 2 is a block diagram of a receiving device according to an embodiment. 4 is a flowchart of a receiving method according to an embodiment. FIG. 2 is a diagram showing an example of an MMT packet and header information according to an embodiment. FIG. 2 is a diagram showing an example of an MMT packet and header information according to an embodiment.

(Findings on which the present invention is based)
In recent years, the resolution of displays such as TVs, smartphones, and tablet terminals has been increasing. In particular, 8K4K (resolution 8K×4K) services are planned for broadcasting in Japan in 2020. Since it is difficult to decode ultra-high resolution moving images such as 8K4K in real time using a single decoder, a method of performing decoding processing in parallel using multiple decoders has been considered.

In video coding methods such as H.264 and H.265 standardized by MPEG and ITU, a transmitting device divides a picture into multiple areas called slices or slice segments, and can encode each divided area so that it can be decoded independently. Therefore, for example, in the case of H.265, a receiving device that receives a broadcast can separate data for each slice segment from the received data and output the data for each slice segment to a separate decoder, thereby achieving parallel decoding.

Figure 1 shows an example of dividing one picture into four slice segments in HEVC. For example, a receiving device has four decoders, and each decoder decodes one of the four slice segments.

In conventional broadcasting, a transmitting device stores one picture (an access unit in the MPEG system standard) in one PES packet and multiplexes the PES packet into a sequence of TS packets. For this reason, a receiving device had to separate the payload of the PES packet, analyze the data of the access unit stored in the payload, separate each slice segment, and output the data of each separated slice segment to a decoder.

However, the inventors have discovered that there is a problem in that analyzing access unit data and separating slice segments requires a large amount of processing, making it difficult to perform this processing in real time.

Figure 2 shows an example of how picture data divided into slice segments is stored in the payload of a PES packet.

As shown in FIG. 2, for example, data from multiple slice segments (slice segments 1 to 4) is stored in the payload of one PES packet. In addition, the PES packet is multiplexed into a sequence of TS packets.

A transmission method according to one aspect of the present invention includes a division step of dividing a picture into a plurality of regions, an encoding step of generating encoded data corresponding to each of the plurality of regions by encoding each of the plurality of regions so that each of the plurality of regions can be decoded independently, a packetization step of storing the generated plurality of encoded data in a plurality of packets, and a transmission step of transmitting the plurality of packets, in which the packetization step stores the plurality of encoded data in the plurality of packets such that the encoded data corresponding to different regions is not stored in one packet.

With this, the encoded data for each region is stored in a different packet, so the receiving device can determine which region's encoded data the data stored in the packet is from without analyzing the encoded data stored in the packet's payload. This allows the receiving device to perform the process of generating data to be decoded by each decoding section with a small amount of processing. In this way, the amount of processing involved in generating data to be decoded in the receiving device is reduced.

For example, in the packetization step, control information commonly used for all decoding units in the picture may be stored in a packet that is different from the packets in which the multiple encoded data are stored.

This allows the receiving device to determine which packets contain control information without analyzing the encoded data stored in the payload of the packets. This reduces the amount of processing required to generate the data to be decoded in the receiving device.

A receiving method according to one aspect of the present invention is a receiving method in a receiving device having a plurality of decoding units, and includes a receiving step of receiving a plurality of packets obtained by encoding a plurality of regions obtained by dividing a picture so that the plurality of encoded data can be decoded independently, and packetizing the plurality of encoded data of different regions so that the encoded data of the different regions is not stored in a single packet; a combining step of generating a plurality of combined data by combining control information included in any of the plurality of packets and used in common for all decoding units in the picture with each of the plurality of encoded data of the plurality of regions; and a decoding step of the plurality of decoding units decoding the plurality of combined data in parallel.

With this, the encoded data for each region is stored in a different packet, so the receiving device can determine which region's encoded data the data stored in the packet is from without analyzing the encoded data stored in the packet's payload. This allows the receiving device to perform the process of generating data to be decoded by each decoding section with a small amount of processing. In this way, the amount of processing involved in generating data to be decoded in the receiving device is reduced.

For example, the control information may be stored in a packet that is different from the packets in which the encoded data are stored.

This allows the receiving device to determine which packets contain control information without analyzing the encoded data stored in the payload of the packets. This reduces the amount of processing required to generate the data to be decoded in the receiving device.

For example, in the combining step, header information of the packet may be used to determine which of the multiple regions the data stored in the packet is encoded data for.

This allows the receiving device to use the packet's header information to determine which area of encoded data the data stored in the packet belongs to.

For example, each of the multiple encoded data is in one-to-one correspondence with a basic data unit, which is a unit of data stored in one or more packets, and each of the multiple encoded data is stored in the one or more packets, and the header information of each of the packets includes identification information indicating whether (1) the basic data unit includes only the packet, (2) the basic data unit includes multiple packets and the packet is the first packet of the basic data unit, (3) the basic data unit includes multiple packets and the packet is a packet other than the first or last packet of the basic data unit, or (4) the basic data unit includes multiple packets and the packet is the last packet of the basic data unit, and in the combining step, the start of payload data included in the packet having the header information including the identification information indicating that (1) the basic data unit includes only the packet, or (2) the basic data unit includes multiple packets and the packet is the first packet of the basic data unit, may be determined to be the start of the encoded data of each of the areas.

This allows the receiving device to use the packet's header information to determine which area of encoded data the data stored in the packet belongs to.

For example, the header information of the packet may further include offset information indicating the bit length from the beginning of the encoded data of the picture including the multiple encoded data to the beginning of the encoded data included in the packet, and in the combining step, the beginning of the payload data included in the packet having the header information including the identification information indicating that (1) the basic data unit includes only the packet, or (2) the basic data unit includes multiple packets and the packet is the first packet of the basic data unit, and the offset information indicating the bit length that is not zero, may be determined to be the beginning of the encoded data for each region.

This allows the receiving device to use the packet's header information to determine which area of encoded data the data stored in the packet belongs to.

For example, the receiving method may further include a determination step of determining the decoding unit that will decode each of the multiple combined data based on at least one of the resolution of the picture, the method of dividing the picture into the multiple regions, and the processing capabilities of the multiple decoding units.

This allows the receiving device to appropriately allocate the encoded data for each region to multiple decoding units.

A transmitting device according to one aspect of the present invention includes a division unit that divides a picture into a plurality of regions, an encoding unit that generates encoded data corresponding to each of the plurality of regions by encoding each of the plurality of regions so that each of the plurality of regions can be decoded independently, a packetization unit that stores the generated plurality of encoded data in a plurality of packets, and a transmitting unit that transmits the plurality of packets, and the packetization unit stores the plurality of encoded data in the plurality of packets such that the encoded data corresponding to different regions is not stored in one packet.

With this, the encoded data for each region is stored in a different packet, so the receiving device can determine which region's encoded data the data stored in the packet is from without analyzing the encoded data stored in the packet's payload. This allows the receiving device to perform the process of generating data to be decoded by each decoding section with a small amount of processing. In this way, the amount of processing involved in generating data to be decoded in the receiving device is reduced.

A receiving device according to one aspect of the present invention includes a receiving unit that receives multiple packets obtained by encoding multiple regions obtained by dividing a picture so that the multiple regions can be decoded independently, and packetizing the multiple encoded data obtained by the division so that the encoded data of different regions is not stored in a single packet; a combining unit that generates multiple combined data by combining control information included in any of the multiple packets and commonly used for all decoding units in the picture with each of the multiple encoded data of the multiple regions; and multiple decoding units that decode the multiple combined data in parallel.

With this, the encoded data for each region is stored in a different packet, so the receiving device can determine which region's encoded data the data stored in the packet is from without analyzing the encoded data stored in the packet's payload. This allows the receiving device to perform the process of generating data to be decoded by each decoding section with a small amount of processing. In this way, the amount of processing involved in generating data to be decoded in the receiving device is reduced.

These comprehensive or specific aspects may be realized as a system, a method, an integrated circuit, a computer program, or a computer-readable recording medium such as a CD-ROM, or may be realized as any combination of a system, a method, an integrated circuit, a computer program, and a recording medium.

The following describes the embodiment in detail with reference to the drawings.

The embodiments described below each show a specific example of the present invention. The numerical values, shapes, materials, components, component placement and connection forms, steps, and order of steps shown in the following embodiments are merely examples and are not intended to limit the present invention. Furthermore, among the components in the following embodiments, components that are not described in an independent claim that shows the highest concept are described as optional components.

(Embodiment)
In the following, an example will be described in which H.265 is used as the video encoding method, but this embodiment can also be applied to cases in which other encoding methods such as H.264 are used.

Figure 3 shows an example of dividing an access unit (picture) into division units in this embodiment. The access unit is divided into two equal parts horizontally and vertically, into a total of four tiles, by a function called tiles introduced by H.265. Also, there is a one-to-one correspondence between slice segments and tiles.

The reason for dividing the data into two equal parts horizontally and vertically will be explained below. First, when decoding, a line memory is generally required to store one horizontal line of data, but when it comes to ultra-high resolution such as 8K4K, the horizontal size becomes large, so the size of the line memory increases. When implementing a receiving device, it is desirable to be able to reduce the size of the line memory. In order to reduce the size of the line memory, vertical division is necessary. A data structure called a tile is required for vertical division. For these reasons, tiles are used.

On the other hand, images generally have high correlation in the horizontal direction, so coding efficiency improves when a wider range can be referenced horizontally. Therefore, from the viewpoint of coding efficiency, it is desirable to divide the access unit horizontally.

By dividing the access unit into two equal parts horizontally and vertically, these two characteristics can be achieved simultaneously, and both implementation and coding efficiency can be taken into consideration. If a single decoder is capable of decoding 4K2K video images in real time, then an 8K4K image can be divided into four equal parts, and each slice segment can be divided into 4K2K, allowing the receiving device to decode the 8K4K image in real time.

Next, we will explain why tiles obtained by dividing an access unit in the horizontal and vertical directions correspond one-to-one to slice segments. In H.265, an access unit is composed of multiple units called NAL (Network Adaptation Layer) units.

The payload of a NAL unit stores any of the following: an access unit delimiter that indicates the start position of an access unit; a sequence parameter set (SPS), which is initialization information used commonly by sequence unit during decoding; a picture parameter set (PPS), which is initialization information used commonly within a picture during decoding; supplemental enhancement information (SEI), which is not required for the decoding process itself but is required for processing and displaying the decoding results; and encoded data of a slice segment. The header of the NAL unit contains type information for identifying the data stored in the payload.

When the transmitting device multiplexes the encoded data using a multiplexing format such as MPEG-2 TS, MMT (MPEG Media Transport), MPEG DASH (Dynamic Adaptive Streaming over HTTP), or RTP (Real-time Transport Protocol), the basic unit can be set to the NAL unit. In order to store one slice segment in one NAL unit, it is desirable to divide the access unit into slice segment units when dividing the access unit into regions. For this reason, the transmitting device associates tiles with slice segments in a one-to-one relationship.

As shown in FIG. 4, the transmitting device can also set tiles 1 to 4 together as one slice segment. In this case, however, all tiles are stored in one NAL unit, making it difficult for the receiving device to separate the tiles in the multiplexing layer.

Note that there are two types of slice segments: independent slice segments that can be decoded independently, and reference slice segments that reference independent slice segments. Here, we will explain the case where independent slice segments are used.

Figure 5 is a diagram showing an example of data of an access unit that has been divided so that the boundaries between tiles and slice segments coincide as shown in Figure 3. The data of the access unit includes a NAL unit in which an access unit delimiter is stored, which is placed at the beginning, followed by NAL units of SPS, PPS, and SEI, and data of a slice segment in which data from tiles 1 to 4 is stored, which is placed after that. Note that the data of the access unit does not have to include some or all of the NAL units of SPS, PPS, and SEI.

Next, the configuration of the transmitting device 100 according to this embodiment will be described. FIG. 6 is a block diagram showing an example of the configuration of the transmitting device 100 according to this embodiment. This transmitting device 100 includes an encoding unit 101, a multiplexing unit 102, a modulation unit 103, and a transmitting unit 104.

The encoding unit 101 generates encoded data by encoding the input image, for example, according to H.265. In addition, the encoding unit 101 divides an access unit into four slice segments (tiles) and encodes each slice segment, for example, as shown in FIG. 3.

The multiplexing unit 102 multiplexes the encoded data generated by the encoding unit 101. The modulation unit 103 modulates the data obtained by the multiplexing. The transmission unit 104 transmits the modulated data as a broadcast signal.

Next, the configuration of the receiving device 200 according to this embodiment will be described. FIG. 7 is a block diagram showing an example of the configuration of the receiving device 200 according to this embodiment. This receiving device 200 includes a tuner 201, a demodulation unit 202, a demultiplexing unit 203, a plurality of decoding units 204A to 204D, and a display unit 205.

The tuner 201 receives a broadcast signal. The demodulation unit 202 demodulates the received broadcast signal. The demodulated data is input to the demultiplexing unit 203.

The demultiplexer 203 separates the demodulated data into division units, and outputs the data for each division unit to the decoders 204A to 204D. Here, a division unit is a divided area obtained by dividing an access unit, for example, a slice segment in H.265. Also, here, an 8K4K image is divided into four 4K2K images. Therefore, there are four decoders 204A to 204D.

The multiple decoding units 204A to 204D operate in synchronization with one another based on a predetermined reference clock. Each decoding unit decodes the coded data of the division unit according to the DTS (Decoding Time Stamp) of the access unit, and outputs the decoding result to the display unit 205.

The display unit 205 generates an 8K4K output image by integrating the multiple decoding results output from the multiple decoding units 204A to 204D. The display unit 205 displays the generated output image according to the PTS (Presentation Time Stamp) of the access unit obtained separately. When integrating the decoding results, the display unit 205 may perform filtering such as a deblocking filter in boundary areas between adjacent division units, such as tile boundaries, so that the boundaries are visually less noticeable.

In the above, the transmitting device 100 and the receiving device 200 that transmit or receive broadcasts have been described as examples, but content may be transmitted and received via a communication network. When the receiving device 200 receives content via a communication network, the receiving device 200 separates the multiplexed data from IP packets received via a network such as Ethernet.

In broadcasting, the transmission path delay from when the broadcast signal is transmitted until it reaches the receiving device 200 is constant. On the other hand, in a communication network such as the Internet, due to the effects of congestion, the transmission path delay from when the data transmitted from the server reaches the receiving device 200 is not constant. Therefore, the receiving device 200 often does not perform strictly synchronized playback based on a reference clock such as the PCR in the MPEG-2 TS of the broadcast. Therefore, the receiving device 200 may display the 8K4K output image on the display unit according to the PTS, without strictly synchronizing each decoding unit.

In addition, due to congestion in the communication network, etc., the decoding process for all division units may not be completed by the time indicated by the PTS of the access unit. In this case, the receiving device 200 skips displaying the access unit, or delays displaying until decoding of at least four division units is completed and generation of the 8K4K image is completed.

Note that content may be transmitted and received using a combination of broadcasting and communication. This method can also be applied when playing back multiplexed data stored on a recording medium such as a hard disk or memory.

Next, we will explain how to multiplex access units divided into slice segments when MMT is used as the multiplexing method.

Figure 8 shows an example of packetizing data of an HEVC access unit into MMT. SPS, PPS, SEI, etc. do not necessarily need to be included in the access unit, but here we will show an example in which they are present.

NAL units that are located before the first slice segment in an access unit, such as the access unit delimiter, SPS, PPS, and SEI, are stored together in MMT packet #1. Subsequent slice segments are stored in separate MMT packets for each slice segment.

As shown in FIG. 9, a NAL unit that is placed before the first slice segment in an access unit may be stored in the same MMT packet as the first slice segment.

In addition, when a NAL unit such as End-of-Sequence or End-of-Bitstream, which indicates the end of a sequence or stream, is added after the final slice segment, it is stored in the same MMT packet as the final slice segment. However, since NAL units such as End-of-Sequence or End-of-Bitstream are inserted at the end point of the decoding process or the connection point of two streams, it may be desirable for the receiving device 200 to easily obtain these NAL units in the multiplexing layer. In this case, these NAL units may be stored in a different MMT packet from the slice segments. This allows the receiving device 200 to easily separate these NAL units in the multiplexing layer.

Note that TS, DASH, RTP, or the like may be used as the multiplexing method. In these methods, the transmitting device 100 stores different slice segments in different packets. This ensures that the receiving device 200 can separate the slice segments at the multiplexing layer.

For example, when TS is used, the encoded data is packetized into PES packets on a slice segment basis. When RTP is used, the encoded data is packetized into RTP packets on a slice segment basis. Even in these cases, the NAL units and slice segments that are placed before the slice segments may be packetized separately, as in MMT packet #1 shown in FIG. 8.

When TS is used, the transmitting device 100 indicates the unit of data stored in the PES packet by using a data alignment descriptor, for example. Also, since DASH is a method of downloading MP4 format data units called segments via HTTP, the transmitting device 100 does not packetize the encoded data when transmitting. For this reason, the transmitting device 100 may create subsamples in slice segment units and store information indicating the storage position of the subsamples in the MP4 header so that the receiving device 200 can detect slice segments at the multiplexing layer in the MP4.

The MMT packetization of slice segments is explained in detail below.

As shown in FIG. 8, by packetizing the encoded data, data that is commonly referenced when decoding all slice segments in an access unit, such as SPS and PPS, is stored in MMT packet #1. In this case, receiving device 200 concatenates the payload data of MMT packet #1 with the data of each slice segment, and outputs the obtained data to the decoding unit. In this way, receiving device 200 can easily generate input data for the decoding unit by concatenating the payloads of multiple MMT packets.

Figure 10 is a diagram showing an example in which input data to the decoders 204A to 204D is generated from the MMT packets shown in Figure 8. The demultiplexer 203 generates data necessary for the decoder 204A to decode slice segment 1 by linking the payload data of MMT packet #1 and MMT packet #2. The demultiplexer 203 similarly generates input data for the decoders 204B to 204D. That is, the demultiplexer 203 generates input data for the decoder 204B by linking the payload data of MMT packet #1 and MMT packet #3. The demultiplexer 203 generates input data for the decoder 204C by linking the payload data of MMT packet #1 and MMT packet #4. The demultiplexer 203 generates input data for the decoder 204D by linking the payload data of MMT packet #1 and MMT packet #5.

In addition, the demultiplexer 203 may remove NAL units that are not necessary for the decoding process, such as the access unit delimiter and SEI, from the payload data of MMT packet #1, and separate only the SPS and PPS NAL units that are necessary for the decoding process and add them to the slice segment data.

When the encoded data is packetized as shown in FIG. 9, the demultiplexer 203 outputs MMT packet #1, which includes the first data of the access unit in the multiplexing layer, to the first decoder 204A. The demultiplexer 203 also analyzes the MMT packet, which includes the first data of the access unit in the multiplexing layer, separates the SPS and PPS NAL units, and adds the separated SPS and PPS NAL units to each of the data of the second and subsequent slice segments to generate input data for each of the second and subsequent decoders.

Furthermore, it is desirable that the receiving device 200 can use information contained in the header of the MMT packet to identify the type of data stored in the MMT payload and the index number of the slice segment in the access unit when the slice segment is stored in the payload. Here, the type of data is either pre-slice segment data (the NAL units arranged before the first slice segment in the access unit are collectively referred to as such) or slice segment data. When storing a unit obtained by fragmenting an MPU such as a slice segment in an MMT packet, a mode for storing MFU (Media Fragment Unit) is used. When using this mode, the transmitting device 100 can set, for example, the Data Unit, which is the basic unit of data in the MFU, to a sample (a data unit in MMT, equivalent to an access unit) or a subsample (a unit obtained by dividing a sample).

At this time, the header of the MMT packet includes a field called a fragmentation indicator and a field called a fragment counter.

The Fragmentation indicator indicates whether the data stored in the payload of the MMT packet is a fragmented Data unit, and if so, whether the fragment is the first or last fragment in the Data unit, or a fragment that is neither the first nor the last. In other words, the fragmentation indicator included in the header of a packet is identification information that indicates whether (1) the packet is included in the basic data unit, that is, the data unit, only the packet in question, (2) the data unit is divided and stored in multiple packets, and the packet in question is the first packet of the data unit, (3) the data unit is divided and stored in multiple packets, and the packet in question is a packet other than the first or last packet of the data unit, or (4) the data unit is divided and stored in multiple packets, and the packet in question is the last packet of the data unit.

The fragment counter is an index number that indicates which fragment in the data unit the data stored in the MMT packet corresponds to.

Therefore, by the transmitting device 100 setting the samples in the MMT to Data units and setting the data before the slice segment and each slice segment to fragment units of Data units, the receiving device 200 can identify the type of data stored in the payload using the information contained in the header of the MMT packet. In other words, the demultiplexing unit 203 can generate input data for each of the decoding units 204A to 204D by referring to the header of the MMT packet.

Figure 11 shows an example where a sample is set to a Data unit, and pre-slice segment data and slice segments are packetized as fragments of a Data unit.

The pre-slice segment data and the slice segment are divided into five fragments, fragment #1 to fragment #5. Each fragment is stored in an individual MMT packet. At this time, the values of the Fragmentation indicator and Fragment counter included in the header of the MMT packet are as shown in the figure.

For example, the Fragment indicator is a 2-bit binary value. The Fragment indicator of MMT packet #1, which is the head of the Data unit, the Fragment indicator of MMT packet #5, which is the last packet, and the Fragment indicators of MMT packets #2 to #4, which are packets in between, are each set to a different value. Specifically, the Fragment indicator of MMT packet #1, which is the head of the Data unit, is set to 01, the Fragment indicator of MMT packet #5, which is the last packet, is set to 11, and the Fragment indicators of MMT packets #2 to #4, which are packets in between, are set to 10. If a Data unit contains only one MMT packet, the Fragment indicator is set to 00.

Fragment counter is 4 in MMT packet #1, which is the total number of fragments (5) minus 1, and is decreased by 1 in each of the subsequent packets, until it is 0 in the final MMT packet #5.

Therefore, the receiving device 200 can identify the MMT packet that stores the pre-slice segment data by using either the fragment indicator or the fragment counter. The receiving device 200 can also identify the MMT packet that stores the Nth slice segment by referring to the fragment counter.

The header of the MMT packet also includes a sequence number within the MPU of the Movie Fragment to which the Data unit belongs, a sequence number for the MPU itself, and a sequence number within the Movie Fragment of the sample to which the Data unit belongs. By referring to these, the demultiplexer 203 can uniquely determine the sample to which the Data unit belongs.

Furthermore, since the demultiplexing unit 203 can determine the index number of the fragment within the data unit from the fragment counter, etc., it can uniquely identify the slice segment stored in the fragment even if packet loss occurs. For example, even if the demultiplexing unit 203 is unable to obtain fragment #4 shown in FIG. 11 due to packet loss, it can determine that the fragment received next after fragment #3 is fragment #5, and therefore can correctly output slice segment 4 stored in fragment #5 to decoding unit 204D instead of decoding unit 204C.

When a transmission path that guarantees no packet loss is used, the demultiplexer 203 can process the arriving packets periodically without referring to the header of the MMT packet to determine the type of data stored in the MMT packet or the index number of the slice segment. For example, when an access unit is transmitted by a total of five MMT packets including pre-slice data and four slice segments, the receiving device 200 can obtain the pre-slice data and the data of the four slice segments in sequence by processing the received MMT packets in sequence after determining the pre-slice data of the access unit from which decoding is to be started.

Below, we explain some variations on packetization.

Slice segments do not necessarily have to be divided both horizontally and vertically within the plane of an access unit; as shown in FIG. 1, they may be divided only horizontally or only vertically.

Also, if the access unit is divided only horizontally, tiles do not need to be used.

The number of divisions within an access unit is arbitrary and is not limited to four. However, the area size of slice segments and tiles must be equal to or larger than the lower limit of coding standards such as H.265.

The transmitting device 100 may store identification information indicating the division method within the surface of the access unit in an MMT message or a TS descriptor. For example, information indicating the number of divisions in the horizontal and vertical directions within the surface may be stored. Alternatively, a unique identification information may be assigned to the division method, such as dividing the access unit into two equal parts in the horizontal and vertical directions as shown in FIG. 3, or dividing the access unit into four equal parts in the horizontal direction as shown in FIG. 1. For example, if the access unit is divided as shown in FIG. 3, the identification information indicates mode 1, and if the access unit is divided as shown in FIG. 1, the identification information indicates mode 1.

In addition, information indicating constraints on coding conditions related to the intra-plane division method may be included in the multiplexing layer. For example, information indicating that one slice segment is composed of one tile may be used. Alternatively, information indicating that the reference block when performing motion compensation during decoding of a slice segment or tile is limited to a slice segment or tile at the same position in the screen, or limited to blocks within a predetermined range in an adjacent slice segment may be used.

The transmitting device 100 may also switch whether to divide an access unit into multiple slice segments depending on the resolution of the video. For example, the transmitting device 100 may not perform intra-plane division when the video to be processed has a resolution of 4K2K, but may divide the access unit into four when the video to be processed has a resolution of 8K4K. By predefining the division method for 8K4K video, the receiving device 200 can acquire the resolution of the video to be received, determine whether to perform intra-plane division and the division method, and switch the decoding operation.

Furthermore, the receiving device 200 can detect whether or not there is intra-plane division by referring to the header of the MMT packet. For example, if the access unit is not divided, if the MMT Data unit is set to a sample, the Data unit is not fragmented. Therefore, the receiving device 200 can determine that the access unit is not divided if the value of the Fragmentation counter included in the header of the MMT packet is always zero. Alternatively, the receiving device 200 may detect whether the value of the Fragmentation indicator is always 01. The receiving device 200 can also determine that the access unit is not divided if the value of the Fragmentation indicator is always 01.

The receiving device 200 can also handle cases where the number of divisions within a plane in an access unit does not match the number of decoding units. For example, if the receiving device 200 has two decoding units 204A and 204B that can decode 8K2K encoded data in real time, the demultiplexing unit 203 outputs two of the four slice segments that make up the 8K4K encoded data to the decoding unit 204A.

Figure 12 is a diagram showing an example of operation when MMT packetized data as shown in Figure 8 is input to two decoders 204A and 204B. Here, it is desirable for the receiving device 200 to be able to integrate and output the decoding results of the decoders 204A and 204B as is. Therefore, the demultiplexer 203 selects slice segments to output to each of the decoders 204A and 204B so that the decoding results of each of the decoders 204A and 204B are spatially continuous.

The demultiplexer 203 may select a decoding unit to use depending on the resolution or frame rate of the encoded data of the video. For example, if the receiving device 200 has four 4K2K decoding units, and the resolution of the input image is 8K4K, the receiving device 200 performs the decoding process using all four decoding units. Also, if the resolution of the input image is 4K2K, the receiving device 200 performs the decoding process using only one decoding unit. Alternatively, even if the plane is divided into four, if 8K4K can be decoded in real time by a single decoding unit, the demultiplexer 203 integrates all the division units and outputs them to one decoding unit.

Furthermore, the receiving device 200 may determine the decoding unit to be used taking into consideration the frame rate. For example, if the receiving device 200 has two decoding units with an upper limit of 60 fps for the frame rate that can be decoded in real time when the resolution is 8K4K, there may be cases where 8K4K encoded data at 120 fps is input. In this case, if the plane is composed of four division units, slice segment 1 and slice segment 2 are input to the decoding unit 204A, and slice segment 3 and slice segment 4 are input to the decoding unit 204B, as in the example of FIG. 12. Since each of the decoding units 204A and 204B can decode up to 120 fps in real time if the resolution is 8K2K (half of 8K4K), the decoding process is performed by these two decoding units 204A and 204B.

Even if the resolution and frame rate are the same, the processing amount differs if the profile or level in the encoding method, or the encoding method itself, such as H.264 or H.265, is different. Therefore, the receiving device 200 may select the decoding unit to be used based on this information. Note that, when the receiving device 200 cannot decode all the encoded data received by broadcasting or communication, or when it cannot decode all the slice segments or tiles that constitute the area selected by the user, it may automatically determine the slice segments or tiles that can be decoded within the processing range of the decoding unit. Alternatively, the receiving device 200 may provide a user interface for the user to select the area to be decoded. At this time, the receiving device 200 may display a warning message indicating that all areas cannot be decoded, or may display information indicating the number of areas, slice segments, or tiles that can be decoded.

The above method can also be applied when MMT packets that store slice segments of the same encoded data are transmitted and received using multiple transmission paths, such as broadcasting and communication.

In addition, the transmitting device 100 may perform coding so that the areas of each slice segment overlap in order to make the boundaries between the division units less noticeable. In the example shown in FIG. 13, an 8K4K picture is divided into four slice segments 1 to 4. Each of slice segments 1 to 3 is, for example, 8K x 1.1K, and slice segment 4 is 8K x 1K. Adjacent slice segments also overlap each other. In this way, motion compensation during coding can be efficiently performed at the boundaries when divided into four, as indicated by the dotted lines, improving the image quality of the boundary parts. In this way, image quality degradation at the boundary parts is reduced.

In this case, the display unit 205 cuts out an 8K x 1K area from the 8K x 1.1K area and integrates the resulting area. Note that the transmitting device 100 may separately transmit information indicating whether the slice segments are coded with overlap and the extent of the overlap, by including the information in the multiplexing layer or coded data.

Note that a similar technique can be applied when tiles are used.

The flow of operation of the transmitting device 100 will be described below. Figure 14 is a flowchart showing an example of the operation of the transmitting device 100.

First, the encoding unit 101 divides a picture (access unit) into a plurality of slice segments (tiles), which are a plurality of regions (S101). Next, the encoding unit 101 generates encoded data corresponding to each of the plurality of slice segments by encoding each of the plurality of slice segments so that the slice segments can be decoded independently (S102). Note that the encoding unit 101 may encode the plurality of slice segments using a single encoding unit, or may process the slice segments in parallel using multiple encoding units.

Next, the multiplexing unit 102 multiplexes the multiple encoded data generated by the encoding unit 101 by storing the multiple encoded data in multiple MMT packets (S103). Specifically, as shown in Figs. 8 and 9, the multiplexing unit 102 stores the multiple encoded data in multiple MMT packets so that encoded data corresponding to different slice segments is not stored in one MMT packet. Also, as shown in Fig. 8, the multiplexing unit 102 stores control information commonly used for all decoding units in a picture in MMT packet #1, which is different from the multiple MMT packets #2 to #5 in which the multiple encoded data are stored. Here, the control information includes at least one of an access unit delimiter, an SPS, a PPS, and an SEI.

In addition, the multiplexing unit 102 may store the control information in the same MMT packet as one of the multiple MMT packets in which the multiple encoded data are stored. For example, as shown in FIG. 9, the multiplexing unit 102 may store the control information in the first MMT packet (MMT packet #1 in FIG. 9) of the multiple MMT packets in which the multiple encoded data are stored.

Finally, the transmitting device 100 transmits multiple MMT packets. Specifically, the modulation unit 103 modulates the data obtained by multiplexing, and the transmitting unit 104 transmits the modulated data (S104).

Fig. 15 is a block diagram showing an example of the configuration of the receiving device 200, and is a diagram showing in detail the configuration of the demultiplexing unit 203 and the subsequent stages shown in Fig. 7. As shown in Fig. 15, the receiving device 200 further includes a decoding command unit 206. The demultiplexing unit 203 also includes a type discrimination unit 211, a control information acquisition unit 212, a slice information acquisition unit 213, and a decoded data generation unit 214.

The flow of operation of the receiving device 200 will be described below. Figure 16 is a flowchart showing an example of operation of the receiving device 200. Here, the operation for one access unit is shown. When decoding processing for multiple access units is performed, the processing of this flowchart is repeated.

First, the receiving device 200 receives, for example, multiple packets (MMT packets) generated by the transmitting device 100 (S201).

Next, the type determination unit 211 obtains the type of encoded data stored in the received packet by analyzing the header of the received packet (S202).

Next, the type discrimination unit 211 determines whether the data stored in the received packet is pre-slice segment data or slice segment data based on the type of the acquired encoded data (S203).

If the data stored in the received packet is pre-slice segment data (Yes in S203), the control information acquisition unit 212 acquires the pre-slice segment data of the access unit to be processed from the payload of the received packet, and stores the pre-slice segment data in memory (S204).

On the other hand, if the data stored in the received packet is data of a slice segment (No in S203), the receiving device 200 uses the header information of the received packet to determine which of the multiple areas the data stored in the received packet is encoded data for. Specifically, the slice information acquisition unit 213 analyzes the header of the received packet to acquire the index number Idx of the slice segment stored in the received packet (S205). Specifically, the index number Idx is the index number within the Movie Fragment of the access unit (sample in MMT).

Note that the processing of step S205 may be performed together with step S202.

Next, the decoding data generation unit 214 determines a decoding unit that will decode the slice segment (S206). Specifically, the index number Idx and multiple decoding units are associated in advance, and the decoding data generation unit 214 determines the decoding unit that will decode the slice segment, which corresponds to the index number Idx obtained in step S205.

As described in the example of FIG. 12, the decoded data generation unit 214 may determine the decoding unit to decode the slice segment based on at least one of the resolution of the access unit (picture), the method of dividing the access unit into multiple slice segments (tiles), and the processing capabilities of the multiple decoding units included in the receiving device 200. For example, the decoded data generation unit 214 determines the method of dividing the access unit based on identification information in a descriptor such as an MMT message or a TS section.

Next, the decoded data generation unit 214 generates multiple pieces of input data (combined data) to be input to the multiple decoding units by combining control information, which is included in one of the multiple packets and is used commonly for all decoding units in the picture, with each of the multiple pieces of encoded data for the multiple slice segments. Specifically, the decoded data generation unit 214 acquires slice segment data from the payload of the received packet. The decoded data generation unit 214 generates input data to the decoding unit determined in step S206 by combining the pre-slice segment data stored in memory in step S204 with the acquired slice segment data (S207).

After step S204 or S207, if the data of the received packet is not the final data of the access unit (No in S208), the process from step S201 onwards is performed again. In other words, the above process is repeated until input data for the multiple decoding units 204A to 204D corresponding to all slice segments included in the access unit is generated.

Note that the timing at which a packet is received is not limited to the timing shown in FIG. 16, and multiple packets may be received in advance or sequentially and stored in a memory, etc.

On the other hand, if the data of the received packet is the final data of the access unit (Yes in S208), the decoding command unit 206 outputs the multiple input data generated in step S207 to the corresponding decoding units 204A to 204D (S209).

Next, the multiple decoding units 204A to 204D generate multiple decoded images by decoding the multiple input data in parallel according to the DTS of the access unit (S210).

Finally, the display unit 205 generates a display image by combining the multiple decoded images generated by the multiple decoding units 204A to 204D, and displays the display image according to the PTS of the access unit (S211).

The receiving device 200 obtains the DTS and PTS of the access unit by analyzing the header information of the MPU or the payload data of the MMT packet that stores the header information of the Movie Fragment. When TS is used as the multiplexing method, the receiving device 200 obtains the DTS and PTS of the access unit from the header of the PES packet. When RTP is used as the multiplexing method, the receiving device 200 obtains the DTS and PTS of the access unit from the header of the RTP packet.

When integrating the decoding results of the multiple decoding units, the display unit 205 may perform a filter process such as a deblocking filter at the boundary between adjacent division units. Since the filter process is not necessary when displaying the decoding results of a single decoding unit, the display unit 205 may switch the process depending on whether or not to perform a filter process at the boundary between the decoding results of the multiple decoding units. Whether or not the filter process is necessary may be specified in advance depending on the presence or absence of division. Alternatively, information indicating whether or not the filter process is necessary may be stored separately in the multiplexing layer. Information necessary for the filter process, such as a filter coefficient, may be stored in the SPS, PPS, SEI, or slice segment. The decoding units 204A to 204D or the demultiplexing unit 203 acquire this information by analyzing the SEI, and output the acquired information to the display unit 205. The display unit 205 performs a filter process using this information. When this information is stored in the slice segment, it is preferable that the decoding units 204A to 204D acquire this information.

In the above explanation, an example was given in which the types of data stored in a fragment are two types: pre-slice segment data and slice segments. However, the types of data may be three or more. In this case, a case distinction is made in step S203 according to the type.

In addition, when the data size of a slice segment is large, the transmitting device 100 may fragment the slice segment and store it in the MMT packet. In other words, the transmitting device 100 may fragment the pre-slice-segment data and the slice segment. In this case, if the access unit and the data unit are set to be equal as in the packetization example shown in FIG. 11, the following problem occurs.

For example, if slice segment 1 is divided into three fragments, slice segment 1 is divided into three packets with Fragment counter values of 1 to 3 and transmitted. Furthermore, from slice segment 2 onwards, the Fragment counter value becomes 4 or more, and it becomes impossible to associate the Fragment counter value with the data stored in the payload. Therefore, the receiving device 200 cannot identify the packet that stores the first data of the slice segment from the information in the header of the MMT packet.

In such a case, the receiving device 200 may analyze the data in the payload of the MMT packet to identify the start position of the slice segment. Here, there are two types of formats for storing NAL units in the multiplexing layer in H.264 or H.265: a format called a byte stream format in which a start code consisting of a specific bit string is added immediately before the NAL unit header, and a format called a NAL size format in which a field indicating the size of the NAL unit is added.

The byte stream format is used in MPEG-2 systems and RTP, etc. The NAL size format is used in MP4, as well as DASH and MMT, which use MP4, etc.

When a byte stream format is used, the receiving device 200 analyzes whether the first data of the packet matches the start code. If the first data of the packet matches the start code, the receiving device 200 can detect whether the data contained in the packet is slice segment data by obtaining the type of NAL unit from the following NAL unit header.

On the other hand, in the case of the NAL size format, the receiving device 200 cannot detect the start position of the NAL unit based on the bit string. Therefore, in order to obtain the start position of the NAL unit, the receiving device 200 needs to shift the pointer by reading data by the size of the NAL unit, starting from the first NAL unit of the access unit.

However, if the size of the subsample unit is indicated in the header of an MPU or Movie Fragment in MMT, and the subsample corresponds to pre-slice data or a slice segment, the receiving device 200 can identify the start position of each NAL unit based on the size information of the subsample. Therefore, the transmitting device 100 may include information indicating whether or not information of the subsample unit is present in an MPU or Movie Fragment in information that the receiving device 200 acquires when it starts receiving data, such as an MPT in MMT.

The MPU data is an extension of the MP4 format. In MP4, there are modes in which parameter sets such as SPS and PPS of H.264 or H.265 can be stored as sample data, and modes in which they cannot be stored. Information for identifying this mode is indicated as the entry name of SampleEntry. When a mode in which it can be stored is used and the parameter set is included in the sample, the receiving device 200 acquires the parameter set by the method described above.

On the other hand, when a mode that cannot store the parameter set is used, the parameter set is stored as Decoder Specific Information in the SampleEntry, or is stored using a stream for the parameter set. Here, since the stream for the parameter set is not generally used, it is preferable that the transmitting device 100 stores the parameter set in the Decoder Specific Information. In this case, the receiving device 200 analyzes the SampleEntry transmitted as metadata of the MPU or metadata of the Movie Fragment in the MMT packet to obtain the parameter set referenced by the access unit.

When the parameter set is stored as sample data, the receiving device 200 can obtain the parameter set required for decoding by referring only to the sample data without referring to the SampleEntry. In this case, the transmitting device 100 does not need to store the parameter set in the SampleEntry. In this way, the transmitting device 100 can use the same SampleEntry in different MPUs, thereby reducing the processing load on the transmitting device 100 when generating an MPU. Another advantage is that the receiving device 200 does not need to refer to the parameter set in the SampleEntry.

Alternatively, the transmitting device 100 may store one default parameter set in SampleEntry, and store the parameter set referenced by the access unit in the sample data. In conventional MP4, it was common to store parameter sets in SampleEntry, so there is a possibility that a receiving device may stop playback if a parameter set does not exist in SampleEntry. This problem can be solved by using the above method.

Alternatively, the transmitting device 100 may store a parameter set in the sample data only when a parameter set different from the default parameter set is used.

In addition, since it is possible to store parameter sets in SampleEntry in both modes, the transmitting device 100 may always store parameter sets in VisualSampleEntry, and the receiving device 200 may always obtain parameter sets from VisualSampleEntry.

In the MMT standard, the header information of MP4 such as Moov and Moof is called MPU meta, but the transmitting device 100 does not necessarily have to transmit MPU meta. Furthermore, the receiving device 200 can also determine whether SPS and PPS are stored in the sample data based on the service of the ARIB (Association of Radio Industries and Businesses) standard, the type of asset, or whether MPU meta is transmitted.

Figure 17 shows an example in which pre-slice segment data and each slice segment are set to different Data units.

In the example shown in FIG. 17, the data sizes of the data before the slice segment and slice segments 1 to 4 are Length #1 to Length #5, respectively. The field values of the Fragmentation indicator, Fragment counter, and Offset included in the header of the MMT packet are as shown in the figure.

Here, Offset is offset information indicating the bit length (offset) from the beginning of the encoded data of the sample (access unit or picture) to which the payload data belongs to to the first byte of the payload data (encoded data) included in the MMT packet. Note that, although the value of the Fragment counter is described as starting from a value obtained by subtracting 1 from the total number of fragments, it may start from another value.

Figure 18 is a diagram showing an example of when a Data unit is fragmented. In the example shown in Figure 18, slice segment 1 is divided into three fragments, which are stored in MMT packets #2 to #4, respectively. In this case, if the data size of each fragment is Length #2_1 to Length #2_3, respectively, the values of each field are as shown in the figure.

In this way, when a data unit such as a slice segment is set to Data unit, the start of the access unit and the start of the slice segment can be determined based on the field values of the MMT packet header as follows:

The start of the payload in a packet with an Offset value of 0 is the start of the access unit.

The start of the payload of a packet whose Offset value is different from 0 and whose Fragmentation indicator value is 00 or 01 is the start of a slice segment.

In addition, if fragmentation of the data unit does not occur and no packet loss occurs, the receiving device 200 can identify the index number of the slice segment to be stored in the MMT packet based on the number of slice segments obtained after detecting the beginning of the access unit.

Furthermore, even if the data unit of the data before the slice segment is fragmented, the receiving device 200 can similarly detect the beginning of the access unit and the slice segment.

In addition, even if packet loss occurs or the SPS, PPS, and SEI included in the slice segment pre-data are set to different data units, the receiving device 200 can identify the MMT packet that stores the first data of the slice segment based on the analysis result of the MMT header, and then analyze the slice segment header to identify the start position of the slice segment or tile within the picture (access unit). The amount of processing involved in analyzing the slice header is small, and the processing load is not a problem.

In this way, each of the multiple coded data of the multiple slice segments is in one-to-one correspondence with a basic data unit (Data unit), which is a unit of data stored in one or more packets. In addition, each of the multiple coded data is stored in one or more MMT packets.

The header information of each MMT packet includes a Fragmentation indicator (identification information) and an Offset (offset information).

The receiving device 200 determines that the start of payload data included in a packet having header information including a Fragmentation indicator whose value is 00 or 01 is the start of the encoded data of each slice segment. Specifically, the receiving device 200 determines that the start of payload data included in a packet having header information including an Offset whose value is not 0 and a Fragmentation indicator whose value is 00 or 01 is the start of the encoded data of each slice segment.

In the example of FIG. 17, the beginning of a Data unit is either the beginning of an access unit or the beginning of a slice segment, and the value of the Fragmentation indicator is 00 or 01. Furthermore, the receiving device 200 can detect the beginning of an access unit or the beginning of a slice segment without referring to the Offset by referring to the type of the NAL unit and determining whether the beginning of a Data Unit is an access unit delimiter or a slice segment.

In this way, the transmitting device 100 performs packetization so that the beginning of the NAL unit always starts from the beginning of the payload of the MMT packet. This allows the receiving device 200 to detect the beginning of the access unit or slice segment by analyzing the Fragmentation indicator and the NAL unit header, including cases where the data before the slice segment is divided into multiple Data units. The type of the NAL unit is present in the first byte of the NAL unit header. Therefore, when analyzing the header portion of the MMT packet, the receiving device 200 can obtain the type of the NAL unit by analyzing an additional byte of data. In the case of audio, the receiving device 200 only needs to detect the beginning of the access unit, and can make a determination based on whether the value of the Fragmentation indicator is 00 or 01.

As described above, when storing encoded data that has been encoded so that it can be divided and decoded in a PES packet of MPEG-2 TS, the transmitting device 100 can use the data alignment descriptor. Below, an example of a method for storing encoded data in a PES packet is described in detail.

For example, in HEVC, the transmitting device 100 can use the data alignment descriptor to indicate whether the data stored in the PES packet is an access unit, a slice segment, or a tile. The alignment types in HEVC are specified as follows:

Alignment type=8 indicates a HEVC slice segment. Alignment type=9 indicates a HEVC slice segment or access unit. Alignment type=12 indicates a HEVC slice segment or tile.

Therefore, by using type 9, for example, the transmitting device 100 can indicate that the data of the PES packet is either a slice segment or pre-slice segment data. Since a type indicating a slice rather than a slice segment is also separately defined, the transmitting device 100 may use a type indicating a slice rather than a slice segment.

The DTS and PTS included in the header of a PES packet are set only in the PES packet that contains the first data of an access unit. Therefore, if the type is 9 and the PES packet contains a DTS or PTS field, the receiving device 200 can determine that the PES packet stores the entire access unit or the first division unit of the access unit.

The transmitting device 100 may also enable the receiving device 200 to distinguish the data contained in the packet using a field such as transport_priority, which indicates the priority of a TS packet that stores a PES packet that includes the first data of an access unit. The receiving device 200 may also determine the data contained in the packet by analyzing whether the payload of the PES packet is an access unit delimiter. The data_alignment_indicator in the PES packet header indicates whether data is stored in the PES packet according to these types. If this flag (data_alignment_indicator) is set to 1, it is guaranteed that the data stored in the PES packet complies with the type indicated in the data alignment descriptor.

In addition, the transmitting device 100 may use the data alignment descriptor only when PES packetization is performed in units that can be divided and decoded, such as slice segments. This allows the receiving device 200 to determine that the coded data has been PES packetized in units that can be divided and decoded if the data alignment descriptor is present, and to determine that the coded data has been PES packetized in access unit units if the data alignment descriptor is not present. Note that if the data_alignment_indicator is set to 1 and the data alignment descriptor is not present, it is specified in the MPEG-2 TS standard that the unit of PES packetization is the access unit.

If the PMT contains a data alignment descriptor, the receiving device 200 can determine that the PES packetization is performed in units that can be divided and decoded, and can generate input data for each decoding unit based on the packetized units. If the PMT does not contain a data alignment descriptor and the receiving device 200 determines that parallel decoding of the encoded data is necessary based on the program information or information in other descriptors, the receiving device 200 generates input data for each decoding unit by analyzing the slice header of the slice segment. If the encoded data can be decoded by a single decoding unit, the receiving device 200 decodes the data of the entire access unit by the decoding unit. If information indicating whether the encoded data is composed of units that can be divided and decoded, such as slice segments or tiles, is separately indicated by a descriptor in the PMT, the receiving device 200 may determine whether the encoded data can be decoded in parallel based on the analysis result of the descriptor.

In addition, the DTS and PTS included in the header of a PES packet are set only in the PES packet that contains the first data of an access unit, so when an access unit is divided and packetized into PES packets, the second and subsequent PES packets do not contain information indicating the DTS and PTS of the access unit. Therefore, when performing decoding processes in parallel, each of the decoding units 204A-204D and the display unit 205 uses the DTS and PTS stored in the header of the PES packet that contains the first data of the access unit.

The transmitting device, receiving device, transmitting method, and receiving method according to the embodiment have been described above, but the present invention is not limited to this embodiment.

Furthermore, each processing unit included in the transmitting device and receiving device according to the above-mentioned embodiment is typically realized as an LSI, which is an integrated circuit. These may be individually implemented as single chips, or may be integrated into a single chip that includes some or all of them.

In addition, the integrated circuit is not limited to LSI, but may be realized by a dedicated circuit or a general-purpose processor. It is also possible to use an FPGA (Field Programmable Gate Array) that can be programmed after LSI manufacturing, or a reconfigurable processor that can reconfigure the connections and settings of circuit cells inside the LSI.

In each of the above embodiments, each component may be configured with dedicated hardware, or may be realized by executing a software program suitable for each component. Each component may be realized by a program execution unit such as a CPU or processor reading and executing a software program recorded on a recording medium such as a hard disk or semiconductor memory.

In other words, the transmitting device and the receiving device include a processing circuit and a storage device electrically connected to the processing circuit (accessible from the processing circuit). The processing circuit includes at least one of dedicated hardware and a program execution unit. In addition, when the processing circuit includes a program execution unit, the storage device stores a software program to be executed by the program execution unit. The processing circuit uses the storage device to execute the transmitting method or the receiving method according to the above embodiment.

Furthermore, the present invention may be the above-mentioned software program, or a non-transitory computer-readable recording medium on which the above-mentioned program is recorded. Needless to say, the above-mentioned program can be distributed via a transmission medium such as the Internet.

Furthermore, all the numbers used above are merely examples to specifically explain the present invention, and the present invention is not limited to the numbers exemplified.

The division of functional blocks in the block diagram is one example, and multiple functional blocks may be realized as one functional block, one functional block may be divided into multiple blocks, or some functions may be transferred to other functional blocks. In addition, the functions of multiple functional blocks having similar functions may be processed in parallel or in a time-shared manner by a single piece of hardware or software.

The order in which the steps included in the above-mentioned transmission method or reception method are executed is merely an example to specifically explain the present invention, and the steps may be executed in an order other than the above. Also, some of the steps may be executed simultaneously (in parallel) with other steps.

The transmitting device, receiving device, transmitting method, and receiving method according to one or more aspects of the present invention have been described above based on the embodiments, but the present invention is not limited to these embodiments. As long as they do not deviate from the spirit of the present invention, various modifications that a person skilled in the art may make to this embodiment, or forms constructed by combining components of different embodiments, may also be included within the scope of one or more aspects of the present invention.

The present invention can be applied to devices or equipment that transport media such as video data and audio data.

REFERENCE SIGNS LIST 100 Transmitting device 101 Encoding unit 102 Multiplexing unit 103 Modulation unit 104 Transmitting unit 200 Receiving device 201 Tuner 202 Demodulation unit 203 Demultiplexing unit 204A, 204B, 204C, 204D Decoding unit 205 Display unit 206 Decoding command unit 211 Type discrimination unit 212 Control information acquisition unit 213 Slice information acquisition unit 214 Decoded data generation unit

Claims (2)

A segmentation step of segmenting a picture into a number of regions;
an encoding step of generating encoded data corresponding to each of the plurality of regions by encoding each of the plurality of regions so that the encoded data can be decoded independently;
a packetizing step of storing the generated plurality of encoded data in a plurality of packets;
and transmitting the plurality of packets;
each of the plurality of encoded data is associated one-to-one with a basic data unit which is a unit of data stored in one or more packets;
Each of the plurality of encoded data is stored in the one or more packets;
The header information of each of the packets includes identification information indicating whether the packet in question is the first packet of the basic data unit, (1) the basic data unit includes only the packet in question, (2) the basic data unit includes multiple packets, and the packet in question is the first packet of the basic data unit, (3) the basic data unit includes multiple packets, and the packet in question is a packet other than the first or last packet of the basic data unit, or (4) the basic data unit includes multiple packets, and the packet in question is the last packet of the basic data unit,
In the packetizing step, control information used for a decoding unit in the picture is stored in one packet different from a plurality of packets in which the plurality of encoded data are stored;
the header information of the packet includes offset information indicating a bit length from a beginning of the encoded data of the picture to a beginning of the encoded data included in the packet;
Transmission method. A receiving method in a receiving device having a plurality of decoding units,
a receiving step of receiving a plurality of packets obtained by packetizing a plurality of coded data obtained by coding a plurality of areas obtained by dividing a picture so as to be independently decodable;
a decoding step in which the plurality of decoding units decode the plurality of encoded data in parallel,
each of the plurality of encoded data is associated one-to-one with a basic data unit which is a unit of data stored in one or more packets;
Each of the plurality of encoded data is stored in the one or more packets;
The header information of each of the packets includes identification information indicating whether the packet in question is the first packet of the basic data unit, (1) the basic data unit includes only the packet in question, (2) the basic data unit includes multiple packets, and the packet in question is the first packet of the basic data unit, (3) the basic data unit includes multiple packets, and the packet in question is a packet other than the first or last packet of the basic data unit, or (4) the basic data unit includes multiple packets, and the packet in question is the last packet of the basic data unit,
control information used for a decoding unit in the picture is stored in one packet different from the plurality of packets in which the plurality of encoded data are stored;
the header information of the packet includes offset information indicating a bit length from a beginning of the encoded data of the picture to a beginning of the encoded data included in the packet;
Receiving method.

Images